Subject: RE: [Syslog] New direction and proposed charter

From: "Alexander Clemm \(alex\)" <[email-masked]>
Date: Mon, 21 Nov 2005 11:49:51 -0800

Hello Chris,

your proposal sounds good.  I don't disagree with any of your items
(which means, I agree with alll of them:-)   

Concerning the particular questions that you raise:  

"Should we continue to have the MSGID in the header, or should that
become 
an SD-ID element?"  

I believe it will be preferrable to have it in the header, not require
an SD-ID for that.  

"Could we add a language tag as an element in an 
SD-ID to indicate the language of the MSG?"

I believe SD-ID would be the right place to put this one.  A
corresponding SDE could be defined at a later point in time.  

"We need a version.  Should that be in the header (after <PRI>), or as
an 
element in an SD-ID?"

I believe it is preferrable to have it in the header, not require an
SD-ID for that.

--- Alex

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of Chris Lonvick
(clonvick)
Sent: Monday, November 21, 2005 11:12 AM
To: [email-masked]
Subject: [Syslog] New direction and proposed charter

Hi Folks,

I'd like for us to come to closure on some things.  I'm going to be a
bit direct on these questions so we can focus quicker.  We really need
for people to send in responses to see who's listening and involved.

>From the meeting, it sounds like we will get many more implementations 
>if
we continue to use <PRI>... at the start of syslog messages.  This will
allow current receivers to continue to receive messages and put them in
the right bins.  Does anyone disagree with this?

The WG has agreed to use the timestamp Rainer has in the current
syslog-protocol.

Using the FQDN and numeric notations for IPv4 and IPv6 addresses has
been 
agreed to.

Putting in the Structured Data still seems like a great evolutionary
step 
for syslog.  I'd like to see that go into the new syslog-protocol.  Does

anyone disagree with this?

Should we continue to have the MSGID in the header, or should that
become 
an SD-ID element?

We've had slow-downs in our prior discussions about truncation and
message 
length.  To complete this work without revisiting those discussions, and

in the spirit of backwards compliance with traditional syslog, I propose

that we accept the length as Rainer has proposed in syslog-protocol, and

disallow any device to truncate the messages.  At some future time, 
someone can augment the syslog-protocol standard and define a mechanism 
and signalling that will allow truncation.  Does anyone disagree with 
this?

Encoding has been discussed and we have agreed upon US-ASCII and UTF-8
in 
appropriate places.  Could we add a language tag as an element in an 
SD-ID to indicate the language of the MSG?

We need a version.  Should that be in the header (after <PRI>), or as an

element in an SD-ID?

One possibility would be to assemble a syslog message as:

<PRI>TIMESTAMP FQDN VERSION MSGID [SD-ID]s MSG

If we can agree to this then I suspect that we can have a working
document 
within Rainer's timeframe.  I'll propose the following charter to keep
us 
focused.

-------- Proposed Charter  --------------

Syslog is a de-facto standard for logging system events.  However, the
protocol component of this event logging system has not been formally
documented.  While the protocol has been very useful and scalable, it
has some known security problems which were documented in RFC 3164.

The goal of this working group is to address the security and
integrity problems of the existing syslog mechanism while not breaking
backwards compatibility.  The most obvious problems that need to be
addressed in the syslog protocol are the timestamp, which has not
formally included a means to indicate the year, and the identification
of the source which has been a hostname without a qualified domain
name.  Additionally, a version, some type of message indicator, and a 
means to convey structured data will be included in the protocol.

syslog has traditionally been transported over UDP and this WG has 
already defined RFC 3195 for the reliable transport for the syslog 
messages.  The WG will separate the UDP transport from the protocol so 
that others may define additional transports in the future.

- A standard will be produced that formally documents the syslog
protocol.  A mechanism will also be defined in this specification
that will provide a means to convey structured data.

- A standard will be produced that documents the UDP transport for
syslog.

- A standard will be produced that documents a mechanism to sign
syslog messages to provide integrity checking and source
authentication.

- A MIB definition for syslog will be produced.

-------------------------------------------

PLEASE review this and respond.

Thanks,
Chris

_______________________________________________
Syslog mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/syslog

_______________________________________________
Syslog mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/syslog

---

From: "Anton Okmianski \(aokmians\)" <[email-masked]>
Date: Mon, 21 Nov 2005 14:50:20 -0500

Chris and all:

So, to sum up the only change to the syslog-protocol that is proposed is adding <PRI> field for backwards compatibility and removing truncation. Maybe a few other minor changes needed to incorporate the <PRI> field back in, like eliminating redundant SEVERITY field and deciding on whether FACILITY becomes EXTENDED_FACILITY or gets eliminated.   

I drafted a few statements on possible charter, which are pretty much aligned with what you described. See attached.  

So, I agree with the charter you proposed and think we are very close to fulfilling it. 

Answers to specific questions below. 

> [mailto:[email-masked]] On Behalf Of Chris 
> Lonvick (clonvick)
> Sent: Monday, November 21, 2005 2:12 PM
> To: [email-masked]
> I'd like for us to come to closure on some things.  I'm going 
> to be a bit direct on these questions so we can focus 
> quicker.  We really need for people to send in responses to 
> see who's listening and involved.
> 
> 
> 
> we continue to use <PRI>... at the start of syslog messages.  
> This will allow current receivers to continue to receive 
> messages and put them in the right bins.  Does anyone 
> disagree with this?
> 
> 
> The WG has agreed to use the timestamp Rainer has in the 
> current syslog-protocol.
> 
> 
> Using the FQDN and numeric notations for IPv4 and IPv6 
> addresses has been 
> agreed to.
> 
> 
> Putting in the Structured Data still seems like a great 
> evolutionary step 

No disagreement - violent support. 

> Should we continue to have the MSGID in the header, or should 
> that become 
> an SD-ID element?

Support for separate field for MSGID as defined now. 

> We've had slow-downs in our prior discussions about 
> truncation and message 

I agree with this approach. Something like a message signature, for example, can indicate message integrity, including the fact that message was truncated.   

> Encoding has been discussed and we have agreed upon US-ASCII 
> and UTF-8 in 

No problem with that.  However, if we get into a rat-hole regarding various standard tags, we could easily strip that whole setion into a separate document. I would hope we could standardize some very essential fields in syslog-protocol, as current draft attempted to do.  

> <PRI>TIMESTAMP FQDN VERSION MSGID [SD-ID]s MSG

This works for me.  But if we agree that we are breaking timestamp compatability anyway, I would suggest the version be put as a first field.  The key factors for me would ease of parsing and eliminating false-positives.  

> If we can agree to this then I suspect that we can have a 
> working document 
> within Rainer's timeframe.  

What is timeframe?

Thanks,
Anton. 

> I'll propose the following 
> charter to keep us 
> focused.
>

---

From: "Steve Chang \(schang99\)" <[email-masked]>
Date: Mon, 21 Nov 2005 11:56:15 -0800

Please see my inline comments.

> From: [email-masked]
[mailto:[email-masked]]
> On Behalf Of Chris Lonvick (clonvick)
bit
> direct on these questions so we can focus quicker.  We really need for
> people to send in responses to see who's listening and involved.
> 
> 
> 
> >From the meeting, it sounds like we will get many more
implementations if
> the right bins.  Does anyone disagree with this?

I agree. With <PRI> retained, our server code will not be broken.
And it can serve as message delimiter so we can pack as many short
messages before placing it to transport layer delivery.

been
step
become
> an SD-ID element?

I support keeping it in the HEADER since MSGID is a main key to
identifying 
the message, it should not be buried too deep. 

Same argument goes to APPNAME.  I suggest keep it in the HEADER part.
Without the APPNAME in the header, one important piece of information
will
have to find its place in the SD section and make it less obvious to
identify uniquely the message since the MSGID field alone can be a
vendor
specific thing and may not be unique enough for our receiver to
differentiate messages. 

message
in
> We need a version.  Should that be in the header (after <PRI>), or as
an

VERSION is here to help differentiate the format changes.
So, I would suggest place it immediately after <PRI> field.

> <PRI>TIMESTAMP FQDN VERSION MSGID [SD-ID]s MSG

With the APPNAME and VERSION I mentioned above, the format will become:

<PRI> VERSION TIMESTAMP FQDN APPNAME MSGID [SD-ID]s MSG

Of course, let's hear opinions from others.

document
us

---

From: "Rainer Gerhards" <[email-masked]>
Date: Mon, 21 Nov 2005 21:21:35 +0100

Reading the other responses to Chris' question, let me phrase my
response a little differently.

I, too, can agree with the charter. The big question is if we really
feel that strong about backward compatibility. If we think it is vitally
important, we have big constraints, which we MUST take care of. If we
consider backward compatibility non-essential, we can of course use what
was recommended. We just need to know that then we have not really
departed from syslog-protocol - just removed some fields. Of course,
that can be benefitial. But it is not what lead to this discussion.

However, I consider it to be contra-productive to keep <PRI> as this
will trick older recievers into thinking it is something that the
understand. Anyhow, I will do some testing this week with some older
receivers to see how they actually react to the format Chris summed up a
s potential consensus.

Rainer

> [mailto:[email-masked]] On Behalf Of Alexander 
> Clemm (alex)
> Sent: Monday, November 21, 2005 8:50 PM
> To: Chris Lonvick (clonvick); [email-masked]
> Subject: RE: [Syslog] New direction and proposed charter
> 
> Hello Chris,
> 
> implementations 
> This will

---

From: "Anton Okmianski \(aokmians\)" <[email-masked]>
Date: Mon, 21 Nov 2005 15:31:50 -0500

Rainer:

> > in appropriate places.  Could we add a language tag as an 
> element in 
> > an SD-ID to indicate the language of the MSG?
> 
> If so, we should include the *character set* not the 
> language. In respect to existing implementations, that would 
> also be usefule. We should strongly consider to allow (but 
> not recommend) other encodings, too (like popular JIS or 
> EUC). I also posted this in my previous mail.

By character sets, do you suggest the use of the various locale-specific encodings instead of using Unicode with some UTF-8?  

I think that horrible legacy of gazillion local-specific encodings should be avoided at all cost! It is a dead-end. Unicode resolved that issue -- we should stick to it.  I thought this was an accepted direction at IETF.  It is in the industry too.

If I understand correctly, Chris was proposing a mere indication of the language(s) used, which could be useful to the person analyzing the message. I don't think Chris was proposing to do something instead of UTF-8, which covers all of Unicode, which in turn covers all languages. Or did I misinterpret?

Thanks,
Anton

---

From: "Rainer Gerhards" <[email-masked]>
Date: Mon, 21 Nov 2005 21:44:56 +0100

Anton,

Please read my message in the spirit of the question on backwards
compatibility I posted after the initial reply. Sorry for not telling
this together with the initla reply.

I too find it not suitable to support a horrendous number of encodings,
but if we really want backwards compatibility, we must do it - simply
because it is currently done.

All it boils down is how important backwards compatibility is.

As of the language, I am not sure if it will be useful to have this. For
example, what about a message that contains mixed language strings
(english/some local language) is not uncommon. I do not see any benefit
that is worth this trouble.

Rainer 

> From: Anton Okmianski (aokmians) [mailto:[email-masked]] 
> Sent: Monday, November 21, 2005 9:32 PM
> To: Rainer Gerhards; Chris Lonvick (clonvick); [email-masked]
> Cc: [email-masked]
> By character sets, do you suggest the use of the various 
> locale-specific encodings instead of using Unicode with some UTF-8?  
> 
> I think that horrible legacy of gazillion local-specific 
> encodings should be avoided at all cost! It is a dead-end. 
> Unicode resolved that issue -- we should stick to it.  I 
> thought this was an accepted direction at IETF.  It is in the 
> industry too.
> 
> If I understand correctly, Chris was proposing a mere 
> indication of the language(s) used, which could be useful to 
> the person analyzing the message. I don't think Chris was 
> proposing to do something instead of UTF-8, which covers all 
> of Unicode, which in turn covers all languages. Or did I misinterpret?
> 
> Thanks,
> Anton
> 
> 
>

---

From: "Anton Okmianski \(aokmians\)" <[email-masked]>
Date: Mon, 21 Nov 2005 15:57:21 -0500

Rainer:

I agree backwards compatibility is a messy subject.  I think the proposal on the table now is to do just <PRI> with the hope that this allows old receivers to at least process message with appropriate severity/facility. Say long to a different file.  For everything else, all bets are off.  

My experience with Solaris/Linux receivers shows they would accept a message with good PRI and then junk.  They just append their own timestamp. But I agree, this won't guarantee backwards compatibility with any receiver because receivers are not standard at this point. Achieving interop with something that is not standard is a tough problem.  

If people want old <PRI> in there, I am ok with that. But I agree with you, let's not be naïve about what level of interop this would achieve. If that very basic level is useful to people - fine.  But there is no grand goal of full backwards compatibility here. We have to be clear about that in the charter.  

To the specific issue... UTF-8 encoding of basic ASCII is the same 7-bit ASCII code.  So, if new sender only sends a subset of characters allowed by RFC 3164 and encodes them as UTF-8, it will look exactly as old encoding.  So, I think that part is backwards compatible for the subset of characters supported by old receivers.  

Thanks,
Anton. 

> From: Rainer Gerhards [mailto:[email-masked]] 
> Sent: Monday, November 21, 2005 3:45 PM
> To: Anton Okmianski (aokmians); Chris Lonvick (clonvick); 
> [email-masked]
> Please read my message in the spirit of the question on 
> backwards compatibility I posted after the initial reply. 
> Sorry for not telling this together with the initla reply.
> 
> I too find it not suitable to support a horrendous number of 
> encodings, but if we really want backwards compatibility, we 
> must do it - simply because it is currently done.
> 
> As of the language, I am not sure if it will be useful to 
> have this. For example, what about a message that contains 
> mixed language strings (english/some local language) is not 
> uncommon. I do not see any benefit that is worth this trouble.
> 
> Rainer 
> 
> > > If so, we should include the *character set* not the language. In 
> > > respect to existing implementations, that would also be 
> usefule. We 
> > > should strongly consider to allow (but not recommend) other 
> > > encodings, too (like popular JIS or EUC). I also posted 
> this in my 
> > > previous mail.
> > 
> > I think that horrible legacy of gazillion local-specific encodings 
> > should be avoided at all cost! It is a dead-end.
> thought this 
> > was an accepted direction at IETF.  It is in the industry too.
> > 
> > If I understand correctly, Chris was proposing a mere indication of 
> > the language(s) used, which could be useful to the person analyzing 
> > the message. I don't think Chris was proposing to do 
> something instead 
> > of UTF-8, which covers all of Unicode, which in turn covers all 
> > languages. Or did I misinterpret?
> > 
> > Thanks,
> > Anton
> > 
> > 
> > 
>

---

From: "Rainer Gerhards" <[email-masked]>
Date: Mon, 21 Nov 2005 22:14:47 +0100

Chris, Anton, WG:

OK, if we just want to retain the <PRI> and hope that this will lead to
the messages go to the right bin and also hope that the syslogd won't be
broken, this might indeed be a solution. My understanding was that
backwards-compatibility was requested and that would be more than this

I'll do some checks hopefully tomorrow, but think this might be a
workable solution.

If we take that route, the charter should state our very limited
understanding of backwards-compatibility, else this could be another
source of a lot of discussion. This is also the reason I am stressing
this point so much (do not pretend to do things you don't do...).

Rainer

> From: Chris Lonvick [mailto:[email-masked]] 
> Sent: Monday, November 21, 2005 9:58 PM
> To: Rainer Gerhards
> Cc: [email-masked]; [email-masked]
> Hi Rainer and all,
> On Mon, 21 Nov 2005, Rainer Gerhards wrote:
> 
> > Chris & WG
> >
> >>
> > ##############################################################
> >> This will
> >> the right bins.
> > ##############################################################
> >
> >> Does anyone disagree with this?
> >
> > Yes, disagreement here. For the reasons outline in mail from Friday,
> > this is *NOT* true. Existing syslog receivers will be 
> broken if we just
> > stick with <PRI> and do not adjust the other header fields.
> >
> > Of course, it is doable. For details, review
> >
> > http://www.mail-archive.com/syslog%40lists.ietf.org/msg00121.html
> >
> > (around the middle of the post).
> >
> 
> From my perspective, I think that having a syslog receiver 
> receive the 
> message and get it into the right bin is acceptable.  If we 
> don't have 
> that then we are breaking very much.  If we can accomplish that, then 
> receivers will continue to receive the messages and the 
> parsers will have 
> to be updated.
> 
> I believe that the alternative that you're saying, Rainer, is 
> that syslog 
> transmitters COULD keep their existing headers and our 
> Working Group only 
> put things in there.  If we go with that, then the parsers 
> will still need 
> to be updated to understand the SD-ID information.  I'd 
> prefer to just 
> keep the <PRI> and modify the rest of the packet.
> 
> We need to hear from more people on this.
> 
> Thanks,
> Chris
>

---

From: "Steve Chang \(schang99\)" <[email-masked]>
Date: Mon, 21 Nov 2005 14:34:25 -0800

Just FYI.

For discussion on this language mixing stuffs, I am including some
found in the junk email I received. I am sure it happens to other 
non-English world too.

In Chinese world, mixing Chinese and English is quite common in our daily
life reading materials! See example below (I am not doing commercial here.
I am including it to make a point.)

野人花園 - 野人花園精選 搶先聽！( KKBOX新歌搶先聽 - 2005/11/01）
願境網訊股份有限公司 2005 Skysoft. All Rights Reserved. 線上客服信箱　客服專線02-2655-8520

I am not sure when such things will start to be seen in syslog messages though.

If you can not read those characters, perhaps your email reader do not support traditional Chinese character set. You need to install that first.

Regards,

Steve

> From: [email-masked] [mailto:[email-masked]]
> On Behalf Of Rainer Gerhards
> Sent: Monday, November 21, 2005 12:45 PM
> To: Anton Okmianski (aokmians); Chris Lonvick (clonvick); [email-masked]

---

From: "Rainer Gerhards" <[email-masked]>
Date: Tue, 22 Nov 2005 08:25:04 +0100

I fully agree with David. That charter would work well and also cover
keeping the <PRI>. It would make clear that we are not trying to get
full backwards compatibility. 

Rainer 

> [mailto:[email-masked]] On Behalf Of David B Harrington
> Sent: Tuesday, November 22, 2005 12:21 AM
> To: 'Chris Lonvick'; [email-masked]
> Hi,
> I am concerned about the emphasis on backwards compatibility. The
> reason people want a standard is that existing server implementations
> have made different design decisions, and device and application
> vendors are forced to either interoperate with one vendor-specific
> server implementation, or need to write code to support different
> server implementations. Device and application vendors would like one
> consistent solution that is vendor-neutral so their implementations
> will interoperate with any standard-compliant server. 
> 
> If we strive for backwards compatibility, we will end up with a
> standard that simply rubber-stamps many of the non-interoperable,
> vendor-specific existing server implementations.  We should be
> striving to produce a standard for syslog that is vendor-neutral and
> that helps server and equipment/application vendors produce
> interoperable implementations using one standard syslog specification.
> 
> I am, however, a strong believer in making it fairly easy for vendors
> to migrate from their existing vendor-specific non-interoperable
> implementations to a vendor-neutral interoperable standard, and to
> make it possible for servers and senders to support both the existing
> syslog and the new standard for syslog. 
> 
> I believe we should emphasize "ease of migration and co-existence"
> rather than "backwards compatibility".
> 
> I suggest changing 
> "The goal of this working group is to address the security and
> backwards compatibility."
> To
> "The goal of this working group is to address the security and
> integrity problems, and to standardize the syslog protocol, transport,
> and a select set of mechanisms in a manner that considers the ease of
> migration between and the co-existence of existing versions and the
> standard."
> 
> Some charter items could be interpreted as documenting existing
> practices, rather than documenting an agreed-to standardized approach
> for these aspects of syslog.
> 
> I suggest changing
> 
> To
> "
> - A document will be produced that describes a standardized syslog
> protocol.  A mechanism will also be defined in this document
> - A document will be produced that describes a standardized UDP
> transport for syslog.
> 
> - A document will be produced that describes a standardized mechanism
> to sign syslog messages to provide integrity checking and source
> authentication."
> 
> David Harrington
> [email-masked]
> 
> 
> 
> 
> 
>

---

From: "Rainer Gerhards" <[email-masked]>
Date: Tue, 22 Nov 2005 08:28:44 +0100

I personally do not think that this will address the
initernationlization issue, but I do not violently object it. I agree
with Steve Chang that it is redundant information at best.
Multi-language messages are commen, at least e.g. in Non-English
localized syslog messages. So this adds some complexity. However, if we
make it optional, nobody needs to write that tag, which could resolve
the ambiguity. This doesn't hurt the standard at all because the
encoding (UTF-8) is well-specified in any case. So a language tag would
just be another extra information for scripts (whatever) and be
non-essential.

Rainer

> Sent: Monday, November 21, 2005 11:49 PM
> To: Alexander Clemm (alex)
> Cc: Anton Okmianski (aokmians); Rainer Gerhards; 
> [email-masked]; [email-masked]
> I was proposing just a SD-ID element that would indicate the natural 
> language of the message content.  I'd suggest using RFC 3066 language 
> tags.
>      http://www.ietf.org/rfc/rfc3066.txt
> 
> ISO 639-1 and -2 codes for languages may be found here:
>      http://www.loc.gov/standards/iso639-2/englangn.html
> 
> I think this would address the internationalization issue.
> 
> Thanks,
> Chris
> 
> 
> On Mon, 21 Nov 2005, Alexander Clemm (alex) wrote:
> 
> > My understanding was also that Chris' comment relates to  natural
> > language - the language used in the message payload, that possibly a
> > system administrator will interpret - not encoding where my
> > understanding is we are set on UTF-8.  Most systems use 
> English, but for
> > internationalization purposes, it is conceivable to send the same
> > message with a message text in a different - local - 
> language.  Since in
> > general the language will be English, plus the language is easily
> > determined from looking at the message text itself, and in 
> addition all
> > syslog messages emitted by the same sender will tend to use the same
> > language, having a language identifier as part of the 
> header would not
> > be appropriate, but allowing for it as an SD-ID if someone does care
> > about it might make sense.
> >
> > --- Alex
> >
> > [mailto:[email-masked]] On Behalf Of Anton Okmianski
> > (aokmians)
> > Sent: Monday, November 21, 2005 12:32 PM
> >> respect to existing implementations, that would also be usefule. We
> >> should strongly consider to allow (but not recommend) 
> other encodings,
> >
> >> too (like popular JIS or EUC). I also posted this in my 
> previous mail.
> >
> locale-specific
> > encodings instead of using Unicode with some UTF-8?
> >
> > should be avoided at all cost! It is a dead-end. Unicode 
> resolved that
> > issue -- we should stick to it.  I thought this was an accepted
> > direction at IETF.  It is in the industry too.
> >
> indication of the
> > language(s) used, which could be useful to the person analyzing the
> > message. I don't think Chris was proposing to do something 
> instead of
> > UTF-8, which covers all of Unicode, which in turn covers 
> all languages.
> > Or did I misinterpret?
> >
> > Thanks,
> > Anton
> >
> >
> >

---

From: "Anton Okmianski \(aokmians\)" <[email-masked]>
Date: Tue, 22 Nov 2005 12:03:45 -0500

Darren:

> > WG,
> > <PRI>VERSION TIMESTAMP HOSTNAME APP-NAME PROCID [SD-ID]s MSG
> 
> I would put the SD-IDs after the message.
> 
> The SD-IDs and detailed bits of meaning to the MSG and 
> without the MSG, are irrelevant.  The exception being a 
> language marker.

I would prefer SD-ID where it is in example.  I would also re-iterate suggestion of having MSGID in the header, which a number of people supported.  Those two combined are arguably more important than the MSG part itself.  For example (in abbreviated syntax:

host.domain.com MyApp proc1234 STARTED_UP [ip=1.1.1.1]: The applications has started

I could live without the MSG here if it got truncated. The MSGID and SD-ID are much more important in this case.  BTW, the possible truncation of text and its variability (possible substitutable variables) is another reason why MSGID is so useful. It makes it easier for intelligent receivers to do things like event correlation. 

Thanks,
Anton. 

> 
> > - replace NUL with an escape sequence upon reception (e.g. <00>)
> 
> Why not \0 ?
> 
> Darren
>

---

From: "Rainer Gerhards" <[email-masked]>
Date: Tue, 22 Nov 2005 18:21:42 +0100

> > If we go for framing, we must use byte-couting, because we have not
> > outruled any sequence. If we go for octet-stuffing, we must 
> define an
> > escape mechanism. Any of this would be helpful for plain 
> tcp syslog, but
> > that is definitely a big departure from current syslog. 
> Please note that
> > currently many syslogds do octet-stuffing and the message 
> TRAILER is LF.
> 
> That's unfortunate :(

I agree, but that's the way it is in current (non-standard)
implementations.

> 
> In nearly all IETF protocols, the message trailer or EOL 
> marker is CR-LF.

If we go for a very simplistic tcp transport, there is nothing that
hinders us in chaning it to CR-LF. That would also be compatible to
existing receivers, as LF thankfully comes after CR...

Rainer

---

From: "Alexander Clemm \(alex\)" <[email-masked]>
Date: Tue, 22 Nov 2005 09:42:28 -0800

My preference would be to keep structured data before MSG.  This way, it
can effectively be treated as an "extended header".  
--- Alex

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of Darren Reed
Sent: Tuesday, November 22, 2005 8:51 AM
To: Rainer Gerhards
Cc: [email-masked]
Subject: Re: [Syslog] New direction and proposed charter

The SD-IDs and detailed bits of meaning to the MSG and without the MSG,
are irrelevant.  The exception being a language marker.

Why not \0 ?

Darren

---

From: "Steve Chang \(schang99\)" <[email-masked]>
Date: Tue, 22 Nov 2005 09:48:01 -0800

Hi,

I would like to second Anton's view on MSGID and SD-ID position.

In my implementation, APP-NAME, MSGID and Severity (part of PRI) are
used to uniquely identify a message.  With those 3 fields, we can have
very good sense on the nature of the event notification at the receiver
side. Even the message could have been truncated; we can use those
fields to look up what could have been included in the message body. So,
it's OK to live with possibility that some part of the message can be
truncated.   The SD section is used to carries other useful and possible
some critical information which may not be directly related to the MSG
itself. By keeping it where it is, i.e. before MSG (which tends to be
long), will make it less likely to be truncated.  

Thanks,

Steve

> On Behalf Of Anton Okmianski (aokmians)
> Sent: Tuesday, November 22, 2005 9:04 AM
> To: Darren Reed; Rainer Gerhards
> Cc: [email-masked]
> I would prefer SD-ID where it is in example.  I would also re-iterate
> suggestion of having MSGID in the header, which a number of people
> supported.  Those two combined are arguably more important than the
MSG
> part itself.  For example (in abbreviated syntax:
> 
> host.domain.com MyApp proc1234 STARTED_UP [ip=1.1.1.1]: The
applications
> has started
> 
> I could live without the MSG here if it got truncated. The MSGID and
SD-ID
> are much more important in this case.  BTW, the possible truncation of
> text and its variability (possible substitutable variables) is another
> reason why MSGID is so useful. It makes it easier for intelligent
> receivers to do things like event correlation.
> 
> Thanks,
> Anton.
> 
> >

---

From: "Rainer Gerhards" <[email-masked]>
Date: Wed, 23 Nov 2005 09:09:30 +0100

comments inline...

And an important question for the WG close to the end of the message!

Rainer 

> From: Darren Reed [mailto:[email-masked]] 
> Sent: Wednesday, November 23, 2005 6:06 AM
> To: Rainer Gerhards
> Cc: Darren Reed; [email-masked]
> > This raises the question of what terminates the MSG part ;)
> 
> Using the above syntax, how do you distinguish between [] at the start
> of the message from actualy SD-ID data?

I used the "short" syntax Chris used in his proposal. The full ABNF
would be:

 <PRI>VERSION SP TIMESTAMP SP HOSTNAME SP APP-NAME SP PROCID SP [SD-ID]s
SP MSG

This follows the IETF-tradition of SP-delemiting header fields. The
SD-IDs are clearly parsable based on that SP. -protcol-15 has the
details (everything from there would still apply).

> 
> I think what's missing from the above, is a ':' and the syntax should
> be:
> 
> <PRI>VERSION TIMESTAMP HOSTNAME APP-NAME PROCID [SD-ID]: MSG
> 
> The protocol document needs to outlaw ':' being in any field before
> the MSG.
> 
> If you mark "VERSION", "PROCID" and "SD-ID" data as all being optional
> then the format comes back to being very close to what's in use today.

PROCID and SD-ID are optional (but need to have "-" if non-present.
Version, IMHO, should NOT be optional because this will be a live-safer
when we do the next iteration of the protocol in the future. It also
shouldn't be a big deal to stick a "1" character in that position...

> 
> > That would
> > mean we would need to introduce byte-counting, at least I think so.
> 
> Well, without the ':' to say where the MSG starts, I'd have argued
> "How do you tell where SD-ID ends and MSG starts?" vs there just being
> a string of bad SD-ID data following some good SD-ID data.

If we have bad SD-ID data, than we have a bad message. I do not think
that it is appropriate to try to process bad data - there has been too
many security vulnerabilities because of this in recent days...

> 
> As for "but the SD has important information and the MSD does not",
> that's simply a matter of how you structure the message.

I agree on that. Trunction is always bad..

> > That's another good choice.
> 
> It's also how data gets escaped, in general, in Internet stuff.
> 
> > That was my main message. Is it better to live
> > with that or introduce a CLR on not allowing NUL?
> 
> I'd like to see NUL outlawed from messages.

That would indeed solve a number of issues. The last time we discussed
this, it was consensus that this would be a CLR ("crappy little rule")
that we wanted to avoid. In the sense of the current discussion: do we
still think this is true? 

Question to WG: should we outlaw NUL from MSG?

Rainer

---

From: "Rainer Gerhards" <[email-masked]>
Date: Wed, 23 Nov 2005 15:04:02 +0100

Glenn,

very interesting approach with the timestamp. I think your ideas can be
the key to maintaining a lot of backwards compatibility by still
retaining new functionality.

First some bad news: I am not sure if by "BSD syslog" you are refering
to RFC 3164 or a specific distribution of BSD. I have created a small
script to test out your recommendation. I used FreeBSD stock syslogd as
the receiver.

It did NOT work as expected. There are two reasons

a) (that) BSD syslogd takes the sender always from the system that send
it
b) even worse, when relaying, it puts "Forwarded from <hostname>: " into

   the hostname part (yes, with all that spaces)

So while the idea sounds excellent, it does not work with stock FreeBSD
syslogd. I am not sure about other BSD variants, nor have I checked the
sysklogd package. I believe it will have less issues in this regard.
   
This was the message I sent (via perl script):

"<148>Oct 11 22:14:15 mymachine.example.com 1 ID47 2003T.003Z  'su root'
failed for lonvick on /dev/pts/8"

this was the raw message received after being relayed once by FreeBSD
stock syslogd:

"<148>Oct 11 22:14:15 Forwarded from 172.19.2.7: mymachine.example.com 1
ID47 2003T.003Z  'su root' failed for lonvick on /dev/pts/8"

As you can see, the message is somewhat distorted - definitely enough
for digital signatures to be broken. [Implementor's side-note: This can
be fixed on a syslog-application layer level, far beyond the IETF scope.
It's straightforward and easy to do, so it will probably happen.]

Even though this actual sample seems not to work, it paves the way to a
very elegant compatibility solution. The key is to add the extra
information (e.g. Timestamp) in a different place. At least I was so
focussed on fields at whole that I did not notice this possibility. I
have experiemented a bit more with Glenn's proposal, shuffeling some
fields. The result was this:

<PRI>BSD_syslog_timestamp FQDN TAG "@#"VERSION MSGID Remainder_Timestamp
[SD-ID]s MSG

or in an actual sample:

"<148>Oct 11 22:14:15 mymachine.example.com su[4711]: @#1 ID47
2003T.003Z [SD-IDs] 'su root' failed for lonvick on /dev/pts/8"

I have used the BSD timestamp and FAQN as Glenn suggested. Then, I have
added the "TAG" again. If we think in the spirit of my mail this morning
on syslog & non-IETF standards, it would not really hurt if we
standardize TAG instead of two fields. If I would like to retain the
APP-NAME and PROCESSID, I could do the following ABNF:

TAG = APP-NAME ["[" PROCESSID "]"] ":"

The side-effect of this is that almost all syslog-messages currently
emited comply with that format. So I suggest that we strongly consider
joining these two again. Out of the sudden, we have the "old" header,
but it is parsable by a hyptothetical new syslogd. Next, I have used a
trick from syslog-sign. I have changed the VERSION from a number to a
number plus a cookie. The version would now be "@#1". I do not care if
the cookie is "@#" or something else. The key point is that it, together
with the version should be very unlikely to exist at that place in
old-style syslog. That would allow a "new" receiver to differentiate
between old and new style syslog messages. The rest of the message is
just applying Glenn's proposal again: it has the MSG, the missing parts
of the timestamp, SD-IDs and MSG. The Remainder_Timestamp looks strange.
We might like it, we might like something else. That's easy to change
and discuss. It's the concept that matters right now, not the exact
format.

If we take the outlined route, we would be able to extend the syslog
protocol with as much backward compatibility as is possible in a
not-yet-standardized world. I find this very desirable. I think we even
have good chances that many existing "old" syslogds would relay such
messages without changing them, thus keeping digital signatures intact.
The required text changes for syslog-protocol should be moderate.

I strongly propose we go in that direction.

Rainer
> [mailto:[email-masked]] On Behalf Of Glenn 
> Mansfield Keeni
> Sent: Wednesday, November 23, 2005 1:39 PM
> To: [email-masked]
> Chris/Rainer,
> Complete agreement.
> In principle I agree with the timestamp format. It is good.
> I may have missed the discussion on this matter,  in that case please
> accept my apologies and ignore the rest of the mail.
> 
> To get existing BSD syslog devices specifically relays into 
> the compatibility
> fold it WILL be good idea to keep the timestamp in two parts
> 
>        RainerTimestamp =  BSD_syslog_timestamp  + RemainderTimestamp
> 
> In the context of what has been said above about the 
> timestamp. I would
> suggest
>   <PRI>BSD_syslog_timestamp FQDN VERSION MSGID 
> Remainder_Timestamp [SD-ID]s MSG
> 
> That would allow existing BSD-syslog relays to handle the new 
> syslog protocol
> in a transparent manner. Everything from VERSION to the end 
> is treated as "message".
> 
> We do not lose information.
>      The Remainder_timestamp carries it - in a slightly less 
> convenient place
>      though.
> 
> On the other hand if we insist on using RainerTimestamp, 
> existing BSD_syslog
> relays will relay the message as
> 
> <PRI>BSD_syslog_timestamp FQDN RainerTimestamp FQDN VERSION 
> MSGID [SD-ID]s MSG
> 
> The message does get distorted to some extent.
> 
> > document within Rainer's timeframe.  I'll propose the 
> following charter
> > to keep us focused.
> > 
> Glenn

---

From: "Anton Okmianski \(aokmians\)" <[email-masked]>
Date: Wed, 23 Nov 2005 10:28:25 -0500

Rainer, Glenn, all:

I second Rainer's findings. Relays don't work as described on Solaris and Linux syslog daemons either.  Each of them does their own things. Linux comes the closest to the described behavior. Only it inserts its own timestamp and hostname regardless of what you send. Solaris inserts a socket identifier (IP + encoded port). Like this:

Oct 11 22:14:15 [161.44.64.231.14.198] mymachine su: my message

I think backwards compatibility on anything other than <PRI> field is a shaky proposition. This certainly has to be settled as part of charter definition because we have to scope our work clearly this time.   

Thanks,
Anton.   

> [mailto:[email-masked]] On Behalf Of Rainer Gerhards
> Sent: Wednesday, November 23, 2005 9:04 AM
> To: Glenn Mansfield Keeni; [email-masked]
> very interesting approach with the timestamp. I think your 
> ideas can be the key to maintaining a lot of backwards 
> compatibility by still retaining new functionality.
> 
> First some bad news: I am not sure if by "BSD syslog" you are 
> refering to RFC 3164 or a specific distribution of BSD. I 
> have created a small script to test out your recommendation. 
> I used FreeBSD stock syslogd as the receiver.
> 
> a) (that) BSD syslogd takes the sender always from the system 
> that send it
> So while the idea sounds excellent, it does not work with 
> stock FreeBSD syslogd. I am not sure about other BSD 
> variants, nor have I checked the sysklogd package. I believe 
> it will have less issues in this regard.
>    
> this was the raw message received after being relayed once by 
> FreeBSD stock syslogd:
> 
> As you can see, the message is somewhat distorted - 
> definitely enough for digital signatures to be broken. 
> [Implementor's side-note: This can be fixed on a 
> syslog-application layer level, far beyond the IETF scope.
> Even though this actual sample seems not to work, it paves 
> the way to a very elegant compatibility solution. The key is 
> to add the extra information (e.g. Timestamp) in a different 
> place. At least I was so focussed on fields at whole that I 
> did not notice this possibility. I have experiemented a bit 
> more with Glenn's proposal, shuffeling some fields. The 
> result was this:
> 
> <PRI>BSD_syslog_timestamp FQDN TAG "@#"VERSION MSGID 
> "<148>Oct 11 22:14:15 mymachine.example.com su[4711]: @#1 
> ID47 2003T.003Z [SD-IDs] 'su root' failed for lonvick on /dev/pts/8"
> 
> I have used the BSD timestamp and FAQN as Glenn suggested. 
> Then, I have added the "TAG" again. If we think in the spirit 
> of my mail this morning on syslog & non-IETF standards, it 
> would not really hurt if we standardize TAG instead of two 
> fields. If I would like to retain the APP-NAME and PROCESSID, 
> I could do the following ABNF:
> 
> The side-effect of this is that almost all syslog-messages 
> currently emited comply with that format. So I suggest that 
> we strongly consider joining these two again. Out of the 
> sudden, we have the "old" header, but it is parsable by a 
> hyptothetical new syslogd. Next, I have used a trick from 
> syslog-sign. I have changed the VERSION from a number to a 
> number plus a cookie. The version would now be "@#1". I do 
> not care if the cookie is "@#" or something else. The key 
> point is that it, together with the version should be very 
> unlikely to exist at that place in old-style syslog. That 
> would allow a "new" receiver to differentiate between old and 
> new style syslog messages. The rest of the message is just 
> applying Glenn's proposal again: it has the MSG, the missing 
> parts of the timestamp, SD-IDs and MSG. The 
> Remainder_Timestamp looks strange.
> We might like it, we might like something else. That's easy 
> to change and discuss. It's the concept that matters right 
> now, not the exact format.
> 
> If we take the outlined route, we would be able to extend the 
> syslog protocol with as much backward compatibility as is 
> possible in a not-yet-standardized world. I find this very 
> desirable. I think we even have good chances that many 
> existing "old" syslogds would relay such messages without 
> changing them, thus keeping digital signatures intact.
> > [mailto:[email-masked]] On Behalf Of Glenn Mansfield 
> > Keeni
> > To get existing BSD syslog devices specifically relays into the 
> > compatibility fold it WILL be good idea to keep the 
> timestamp in two 
> > parts
> > 
> > In the context of what has been said above about the timestamp. I 
> > would suggest
> >   <PRI>BSD_syslog_timestamp FQDN VERSION MSGID Remainder_Timestamp 
> > [SD-ID]s MSG
> > 
> > That would allow existing BSD-syslog relays to handle the 
> new syslog 
> > protocol in a transparent manner. Everything from VERSION 
> to the end 
> > is treated as "message".
> > 
> > On the other hand if we insist on using RainerTimestamp, existing 
> > BSD_syslog relays will relay the message as
> > 
> > <PRI>BSD_syslog_timestamp FQDN RainerTimestamp FQDN VERSION MSGID 
> > [SD-ID]s MSG
> >

---

From: "Anton Okmianski \(aokmians\)" <[email-masked]>
Date: Wed, 23 Nov 2005 19:01:44 -0500

Andrew:

The \u0000 is a valid Unicode and UTF-8 character. 

See section 1 (first bullet point) in UTF-8 RFC:
http://www.ietf.org/rfc/rfc3629.txt

Also defined as valid in Unicode standard:
http://www.unicode.org/Public/UNIDATA/Blocks.txt

I can't tell what it could possibly be used to indicate now and in the future. 

I can see something like this error message:

 ...MyApp pid123 MALFORMED_INPUT: Bad input received from client: [aaa\u0000]

The question is does not it need to be escaped for transport or only in display.  I'd personally prefer deferring any kind of escaping until it is really needed, like when displaying binary to the user.  

Thanks,
Anton.  

> [mailto:[email-masked]] On Behalf Of Andrew Ross
> Sent: Wednesday, November 23, 2005 6:17 PM
> To: 'Rainer Gerhards'
> Cc: [email-masked]
> >The only tricky issue that remains is the NUL octet. The 
> more I think 
> >about it, the more I think the CLR to disallow it is less 
> evil than to 
> >make it stay...
> 
> I agree that having the CLR for NUL octet exclusion is OK.
> 
> Quick question, if someone is sending international data in 
> UTF-8 format, can there ever be a valid UTF-8 sequence that 
> includes a NUL octet value? If so, we need to allow NUL 
> values in the MSG. 
> 
> Does anyone on the list know UTF-8 well enough to answer this?
> 
> Cheers
> 
> Andrew
> 
>

---

From: "Steve Chang \(schang99\)" <[email-masked]>
Date: Thu, 24 Nov 2005 11:33:07 -0800

Rainer:

OK. Here are a few of my imagined uses of the MSG with mixed text and
binary data.

1.) With the option to include binary data in the MSG, a new set of
diagnostic messages can include some system internal data structure 
	values of concern for analysis. The beginning text in MSG will
state 
	detailed description of the following binary data structure
values.  	With proper tools provided, the binary data structure
values can be
	examined in great details at the receiving end.  

2.) Or the MSG can be used to capture the original incoming packet data 
	detected containing certain pattern. 

3.) We can also include some voice data (binary) in MSG for certain
	situations. 

4.) It can become backup route for SNMP traps because it can now carry
	text and binary data in MSG.

The possible usage is basically up to device vendor's imaginations.  
But if you only allow for text data because of the concern at hand 
over NULL character, then the door is shut.

With potentially rich feature sets to be imagined and to come with this
binary-data-capable MSG, the syslog message receiver will have to be
upgraded to be able to enjoy the benefits. I will leave it to 
sales/marketing people to convince the customers so NULL sensitive 
syslog message receiver will not be in the way.

Thanks,

Steve

> Sent: Thursday, November 24, 2005 3:50 AM
> To: [email-masked]
> Steve,
> no reply, but a question very important to me. What do you consider a
> valid use case for the US-ASCII NUL character inside MSG? If I had a
> good sample, I'd probably have a better time understanding why this is
> important...
> 
> Rainer
> 
> On Thu, 2005-11-24 at 11:48, Steve Chang (schang99) wrote:
> > Rainer:
> >
> > > Sent: Thursday, November 24, 2005 1:25 AM
> > > To: Balazs Scheidler
> > > Cc: [email-masked]
> > > > On Thu, 2005-11-24 at 09:36 +0100, Rainer Gerhards wrote:
> > > > > Anton:
> > > >
> > > > > So I wonder if it wouldn't be wiser to accept that CLR here
> > > > and disallow
> > > > > NUL. After all, I can not see a valid use case for it
> > > > either... (in the
> > > > > sample you provided it honestly believe the sender should
> > > > not send a NUL
> > > > > octet but a "\u0000" string).
> > > > >
> > > >
> > > > \u0000 is a NUL octet at least in its canonical encoding.
> > >
> > > OK, looks like I need to dig up the ASCII table. My fault I didn't
do
> > > in the first place.
> > > Anston's sample was
> > > For simplicity, let me strip the rest and just look at that part
> > >
> > > \u0000
> > >
> > > I think the sender of the sample message should not encode it as
> > >
> > > NUL (0x00)
> > >
> > > but instead as
> > >
> > > 0x5c-0x75-0x30-0x30-0x30-0x30 (\-u-0-0-0-0)
> > >
> > > Encoding it as NUL in an otherwise human-readable message makes no
> > sense
> > > to me.
> > >
> >
> > From the perspective of formatting a piece of device internal raw
data
> > into an outgoing syslog message and considering the CPU cycles
needed to
> > encode a NULL character into some other form to keep the whole MSG
> > intact,
> > I would say prepend the length of MSG (or say, add a fixed 4 digits
byte
> > length of the MSG between SD-IDs and MSG) will be much easier for
the
> > sender.  With this, I think the new IETF syslog compliant receiver
> > should also have easier time fetching in the complete MSG,
regardless
> > it's a plain text, XML-formated, or even binary data.
> > [** side note: a fixed 4 digit MSG length can handle up to 9999
bytes.
> > Big enough for foreseeable future. Also, the first MSG can be
appended
> > with other feature specific information before it gets sent out. And
> > hence updating the final MSG length can be easier.]
> >
> > As to what will happen to the old syslog receivers receiving the
message
> > will NULL character?  The message will probably be cut short right
at
> > reading the NULL character as you have explained. And I don't think
> > there will be a good compromised solution for it.
> >
> > At the sender side, the overhead in checking for a NULL character in
any
> > of the incoming raw message and encode it to other form is just not
> > practical. I would rather not have to pay that overhead if possible.
> > Besides, if mandated, you will be limiting how MSG is to be used.
> >
> > I would say it should be sufficient to just explicitly point out the
> > danger of having the NUL in the MSG part in the new IETF syslog
> > protocol.
> > And you can be kind to add some suggestions as to how the syslog
message
> > initial sending device can choose to do to handle this situation
> > themselves.
> > Vendors should make their own decisions on this issue.
> >
> > With this caveat, we get to keep the flexibility of having the MSG
part
> > in plain text, or in binary for the IETF syslog protocol.
> >
> > It's tradeoff for WG to debate.
> >
> >
> > > I am questioning if there is any legitimate case where an actual
NUL
> > > needs to be part of MSG.
> > >
> > > > UTF-8 allows
> > > > the same character to be encoded in multiple ways, see a
> > > > description on
> > > > the encoding format:
> > > >
> > > > http://www1.tip.nl/~t876506/utf8tbl.html
> > > >
> > > > And I think it would be wise to require the canonical encoding
to be
> > > > used and treat all other encodings of the same characters as
errors.
> > > > this is a large can of worms, lot's of security problems arised
from
> > > > this issue. (see the IIS problems couple of years ago)
> > >
> > > "Shortest possible form" thanksfully is already required in
> > > syslog-protocol-15.
> > >
> > > Rainer
> > >

---

From: "Rainer Gerhards" <[email-masked]>
Date: Thu, 24 Nov 2005 21:40:52 +0100

Steve:

Sorry to be blunt: this is *faaaaaaaaaaar* bejond what we currently try
to acomplish. I guess this should be defined in a smime to syslog
mapping document. I am just a bit concerend about your proposed limit of
9999 octets. I am not sure how much voice will fit into it...

Honestly - this is no payload for syslog. I also see no point in SNMP
backup. 

Some people might find that some of the samples are actual valid use
cases, and I may become convinced over time. But - to say it with Darren
- we have smaller fish to fry today ;)

I also do not find it very consistent to vote in a meeting for backwards
compatibility and then ask on list for a syslog revolution...

Rainer

> From: Steve Chang (schang99) [mailto:[email-masked]] 
> Sent: Thursday, November 24, 2005 8:33 PM
> To: Rainer Gerhards; [email-masked]
> > > >  ...MyApp pid123 MALFORMED_INPUT: Bad input received 
> from client:
> > > > [aaa\u0000]
> > > >

---

From: "Rainer Gerhards" <[email-masked]>
Date: Thu, 24 Nov 2005 23:21:49 +0100

Andrew,

That's exactly our experience. 100% same story...

Rainer 

> From: Andrew Ross [mailto:[email-masked]] 
> Sent: Thursday, November 24, 2005 11:20 PM
> To: Rainer Gerhards
> Cc: [email-masked]
> Rainer,
> I agree that 3164 is only really valid with respect to the 
> <PRI>. When we
> implemented it in Kiwi Syslog we found no device actually 
> used the 3164
> format exactly. Sometimes the hostname was there, sometimes 
> not. Having to
> write parsing code to work out if a hostname was actually a 
> TAG or not was a
> huge processing overhead. We now disable 3164 parsing by default.
> 
> Cheers
> 
> Andrew
> 
> 
>

---

From: "Anton Okmianski \(aokmians\)" <[email-masked]>
Date: Mon, 28 Nov 2005 10:34:34 -0500

For whatever its worth... I ran a test on Solaris.  It syslogd cuts off message at the nul character, but handles the left part of it fine. 

Thanks,
Anton.  

> [mailto:[email-masked]] On Behalf Of Steve 
> Chang (schang99)
> Sent: Thursday, November 24, 2005 5:48 AM
> To: Rainer Gerhards; Balazs Scheidler
> Cc: [email-masked]
> > OK, looks like I need to dig up the ASCII table. My fault I 
> didn't do
> it
> > in the first place.
> > 
> > Anston's sample was
> > 
> >From the perspective of formatting a piece of device 
> internal raw data
> into an outgoing syslog message and considering the CPU 
> cycles needed to encode a NULL character into some other form 
> to keep the whole MSG intact, I would say prepend the length 
> of MSG (or say, add a fixed 4 digits byte length of the MSG 
> between SD-IDs and MSG) will be much easier for the sender.  
> With this, I think the new IETF syslog compliant receiver 
> regardless it's a plain text, XML-formated, or even binary data.
> [** side note: a fixed 4 digit MSG length can handle up to 9999 bytes.
> appended with other feature specific information before it 
> gets sent out. And hence updating the final MSG length can be 
> easier.] 
> 
> As to what will happen to the old syslog receivers receiving 
> the message will NULL character?  The message will probably 
> be cut short right at reading the NULL character as you have 
> explained. And I don't think there will be a good compromised 
> solution for it. 
> 
> At the sender side, the overhead in checking for a NULL 
> character in any of the incoming raw message and encode it to 
> other form is just not practical. I would rather not have to 
> pay that overhead if possible.
> I would say it should be sufficient to just explicitly point 
> out the danger of having the NUL in the MSG part in the new 
> IETF syslog protocol.
> And you can be kind to add some suggestions as to how the 
> syslog message initial sending device can choose to do to 
> handle this situation themselves. 
> With this caveat, we get to keep the flexibility of having 
> the MSG part in plain text, or in binary for the IETF syslog protocol.
> 
> > I am questioning if there is any legitimate case where an 
> actual NUL 
> > needs to be part of MSG.
> > 
> > > UTF-8 allows
> description 
> > > on the encoding format:
> > > And I think it would be wise to require the canonical 
> encoding to be 
> > > used and treat all other encodings of the same characters 
> as errors.
> > > this is a large can of worms, lot's of security problems 
> arised from

---

From: "Anton Okmianski \(aokmians\)" <[email-masked]>
Date: Mon, 28 Nov 2005 10:59:17 -0500

Rainer:

They are valid use-cases.  I believe Cisco IOS logs binary diagnostic messages today, and for good reasons. I am not sure how they are encoded (you can of course encode binary in ASCII). I am not fresh on details, but it seems to be a valid use-case to me. 

Is not it a problem if logging library has to allocate memory (for possible character substitutions) and has to scan the message in search of invalid characters to escape?  When you are running out of memory and are trying to log a critical message, this could be the difference between being able to send it or not.    

Thanks,
Anton.   

> Sent: Thursday, November 24, 2005 3:41 PM
> Cc: [email-masked]
> Sorry to be blunt: this is *faaaaaaaaaaar* bejond what we 
> currently try to acomplish. I guess this should be defined in 
> a smime to syslog mapping document. I am just a bit concerend 
> about your proposed limit of
> Honestly - this is no payload for syslog. I also see no point 
> in SNMP backup. 
> 
> Some people might find that some of the samples are actual 
> valid use cases, and I may become convinced over time. But - 
> to say it with Darren
> I also do not find it very consistent to vote in a meeting 
> for backwards compatibility and then ask on list for a syslog 
> revolution...
> 
> Rainer
> 
> > 	values of concern for analysis. The beginning text in 
> MSG will state
> > With potentially rich feature sets to be imagined and to come with 
> > this binary-data-capable MSG, the syslog message receiver 
> will have to 
> > be upgraded to be able to enjoy the benefits. I will leave it to 
> internal raw
> > > > intact, I would say prepend the length of MSG (or say, add a
> > fixed 4 digits
> > byte
> > character in
> > > > > > description on the encoding format:
> > > > > >

---

From: "Rainer Gerhards" <[email-masked]>
Date: Mon, 28 Nov 2005 19:58:14 +0100

Anton:

If we agree this is a valid use case, then we must reconsider the UTF-8
requirement/recommendation. Because binary data will not result in
proper UTF-8 encoding. I have to admit that I strongly doubt this is
something that should be in syslog-protocol - maybe something for a
later add-on using structured data (base-64 encoded binary
structured-data would be an option)...

Please also check my implementation report on other difficulties with
UTF-8 encoding, only.

Rainer

> Sent: Monday, November 28, 2005 4:59 PM
> To: Rainer Gerhards
> Cc: [email-masked]
> They are valid use-cases.  I believe Cisco IOS logs binary 
> diagnostic messages today, and for good reasons. I am not 
> sure how they are encoded (you can of course encode binary in 
> ASCII). I am not fresh on details, but it seems to be a valid 
> use-case to me. 
> 
> Is not it a problem if logging library has to allocate memory 
> (for possible character substitutions) and has to scan the 
> message in search of invalid characters to escape?  When you 
> are running out of memory and are trying to log a critical 
> message, this could be the difference between being able to 
> send it or not.    
> 
> Thanks,
> Anton.   
> 
> > [email-masked] https://www1.ietf.org/mailman/listinfo/syslog
> > 
>