Subject: RE: [Syslog] #3 NUL octets, #4 binary data, #8 octet-counting

From: "Rainer Gerhards" <[email-masked]>
Date: Wed, 30 Nov 2005 10:05:29 +0100

Darren, 

> > I have received notes via private mail telling me there 
> seem to be some
> > existing (and eventually soon upcoming) valid use cases for 
> binary data
> > in syslog. I think there is no point in arguing whether 
> that's fortunate
> > or not. It simply looks like that's the way it is. I do not like the
> > idea of breaking existing use cases for syslog (because 
> that will only
> > lead to implementors ignoring the spec and the story of syslog
> > inconsistencies continues...). As such, I think we need to 
> provide at
> > least some minimal support for it (aka "not outlaw it").
> ..
> 
> This is the wrong approach to take.
> 
> Furthermore, if 3164 is anything to go by, conveying of 
> control characters
> by the existing protocol is not well documented at present.

That is right, RFC 3164 does NOT well document existing behaviour. Maybe
that's why it is titled "the BSD syslog protocol" - just kidding. I have
made considerable effort the past days to see why there is such a bad
mapping between RFC 3164 and reality. In fact, I consider this to be a
major problem that surfaces often during discussions. I myself based
lots of assumptions on RFC 3164 and now testing has shown that these are
invalid. If I only had gone to the lab earlier...

I've read through the full mailing list archive yesterday. I have seen
that there was put a lot of effort in RFC 3164, Chris has even talked
several times with Eric Allman, who unquestionable knows pretty well
about syslog ;) So why this gap between 3164 and practice? I have come
to the conclusion that after Eric invented it, other folks have redone
his work on other platforms (e.g. Linux, Windows and of course embedded
devices). While all of these implementors had Eric's ideas on their
mind, there was no spec to follow. Thus, each one introduced subtle
differences and finally even BSD syslogd was modified in subtle ways. At
the time 3164 was defined, nobody went to the lab and verified the
results. Nor did I when I started with syslog-protocol. We all were so
convinced that when Eric agreed to the document, it must be right. I
think we do not need to put finger at anyone. In fact, 3164 is a fine
piece of work. Even if the format specification can not be considered to
be universally, it describes the security issues inherent with syslog.
As far as I understand, that was the primary goal of that document.

But now that we know practice is different, we should apply that
knowledge to new documents.
> 
> I think it is incumbant upon the WG to convey the message to 
> implementors
> that while it will listen to their needs and plans for 
> implementation, it
> is not up to them to choose which way this group proceeds 
> merely because
> they decide to provide a particular implementation.

In general, I agree. But I think we should support the currently
existing use cases for syslog. After all, was this effort not put on
hold because it was said it is not backwards-compatible enough?

I do not intend to encourage that, that's also why I suggest wording on
the restrictions (like no signatures) it comes with.

> 
> ..
> > Chris proposal for #5 (character encoding) also provides an elegant
> > solution for binary data. We can use something like:
> > 
> > [enc="binary"]
> > 
> > or
> > 
> > [enc="base-64"]
> > 
> > I do NOT intend to specify this - I think it should be in 
> the scope of a
> > separate document specifying the use of binary data. Then 
> would also be
> > the right time to discuss all issues that arise out of it. 
> For now, I
> > just would like to keep the door open.
> > 
> > Finally, I propose to extend Chris format so that the 
> message size can
> > be conveyed. This has been brought up several times and I 
> think a clean
> > solution is now obvious:
> > 
> > [enc="utf-8" lang="en" size="MSG-size-in-octets"]
> > 
> > MSG-size-in-octets would be the size of the MSG part (just that!) in
> > octets. Counting just the MSG part is sufficient, as the rest of the
> > message consists of fields properly delimited. The size is 
> probably most
> > useful for binary data.
> 
> What happens if the message is truncated?

Good question. I'd say the size would need to be adjusted.

> What value is the message size really providing here?
> If we have a natural EOR marker, LF, what do we need the size for?

We do NOT have a EOR marker. As of the current draft, LF is just a
regular character like any other. We explicitely wanted the capability
to surface, now we have it. I even think its not bad to have it. I do
NOT like the idea to re-iterate that long discussion of whether or not
full UTF-8 should be supported. Please review the mailing list archive
for that.

If you violently object, please do so. In this case, I would ask Chris
how we can proceed in such a case. I myself violently object restricting
to printable characters only because of the multiple reasons found in
the archive (and I don't intend to re-iterate that for the 6th time...).
Sorry to be blunt, be if we always re-iterate everything, we will never
arrive somewhere.

> 
> Given that a syslog message is a single record, I don't 
> believe that it
> makes any sense to include a "size" parameter.
> 
> Darren
> 

_______________________________________________
Syslog mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/syslog

---

From: Darren Reed <[email-masked]>
Date: Thu, 1 Dec 2005 03:58:57 +1100 (EST)

..
> So why this gap between 3164 and practice? I have come
> think we do not need to put finger at anyone.
..

I agree with the summary here.

> hold because it was said it is not backwards-compatible enough?

Indeed.  But I think there were (or are) protocol subtlties that
many people are unaware of.

Maybe in your testing of syslog messages you should include sending
a message that is 255 characters long 0x1 - 0xff and see what comes
out the other end.

I'd also recommend that you include Solaris (preferably Solaris 10)
in your test suite if you can't include any of HP-UX or AIX.  It's
probably the most common commercial Unix and people who deploy it
are loathe to replace system components with open source bits.
Solaris 10 for x86 is a free download.

> Good question. I'd say the size would need to be adjusted.

What happens if/when the SD data is truncated and either the MSG size
is lost or truncated half way through ?

I believe there is very little added value of having the message
size as part of SD data - or at least not enough to warrant it
being treated as a mandatory/required field to include.

> full UTF-8 should be supported. Please review the mailing list archive

Allowing LF inside a native message is bad, now that I think of it.

Supporting UTF-8 doesn't impact what we use as an EOR marker unless
we are going to say that the wire format must not be changed from the
message supplied by the application.  If we're going to be backwards
compatible with the use of ^ (and I think we should and continue to
use this) then this already points to the on-wire content of the
message being different to what is supplied by an application.

I can't see any reason why the on-wire format of the message needs
to be the same as that supplied by the application.  It's not like
we use tcpdump or ethereal to do our loging.

Darren