Subject: RE: [Syslog] byte-counting vs special character

From: "Anton Okmianski \(aokmians\)" <[email-masked]>
Date: Tue, 15 Aug 2006 14:04:43 -0400

I second these concerns.  Escaping requirements result in a more
interdependent layering, which is a weaker architecture (not to mention
the overhead to a new standard). The syslog transport would need to mess
with payload instead of treating it as opaque blob with easily known
length. Not nice. Imagine TCP/IP escaping all payload to separate
datagrams and segments.

Escaping of magic characters is IMHO clearly a hack and should not be
put into a standard! If the argument was to accommodate some real
established standard and there was no way to version things - maybe.
Current syslog is not standardized (not in IETF, not in reality). The
transition to a cleaner design is trivial.  A given implementation can
support whatever legacy compatibility modes it wishes to support. Syslog
is sent to a known destination configured by administrator, not some
kind of broadcast to world-wide-web where you don't know what kind of
receiver will get it. This problem is easily administrated in mixed
legacy/standard environment if someone chooses to deploy one.     

Anton.   

> -----Original Message-----
> From: David Harrington [mailto:[email-masked]] 
> Sent: Tuesday, August 15, 2006 1:25 PM
> To: 'Rainer Gerhards'
> Cc: [email-masked]
> Subject: [Syslog] byte-counting vs special character
> 
> Hi Rainer,
> 
> [speaking as co-chair]
> Can we change the subject line to "byte-counting vs special character"
> for this topic so it is easier to find comments on this topic 
> as compared to other things in the timeline? That will make 
> it nuch easier for the chairs.
> 
> This WG has already gotten stuck, and had WG progress stall, 
> trying to provide backwards compatibility to a bunch of 
> incompatible implementations. I argued on this list before 
> becoming co-chair that backwarsd compatibility may not be 
> achieveable for some features and we need to stop getting 
> hung up on it. Sometimes to build a good standard, you need 
> to choose something that will break some existing implementations. 
> 
> I raised this concern with Chris when I started as co-chair. 
> I do not want to see backwards compatibility arguments stall 
> progress again. I made sure this was reflected in the 
> timeline, which says that by Friday (if you decide to use a 
> special character) you must reach consensus on which special 
> character to use.
> [/speaking as co-chair]
> 
> [speaking as contributor]
> I like the argument that the LF solution will not break 
> existing implementations, but I would like to know this is 
> actually true. Have you actually tested this against multiple 
> implementations, or is it a theoretical argument?
> 
> I know you have tested a number of other proposed ways of 
> doing things against multiple implementations to try to 
> verify backwards compatibility. Have you actually tested 
> multiple existing implementations with the LF and found that 
> they do continue to work without significant problems? Can 
> you tell the WG which ones you have tested? Are there 
> implementations that break when using this solution?
> 
> 
> In a different message you argue that syslog-sign only needs 
> to support -protocol because the implementations will need to 
> have new code added anyway, and the added complexity of 
> backwards compatibility to rfc3164 implementations is not needed.
> 
> You only mention one implementation explicitly that provides 
> syslog/tls, syslog-ng over stunnel. Would all the other 
> implementations need to be modified to support a TLS 
> substrate anyway, so it would not make a difference to them 
> which special character was chosen or if byte-counting was 
> chosen? Which syslog/tls implementations will be impacted by 
> our choice here? 
> 
> Ideally, only the implementations that support a feature need 
> to pay the price for the feature.
> 
> A syslog message delineator will only be needed in 
> implemntations that add support for syslog/tls (or other 
> streams). Whether one supports byte-counting or a special 
> character, only implementations that support multiple 
> messages in a stream, e.g. syslog/tls, should need to be 
> modified to support the choice. 
> 
> Personally I find the octet-counting cleaner because previous 
> discussions on terminators showed that available 
> implementations are highly inconsistent with their special 
> characters. 
> 
> Since byte-counting happens outside the syslog message, and 
> only for syslog in a delimited stream coming in over the 
> syslog/tls port, it would not impact existing code, only code 
> that supports syslog/tls.
> New syslog/tls code would be needed to extract each "normal" 
> syslog message from the stream and then send it on for 
> processing through the existing parser. 
> 
> If message originators start escaping special characters in 
> the message content, existing receivers may need to be 
> modified to un-escape the characters. Think about 
> applications, such as intrusion detection systems, that 
> search for special patterns in syslog content
> - those could be broken by having the content changed by 
> escaping characters in the middle of the content.
> 
> What happens at a store-then-forward relay? Does the relay 
> store the escaped characters or does it un-escape them first? 
> Does it escape them again when it forwards the message? Does 
> it escape them if it un-escaped them on input? Does it escape 
> them if it did not un-escape them on input? What if the 
> sender included some escaped characaters (like '\') for other 
> purposes? Should the relay unescape them as well?
> Will it know to re-escape them on output? What should the 
> ultimate reciever expect to receive?
> 
> Byte-counting looks way less complex to me than choosing a 
> special character and escaping/unescaping special characters 
> in the content during the sending/forwarding/recieving 
> process. (And my opinion has nothing to do with the H**** in 
> my email address.)
> 
> dbh
> 
> > -----Original Message-----
> > From: Rainer Gerhards [mailto:[email-masked]]
> > Sent: Tuesday, August 15, 2006 12:41 AM
> > To: Miao Fuyou
> > Cc: [email-masked]
> > Subject: RE: [Syslog] timeline
> > 
> > Miao,
> > 
> > I am actually concerned about backward compatibility with existing 
> > code
> > *without* the need to upgrade any of that code. As you know,
> deployed
> > software tends to stick.
> > 
> > If we use just LF, existing, deployed technology (e.g. syslog-ng
> with
> > stunnel) would be able to understand a message sent from a "new
> style"
> > syslogd. Having the octet count in front of the message 
> removes that 
> > ability, as the old syslogd will no longer see the <pri> at 
> the start 
> > of the message.
> > 
> > I agree that it is trivial to modify code to take care for 
> the octet 
> > counter. But this is not my concern. My concern is that I 
> would like 
> > to achive as good as possible compatibility with existing deployed 
> > (aka
> > "unmodified") technology. I should have been more specific on that.
> > Sorry for the omission...
> > 
> > I am also unaware of any implementation that mandates CR LF over
> just
> > LF. Could you let me know which ones are these?
> > 
> > Rainer
> > 
> > > -----Original Message-----
> > > From: Miao Fuyou [mailto:[email-masked]]
> > > Sent: Monday, August 14, 2006 7:07 PM
> > > To: Rainer Gerhards
> > > Cc: [email-masked]
> > > Subject: RE: [Syslog] timeline
> > > 
> > >  
> > > Hi, Rainer,
> > > 
> > > A new implementation could rely on byte-counting only and then 
> > > delete LF from the frame(appplication knows exactly where the LF 
> > > is), it may not force us to use escapes. For LF, I think it is 
> > > difficult to get 100% compatibility for a legacy 
> implementation to 
> > > comply TLS-transport without any change to the code. At 
> least, some 
> > > imlementation may need to change
> > CR LF to LF
> > > because some implementations use CR LF rather than LF. 
> So, it may be 
> > > ok to add several LOC to delete FRAME-LEN SP from the frame.
> > > 
> > > I still prefer byte-counting only to byte-counting+LF even
> > if it is a
> > > feasible tradeoff.  
> > > 
> > > Miao
> > > 
> > > > -----Original Message-----
> > > > From: Rainer Gerhards [mailto:[email-masked]]
> > > > Sent: Monday, August 14, 2006 10:18 PM
> > > > To: Miao Fuyou
> > > > Subject: RE: [Syslog] timeline
> > > > 
> > > > We should not go byte-counting + LF. This is the worst choice:
> it 
> > > > 
> > > > A) breaks compatibility
> > > > B) Forces us to use escapes
> > > > 
> > > > So we get the bad of both worlds, without any benefits.
> > > > 
> > > > Rainer
> > > > 
> > > > > -----Original Message-----
> > > > > From: Miao Fuyou [mailto:[email-masked]]
> > > > > Sent: Monday, August 14, 2006 12:58 AM
> > > > > To: 'Anton Okmianski (aokmians)'; 'David Harrington';
> > > > [email-masked]
> > > > > Subject: RE: [Syslog] timeline
> > > > > 
> > > > > 
> > > > > My vote: byte-counting only > byte-counting + LF > LF
> > > >  
> > > > 
> > > 
> > > 
> > > 
> > 
> > _______________________________________________
> > Syslog mailing list
> > [email-masked]
> > https://www1.ietf.org/mailman/listinfo/syslog
> > 
> 
> 
> _______________________________________________
> Syslog mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/syslog
> 

_______________________________________________
Syslog mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/syslog

---

From: Carson Gaspar <[email-masked]>
Date: Tue, 15 Aug 2006 13:50:40 -0700

I've been mostly lurking lately, but I would like to re-iterate my support 
for byte counting. It's the Right Thing To Do, and theoretical "backwards 
compatibility" with existing non-standard non-widely-deployed syslog over 
TCP solutions isn't a good enough reason to put bad ideas in the standard.

---

From: Balazs Scheidler <[email-masked]>
Date: Wed, 16 Aug 2006 12:24:51 +0200

> TCP solutions isn't a good enough reason to put bad ideas in the standard.

Then why on earth did the BEEP based syslog/RAW use LF as line
terminator?

Quoting RFC3195:

   If multiple syslog messages are included in a single ANS reply, each
   is separated from the preceding with a CRLF.  There is no ending
   delimiter, but each syslog event message body length MUST be 1024
   bytes or less, excluding BEEP framing overhead.  Note that there MUST
   NOT be a CRLF between the text of the final syslog event message and
   the "END" marking the trailer of the BEEP frame.

As it currently stands:
  * nonstandard, but widely deployed syslog over TCP use LF
  * nonstandard, less widely deployed syslog-over-TCP-over-SSL use LF
  * standard, even less widely deployed syslog/RAW uses CRLF
  * semi-standard, UDP based syslog uses UDP packet framing for message
termination

Syslog-ng has supported TCP transport since its first release in 1998, a
number of devices interoperate with it. Whatever the decision of the
working group, syslog-ng will need to support this legacy TCP mode of
operation for backward compatibility. And as there is little incentive
for a user to change to the byte-counted form, I think the legacy mode
will still be more widely used for a couple of years to come.

Of course syslog-ng might not be the most widely deployed syslog
implementation, but I'm sure it is in the top three. Some numbers using
google search, which I know is not really representative:

sysklogd    607000 (default on Linux, no TCP support)
syslog-ng   593000
kiwi syslog 321000
msyslog     33400
sdsc syslog 21900
rsyslog     21500

In my opinion we either design something that is really sexy
(application layer acknowledgements, authentication, compression,
something that BEEP based syslog provides but which has other problems)
or we should be as compatible with existing installations as possible.

In my opinion the goal here is that the user will not need to change its
configuration file, or will not need a separate port on her firewalls, a
simple software upgrade on either receiver/sender and be done with it.

The current TCP based protocol _is_ implemented in closed-box devices
where replacing the sender software component is not possible or
feasible, if the new feature is incompatible, it will be difficult to
convince users to change, unless some real value is also added, other
than "byte-counted" form is cleaner. I tend to agree that byte-counting
is cleaner, I just don't see the advantate in the current state of
affairs.

-- 
Bazsi

---

From: "tom.petch" <[email-masked]>
Date: Thu, 17 Aug 2006 12:53:25 +0200

<inline tp>
----- Original Message -----
From: "Anton Okmianski (aokmians)" <[email-masked]>
To: "David Harrington" <[email-masked]>; "Rainer Gerhards"
<[email-masked]>
Cc: <[email-masked]>
Sent: Tuesday, August 15, 2006 8:04 PM
Subject: RE: [Syslog] byte-counting vs special character

put into a standard!

<tp>
Well, I think you just wrote off most of the IETF STANDARDs that deal with
character-based protocols (like the e-mail we are using to communicate(?)).

A set of characters, of symbols, in a 'message' is encoded, given a number, be
it 6 or 8 or 16 or whatever number of bits.  If that bit pattern conflicts with
the 'control' aspects of a protocol, then that bit pattern must be
'transfer-encoded' so that it does not appear per se on the wire.  That is what
base64 or quoted-printable do for e-mail.

So we are talking of using a well-understood, widely deployed piece of protocol
architecture to solve a common problem.

</tp>
<snip/>