Subject: [Syslog] Small draft for Syslog File Storage?

From: "Rainer Gerhards" <[email-masked]>
Date: Wed, 10 Nov 2010 07:24:17 +0100

Hi all,

In what we did, we specified the on-the-wire format. However, we did not
specify any format to use when persisting syslog data to a file.

Note that we were very generous when specifying the on-the-wire format, for
example we permit LF, CR, NUL and many other characters considered dangerous
in file formats.

There are many tools available which interpret syslog data stored in text
files. However, different syslog implementations may use slightly different
file formats.

Together with the control character issue, the file format question both has
interoperability AND security issues. I think these would be very easy to fix
if we write a small RFC that specifies how text is to be encoded. It would be
similar, but much smaller to RFC4627 (JSON). Actually, I think we would need
to carry over primarily its section 2.5.

I would volunteer to write an initial draft, but would first like to get some
feedback if this effort has any chance of getting through.

Rainer

---

From: "David Harrington" <[email-masked]>
Date: Wed, 10 Nov 2010 14:51:43 +0800

How many syslog sender/receiver implementers would be willing to
support such a common format? 

How many log anaysis application vendors would like such a common
format? or do they consider it unneccesray because they convert
incoming info into their own proprietary database formats anyway?

dbh

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Rainer Gerhards
> Sent: Wednesday, November 10, 2010 2:24 PM
> To: [email-masked]
> Subject: [Syslog] Small draft for Syslog File Storage?
> 
> Hi all,
> 
> _______________________________________________
> Syslog mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/syslog

---

From: "Rainer Gerhards" <[email-masked]>
Date: Wed, 10 Nov 2010 08:38:51 +0100

> From: David Harrington [mailto:[email-masked]]
> Sent: Wednesday, November 10, 2010 7:52 AM
> To: Rainer Gerhards; [email-masked]
> Subject: RE: [Syslog] Small draft for Syslog File Storage?

Good questions, as usual. Obviously I have only one voice here, so for the
most part, I do not know. Would the OPS area be the right area to ask this in
addition to here?

My question was motivated by the Mitre CEE effort:

http://cee.mitre.org/

In very short words, CEE tries to define a standard event format, where what
syslog carries is a subset of the events possible. CEE will also define
syntaxes for log storage. We will most probably support XML, CSV, JSON and
syslog, with syslog being the only format where only a on-the-wire but no
file format standard exists.

I am on the CEE board and one thing we currently try to accomplish is define
a CEE-to-syslog mapping. There are a couple of the larger vendors interested
in logging on the board and the overall consensus seems to be that text files
play an important role when it comes to

a) storing log messages
b) feeding log messages into analysis backends

My own experience in the Linux environment and working with larger users
confirms that. I have some very large customers (which I cannot name due to
NDA) which store logs in (zipped) text file format because any other store is
impractical for their needs. Of course, that doesn't exclude representations
of other subsets in other formats for other needs.

I will try to gather feedback at least from the CEE community, but would
appreciate comments from others as well.

Rainer

---

From: Simon Josefsson <[email-masked]>
Date: Wed, 10 Nov 2010 12:52:34 +0100

> feedback if this effort has any chance of getting through.

I would support that effort.  Multiple incompatible syslog formats are a
pain, and it has bitten me several times in multiple jobs.

One approach that would be easy to move forward with is that you write a
document registering a MIME media sub-type, e.g., text/syslog or
application/syslog.  Describing one reasonable syslog text format in
that document.  With this approach, there is less pressure on all syslog
vendors to agree on a particular format, and you can just invent one
format that makes sense and let people adopt it on an opt-in basis.
I've registered some MIME media types, and learned some details when
doing that, so I could help with this approach if you want.  Having a
MIME media type specified would also enable reliable transfer of syslog
data in protocols that are MIME aware (e.g., HTTP or e-mail).  With this
approach, there is also less of a requirement to be backwards compatible
with existing (often sub-optimal) formats.

/Simon

---

From: Anton Chuvakin <[email-masked]>
Date: Thu, 11 Nov 2010 10:48:45 -0800

> specify any format to use when persisting syslog data to a file.

How big of a deal is this, really? Now, I don't question that
consistent representation is alway better (whether stored or flying
over the wire),  but that does not seem like a 'top syslog problem'
kinda issue to me.

Storage consistency might be way down on the list of issues that will
make syslog more useful for producers/consumers of the data - while
still allow the changes to be adopted....

-- 
Dr. Anton Chuvakin
Site: http://www.chuvakin.org
Blog: http://www.securitywarrior.org
LinkedIn: http://www.linkedin.com/in/chuvakin
Consulting: http://www.securitywarriorconsulting.com
Twitter: @anton_chuvakin
Google Voice: +1-510-771-7106

---

From: [email-masked]
Date: Thu, 11 Nov 2010 17:39:13 -0500

A use case that is coming up fairly often in the health care world is the 
following:

1) System A is gathering syslog reports and archiving them into an 
internal data structure.  The incoming messages comply with the Syslog 
standard.

2) At some later time, an administrative request is made for "all syslog 
reports related to Event X".  The mechanism for making such a request is 
not covered by Syslog, but it's a feature of System A.

3) These events are extracted and put into a file format for transfer to 
System B.  This transfer may use media, ftp, or any other method suitable 
for transfering files.

4) System B analyzes the event reports.

A standard format for syslog file storage would cover the file transferred 
in step 3. 

Kind Regards,

Robert Horn | Agfa HealthCare
Research Scientist | HE/Technology Office
T  +1 978 897 4860

Agfa HealthCare Corporation, Gotham Parkway 580, Carlstadt, NJ 07072-2405, 
USA
http://www.agfa.com/healthcare/
Click on link to read important disclaimer: 
http://www.agfa.com/healthcare/maildisclaimer 

"Rainer Gerhards" <[email-masked]> 
Sent by: [email-masked]
11/10/2010 02:38 AM

To
"David Harrington" <[email-masked]>, <[email-masked]>
cc

Subject
Re: [Syslog] Small draft for Syslog File Storage?

Good questions, as usual. Obviously I have only one voice here, so for the
most part, I do not know. Would the OPS area be the right area to ask this 
in
addition to here?

http://cee.mitre.org/

In very short words, CEE tries to define a standard event format, where 
what

I am on the CEE board and one thing we currently try to accomplish is 
define
a CEE-to-syslog mapping. There are a couple of the larger vendors 
interested
in logging on the board and the overall consensus seems to be that text 
files
play an important role when it comes to

a) storing log messages

My own experience in the Linux environment and working with larger users
confirms that. I have some very large customers (which I cannot name due 
to
NDA) which store logs in (zipped) text file format because any other store 
is
impractical for their needs. Of course, that doesn't exclude 
representations
of other subsets in other formats for other needs.

Rainer