From ietfc@btconnect.com Mon Nov 1 11:04:05 2010 Return-Path: X-Original-To: syslog@core3.amsl.com Delivered-To: syslog@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 3365B3A6A51 for ; Mon, 1 Nov 2010 11:04:05 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: 0.001 X-Spam-Level: X-Spam-Status: No, score=0.001 tagged_above=-999 required=5 tests=[BAYES_50=0.001] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1Q38O5NJv9bu for ; Mon, 1 Nov 2010 11:04:04 -0700 (PDT) Received: from mail.btconnect.com (c2beaomr10.btconnect.com [213.123.26.188]) by core3.amsl.com (Postfix) with ESMTP id E5CCB3A6A50 for ; Mon, 1 Nov 2010 11:04:03 -0700 (PDT) Received: from host86-150-161-23.range86-150.btcentralplus.com (HELO pc6) ([86.150.161.23]) by c2beaomr10.btconnect.com with SMTP id AMH03334; Mon, 01 Nov 2010 18:03:51 +0000 (GMT) Message-ID: <001101cb79e6$8d321620$4001a8c0@gateway.2wire.net> From: "t.petch" To: "Chris Lonvick" , References: Date: Mon, 1 Nov 2010 18:01:49 +0100 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1106 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 X-Mirapoint-IP-Reputation: reputation=Neutral-1, source=Queried, refid=tid=0001.0A0B0302.4CCF0107.0016, actions=tag X-Junkmail-Status: score=10/50, host=c2beaomr10.btconnect.com X-Junkmail-Signature-Raw: score=unknown, refid=str=0001.0A0B0201.4CCF0112.0242,ss=1,fgs=0, ip=0.0.0.0, so=2010-07-22 22:03:31, dmn=2009-09-10 00:05:08, mode=single engine X-Junkmail-IWF: false Subject: Re: [Syslog] New Version Notification for draft-gerhards-syslog-plain-tcp-05 (fwd) X-BeenThere: syslog@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Security Issues in Network Event Logging List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Nov 2010 18:04:05 -0000 Chris I had not noticed before but this seems to have changed direction during the summer; Informational not Standards Track, and stressing byte-counting more, byte-stuffing less. I do find it less clear. I think that the Introduction needs more work in the light of the changes to the rest of the document. I read "This specification includes descriptions of both format options in an attempt to ensure that standardized syslog transport receivers can receive and properly interpret messages sent from legacy syslog senders." got to the end of the document and thought 'oh no it does not!' and then realised that this is now an Appendix whereas before it was in the main body. Of course, if you never knew it was in the body, you might not be as confused as I. But really, the emphasis on standardised and legacy syslog seems misplaced. The carriage over TCP is the same whether the carried is SYSLOG-3164 or SYSLOG-MSG so the distinction seems spurious. And SYSLOG-3164 does not appear in any RFC or I-D I can find. Rather, you have two forms of adaptation to carry a message, and what that message is is mostly academic. Separately, I think that more is needed on Security. It is easier to sabotage TCP than it is UDP; spurious FIN, RST etc. And I think more is needed on closing the session. The transport receiver detects a format error (well, the transport sender is not going to) sends FIN, gets FIN-ACK and .... the transport sender carries merrily on. I think that there should be a recommendation that the transport sender closes the connection and reopens it if it wants to. Tom Petch ----- Original Message ----- From: "Chris Lonvick" To: Sent: Friday, October 01, 2010 9:16 PM Subject: [Syslog] New Version Notification for draft-gerhards-syslog-plain-tcp-05 (fwd) > Hi Folks, > > While this is a non-WG item, there are some people interested. I've > updated the syslog/tcp draft and I'll invite reviews and comments. > > Thanks, > Chris > > ---------- Forwarded message ---------- > Date: Thu, 30 Sep 2010 09:04:15 -0700 (PDT) > From: IETF I-D Submission Tool > To: clonvick@cisco.com > Cc: rgerhards@adiscon.com > Subject: New Version Notification for draft-gerhards-syslog-plain-tcp-05 > > > A new version of I-D, draft-gerhards-syslog-plain-tcp-05.txt has been successfully submitted by Chris Lonvick and posted to the IETF repository. > > Filename: draft-gerhards-syslog-plain-tcp > Revision: 05 > Title: Transmission of Syslog Messages over TCP > Creation_date: 2010-09-30 > WG ID: Independent Submission > Number_of_pages: 14 > > Abstract: > There have been many implementations and deployments of legacy syslog > over TCP for many years. That protocol has evolved without being > standardized and has proven to be quite interoperable in practice. > > The aim of this specification is to document three things: how to > transmit standardized syslog over TCP, how TCP has been used as a > transport for legacy syslog, and how to correlate these usages. > > > > The IETF Secretariat. > > > _______________________________________________ > Syslog mailing list > Syslog@ietf.org > https://www.ietf.org/mailman/listinfo/syslog From ietfdbh@comcast.net Mon Nov 1 22:52:41 2010 Return-Path: X-Original-To: syslog@core3.amsl.com Delivered-To: syslog@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 134763A67EC for ; Mon, 1 Nov 2010 22:52:41 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.258 X-Spam-Level: X-Spam-Status: No, score=-102.258 tagged_above=-999 required=5 tests=[AWL=-0.259, BAYES_00=-2.599, J_CHICKENPOX_15=0.6, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tIuUqe0YFnWH for ; Mon, 1 Nov 2010 22:52:33 -0700 (PDT) Received: from QMTA11.westchester.pa.mail.comcast.net (qmta11.westchester.pa.mail.comcast.net [76.96.59.211]) by core3.amsl.com (Postfix) with ESMTP id C29AB3A67E4 for ; Mon, 1 Nov 2010 22:52:24 -0700 (PDT) Received: from omta17.westchester.pa.mail.comcast.net ([76.96.62.89]) by QMTA11.westchester.pa.mail.comcast.net with comcast id S5q21f0011vXlb85B5sPEu; Tue, 02 Nov 2010 05:52:23 +0000 Received: from 23FX1C1 ([67.189.235.106]) by omta17.westchester.pa.mail.comcast.net with comcast id S5sP1f0072JQnJT3d5sPLw; Tue, 02 Nov 2010 05:52:23 +0000 From: "David Harrington" To: "'t.petch'" , "'Chris Lonvick'" , Date: Tue, 2 Nov 2010 13:52:28 +0800 Message-ID: <82F9FF37AFF94FB8AC90606D1124D63A@23FX1C1> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook 11 X-MIMEOLE: Produced By Microsoft MimeOLE V6.00.2900.5994 Thread-Index: Act57y9sNoDu5xDVRhWhEqoAXQpm7gAXvs9AAABhayA= Subject: Re: [Syslog] New Version Notification fordraft-gerhards-syslog-plain-tcp-05 (fwd) X-BeenThere: syslog@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Security Issues in Network Event Logging List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Nov 2010 05:52:41 -0000 Hi, Many of the changes were made at my request. I believe the document as written would not have made it through IESG approval. 1) the IETF has defined a standard syslog; how to make your legacy proprietary version work is not an IETF problem. 2) the syslog WG was created to develop a secure syslog solution with secure transport and signing capability. How to make your legacy proprietary version work over non-secure transport is not an IETF problem. 3) Publishing this as a proposed standard seems to violate BCP 61. syslog/tls already provides "strong security" over tcp, so syslog/tcp is not needed to meet IETF goals. Under what circumstances is it **desirable** to use this specification (with no strong security available) in the Internet? Why not use the syslog/TLS specification, with the security features administratively turned off within secure environments? You cannot justify implementing this by saying things like "syslog/TLS is required and this is optional", and not explain WHY this additional non-bcp61-compliant specification is needed. 4) The aim of this IETF specification should be to document "how TCP MAY be used as a transport for standardized syslog", when the standard secure transport may not apply. (But I expect serious pushback from the IESG on this; see #3) Because this might have to work with legacy deployments, we also include as an appendix "how to correlate the legacy and standard usages." 5) RFC3164 is just a survey, not a specification. 6) RFC2119 language needed to be cleaned up. David Harrington Director, IETF Transport Area ietfdbh@comcast.net (preferred for ietf) dbharrington@huaweisymantec.com +1 603 828 1401 (cell) > -----Original Message----- > From: syslog-bounces@ietf.org > [mailto:syslog-bounces@ietf.org] On Behalf Of t.petch > Sent: Tuesday, November 02, 2010 1:02 AM > To: Chris Lonvick; syslog@ietf.org > Subject: Re: [Syslog] New Version Notification > fordraft-gerhards-syslog-plain-tcp-05 (fwd) > > Chris > > I had not noticed before but this seems to have changed > direction during the > summer; Informational not Standards Track, and stressing > byte-counting more, > byte-stuffing less. > > I do find it less clear. I think that the Introduction needs > more work in the > light of the changes to the rest of the document. I read > "This specification includes descriptions of both > format options in an attempt to ensure that standardized syslog > transport receivers can receive and properly interpret > messages sent > from legacy syslog senders." > got to the end of the document and thought 'oh no it does > not!' and then > realised that this is now an Appendix whereas before it was > in the main body. > Of course, if you never knew it was in the body, you might > not be as confused as > I. > > But really, the emphasis on standardised and legacy syslog > seems misplaced. The > carriage over TCP is the same whether the carried is > SYSLOG-3164 or SYSLOG-MSG > so the distinction seems spurious. And SYSLOG-3164 does not > appear in any RFC > or I-D I can find. > > Rather, you have two forms of adaptation to carry a message, > and what that > message is is mostly academic. > > Separately, I think that more is needed on Security. It is > easier to sabotage > TCP than it is UDP; spurious FIN, RST etc. > > And I think more is needed on closing the session. The > transport receiver > detects a format error (well, the transport sender is not > going to) sends FIN, > gets FIN-ACK and .... the transport sender carries merrily > on. I think that > there should be a recommendation that the transport sender > closes the connection > and reopens it if it wants to. > > Tom Petch > ----- Original Message ----- > From: "Chris Lonvick" > To: > Sent: Friday, October 01, 2010 9:16 PM > Subject: [Syslog] New Version Notification for > draft-gerhards-syslog-plain-tcp-05 (fwd) > > > > Hi Folks, > > > > While this is a non-WG item, there are some people interested. I've > > updated the syslog/tcp draft and I'll invite reviews and comments. > > > > Thanks, > > Chris > > > > ---------- Forwarded message ---------- > > Date: Thu, 30 Sep 2010 09:04:15 -0700 (PDT) > > From: IETF I-D Submission Tool > > To: clonvick@cisco.com > > Cc: rgerhards@adiscon.com > > Subject: New Version Notification for > draft-gerhards-syslog-plain-tcp-05 > > > > > > A new version of I-D, > draft-gerhards-syslog-plain-tcp-05.txt has been > successfully submitted by Chris Lonvick and posted to the > IETF repository. > > > > Filename: draft-gerhards-syslog-plain-tcp > > Revision: 05 > > Title: Transmission of Syslog Messages over TCP > > Creation_date: 2010-09-30 > > WG ID: Independent Submission > > Number_of_pages: 14 > > > > Abstract: > > There have been many implementations and deployments of > legacy syslog > > over TCP for many years. That protocol has evolved without being > > standardized and has proven to be quite interoperable in practice. > > > > The aim of this specification is to document three things: how to > > transmit standardized syslog over TCP, how TCP has been used as a > > transport for legacy syslog, and how to correlate these usages. > > > > > > > > The IETF Secretariat. > > > > > > _______________________________________________ > > Syslog mailing list > > Syslog@ietf.org > > https://www.ietf.org/mailman/listinfo/syslog > > _______________________________________________ > Syslog mailing list > Syslog@ietf.org > https://www.ietf.org/mailman/listinfo/syslog From rgerhards@hq.adiscon.com Tue Nov 9 22:24:04 2010 Return-Path: X-Original-To: syslog@core3.amsl.com Delivered-To: syslog@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id A4B5E3A68E6 for ; Tue, 9 Nov 2010 22:24:03 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.599 X-Spam-Level: X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nsmlY07h-dS7 for ; Tue, 9 Nov 2010 22:23:59 -0800 (PST) Received: from vmmail.adiscon.com (vmmail.adiscon.com [178.63.79.189]) by core3.amsl.com (Postfix) with ESMTP id 824763A68EA for ; Tue, 9 Nov 2010 22:23:54 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by vmmail.adiscon.com (Postfix) with ESMTP id 021CE74A46D for ; Wed, 10 Nov 2010 07:24:20 +0100 (CET) Received: from vmmail.adiscon.com ([127.0.0.1]) by localhost (vmmail.adiscon.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id dXbT6N4jTA2G for ; Wed, 10 Nov 2010 07:24:19 +0100 (CET) Received: from GRFEXC.intern.adiscon.com (pd95c774a.dip0.t-ipconnect.de [217.92.119.74]) by vmmail.adiscon.com (Postfix) with ESMTPA id C4A5274A466 for ; Wed, 10 Nov 2010 07:24:19 +0100 (CET) MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable x-cr-puzzleid: {435F3908-A628-4949-863B-181A7905EC06} X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message x-cr-hashedpuzzle: /D4= AlaA BCp3 BfgA Bsh4 ByVD CLv4 FGZz FJH4 GwZH HfHH H12R H8rG H+JF IiKy Is+E; 1; cwB5AHMAbABvAGcAQABpAGUAdABmAC4AbwByAGcA; Sosha1_v1; 7; {435F3908-A628-4949-863B-181A7905EC06}; cgBnAGUAcgBoAGEAcgBkAHMAQABoAHEALgBhAGQAaQBzAGMAbwBuAC4AYwBvAG0A; Wed, 10 Nov 2010 06:24:17 GMT; UwBtAGEAbABsACAAZAByAGEAZgB0ACAAZgBvAHIAIABTAHkAcwBsAG8AZwAgAEYAaQBsAGUAIABTAHQAbwByAGEAZwBlAD8A Date: Wed, 10 Nov 2010 07:24:17 +0100 Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD6C5@GRFEXC.intern.adiscon.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Small draft for Syslog File Storage? Thread-Index: AcuAn+bP3HAgEVt4R0ibTtuACoge8Q== From: "Rainer Gerhards" To: Subject: [Syslog] Small draft for Syslog File Storage? X-BeenThere: syslog@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Security Issues in Network Event Logging List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Nov 2010 06:24:04 -0000 Hi all, In what we did, we specified the on-the-wire format. However, we did not specify any format to use when persisting syslog data to a file. Note that we were very generous when specifying the on-the-wire format, = for example we permit LF, CR, NUL and many other characters considered = dangerous in file formats. There are many tools available which interpret syslog data stored in = text files. However, different syslog implementations may use slightly = different file formats. Together with the control character issue, the file format question both = has interoperability AND security issues. I think these would be very easy = to fix if we write a small RFC that specifies how text is to be encoded. It = would be similar, but much smaller to RFC4627 (JSON). Actually, I think we would = need to carry over primarily its section 2.5. I would volunteer to write an initial draft, but would first like to get = some feedback if this effort has any chance of getting through. Rainer From ietfdbh@comcast.net Tue Nov 9 22:51:37 2010 Return-Path: X-Original-To: syslog@core3.amsl.com Delivered-To: syslog@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 713A23A692B for ; Tue, 9 Nov 2010 22:51:37 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.435 X-Spam-Level: X-Spam-Status: No, score=-102.435 tagged_above=-999 required=5 tests=[AWL=0.164, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id p-2gz2mJNJGn for ; Tue, 9 Nov 2010 22:51:36 -0800 (PST) Received: from QMTA11.westchester.pa.mail.comcast.net (qmta11.westchester.pa.mail.comcast.net [76.96.59.211]) by core3.amsl.com (Postfix) with ESMTP id 238773A696A for ; Tue, 9 Nov 2010 22:51:35 -0800 (PST) Received: from omta05.westchester.pa.mail.comcast.net ([76.96.62.43]) by QMTA11.westchester.pa.mail.comcast.net with comcast id VJrX1f0010vyq2s5BJryEt; Wed, 10 Nov 2010 06:51:58 +0000 Received: from 23FX1C1 ([130.129.118.241]) by omta05.westchester.pa.mail.comcast.net with comcast id VJrl1f00B5CaykJ3RJroGC; Wed, 10 Nov 2010 06:51:56 +0000 From: "David Harrington" To: "'Rainer Gerhards'" , References: <9B6E2A8877C38245BFB15CC491A11DA71DD6C5@GRFEXC.intern.adiscon.com> Date: Wed, 10 Nov 2010 14:51:43 +0800 Message-ID: <108C7C8C45254453AB931B12C5E247A6@23FX1C1> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook 11 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.5994 In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD6C5@GRFEXC.intern.adiscon.com> Thread-index: AcuAn+bP3HAgEVt4R0ibTtuACoge8QAA4QLw Subject: Re: [Syslog] Small draft for Syslog File Storage? X-BeenThere: syslog@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Security Issues in Network Event Logging List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Nov 2010 06:51:37 -0000 How many syslog sender/receiver implementers would be willing to support such a common format? How many log anaysis application vendors would like such a common format? or do they consider it unneccesray because they convert incoming info into their own proprietary database formats anyway? dbh > -----Original Message----- > From: syslog-bounces@ietf.org > [mailto:syslog-bounces@ietf.org] On Behalf Of Rainer Gerhards > Sent: Wednesday, November 10, 2010 2:24 PM > To: syslog@ietf.org > Subject: [Syslog] Small draft for Syslog File Storage? > > Hi all, > > In what we did, we specified the on-the-wire format. However, > we did not > specify any format to use when persisting syslog data to a file. > > Note that we were very generous when specifying the > on-the-wire format, for > example we permit LF, CR, NUL and many other characters > considered dangerous > in file formats. > > There are many tools available which interpret syslog data > stored in text > files. However, different syslog implementations may use > slightly different > file formats. > > Together with the control character issue, the file format > question both has > interoperability AND security issues. I think these would be > very easy to fix > if we write a small RFC that specifies how text is to be > encoded. It would be > similar, but much smaller to RFC4627 (JSON). Actually, I > think we would need > to carry over primarily its section 2.5. > > I would volunteer to write an initial draft, but would first > like to get some > feedback if this effort has any chance of getting through. > > Rainer > _______________________________________________ > Syslog mailing list > Syslog@ietf.org > https://www.ietf.org/mailman/listinfo/syslog From rgerhards@hq.adiscon.com Tue Nov 9 23:38:27 2010 Return-Path: X-Original-To: syslog@core3.amsl.com Delivered-To: syslog@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id ECD8A3A6802 for ; Tue, 9 Nov 2010 23:38:27 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.599 X-Spam-Level: X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ps4xHlTkN2HW for ; Tue, 9 Nov 2010 23:38:26 -0800 (PST) Received: from vmmail.adiscon.com (vmmail.adiscon.com [178.63.79.189]) by core3.amsl.com (Postfix) with ESMTP id 361FB3A67C2 for ; Tue, 9 Nov 2010 23:38:24 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by vmmail.adiscon.com (Postfix) with ESMTP id 5C3B274A470; Wed, 10 Nov 2010 08:38:50 +0100 (CET) Received: from vmmail.adiscon.com ([127.0.0.1]) by localhost (vmmail.adiscon.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1UWh1nXm9D-K; Wed, 10 Nov 2010 08:38:50 +0100 (CET) Received: from GRFEXC.intern.adiscon.com (pd95c774a.dip0.t-ipconnect.de [217.92.119.74]) by vmmail.adiscon.com (Postfix) with ESMTPA id 258CC74A46D; Wed, 10 Nov 2010 08:38:50 +0100 (CET) Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-MimeOLE: Produced By Microsoft Exchange V6.5 Date: Wed, 10 Nov 2010 08:38:51 +0100 Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD6C8@GRFEXC.intern.adiscon.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [Syslog] Small draft for Syslog File Storage? Thread-Index: AcuAn+bP3HAgEVt4R0ibTtuACoge8QAA4QLwAAFFPpA= References: <9B6E2A8877C38245BFB15CC491A11DA71DD6C5@GRFEXC.intern.adiscon.com> <108C7C8C45254453AB931B12C5E247A6@23FX1C1> From: "Rainer Gerhards" To: "David Harrington" , Subject: Re: [Syslog] Small draft for Syslog File Storage? X-BeenThere: syslog@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Security Issues in Network Event Logging List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Nov 2010 07:38:28 -0000 > -----Original Message----- > From: David Harrington [mailto:ietfdbh@comcast.net] > Sent: Wednesday, November 10, 2010 7:52 AM > To: Rainer Gerhards; syslog@ietf.org > Subject: RE: [Syslog] Small draft for Syslog File Storage? Good questions, as usual. Obviously I have only one voice here, so for = the most part, I do not know. Would the OPS area be the right area to ask = this in addition to here? My question was motivated by the Mitre CEE effort: http://cee.mitre.org/ In very short words, CEE tries to define a standard event format, where = what syslog carries is a subset of the events possible. CEE will also define syntaxes for log storage. We will most probably support XML, CSV, JSON = and syslog, with syslog being the only format where only a on-the-wire but = no file format standard exists. I am on the CEE board and one thing we currently try to accomplish is = define a CEE-to-syslog mapping. There are a couple of the larger vendors = interested in logging on the board and the overall consensus seems to be that text = files play an important role when it comes to a) storing log messages b) feeding log messages into analysis backends My own experience in the Linux environment and working with larger users confirms that. I have some very large customers (which I cannot name due = to NDA) which store logs in (zipped) text file format because any other = store is impractical for their needs. Of course, that doesn't exclude = representations of other subsets in other formats for other needs. I will try to gather feedback at least from the CEE community, but would appreciate comments from others as well. Rainer > How many syslog sender/receiver implementers would be willing to > support such a common format? >=20 > How many log anaysis application vendors would like such a common > format? or do they consider it unneccesray because they convert > incoming info into their own proprietary database formats anyway? >=20 > dbh >=20 > > -----Original Message----- > > From: syslog-bounces@ietf.org > > [mailto:syslog-bounces@ietf.org] On Behalf Of Rainer Gerhards > > Sent: Wednesday, November 10, 2010 2:24 PM > > To: syslog@ietf.org > > Subject: [Syslog] Small draft for Syslog File Storage? > > > > Hi all, > > > > In what we did, we specified the on-the-wire format. However, > > we did not > > specify any format to use when persisting syslog data to a file. > > > > Note that we were very generous when specifying the > > on-the-wire format, for > > example we permit LF, CR, NUL and many other characters > > considered dangerous > > in file formats. > > > > There are many tools available which interpret syslog data > > stored in text > > files. However, different syslog implementations may use > > slightly different > > file formats. > > > > Together with the control character issue, the file format > > question both has > > interoperability AND security issues. I think these would be > > very easy to fix > > if we write a small RFC that specifies how text is to be > > encoded. It would be > > similar, but much smaller to RFC4627 (JSON). Actually, I > > think we would need > > to carry over primarily its section 2.5. > > > > I would volunteer to write an initial draft, but would first > > like to get some > > feedback if this effort has any chance of getting through. > > > > Rainer > > _______________________________________________ > > Syslog mailing list > > Syslog@ietf.org > > https://www.ietf.org/mailman/listinfo/syslog From simon@josefsson.org Wed Nov 10 03:52:39 2010 Return-Path: X-Original-To: syslog@core3.amsl.com Delivered-To: syslog@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id EE0D528C0D6 for ; Wed, 10 Nov 2010 03:52:39 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.599 X-Spam-Level: X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RpwINkJgX2An for ; Wed, 10 Nov 2010 03:52:39 -0800 (PST) Received: from yxa-v.extundo.com (yxa-v.extundo.com [83.241.177.39]) by core3.amsl.com (Postfix) with ESMTP id D754A3A6AA2 for ; Wed, 10 Nov 2010 03:52:38 -0800 (PST) Received: from latte.josefsson.org (c80-216-27-64.bredband.comhem.se [80.216.27.64]) (authenticated bits=0) by yxa-v.extundo.com (8.14.3/8.14.3/Debian-5+lenny1) with ESMTP id oAABqVss012806 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Wed, 10 Nov 2010 12:52:32 +0100 From: Simon Josefsson To: "Rainer Gerhards" References: <9B6E2A8877C38245BFB15CC491A11DA71DD6C5@GRFEXC.intern.adiscon.com> OpenPGP: id=B565716F; url=http://josefsson.org/key.txt X-Hashcash: 1:22:101110:rgerhards@hq.adiscon.com::nxiqM2YuLJP0RdvG:08kh X-Hashcash: 1:22:101110:syslog@ietf.org::S0h0pLUZxrZPll4T:RqP4 Date: Wed, 10 Nov 2010 12:52:34 +0100 In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD6C5@GRFEXC.intern.adiscon.com> (Rainer Gerhards's message of "Wed, 10 Nov 2010 07:24:17 +0100") Message-ID: <87mxphe6pp.fsf@latte.josefsson.org> User-Agent: Gnus/5.110011 (No Gnus v0.11) Emacs/23.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Virus-Scanned: clamav-milter 0.96.3 at yxa-v X-Virus-Status: Clean X-Mailman-Approved-At: Wed, 10 Nov 2010 08:04:12 -0800 Cc: syslog@ietf.org Subject: Re: [Syslog] Small draft for Syslog File Storage? X-BeenThere: syslog@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Security Issues in Network Event Logging List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Nov 2010 11:52:40 -0000 "Rainer Gerhards" writes: > Hi all, > > In what we did, we specified the on-the-wire format. However, we did not > specify any format to use when persisting syslog data to a file. > > Note that we were very generous when specifying the on-the-wire format, for > example we permit LF, CR, NUL and many other characters considered dangerous > in file formats. > > There are many tools available which interpret syslog data stored in text > files. However, different syslog implementations may use slightly different > file formats. > > Together with the control character issue, the file format question both has > interoperability AND security issues. I think these would be very easy to fix > if we write a small RFC that specifies how text is to be encoded. It would be > similar, but much smaller to RFC4627 (JSON). Actually, I think we would need > to carry over primarily its section 2.5. > > I would volunteer to write an initial draft, but would first like to get some > feedback if this effort has any chance of getting through. I would support that effort. Multiple incompatible syslog formats are a pain, and it has bitten me several times in multiple jobs. One approach that would be easy to move forward with is that you write a document registering a MIME media sub-type, e.g., text/syslog or application/syslog. Describing one reasonable syslog text format in that document. With this approach, there is less pressure on all syslog vendors to agree on a particular format, and you can just invent one format that makes sense and let people adopt it on an opt-in basis. I've registered some MIME media types, and learned some details when doing that, so I could help with this approach if you want. Having a MIME media type specified would also enable reliable transfer of syslog data in protocols that are MIME aware (e.g., HTTP or e-mail). With this approach, there is also less of a requirement to be backwards compatible with existing (often sub-optimal) formats. /Simon From rgerhards@hq.adiscon.com Wed Nov 10 09:06:29 2010 Return-Path: X-Original-To: syslog@core3.amsl.com Delivered-To: syslog@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 0A0443A69B0 for ; Wed, 10 Nov 2010 09:06:29 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.599 X-Spam-Level: X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id M6Z4d7+4pjhs for ; Wed, 10 Nov 2010 09:06:28 -0800 (PST) Received: from vmmail.adiscon.com (vmmail.adiscon.com [178.63.79.189]) by core3.amsl.com (Postfix) with ESMTP id F22243A6978 for ; Wed, 10 Nov 2010 09:06:27 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by vmmail.adiscon.com (Postfix) with ESMTP id 2A67274A4CA; Wed, 10 Nov 2010 18:06:55 +0100 (CET) Received: from vmmail.adiscon.com ([127.0.0.1]) by localhost (vmmail.adiscon.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id eqyXC+eiuzaw; Wed, 10 Nov 2010 18:06:55 +0100 (CET) Received: from GRFEXC.intern.adiscon.com (pd95c774a.dip0.t-ipconnect.de [217.92.119.74]) by vmmail.adiscon.com (Postfix) with ESMTPA id F118E74A4BE; Wed, 10 Nov 2010 18:06:54 +0100 (CET) Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-MimeOLE: Produced By Microsoft Exchange V6.5 Date: Wed, 10 Nov 2010 18:06:54 +0100 Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD6D6@GRFEXC.intern.adiscon.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Re: [Syslog] Small draft for Syslog File Storage? Thread-Index: AcuA+ayJB7ZcYYfZTVmNr9XzdwhXTg== From: "Rainer Gerhards" To: , Subject: Re: [Syslog] Small draft for Syslog File Storage? X-BeenThere: syslog@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Security Issues in Network Event Logging List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Nov 2010 17:06:29 -0000 Hi Simon, it looks like I accidently deleted you mail while being on the phone. So = I can not directly reply to it. The idea of a mime encoding is interesting. Can you point me to a RFC = where you did this? I would be very happy if you could care about the mime = parts of a draft, while I provide some of the encoding that fits what I see as = needs for CEE and many users I know (basic things like no control characters present, one message per text file line and so). Thanks, Rainer From heinbockel@mitre.org Wed Nov 10 11:28:36 2010 Return-Path: X-Original-To: syslog@core3.amsl.com Delivered-To: syslog@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 275A23A6887 for ; Wed, 10 Nov 2010 11:28:36 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -5.949 X-Spam-Level: X-Spam-Status: No, score=-5.949 tagged_above=-999 required=5 tests=[AWL=0.650, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jZmaPHpeV-o9 for ; Wed, 10 Nov 2010 11:28:34 -0800 (PST) Received: from smtp-bedford.mitre.org (smtp-bedford.mitre.org [129.83.20.191]) by core3.amsl.com (Postfix) with ESMTP id C699B3A6821 for ; Wed, 10 Nov 2010 11:28:33 -0800 (PST) Received: from smtp-bedford.mitre.org (localhost.localdomain [127.0.0.1]) by smtp-bedford.mitre.org (8.13.1/8.13.1) with ESMTP id oAAJT1T2025539 for ; Wed, 10 Nov 2010 14:29:01 -0500 Received: from imchub2.MITRE.ORG (imchub2.mitre.org [129.83.29.74]) by smtp-bedford.mitre.org (8.13.1/8.13.1) with ESMTP id oAAJT1xS025534 for ; Wed, 10 Nov 2010 14:29:01 -0500 Received: from IMCMBX3.MITRE.ORG ([129.83.29.206]) by imchub2.MITRE.ORG ([129.83.29.74]) with mapi; Wed, 10 Nov 2010 14:29:01 -0500 From: "Heinbockel, Bill" To: "syslog@ietf.org" Date: Wed, 10 Nov 2010 14:26:13 -0500 Thread-Topic: Re: [Syslog] Small draft for Syslog File Storage? Thread-Index: AcuBDSGslkXMwmgqScKcXKhlp6ijLw== Message-ID: <93ED0A84F9A1D74FA65021D940AA5884053E827EBF@IMCMBX3.MITRE.ORG> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: multipart/signed; protocol="application/x-pkcs7-signature"; micalg=SHA1; boundary="----=_NextPart_000_0005_01CB80E3.38DFBE70" MIME-Version: 1.0 Subject: Re: [Syslog] Small draft for Syslog File Storage? X-BeenThere: syslog@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Security Issues in Network Event Logging List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Nov 2010 19:29:20 -0000 ------=_NextPart_000_0005_01CB80E3.38DFBE70 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sounds like a good idea to me The biggest step that you need to make from the on-the-wire RFC5424 Syslog is the specification of a Syslog record separator. In most Syslog log files (as well as CSV and other multi-record file formats), the typical record separator is LF or CRLF. Regardless, in order to define the record separator, you will have to add at least one more encoding or Syslog syntax requirement on top of the existing RFC5424 specification, as currently all characters are valid in a Syslog message portion. The specification would be fairly straight-forward, as you could just standardize on the approaches taken by rsyslog and Syslog-ng. Also, RFC5424 provides enough flexibility in character escaping to build on further escaping for control characters (U+0000 through U+001F) to make this a possibility In addition, I would like to suggest the addition of an optional file header for Syslog files. This would allow for easy versioning of the file, allow a place for products to include additional information, and be able to hold information such as the vendor, name, and version of the application producing the log. This would be an especially nice feature when digging through and parsing old Syslog records Regardless of the outcome of this discussion, I would like to see a couple of more optional encodings added to the RFC5424 specification to handle U+0000 through U+001F characters maybe: \n, \r, \t, and some generic hex encoding for the others \x00 \x01 ... \x1F > -----Original Message----- > From: syslog-bounces at ietf.org > [mailto:syslog-bounces at ietf.org] On Behalf Of Rainer Gerhards > Sent: Wednesday, November 10, 2010 2:24 PM > To: syslog at ietf.org > Subject: [Syslog] Small draft for Syslog File Storage? > > Hi all, > > In what we did, we specified the on-the-wire format. However, > we did not > specify any format to use when persisting syslog data to a file. > > Note that we were very generous when specifying the > on-the-wire format, for > example we permit LF, CR, NUL and many other characters > considered dangerous > in file formats. > > There are many tools available which interpret syslog data > stored in text > files. However, different syslog implementations may use > slightly different > file formats. > > Together with the control character issue, the file format > question both has > interoperability AND security issues. I think these would be > very easy to fix > if we write a small RFC that specifies how text is to be > encoded. It would be > similar, but much smaller to RFC4627 (JSON). Actually, I > think we would need > to carry over primarily its section 2.5. > > I would volunteer to write an initial draft, but would first > like to get some > feedback if this effort has any chance of getting through. > > Rainer > _______________________________________________ > Syslog mailing list > Syslog at ietf.org > https://www.ietf.org/mailman/listinfo/syslog ------=_NextPart_000_0005_01CB80E3.38DFBE70 Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIKxTCCA2Qw ggJMoAMCAQICAQEwDQYJKoZIhvcNAQEFBQAwWjESMBAGA1UEChMJbWl0cmUub3JnMR4wHAYDVQQL ExVDZXJ0aWZpY2F0ZSBBdXRob3JpdHkxJDAiBgNVBAMTG01JVFJFIENvcnBvcmF0aW9uIFJvb3Qg Q0EtMTAeFw0wNjA2MDEwNDAwMDBaFw0xODA2MDEwNDAwMDBaMFoxEjAQBgNVBAoTCW1pdHJlLm9y ZzEeMBwGA1UECxMVQ2VydGlmaWNhdGUgQXV0aG9yaXR5MSQwIgYDVQQDExtNSVRSRSBDb3Jwb3Jh dGlvbiBSb290IENBLTEwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCva1qWPZiEJv5v MtCbjt0cTu0Nbn15Q1cKqQBXKi8VSH9zZPmPxfWizJJ7JSqFJ5/sLUz3NsnUVjpLYBdFcxNXnOLj XtmDPFOewm5T98NZc9wRRCiDzt4f8qsHFI19ShPiK3cN5UqtJf+i66QVLA1S6CNL6o2eGAsAl5Wn xwOh2BfcWU5fNlHDVc9KKAlDDWpHjC2LLHAUbLP4ZzMIJKcLgLKFMtgM2AEfaSHzmi7WUdUHRCtC blrF7qzPsy/jBLFrr8VcX+mb7saq95pEOilgcix0/naW7kJfM5ph7UBB+S1O/OhH+ZjQ4MjWnwE8 A/YDrQx1OVLAOi29Bsho/l8lAgMBAAGjNTAzMBIGA1UdEwEB/wQIMAYBAf8CAQMwHQYDVR0OBBYE FMdwUQDYTf7kAdRolsU9n5qX/nQvMA0GCSqGSIb3DQEBBQUAA4IBAQAa+fVfCljimBlcfWwkfJXu XNKWun9xloFKjnq6SPGgAIKi5LUDil60a0NaNGoGSO3I1xzYt7ncayh21qXulcVTDFqubSJdv51a HTuJYcYUX72LN/gSq03UVLBCJzYm7ZLUlkb2YLo7xUeZ3coLFcT5AHR36kjG4cYHqXgH0liBl8jx pN0gwgaci4sgPLUj1w4t8zoKH+zxGFwXwTP/P+etQqiJZ5T00fLLm5kz9mmnxxmmIvUGNdsCAhGh dnF24pcrR43LNgyOBJ9DPUHBNq3kUQRO48WBKxBxflOtKzsICx/HEtIABcZn7deADHcY9spULZfB nQYdEpyz5tgh7Y2qMIIDcTCCAlmgAwIBAgICL4swDQYJKoZIhvcNAQEFBQAwXTESMBAGA1UEChMJ bWl0cmUub3JnMR4wHAYDVQQLExVDZXJ0aWZpY2F0ZSBBdXRob3JpdHkxJzAlBgNVBAMTHk1JVFJF IENvcnBvcmF0aW9uIFByaW1hcnkgQ0EtMTAeFw0wOTA3MDIxNzAxNTVaFw0xMDEyMjQxNzAxNTVa MGExEjAQBgNVBAoTCW1pdHJlLm9yZzEPMA0GA1UECxMGcGVvcGxlMRowGAYKCZImiZPyLGQBARMK aGVpbmJvY2tlbDEeMBwGA1UEAxMVSGVpbmJvY2tlbCBXaWxsaWFtIEouMIGfMA0GCSqGSIb3DQEB AQUAA4GNADCBiQKBgQCiPxyMbPx4hedlMNuj2CkE0lREfhUCAVkpyUGSAEoLgMLrHb8j7Abcp7EE JmFyIyZkE8GVucA0JVWqj784ldfRpYH6PXmj57BBTFdgA0/WLp8zZWHBAGsv/l+QMuars7nEIYuK F0Hlu9RjlFLoElcqv6O1RbmNAzJb92yuNPBVqQIDAQABo4G6MIG3MA4GA1UdDwEB/wQEAwIF4DAd BgNVHQ4EFgQUX9UwKQB9ivecYmlVUaspMPpnGNAwHwYDVR0jBBgwFoAUh7QPSI1iM0LBLVEaSB7C nrsKsa0wRAYDVR0fBD0wOzA5oDegNYYzaHR0cDovL3d3dy5taXRyZS5vcmcvdGVjaC9taWkvcGtp L2NhMV9taXRyZV9vcmcuY3JsMB8GA1UdEQQYMBaBFGhlaW5ib2NrZWxAbWl0cmUub3JnMA0GCSqG SIb3DQEBBQUAA4IBAQBJCxovAdq1uUba8ejgDii2JG65gjdIDpluQrhsGQNroNpdnXev8MNecCEw /2RH7kuwHx2lN0G7kSZkp0PTLe+X1vk0fL7m/NWD2G/FVe0umSbrg6Szy++tnkfcTojCodQrY6e8 pjAeMSdlF+3HHuh+wopRDZU+KBUykLqeGI91TZkGQOH66VphXvH7NtwkMnm131pwKLS2myfw9WJu PYE4pdvLuHb1EqB81RaOD9gNPf5gZBulLpK0XBbmx15c8x3lP0JydjKSsCV4+Y5YwUYcbpMB0nhd MuRYTmb/6JQFrUwrE+I74xH8i3jL6Vmx3QyooGFvoTp+hfS8PYti0PbyMIID5DCCAsygAwIBAgIB BTANBgkqhkiG9w0BAQUFADBaMRIwEAYDVQQKEwltaXRyZS5vcmcxHjAcBgNVBAsTFUNlcnRpZmlj YXRlIEF1dGhvcml0eTEkMCIGA1UEAxMbTUlUUkUgQ29ycG9yYXRpb24gUm9vdCBDQS0xMB4XDTA2 MDYwMzE3MTMyMloXDTEyMDYwMzE3MTMyMlowXTESMBAGA1UEChMJbWl0cmUub3JnMR4wHAYDVQQL ExVDZXJ0aWZpY2F0ZSBBdXRob3JpdHkxJzAlBgNVBAMTHk1JVFJFIENvcnBvcmF0aW9uIFByaW1h cnkgQ0EtMTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMjwe1ZdEIKoELdLvENnbkbO 3mVD54ZX5SjxTzFxhvoqhqSJmLOp32xT7J9KvBapJRbJv1BFdiSUN3O0q8H66nvQqwm1RYe+tTsW SO351FolGrDT9iDRtfWgH60PCKCbABLRsx3iGnEvjOQjeAtMn1AugqQWU3PWZXYy1Grbya+NOytY vu1p60bDFSoSAn4Jojv14lUfWHl8s3kahay8umLSQibiXdEEX8CrSkaXpOaFOuyM6+wpRw7TybO1 Dk4zGAUtn0887QvsMDk6evgN2WxMprkHAGUcJhpI1QXtkfDIl9ukdNiIoM/vdN2QC/+m6b8dA55K 5UdlBa9SgRnwapkCAwEAAaOBsTCBrjASBgNVHRMBAf8ECDAGAQH/AgECMA4GA1UdDwEB/wQEAwIB hjAdBgNVHQ4EFgQUh7QPSI1iM0LBLVEaSB7CnrsKsa0wHwYDVR0jBBgwFoAUx3BRANhN/uQB1GiW xT2fmpf+dC8wSAYDVR0fBEEwPzA9oDugOYY3aHR0cDovL3d3dy5taXRyZS5vcmcvdGVjaC9taWkv cGtpL3Jvb3RjYTFfbWl0cmVfb3JnLmNybDANBgkqhkiG9w0BAQUFAAOCAQEATW5u664p7N0iAj27 Xl/akjdfkSQpaosf6cNyAHu7utCytFfY1WfRNmvnNDGYkqI3XMFOa18SNjiNsMCH+sFQaO+oyDnP iIkEZQvlfGGrRpqIm6j//Fgz85bnf1kAM5I61Np7ofCnciRvp9ZB/+u+9i262tgiJPJrvBcqXmge T9riCc3RPjxqPNmYslOvNLpIifchelJhF7nIge+7RkAUcTJenj8yKwK0J3+PEpgYRQ+V2C62rnjo huxPgMw/fYoNTOlh3MVl7adwyK1ahPw2a9eOjSWglqoPTaBNeHJqRJZZ6Vi7S55+VAWCfkAqM5m3 tUiVzjsp2dFcTJxnYezaoDGCAr0wggK5AgEBMGMwXTESMBAGA1UEChMJbWl0cmUub3JnMR4wHAYD VQQLExVDZXJ0aWZpY2F0ZSBBdXRob3JpdHkxJzAlBgNVBAMTHk1JVFJFIENvcnBvcmF0aW9uIFBy aW1hcnkgQ0EtMQICL4swCQYFKw4DAhoFAKCCAbAwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAc BgkqhkiG9w0BCQUxDxcNMTAxMTEwMTkyNjExWjAjBgkqhkiG9w0BCQQxFgQUX7Q9eeDIHdEupoJR hf30LjilHQ8wZwYJKoZIhvcNAQkPMVowWDAKBggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYI KoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgwBwYFKw4DAhowCgYIKoZIhvcNAgUw cgYJKwYBBAGCNxAEMWUwYzBdMRIwEAYDVQQKEwltaXRyZS5vcmcxHjAcBgNVBAsTFUNlcnRpZmlj YXRlIEF1dGhvcml0eTEnMCUGA1UEAxMeTUlUUkUgQ29ycG9yYXRpb24gUHJpbWFyeSBDQS0xAgIv izB0BgsqhkiG9w0BCRACCzFloGMwXTESMBAGA1UEChMJbWl0cmUub3JnMR4wHAYDVQQLExVDZXJ0 aWZpY2F0ZSBBdXRob3JpdHkxJzAlBgNVBAMTHk1JVFJFIENvcnBvcmF0aW9uIFByaW1hcnkgQ0Et MQICL4swDQYJKoZIhvcNAQEBBQAEgYAmi75GI8chzEy2nsvUGvmE9P7aDDOSCks58NqmVcDRR93L RCMeFTsnb9Yh3NT+Lenq0NKrTTUxtLLWMii1iMnuTBshGpii9UDS5WZAIruXqLAwSBw8SCuECOc4 5RMan9t7XWxeZqC3c0lL6d/lOBrCxZ7X6VPNL/mPpTqVUW3erAAAAAAAAA== ------=_NextPart_000_0005_01CB80E3.38DFBE70-- From simon@josefsson.org Wed Nov 10 10:25:29 2010 Return-Path: X-Original-To: syslog@core3.amsl.com Delivered-To: syslog@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 9EC103A6971 for ; Wed, 10 Nov 2010 10:25:29 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.599 X-Spam-Level: X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qOBSyU4G07H2 for ; Wed, 10 Nov 2010 10:25:27 -0800 (PST) Received: from yxa-v.extundo.com (yxa-v.extundo.com [83.241.177.39]) by core3.amsl.com (Postfix) with ESMTP id 79A193A698D for ; Wed, 10 Nov 2010 10:25:24 -0800 (PST) Received: from latte.josefsson.org (c80-216-27-64.bredband.comhem.se [80.216.27.64]) (authenticated bits=0) by yxa-v.extundo.com (8.14.3/8.14.3/Debian-5+lenny1) with ESMTP id oAAIPikR003521 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Wed, 10 Nov 2010 19:25:46 +0100 From: Simon Josefsson To: "Rainer Gerhards" References: <9B6E2A8877C38245BFB15CC491A11DA71DD6D6@GRFEXC.intern.adiscon.com> OpenPGP: id=B565716F; url=http://josefsson.org/key.txt X-Hashcash: 1:22:101110:rgerhards@hq.adiscon.com::iCqVjIf8W4f9ZKyU:4eIc X-Hashcash: 1:22:101110:syslog@ietf.org::1J18lp9pOXm5EtgS:Exq8 Date: Wed, 10 Nov 2010 19:25:50 +0100 In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD6D6@GRFEXC.intern.adiscon.com> (Rainer Gerhards's message of "Wed, 10 Nov 2010 18:06:54 +0100") Message-ID: <87vd45828h.fsf@latte.josefsson.org> User-Agent: Gnus/5.110011 (No Gnus v0.11) Emacs/23.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Virus-Scanned: clamav-milter 0.96.3 at yxa-v X-Virus-Status: Clean X-Mailman-Approved-At: Thu, 11 Nov 2010 08:05:48 -0800 Cc: syslog@ietf.org Subject: Re: [Syslog] Small draft for Syslog File Storage? X-BeenThere: syslog@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Security Issues in Network Event Logging List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Nov 2010 18:25:29 -0000 "Rainer Gerhards" writes: > Hi Simon, > > it looks like I accidently deleted you mail while being on the phone. So I > can not directly reply to it. > > The idea of a mime encoding is interesting. Can you point me to a RFC where > you did this? I would be very happy if you could care about the mime parts of > a draft, while I provide some of the encoding that fits what I see as needs > for CEE and many users I know (basic things like no control characters > present, one message per text file line and so). Hi Rainer. I did this for DNS data, see RFC 4027. There are some idiosyncratic aspects of MIME that needs to be handled, but I don't see anything that would be a show stopper for a syslog format. Oh, and please use a timestamp format that embeds the year! How about the RFC 3339 format? I hate how it is impossible to know what year a log entry was written on modern Linux systems. /Simon From clonvick@cisco.com Thu Nov 11 08:18:24 2010 Return-Path: X-Original-To: syslog@core3.amsl.com Delivered-To: syslog@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id B02A33A6A71 for ; Thu, 11 Nov 2010 08:18:24 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -110.599 X-Spam-Level: X-Spam-Status: No, score=-110.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vJ2JOVvCT+GZ for ; Thu, 11 Nov 2010 08:18:23 -0800 (PST) Received: from sj-iport-4.cisco.com (sj-iport-4.cisco.com [171.68.10.86]) by core3.amsl.com (Postfix) with ESMTP id EA3B63A6822 for ; Thu, 11 Nov 2010 08:18:23 -0800 (PST) Authentication-Results: sj-iport-4.cisco.com; dkim=neutral (message not signed) header.i=none X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AvsEAMql20yrR7Ht/2dsb2JhbACiRXGld5tUhUoEhFo X-IronPort-AV: E=Sophos;i="4.59,183,1288569600"; d="scan'208";a="215612781" Received: from sj-core-1.cisco.com ([171.71.177.237]) by sj-iport-4.cisco.com with ESMTP; 11 Nov 2010 16:18:54 +0000 Received: from sjc-cde-011.cisco.com (sjc-cde-011.cisco.com [171.69.16.68]) by sj-core-1.cisco.com (8.13.8/8.14.3) with ESMTP id oABGIsEC004885; Thu, 11 Nov 2010 16:18:54 GMT Date: Thu, 11 Nov 2010 08:18:54 -0800 (PST) From: Chris Lonvick To: Simon Josefsson In-Reply-To: <87vd45828h.fsf@latte.josefsson.org> Message-ID: References: <9B6E2A8877C38245BFB15CC491A11DA71DD6D6@GRFEXC.intern.adiscon.com> <87vd45828h.fsf@latte.josefsson.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: syslog@ietf.org Subject: Re: [Syslog] Small draft for Syslog File Storage? X-BeenThere: syslog@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Security Issues in Network Event Logging List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Nov 2010 16:18:24 -0000 Hi Simon, On Wed, 10 Nov 2010, Simon Josefsson wrote: > Oh, and please use a timestamp format that embeds the year! How about > the RFC 3339 format? I hate how it is impossible to know what year a > log entry was written on modern Linux systems. Take a look at RFC 5424. The timestamp is from RFC 3339. Thanks, Chris From rgerhards@hq.adiscon.com Thu Nov 11 08:21:18 2010 Return-Path: X-Original-To: syslog@core3.amsl.com Delivered-To: syslog@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 0FD763A6A85 for ; Thu, 11 Nov 2010 08:21:18 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.599 X-Spam-Level: X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zwX0Dy43LGja for ; Thu, 11 Nov 2010 08:21:14 -0800 (PST) Received: from vmmail.adiscon.com (vmmail.adiscon.com [178.63.79.189]) by core3.amsl.com (Postfix) with ESMTP id 024C13A6822 for ; Thu, 11 Nov 2010 08:21:12 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by vmmail.adiscon.com (Postfix) with ESMTP id BE6CA74A4DB; Thu, 11 Nov 2010 17:21:41 +0100 (CET) Received: from vmmail.adiscon.com ([127.0.0.1]) by localhost (vmmail.adiscon.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tc72dwRK3Quu; Thu, 11 Nov 2010 17:21:41 +0100 (CET) Received: from GRFEXC.intern.adiscon.com (pd95c774a.dip0.t-ipconnect.de [217.92.119.74]) by vmmail.adiscon.com (Postfix) with ESMTPA id 9834A74A4DA; Thu, 11 Nov 2010 17:21:41 +0100 (CET) Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-MimeOLE: Produced By Microsoft Exchange V6.5 Date: Thu, 11 Nov 2010 17:21:40 +0100 Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD6E3@GRFEXC.intern.adiscon.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [Syslog] Small draft for Syslog File Storage? Thread-Index: AcuBvC8OvDPl90MTQjG4Q4J3YI++egAABwFw References: <9B6E2A8877C38245BFB15CC491A11DA71DD6D6@GRFEXC.intern.adiscon.com> <87vd45828h.fsf@latte.josefsson.org> From: "Rainer Gerhards" To: "Chris Lonvick" , "Simon Josefsson" Cc: syslog@ietf.org Subject: Re: [Syslog] Small draft for Syslog File Storage? X-BeenThere: syslog@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Security Issues in Network Event Logging List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Nov 2010 16:21:18 -0000 > -----Original Message----- > From: Chris Lonvick [mailto:clonvick@cisco.com] > Sent: Thursday, November 11, 2010 5:19 PM > To: Simon Josefsson > Cc: Rainer Gerhards; syslog@ietf.org > Subject: Re: [Syslog] Small draft for Syslog File Storage? >=20 > Hi Simon, >=20 > On Wed, 10 Nov 2010, Simon Josefsson wrote: > > Oh, and please use a timestamp format that embeds the year! How > about > > the RFC 3339 format? I hate how it is impossible to know what year = a > > log entry was written on modern Linux systems. >=20 > Take a look at RFC 5424. The timestamp is from RFC 3339. Sorry for the silence today. I am currently working very hard on very = complex code for log normalization. But one thing quickly: the timestamp is a typical example of how the = real world is hesitant to change. Rsyslog has become the default syslogd on = almost all modern linux distros. Rsyslog emits RFC3339 stamps be default, and = also uses them by default inside log files. But *all* distros have configured = it to use the old-style timestamp... Rainer From simon@josefsson.org Thu Nov 11 08:24:38 2010 Return-Path: X-Original-To: syslog@core3.amsl.com Delivered-To: syslog@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id DCCDD3A69AA for ; Thu, 11 Nov 2010 08:24:38 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.599 X-Spam-Level: X-Spam-Status: No, score=-102.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ng6Ta1tuEQzl for ; Thu, 11 Nov 2010 08:24:38 -0800 (PST) Received: from yxa-v.extundo.com (yxa-v.extundo.com [83.241.177.39]) by core3.amsl.com (Postfix) with ESMTP id C72B63A6A69 for ; Thu, 11 Nov 2010 08:24:37 -0800 (PST) Received: from latte.josefsson.org (c80-216-27-64.bredband.comhem.se [80.216.27.64]) (authenticated bits=0) by yxa-v.extundo.com (8.14.3/8.14.3/Debian-5+lenny1) with ESMTP id oABGOwex017053 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Thu, 11 Nov 2010 17:25:00 +0100 From: Simon Josefsson To: "Rainer Gerhards" References: <9B6E2A8877C38245BFB15CC491A11DA71DD6D6@GRFEXC.intern.adiscon.com> <87vd45828h.fsf@latte.josefsson.org> <9B6E2A8877C38245BFB15CC491A11DA71DD6E3@GRFEXC.intern.adiscon.com> OpenPGP: id=B565716F; url=http://josefsson.org/key.txt X-Hashcash: 1:22:101111:syslog@ietf.org::tx+AuoI0EsqAZcI9:1ZLq X-Hashcash: 1:22:101111:clonvick@cisco.com::9AEloECqX0tbT9KL:BKfK X-Hashcash: 1:22:101111:rgerhards@hq.adiscon.com::ff5QJVKDNqKyK2it:B8bP Date: Thu, 11 Nov 2010 17:25:10 +0100 In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD6E3@GRFEXC.intern.adiscon.com> (Rainer Gerhards's message of "Thu, 11 Nov 2010 17:21:40 +0100") Message-ID: <87oc9vj09l.fsf@latte.josefsson.org> User-Agent: Gnus/5.110011 (No Gnus v0.11) Emacs/23.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Virus-Scanned: clamav-milter 0.96.4 at yxa-v X-Virus-Status: Clean Cc: syslog@ietf.org Subject: Re: [Syslog] Small draft for Syslog File Storage? X-BeenThere: syslog@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Security Issues in Network Event Logging List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Nov 2010 16:24:39 -0000 "Rainer Gerhards" writes: >> -----Original Message----- >> From: Chris Lonvick [mailto:clonvick@cisco.com] >> Sent: Thursday, November 11, 2010 5:19 PM >> To: Simon Josefsson >> Cc: Rainer Gerhards; syslog@ietf.org >> Subject: Re: [Syslog] Small draft for Syslog File Storage? >> >> Hi Simon, >> >> On Wed, 10 Nov 2010, Simon Josefsson wrote: >> > Oh, and please use a timestamp format that embeds the year! How >> about >> > the RFC 3339 format? I hate how it is impossible to know what year a >> > log entry was written on modern Linux systems. >> >> Take a look at RFC 5424. The timestamp is from RFC 3339. > > Sorry for the silence today. I am currently working very hard on very complex > code for log normalization. > > But one thing quickly: the timestamp is a typical example of how the real > world is hesitant to change. Rsyslog has become the default syslogd on almost > all modern linux distros. Rsyslog emits RFC3339 stamps be default, and also > uses them by default inside log files. But *all* distros have configured it > to use the old-style timestamp... Yes, and that is annoying. Using the RFC 3339 format for stored data seems like the obvious choice if this is what RFC 5424 is using already. /Simon From rgerhards@hq.adiscon.com Thu Nov 11 08:25:28 2010 Return-Path: X-Original-To: syslog@core3.amsl.com Delivered-To: syslog@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 1E3AA3A6A74 for ; Thu, 11 Nov 2010 08:25:28 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.599 X-Spam-Level: X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fhGBLMBcKcHl for ; Thu, 11 Nov 2010 08:25:27 -0800 (PST) Received: from vmmail.adiscon.com (vmmail.adiscon.com [178.63.79.189]) by core3.amsl.com (Postfix) with ESMTP id 3873C3A69AA for ; Thu, 11 Nov 2010 08:25:27 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by vmmail.adiscon.com (Postfix) with ESMTP id 2DEF874A4DB; Thu, 11 Nov 2010 17:25:57 +0100 (CET) Received: from vmmail.adiscon.com ([127.0.0.1]) by localhost (vmmail.adiscon.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GiyEulQn6wip; Thu, 11 Nov 2010 17:25:57 +0100 (CET) Received: from GRFEXC.intern.adiscon.com (pd95c774a.dip0.t-ipconnect.de [217.92.119.74]) by vmmail.adiscon.com (Postfix) with ESMTPA id 0202874A4DA; Thu, 11 Nov 2010 17:25:56 +0100 (CET) Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-MimeOLE: Produced By Microsoft Exchange V6.5 Date: Thu, 11 Nov 2010 17:25:55 +0100 Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DD6E4@GRFEXC.intern.adiscon.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Small draft for Syslog File Storage? Thread-Index: AcuBvPqhefGg/l/jSyGR8n4mMCT9EAAABRBQ References: <9B6E2A8877C38245BFB15CC491A11DA71DD6D6@GRFEXC.intern.adiscon.com><87vd45828h.fsf@latte.josefsson.org><9B6E2A8877C38245BFB15CC491A11DA71DD6E3@GRFEXC.intern.adiscon.com> <87oc9vj09l.fsf@latte.josefsson.org> From: "Rainer Gerhards" To: "Simon Josefsson" Cc: syslog@ietf.org Subject: Re: [Syslog] Small draft for Syslog File Storage? X-BeenThere: syslog@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Security Issues in Network Event Logging List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Nov 2010 16:25:28 -0000 > -----Original Message----- > From: Simon Josefsson [mailto:simon@josefsson.org] > Sent: Thursday, November 11, 2010 5:25 PM > To: Rainer Gerhards > Cc: Chris Lonvick; syslog@ietf.org > Subject: Re: Small draft for Syslog File Storage? >=20 > "Rainer Gerhards" writes: >=20 > >> -----Original Message----- > >> From: Chris Lonvick [mailto:clonvick@cisco.com] > >> Sent: Thursday, November 11, 2010 5:19 PM > >> To: Simon Josefsson > >> Cc: Rainer Gerhards; syslog@ietf.org > >> Subject: Re: [Syslog] Small draft for Syslog File Storage? > >> > >> Hi Simon, > >> > >> On Wed, 10 Nov 2010, Simon Josefsson wrote: > >> > Oh, and please use a timestamp format that embeds the year! How > >> about > >> > the RFC 3339 format? I hate how it is impossible to know what > year a > >> > log entry was written on modern Linux systems. > >> > >> Take a look at RFC 5424. The timestamp is from RFC 3339. > > > > Sorry for the silence today. I am currently working very hard on = very > complex > > code for log normalization. > > > > But one thing quickly: the timestamp is a typical example of how the > real > > world is hesitant to change. Rsyslog has become the default syslogd > on almost > > all modern linux distros. Rsyslog emits RFC3339 stamps be default, > and also > > uses them by default inside log files. But *all* distros have > configured it > > to use the old-style timestamp... >=20 > Yes, and that is annoying. Using the RFC 3339 format for stored data > seems like the obvious choice if this is what RFC 5424 is using > already. Actually, I made the switch in rsyslog roughly 4 to 5 years ago, even = before we had RFC5424... :( Rainer From ietfc@btconnect.com Thu Nov 11 09:29:01 2010 Return-Path: X-Original-To: syslog@core3.amsl.com Delivered-To: syslog@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id AA29F3A6909 for ; Thu, 11 Nov 2010 09:29:01 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.413 X-Spam-Level: X-Spam-Status: No, score=-2.413 tagged_above=-999 required=5 tests=[AWL=0.186, BAYES_00=-2.599] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Ecz3GK9JG31l for ; Thu, 11 Nov 2010 09:29:00 -0800 (PST) Received: from mail.btconnect.com (c2bthomr13.btconnect.com [213.123.20.131]) by core3.amsl.com (Postfix) with ESMTP id 6C2603A6969 for ; Thu, 11 Nov 2010 09:28:59 -0800 (PST) Received: from host86-154-167-163.range86-154.btcentralplus.com (HELO pc6) ([86.154.167.163]) by c2bthomr13.btconnect.com with SMTP id APV20853; Thu, 11 Nov 2010 17:29:11 +0000 (GMT) Message-ID: <003001cb81bd$55518900$4001a8c0@gateway.2wire.net> From: "t.petch" To: "Simon Josefsson" , "Rainer Gerhards" References: <9B6E2A8877C38245BFB15CC491A11DA71DD6D6@GRFEXC.intern.adiscon.com> <87vd45828h.fsf@latte.josefsson.org> Date: Thu, 11 Nov 2010 17:23:41 +0100 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1106 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 X-Mirapoint-IP-Reputation: reputation=Fair-1, source=Queried, refid=tid=0001.0A0B0301.4CDC27DB.00B3, actions=tag X-Junkmail-Status: score=10/50, host=c2bthomr13.btconnect.com X-Junkmail-Signature-Raw: score=unknown, refid=str=0001.0A0B0206.4CDC27E9.01B0,ss=1,fgs=0, ip=0.0.0.0, so=2010-07-22 22:03:31, dmn=2009-09-10 00:05:08, mode=single engine X-Junkmail-IWF: false Cc: syslog@ietf.org Subject: Re: [Syslog] Small draft for Syslog File Storage? X-BeenThere: syslog@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Security Issues in Network Event Logging List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Nov 2010 17:29:01 -0000 Rainer You might also want to look at RFC4288 and RFC4289. Tom Petch ----- Original Message ----- From: "Simon Josefsson" To: "Rainer Gerhards" Cc: Sent: Wednesday, November 10, 2010 7:25 PM Subject: Re: [Syslog] Small draft for Syslog File Storage? > "Rainer Gerhards" writes: > > > Hi Simon, > > > > it looks like I accidently deleted you mail while being on the phone. So I > > can not directly reply to it. > > > > The idea of a mime encoding is interesting. Can you point me to a RFC where > > you did this? I would be very happy if you could care about the mime parts of > > a draft, while I provide some of the encoding that fits what I see as needs > > for CEE and many users I know (basic things like no control characters > > present, one message per text file line and so). > > Hi Rainer. I did this for DNS data, see RFC 4027. There are some > idiosyncratic aspects of MIME that needs to be handled, but I don't see > anything that would be a show stopper for a syslog format. > > Oh, and please use a timestamp format that embeds the year! How about > the RFC 3339 format? I hate how it is impossible to know what year a > log entry was written on modern Linux systems. > > /Simon > _______________________________________________ > Syslog mailing list > Syslog@ietf.org > https://www.ietf.org/mailman/listinfo/syslog From anton.chuvakin@gmail.com Thu Nov 11 10:48:48 2010 Return-Path: X-Original-To: syslog@core3.amsl.com Delivered-To: syslog@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id C3CC13A6950 for ; Thu, 11 Nov 2010 10:48:48 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.977 X-Spam-Level: X-Spam-Status: No, score=-1.977 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, FM_FORGED_GMAIL=0.622] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id D5jyqTAsMnAT for ; Thu, 11 Nov 2010 10:48:47 -0800 (PST) Received: from mail-ww0-f44.google.com (mail-ww0-f44.google.com [74.125.82.44]) by core3.amsl.com (Postfix) with ESMTP id AB2E13A67E6 for ; Thu, 11 Nov 2010 10:48:46 -0800 (PST) Received: by wwb34 with SMTP id 34so181179wwb.13 for ; Thu, 11 Nov 2010 10:49:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:mime-version:sender:received :in-reply-to:references:from:date:x-google-sender-auth:message-id :subject:to:content-type; bh=eW1vF9kV51GKodymjJGWzUANcgIyuqwHdryImASf7jk=; b=u+Ui7IbRqLEIxGwQeaqzhfCR0x4yyRBE7weh4MTDPMzPt8uuyvycP6Xss91YOV17Bd fE+McCueueIZCG1ALGkQOPGcTQrOH0uQPbhSSsUT7/AkFeAY34JZW6Ux05sfXIAf60ur hUzeX5zSsVJi2U24bJhAy/QSfyLdzlVNP9SUA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:content-type; b=uVrbscB2Jdb2c4ehPmzuG2flobed12XIJI/SbFEcnYVoYl96tWIq9QiZeLYFrTRlG6 4UZ+3w22b2DC8HQLyzK/unA0KKdMow5du10TAG5eM/0MFVQShpNkcm+xDaqOch+dLeVB UZYYo7Ld+eTVP+Mlrr+rL61jjBRtZnGliu6CE= Received: by 10.227.63.7 with SMTP id z7mr1280660wbh.21.1289501355626; Thu, 11 Nov 2010 10:49:15 -0800 (PST) MIME-Version: 1.0 Sender: anton.chuvakin@gmail.com Received: by 10.227.136.17 with HTTP; Thu, 11 Nov 2010 10:48:45 -0800 (PST) In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD6C5@GRFEXC.intern.adiscon.com> References: <9B6E2A8877C38245BFB15CC491A11DA71DD6C5@GRFEXC.intern.adiscon.com> From: Anton Chuvakin Date: Thu, 11 Nov 2010 10:48:45 -0800 X-Google-Sender-Auth: qR2Ehh2_nr1ysvvitgSkxXBXHLI Message-ID: To: syslog@ietf.org Content-Type: text/plain; charset=ISO-8859-1 Subject: Re: [Syslog] Small draft for Syslog File Storage? X-BeenThere: syslog@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Security Issues in Network Event Logging List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Nov 2010 18:50:36 -0000 > In what we did, we specified the on-the-wire format. However, we did not > specify any format to use when persisting syslog data to a file. How big of a deal is this, really? Now, I don't question that consistent representation is alway better (whether stored or flying over the wire), but that does not seem like a 'top syslog problem' kinda issue to me. Storage consistency might be way down on the list of issues that will make syslog more useful for producers/consumers of the data - while still allow the changes to be adopted.... -- Dr. Anton Chuvakin Site: http://www.chuvakin.org Blog: http://www.securitywarrior.org LinkedIn: http://www.linkedin.com/in/chuvakin Consulting: http://www.securitywarriorconsulting.com Twitter: @anton_chuvakin Google Voice: +1-510-771-7106 From prvs=924a79796=robert.horn@agfa.com Thu Nov 11 14:39:46 2010 Return-Path: X-Original-To: syslog@core3.amsl.com Delivered-To: syslog@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 6001D3A6358; Thu, 11 Nov 2010 14:39:46 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.598 X-Spam-Level: X-Spam-Status: No, score=-6.598 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EM585TQHtHN5; Thu, 11 Nov 2010 14:39:41 -0800 (PST) Received: from mornm01-out.agfa.com (mornm01-out.agfa.com [134.54.1.75]) by core3.amsl.com (Postfix) with ESMTP id D82663A6A84; Thu, 11 Nov 2010 14:39:28 -0800 (PST) X-IronPort-AV: E=Sophos;i="4.59,185,1288566000"; d="scan'208";a="114268851" Received: from morswa037.agfa.be (HELO morswa037.be.local) ([10.232.220.21]) by mornm01-out.agfa.com with ESMTP; 11 Nov 2010 23:39:12 +0100 In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DD6C8@GRFEXC.intern.adiscon.com> To: rgerhards@hq.adiscon.com MIME-Version: 1.0 Message-ID: From: robert.horn@agfa.com Date: Thu, 11 Nov 2010 17:39:13 -0500 Content-Type: multipart/alternative; boundary="=_alternative 007C7242852577D8_=" Cc: syslog@ietf.org, syslog-bounces@ietf.org Subject: Re: [Syslog] Small draft for Syslog File Storage? X-BeenThere: syslog@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Security Issues in Network Event Logging List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Nov 2010 22:39:46 -0000 This is a multipart message in MIME format. --=_alternative 007C7242852577D8_= Content-Type: text/plain; charset="US-ASCII" A use case that is coming up fairly often in the health care world is the following: 1) System A is gathering syslog reports and archiving them into an internal data structure. The incoming messages comply with the Syslog standard. 2) At some later time, an administrative request is made for "all syslog reports related to Event X". The mechanism for making such a request is not covered by Syslog, but it's a feature of System A. 3) These events are extracted and put into a file format for transfer to System B. This transfer may use media, ftp, or any other method suitable for transfering files. 4) System B analyzes the event reports. A standard format for syslog file storage would cover the file transferred in step 3. Kind Regards, Robert Horn | Agfa HealthCare Research Scientist | HE/Technology Office T +1 978 897 4860 Agfa HealthCare Corporation, Gotham Parkway 580, Carlstadt, NJ 07072-2405, USA http://www.agfa.com/healthcare/ Click on link to read important disclaimer: http://www.agfa.com/healthcare/maildisclaimer "Rainer Gerhards" Sent by: syslog-bounces@ietf.org 11/10/2010 02:38 AM To "David Harrington" , cc Subject Re: [Syslog] Small draft for Syslog File Storage? > -----Original Message----- > From: David Harrington [mailto:ietfdbh@comcast.net] > Sent: Wednesday, November 10, 2010 7:52 AM > To: Rainer Gerhards; syslog@ietf.org > Subject: RE: [Syslog] Small draft for Syslog File Storage? Good questions, as usual. Obviously I have only one voice here, so for the most part, I do not know. Would the OPS area be the right area to ask this in addition to here? My question was motivated by the Mitre CEE effort: http://cee.mitre.org/ In very short words, CEE tries to define a standard event format, where what syslog carries is a subset of the events possible. CEE will also define syntaxes for log storage. We will most probably support XML, CSV, JSON and syslog, with syslog being the only format where only a on-the-wire but no file format standard exists. I am on the CEE board and one thing we currently try to accomplish is define a CEE-to-syslog mapping. There are a couple of the larger vendors interested in logging on the board and the overall consensus seems to be that text files play an important role when it comes to a) storing log messages b) feeding log messages into analysis backends My own experience in the Linux environment and working with larger users confirms that. I have some very large customers (which I cannot name due to NDA) which store logs in (zipped) text file format because any other store is impractical for their needs. Of course, that doesn't exclude representations of other subsets in other formats for other needs. I will try to gather feedback at least from the CEE community, but would appreciate comments from others as well. Rainer > How many syslog sender/receiver implementers would be willing to > support such a common format? > > How many log anaysis application vendors would like such a common > format? or do they consider it unneccesray because they convert > incoming info into their own proprietary database formats anyway? > > dbh > > > -----Original Message----- > > From: syslog-bounces@ietf.org > > [mailto:syslog-bounces@ietf.org] On Behalf Of Rainer Gerhards > > Sent: Wednesday, November 10, 2010 2:24 PM > > To: syslog@ietf.org > > Subject: [Syslog] Small draft for Syslog File Storage? > > > > Hi all, > > > > In what we did, we specified the on-the-wire format. However, > > we did not > > specify any format to use when persisting syslog data to a file. > > > > Note that we were very generous when specifying the > > on-the-wire format, for > > example we permit LF, CR, NUL and many other characters > > considered dangerous > > in file formats. > > > > There are many tools available which interpret syslog data > > stored in text > > files. However, different syslog implementations may use > > slightly different > > file formats. > > > > Together with the control character issue, the file format > > question both has > > interoperability AND security issues. I think these would be > > very easy to fix > > if we write a small RFC that specifies how text is to be > > encoded. It would be > > similar, but much smaller to RFC4627 (JSON). Actually, I > > think we would need > > to carry over primarily its section 2.5. > > > > I would volunteer to write an initial draft, but would first > > like to get some > > feedback if this effort has any chance of getting through. > > > > Rainer > > _______________________________________________ > > Syslog mailing list > > Syslog@ietf.org > > https://www.ietf.org/mailman/listinfo/syslog _______________________________________________ Syslog mailing list Syslog@ietf.org https://www.ietf.org/mailman/listinfo/syslog --=_alternative 007C7242852577D8_= Content-Type: text/html; charset="US-ASCII"
A use case that is coming up fairly often in the health care world is the following:

1) System A is gathering syslog reports and archiving them into an internal data structure.  The incoming messages comply with the Syslog standard.

2) At some later time, an administrative request is made for "all syslog reports related to Event X".  The mechanism for making such a request is not covered by Syslog, but it's a feature of System A.

3) These events are extracted and put into a file format for transfer to System B.  This transfer may use media, ftp, or any other method suitable for transfering files.

4) System B analyzes the event reports.

A standard format for syslog file storage would cover the file transferred in step 3.  

Kind Regards,

Robert Horn | Agfa HealthCare
Research Scientist | HE/Technology Office
T  +1 978 897 4860

Agfa HealthCare Corporation, Gotham Parkway 580, Carlstadt, NJ 07072-2405, USA
http://www.agfa.com/healthcare/
Click on link to read important disclaimer: http://www.agfa.com/healthcare/maildisclaimer


"Rainer Gerhards" <rgerhards@hq.adiscon.com>
Sent by: syslog-bounces@ietf.org

11/10/2010 02:38 AM

To
"David Harrington" <ietfdbh@comcast.net>, <syslog@ietf.org>
cc
Subject
Re: [Syslog] Small draft for Syslog File Storage?






> -----Original Message-----
> From: David Harrington [mailto:ietfdbh@comcast.net]
> Sent: Wednesday, November 10, 2010 7:52 AM
> To: Rainer Gerhards; syslog@ietf.org
> Subject: RE: [Syslog] Small draft for Syslog File Storage?

Good questions, as usual. Obviously I have only one voice here, so for the
most part, I do not know. Would the OPS area be the right area to ask this in
addition to here?

My question was motivated by the Mitre CEE effort:

http://cee.mitre.org/

In very short words, CEE tries to define a standard event format, where what
syslog carries is a subset of the events possible. CEE will also define
syntaxes for log storage. We will most probably support XML, CSV, JSON and
syslog, with syslog being the only format where only a on-the-wire but no
file format standard exists.

I am on the CEE board and one thing we currently try to accomplish is define
a CEE-to-syslog mapping. There are a couple of the larger vendors interested
in logging on the board and the overall consensus seems to be that text files
play an important role when it comes to

a) storing log messages
b) feeding log messages into analysis backends

My own experience in the Linux environment and working with larger users
confirms that. I have some very large customers (which I cannot name due to
NDA) which store logs in (zipped) text file format because any other store is
impractical for their needs. Of course, that doesn't exclude representations
of other subsets in other formats for other needs.

I will try to gather feedback at least from the CEE community, but would
appreciate comments from others as well.

Rainer

> How many syslog sender/receiver implementers would be willing to
> support such a common format?
>
> How many log anaysis application vendors would like such a common
> format? or do they consider it unneccesray because they convert
> incoming info into their own proprietary database formats anyway?
>
> dbh
>
> > -----Original Message-----
> > From: syslog-bounces@ietf.org
> > [mailto:syslog-bounces@ietf.org] On Behalf Of Rainer Gerhards
> > Sent: Wednesday, November 10, 2010 2:24 PM
> > To: syslog@ietf.org
> > Subject: [Syslog] Small draft for Syslog File Storage?
> >
> > Hi all,
> >
> > In what we did, we specified the on-the-wire format. However,
> > we did not
> > specify any format to use when persisting syslog data to a file.
> >
> > Note that we were very generous when specifying the
> > on-the-wire format, for
> > example we permit LF, CR, NUL and many other characters
> > considered dangerous
> > in file formats.
> >
> > There are many tools available which interpret syslog data
> > stored in text
> > files. However, different syslog implementations may use
> > slightly different
> > file formats.
> >
> > Together with the control character issue, the file format
> > question both has
> > interoperability AND security issues. I think these would be
> > very easy to fix
> > if we write a small RFC that specifies how text is to be
> > encoded. It would be
> > similar, but much smaller to RFC4627 (JSON). Actually, I
> > think we would need
> > to carry over primarily its section 2.5.
> >
> > I would volunteer to write an initial draft, but would first
> > like to get some
> > feedback if this effort has any chance of getting through.
> >
> > Rainer
> > _______________________________________________
> > Syslog mailing list
> > Syslog@ietf.org
> > https://www.ietf.org/mailman/listinfo/syslog

_______________________________________________
Syslog mailing list
Syslog@ietf.org
https://www.ietf.org/mailman/listinfo/syslog
--=_alternative 007C7242852577D8_=-- From clonvick@cisco.com Tue Nov 16 11:46:00 2010 Return-Path: X-Original-To: syslog@core3.amsl.com Delivered-To: syslog@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 653913A6E2A for ; Tue, 16 Nov 2010 11:46:00 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -110.299 X-Spam-Level: X-Spam-Status: No, score=-110.299 tagged_above=-999 required=5 tests=[AWL=-0.300, BAYES_00=-2.599, J_CHICKENPOX_15=0.6, RCVD_IN_DNSWL_HI=-8, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id f-ngPGw++Hbh for ; Tue, 16 Nov 2010 11:45:59 -0800 (PST) Received: from sj-iport-6.cisco.com (sj-iport-6.cisco.com [171.71.176.117]) by core3.amsl.com (Postfix) with ESMTP id 9EC1B3A6E23 for ; Tue, 16 Nov 2010 11:45:59 -0800 (PST) Authentication-Results: sj-iport-6.cisco.com; dkim=neutral (message not signed) header.i=none X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AvsEAEdu4kyrR7Hu/2dsb2JhbACiY3GkY5snhUsEhFo X-IronPort-AV: E=Sophos;i="4.59,207,1288569600"; d="scan'208";a="621120873" Received: from sj-core-5.cisco.com ([171.71.177.238]) by sj-iport-6.cisco.com with ESMTP; 16 Nov 2010 19:46:43 +0000 Received: from sjc-cde-011.cisco.com (sjc-cde-011.cisco.com [171.69.16.68]) by sj-core-5.cisco.com (8.13.8/8.14.3) with ESMTP id oAGJkh7q021406; Tue, 16 Nov 2010 19:46:43 GMT Date: Tue, 16 Nov 2010 11:46:43 -0800 (PST) From: Chris Lonvick To: "t.petch" In-Reply-To: <001101cb79e6$8d321620$4001a8c0@gateway.2wire.net> Message-ID: References: <001101cb79e6$8d321620$4001a8c0@gateway.2wire.net> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: syslog@ietf.org Subject: Re: [Syslog] New Version Notification for draft-gerhards-syslog-plain-tcp-05 (fwd) X-BeenThere: syslog@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Security Issues in Network Event Logging List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Nov 2010 19:46:00 -0000 Hi Tom, On Mon, 1 Nov 2010, t.petch wrote: > Chris > > I had not noticed before but this seems to have changed direction during the > summer; Informational not Standards Track, and stressing byte-counting more, > byte-stuffing less. > > I do find it less clear. I think that the Introduction needs more work in the > light of the changes to the rest of the document. I read > "This specification includes descriptions of both > format options in an attempt to ensure that standardized syslog > transport receivers can receive and properly interpret messages sent > from legacy syslog senders." > got to the end of the document and thought 'oh no it does not!' and then > realised that this is now an Appendix whereas before it was in the main body. > Of course, if you never knew it was in the body, you might not be as confused as > I. > > But really, the emphasis on standardised and legacy syslog seems misplaced. The > carriage over TCP is the same whether the carried is SYSLOG-3164 or SYSLOG-MSG > so the distinction seems spurious. And SYSLOG-3164 does not appear in any RFC > or I-D I can find. > > Rather, you have two forms of adaptation to carry a message, and what that > message is is mostly academic. I was trying to clean it up so that it would only show that one mechanism was available for transporting RFC 5424 syslog messages over TCP, and that two mechanisms have been seen for transporting legacy syslog over TCP. The one mechanism for RFC 5424 over TCP is "bit counting" and is exactly like syslog/tls and syslog/dtls. The other method of "bit stuffing" has been seen in the wild but has problems. I'll take another look at it and see if I can explain that more clearly. > > Separately, I think that more is needed on Security. It is easier to sabotage > TCP than it is UDP; spurious FIN, RST etc. Yeah... Easier said than done. A lot of TCP attacks are not documented, and there are a lot of them. I'll see what I can do on that as well. > > And I think more is needed on closing the session. The transport receiver > detects a format error (well, the transport sender is not going to) sends FIN, > gets FIN-ACK and .... the transport sender carries merrily on. I think that > there should be a recommendation that the transport sender closes the connection > and reopens it if it wants to. Good point. I'll work on that. I'll dig through it after the US Thanksgiving holiday. Thanks, Chris