From turners@ieca.com Tue Feb 1 05:43:58 2011 Return-Path: X-Original-To: syslog@core3.amsl.com Delivered-To: syslog@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 94DA53A6CF4 for ; Tue, 1 Feb 2011 05:43:58 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.243 X-Spam-Level: X-Spam-Status: No, score=-102.243 tagged_above=-999 required=5 tests=[AWL=-0.245, BAYES_00=-2.599, J_CHICKENPOX_41=0.6, UNPARSEABLE_RELAY=0.001, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id skcE4JgR6QTb for ; Tue, 1 Feb 2011 05:43:57 -0800 (PST) Received: from nm9.bullet.mail.ac4.yahoo.com (nm9.bullet.mail.ac4.yahoo.com [98.139.52.206]) by core3.amsl.com (Postfix) with SMTP id E00513A6CE3 for ; Tue, 1 Feb 2011 05:43:56 -0800 (PST) Received: from [98.139.52.190] by nm9.bullet.mail.ac4.yahoo.com with NNFMP; 01 Feb 2011 13:47:11 -0000 Received: from [98.139.52.130] by tm3.bullet.mail.ac4.yahoo.com with NNFMP; 01 Feb 2011 13:47:11 -0000 Received: from [127.0.0.1] by omp1013.mail.ac4.yahoo.com with NNFMP; 01 Feb 2011 13:47:11 -0000 X-Yahoo-Newman-Id: 92017.6076.bm@omp1013.mail.ac4.yahoo.com Received: (qmail 1918 invoked from network); 1 Feb 2011 13:47:11 -0000 Received: from thunderfish.local (turners@96.241.4.28 with plain) by smtp112.biz.mail.re2.yahoo.com with SMTP; 01 Feb 2011 05:47:10 -0800 PST X-Yahoo-SMTP: ZrP3VLSswBDL75pF8ymZHDSu9B.vcMfDPgLJ X-YMail-OSG: 6RTOAuEVM1nu_5G3.rzTQrmHxUjE_CyL2XXcIgz9W04NYym 0PhY53F0IlUl.d3WcFMRdLc2pEz9uQbYnuCTa7actJDujoYsaGPRZC4RLHn. AuUFc6l1pPKESAcQes7Lc2BqfExg_Rj24uA_dzEYB4yxSytKIvkr1vL5POjk qP20VmgUxcAOjYjUBYz3nICzx23X6q53wYLUd4snGMU_q1WbwgFeVbaJH.LF JKj..UrP0qXY6zVWWNe1XbtGU5nj0PvD8DwyRHM2xEtyGyXWwDoqy9193Xm0 rEEOogQ-- X-Yahoo-Newman-Property: ymail-3 Message-ID: <4D480EDE.7030004@ieca.com> Date: Tue, 01 Feb 2011 08:47:10 -0500 From: Sean Turner User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.13) Gecko/20101207 Lightning/1.0b2 Thunderbird/3.1.7 MIME-Version: 1.0 To: Chris Lonvick References: <4D459BF9.9050407@ieca.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: syslog@ietf.org Subject: Re: [Syslog] New syslog/tcp draft available X-BeenThere: syslog@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Security Issues in Network Event Logging List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Feb 2011 13:43:58 -0000 Chris, I am sympathetic to not wanting to get stuck waiting for a reference. I think the informative reference you cite would past muster. spt On 1/31/11 9:48 PM, Chris Lonvick wrote: > Hi Sean, > > I've seen that but I don't want this document to sit idle for the next > couple of years while that matures and becomes a normative and stable > reference via becoming an RFC. > > I'm really thinking that putting in definitive references for transport > layer vulnerabilities is going a bit beyond what is expected of an > INFORMATIONAL document. That being said, I think it's a good idea and am > willing to pursue it within reason. > > Gont's document does reference a paper by Steve Bellovin: > Bellovin, S. M. 1989. Security Problems in the TCP/IP Protocol > Suite. Computer Communication Review, Vol. 19, No. 2, pp. 32-48. > That may be found here: > http://portal.acm.org/citation.cfm?id=378449 > > What would you think about referencing that document as an INFORMATIVE > reference in the third subsection of the Security Considerations section? > > Thanks, > Chris > > On Sun, 30 Jan 2011, Sean Turner wrote: > >> Chris, >> >> Not sure if this is what you're looking for, but have you checked out: >> http://datatracker.ietf.org/doc/draft-ietf-tcpm-tcp-security/ >> >> spt >> >> >> On 1/30/11 12:01 PM, Chris Lonvick wrote: >>> Hi Folks, >>> >>> We've finally gotten around to revising draft-gerhards-syslog-plain-tcp. >>> : -) >>> >>> This addresses the issues that Tom raised about >>> - the intro specifically stating what to expect in the body of the text >>> - a note on the transport security. >>> >>> For the first, we just sort'a straightened things out with a few edits. >>> For the latter, I looked in many places for a list of TCP >>> vulnerabilities but couldn't find anything substantial. The US-CERT had >>> a few implementation things and there were a scattering of other things. >>> In the end, I just added a subsection to warn impelemters to look >>> closely before writing code. If anyone has any other suggestions, please >>> let us know. >>> >>> Thanks, >>> Chris >>> _______________________________________________ >>> Syslog mailing list >>> Syslog@ietf.org >>> https://www.ietf.org/mailman/listinfo/syslog >>> >> > From clonvick@cisco.com Wed Feb 2 06:36:37 2011 Return-Path: X-Original-To: syslog@core3.amsl.com Delivered-To: syslog@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id AB9C23A6D2A for ; Wed, 2 Feb 2011 06:36:37 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -110.479 X-Spam-Level: X-Spam-Status: No, score=-110.479 tagged_above=-999 required=5 tests=[AWL=0.120, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id OjG-jQvpotAx for ; Wed, 2 Feb 2011 06:36:36 -0800 (PST) Received: from sj-iport-4.cisco.com (sj-iport-4.cisco.com [171.68.10.86]) by core3.amsl.com (Postfix) with ESMTP id B128C3A6D28 for ; Wed, 2 Feb 2011 06:36:36 -0800 (PST) Authentication-Results: sj-iport-4.cisco.com; dkim=neutral (message not signed) header.i=none X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AgkGAFP7SE2rR7H+/2dsb2JhbACWdAEBjiNzoQObFoVTBIR4 Received: from sj-core-2.cisco.com ([171.71.177.254]) by sj-iport-4.cisco.com with ESMTP; 02 Feb 2011 14:39:56 +0000 Received: from sjc-cde-011.cisco.com (sjc-cde-011.cisco.com [171.69.16.68]) by sj-core-2.cisco.com (8.13.8/8.14.3) with ESMTP id p12EduZB028563 for ; Wed, 2 Feb 2011 14:39:56 GMT Date: Wed, 2 Feb 2011 06:39:56 -0800 (PST) From: Chris Lonvick To: syslog@ietf.org Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; CHARSET=US-ASCII; format=flowed Content-ID: Subject: [Syslog] I-D Action:draft-gerhards-syslog-plain-tcp-08.txt (fwd) X-BeenThere: syslog@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Security Issues in Network Event Logging List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Feb 2011 14:36:37 -0000 Hi Folks, I've updated the document to include the reference from Steve Bellovin. We'd appreciate reviews and feedback. Thanks, Chris ---------- Forwarded message ---------- Date: Tue, 01 Feb 2011 18:15:01 -0800 From: Internet-Drafts@ietf.org To: i-d-announce@ietf.org Subject: I-D Action:draft-gerhards-syslog-plain-tcp-08.txt A New Internet-Draft is available from the on-line Internet-Drafts directories. Title : Transmission of Syslog Messages over TCP Author(s) : R. Gerhards, C. Lonvick Filename : draft-gerhards-syslog-plain-tcp-08.txt Pages : 13 Date : 2011-02-01 There have been many implementations and deployments of legacy syslog over TCP for many years. That protocol has evolved without being standardized and has proven to be quite interoperable in practice. The aim of this specification is to document three things: how to transmit standardized syslog over TCP, how TCP has been used as a transport for legacy syslog, and how to correlate these usages to ensure interoperability. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-gerhards-syslog-plain-tcp-08.txt Internet-Drafts are also available by anonymous FTP at: ftp://ftp.ietf.org/internet-drafts/ Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. From jeroen@unfix.org Tue Feb 15 03:17:32 2011 Return-Path: X-Original-To: syslog@core3.amsl.com Delivered-To: syslog@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 2D55F3A6A71 for ; Tue, 15 Feb 2011 03:17:32 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -101.392 X-Spam-Level: X-Spam-Status: No, score=-101.392 tagged_above=-999 required=5 tests=[AWL=1.207, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hyV70iMpb5k6 for ; Tue, 15 Feb 2011 03:17:31 -0800 (PST) Received: from abaddon.unfix.org (abaddon.unfix.org [62.220.146.203]) by core3.amsl.com (Postfix) with ESMTP id 9037B3A6C9F for ; Tue, 15 Feb 2011 03:17:28 -0800 (PST) Received: from [IPv6:2001:620:20:1001:216:d3ff:fe25:14da] (spaghetti.zurich.ibm.com [IPv6:2001:620:20:1001:216:d3ff:fe25:14da]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: jeroen) by abaddon.unfix.org (Postfix) with ESMTPSA id 97416205D1; Tue, 15 Feb 2011 12:17:24 +0100 (CET) Message-ID: <4D5A60C8.3090000@unfix.org> Date: Tue, 15 Feb 2011 12:17:28 +0100 From: Jeroen Massar Organization: Unfix User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: syslog@ietf.org X-Enigmail-Version: 1.1.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: Sam Johnston , cee@mitre.org Subject: [Syslog] draft-cloud-log-00 / CEE - why not IPFIX? X-BeenThere: syslog@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Security Issues in Network Event Logging List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Feb 2011 11:17:32 -0000 Hi, As the subject states, for both this cloud[1] and CEE[2] proposals, why not use IPFIX instead for structured logging data!? Greets, Jeroen [1] http://www.ietf.org/id/draft-cloud-log-00.txt [2] http://cee.mitre.org/ From jeroen@unfix.org Wed Feb 16 02:34:28 2011 Return-Path: X-Original-To: syslog@core3.amsl.com Delivered-To: syslog@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 3FD173A6DD2 for ; Wed, 16 Feb 2011 02:34:28 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -101.996 X-Spam-Level: X-Spam-Status: No, score=-101.996 tagged_above=-999 required=5 tests=[AWL=0.604, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 0cxFejZx3YPJ for ; Wed, 16 Feb 2011 02:34:27 -0800 (PST) Received: from abaddon.unfix.org (abaddon.unfix.org [62.220.146.203]) by core3.amsl.com (Postfix) with ESMTP id D289D3A6B59 for ; Wed, 16 Feb 2011 02:34:26 -0800 (PST) Received: from [IPv6:2001:41e0:ff42:99:222:cfff:fe31:ce41] (spaghetti.ch.unfix.org [IPv6:2001:41e0:ff42:99:222:cfff:fe31:ce41]) (using SSLv3 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: jeroen) by abaddon.unfix.org (Postfix) with ESMTPSA id 48ECB20B3E; Wed, 16 Feb 2011 11:34:19 +0100 (CET) Message-ID: <4D5BA85B.7040007@unfix.org> Date: Wed, 16 Feb 2011 11:35:07 +0100 From: Jeroen Massar Organization: Unfix User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: "Heinbockel, Bill" References: <4D5A60C8.3090000@unfix.org> <93ED0A84F9A1D74FA65021D940AA588405446C41F9@IMCMBX3.MITRE.ORG> In-Reply-To: <93ED0A84F9A1D74FA65021D940AA588405446C41F9@IMCMBX3.MITRE.ORG> X-Enigmail-Version: 1.1.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: Sam Johnston , cee@mitre.org, syslog@ietf.org Subject: Re: [Syslog] draft-cloud-log-00 / CEE - why not IPFIX? X-BeenThere: syslog@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Security Issues in Network Event Logging List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Feb 2011 10:34:28 -0000 On 2011-02-16 06:21, Heinbockel, Bill wrote: > From what I understand, IPFIX is for expression of IP flows from network sensing > devices. For a short bit forget about the history of IPFIX, it indeed comes from NetFlow, and thus is used quite in a network centric way, but effectively it is a structured streaming data format. > Could you please explain how IPFIX is relevant to event and cloud logging data? > I understand how CEE and IPFIX may overlap for describing networking events, but > it is unclear to me how IPFIX could handle things like Windows Event Logs and > RHEL audit logs. There are two parts to IPFIX: Templates + Data The template describes how the data looks like, for instance, lets take an Apache CLF log entry: 66.249.66.174 - - [16/Feb/2011:10:48:11 +0100] "GET /robots.txt HTTP/1.1" 200 2629 "-" "Googlebot-Image/0" We can make an IPFIX template for that [ {4, IPv4_SRC }, {4, TIMESTAMP}, {4, HTTP_METHOD}, {v, URL}, {v, HTTP_PROTOCOL}, {2, HTTP_RESULT}, {8, OCTETS}, {v, HTTP_REFER}, {v, HTTP_USERAGENT}, ] The 'v' markers indicate variable fieldlengths, the others indicates the number of bytes such a field takes. The data is then just encoded in the above format, presto. The above is a simple example, one can also have repeating lists and of course you could make a variable template which just includes the fields that you actually want to look at or you could already do some aggregation and add other fields. Templates are only sent every now and then, as they should not change. The data is the important bit. The fieldnames are actually numbers in the data, thus very compact, and are mapped to descriptions, data types etc, per a nice XML file http://www.iana.org/assignments/ipfix/ipfix.xml (or .xhtml or .txt for a more human readable version ;) for the official IANA list and with the help of Enterprise IDs any others can easily be added. The big advantage is that you can more or less do static templates if you want and you only need one single parser on the collector side, thus one does not have to create another parser and collector again for decoding other protocols, just one, the IPFIX one, and you can optimize that really well for all kinds of scenarios. Greets, Jeroen From rgerhards@hq.adiscon.com Wed Feb 16 02:39:20 2011 Return-Path: X-Original-To: syslog@core3.amsl.com Delivered-To: syslog@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 0D7CA3A6DF8 for ; Wed, 16 Feb 2011 02:39:20 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.599 X-Spam-Level: X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NZWw5rYlnz-V for ; Wed, 16 Feb 2011 02:39:18 -0800 (PST) Received: from vmmail.adiscon.com (vmmail.adiscon.com [178.63.79.189]) by core3.amsl.com (Postfix) with ESMTP id 97F8E3A6DEA for ; Wed, 16 Feb 2011 02:39:18 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by vmmail.adiscon.com (Postfix) with ESMTP id C16E474A478; Wed, 16 Feb 2011 11:39:44 +0100 (CET) Received: from vmmail.adiscon.com ([127.0.0.1]) by localhost (vmmail.adiscon.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Yksyx456kXvS; Wed, 16 Feb 2011 11:39:44 +0100 (CET) Received: from GRFEXC.intern.adiscon.com (pd95c774a.dip0.t-ipconnect.de [217.92.119.74]) by vmmail.adiscon.com (Postfix) with ESMTPA id 8D82774A470; Wed, 16 Feb 2011 11:39:44 +0100 (CET) Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-MimeOLE: Produced By Microsoft Exchange V6.5 Date: Wed, 16 Feb 2011 11:39:43 +0100 Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDC71@GRFEXC.intern.adiscon.com> In-Reply-To: <4D5BA85B.7040007@unfix.org> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [Syslog] draft-cloud-log-00 / CEE - why not IPFIX? Thread-Index: AcvNxTiINNQbVAD0QfeO4cX+AjASbwAAFsNw References: <4D5A60C8.3090000@unfix.org><93ED0A84F9A1D74FA65021D940AA588405446C41F9@IMCMBX3.MITRE.ORG> <4D5BA85B.7040007@unfix.org> From: "Rainer Gerhards" To: "Jeroen Massar" , "Heinbockel, Bill" Cc: Sam Johnston , cee@mitre.org, syslog@ietf.org Subject: Re: [Syslog] draft-cloud-log-00 / CEE - why not IPFIX? X-BeenThere: syslog@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Security Issues in Network Event Logging List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Feb 2011 10:39:20 -0000 The SIP CLF WG has just recently rejected IPFIX for it being binary and chosen indexed ASCII instead for their format. Their reasoning (after a = long struggle) is probably educating: http://www.ietf.org/mail-archive/web/sip-clf/current/msg00364.html I don't think that IPFIX is a good solution *in the syslog context*. It = is very far from what people expect. Other than that, I'd probably need to re-iterate the arguments made on the SIP CLF mailing list, so it = probably is better to refer to their archive ;) Rainer > -----Original Message----- > From: syslog-bounces@ietf.org [mailto:syslog-bounces@ietf.org] On > Behalf Of Jeroen Massar > Sent: Wednesday, February 16, 2011 11:35 AM > To: Heinbockel, Bill > Cc: Sam Johnston; cee@mitre.org; syslog@ietf.org > Subject: Re: [Syslog] draft-cloud-log-00 / CEE - why not IPFIX? >=20 > On 2011-02-16 06:21, Heinbockel, Bill wrote: > > From what I understand, IPFIX is for expression of IP flows from > network sensing > > devices. >=20 > For a short bit forget about the history of IPFIX, it indeed comes = from > NetFlow, and thus is used quite in a network centric way, but > effectively it is a structured streaming data format. >=20 > > Could you please explain how IPFIX is relevant to event and cloud > logging data? > > I understand how CEE and IPFIX may overlap for describing networking > events, but > > it is unclear to me how IPFIX could handle things like Windows Event > Logs and > > RHEL audit logs. >=20 > There are two parts to IPFIX: Templates + Data >=20 > The template describes how the data looks like, for instance, lets = take > an Apache CLF log entry: >=20 > 66.249.66.174 - - [16/Feb/2011:10:48:11 +0100] "GET /robots.txt > HTTP/1.1" 200 2629 "-" "Googlebot-Image/0" >=20 > We can make an IPFIX template for that >=20 > [ > {4, IPv4_SRC }, > {4, TIMESTAMP}, > {4, HTTP_METHOD}, > {v, URL}, > {v, HTTP_PROTOCOL}, > {2, HTTP_RESULT}, > {8, OCTETS}, > {v, HTTP_REFER}, > {v, HTTP_USERAGENT}, > ] >=20 > The 'v' markers indicate variable fieldlengths, the others indicates > the > number of bytes such a field takes. The data is then just encoded in > the > above format, presto. >=20 > The above is a simple example, one can also have repeating lists and = of > course you could make a variable template which just includes the > fields > that you actually want to look at or you could already do some > aggregation and add other fields. Templates are only sent every now = and > then, as they should not change. The data is the important bit. >=20 > The fieldnames are actually numbers in the data, thus very compact, = and > are mapped to descriptions, data types etc, per a nice XML file > http://www.iana.org/assignments/ipfix/ipfix.xml (or .xhtml or .txt = for > a more human readable version ;) for the official IANA list and with > the > help of Enterprise IDs any others can easily be added. >=20 > The big advantage is that you can more or less do static templates if > you want and you only need one single parser on the collector side, > thus > one does not have to create another parser and collector again for > decoding other protocols, just one, the IPFIX one, and you can = optimize > that really well for all kinds of scenarios. >=20 > Greets, > Jeroen > _______________________________________________ > Syslog mailing list > Syslog@ietf.org > https://www.ietf.org/mailman/listinfo/syslog From jeroen@unfix.org Wed Feb 16 02:55:43 2011 Return-Path: X-Original-To: syslog@core3.amsl.com Delivered-To: syslog@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 6DEB23A6C9B for ; Wed, 16 Feb 2011 02:55:43 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.197 X-Spam-Level: X-Spam-Status: No, score=-102.197 tagged_above=-999 required=5 tests=[AWL=0.402, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fqD7J8i0msHb for ; Wed, 16 Feb 2011 02:55:42 -0800 (PST) Received: from abaddon.unfix.org (abaddon.unfix.org [62.220.146.203]) by core3.amsl.com (Postfix) with ESMTP id 1F9153A6C8F for ; Wed, 16 Feb 2011 02:55:42 -0800 (PST) Received: from [IPv6:2001:41e0:ff42:99:222:cfff:fe31:ce41] (spaghetti.ch.unfix.org [IPv6:2001:41e0:ff42:99:222:cfff:fe31:ce41]) (using SSLv3 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: jeroen) by abaddon.unfix.org (Postfix) with ESMTPSA id BB7C921780; Wed, 16 Feb 2011 11:55:42 +0100 (CET) Message-ID: <4D5BAD69.2060608@unfix.org> Date: Wed, 16 Feb 2011 11:56:41 +0100 From: Jeroen Massar Organization: Unfix User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: Rainer Gerhards References: <4D5A60C8.3090000@unfix.org><93ED0A84F9A1D74FA65021D940AA588405446C41F9@IMCMBX3.MITRE.ORG> <4D5BA85B.7040007@unfix.org> <9B6E2A8877C38245BFB15CC491A11DA71DDC71@GRFEXC.intern.adiscon.com> In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDC71@GRFEXC.intern.adiscon.com> X-Enigmail-Version: 1.1.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: Sam Johnston , cee@mitre.org, syslog@ietf.org Subject: Re: [Syslog] draft-cloud-log-00 / CEE - why not IPFIX? X-BeenThere: syslog@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Security Issues in Network Event Logging List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Feb 2011 10:55:43 -0000 On 2011-02-16 11:39, Rainer Gerhards wrote: > The SIP CLF WG has just recently rejected IPFIX for it being binary and > chosen indexed ASCII instead for their format. Their reasoning (after a long > struggle) is probably educating: > > http://www.ietf.org/mail-archive/web/sip-clf/current/msg00364.html > > I don't think that IPFIX is a good solution *in the syslog context*. It is > very far from what people expect. Other than that, I'd probably need to > re-iterate the arguments made on the SIP CLF mailing list, so it probably is > better to refer to their archive ;) Why would they expect anything about the *DATA* format of a protocol? Note that the whole point that IPFIX (or any other structured data format for that matter) 'solves' is that one has to make a parser for every single log file format out there. Doing this at the meter tends to be cheaper due to the ability to distribute that than at the aggregated part. (then again sFlow as an example does it exactly the other way around, just pushing packets and letting the collector do the hard parsing part, but we are talking about sampled flows here thus you will miss out on events which is not a decision you can make at the meter if you are looking at say breaking attempts or failures ;) I think the pro-ascii versus binary argument comes effectively primarily from organizations who process large amounts of variable-string ascii data already and who do not really care about a few extra bits or a bit more overhead in processing data as they have large global clusters of hosts already doing that work. Their programming languages tend to be of a scripted-style too which tend to make it harder / less efficient to work on binary data but work great with ascii-alike data. Nevertheless, I've a generic logline parser which simply converts syslog and other log file formats into IPFIX. The problem with the whole ascii thing though is that one has to teach the parser what fields are what, and in the case of for instance the Apache CLF teach it the weird delimiters that are present. These are all special cases, something that one would really like to avoid if one wants to keep it speedy. My model partially solves that as I only have to do the special casing at the edge, where the log file gets converted into IPFIX. As those are considered 'meters' I just deploy more and more of those, while I can keep the collector side generally either a single box and otherwise easily distribute the data amongst them. And of course, the conversion goes the other way too, it can spit out reformatted 'ascii' again if needed. Greets, Jeroen (who finds it funny to see ASCII btw, as there is this thing called UTF-8 that makes it possible to express things in all languages of the world. I guess those people have to live with punycode etc...) From rgerhards@hq.adiscon.com Wed Feb 16 03:33:37 2011 Return-Path: X-Original-To: syslog@core3.amsl.com Delivered-To: syslog@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id C3D3D3A6C9D for ; Wed, 16 Feb 2011 03:33:37 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.599 X-Spam-Level: X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FBYYXY2p2sqa for ; Wed, 16 Feb 2011 03:33:36 -0800 (PST) Received: from vmmail.adiscon.com (vmmail.adiscon.com [178.63.79.189]) by core3.amsl.com (Postfix) with ESMTP id 40FB73A6B6B for ; Wed, 16 Feb 2011 03:33:36 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by vmmail.adiscon.com (Postfix) with ESMTP id C25B374A478; Wed, 16 Feb 2011 12:34:03 +0100 (CET) Received: from vmmail.adiscon.com ([127.0.0.1]) by localhost (vmmail.adiscon.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HxpTKJj5NrpU; Wed, 16 Feb 2011 12:34:03 +0100 (CET) Received: from GRFEXC.intern.adiscon.com (pd95c774a.dip0.t-ipconnect.de [217.92.119.74]) by vmmail.adiscon.com (Postfix) with ESMTPA id 8011B74A44A; Wed, 16 Feb 2011 12:34:03 +0100 (CET) Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: base64 X-MimeOLE: Produced By Microsoft Exchange V6.5 Date: Wed, 16 Feb 2011 12:34:06 +0100 Message-ID: <9B6E2A8877C38245BFB15CC491A11DA71DDC72@GRFEXC.intern.adiscon.com> In-Reply-To: <4D5BAD69.2060608@unfix.org> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [Syslog] draft-cloud-log-00 / CEE - why not IPFIX? Thread-Index: AcvNyYYPywr6CMt9RIG7zG/571u9KwAAuVEA References: <4D5A60C8.3090000@unfix.org><93ED0A84F9A1D74FA65021D940AA588405446C41F9@IMCMBX3.MITRE.ORG> <4D5BA85B.7040007@unfix.org> <9B6E2A8877C38245BFB15CC491A11DA71DDC71@GRFEXC.intern.adiscon.com> <4D5BAD69.2060608@unfix.org> From: "Rainer Gerhards" To: "Jeroen Massar" Cc: Sam Johnston , cee@mitre.org, syslog@ietf.org Subject: Re: [Syslog] draft-cloud-log-00 / CEE - why not IPFIX? X-BeenThere: syslog@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Security Issues in Network Event Logging List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Feb 2011 11:33:37 -0000 V2VsbCBJIHJlYWxseSBkb24ndCBsaWtlIHRvIHJlc3RhcnQgdGhhdCBkaXNjdXNzaW9uIGluIHRo aXMgY29udGV4dCBoZXJlDQphZ2FpbiwgYnV0IGxldCBtZSBub3RlIHRoYXQgd2hhdCB5b3UgYXJl IGRvaW5nIHdpdGggeW91ciBjb252ZXJ0ZXIgaXMgdmVyeQ0KdXNlZnVsLiBJdCBhY3R1YWxseSBu b3JtYWxpemVzIGRhdGEgaW50byBhIGNhbm9uaWNhbCBmb3JtYXQuIFRoaXMgaXMNCnNvbWV0aGlu ZyB0aGF0IENFRSB0ZW5kcyB0byBkbywgYnV0IGluIGEgcHJvdG9jb2wgYWdub3N0aWMgZm9ybWF0 IChhbmQgSSBkbw0Kc2ltaWxhciB0aGluZ3MgaW4gbXkgcHJvamVjdHMgYXMgd2VsbCkuIFRoZSBn ZW5lcmFsIHV0aWxpdHkgaXMgdW5xdWVzdGlvbmVkLg0KVGhlIHF1ZXN0aW9uIGlzIGlmIHN1Y2gg YW4gZWZmb3J0IG11c3QgYmUgYm91bmQgcmVzdHJpY3RlZCB0byBhIHNpbmdsZQ0KcHJvdG9jb2wu IE15IFBvViBpcyB0aGF0IHRoaXMgaXMgY291bnRlci1wcm9kdWN0aXZlLg0KDQpZb3UgZGVmaW5p dGVseSBoYXZlIGEgcG9pbnQgaW4gdGhhdCBJUEZJWCBtYXkgYmUgc3VwZXJpb3IgdGhhbiBzeXNs b2cgaW4gbWFueQ0KcmVnYXJkcy4gSSBkbyBub3QgaW50ZW5kIHRvIGFyZ3VlIGFnYWluc3QgdGhp cy4gQnV0IG9mdGVuIGEgc2ltcGxlciBzb2x1dGlvbg0KaXMgYWJsZSB0byBkcmF3IG1vcmUgYXR0 ZW50aW9uLCBhbmQgdGh1cyBkZXBsb3ltZW50cywgdGhhbiBhIChwb3RlbnRpYWxseSBvcg0KYWN0 dWFsbHkpIHRlY2huaWNhbCBzdXBlcmlvciBvbmUgKHNob3VsZG4ndCB3ZSBhbGwgdXNlIHRoZSBP U0kgc3RhY2sgYnkgbm93LA0KanVzdCBhcyBvbmUgZXhhbXBsZS4uLikuIA0KDQpJIGRvbid0IHRo aW5rIGl0IGlzIHVzZWZ1bCB0byBpbmNsdWRlIElQRklYIGluIHN5c2xvZy4gQnV0IGl0IG1heSBi ZSBhbg0Kb3B0aW9uIHRoYXQgSVBGSVggbWFrZXMgc3lzbG9nIG9ic29sZXRlLiBJIHRoaW5rIHlv dSBzaG91bGQgdGFrZSB0aGF0IGxhdGVyDQpyb3V0ZS4NCg0KQnV0IGFzIEkgc2FpZCAtLSBJIGRv IG5vdCBpbnRlbmQgdG8gc3Bhd24gYW5vdGhlciBpdGVyYXRpb24gb2YgdGhpcyBsZW5ndGh5DQpk aXNjdXNzaW9uLiBJdCBoYXMgb2NjdXJyZWQgc29vbyBvZnRlbiBpbiB0aGUgcGFzdCB5ZWFycy4N Cg0KUmFpbmVyDQoNClBTOiBZb3UgYXJlIHJpZ2h0IGluIG9uZSBtb3JlIHRoaW5nICJBU0NJSSIg aXMgdGhlIHdyb25nIHRlcm0uIE1vc3QgZm9sa3MNCihpbmNsdWRpbmcgbWUpIHNlZW0gdG8gYmUg c2xvcHB5IGFuZCBzYXkgQVNDSUkgd2hlbiB0aGV5IGFjdHVhbGx5IG1lYW4NCnByaW50YWJsZSB0 ZXh0IGRhdGEsIG9mIGNvdXJzZSBpbmNsdWRpbmcgVVRGLTguDQoNCg0KPiAtLS0tLU9yaWdpbmFs IE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiBKZXJvZW4gTWFzc2FyIFttYWlsdG86amVyb2VuQHVuZml4 Lm9yZ10NCj4gU2VudDogV2VkbmVzZGF5LCBGZWJydWFyeSAxNiwgMjAxMSAxMTo1NyBBTQ0KPiBU bzogUmFpbmVyIEdlcmhhcmRzDQo+IENjOiBIZWluYm9ja2VsLCBCaWxsOyBTYW0gSm9obnN0b247 IGNlZUBtaXRyZS5vcmc7IHN5c2xvZ0BpZXRmLm9yZw0KPiBTdWJqZWN0OiBSZTogW1N5c2xvZ10g ZHJhZnQtY2xvdWQtbG9nLTAwIC8gQ0VFIC0gd2h5IG5vdCBJUEZJWD8NCj4gDQo+IE9uIDIwMTEt MDItMTYgMTE6MzksIFJhaW5lciBHZXJoYXJkcyB3cm90ZToNCj4gPiBUaGUgU0lQIENMRiBXRyBo YXMganVzdCByZWNlbnRseSByZWplY3RlZCBJUEZJWCBmb3IgaXQgYmVpbmcgYmluYXJ5DQo+IGFu ZA0KPiA+IGNob3NlbiBpbmRleGVkIEFTQ0lJIGluc3RlYWQgZm9yIHRoZWlyIGZvcm1hdC4gVGhl aXIgcmVhc29uaW5nIChhZnRlcg0KPiBhIGxvbmcNCj4gPiBzdHJ1Z2dsZSkgaXMgcHJvYmFibHkg ZWR1Y2F0aW5nOg0KPiA+DQo+ID4gaHR0cDovL3d3dy5pZXRmLm9yZy9tYWlsLWFyY2hpdmUvd2Vi L3NpcC1jbGYvY3VycmVudC9tc2cwMDM2NC5odG1sDQo+ID4NCj4gPiBJIGRvbid0IHRoaW5rIHRo YXQgSVBGSVggaXMgYSBnb29kIHNvbHV0aW9uICppbiB0aGUgc3lzbG9nIGNvbnRleHQqLg0KPiBJ dCBpcw0KPiA+IHZlcnkgZmFyIGZyb20gd2hhdCBwZW9wbGUgZXhwZWN0LiBPdGhlciB0aGFuIHRo YXQsIEknZCBwcm9iYWJseSBuZWVkDQo+IHRvDQo+ID4gcmUtaXRlcmF0ZSB0aGUgYXJndW1lbnRz IG1hZGUgb24gdGhlIFNJUCBDTEYgbWFpbGluZyBsaXN0LCBzbyBpdA0KPiBwcm9iYWJseSBpcw0K PiA+IGJldHRlciB0byByZWZlciB0byB0aGVpciBhcmNoaXZlIDspDQo+IA0KPiBXaHkgd291bGQg dGhleSBleHBlY3QgYW55dGhpbmcgYWJvdXQgdGhlICpEQVRBKiBmb3JtYXQgb2YgYSBwcm90b2Nv bD8NCj4gDQo+IE5vdGUgdGhhdCB0aGUgd2hvbGUgcG9pbnQgdGhhdCBJUEZJWCAob3IgYW55IG90 aGVyIHN0cnVjdHVyZWQgZGF0YQ0KPiBmb3JtYXQgZm9yIHRoYXQgbWF0dGVyKSAnc29sdmVzJyBp cyB0aGF0IG9uZSBoYXMgdG8gbWFrZSBhIHBhcnNlciBmb3INCj4gZXZlcnkgc2luZ2xlIGxvZyBm aWxlIGZvcm1hdCBvdXQgdGhlcmUuIERvaW5nIHRoaXMgYXQgdGhlIG1ldGVyIHRlbmRzDQo+IHRv DQo+IGJlIGNoZWFwZXIgZHVlIHRvIHRoZSBhYmlsaXR5IHRvIGRpc3RyaWJ1dGUgdGhhdCB0aGFu IGF0IHRoZSBhZ2dyZWdhdGVkDQo+IHBhcnQuICh0aGVuIGFnYWluIHNGbG93IGFzIGFuIGV4YW1w bGUgZG9lcyBpdCBleGFjdGx5IHRoZSBvdGhlciB3YXkNCj4gYXJvdW5kLCBqdXN0IHB1c2hpbmcg cGFja2V0cyBhbmQgbGV0dGluZyB0aGUgY29sbGVjdG9yIGRvIHRoZSBoYXJkDQo+IHBhcnNpbmcg cGFydCwgYnV0IHdlIGFyZSB0YWxraW5nIGFib3V0IHNhbXBsZWQgZmxvd3MgaGVyZSB0aHVzIHlv dSB3aWxsDQo+IG1pc3Mgb3V0IG9uIGV2ZW50cyB3aGljaCBpcyBub3QgYSBkZWNpc2lvbiB5b3Ug Y2FuIG1ha2UgYXQgdGhlIG1ldGVyIGlmDQo+IHlvdSBhcmUgbG9va2luZyBhdCBzYXkgYnJlYWtp bmcgYXR0ZW1wdHMgb3IgZmFpbHVyZXMgOykNCj4gDQo+IEkgdGhpbmsgdGhlIHByby1hc2NpaSB2 ZXJzdXMgYmluYXJ5IGFyZ3VtZW50IGNvbWVzIGVmZmVjdGl2ZWx5DQo+IHByaW1hcmlseQ0KPiBm cm9tIG9yZ2FuaXphdGlvbnMgd2hvIHByb2Nlc3MgbGFyZ2UgYW1vdW50cyBvZiB2YXJpYWJsZS1z dHJpbmcgYXNjaWkNCj4gZGF0YSBhbHJlYWR5IGFuZCB3aG8gZG8gbm90IHJlYWxseSBjYXJlIGFi b3V0IGEgZmV3IGV4dHJhIGJpdHMgb3IgYSBiaXQNCj4gbW9yZSBvdmVyaGVhZCBpbiBwcm9jZXNz aW5nIGRhdGEgYXMgdGhleSBoYXZlIGxhcmdlIGdsb2JhbCBjbHVzdGVycyBvZg0KPiBob3N0cyBh bHJlYWR5IGRvaW5nIHRoYXQgd29yay4gVGhlaXIgcHJvZ3JhbW1pbmcgbGFuZ3VhZ2VzIHRlbmQg dG8gYmUNCj4gb2YNCj4gYSBzY3JpcHRlZC1zdHlsZSB0b28gd2hpY2ggdGVuZCB0byBtYWtlIGl0 IGhhcmRlciAvIGxlc3MgZWZmaWNpZW50IHRvDQo+IHdvcmsgb24gYmluYXJ5IGRhdGEgYnV0IHdv cmsgZ3JlYXQgd2l0aCBhc2NpaS1hbGlrZSBkYXRhLg0KPiANCj4gTmV2ZXJ0aGVsZXNzLCBJJ3Zl IGEgZ2VuZXJpYyBsb2dsaW5lIHBhcnNlciB3aGljaCBzaW1wbHkgY29udmVydHMNCj4gc3lzbG9n DQo+IGFuZCBvdGhlciBsb2cgZmlsZSBmb3JtYXRzIGludG8gSVBGSVguIFRoZSBwcm9ibGVtIHdp dGggdGhlIHdob2xlIGFzY2lpDQo+IHRoaW5nIHRob3VnaCBpcyB0aGF0IG9uZSBoYXMgdG8gdGVh Y2ggdGhlIHBhcnNlciB3aGF0IGZpZWxkcyBhcmUgd2hhdCwNCj4gYW5kIGluIHRoZSBjYXNlIG9m IGZvciBpbnN0YW5jZSB0aGUgQXBhY2hlIENMRiB0ZWFjaCBpdCB0aGUgd2VpcmQNCj4gZGVsaW1p dGVycyB0aGF0IGFyZSBwcmVzZW50LiBUaGVzZSBhcmUgYWxsIHNwZWNpYWwgY2FzZXMsIHNvbWV0 aGluZw0KPiB0aGF0DQo+IG9uZSB3b3VsZCByZWFsbHkgbGlrZSB0byBhdm9pZCBpZiBvbmUgd2Fu dHMgdG8ga2VlcCBpdCBzcGVlZHkuDQo+IA0KPiBNeSBtb2RlbCBwYXJ0aWFsbHkgc29sdmVzIHRo YXQgYXMgSSBvbmx5IGhhdmUgdG8gZG8gdGhlIHNwZWNpYWwgY2FzaW5nDQo+IGF0IHRoZSBlZGdl LCB3aGVyZSB0aGUgbG9nIGZpbGUgZ2V0cyBjb252ZXJ0ZWQgaW50byBJUEZJWC4gQXMgdGhvc2Ug YXJlDQo+IGNvbnNpZGVyZWQgJ21ldGVycycgSSBqdXN0IGRlcGxveSBtb3JlIGFuZCBtb3JlIG9m IHRob3NlLCB3aGlsZSBJIGNhbg0KPiBrZWVwIHRoZSBjb2xsZWN0b3Igc2lkZSBnZW5lcmFsbHkg ZWl0aGVyIGEgc2luZ2xlIGJveCBhbmQgb3RoZXJ3aXNlDQo+IGVhc2lseSBkaXN0cmlidXRlIHRo ZSBkYXRhIGFtb25nc3QgdGhlbS4NCj4gDQo+IEFuZCBvZiBjb3Vyc2UsIHRoZSBjb252ZXJzaW9u IGdvZXMgdGhlIG90aGVyIHdheSB0b28sIGl0IGNhbiBzcGl0IG91dA0KPiByZWZvcm1hdHRlZCAn YXNjaWknIGFnYWluIGlmIG5lZWRlZC4NCj4gDQo+IEdyZWV0cywNCj4gIEplcm9lbg0KPiANCj4g ICh3aG8gZmluZHMgaXQgZnVubnkgdG8gc2VlIEFTQ0lJIGJ0dywgYXMgdGhlcmUgaXMgdGhpcyB0 aGluZyBjYWxsZWQNCj4gICBVVEYtOCB0aGF0IG1ha2VzIGl0IHBvc3NpYmxlIHRvIGV4cHJlc3Mg dGhpbmdzIGluIGFsbCBsYW5ndWFnZXMgb2YNCj4gICB0aGUgd29ybGQuIEkgZ3Vlc3MgdGhvc2Ug cGVvcGxlIGhhdmUgdG8gbGl2ZSB3aXRoIHB1bnljb2RlIGV0Yy4uLikNCg== From jeroen@unfix.org Wed Feb 16 04:53:58 2011 Return-Path: X-Original-To: syslog@core3.amsl.com Delivered-To: syslog@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 5D29E3A6CAA; Wed, 16 Feb 2011 04:53:58 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.297 X-Spam-Level: X-Spam-Status: No, score=-102.297 tagged_above=-999 required=5 tests=[AWL=0.302, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xjbNwU+UPS5a; Wed, 16 Feb 2011 04:53:55 -0800 (PST) Received: from abaddon.unfix.org (abaddon.unfix.org [62.220.146.203]) by core3.amsl.com (Postfix) with ESMTP id 896D13A6CC6; Wed, 16 Feb 2011 04:53:54 -0800 (PST) Received: from [IPv6:2001:41e0:ff42:99:222:cfff:fe31:ce41] (spaghetti.ch.unfix.org [IPv6:2001:41e0:ff42:99:222:cfff:fe31:ce41]) (using SSLv3 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: jeroen) by abaddon.unfix.org (Postfix) with ESMTPSA id BEAC621778; Wed, 16 Feb 2011 13:53:52 +0100 (CET) Message-ID: <4D5BC8F8.70008@unfix.org> Date: Wed, 16 Feb 2011 13:54:16 +0100 From: Jeroen Massar Organization: Unfix User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: Rainer Gerhards References: <4D5A60C8.3090000@unfix.org><93ED0A84F9A1D74FA65021D940AA588405446C41F9@IMCMBX3.MITRE.ORG> <4D5BA85B.7040007@unfix.org> <9B6E2A8877C38245BFB15CC491A11DA71DDC71@GRFEXC.intern.adiscon.com> <4D5BAD69.2060608@unfix.org> <9B6E2A8877C38245BFB15CC491A11DA71DDC72@GRFEXC.intern.adiscon.com> In-Reply-To: <9B6E2A8877C38245BFB15CC491A11DA71DDC72@GRFEXC.intern.adiscon.com> X-Enigmail-Version: 1.1.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: cee@mitre.org, syslog@ietf.org, sip-clf@ietf.org Subject: Re: [Syslog] draft-cloud-log-00 / CEE - why not IPFIX? X-BeenThere: syslog@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Security Issues in Network Event Logging List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Feb 2011 12:53:58 -0000 On 2011-02-16 12:34, Rainer Gerhards wrote: [..] > The question is if such an effort must be bound restricted to a single > protocol. My PoV is that this is counter-productive. I might be missing the parsing here, but can you explain what part is counter-productive to have a single protocol and having to write only a single parser versus having a lot of different protocols who require parsers which can resolve ambiguity? > You definitely have a point in that IPFIX may be superior than syslog in many > regards. I do not intend to argue against this. But often a simpler solution > is able to draw more attention, and thus deployments, than a (potentially or > actually) technical superior one (shouldn't we all use the OSI stack by now, > just as one example...). I wonder what the "E" in IETF stands for if I see the above statement. Why not simply use CSV or hey just pack it in XML! :) Please note that IPFIX has gathered quite some attention already. As for the OSI stack argument, the world is not ready for IPv6 either. I like to also quote: http://www.ietf.org/mail-archive/web/sip-clf/current/msg00430.html 8<----------------------------------------------------------------------------- 1. Shall we use a TAB or a SPACE (or any LWS) as field delimiters? Considerations and points raised: - TABs don’t survive Telnet or web pages very well, especially when copy/pasting. - spaces can appear inside fields (as can most other delimiters we choose) - how do TAB and SPACE delimiters interact with shell tools (rep, cut, awk) ----------------------------------------------------------------------------->8 The moment you have to worry about a delimiter, you have big problems.... that is why char format is not a good idea. Yes, you can use all the standard unix command tools to process them, but are you really going to do this on the big scale? I certainly hope not. The same with Apache logs, very useful to have to parse the IP address again to get the information you want. Especially with IPv6 where the address can have quite a number of formats based on how the address is represented, not even going talking about the point. Oh and those unix tools, they do not support UTF-8 in a lot of cases, thus they are suddenly quite useless. > I don't think it is useful to include IPFIX in syslog. But it may be an > option that IPFIX makes syslog obsolete. I think you should take that later > route. If there is going to be new effort for syslog, is that then not the route it should be taking? Structured data is so much more useful than unstructured data, and that is the big problem that > But as I said -- I do not intend to spawn another iteration of this lengthy > discussion. It has occurred sooo often in the past years. Is there a summary of this discussion with a clear listing of pro/cons? Would be good to have that in a draft, is there one? If there isn't please write one with those arguments so that it can referred to instead of stating 'we did this before, EOD'... Greets, Jeroen From heinbockel@mitre.org Thu Feb 17 08:36:04 2011 Return-Path: X-Original-To: syslog@core3.amsl.com Delivered-To: syslog@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 17B5D3A6C7D for ; Thu, 17 Feb 2011 08:36:04 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.599 X-Spam-Level: X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ClNBL3UcbgSy for ; Thu, 17 Feb 2011 08:36:01 -0800 (PST) Received: from smtpksrv1.mitre.org (smtpksrv1.mitre.org [198.49.146.77]) by core3.amsl.com (Postfix) with ESMTP id AC3AD3A6C3C for ; Thu, 17 Feb 2011 08:36:01 -0800 (PST) Received: from smtpksrv1.mitre.org (localhost.localdomain [127.0.0.1]) by localhost (Postfix) with SMTP id A1DBC21B17F3; Thu, 17 Feb 2011 11:36:32 -0500 (EST) Received: from imchub2.MITRE.ORG (imchub2.mitre.org [129.83.29.74]) by smtpksrv1.mitre.org (Postfix) with ESMTP id 8F71321B17D3; Thu, 17 Feb 2011 11:36:32 -0500 (EST) Received: from IMCMBX3.MITRE.ORG ([129.83.29.206]) by imchub2.MITRE.ORG ([129.83.29.74]) with mapi; Thu, 17 Feb 2011 11:36:32 -0500 From: "Heinbockel, Bill" To: Jeroen Massar Date: Thu, 17 Feb 2011 11:35:19 -0500 Thread-Topic: [Syslog] draft-cloud-log-00 / CEE - why not IPFIX? Thread-Index: AcvNxSVcMeG8rWBOQCCWiPHcleUWdwA+p2yQ Message-ID: <93ED0A84F9A1D74FA65021D940AA5884054477F158@IMCMBX3.MITRE.ORG> References: <4D5A60C8.3090000@unfix.org> <93ED0A84F9A1D74FA65021D940AA588405446C41F9@IMCMBX3.MITRE.ORG> <4D5BA85B.7040007@unfix.org> In-Reply-To: <4D5BA85B.7040007@unfix.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: multipart/signed; micalg=SHA1; protocol="application/x-pkcs7-signature"; boundary="----=_NextPart_000_0033_01CBCE96.C12A9240" MIME-Version: 1.0 Cc: Sam Johnston , Event Expression , "syslog@ietf.org" , Common Subject: Re: [Syslog] draft-cloud-log-00 / CEE - why not IPFIX? X-BeenThere: syslog@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Security Issues in Network Event Logging List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Feb 2011 16:36:04 -0000 ------=_NextPart_000_0033_01CBCE96.C12A9240 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Thanks for that, I did not realize that IPFIX had been extended beyond its netflow past. I don't have the time now, but I am interested in looking into it further. It does kind of remind me of ASN.1/SNMP, where we need to worry about the names/OID translation Also, a lot of vendors and users seem to prefer the ease of text-based protocols like Syslog for logging. I am not saying this is good or bad, but it seems to be the sweetspot -- binary is not natively human readable and XML has too much overhead. William Heinbockel The MITRE Corporation >-----Original Message----- >From: Jeroen Massar [mailto:jeroen@unfix.org] >Sent: Wednesday, 16 February 2011 05:35 >To: Heinbockel, Bill >Cc: syslog@ietf.org; Chris Lonvick; Gene Golovinsky; Sam Johnston; Common >Event Expression >Subject: Re: [Syslog] draft-cloud-log-00 / CEE - why not IPFIX? > >On 2011-02-16 06:21, Heinbockel, Bill wrote: >> From what I understand, IPFIX is for expression of IP flows from network >sensing >> devices. > >For a short bit forget about the history of IPFIX, it indeed comes from >NetFlow, and thus is used quite in a network centric way, but >effectively it is a structured streaming data format. > >> Could you please explain how IPFIX is relevant to event and cloud logging >data? >> I understand how CEE and IPFIX may overlap for describing networking >events, but >> it is unclear to me how IPFIX could handle things like Windows Event Logs >and >> RHEL audit logs. > >There are two parts to IPFIX: Templates + Data > >The template describes how the data looks like, for instance, lets take >an Apache CLF log entry: > >66.249.66.174 - - [16/Feb/2011:10:48:11 +0100] "GET /robots.txt >HTTP/1.1" 200 2629 "-" "Googlebot-Image/0" > >We can make an IPFIX template for that > >[ > {4, IPv4_SRC }, > {4, TIMESTAMP}, > {4, HTTP_METHOD}, > {v, URL}, > {v, HTTP_PROTOCOL}, > {2, HTTP_RESULT}, > {8, OCTETS}, > {v, HTTP_REFER}, > {v, HTTP_USERAGENT}, >] > >The 'v' markers indicate variable fieldlengths, the others indicates the >number of bytes such a field takes. The data is then just encoded in the >above format, presto. > >The above is a simple example, one can also have repeating lists and of >course you could make a variable template which just includes the fields >that you actually want to look at or you could already do some >aggregation and add other fields. Templates are only sent every now and >then, as they should not change. The data is the important bit. > >The fieldnames are actually numbers in the data, thus very compact, and >are mapped to descriptions, data types etc, per a nice XML file > http://www.iana.org/assignments/ipfix/ipfix.xml (or .xhtml or .txt for >a more human readable version ;) for the official IANA list and with the >help of Enterprise IDs any others can easily be added. > >The big advantage is that you can more or less do static templates if >you want and you only need one single parser on the collector side, thus >one does not have to create another parser and collector again for >decoding other protocols, just one, the IPFIX one, and you can optimize >that really well for all kinds of scenarios. > >Greets, > Jeroen ------=_NextPart_000_0033_01CBCE96.C12A9240 Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIKxTCCA2Qw ggJMoAMCAQICAQEwDQYJKoZIhvcNAQEFBQAwWjESMBAGA1UEChMJbWl0cmUub3JnMR4wHAYDVQQL ExVDZXJ0aWZpY2F0ZSBBdXRob3JpdHkxJDAiBgNVBAMTG01JVFJFIENvcnBvcmF0aW9uIFJvb3Qg Q0EtMTAeFw0wNjA2MDEwNDAwMDBaFw0xODA2MDEwNDAwMDBaMFoxEjAQBgNVBAoTCW1pdHJlLm9y ZzEeMBwGA1UECxMVQ2VydGlmaWNhdGUgQXV0aG9yaXR5MSQwIgYDVQQDExtNSVRSRSBDb3Jwb3Jh dGlvbiBSb290IENBLTEwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCva1qWPZiEJv5v MtCbjt0cTu0Nbn15Q1cKqQBXKi8VSH9zZPmPxfWizJJ7JSqFJ5/sLUz3NsnUVjpLYBdFcxNXnOLj XtmDPFOewm5T98NZc9wRRCiDzt4f8qsHFI19ShPiK3cN5UqtJf+i66QVLA1S6CNL6o2eGAsAl5Wn xwOh2BfcWU5fNlHDVc9KKAlDDWpHjC2LLHAUbLP4ZzMIJKcLgLKFMtgM2AEfaSHzmi7WUdUHRCtC blrF7qzPsy/jBLFrr8VcX+mb7saq95pEOilgcix0/naW7kJfM5ph7UBB+S1O/OhH+ZjQ4MjWnwE8 A/YDrQx1OVLAOi29Bsho/l8lAgMBAAGjNTAzMBIGA1UdEwEB/wQIMAYBAf8CAQMwHQYDVR0OBBYE FMdwUQDYTf7kAdRolsU9n5qX/nQvMA0GCSqGSIb3DQEBBQUAA4IBAQAa+fVfCljimBlcfWwkfJXu XNKWun9xloFKjnq6SPGgAIKi5LUDil60a0NaNGoGSO3I1xzYt7ncayh21qXulcVTDFqubSJdv51a HTuJYcYUX72LN/gSq03UVLBCJzYm7ZLUlkb2YLo7xUeZ3coLFcT5AHR36kjG4cYHqXgH0liBl8jx pN0gwgaci4sgPLUj1w4t8zoKH+zxGFwXwTP/P+etQqiJZ5T00fLLm5kz9mmnxxmmIvUGNdsCAhGh dnF24pcrR43LNgyOBJ9DPUHBNq3kUQRO48WBKxBxflOtKzsICx/HEtIABcZn7deADHcY9spULZfB nQYdEpyz5tgh7Y2qMIIDcTCCAlmgAwIBAgICTU0wDQYJKoZIhvcNAQEFBQAwXTESMBAGA1UEChMJ bWl0cmUub3JnMR4wHAYDVQQLExVDZXJ0aWZpY2F0ZSBBdXRob3JpdHkxJzAlBgNVBAMTHk1JVFJF IENvcnBvcmF0aW9uIFByaW1hcnkgQ0EtMTAeFw0xMDEyMjQxNzAyMDhaFw0xMjA2MDMxNzEzMjJa MGExEjAQBgNVBAoTCW1pdHJlLm9yZzEPMA0GA1UECxMGUGVvcGxlMRowGAYKCZImiZPyLGQBARMK aGVpbmJvY2tlbDEeMBwGA1UEAxMVSGVpbmJvY2tlbCBXaWxsaWFtIEouMIGfMA0GCSqGSIb3DQEB AQUAA4GNADCBiQKBgQD1fnZZxPrtC6BEKYX33ph9tumgzYQTNNIWCUCR+h0AudF0Pxb+wRDeDZps VIylRejdkCQ4td+J2WZI/QdeqMREpYWHVDR64TQ7kgIImOPgzYa5Ttu7byFBtGkaBJnRvcqgK23j pBiPCfIY6k0GXGdQjKpEn5kTVv1zm2HpyPZcewIDAQABo4G6MIG3MA4GA1UdDwEB/wQEAwIF4DAd BgNVHQ4EFgQU6RMrsHGS4C3d+ljqPBM8ENcv+lIwHwYDVR0jBBgwFoAUh7QPSI1iM0LBLVEaSB7C nrsKsa0wRAYDVR0fBD0wOzA5oDegNYYzaHR0cDovL3d3dy5taXRyZS5vcmcvdGVjaC9taWkvcGtp L2NhMV9taXRyZV9vcmcuY3JsMB8GA1UdEQQYMBaBFGhlaW5ib2NrZWxAbWl0cmUub3JnMA0GCSqG SIb3DQEBBQUAA4IBAQAvr07pYzYYAx+28PYGnbmgA32RiE8NAzc64pGLXDNa34xFWTjDFIY9levD TQhoZ1fXo1dNtXvB45Ilh+Jsf3+9XTe2vYNf8vcEnxu+buddzsdxz2U2KvGbQNlcxat+zEtY0eWn qZqmMXbhyTiVyziGGJSWnFKDyFOJFkq3cuK29IOYU+9/vZDQE2QBAzCViMvWcAGs6QfTlWS4gqV3 kdkBAmqeV6MgFr+BhUPjjdmYYgSU0CPH6ql0j1j/y2Yd4IK/Er+SFaU6T/ri+7ytSHDjAI+Cub5x Pv8AiiaErv7KH+3RhG+suNJ/QlNjzuXLgr5RcufHUt38MT5ibca6xBebMIID5DCCAsygAwIBAgIB BTANBgkqhkiG9w0BAQUFADBaMRIwEAYDVQQKEwltaXRyZS5vcmcxHjAcBgNVBAsTFUNlcnRpZmlj YXRlIEF1dGhvcml0eTEkMCIGA1UEAxMbTUlUUkUgQ29ycG9yYXRpb24gUm9vdCBDQS0xMB4XDTA2 MDYwMzE3MTMyMloXDTEyMDYwMzE3MTMyMlowXTESMBAGA1UEChMJbWl0cmUub3JnMR4wHAYDVQQL ExVDZXJ0aWZpY2F0ZSBBdXRob3JpdHkxJzAlBgNVBAMTHk1JVFJFIENvcnBvcmF0aW9uIFByaW1h cnkgQ0EtMTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMjwe1ZdEIKoELdLvENnbkbO 3mVD54ZX5SjxTzFxhvoqhqSJmLOp32xT7J9KvBapJRbJv1BFdiSUN3O0q8H66nvQqwm1RYe+tTsW SO351FolGrDT9iDRtfWgH60PCKCbABLRsx3iGnEvjOQjeAtMn1AugqQWU3PWZXYy1Grbya+NOytY vu1p60bDFSoSAn4Jojv14lUfWHl8s3kahay8umLSQibiXdEEX8CrSkaXpOaFOuyM6+wpRw7TybO1 Dk4zGAUtn0887QvsMDk6evgN2WxMprkHAGUcJhpI1QXtkfDIl9ukdNiIoM/vdN2QC/+m6b8dA55K 5UdlBa9SgRnwapkCAwEAAaOBsTCBrjASBgNVHRMBAf8ECDAGAQH/AgECMA4GA1UdDwEB/wQEAwIB hjAdBgNVHQ4EFgQUh7QPSI1iM0LBLVEaSB7CnrsKsa0wHwYDVR0jBBgwFoAUx3BRANhN/uQB1GiW xT2fmpf+dC8wSAYDVR0fBEEwPzA9oDugOYY3aHR0cDovL3d3dy5taXRyZS5vcmcvdGVjaC9taWkv cGtpL3Jvb3RjYTFfbWl0cmVfb3JnLmNybDANBgkqhkiG9w0BAQUFAAOCAQEATW5u664p7N0iAj27 Xl/akjdfkSQpaosf6cNyAHu7utCytFfY1WfRNmvnNDGYkqI3XMFOa18SNjiNsMCH+sFQaO+oyDnP iIkEZQvlfGGrRpqIm6j//Fgz85bnf1kAM5I61Np7ofCnciRvp9ZB/+u+9i262tgiJPJrvBcqXmge T9riCc3RPjxqPNmYslOvNLpIifchelJhF7nIge+7RkAUcTJenj8yKwK0J3+PEpgYRQ+V2C62rnjo huxPgMw/fYoNTOlh3MVl7adwyK1ahPw2a9eOjSWglqoPTaBNeHJqRJZZ6Vi7S55+VAWCfkAqM5m3 tUiVzjsp2dFcTJxnYezaoDGCAr0wggK5AgEBMGMwXTESMBAGA1UEChMJbWl0cmUub3JnMR4wHAYD VQQLExVDZXJ0aWZpY2F0ZSBBdXRob3JpdHkxJzAlBgNVBAMTHk1JVFJFIENvcnBvcmF0aW9uIFBy aW1hcnkgQ0EtMQICTU0wCQYFKw4DAhoFAKCCAbAwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAc BgkqhkiG9w0BCQUxDxcNMTEwMjE3MTYzNTE5WjAjBgkqhkiG9w0BCQQxFgQU4hk/CwNI23V0m+GV fEcPtyJUX6gwZwYJKoZIhvcNAQkPMVowWDAKBggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYI KoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgwBwYFKw4DAhowCgYIKoZIhvcNAgUw cgYJKwYBBAGCNxAEMWUwYzBdMRIwEAYDVQQKEwltaXRyZS5vcmcxHjAcBgNVBAsTFUNlcnRpZmlj YXRlIEF1dGhvcml0eTEnMCUGA1UEAxMeTUlUUkUgQ29ycG9yYXRpb24gUHJpbWFyeSBDQS0xAgJN TTB0BgsqhkiG9w0BCRACCzFloGMwXTESMBAGA1UEChMJbWl0cmUub3JnMR4wHAYDVQQLExVDZXJ0 aWZpY2F0ZSBBdXRob3JpdHkxJzAlBgNVBAMTHk1JVFJFIENvcnBvcmF0aW9uIFByaW1hcnkgQ0Et MQICTU0wDQYJKoZIhvcNAQEBBQAEgYC2urY2WkzIQJpCWNNiODQrvmsz0R7mUIWEnMsM2ADRUsWQ wKxBXJ91tSwU2jGvVttSbalKseabyyTqDw3OzYhhM7w9fUexKlPZZJTCjJcXzqsTdaMjZ+oJm2+K RlzG2PLlxMatU5HdPX/ZB20RPu21VxWLcYy4Mmq0n4j9XsTekwAAAAAAAA== ------=_NextPart_000_0033_01CBCE96.C12A9240-- From schlitt@world.std.com Thu Feb 17 09:20:47 2011 Return-Path: X-Original-To: syslog@core3.amsl.com Delivered-To: syslog@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 412ED3A6D1C for ; Thu, 17 Feb 2011 09:20:47 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.599 X-Spam-Level: X-Spam-Status: No, score=-3.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nIykDw-v6GUy for ; Thu, 17 Feb 2011 09:20:45 -0800 (PST) Received: from TheWorld.com (pcls4.std.com [192.74.137.84]) by core3.amsl.com (Postfix) with ESMTP id 9A04F3A6CB8 for ; Thu, 17 Feb 2011 09:20:45 -0800 (PST) Received: from shell.TheWorld.com (root@shell01.TheWorld.com [192.74.137.71]) by TheWorld.com (8.13.6/8.13.6) with ESMTP id p1HHL3M4031579; Thu, 17 Feb 2011 12:21:05 -0500 Received: from shell01.TheWorld.com (localhost.theworld.com [127.0.0.1]) by shell.TheWorld.com (8.13.6/8.12.8) with ESMTP id p1HHL2LK5838276; Thu, 17 Feb 2011 12:21:02 -0500 (EST) Received: from localhost (schlitt@localhost) by shell01.TheWorld.com (8.13.6/8.13.6/Submit) with ESMTP id p1HHKxsS5828902; Thu, 17 Feb 2011 12:20:59 -0500 (EST) X-Authentication-Warning: shell01.TheWorld.com: schlitt owned process doing -bs Date: Thu, 17 Feb 2011 12:20:58 -0500 From: Dan Schlitt X-X-Sender: schlitt@shell01.TheWorld.com To: "Heinbockel, Bill" In-Reply-To: <93ED0A84F9A1D74FA65021D940AA5884054477F158@IMCMBX3.MITRE.ORG> Message-ID: References: <4D5A60C8.3090000@unfix.org> <93ED0A84F9A1D74FA65021D940AA588405446C41F9@IMCMBX3.MITRE.ORG> <4D5BA85B.7040007@unfix.org> <93ED0A84F9A1D74FA65021D940AA5884054477F158@IMCMBX3.MITRE.ORG> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Sam Johnston , Event Expression , "syslog@ietf.org" , Common@core3.amsl.com Subject: Re: [Syslog] draft-cloud-log-00 / CEE - why not IPFIX? X-BeenThere: syslog@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Security Issues in Network Event Logging List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Feb 2011 17:20:47 -0000 On this topic it might be well to look at Tom Limoncelli's article in the February Communications of the ACM. He is highly respected in the system administrator community and speaks for many of us. /dan -- Dan Schlitt schlitt@world.std.com On Thu, 17 Feb 2011, Heinbockel, Bill wrote: > Thanks for that, I did not realize that IPFIX had been extended beyond its netflow > past. > > I don't have the time now, but I am interested in looking into it further. It does > kind of remind me of ASN.1/SNMP, where we need to worry about the names/OID > translation > > Also, a lot of vendors and users seem to prefer the ease of text-based protocols > like Syslog for logging. I am not saying this is good or bad, but it seems to be > the sweetspot -- binary is not natively human readable and XML has too much > overhead. > > > William Heinbockel > The MITRE Corporation > > >> -----Original Message----- >> From: Jeroen Massar [mailto:jeroen@unfix.org] >> Sent: Wednesday, 16 February 2011 05:35 >> To: Heinbockel, Bill >> Cc: syslog@ietf.org; Chris Lonvick; Gene Golovinsky; Sam Johnston; Common >> Event Expression >> Subject: Re: [Syslog] draft-cloud-log-00 / CEE - why not IPFIX? >> >> On 2011-02-16 06:21, Heinbockel, Bill wrote: >>> From what I understand, IPFIX is for expression of IP flows from network >> sensing >>> devices. >> >> For a short bit forget about the history of IPFIX, it indeed comes from >> NetFlow, and thus is used quite in a network centric way, but >> effectively it is a structured streaming data format. >> >>> Could you please explain how IPFIX is relevant to event and cloud logging >> data? >>> I understand how CEE and IPFIX may overlap for describing networking >> events, but >>> it is unclear to me how IPFIX could handle things like Windows Event Logs >> and >>> RHEL audit logs. >> >> There are two parts to IPFIX: Templates + Data >> >> The template describes how the data looks like, for instance, lets take >> an Apache CLF log entry: >> >> 66.249.66.174 - - [16/Feb/2011:10:48:11 +0100] "GET /robots.txt >> HTTP/1.1" 200 2629 "-" "Googlebot-Image/0" >> >> We can make an IPFIX template for that >> >> [ >> {4, IPv4_SRC }, >> {4, TIMESTAMP}, >> {4, HTTP_METHOD}, >> {v, URL}, >> {v, HTTP_PROTOCOL}, >> {2, HTTP_RESULT}, >> {8, OCTETS}, >> {v, HTTP_REFER}, >> {v, HTTP_USERAGENT}, >> ] >> >> The 'v' markers indicate variable fieldlengths, the others indicates the >> number of bytes such a field takes. The data is then just encoded in the >> above format, presto. >> >> The above is a simple example, one can also have repeating lists and of >> course you could make a variable template which just includes the fields >> that you actually want to look at or you could already do some >> aggregation and add other fields. Templates are only sent every now and >> then, as they should not change. The data is the important bit. >> >> The fieldnames are actually numbers in the data, thus very compact, and >> are mapped to descriptions, data types etc, per a nice XML file >> http://www.iana.org/assignments/ipfix/ipfix.xml (or .xhtml or .txt for >> a more human readable version ;) for the official IANA list and with the >> help of Enterprise IDs any others can easily be added. >> >> The big advantage is that you can more or less do static templates if >> you want and you only need one single parser on the collector side, thus >> one does not have to create another parser and collector again for >> decoding other protocols, just one, the IPFIX one, and you can optimize >> that really well for all kinds of scenarios. >> >> Greets, >> Jeroen > From gene@alertlogic.com Thu Feb 17 10:47:04 2011 Return-Path: X-Original-To: syslog@core3.amsl.com Delivered-To: syslog@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id A16573A6D6C for ; Thu, 17 Feb 2011 10:47:04 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.265 X-Spam-Level: X-Spam-Status: No, score=-3.265 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, IP_NOT_FRIENDLY=0.334, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id D1QYqDnzFIZH for ; Thu, 17 Feb 2011 10:47:03 -0800 (PST) Received: from smtp205.dfw.emailsrvr.com (smtp205.dfw.emailsrvr.com [67.192.241.205]) by core3.amsl.com (Postfix) with ESMTP id 672793A6D31 for ; Thu, 17 Feb 2011 10:47:03 -0800 (PST) Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp20.relay.dfw1a.emailsrvr.com (SMTP Server) with ESMTP id DE38B2581AF; Thu, 17 Feb 2011 13:47:34 -0500 (EST) X-Virus-Scanned: OK Received: from smtp192.mex07a.mlsrvr.com (smtp192.mex07a.mlsrvr.com [67.192.133.192]) by smtp20.relay.dfw1a.emailsrvr.com (SMTP Server) with ESMTPS id BD318258118; Thu, 17 Feb 2011 13:47:34 -0500 (EST) Received: from 34093-MBX-C01.mex07a.mlsrvr.com ([192.168.1.63]) by DFW1HUB12.mex07a.mlsrvr.com ([192.168.1.208]) with mapi; Thu, 17 Feb 2011 12:47:34 -0600 From: Gene Golovinsky To: Dan Schlitt , "Heinbockel, Bill" , "clouds@ietf.org" Date: Thu, 17 Feb 2011 12:47:33 -0600 Thread-Topic: [Syslog] draft-cloud-log-00 / CEE - why not IPFIX? Thread-Index: AcvOxxvimbNTmqoKTxOIAS4W5JyZrQAC/PPQ Message-ID: References: <4D5A60C8.3090000@unfix.org> <93ED0A84F9A1D74FA65021D940AA588405446C41F9@IMCMBX3.MITRE.ORG> <4D5BA85B.7040007@unfix.org> <93ED0A84F9A1D74FA65021D940AA5884054477F158@IMCMBX3.MITRE.ORG> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: Sam Johnston , Event Expression , "syslog@ietf.org" , "Common@core3.amsl.com" Subject: Re: [Syslog] draft-cloud-log-00 / CEE - why not IPFIX? X-BeenThere: syslog@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Security Issues in Network Event Logging List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Feb 2011 18:47:04 -0000 Is there a link to the article? --Gene -----Original Message----- From: syslog-bounces@ietf.org [mailto:syslog-bounces@ietf.org] On Behalf Of= Dan Schlitt Sent: Thursday, February 17, 2011 11:21 AM To: Heinbockel, Bill Cc: Sam Johnston; Event Expression; syslog@ietf.org; Common@core3.amsl.com Subject: Re: [Syslog] draft-cloud-log-00 / CEE - why not IPFIX? On this topic it might be well to look at Tom Limoncelli's article in the F= ebruary Communications of the ACM. He is highly respected in the system adm= inistrator community and speaks for many of us. /dan --=20 Dan Schlitt schlitt@world.std.com On Thu, 17 Feb 2011, Heinbockel, Bill wrote: > Thanks for that, I did not realize that IPFIX had been extended beyond=20 > its netflow past. > > I don't have the time now, but I am interested in looking into it=20 > further. It does kind of remind me of ASN.1/SNMP, where we need to=20 > worry about the names/OID translation > > Also, a lot of vendors and users seem to prefer the ease of text-based=20 > protocols like Syslog for logging. I am not saying this is good or=20 > bad, but it seems to be the sweetspot -- binary is not natively human=20 > readable and XML has too much overhead. > > > William Heinbockel > The MITRE Corporation > > >> -----Original Message----- >> From: Jeroen Massar [mailto:jeroen@unfix.org] >> Sent: Wednesday, 16 February 2011 05:35 >> To: Heinbockel, Bill >> Cc: syslog@ietf.org; Chris Lonvick; Gene Golovinsky; Sam Johnston;=20 >> Common Event Expression >> Subject: Re: [Syslog] draft-cloud-log-00 / CEE - why not IPFIX? >> >> On 2011-02-16 06:21, Heinbockel, Bill wrote: >>> From what I understand, IPFIX is for expression of IP flows from=20 >>> network >> sensing >>> devices. >> >> For a short bit forget about the history of IPFIX, it indeed comes=20 >> from NetFlow, and thus is used quite in a network centric way, but=20 >> effectively it is a structured streaming data format. >> >>> Could you please explain how IPFIX is relevant to event and cloud=20 >>> logging >> data? >>> I understand how CEE and IPFIX may overlap for describing networking >> events, but >>> it is unclear to me how IPFIX could handle things like Windows Event=20 >>> Logs >> and >>> RHEL audit logs. >> >> There are two parts to IPFIX: Templates + Data >> >> The template describes how the data looks like, for instance, lets=20 >> take an Apache CLF log entry: >> >> 66.249.66.174 - - [16/Feb/2011:10:48:11 +0100] "GET /robots.txt=20 >> HTTP/1.1" 200 2629 "-" "Googlebot-Image/0" >> >> We can make an IPFIX template for that >> >> [ >> {4, IPv4_SRC }, >> {4, TIMESTAMP}, >> {4, HTTP_METHOD}, >> {v, URL}, >> {v, HTTP_PROTOCOL}, >> {2, HTTP_RESULT}, >> {8, OCTETS}, >> {v, HTTP_REFER}, >> {v, HTTP_USERAGENT}, >> ] >> >> The 'v' markers indicate variable fieldlengths, the others indicates=20 >> the number of bytes such a field takes. The data is then just encoded=20 >> in the above format, presto. >> >> The above is a simple example, one can also have repeating lists and=20 >> of course you could make a variable template which just includes the=20 >> fields that you actually want to look at or you could already do some=20 >> aggregation and add other fields. Templates are only sent every now=20 >> and then, as they should not change. The data is the important bit. >> >> The fieldnames are actually numbers in the data, thus very compact,=20 >> and are mapped to descriptions, data types etc, per a nice XML file=20 >> http://www.iana.org/assignments/ipfix/ipfix.xml (or .xhtml or .txt=20 >> for a more human readable version ;) for the official IANA list and=20 >> with the help of Enterprise IDs any others can easily be added. >> >> The big advantage is that you can more or less do static templates if=20 >> you want and you only need one single parser on the collector side,=20 >> thus one does not have to create another parser and collector again=20 >> for decoding other protocols, just one, the IPFIX one, and you can=20 >> optimize that really well for all kinds of scenarios. >> >> Greets, >> Jeroen > _______________________________________________ Syslog mailing list Syslog@ietf.org https://www.ietf.org/mailman/listinfo/syslog