Subject: RE: [ipfix-eval] Split reporting, memory consumption, and "snapshot" attributes [was: Discussion of Candidate Protocols]

From: "Carter Bullard" <[email-masked]>
Date: Fri, 25 Oct 2002 19:40:45 -0400

Hey Tal,
   Yes, this is but one of the approaches that can be
used.  My example was intended to show how a simple
apparently benign reported value, TTL, has complex
reporting issues that must be considered in order to
"do it right".  I'm confident that the IPFIX WG charter
doesn't have this work in its scope.

   I agree that sticking to the WG's simple charter
is the key, and concepts like split reporting, memory
consumption, and "snapshot" attributes, are not
in scope.

   Being able to transport existing IP flow data, is, in
my humble opinion, the most important thing IPFIX needs
to do.  But, if all it does is standardize one existing
protocol, then its work is pretty much useless to the
industry.

Carter

> -----Original Message-----
> From: Tal Givoly [mailto:[email-masked]] 
> Sent: Thursday, October 24, 2002 11:01 PM
> To: [email-masked]; 'Simon Leinen'; [email-masked]
> Cc: 'Michael MacFaden'; [email-masked]
> Subject: RE: [ipfix-eval] Split reporting, memory 
> consumption, and "snapshot" attributes [was: Discussion of 
> Candidate Protocols]
> 
> 
> Carter,
> 
> Your point is well taken. In our system, we have unique keys 
> to describe the
> values with different meaning to identify the type of 
> aggregation performed
> (key, first, last, max, min, sum, etc.). In our user presentation, we
> clearly distinguish between the value by denoting it with a 
> prefix modifier
> (when presented in textual form to a user, for instance). There is a
> different key to denote "Source AS" and there is another key to denote
> "First Source AS". When we accumulate "Bytes" we actually say 
> "Sum of Bytes"
> rather than leaving this implicit.
> 
> The same type of issue may exist when defining units of data 
> reported (some
> devices/cards report in K bytes, etc.). Ambiguity of the meaning of an
> attribute can be eliminated by denoting them with different 
> key attribute
> IDs.
> 
> Would you agree that using methods like the one mentioned above, it is
> possible disambiguate the data rather than settle for a 
> situation where the
> precise meaning cannot be correct/accurate.
> 
> Tal
> 
> -----Original Message-----
> From: Carter Bullard [mailto:[email-masked]]
> Sent: Thursday, October 24, 2002 5:55 PM
> To: 'Tal Givoly'; 'Simon Leinen'; [email-masked]
> Cc: 'Michael MacFaden'; [email-masked]
> Subject: RE: [ipfix-eval] Split reporting, memory consumption, and
> "snapshot" attributes [was: Discussion of Candidate Protocols]
> 
> 
> Gentle people,
>    Everyone seems to take it for granted that its cool to bind
> non-key attributes to aggregated data, but there are some
> relational algebraic gottchas that you have to solve if
> you want IPFIX data to be "correct", and Tal has touched on
> one of them.
> 
>    Split reporting assumes that there is a reason to report
> transitions in non-key attribute data, but is there really
> a relational algebraic basis to justify "going to the trouble"?
> It really hinges on the concept of data "correctness", and
> whether non-key attributes can every achieve any level of
> correctness in IPFIX data.  Let me use an example to demonstrate
> what I'm getting at.
> 
>    TTL will most likely be a perennial non-key attribute that
> will be directly supported in the IPFIX data model.  It is one
> of the simplest of the non-key attributes to deal with, because
> the value is contained in the actual observed data, so the value
> has some "correctness".  But, TTL can change often during the
> brief life of a flow, and as a result, when an IPFIX device
> reports a single value for TTL in a flow report, the value has
> only limited applicability as an attribute.  It doesn't really
> apply to the whole flow, it is safe to say that it applies to
> at most, only one packet of the flow.
> 
>    Now reporting this non-key attribute is very useful, and it
> should be done, but attempting to engineer some level of
> "correctness" into this value would definitely be futile.  The
> reason is that as transitions in non-key attributes reach a
> certain level, and a pretty low level at that, the attempt to
> maintain attribute "correctness" overwhelms the system.  In
> the analysis, this is the general case for any non-key attribute
> that changes, even AS number.
> 
>    So what to do?  My suggestion is to understand that non-key
> attributes are not relational attributes, and as such little
> engineering should be done to try to improve on the data's
> "correctness".  They are interesting bits of information and can
> be of value using appropriate statistical methods.  Maybe just
> put a bit in the record that indicates if the value is correct
> or not is all you need to do.
> 
> I hope this has been of interest.
> 
> Carter
> 
> 
> 
> 
> 
> 
> 
> 
> > -----Original Message-----
> > From: majordomo listserver
> > [mailto:[email-masked]] On Behalf Of Tal Givoly
> > Sent: Thursday, October 24, 2002 7:24 PM
> > To: Simon Leinen; [email-masked]
> > Cc: [email-masked]; 'Michael MacFaden';
> > [email-masked]
> > Subject: RE: [ipfix-eval] Split reporting, memory
> > consumption, and "snapshot" attributes [was: Discussion of
> > Candidate Protocols]
> >
> >
> > Simon/Paul,
> >
> > As there's been some debate regarding terminology, may I lend
> > a slightly DB-centric perspective: In SQL terminology, the
> > attributes that are shared would be identified by "First()"
> > aggregate functions while most aggregates would be result of
> > aggregate functions such as "Sum()" functions.
> >
> > Notice that if a device does employ "split reporting",
> > obviously the complexity of the collector is increased and it
> > must create a "join" between the two flows.
> >
> > Split reporting would no longer be FNF, but instead would be
> > 3NF - increasing complexity in the metering process as well,
> > if this is a requirement.
> >
> > In order to keep things simple and more broadly accepted, it
> > seems to me that leaving things in FNF is more likely to be
> > adopted and implemented - a desirable trait of any standard.
> >
> > My $0.02.
> >
> > Tal
> >
> >
> > --
> > Help        mailto:[email-masked] and say "help"
> > in message body
> > Unsubscribe mailto:[email-masked] and say
> > "unsubscribe ipfix" in message body
> > Archive     http://ipfix.doit.wisc.edu/archive/
> >
> 
> 

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/