Subject: RE: [ipfix-eval] Split reporting, memory consumption, and "snapshot" attributes [was: Discussion of Candidate Protocols]

From: "Carter Bullard" <[email-masked]>
Date: Thu, 24 Oct 2002 20:55:14 -0400

Gentle people,
   Everyone seems to take it for granted that its cool to bind
non-key attributes to aggregated data, but there are some
relational algebraic gottchas that you have to solve if
you want IPFIX data to be "correct", and Tal has touched on
one of them.  

   Split reporting assumes that there is a reason to report
transitions in non-key attribute data, but is there really
a relational algebraic basis to justify "going to the trouble"?
It really hinges on the concept of data "correctness", and
whether non-key attributes can every achieve any level of
correctness in IPFIX data.  Let me use an example to demonstrate
what I'm getting at.

   TTL will most likely be a perennial non-key attribute that
will be directly supported in the IPFIX data model.  It is one
of the simplest of the non-key attributes to deal with, because
the value is contained in the actual observed data, so the value
has some "correctness".  But, TTL can change often during the
brief life of a flow, and as a result, when an IPFIX device
reports a single value for TTL in a flow report, the value has
only limited applicability as an attribute.  It doesn't really
apply to the whole flow, it is safe to say that it applies to
at most, only one packet of the flow.

   Now reporting this non-key attribute is very useful, and it
should be done, but attempting to engineer some level of
"correctness" into this value would definitely be futile.  The
reason is that as transitions in non-key attributes reach a
certain level, and a pretty low level at that, the attempt to
maintain attribute "correctness" overwhelms the system.  In
the analysis, this is the general case for any non-key attribute
that changes, even AS number.

   So what to do?  My suggestion is to understand that non-key
attributes are not relational attributes, and as such little
engineering should be done to try to improve on the data's
"correctness".  They are interesting bits of information and can
be of value using appropriate statistical methods.  Maybe just
put a bit in the record that indicates if the value is correct
or not is all you need to do.

I hope this has been of interest.

Carter

> -----Original Message-----
> From: majordomo listserver 
> [mailto:[email-masked]] On Behalf Of Tal Givoly
> Sent: Thursday, October 24, 2002 7:24 PM
> To: Simon Leinen; [email-masked]
> Cc: [email-masked]; 'Michael MacFaden'; 
> [email-masked]
> Subject: RE: [ipfix-eval] Split reporting, memory 
> consumption, and "snapshot" attributes [was: Discussion of 
> Candidate Protocols]
> 
> 
> Simon/Paul,
> 
> As there's been some debate regarding terminology, may I lend 
> a slightly DB-centric perspective: In SQL terminology, the 
> attributes that are shared would be identified by "First()" 
> aggregate functions while most aggregates would be result of 
> aggregate functions such as "Sum()" functions.
> 
> Notice that if a device does employ "split reporting", 
> obviously the complexity of the collector is increased and it 
> must create a "join" between the two flows.
> 
> Split reporting would no longer be FNF, but instead would be 
> 3NF - increasing complexity in the metering process as well, 
> if this is a requirement.
> 
> In order to keep things simple and more broadly accepted, it 
> seems to me that leaving things in FNF is more likely to be 
> adopted and implemented - a desirable trait of any standard.
> 
> My $0.02.
> 
> Tal
> 
> 
> --
> Help        mailto:[email-masked] and say "help" 
> in message body
> Unsubscribe mailto:[email-masked] and say 
> "unsubscribe ipfix" in message body
> Archive     http://ipfix.doit.wisc.edu/archive/
> 

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: "Tal Givoly" <[email-masked]>
Date: Thu, 24 Oct 2002 20:01:09 -0700

Carter,

Your point is well taken. In our system, we have unique keys to describe the
values with different meaning to identify the type of aggregation performed
(key, first, last, max, min, sum, etc.). In our user presentation, we
clearly distinguish between the value by denoting it with a prefix modifier
(when presented in textual form to a user, for instance). There is a
different key to denote "Source AS" and there is another key to denote
"First Source AS". When we accumulate "Bytes" we actually say "Sum of Bytes"
rather than leaving this implicit.

The same type of issue may exist when defining units of data reported (some
devices/cards report in K bytes, etc.). Ambiguity of the meaning of an
attribute can be eliminated by denoting them with different key attribute
IDs.

Would you agree that using methods like the one mentioned above, it is
possible disambiguate the data rather than settle for a situation where the
precise meaning cannot be correct/accurate.

Tal

-----Original Message-----
From: Carter Bullard [mailto:[email-masked]]
Sent: Thursday, October 24, 2002 5:55 PM
To: 'Tal Givoly'; 'Simon Leinen'; [email-masked]
Cc: 'Michael MacFaden'; [email-masked]
Subject: RE: [ipfix-eval] Split reporting, memory consumption, and
"snapshot" attributes [was: Discussion of Candidate Protocols]

Gentle people,

Carter