Subject: RE: [ipfix-eval] Discussion of Candidate Protocols

From: "Carter Bullard" <[email-masked]>
Date: Thu, 3 Oct 2002 20:07:35 -0400

Hey Mike,
   Thanks, but I want to note that netflow cannot support
much in the way of extensions, so simply describing existing
practice and defining a standard to support those observations
is not going to do the job for my customers.

   In writing this response I found myself criticizing LFAP
quite a bit, and I didn't really want to do that, but I'm going to
leave some of my issues in hopes that it will help the process.
I hope that some do find it useful, and I welcome any comment.

   One question.  None of the candidates are perfect.  If we
adopt one, what will be the process that we go through to
'correct' any issues?  I'm sure it has been described, but
could someone paraphrase?

Thanks,

Carter

Comments on LFAP as IPFIX candidate.

I don't see that LFAP is a solution to the flow record transport
problems that I worry about, which is the efficient, reliable
and secure transport of any vendors flow records, and I'm
particularly concerned with at least one of the basic architectural
positions that LFAP takes, the protocol complexity and
its Information Model.

The biggest issue with the architecture is timestamps.  My
understanding reading the lfap-eval, is that timestamps 
relate to the flow cache rather than the wire-line timestamps of
the packet stream that is being monitored.  If true, I believe
that this is huge problem, as it will introduce a lot of
timestamp variance that cannot be adjusted for, rendering
the data very limited in its usefulness.  The IPPM framework
devotes a large amount of time and effort describing how a monitor
should deal with timestamps of packets, and an IPFIX monitor
should comply with the IPPM framework on this.  Anything less
would be networking 'malpractice', if there ever could be such
a thing.

I think the protocol has too many message types.  Don't need
a separate turn for VR/VRA, and AR/ARA you should be able to
present the VR/CR/AR information elements in a single message and
then get a single response.

The concept that you can support the division of FAR and FUN
messages generate great potential problems.  If I connect at some
arbitrary point and don't receive the FAR message, how do I
process FUN messages, which don't have flow descriptors?

With regard to the Information Model,  I sense that I would
put my entire record in a single vendor specific IE and leave it
at that.  While a drastic statement, its not too far from the
truth.

Just to innumerate a few problems I have with LFAP's basic
data definitions, and this is not an exhaustive set, of course.

Best time granularity of 10's of milliseconds (1/100th of a sec)
is not good, uSec minimum, nanoSec preferable.

The requirement that the FAS/CCE's must be globally unique, and then
you start adding ones to them on roll over (how do you ensure that
the FAS is still globally unique?) is a big issue (locally unique?)

The concepts of the Flow Qualifier checkpoint and priority are
completely proprietary, they should be vendor specific OIDs?

64 bit counters for packets and bytes in every record, when over
90% of all flows have less than 256 packets and 8K of bytes in the
entire flow?  We should have 1, 2, 4 and 8 byte counter IEs as
needed, don't you think?

The IpId field is 16 bits but the IE to report it is 64 bits long.
Why not a 32 bit field for those special values that are 8-16 bits
long?  Why not have a composite IE, like an IP header attribute
IE, which could have the IpId, Tos, TTL, and option indicators all
in a single IE?  Why not an IE for the classic 5-tuple flow?

Minor points, but important points for my products.

Carter

-----Original Message-----
From: majordomo listserver [mailto:[email-masked]] On
Behalf Of Michael MacFaden
Sent: Thursday, October 03, 2002 6:10 PM
To: [email-masked]
Subject: Re: [ipfix-eval] Discussion of Candidate Protocols

On Thu, Oct 03, 2002 at 05:31:48PM -0400, Carter Bullard wrote:
>My existing flow record generators provide jitter and loss information 
>in flow data records.  How do I do that with an LFAP or netflow 
>strategy?

For LFAP, build your collector as one normally would and
then just follow section 2.45 of 
http://www.ietf.org/internet-drafts/draft-riverstone-lfap-data-01.txt

My point was/is simply this. 
We should consider the feature/complexity tradeoff 
for each criteria considered in context of the problem 
being solved. 

My personal belief is that IPFIX does not stand for 
GDMP (Generic Data Mover Protocol). 

Regards,
Mike MacFaden

--
Help        mailto:[email-masked] and say "help" in message
body
Unsubscribe mailto:[email-masked] and say "unsubscribe
ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: [email-masked]
Date: Fri, 04 Oct 2002 12:17:14 -0400

> could someone paraphrase?

	I assumed the adopted protocol would be a starting point
	and need modification to better suit IPFIX needs. Just
	my 2 cents.

	More below...
> positions that LFAP takes, the protocol complexity and

	It is a bit difficult to address these general comments so
	I'll stick to addressing specific items you mentioned
	below. If you care to elaborate on any of the above I'll
	do my best to clarify.
> the packet stream that is being monitored.  

	Not true at all. In the eval I mentioned that timestamps
	need some clarification and here is why. There are 2 types.

		1) cache timeout
		2) wire-line timestamps.

	My main point is that we (IPFIX) need to support both types
	and we need to know which which one is being used.
	But both can be handled via LFAP. 

> If true, I believe

	When you are negotiating version number you cannot
	do anything until the version is agreed upon. Otherwise
	you cannot reliably decode the message formats. Hence VR/VRA
	first.

	There are a couple of benefits of the CR message. First it provides 
	a list of other servers known to the device. **IF** the servers 
	attempt to do any load balancing the device can now be redirected 
	to another known server early on. The CR is also here for security 
	purposes. You don't want to allow someone to do a DOS attack on your 
	sever by connecting and blasting it with messages. The protocol message 
	sequence was designed to help with that problem. 

	The AR/ARA then allows an exchange of information and was
	designed for flexibility and extensibility.

	The amount of time it takes to send the messages is a non-issue
	since it is only done once at session start and is infinitesimal 
	when compared to the rest of the session.

	So there are good reasons for the message exchange. It is 
	a balance between simplicity, extensibility and security.
	Big topics handled by 3 little messages.

> process FUN messages, which don't have flow descriptors?

	You are absolutely correct. This is probably the biggest
	difference for LFAP. 

	As I mentioned in the eval, I think the LFAP spec
	should be modified to allow measured attributes (bytes,
	packets, etc...) in the FAR. This is a trivial change
	and provides the ability to send complete flows in one message. 
	This is the easy problem to solve.

	The problem with the one shot messages is storage. The device
	now needs to store all the attributes until it is time to
	send the message. When you have millions of flows this can
	be a huge problem! Or if the device is small it can be a large
	problem. And as the number of attributes to report grows it
	makes the problem worse.

	Solving this problem is much more difficult than allowing
	LFAP to send one shot messages.

	LFAP has solved this problem by splitting up the attributes 
	and allowing them to be reported independently. 

	The answer to your specific question "you" don't connect
	to the device, the device connects to you. But that just
	moves the problem, what happens if the device looses the
	connection to one server and connects to another.

	You could resend all the FAR's but that would create
	a lot of extra burden on the device. So we chose to 
	let the servers handle the failover case. FUN records
	with no descriptior are sent to a single sever.
	Similarly, FAR records which don't have a closing FUN
	are sent to the single server.  (FYI - long lived flows, 
	may have several FUNs.) In this way only failover flows
	need to get resolved and that should be a fraction 
	of the total number of flows.

> at that.  While a drastic statement, its not too far from the

	I understand the statement but I don't understand how
	it fits in the IPFIX context.

	IPFIX is trying to standardize 2 things...

		1) The data that is exported
		2) The way in which data is exported. 

	If you have to put all your data in the vendor-specific field 
	then IPFIX only accomplished 1 of those 2 things. LFAP's data
	model provides a better starting point than most but is by no 
	means perfect. If adopted, I'm sure it will need to be modified.

> is not good, uSec minimum, nanoSec preferable.

	No problem. 1/100th of a second is good enough for
	cache timeout. The definition for wire-line timestamps
	can be nanoSec.
> the FAS is still globally unique?) is a big issue (locally unique?)

	I'm not sure what you mean by a unique FAS/CCE. What
	part are you referencing?	

> completely proprietary, they should be vendor specific OIDs?

	You may be right on that.
> entire flow?  We should have 1, 2, 4 and 8 byte counter IEs as

	I would agree to that.
> Minor points, but important points for my products.

	All good points. As I said LFAP has a good data model 
	starting point but would need the input of people such 
	as yourself.

	Paul

---

From: "Peter Ludemann" <[email-masked]>
Date: Sat, 5 Oct 2002 09:49:16 -0700

> positions that LFAP takes, the protocol complexity and

As a general comment, could we leave out the Information Model in
these discussion, unless we're discussing data types. I thought
that we were just discussing the protocols and not the semantics
of the data (i.e., the form, not the content).
 
> the packet stream that is being monitored.  ...  Anything less

This is an example of a statement that does not belong in this
discussion, but does belong in a discussion on the data model,
information model, or architecture. IMHO.
 
...

> at that.  While a drastic statement, its not too far from the

This is true of any protocol that supports a "blob" data type,
although it's arguably violating the spirit of the specification.
 
...
 
> is not good, uSec minimum, nanoSec preferable.

Agreed ... for our PacketSight probe, we use nanoseconds.

I should add that the CRANE specification is lacking a few useful
time types also. We hadn't decided on the full list, so left them
out of the draft. (e.g., a nanosecond time that uses two 32-bit
values [seconds, nanoseconds] instead of a 64-bit value). Adding
them to CRANE is trivial.

...

> entire flow?  We should have 1, 2, 4 and 8 byte counter IEs as

This is why we provided the full range of 8, 16, 32, 64 bit counters,
both signed and unsigned. (We were also debating whether these would
be described as gauges or counters but decided that that would be left
to the semantics of the fields; this decision might be revisited in
the future.) Also, we decided to output in host byte order, not in
network byte order, so that little-endian machines wouldn't pay
a performance penalty (the assumption being that the receiver has more
CPU and also has fewer realtime constraints).

- peter

---

From: Benoit Claise <[email-masked]>
Date: Wed, 09 Oct 2002 13:54:51 +0200

Hi Carter and all,

>is not going to do the job for my customers.
I'm not sure I understand the statement above.

First of all, from the charter:

    "The reported flow information will include both (1) those
    attributes derived from the IP packet headers such as source and
    destination address, protocol, and port number and (2) those
    attributes often known only to the exporter such as ingress and
    egress ports, IP (sub)net mask,
    autonomous system numbers and perhaps sub-IP-layer information. "

Netflow (obviously) contains all this data types, and much more are in 
the pipe. I don't think we are lacking many data types regarding _flow 
record._
So yes, we are compliant with the requirements. BTW, I agree with Paul 
that we maybe don't want to standardize the "generalized data movers" 
protocol.
Note: Looking at the latest emails on the mailing list, I sometimes feel 
that we are trying to solve just too many problems at the same time and 
that prevents us from going forward! Example: fail-over, data 
synchronization at the collectors, load balancing, extensibility of 
everything you can think of, etc... I think we should focus on the 
charter and on the requirement draft. Or at  least work step by step: 
some work items could be done in a phase 2.

Second of all, it seems that netflow is seen on this mailing list as 
being non extensible. This is not correct!
Netflow version 9 has specifically been designed to be extensible.
You gave the example of the jitter. Well if you have this information on 
the exporter, nothing prevents the netflow v9 protocol to send this 
information to the collector.
Netflow version 9 can be used: non only to send flow records information 
but also information about the metering process (ex: number of flows 
exported/non exported). From there, again, we can export basically 
anything... even a MIB variable. Just define a new data type and you can 
use in the template.
An example? you would like to send the CPU utilization with every single 
flow records, this is possible ... in theory... I wrote "an example", 
not "a good example" :)

Regards, Benoit.

---

From: Vamsidhar Valluri <[email-masked]>
Date: Thu, 10 Oct 2002 10:38:37 -0700 (PDT)

Hi Carter,

> is not going to do the job for my customers.

sorry for bringing this up late but are you saying even with netflow
version 9 we cannot provide support for additional features??? Have you
considered Option template secyion in version 9 Spec???

thanks
-vamsi

---

From: Simon Leinen <[email-masked]>
Date: 24 Oct 2002 23:49:49 +0200

On Fri, 04 Oct 2002 12:17:14 -0400, [email-masked] said:
> 	The problem with the one shot messages is storage.

Sorry for the late response.  I've been trying to understand the
tradeoffs of "split reporting" (as LFAP does with its FAR/FUN
messages) and what you call "one-shot reporting", which is what
e.g. NetFlow does.

>       The device now needs to store all the attributes until it is
> 	time to send the message. When you have millions of flows this
> 	can be a huge problem! Or if the device is small it can be a
> 	large problem. And as the number of attributes to report grows
> 	it makes the problem worse.

At first I thought that this doesn't matter, because you I only
thought about two types of attributes for a flow:

1.) Attributes such as src/dst addresses etc., which are part of the
    flow "key", i.e. distinguish the flow from others.  Those must be
    stored over the entire lifetime of the flow anyway, or the meter
    wouldn't be able to match packets to flows.  With split reporting,
    they can be exported at the start of the flow versus at the end -
    no benefit here.

2.) Attributes such as octet and packet counters, which represent
    "measurements" about the flow and which have to be updated for
    every packet.  Those have to be stored and updated by the meter,
    and exported by the exporter in any case.

For these types of attributes, I think split reporting doesn't provide
any storage benefit on the exporter.  However, there is a third type
of attribute:

3.) Attributes that are neither part of the "flow key" (this notion
    isn't part of our terminology, but I find it very useful), but
    that aren't updated for every packet either.  Let's call those
    "snapshot" attributes for the sake of the discussion.

Such attributes exist in practice, although they are somewhat
problematic if you think about it.

Examples of such attributes might include the source/destination AS
number (which might either be the upstream/downstream or
origin/terminus AS number :-), source or destination address prefix
length, which are thought of as "usually" stable over the lifetime of
a flow.

In split reporting, they can be reported when the flow is created at
the meter, using something like LFAP's "FAR" message, and then the
meter (and exporter) can forget about them.

With one-shot reporting (I'm tempted to call this "unitary reporting"
because that sounds way more scientific), the meter must hold on to
these attributes until the flow can be exported (usually when it is
expired from the meter).

However...

Some of these attributes don't actually have to be stored, as they can
be computed from other attributes (that must be stored anyway) when
the flow is exported.  For example, an AS number can be computed from
the corresponding flow's IP address using a BGP table lookup on
export.

The difference is that split reporting would report those attributes
as they were valid for the first packet of a flow, while "unitary flow
export" would probably report them as of the time of export of the
flow - which usually happens closely after the last packet.  Since we
assume that these attributes generally remain stable over the lifetime
of a flow, the result should be the same in most cases.   When it
isn't (during e.g. routing changes over the lifetime of the flow), I
cannot see a clear advantage for either.

So for this sub-type of the third class of attributes there doesn't
seem to be a clear benefit with split reporting either.

For the remaining attributes in the third class, split reporting would
provide the possibility of space savings at the meter/exporter.  One
obvious such attribute is the observation time of the first packet.

The question is are there many other attributes that would fall into
this (sub-)class, and are the possible savings sufficient to make
split reporting an obvious win?

Besides complicating the protocol, split reporting does seem to
require more memory at the collector, which has to match FUNs with
FARs if it wants to combine data from both.

I'm just trying to understand the tradeoff.

> 	Solving this problem is much more difficult than allowing LFAP
> 	to send one shot messages.

(If you think there actually *is* a problem, that is :-)

-- 
Simon.

---

From: "Tal Givoly" <[email-masked]>
Date: Thu, 24 Oct 2002 16:23:52 -0700

Simon/Paul,

As there's been some debate regarding terminology, may I lend a slightly
DB-centric perspective: In SQL terminology, the attributes that are shared
would be identified by "First()" aggregate functions while most aggregates
would be result of aggregate functions such as "Sum()" functions.

Notice that if a device does employ "split reporting", obviously the
complexity of the collector is increased and it must create a "join" between
the two flows.

Split reporting would no longer be FNF, but instead would be 3NF -
increasing complexity in the metering process as well, if this is a
requirement.

In order to keep things simple and more broadly accepted, it seems to me
that leaving things in FNF is more likely to be adopted and implemented - a
desirable trait of any standard.

My $0.02.

Tal

---

From: [email-masked]
Date: Fri, 25 Oct 2002 09:35:55 -0400

Very nice analysis!

One thing you left out is that many of the snapshot attributes
are already known by the device at the time the flow is created.
For example the dst AS has already been looked up. By waiting
until the end, you actually cause more work by doing operations
twice. Another example is we get some information from the ACL
the packet matched. This ACL processing would also have to
be done again.

In addition, there are other packet attributes such tcp flags, 
MAC address, etc... that cannot be retrieved later. Since they
are not part of the key their use would be more of a heuristic,
but may be useful none the less depending on the application.

Paul