Subject: [ipfix-eval] Simon's and Ram's evaluations

From: Juergen Quittek <[email-masked]>
Date: Tue, 22 Oct 2002 16:08:11 +0200

Dear all,

I agree almost completely with Simons evaluation. In Ram's
evaluation I like the classification of compliancy per item.
Maybe it can be formatted a bit more compact, for example:

  "3.1.1.Reliability (5.1)

      LFAP: T     CRANE: E     IDPR: E     NetFlow: E     Diameter: E"

In the conclusion section I think a table summarizing the
classification would be helpful. I put my evaluation into an example
for such a table below. It is almost completely consistent with Simon's
evaluation and has a few differences to Ram's one:

   LFAP   CRANE  IDPR   Net-   Dia-
                        Flow   meter
    T      E      E      E      E     Reliability (5.1)
    E      F      F      T      E     Sampling (5.2)
    T      E      E      T      E     Overload Behavior (5.3)
    T      T      T      T      T     Timestamps (5.4)
    T      T      E      T      T     Time Synchronization (5.5)
    T      T      E      T      T     Flow Expiration (5.6)
    ?      ?      ?      ?      ?     Multicast Flows (5.7)
    ?      ?      ?      ?      ?     Ignore Port Copy (5.8)
    T      T      T      T      T     Information Model (6.1)
    T      T      T      T      T     Data Model (6.2)
    T      T      T      T      T     Congestion Awareness (6.3.1)
    T      T      T      P      T     Reliability (6.3.2)
    T      T      T      P      T     Security (6.3.3)
    T      T      T      T      T     Push Mode Reporting (6.4)
    F      F      E      F      E     Pull Mode Reporting (6.4)
    T      T      T      T      T     Regular Report. Interval (6.5)
    T      T      T      T      T     Notif. on Specif. Events (6.6)
    E      E      E      E      E     Anonymization (6.7)
    T      T      T      T      T     Openness (8.1)
    T      T      T      T      T     Scalability (8.2)
    T      T      P      T      T     Several Collecting Proc (8.2)
-------------------------------------------------------------------
   16     14     11     14     14     number of Ts
    2      3      6      2      5     number of Es
    0      0      1      2      0     number of Ps
    1      2      1      1      0     number of Fs

Comments (comparing to Ram's and Simon's evaluation):
5.1: Only LFAP already indicates lask of reliability
     of the metering process. The other can be extended
     to do so.
5.2: CRANE, IPDR, Diameter does not address overload behavior.
     But they can be extended to report on it.

Juergen's Conclusion

My view is a bit bit biased, because it is the view of one having to
implement the protocol on devices. Therefore, I give more weight than
others might do to the cost of the implementation, the consumption of
processing power and the memory consumption.

Therefore, I estimate simple approaches. And NetFlow is the most simple
one. Also I learned that simplicity increases reliability (or safety
of design and implementation).

I am not sure whether or not I could build a compatible implementation of
CRANE or LFAP that just ignores all configuration messages received from
a collector.

If we go for a more complex prototcol, there is the choice between
problem-specific protocols on one side (IPDR, LFAP, CRANE) and the
more general Diameter on the other side. Diameter would be in favor
if it was used widely already and anyway be implemented on most boxes.
Since this is not the case I would go for problem-psecific protocols.

Concerning the produced flows, I clearly prefer the efficient NetFlow,
IPDR, and CRANE to LFAP and Diameter.

The item level comparison did not show a clear winner.

Considering all this I would recommend to go for NetFlow v9. If this
is not accepted because NetFlow is considered as too simple or lacking
functionality, I would recommend using IDPR.

    Juergen

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: [email-masked]
Date: Tue, 22 Oct 2002 11:23:10 -0400

Juergen Quittek wrote:

[snip]

> processing power and the memory consumption.

	Netflow V9 will need to me modified to allow split reporting.
	Many devices do not have the ability to store attributes per flow
	until expiration.

	This is a key diffentiator for LFAP as it allows both split reporting
	and one shot reporting.

> CRANE or LFAP that just ignores all configuration messages received from

	Can you clarify what configuration messages you are refering to?

> IPDR, and CRANE to LFAP and Diameter.

	I disagree. LFAP's multi-record format allows the
	type information to be given once for many flows. Furthermore,
	if bandwidth is truly an issue the incremental savings of
	sending type info once per session versus once per message
	will not solve it. A higher level of aggregation will be
	the solution.

---

From: Juergen Quittek <[email-masked]>
Date: Tue, 22 Oct 2002 18:08:53 +0200

Paul,

> 	This is a key diffentiator for LFAP as it allows both split reporting

I see that split reporting is an advantae of LFAP concerning memory consumption.
We should mention this in the evaluation.

> 	Can you clarify what configuration messages you are refering to?

"Control message" was bad phrasing. I meant AR and KA.

> 	will not solve it. A higher level of aggregation will be

My point is that LPFAP sends at least two messages per flow (FAR and FUN).
But most of the flows on the Internet are very short-lived and can usually
be reported by a single NetFlow or IPDR message.

---

From: [email-masked]
Date: Tue, 22 Oct 2002 12:27:27 -0400

> "Control message" was bad phrasing. I meant AR and KA.

	If the client handles connection loss in someother way, 
	KA messages can be ignored.

	If the collector does not need time sync from the device
	then no AR messages are needed from the Collector. So
	the minimal message exchange would be...

	VR
	VRA
	CR  - we could even get rid of this assuming security is handled at
		the lower layers
	FER
	
> be reported by a single NetFlow or IPDR message.

	Not true. LFAP can allow all attributes to be reported
	in the FAR message.

---

From: Vamsidhar Valluri <[email-masked]>
Date: Tue, 22 Oct 2002 11:50:39 -0700 (PDT)

Jeurgen,
  Please see inline

> > 	Many devices do not have the ability to store attributes per flow

At the high level, if the goal is to send periodic updates of certain
statistics related to flow then it can be done with CURRENT Netflow by
adjusting the "active timeout" value of flow entries. Ofcourse we do send
the key fields also with it unlike FUN/FAR messages. Memory consumption on
the device is specific to implemention ( i agree with Carter) and if we
 run out of memory for some reason there is always the Overload Condition
indication mentioned in the requirements.

-vamsi

---

From: Vamsidhar Valluri <[email-masked]>
Date: Tue, 22 Oct 2002 12:17:16 -0700 (PDT)

On Tue, 22 Oct 2002 [email-masked] wrote:

> Vamsidhar Valluri wrote:
> 	Call it implementation specific or whatever, but if I need
> 	20 extra bytes per flow for a million flows, consuming
> 	20mb of memory is going to be an issue. Not to mention the
> 	memory and CPU time to maintain the list.

Paul, as pointed out by Juergen lifetime of many flows is small.

-vamsi

> 	Paul

---

From: [email-masked]
Date: Tue, 22 Oct 2002 15:05:51 -0400

Paul

[snip]

---

From: [email-masked]
Date: Tue, 22 Oct 2002 15:23:47 -0400

> Paul, as pointed out by Juergen lifetime of many flows is small.

	How does that change the memory requirement? If I've got on average
	N active flows then I have N*extra-bytes of memory at any point
	in time.

---

From: Michael MacFaden <[email-masked]>
Date: Tue, 22 Oct 2002 14:13:42 -0700

>Paul, as pointed out by Juergen lifetime of many flows is small.

I agree the protocol should behave well in the most common case,
but this does not appear to be a sound engineering tradeoff 
(ie having a protocol behave well for only a particular flow pattern). 

And while going to overflow is a sound last
resort, maybe there are some intermediate solutions 
that indeed would work better in practice.

Regards,
Mike MacFaden

---

From: Vamsidhar Valluri <[email-masked]>
Date: Tue, 22 Oct 2002 14:22:02 -0700 (PDT)

> 	N active flows then I have N*extra-bytes of memory at any point

I agree extra memory will be consumed for Average N value. But consider
the following example(simplified) related to lifetime of flows for
statically allocated memory. Suppose cache
can hold 5K entries at a time. If there are 2K long standing flows,
whose lifetime is > 5min and 3K short lived flows whose life time
is ~1min. Considering a trivial scenario where at the start of every
minute ~3K flows are created which expire after 1 min, over a period of
5min this cache would have exported information related to 17K flows
whereas if you have 5K long standing flows, you could send information
related to only 5K flows (assuming you don't create new flows if cache is
full by expiring the old ones).

-vamsi

---

From: "Tal Givoly" <[email-masked]>
Date: Wed, 23 Oct 2002 00:22:46 -0700

Juergen,

<snip>
<snip>

Quoting from the requirements:

5.1.  Reliability

   The metering process MUST either be reliable or missing reliability
   MUST be known and indicated. The metering process is reliable if each
   packet passing the observation point is metered according to the
   configuration of the metering process. If, e.g. due to some overload,
   not all passing packets can be included into the metering process,
   then the metering process MUST be able to detect this failure and to
   report about it.

Tal> The way I interpret this requirement is is that EITHER an accurate
depiction of events observed by the Observation Point and metered by the
Metering Process MUST be delivered to a Collecting Process OR an indication
that the Metering Process didn't process the data reliably MUST be delivered
to the collection process.

Basically - that either of these messages MUST be delivered to the
Collection
Process.

CRANE obviously supports this. CRANE imposes no constraints on the Metering
Process. Therefore, it is the Metering Process' responsibility to indicate,
perhaps with two different types of events whether it reliably metered or
not.
In fact, this indication MUST get to a Collection Process.

I contest that NetFlow FAILS this requirement while CRANE nor Diameter fail
this requirement. It is subject to debate on whether LFAP or IPDR meet this
requirement or must be extended to meet this requirement.

Therefore, I ask you to reconsider your scoring of this important item.

Tal
____________________________________________

Tal Givoly
Chief Scientist
XACCT Technologies, Inc.
2900 Lakeside Drive
Santa Clara, CA 95054
http://www.xacct.com

Direct:  408-330-5747
Tel:     408-654-9900 x 5747
Fax:     408-654-9904
E-mail:  mailto:[email-masked]
____________________________________________

---

From: "Tal Givoly" <[email-masked]>
Date: Wed, 23 Oct 2002 00:22:48 -0700

Juergen,

<snip>

I believe there may be a typo - you mention "5.2 overload behavior", however
5.2 is, in fact, sampling behavior. Which did you mean that you differ with
compared to Simon's review?

Regarding Sampling Behavior (and this is in response to both your
evaluations
as well as Simon's), both evaluations conclude that that CRANE Fails to meet
this requirement.

CRANE clearly allows distinguishing between sampled data and non-sampled
data.
As CRANE, similarly to IPDR and Diameter, doesn't impose any restrictions
on the Metering Process, the latter can easily submit sampled data in
a dedicated Template. Even if you consider that there is no intrinsic
support
specifically for Sampling Behavior within the protocol itself, it is
definitely
Extendable to that degree, would you not agree?

On the other hand, I don't quite understand how NetFlow v9 actually supports
multiple concurrent Templates by the same Export Process. As NetFlow v9
Templates are indistinguishable (they are changed rapidly and not negotiated
with no uniquely describing attributes!). Therefore, NetFlow v9 cannot
support both sampled and non-sampled mode of operations in any
combination - a requirement that is implied by the Sampling Requirement
(for at least some length of overlap time).

Perhaps this question should be directed to the NetFlow protocol advocate:

- How can multiple Templates coexist?
- If, in fact, it is possible for multiple Templates to coexist, please
  elaborate how are these distinguished by recipients based on the current
  protocol draft?

  For instance, how would a Collection Process differentiate between sampled
  and non-sampled data? How would this be done if both are being exported
  concurrently?

Thanks for any clarification.

Tal

-----Original Message-----
From: majordomo listserver [mailto:[email-masked]]On Behalf
Of Juergen Quittek
Sent: Tuesday, October 22, 2002 7:08 AM
To: [email-masked]
Subject: [ipfix-eval] Simon's and Ram's evaluations

Dear all,

  "3.1.1.Reliability (5.1)

Juergen's Conclusion

    Juergen

--
Help        mailto:[email-masked] and say "help" in message
body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: "Tal Givoly" <[email-masked]>
Date: Wed, 23 Oct 2002 01:37:29 -0700

Juergen,

<snip>

5.1.  Reliability

More concisely, that either of these messages MUST be delivered to the
Collection Process.

CRANE obviously supports this. CRANE imposes no constraints on the Metering
Process. Therefore, it is the Metering Process' responsibility to indicate,
perhaps with two different types of events whether it reliably metered or
not. In fact, this indication MUST get to a Collection Process.

Tal

Tal Givoly
Chief Scientist
XACCT Technologies, Inc.
2900 Lakeside Drive
Santa Clara, CA 95054
http://www.xacct.com

Direct:  408-330-5747
Tel:     408-654-9900 x 5747
Fax:     408-654-9904

---

From: Juergen Quittek <[email-masked]>
Date: Wed, 23 Oct 2002 11:38:24 +0200

Tal,

The way I see it, all protocols can easily be extended to meet
this requirement, by adding a reliability attribute to each
flow record (just one bit), to the templates, or other messages.

My point in the evaluation was, that only LFAP has this feature
already defined. And only LFAP meets it without further extension.

    Juergen

---

From: Juergen Quittek <[email-masked]>
Date: Wed, 23 Oct 2002 11:46:36 +0200

Tal,

> compared to Simon's review?

My apologies for the typo. It should be "5.2 Sampling". I wrote the note
because my evaluation differs from Ram's one.

> Extendable to that degree, would you not agree?

I see your point. Both CRANE and IPDR can be extended by flow attributes
indicating sampling configuration. Therefore, they should get an 'E' for 5.2
instead of an 'F'.

    Juergen

---

From: Vamsidhar Valluri <[email-masked]>
Date: Wed, 23 Oct 2002 12:16:02 -0700 (PDT)

Please see inline

> requirement or must be extended to meet this requirement.

This needs further explanation than what is presented in our netflow-eval-draft.
There are two scenarios (both are implementation specific)

1. flow metering process is implemented in such a way that on certain
overload conditions it will switch states from a) full flow accounting to
b) no-flow accounting. In state b) metering process maintain somekind of
counters which keep track of number of packets for which flow accountig
was not performed. I guess LFAB does in this fashion.

2. flow metering process is implemented in such a way that all packets
going through that observation point are always accounted. In case of
abnormal conditions packets may be dropped before any packet
forwarding process (here control has not even reached flow metering
process) can be done. I am not sure how CRANE can send this information to
collector????

Regarding the contest that Netflow "FAILS" this requirement, i would like
to point to Netflow on Cat6K which maintains counters in terms of packets
for netflow-cache misses due to overload conditions.This was primarily
used for debugging but with Netflow Version 9, using
option template we can send these counters to the collector. This meets
the requirement.

thanks
-vamsi

---

From: Vamsidhar Valluri <[email-masked]>
Date: Wed, 23 Oct 2002 12:21:42 -0700 (PDT)

> requirement or must be extended to meet this requirement.

Please see my previous email. Same explanation applies here.

-vamsi

---

From: Vamsidhar Valluri <[email-masked]>
Date: Wed, 23 Oct 2002 14:57:38 -0700 (PDT)

Tal,
  Please see inline

> (for at least some length of overlap time).

Every Template has an associated ID. Each template identifies unique set
of attributes. For packets accounted through Sampling mechanism we create
a new Template with an additional attribute called "Sampler ID". This
"Sampler ID" uniquely identifies set of Sampling attributes (Sampling
rate, Sampling algorithm etc). These sampling attributes mapped to ID are
sent in the form of option templates to the collector. Collector will map Sampling
ID to it's corresponding Option data which contains specific values
of sampling attributes which will give an idea of type of
sampling performed on that flow. If there are changes to Sampling
attributes we create new Option templates with new IDs and send to the
collector before any data corresponding to that Sampling mechanism is
exported to collector. We can do this on the fly. We need not exchange
these Template before we start Netflow accounting on the box. In short if
Metering process is doing Sampling and non-sampling of data, there exists
two templates which will uniquely identify Sampled and non-sampled flows.

Some more information:
Flows can have only IPV4, or IPV4 + BGP related, or IPV4+BGP + Multicast,
or IPV4+Multicast, or IPV6, or MPLS, or IPV6+Multicast, sampled .......

So there are lot of possible combinations, we create Template only when we
see these combinations and do not negatiate all these combinations before
hand. If the Collector can understand these attributes then our mechanism
is much more flexible.

> - How can multiple Templates coexist?

Multiple Templates will exist because the attribute set is different. One
with Sampler ID and one without.

>   elaborate how are these distinguished by recipients based on the current

Please see above explanation. On CCO you will find new Sampling related
Field Types, if not then they are in the process of updation.

>   and non-sampled data? How would this be done if both are being exported

Presence of "sampler ID" attribute will differenciate Sampled from
non-sampled Data.

thanks
-vamsi

---

From: "Tal Givoly" <[email-masked]>
Date: Thu, 24 Oct 2002 14:53:51 -0700

Vamsi,

Let me just verify my understanding:

<snip>
packets accounted through Sampling mechanism we create a new Template with
an additional attribute called "Sampler ID".
<snip>

The templates are distinguished one from the other by the existence of
particular attributes?

Supposing you want to support many different templates (based on the
examples you provided):
1. IPv4
2. IPv4+BGP
3. IPv4+BGP+Multicast
4. IPv4+Multicast
5. IPv6
6. MPLS
7. IPv6+Multicast
8-14. Sampled (for each of above combinations) - another 7 combinations

This will all IMPLICITLY be deducted by the existence of particular
attributes?

In NFv8, for instance, there are various aggregation schemes supported by
the routers.
- Are you implying that the aggregation scheme used by the network element
would be deduced by the collector by virtue of which fields are MISSING from
the flow record?
- How would you propose extending this to have different aggregation keys
with the same attributes?

Tal

-----Original Message-----
From: Vamsidhar Valluri [mailto:[email-masked]]
Sent: Wednesday, October 23, 2002 2:58 PM
To: Tal Givoly
Cc: Juergen Quittek; [email-masked]
Subject: RE: [ipfix-eval] Simon's and Ram's evaluations - Sampling vs.
Overload

Tal,
  Please see inline

sent in the form of option templates to the collector. Collector will map
Sampling

Some more information:

thanks
-vamsi

body

---

From: Vamsidhar Valluri <[email-masked]>
Date: Thu, 24 Oct 2002 15:15:15 -0700 (PDT)

Tal,
  Please see inline

> The templates are distinguished one from the other by the existence of

In current netflow implementation attributes includes Key fields + flow
related statistics

> This will all IMPLICITLY be deducted by the existence of particular

A Template ID is decided on the basis of "Key fields + other flow fields".
If customer is not interested in receiving key fields and is interested
only in other flow fields like for example Bytes and Packet count. Then
for each different set of key fields we generate different Template
distinquished by template ID. And the mapping between Key fields and the
Template ID can be sent as Option Template and Option Data records.

> would be deduced by the collector by virtue of which fields are MISSING from

two cases here - currently we send key fields in the flow record. If
customer does not want key fields or wants only subset of them then the
mapping between flow Key and the Template ID will be sent as OPtion data,
since the Template ID is decided on key fields + other attributes.

> with the same attributes?

Above should explain this.

-vamsi

---

From: Benoit Claise <[email-masked]>
Date: Mon, 28 Oct 2002 16:39:42 +0100

Paul,

>	This is a key diffentiator for LFAP as it allows both split reporting
Do I understand correctly that with split reporting, you report a FUN 
and (at least) one FAR messages.
So basically, in order to gain some memory on the devices for long lived 
flows, you export 2 smaller messages (whose sum can only be bigger for 
short flows) to the collector for all flows, included the short ones?
We've seen on our devices that the action of exporting packets will eat 
some extra non-negligible CPU. And I think this is a concern!

Now regarding the long-lived flows.
I know that the P2P problem is growing but I would refer to this paper 
http://www.research.att.com/~duffield/pubs/DLT02-flows.pdf
You will see in there that the highest flows in terms of average packet 
number are napster and real-audio with respectively 455 and 467 packets.
What is the chance that they will produce long-lived flows? knowing that 
this timer is 30 minutes in the netflow metering process (btw, this is 
configurable)
Not much I should say!

Regards, Benoit.

---

From: [email-masked]
Date: Mon, 28 Oct 2002 11:18:46 -0500

> some extra non-negligible CPU. And I think this is a concern!

	I am not, repeat NOT, advocating mandatory split reporting of flows!

	In fact, I suggested in my eval that the LFAP spec should be
	modified so that "unitary" flow reporting could happen via
	one FAR message. 
> this timer is 30 minutes in the netflow metering process (btw, this is

	You missed one key point, for each flow you would still need 
	to hold on to the attributes for 30 minutes until the flow timed out
	(or how ever long it took to determine the flow should be
	deleted). 

	Plus keep in mind that devices may be reporting aggregated 
	flows where the flows stay around much longer. While the number
	may be lower than a very granular key, there may still be large
	numbers of them. Furthermore, with aggregation you need to use
	a time-out since TCP SYN,FIN don't help. Shorter time-outs cause
	a lots of flow creation which is probably what you're trying
	to avoid so the norm will likely be longer time-outs.
	
	Thus, with a split reporting option you provide the device
	implementor the ability to split report or one shot report
	(or I suppose some fancy implementation may use a combination).

	Since we would like this protocol to be implemented on lots
	of different devices, memory and CPU consumption are show stopper
	type issues. If a device can't support IPFIX without chewing up
	too many device resources it either wont get implemented or
	wont get turned on.

	Split reporting provides a necessary degree of implementation
	flexibility for both granular and aggregated flows. Yes it makes
	the Collector's job more difficult but its better than not having
	some devices support to begin with.

	Paul

---

From: Benoit Claise <[email-masked]>
Date: Tue, 29 Oct 2002 16:22:12 +0100

Hi Paul,

>	I am not, repeat NOT, advocating mandatory split reporting of flows!
My misunderstanding, sorry Paul.
As a consequence, I will stop this email thread here.

Regards,  Benoit.