Subject: [ipfix-req] Re: IPFIX Requirements Questions

From: [email-masked]
Date: Wed, 04 Sep 2002 10:21:53 -0400

I wanted to summarize where we are with the issues
I raised a while back. A few still need to be resolved.

[email-masked] wrote:
> 
> As I work on the advocates document some questions
> have come to mind on the requirements document.
> 
> 4.3 Distinguishing flows using Transport Header Fields
> 
>         In the case of fragmented packets there are
>         no port numbers. Are we saying flow state information
>         MUST be maintained? In general, it is not clear from
>         the document what the requirements are for fragmented
>         packets.

	RESOLVED: State information MAY be kept for 
		  proper flow designation.

> 
> 4.4 MPLS Label
> 
>         In the case of dynamic LSP's the label is not useful.
>         Why require distinguishing flows based on label.
>         What can be done with that information?

	RESOLVED: We agreed to distinguish by label and to report label.

> 
> 5.3 Overload Behavior
> 
>         This section needs to be re-worked a little. Some
>         of the MUST's impose undue burden on the device.
> 
>         1. Must distinguish flow records before and after the
>            behavior change.
> 
>                 In the case where the behavior is to drop
>                 overflow flows, why do we need to distinguish
>                 flows. Perhaps I misunderstood something.
> 
>         2. All flows from previous behavior MUST be terminated.
> 
> 
>                 Again, if I simply dropped excess flows why terminate
>                 all existing ones. This will likely cause more overflow
>                 as the get reestablished. This also seems to go to
>                 far towards implementation.
> 
>         3. Meeting process MUST NOT merge previous records.
> 
>         I think what we are trying to say is flows with a different
>         definition MUST be distinguishable. How that is done is not
>         part of the requirements doc.

	OPEN

	We agreed that not all behaviors require terminating flows.

	We still need more dialogue on the following

	1. Do we need to define a term for the set of properties
	   that is used to distinguish flows.

	2. Do we need to require that flows can be mapped to the
	   term defined in #1

> 
> 5.4 Timestamps
> 
>         Time stamps to not always mapped to the first packet and last
>         packet. In many cases, the end timestamp is merely when
>         the flow timed out and has nothing to do with when the
>         last packet was seen. Allowing both semantics may be useful.
>         A re-wording like this...
> 
>                 The metering process MUST be able to generate timestamps for the
>                 start and end of a flow. The metering process MAY also provide
>                 timestamps for the first and the last observed packet
>                 of a flow. The timestamp resolution MUST be at least the one of
>                 the sysUpTime [RFC1213], which is one centisecond.
> 
>         Note - this does not mandate 2 timestamp fields for a flow. You
>         could have one element for each meaning (e.g. Flow-start-time,
>         first-packet-time, flow-start-and-first-packet-time) and use
>         the one with the desired meaning.

	OPEN 

	This issue still needs more dialogue to gain clarity. 

	Ganesh believes it is possible to detect last packet time within
	msec accuracy.

	I believe it is not possible without hardware assist and not
	all platform will have that capability.

> 
> 6.1 information Model
> 
>         9  Packet Counter
>         10 Byte counter
> 
>                 As mentioned earlier, for fragments this
>                 requires state information? If there is no
>                 state info, then what flow are the counted
>                 towards?

	RESOLVED(I think): no mention needs to me made in the packet or byte 
		           counter section since fragment counting is a matter 
		           for flow designation.

> 
>         13 BGP AS number
> 
>                 Is this the AS number for the observation point, the
>                 source/destination address in the packet or some
>                 combination?

	RESOLVED: These MAY be reported.

		1. Source IP AS
		2. Destination IP AS
		3. Observation Point AS (which may be a a list if it 
		   			 participates on more than one)

	I would like to propose 2 more

		Next hop AS
		previous hop AS

> 
>         14. MPLS Label
> 
>                 As stated earlier, in the case of dynamic LSP's the
>                 label has no meaning.
> 
>         21 multicast replication number
> 
>                 The document stated this can change over the life
>                 of the flow. But no mention of what happens when
>                 it changes. Is that a new flow? If not, what meaning
>                 does it have when reported? What can I do with it?

	RESOLVED: If a new flow is desired, it becomes part of the 
		"Distinguishing Properties". If not, then it can still 
		be reported and ignored when it changes.
> 
> 
> 6.2 Data Model
> 
>         What about allowing Vendors to add information independently?
>         Some data to be transported may be vendor specific and never
>         make it into the IPFIX spec.

	RESOLVED: It is allowed now and we do not need to mention 
		it specifically.

> 
> 8.3 Several Collecting Processes
> 
>         No mention is made when reporting to several devices
>         of ensuring the duplicate data does not cause a double
>         counting problem.

	RESOLVED: This requirement needs to be added.

> 
> 9. Special Device Considerations
> 
>         I'm not sure of the meaning for the following...
> 
>         ...
> 
>    Please note that here, the observation
>    point of a single flow cannot exceed the set of most fine-granular
>    observation points linked to a single metering process, because only
>    the metering process can merge packets observed at different fine-
>    granular observation points to a joint flow.
> 
>         ...
> 
>    Also the
>    locations of metering processes are not of any relevance for this
>    document (in contrast to the locations of observation points and the
>    exporting processes).
> 

	These were just for my understanding.

> Paul

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Ganesh Sadasivan <[email-masked]>
Date: Wed, 25 Sep 2002 19:20:08 -0700

>            that is used to distinguish flows.

Is it enough if we give an exporter state along with the export packet.
This assumes that a control infromation which explaining the action taken
corresponding to the exporter state has been send apriori. This way
the flows itself need not be changed.
The other way is to send a different flow. Even though the export record
may look  the same, it is identified by a different id and the id->config mapping
is assumed to have been send when the overload condition started.

>         2. Do we need to require that flows can be mapped to the

It depends on the apporach taken.

>         Ganesh believes it is possible to detect last packet time within

Well, as you mentioned a while ago if some one implements an ager
which ages out a million cache entries in s/w, the answer is no. It
seems to me it is not something a vendor would venture into doing.
It is a good time for some doing something in the similar lines to speak up;)
Thanks
Ganesh

>         I would like to propose 2 more

Why not the AS PATH?

---

From: [email-masked]
Date: Thu, 26 Sep 2002 10:16:07 -0400

> Is it enough if we give an exporter state along with the export packet.

	What do you mean by "exporter state"?

> corresponding to the exporter state has been send apriori. 

	Statement #1 above doesn't say anything about what is sent
	when or where. 

	I'm asking more of a spec question. We don't define a term for the
	set of properties that are used to uniquely identify a flow.

> This way
> The other way is to send a different flow. 

	This is the same as terminating exiting flows, which I'm
	trying to avoid.

> Even though the export record

	I don't think it does. Statement #2 say we need to ba able to
	tell which flow mapped to which definition. It doesn't
	say anything about how that mapping is done, only that
	it can be done somehow.

> It is a good time for some doing something in the similar lines to speak up;)

	Some vendors already do this. It is not a theoretical problem
	but a real one. And you don't need a million flows for it
	to be a problem. You could have just one flow. If the
	ager checks say every 30 seconds, then your end time has
	a maximum of a 30 second delay.

	Paul

---

From: Ganesh Sadasivan <[email-masked]>
Date: Thu, 26 Sep 2002 08:39:44 -0700

>         What do you mean by "exporter state"?

Whether it is in "overload condition" , "normal" etc. Sorry for being unclear.
What I meant was, use the exporter state along with the other information  defined
in section 4.0 to distinguish flows. The methods that I mentioned below are just
examples of implementation.

>         This is the same as terminating exiting flows, which I'm

Why so?

>         a maximum of a 30 second delay.

Is that for some reason or is it a limitation in the implementation?
-ganesh

---

From: [email-masked]
Date: Thu, 26 Sep 2002 12:14:16 -0400

> examples of implementation.

	I think this points to an area of weakness for us (IPFIX). We are
	having a hard time having a discussion.

	Lets say that DP = "Distinguising Properties". Where DP is
	the set of fields and functions, etc... that are used
	to define a flow. Lets assume for now that the set of possible
	fields and functions, etc... is fully defined somewhere
	(it's not, but assume it is for now.).

	If we require that each flow can be mapped to its DP
	then the discussion moves to how that happens. 

	There are many ways. Here are a few which come to mind...

	1) send the DP with each flow.
	2) send the DP with an ID. Then each flow contains
	   a DP-ID field.
	3) Send the DP for noraml operation and the DP for
	   overload situation. Then only state changes need
	   to be delivered. 

	#3 above seems to be what you are talking about.

	However, I think we need to define DP as a term and the set of
	possible fields and functions first. Add the requirement of mapping
	a flow to a DP. Then the discussion can move to how we
	can best do the mapping. Then the discussion of what happens
	under overload conditions will be clearer.

> >         This is the same as terminating exiting flows, which I'm

	Because you are going to send a new flow for all existing flows
	during overload sitiation. It will probably not be 1 for 1 but
	rather many flows now being collapsed into one during overload.
	
	Isn't that what you meant by "send a different flow".
> Is that for some reason or is it a limitation in the implementation?

	It is merely software based aging. Probably just an initial
	system design choice.

	Paul

---

From: Ganesh Sadasivan <[email-masked]>
Date: Thu, 26 Sep 2002 11:54:03 -0700

>         There are many ways. Here are a few which come to mind...

Does it matter how the DP is identified & send (whether with each flow or  split
across flow & control or otherwise) is sent, from a req. doc perspective?
As long as we define DP (which is what you are suggesting to do) and tell that DP
must give enough information so that the full flow context can be  derived at the
collector, will it not solve this issue?

>         a flow to a DP.

As I see DP is a function of  {1.metering process(s) employed,  2. exporter state,
3. fields collected}.
There is a furthur inter dependency between
* 2&1 (with the exporter state change metering could change),
* 2 & 3 (with the exporter state change fields collected could change),
* 1 & 3 (fields are input tometering functions).

Am I missing anything?

> Then the discussion can move to how we

Yes. But why do we not leave room for such a case?

>         It is merely software based aging. Probably just an initial

Are you indicating that there could be products in which the design
cannot be changed to suit the msec granularity timing requirements
of IPFIX?
-ganesh

---

From: [email-masked]
Date: Thu, 26 Sep 2002 15:56:40 -0400

> collector, will it not solve this issue?

	Yes it will. I gave example to make sure we were on the
	same page. They would not go in the req doc.

> As I see DP is a function of  {1.metering process(s) employed,  2. exporter state,

	I don't see a direct relationship #1 and #2/#3. For example
	I could have a DP of source and dest IP but collect (i.e. report)
	Src/Dst AS number. Also, the state may change from normal
	to overload with no change in DP but rather a dropping of
	additional flows.

	I think DP stands pretty much on its own. In other words
	fields collected and the state are not part of the DP
	definition. Those things may influence which DP is used
	but they are not part of the DP definition itself.

	But I think we are getting a little ahead of the game.
	We need to define DP and its fields and functions.

	Once we have consensus that we need to define DP and its
	fields and functions and we agree we need to map flow
	to DP, then we can kick off the discussion.

> Yes. But why do we not leave room for such a case?

	The current document says all flows must be terminated. I
	was pushing to allow for other alternatives, including
	terminating all flows.
> cannot be changed to suit the msec granularity timing requirements

	Yes, because if there is no current hardware support and there
	can be lots of flows then you don't have much choice but to increase
	the aging time. That type of platform exists today.

	Paul

---

From: Ganesh Sadasivan <[email-masked]>
Date: Mon, 30 Sep 2002 09:13:37 -0700

Just taking the topic of DP alone.

>         to overload with no change in DP but rather a dropping of

It depends on the overload policy.
* If it is such that say more aggregation (by using less fields) is done in
the case of an overload (which effecively reduces the # of records
exported) then there is a relation between 1 & fields exported (may not
be the fields collected as I mentioned above).
* If there are fields extracted in the packet path which are not directly
derived from the packet (like AS #) in the case of an overload, one may
not collect these fields in the packet path itself.
* When the exporter state is changed to overload, the metering process
may be changed from sampling interval X to Y.

Thanks
Ganesh

>
>
>         I think DP stands pretty much on its own.

> In other words