Subject: RE: [ipfix] Questions regarding draft-gsadasiv-ipfix-proposal-00

From: "Norseth, KC" <[email-masked]>
Date: Tue, 12 Feb 2002 11:50:38 -0500

Hi Robert,  

Lurkers are welcome.

|-----Original Message-----
|From: Robert Lowe [mailto:[email-masked]] 
|Sent: Tuesday, February 12, 2002 8:56 AM
|To: [email-masked]
|Subject: [ipfix] Questions regarding draft-gsadasiv-ipfix-proposal-00
|
|
|Hi!
|
|Just a few simple questions from a lurker here, that perhaps 
|Ganesh or Benoit 
|can best answer.  Please excuse me, if I've overlooked 
|something in my rather 
|quick reading of this.
|
|Section 4.1
|
|If detection of a collector crash is impossible only because 
|of the underlying transport, e.g. UDP, why not add an 
|_optional_ keepalive exchange between exporter and collector?  
|This would still not introduce as much overhead as using a 
|reliable transport, or introduce multiple control information 
|connections when using redundant collectors, as you outline in 
|section 4.2.

This is a section I am working on.  I have not had a chance to finish it
yet.  We need some way of communicating between the collector and the
exporter, if just to let the exporter know the collector is not there for
failover.

|Section 5.4
|
|You mention that templates should be periodically sent from 
|exporter to 
|collector.  If the collector were to acknowledge receipt of 
|these, could this not act as a simple keepalive as well?  
|Granted, this is not something one would want to have happen 
|frequently.  Am I to assume that the collector can initiate 
|these as well?  I'm not advocating this as a form of 
|capability negotiation (see below).

Good point.  Some response from the collector is a signal there.
|
|---------
|
|A few more comments regarding templates, although ones which 
|probably fall within the realm of operational vs. protocol issues...
|
|Early on there was some discussion regarding the impact of 
|IPFIX on the network itself, and if/how the exporter or 
|collector should react to overload conditions.  As I recall, 
|Peter Phaal indicated that from an analysis point of view, it 
|would not be good to change the sample rate. However, when 
|faced with an overload condition:
|
|1) On the exporter, should it:
|	a. continue as is, and drop packets when it has to, or
|        b. fall-back to a user pre-defined sampling rate, compromising
|           IPFIX data to avoid dropping packets ??
|
|2) On the collector, should it:
|        a. continue, and lose flow data because it cannot 
|process it all, or
|        b. signal a fall-back to a different sampling rate via 
|a new template?
|
|If the solution uses the verb "buy", don't mention it!  ;-)  I 
|can accept a good argument that these are operational issues.

This is very dangerous to have the exporter switch to sampling
automatically.  This could really confuse the information being gathered.
You could get the message from the reports that volume has dropped down
during a period of time, but in reality, it is dnagerously high.

K.C.

|Also, this could be related to how an exporter deals with DoS 
|attacks, which I haven't heard anything about recently.  
|Perhaps all of you gentle folk talk about this stuff at IETF 
|meetings, which I don't get to attend.
|
|-Robert
|
|--
|Help        mailto:[email-masked] and say "help" 
|in message body
|Unsubscribe mailto:[email-masked] and say 
|"unsubscribe ipfix" in message body
|Archive     http://ipfix.doit.wisc.edu/archive/
|

---

From: Robert Lowe <[email-masked]>
Date: Tue, 12 Feb 2002 13:22:48 -0600

> "Norseth, KC" wrote:

Hi KC!

> This is very dangerous to have the exporter switch to sampling automatically.  This
> could really confuse the information being gathered.  You could get the message from the
> reports that volume has dropped down during a period of time, but in reality, it is
> dnagerously high.

No, I was assuming that the new template would somehow be recorded, or available
during analysis, so that it would be recognized that the sampling rate changed
and could be properly reflected in a report.  I agree that not having this change 
record available during analysis makes the analysis pretty useless.  

But this is a kind of in-band config that would add more complexity than would 
seem to outweigh its benefit.  I think I need to re-read a few things to better
understand the use of templates in general too.

Thanks!

-Robert

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: "K.C. Norseth" <[email-masked]>
Date: Tue, 12 Feb 2002 23:31:42 -0700

Hi Robert,

----- Original Message -----
From: Robert Lowe <[email-masked]>
To: Norseth, KC <[email-masked]>
Cc: <[email-masked]>
Sent: Tuesday, February 12, 2002 12:22 PM
Subject: Re: [ipfix] Questions regarding draft-gsadasiv-ipfix-proposal-00

| > "Norseth, KC" wrote:
|
| Hi KC!
|
| > |A few more comments regarding templates, although ones which
| > |probably fall within the realm of operational vs. protocol issues...
| > |
| > |Early on there was some discussion regarding the impact of
| > |IPFIX on the network itself, and if/how the exporter or
| > |collector should react to overload conditions.  As I recall,
| > |Peter Phaal indicated that from an analysis point of view, it
| > |would not be good to change the sample rate. However, when
| > |faced with an overload condition:
| > |
| > |1) On the exporter, should it:
| > |       a. continue as is, and drop packets when it has to, or
| > |        b. fall-back to a user pre-defined sampling rate, compromising
| > |           IPFIX data to avoid dropping packets ??
| > |
| > |2) On the collector, should it:
| > |        a. continue, and lose flow data because it cannot
| > |process it all, or
| > |        b. signal a fall-back to a different sampling rate via
| > |a new template?
| > |
| > |If the solution uses the verb "buy", don't mention it!  ;-)  I
| > |can accept a good argument that these are operational issues.
| >
| > This is very dangerous to have the exporter switch to sampling
automatically.  This
| > could really confuse the information being gathered.  You could get the
message from the
| > reports that volume has dropped down during a period of time, but in
reality, it is
| > dnagerously high.
|
| No, I was assuming that the new template would somehow be recorded, or
available
| during analysis, so that it would be recognized that the sampling rate
changed
| and could be properly reflected in a report.  I agree that not having this
change
| record available during analysis makes the analysis pretty useless.
|
| But this is a kind of in-band config that would add more complexity than
would
| seem to outweigh its benefit.  I think I need to re-read a few things to
better
| understand the use of templates in general too.

This could be a SHOULD if people want it to be. I think it could be
dangerous, but you are right, a template could inform the collector that
this is sampled data.  The collector would have to be smart enough to
explain this to the readers of the reports when it changes though.  It would
also have to be smart to switch back and once again the collector would have
to be smart enough to report this to the users.  I don't think the
collectors/applications out there now are smart enough for this.

K.C.

| Thanks!
|
| -Robert
|
| --
| Help        mailto:[email-masked] and say "help" in message
body
| Unsubscribe mailto:[email-masked] and say
| "unsubscribe ipfix" in message body
| Archive     http://ipfix.doit.wisc.edu/archive/
|

---

From: [email-masked]
Date: Wed, 13 Feb 2002 09:02:29 -0500

> collectors/applications out there now are smart enough for this.

	What to do in overload situations has not been discussed
	much to date. A few ideas have been tossed around but that's
	it as for as I can tell. I don't think much more than
	a TBD can go in the doc at this point.

---

From: Robert Lowe <[email-masked]>
Date: Wed, 13 Feb 2002 14:26:36 -0600

"K.C. Norseth" wrote:

Hi K.C.!

> collectors/applications out there now are smart enough for this.

Won't they have to become smarter?  Besides a changing sample rate from
one exporter, how will they aggregate flow data from multiple exporters, 
each potentially using a different sample rate?  Or do all exporters 
within a "domain" have to use the sample rate?

-Robert

---

From: "Peter Phaal" <[email-masked]>
Date: Wed, 13 Feb 2002 13:57:14 -0800

> within a "domain" have to use the sample rate?

This raises an interesting point. The IPFIX charter doesn't address the
algorithms used by the flow collector, but there are system-wide issues that
should be explored. Design choices made in the IPFIX protocol specification
can have a fundamental effect on the capabilities of traffic management
systems collecting IPFIX data (scalability, latency, accuracy, multi-point
correlation, etc.)

Specifically here are some issues to consider with regard to agents
dynamically altering sampling rates in response to network load:

1. It may introduce correlation into the measurements that biases results
and makes the computation of variances impossible.
2. If the agent changes sampling rate, will it flush its flow cache? If it
doesn't there will be some flows that have ill-defined sampling rates.
Forcing a cache flush at the point where you are starting to sample because
of a surge in traffic seems like a bad idea.
3. Network traffic is variable. The load in one interval is not very
predictive of the load in the next interval. The agent is likely to always
have the wrong sampling rate, reducing sampling interval before a burst and
increasing the sampling interval before a lull.

In general, it is important that a measurement agent should behave
predictability. The IPFIX agents and the collectors need to be configured
consistently on a network-wide basis. Selecting appropriate sampling rates
for each observation point is a configuration issue and it should be up the
the network manager to determine the appropriate sampling rate, taking into
account the measurement load on exporters, load on the collector and the
desired accuracy of measurements.

Peter
----------------------
Peter Phaal
InMon Corp.

[email-masked]