Subject: [ipfix-req] RE: Data Export Reliability Requirement [was: Re: IPFIX Requireme nt draft status - section 6.3.2]

From: "Reinaldo Penno" <[email-masked]>
Date: Fri, 4 Oct 2002 18:44:10 -0700

> -----Original Message-----
> From: Simon Leinen [mailto:[email-masked]]
> Sent: Friday, October 04, 2002 3:30 PM
> To: [email-masked]
> Cc: Penno, Reinaldo [BL60:0430:EXCH]; 
> [email-masked]
> Subject: Data Export Reliability Requirement [was: Re: IPFIX 
> Requirement
> draft status - section 6.3.2]
> 
> 
> On Mon, 30 Sep 2002 03:00:03 -0700, "Reinaldo Penno" 
> <[email-masked]> said:
> > I'm looking to understand/really clarify this requirement. It seems
> > ambiguous...at least to me. I do not understand what is the
> > meaning/purpose of "-->abscence<-- of reliability....MUST be
> > indicated".
> 
> Yes, that just sounds weird.  (High) reliability should be a goal for
> the exporting process, not something that can be readily measured, or
> even less is "present" or "absent" at a given time during the
> operation of an IPFIX implementation.
> 
> reqs-06> Absence of reliability of the data transfer, for 
> example caused by
> reqs-06> loss or reordering of packets containing flow 
> information, MUST be
> reqs-06> indicated.
> 
> Why not write something like this instead:
> 
>   "If some flow information data cannot be delivered to the collector
>    in a timely manner, the amount of missing flow information data
>    MUST be indicated to the collector.  Possible reasons for failure
>    to deliver data include loss of packets on the network path, or
>    resource shortage at the exporter."

But how the exporter would know which records were lost in the network
unless we have (application layer) ACKs? 

> 
> I removed packet reordering as an example for reasons to lose data
> because I don't think it's that relevant in practice.  From
> operational experience, I think overflows at the exporter are a much
> more likely source of data loss.
> 
> An indication of the amount of data lost is more useful than an
> unspecific indication of "absence of reliability" and should be
> relatively easy to provide.
> 
> reqs-06> Please note that if an unreliable transport protocol is used,
> reqs-06> reliability can be provided by higher layers. In such a case
> reqs-06> only lack of overall reliability MUST be indicated.
> 
> And end-to-end purist might argue that even if the underlying
> transport protocol is used, reliability must be provided at higher
> layers too.  See the "application layer ACK" debate on the ipfix-req
> list a few days ago.
> 
>   "Reliability can be supported by suitable transport protocols.  In
>   any case only end-to-end flow information loss between exporter and
>   collector need be indicated."
> 
> reqs-06> For example reordering could be dealt with by adding a
> reqs-06> sequence number to each packet.

when I read this whole section of the requirement draft I'm not sure what
are "exactly" the requirements. It's vague, full of loopholes due to the
"CANs" associated with specific situations...

> 
> Or order may not be relevant since the exact arrival time of a piece
> of flow information doesn't matter - the timestamps in the flow record
> will provide sufficient sequence information for the collector.
> -- 
> Simon.
>

---

From: Simon Leinen <[email-masked]>
Date: 05 Oct 2002 11:41:58 +0200

> But how the exporter would know which records were lost in the
> network unless we have (application layer) ACKs?

I see - my "MUST be indicated" is confusing too.  The exporter
wouldn't need to know, but the collector must be able to determine
that something was lost, and "how much", where I intentionally leave
open the accuracy - the collector won't be able to reconstruct
everything anyway.  So what about this:

   "If some flow information data cannot be delivered to the collector
   in a timely manner, the collector MUST be able to estimate the
   amount of data missing.  Possible reasons for failure to deliver
   data include loss of packets on the network path, or resource
   shortage at the exporter."

As an example, the flow export mechanism I use sends me packets of
flow records, and each packet has a flow record sequence number.  So I
can determine exactly how many flows I have missed.  This information
is important for diagnosis of situations where there is information
loss (some, but not all of this information loss would have been
prevented by using a reliable transport protocol).

For my particular applications, I'm mostly interested in the byte
amounts of metered traffic.  So if a flow export protocol had sequence
numbers representing the cumulative total bytes metered, that would be
very useful to me - I really would like to know what percentage of
traffic could not be accounted for.  (In addition to that, I would
still want to know how many flow records I missed.)
-- 
Simon Leinen				       [email-masked]
SWITCH				   http://www.switch.ch/misc/leinen/

	       Computers hate being anthropomorphized.

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: "Peter Ludemann" <[email-masked]>
Date: Sat, 5 Oct 2002 11:30:55 -0700

This note is in two parts: comments on reliability requirements as for the
export protocol and comments on how data loss is reported. I've added
snippets on how we do this in CRANE and suggest that the other protocol
experts describe how they support these functionalities.

At the protocol level, the minimal requirement is that there be some kind of
ACK from the collector to the exporter (as Reinaldo Penno noted). We also
need some kind of unique key that allows de-duplication at the collector.
[In CRANE, we use the exporter "boot time" + 32-bit record sequence number;
the de-duplicator takes multiple data streams from the CRANE collectors and
de-duplicates them.]

(I assume that everyone agrees that de-duplication is required at the
collector.)

For a high-availability solution, we should have the ability to send to
multiple collectors and for the exporter to be able to switch from one
collector to another. The timing issues for fail-over are outside the
protocol; but the capabilities must be part of it. [In our CRANE
implementation, we use a timeout on ACKs for switching to the alternate
receiver. We intend to improve this in future with some kind of load
balancing by observing the rate at which data is being ACKed. It turns out
to be a bit tricky to get this working well for both high-speed transmission
(3000+ rec/sec) and low-speed (1 rec/sec or slower). We only send an ACK
only when the data has been persisted - this behavior is outside the scope
of the protocol itself and therefore should be a SHOULD, not a MUST in the
standard.]

However, there is still the possibility of data loss: the exporter may have
data areas fill up before an ACK arrives. The exporter must therefore throw
away some data. The exporter knows AT LEAST how many records were received
(it is possible that more were received, but the ACKs were lost). So, if the
exporter is forced to throw away data, it can report about what it threw
away.
   (There are other causes of data loss than running of memory:
    such as running out of CPU, or acting as a proxy and losing
    data; these can all be covered by a similar mechanism.)

There will inevitably be some information loss in reporting data loss. The
minimal information that the exporter needs to keep is: number of records
not ACKed, totals for the metrics that were not ACKed.

The information that are kept about data loss are application-dependent --
which totals are important? is some high-level aggregation on the exporter
possible? etc. For a specific data model (e.g., the standard data exported
by Cisco routers using NetFlow v8), a separate specification could state
exactly what must be reported for data loss. [Note: NetFlow v8 is only being
used as an example of data *content* - because the protocol lacks
reliability, the collector can only report statistics such as number of
records that were lost, detected by sequence number gaps.] The separate
specification could also state the rules for throwing away data: e.g., throw
away oldest, newest, smallest, etc.

As an example of data loss reporting, suppose that we are exporting a single
metric and that the statistics on the exporter are:
     # flow records sent (not counting retransmitted) 1000
     # flow records ACKed                              900
     sum of metric all records                       10000
     sum of metric in all ACKed records               8000
From which we derive the following:
     average metric per record                          10
     average metric per ACKed record                     8.889
     average metric per un-ACKed record                 20
And on the receiver, we have:
     # flow records received (de-duplicated)           950
     sum of metric in all received records            9200
     average metric per received record                  9.684
The receiver notes that it did not receive 50 records; it can estimate the
sum of metric in these as 50 * 20 = 1000, for an estimated total of the
metric of 10200. (Something fancier could be done by using variance
calculations, combined with other metrics.)

[In CRANE, we provide two mechanisms for reporting loss:
 - the exporter can define an additional set of fields that it
   exports periodically with data loss values;
 - a status request from the collector, also reported as a set
   of fields.
 The content of these records is defined by the exporter, using the general
template mechanism.]

So, to sum up:
   - require application-level ACKs in the protocol (SHOULD be after
     data are persisted)
   - require enable de-duplication of received records
   - require mechanism for reporting data loss in the exporter -
     the details of this mechanism are outside the scope of the
     protocol except that the protocol must be general enough to
     allow them
   - individual information models (which use the flow export protocol)
     SHOULD describe data loss reporting

- peter

---

From: [email-masked]
Date: Mon, 07 Oct 2002 10:23:02 -0400

>    shortage at the exporter."

	Not sure I like that. How about when the exporter starts
	talking to another collector. How about in overload mode
	when the collector is just dropping flows. 

	I think the exporter is in the best position to estimate how 
	much data was lost. Once more of the protocol is nailed down 
	we can better define how to measure loss.

---

From: Benoit Claise <[email-masked]>
Date: Mon, 07 Oct 2002 16:34:40 +0200

All,

[In my previous email, I thought I took the latest email in the thread, 
I was wrong. Sorry!]

>prevented by using a reliable transport protocol).
The sequence ID per observation domain on the exporter is exactly what 
we use in NetFlow to detect we lost some export packets.
And we don't need application level ACK.

Regards, Benoit.

---

From: Benoit Claise <[email-masked]>
Date: Mon, 07 Oct 2002 17:24:00 +0200

Paul,

>>   shortage at the exporter."
I even prefer this proposal compared to the previous one from Simon.

>	when the collector is just dropping flows.
What is the issue, you could add "overload on the collector" in the list 
of example?
What is important is the collector must know the amount of data missing, 
whereever it comes from: exporter, path or collector.
If you send this information from the exporter, you could possibly not 
report missing information.
Ex:
- the transport session breaks, what happened to the flow records which 
were sent?
- the transport protocol acknowlegde some packets but they are not 
handled correctly by the collector (CPU too busy)

BTW, I agree with you Paul about the "timely manner" which is confusing 
and maybe not necessary.

Regards, Benoit.

---

From: [email-masked]
Date: Mon, 07 Oct 2002 11:50:45 -0400

> - the transport session breaks, what happened to the flow records which

	Exaclty, see below.

> handled correctly by the collector (CPU too busy)

	Again, I think it is difficult to describe how to measure loss
	without a protocol definition to analyze. 

	But my view is that we will be in a **better** position (not perfect)
	to design a solution by measuring data loss from the exporter's
perspective
	rather than the collector. The reason being the exporter talks
	to many collectors and thus has the common view, not the collector.

	So lets assume we have a simple message level ACK for every 100
	messages. I have an outstanding ACK when the device looses connectivity
	to the collector. So the device resends the 100 messages to 
	collector #2. No need to report any data loss.

	If on the other hand if the device never resends those messages
(because of
	a design choice) then the device needs to report the potential loss 
	to the new collector (or how ever that info is to be retrieved, maybe
SNMP).
	
	If we examine this from the collector perspective, lets assume it never
	receives any of the 100 messages that were outstanding. How does it
	even know to report a loss. 

	If somehow it detected the loss, how does it know the messages
	were resent?

	If the collector recieved the messages but just the ACK got lost and 
	the exporter resent the messages, it didin't report a loss anyway. If 
	the exporter didn't resend then it falsely reported the loss. But I 
	think this is preferrable to the collector scenario where no loss was 
	reported when there was one. If I as the administrator know that the
	potential loss is the upper bound then the operator can make an
informed 
	decision to look into the matter futher or not.

	I don't think we can design a 100% guaranteed solution but one
	from the exporter's perspective I think lends itself to better
	possibilities.

	Paul

---

From: Benoit Claise <[email-masked]>
Date: Tue, 08 Oct 2002 12:24:17 +0200

Paul,

>	without a protocol definition to analyze. 
Agreed. So maybe we should maybe postpone the discussion...

>	even know to report a loss. 
For example, by using a meaningfull sequence ID; increased by the number 
of flow records in the export packet

>	If somehow it detected the loss, how does it know the messages
Again with the sequence ID because there is no gap.

>	from the exporter's perspective I think lends itself to better
I really think that the solution is a mix of both: report loss on the 
collector and on the exporter.
Collector loss can be due to: exporter, path, collector
Exporter loss can be due to: exporter, path
The collector loss will be greather than the exporter loss.
But for me, what really counts is the loss of flow records saved on my 
hard disk, so from a collector point of view.

Regards, Benoit.

---

From: [email-masked]
Date: Tue, 08 Oct 2002 09:55:03 -0400

> Agreed. So maybe we should maybe postpone the discussion...

	Lets do that. 

	However, I think this discussion shows the req section should be a 
	little more general to leave open implementation details. Here is 
	my proposal..

 	If flow information is lost, a means to gauge the amount of
	loss MUST be available from the Exporter, Collector or both. 
	Possible reasons for flow data loss include loss of packets on 
	the network path, resource shortage at the exporter, Collector
	crash, etc...

	Paul

---

From: Benoit Claise <[email-masked]>
Date: Wed, 09 Oct 2002 10:33:34 +0200

>	the network path, resource shortage at the exporter, Collector
I would prefer Simon's proposal.

Unless you want to say:
 	If flow information is lost, a means to gauge the amount of
	loss MUST be available from the Exporter _or the_ Collector or both. 
	Possible reasons for flow data loss include loss of packets on 
	the network path, resource shortage at the exporter, Collector
	crash, etc...

Regards, Benoit.

---

From: [email-masked]
Date: Wed, 09 Oct 2002 08:33:27 -0400

>   shortage at the exporter."

	I have 3 concerns with Simon's proposal. 

	1) It appears to limit info on data loss to delivery of messages. 
	   I think the door should be left open, in the requiremetns anyway,
	   for application level info  not just message delivery.

	2) It limits the data loss info to the collector. As I said before
	   it **may** be better at the Exporter. We wont know until we have
	    more detail on the protocol.

	3) The added phrase "in a timely manner" is a bit confusing to me.

>         the network path, resource shortage at the exporter, Collector

	This is fine with me.

	Paul