Subject: RE: Re: [ipfix-eval] Simon's evaluation draft contribution

From: "Tal Givoly" <[email-masked]>
Date: Mon, 14 Oct 2002 22:45:24 -0700

Simon,

I truly appreciate your comprehensive analysis of all protocols in question.

May I try to make a few corrections regarding CRANE:

One of its main design goals was, in fact, efficiency. Under circumstances
in which exists no network congestion or failure in receivers (servers) it
has been benchmarked to perform 90% performance of NetFlow v5 (which is even
more efficient than NetFlow v9). In fact, performance/efficiency/scalability
has been a design goal. The goals of CRANE are, in the following order:
- Reliability - working with carriers on absorbing usage information from
routers for the purpose of usage-sensitive billing has lead us to believe
that the weakest link in the chain is the interface between the mediation
system and the network/service element. This was the prime goal of
developing an additional protocol where NetFlow provided an efficient
solution.
- Scalable - both in the sense of a large, distributed network as well as in
the sense of efficiency - making the overall cost of collecting usage
information least while maintaining the rest of the requirements.

Basically, CRANE does embody your main conclusion - we came to develop it
due to the shortcoming of all other existing flow export protocols at the
time (NetFlow v1-v8, LFAP v1-4, RADIUS, DIAMETER, SNMP and TopFlow as well
as other protocols not evaluated here used in Telco applications, such as
AMA, ASN.1, etc.). All we essentially did was add reliability and
extensibility. Our desire was not to necessitate the need for any further
protocol replacement, so for this we added the version and template
negotiation.

I believe not enough emphasis has been placed on the backward and forward
compatibility, available in modem communications and in fax communications
today due to negotiation protocols that then set the feature/capabilities of
the subsequent dialog. The template negotiation, and the preceding UDP-based
protocol-version negotiation, both offer the protocol the ability to be
upgraded over time with minimum consequences. It is our expectation that
network/service elements would implement one version of the protocol while
servers would implement any number of versions concurrently and communicate
freely with network/service elements of various versions.

Regarding NetFlow v9. I don't believe that adding reliability is a simple
thing based on our extensive experience hardening NetFlow-based solutions
and analyzing their reliability. Today, the naive assumption is that this
would be done based on some distributed voting algorithm. Those aware of the
domain of distributed voting must realize that a majority vote would require
3 redundant collectors. Other more computationally complex and less reliable
voting methods can settle for less redundancy, but these introduce
compromise. First of all, NetFlow currently offers only two. Performing
back-end de-duplication may require persistence of all records and detailed
comparison. Basically, adding reliability mechanisms over NetFlow v9 offsets
the benefit of having a simple protocol. The network/service element can
export data at a higher rate, but there is excessive cost at the receiving
end. This is not required by LFAP, CRANE, DIAMETER or IPDR to achieve higher
levels of reliability.

Regarding LFAP. To best of my understanding, fail-over may cause some data
loss during fail-over (Paul/Mike, please correct me if I'm wrong).
Therefore, the protocol accepts this and reports on potential data loss.
Using templates, or any form of reporting via protocol extensions, supported
by CRANE, IPDR and DIAMETER, it is possible to report loss of usage data,
sampling, or any other form of modification of data attributes. The fact
that templates can be negotiated during sessions with CRANE, provides the
added value that features can be added that weren't configured originally
(perhaps due to user intervention). This avoids the need to bring down the
connection during configuration changes that are likely to occur throughout
a deployment.

Minor corrections:
2.1.2 - you mention that the client will suppress keys unselected by Server,
however, the client can do this optionally, to allow the client the utmost
flexibility to achieve best efficiency from perspective of client. It may
make sense for clients to know if no consumer is interested in the
information provided. For fields that are expensive to populate, the client
may elect to avoid generating their value if no consumer is interested in
them, however, in other cases, clients may find it more efficient to export
everything and have the consumer sort out what to do with the data. This
provides flexibility as well as efficiency.

4.2 - CRANE is oblivious as to whether sampling has taken place or not. An
easy extension would be to use different templates - one for sampled data
and one for non-sampled data and to have both work alternatively or
concurrently. Our experience leads us to believe that sample-based billing
is a very unlikely usage despite the prolonged debate on the matter.
However, I completely agree that there are many other applications that may
be satisfied by sampled data, and there is no limitation on whether to use
them or not. I believe there is great benefit in separating the data models
and the transport. Flow control available within CRANE implementations can
be used to throttle producer and to allow for transition between sampled and
non-sampled mode - if both are supported. As this is subject to specific
applications, it may need additional refinement as part of the requirement
specification.

4.3 - If distinct templates are added for sampled mode of operation, then
providing attributes to describe the sampling parameters can be easily
included or implied by selection of specific templates. Flow control can be
used to select mode of sampling.

5. As mentioned above, evolution of CRANE came from the deficiencies in
NetFlow v1-v8, LFAP v1-4, RADIUS, DIAMETER, SNMP and TopFlow as well as
other protocols not evaluated here. Carriers have much criticism as to the
cost of converting non-fault-tolerant flow-based exports mechanisms to
carrier-grade-reliable. We have attempted to address this need with CRANE
while not compromising the performance. Since we expect that carriers and
equipment vendors would want a protocol that is extensible to support any
future data export need, we added requirements such as flexibility and
extensibility.

I contend that what we did with CRANE is, in fact, exactly what you actually
recommend to do - that is: start with something simple like NetFlow and
update it by adding transport, reliability, and redundant configurations. We
also added extensibility. What other requirements have we dealt with? None,
essentially. What has been added in CRANE beyond these requirements that is
not truly required in a protocol we intend to avoid replacing?

Thanks again for your comprehensive analysis. Comments are more than
welcome.

Tal
____________________________________________

Tal Givoly
Chief Scientist
XACCT Technologies, Inc.
2900 Lakeside Drive
Santa Clara, CA 95054
http://www.xacct.com

Direct:  408-330-5747
Tel:     408-654-9900 x 5747
Fax:     408-654-9904
E-mail:  mailto:[email-masked]
____________________________________________

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Vamsidhar Valluri <[email-masked]>
Date: Wed, 16 Oct 2002 23:47:24 -0700 (PDT)

Tal,
  Please see inline

Regarding NetFlow v9. I don't believe that adding reliability is a simple
thing based on our extensive experience hardening NetFlow-based solutions
and analyzing their reliability.

vamsi>CRANE has done a good job in addressing drawbacks of earlier netflow
versions w.r.t reliability. Hopefully this experience will help us to
comeup with a good solution.

 Today, the naive assumption is that this
would be done based on some distributed voting algorithm. Those aware of
the
domain of distributed voting must realize that a majority vote would
require
3 redundant collectors. Other more computationally complex and less
reliable
voting methods can settle for less redundancy, but these introduce
compromise.

vamsi> I guess link level fail over can be addressed by transport
protocols like SCTP (may be multi-homing). For server(collector)
redundancy we can look at RSERPOOL.

 First of all, NetFlow currently offers only two. Performing
back-end de-duplication may require persistence of all records and
detailed comparison. Basically, adding reliability mechanisms over NetFlow
v9 offsets the benefit of having a simple protocol.

vamsi> Sure, it depends on how simple are the extensions related to
reliability.

thanks
-vamsi

 The network/service element can
export data at a higher rate, but there is excessive cost at the receiving
end. This is not required by LFAP, CRANE, DIAMETER or IPDR to achieve
higher
levels of reliability.

---

From: "Tal Givoly" <[email-masked]>
Date: Fri, 18 Oct 2002 17:01:13 -0700

Vamsi,

Please see inline:

tal> CRANE would not have become such an efficient and reliable solution had
we not learned from our experience working with NetFlow, efficiently
processing it and hardening its usage for business-critical applications.
NetFlow definitely helped pave the way for the industry awareness to
flow-level granularity of information as well as meaningful aggregates (as
of v8). We also worked closely with Cisco on extending it to support
customer needs with features added in v8 and v9.

tal> One of the problems is that the whole issue of application-level
acknowledgement must be added and actual bi-directional communication is
implied by this - something totally absent of NetFlow v9. Just adding SCTP
doesn't address the reliability requirement without more significant
modifications. Furthermore, the SCTP layer would not provide hints for
efficient de-duplication. The flow export protocol itself must be
instrumented to support efficient fault tolerance through minimal redundancy
(typically, it is more efficient if it isn't even co-located).

Furthermore, as I commented earlier, replicating the stream in any form to
multiple recipients creates one of the most expensive forms of
fault-tolerance offered by a protocol. Multiple receivers would need to
consume the data and a voting algorithm would need to be used to select the
correct result. With majority voting (or other voting algorithms), this
would require 2n+1 redundancy, i.e. 3 nodes to sustain a single failure.

For all these replicated streams, in order to avoid single-point failures,
you would need:
- separate controllers/network interfaces
- network bandwidth allocated to all receivers throughout the WAN, including
independent paths (VCs to all redundant collection units)
- computer systems running the receivers, processing full load (not just
load of a potentially failed node).
- excessive persistence at nodes and/or "thick" communication pipes between
nodes - to perform voting algorithms

This basically prevents N-M redundancy (N active nodes and M standby nodes).
For instance, using reliability methods employed in CRANE it is perfectly
feasible to have 10 distributed primary collection points with 2 regional or
centralized secondary units. If any of the primary distributed collection
points fails, data is sent to one of the 2 secondary regionally located
servers. Data can be efficiently de-duplicated between these streams.

The issue of cost-effective reliability is a major concern to carriers.
One-to-one or, as in NetFlow v9's case, One-to-two redundancy, is
unacceptable in terms of cost.

When considering the cost of a solution, the cost of software development
should also be factored in. The cost to develop NetFlow v9-based solutions
that do not include reliability are, indeed, low. However, the cost to
develop a reliable solution for NetFlow is very high due to complexity of
distributed voting algorithms and fault-tolerant on non-cooperating sensors
(which is what we have here).

Rather than make the network/service element non-cooperative to reliable
data collection, I believe IPFIX would benefit dramatically by making it
cost-effective to develop and deploy both efficient and reliable solutions.

tal> We have several proposed protocol options here. We can either begin
with a non-reliable solution and extend it by bolting on reliability
extensions or take a solution that already has reliability designed-in. I
admit that complexity of a protocol does increase somewhat when reliability
is introduced. However, the runtime benefits are very significant (as
described above) when deploying reliable solutions. Furthermore, the impact
when the reliability is not needed is negligible on the network/service
element.

Tal

-----Original Message-----
From: Vamsidhar Valluri [mailto:[email-masked]]
Sent: Wednesday, October 16, 2002 11:47 PM
To: Tal Givoly
Cc: [email-masked]; [email-masked]; Peter Ludemann
Subject: RE: Re: [ipfix-eval] Simon's evaluation draft contribution

Tal,
  Please see inline

thanks
-vamsi

---

From: Simon Leinen <[email-masked]>
Date: 25 Oct 2002 09:42:56 +0200

> I truly appreciate your comprehensive analysis of all protocols in

Thanks! (I wished all specifications were as readable as the CRANE one ;-)

> One of its main design goals was, in fact, efficiency.

I'm not worried about CRANE's on-the-wire efficiency or the
possibility of high-performance implementations.

What I'm concerned about is the complexity (and associated cost of
implementation) of the CRANE protocol, in particular template
negotiation, but also status requests.

> Under circumstances in which exists no network congestion or failure
> in receivers (servers) it has been benchmarked to perform 90%
> performance of NetFlow v5 (which is even more efficient than NetFlow
> v9). In fact, performance/efficiency/scalability has been a design
> goal. The goals of CRANE are, in the following order:

> - Reliability - working with carriers on absorbing usage information
> from routers for the purpose of usage-sensitive billing has lead us
> to believe that the weakest link in the chain is the interface
> between the mediation system and the network/service element. This
> was the prime goal of developing an additional protocol where
> NetFlow provided an efficient solution.

> - Scalable - both in the sense of a large, distributed network as
> well as in the sense of efficiency - making the overall cost of
> collecting usage information least while maintaining the rest of the
> requirements.
[...]
-- 
Simon.

---

From: Simon Leinen <[email-masked]>
Date: 25 Oct 2002 09:46:25 +0200

> One of its main design goals was, in fact, efficiency. Under
> circumstances in which exists no network congestion or failure in
> receivers (servers) it has been benchmarked to perform 90%

> NetFlow provided an efficient solution.

Yes, CRANE presumably (I'm not an expert in engineering protocols for
high reliability) provides features that allow telco-billing-grade
accounting systems to be built.

On the other hand (1) unreliable flow information export protocols are
heavily used, (2) even some applications that involve charging
tolerate a modest amount of data loss, and (3) total reliability
cannot be achieved in this world.

My position is that we should be careful in how far we should depart
from current (unreliable) practice in order to make the protocol
useful for applications that seem to have higher, although probably
not infinitely high, demands on reliability.

Mandating support of a "reliable" transport protocol (TCP or SCTP)
seems to be a very good first step, because this makes the
congestion-friendliness requirement easy to fulfill, uses
well-understood technology (at least for TCP :-), and creates a
feedback loop between the exporter and the collector (including the
network path between them).  This enables the entire system to adapt
to different bottlenecks.  Note that reasonable methods of adaptation
include the metering system shedding load for the export/collection
system, for example by changing to sampled accounting.  This would
probably seem scary to mediation/billing folks but is widely accepted
in the community (statically configured sampling; I think sampling
that adapts to load would be found even more acceptable).

Application-level acknowledgements are a way to further increase
reliability at relatively low cost.  (Whether those could already be
sent "when data has been written to disk" or only when the bill has
been paid is debatable.)

Sequence numbers for flow records, or bunches of flow records, are
another approach that is lightweight.  It has proven useful with
unreliable export protocols, but is also useful for de-duplication in
redundant configurations.

Beyond this I see diminishing returns.  Mechanisms for maintaining
total consistency between redundant components tend to introduce their
own reliability issues.

> collecting usage information least while maintaining the rest of the

Frankly I cannot make out huge difference between the protocols with
respect to scalability.  Probably you are hinting at 
 
> Basically, CRANE does embody your main conclusion - we came to
> develop it due to the shortcoming of all other existing flow export
> protocols at the time (NetFlow v1-v8, LFAP v1-4, RADIUS, DIAMETER,
> SNMP and TopFlow as well as other protocols not evaluated here used
> in Telco applications, such as AMA, ASN.1, etc.). All we essentially
> did was add reliability and extensibility. Our desire was not to
> necessitate the need for any further protocol replacement, so for
> this we added the version and template negotiation.

So CRANE is like (previous) LFAP + reliability + extensibility?
(and IPDR is like (previous) NetFlow + reliability + extensibility?)
-- 
Simon.

---

From: Randy Bush <[email-masked]>
Date: Fri, 25 Oct 2002 05:21:35 -0700

> not infinitely high, demands on reliability.

indeed.  folk might find it useful look at this wg's charter.  this was
meant to be a rather focused effort.

randy

---

From: Robert Lowe <[email-masked]>
Date: Fri, 25 Oct 2002 12:54:10 -0500

Simon Leinen wrote:

...

> that adapts to load would be found even more acceptable).

Dynamically adjusting sampling to load was discussed briefly before, 
but without a clear indication that it could be safely done.  Do you 
have experience that suggests it can be done without compromising 
data integrity?  I agree that switching to a sampling strategy under 
load could a good thing.  See:

    http://ipfix.doit.wisc.edu/archive/0079.html (and other postings 
    by Peter Phaal regarding sampling in general) 

Interestingly enough, he seemed to indicate that using UDP actually 
worked better in practice when employing packet sampling.  However, 
moving to sampling requires reliable notification/communication between 
the exporter and collector.

-Robert

---

From: Benoit Claise <[email-masked]>
Date: Mon, 28 Oct 2002 16:38:28 +0100

Dear all,

>meant to be a rather focused effort.
I could know agree with Simon and Randy.
Even if I understand the CRANE people to defend at all costs the 
reliability at all levels because their specific applications are mainly 
focused on billing, IP flows could be  used for a lot of applications 
where a few flow records lost will not be a big deal. Ex: capacity 
planning, user/application profiling, traffic engineering, etc...

Regards, Benoit.