Subject: Re: [ipfix-req] Collector Failover as a requirement

From: Mark Fullmer <[email-masked]>
Date: Sun, 3 Mar 2002 17:12:38 -0500

(sorry for the late reply...)

On Mon, Feb 25, 2002 at 12:21:29AM -0800, Peter Ludemann wrote:
> 
> In a normal deployment, with the collector close to the sender, "reliable"
> isn't more resource-intensive. And TCP is a reasonable choice; the
> latest NFS is done over TCP because it's as efficient as using
> UDP and avoids re-inventing wheels for things like congestion control.
> When a connection is broken, TCP uses longer and longer intervals when
> retransmitting, so the execution overhead remains small.

In an _ideal_ deployment the collector is directly connected to
the exporter.  Many times this is not the case.  Rack space in
co-lo facilities may not be availabe, the cost of maintaining
collectors around the country may be too high, routers may not
have the space for a linecard with a LAN interface for the
collector, etc.  Most NetFlow deployments I've seen use
centralized collectors...

The buffer space for implementing exports over TCP may not
be available on the hardware -- ie routers that have been
augmented with flow export capability.  Flow exports can
be very bursty and require a lot of socket buffer space
that is not available to some vendor implementations.

Depending on the application a small percentage of lost flow
records may not be an issue.  Example, one router exported
around 70,000,000 flows in a day (a few days ago).  The
collector stats show that around 19,000 were lost, an export
record rate of around 2.7x10^-4, or a little over 600
packets.  For traffic engineering purposes this doesn't
matter, especially since the flows are generated from a
sample rate of 1/100.  If this were an issue the collector
could be directly connected to the router to reduce the loss.

For an IDS or other applications lost export records may be more of
an issue.  An IDS exporter would probably have more resources
and be able to use TCP, a disk based backing store, or some other
methods to make the exporter -> collector connection more reliable.

There are two camps of vendors here, one which has the resources
to implement TCP or some other form of retransmits for the
flow exports, and one that does not.  For IPFIX to be well
adopted the export mechanism needs to satisfy both.  This
isn't difficult with a separate control and export mechanism,
and can easly be added to draft-gsadasiv-ipfix-proposal-00.txt.

> One more thing about unreliable protocols ... they're attractive in theory
> but in practice we find ourselves building workarounds to make them
> seem reliable (this applies to both NetFlow and SNMP). Any data delivery
> method that avoids reliability simply pushes things on to another
> part of the system.

If using a reliable and congestion aware protocol were that
big of an issue for the applications NetFlow is used for
today it would not be so widely deployed.  IPFIX
needs to take this into account while being flexible
enough to support a "better" (TCP/SCTP) transport for
the exports.

> 
> >   o make the control connection in draft-gsadasiv-ipfix-proposal-00.txt
> >     required and TCP.  This removes the need for retransmits or
> >     periodic updates of templates to the collector.  When the collector
> >     boots up it won't have to wait for templates.  This also can
> >     stop the router from generating flows to a non existant collector.
> 
> I don't think that only TCP should be required. I would also allow SCTP
> ... it's better than TCP for fail-over, but not widely available yet.

TCP with a keepalive mechanism will work fine.  Requiring SCTP is
an unnecessary deployment barrier.

char:~% uname -r
4.4-RELEASE
char:~% man -k sctp
sctp: nothing appropriate

(yes, I know there are SCTP implementation for FreeBSD, Linux, etc)
They're not widely available/integrated/burned in yet.

> >   o Implement a keepalive mechanism where the collector will ACK the
> >     highest sequence number flow record it has processed per (any?)
> >     "observation domain"   The keepalive can be optional -- ie the
> >     exporter can send a keepalive of N to inform the collector it
> >     expects a keepalive every N seconds or 0 to disable it.  Default
> >     disabled.
> 
> This is essentially the way that CRANE works. However, I don't think
> that a keepalive gives any advantage ... the sender should just time
> out and failover if it doesn't receive a data ACK in time (there's no
> need for keepalives when no data are being sent). [Downstream
> deduplication will always be needed for handling failures because
> ACKs can get lost.]

Acking every data packet is a lot different from a keepalive...

mark

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Mark Fullmer <[email-masked]>
Date: Sun, 3 Mar 2002 17:24:39 -0500

On Mon, Feb 25, 2002 at 10:28:43PM -0500, Kevin Zhang wrote:

> This is a very good idea.  keepalive can be piggybacked through ACKs from collectors.  This will not only serve as indication of collector fault conditions as well as congestion, it can also improve transport reliability.  This should be done on the IPFIX protocol layer whether TCP or UDP is used as the transport protocol.

Yes, a keepalive will be required regardless of the flow export
(data) channel to support collector failover.  This complicates
multicast a little bit since the multicast collectors will need
some way to access the templates.  Multicast clients may need
to connect to the exporter for this information...

> > 
> Simply counting the lost flow records may not be enough as values associated with each record are different (e.g. start of a flow, end of a flow, interim records).  We should really base our design over a reliable transport layer that offers congestion control.  UDP can optionally be used for limited configurations.

Ideally the exporter will periodically send down other
information on the control channel such flows dropped
internally (to transmit buffer, etc), packet size 
distributions, etc.  draft-gsadasiv-ipfix-proposal-00
seems to allude to this.

mark

---

From: Mark Fullmer <[email-masked]>
Date: Sun, 3 Mar 2002 17:41:23 -0500

On Mon, Feb 25, 2002 at 10:30:03AM -0800, Ganesh Sadasivan wrote:

> > >   o Change the sequence number in draft-gsadasiv-ipfix-proposal-00.txt
> > >     to number of flows instead of number of packets so that the collector
> > >     can count lost flow records as in NetFlow v5,6,7,8.
> 
> There are other means to achive the same in the new protocol. Basically the
> flowsets tell the length and using the size of each record in the flowset, one
> can calculate the # of records.

I've read through the draft again and I don't see how a collector
can figure out how many flow exports it missed when a packet
is dropped. The size of the record and the flowset(s) are
unknown for a missing packet.

mark

---

From: Ganesh Sadasivan <[email-masked]>
Date: Sun, 03 Mar 2002 19:41:27 -0800

Hi Mark,

> unknown for a missing packet.

An export packet can have records of more than one flow set (type).
Probably a cumulative flow record # per flow set is what is necessary
to figure the loss of flow records per type. Getting a general count I
guess is not much useful.
This is my initial thought. Will get back to you with a clearer answer.
Ganesh

---

From: "Peter Ludemann" <[email-masked]>
Date: Sun, 3 Mar 2002 23:06:59 -0800

Keepalives are not needed with TCP; ACKs with timeouts are needed (TCP's
ACKs are not sufficient). In a multicast situation, the ACKs can be kept on
a per-receiver basis (but why do you need this? failover can be done other
ways).

As to lost data ... with multiple output "templates", one could output
information about losses in a separate template (which begs the question
about losing loss data).. Alternatively, the information could be available
in a MIB.

- peter

> -----Original Message-----
> From: majordomo listserver [mailto:[email-masked]]On Behalf
> Of Mark Fullmer
> Sent: Sunday, March 03, 2002 2:25 PM
> To: Kevin Zhang
> Cc: Peter Ludemann; [email-masked]
> Subject: Re: [ipfix-req] Collector Failover as a requirement
>
>
> > This is a very good idea.  keepalive can be piggybacked through
> ACKs from collectors.  This will not only serve as indication of
> collector fault conditions as well as congestion, it can also
> improve transport reliability.  This should be done on the IPFIX
> protocol layer whether TCP or UDP is used as the transport protocol.
>
> > Simply counting the lost flow records may not be enough as
> values associated with each record are different (e.g. start of a
> flow, end of a flow, interim records).  We should really base our
> design over a reliable transport layer that offers congestion
> control.  UDP can optionally be used for limited configurations.
>

---

From: Mark Fullmer <[email-masked]>
Date: Tue, 5 Mar 2002 01:10:46 -0500

> a per-receiver basis (but why do you need this? failover can be done other

The context of my original message was using TCP for the control channel.
Regardless, a periodic keepalive is still needed even if the collector
is ACKing messages from the exporter.  Consider the case when the exporter
is not sending any data.  It's also important for the collector to be
able to detect a dead exporter so it can mark the data period as such
instead of reporting no flow activity.

Reliable multicast is an entirely different topic.  I'm assuming a
simple case where the exporter is using unreliable (no ACK's) UDP
to a multicast address.  With the current NetFlow (v1,5,6,7,8.x) 
multicast is a non issue since the templates are static C struct's.
The exports packets can be sent (actually reflected from unicast since
Cisco doesn't implement this) out unmodified.

I've used multicast NetFlow to reduce fanout when multiple collectors
are processing the same data, not for failover.

> about losing loss data).. Alternatively, the information could be available

Or as a field in the template...

mark