Subject: [ipfix-req] Collector Failover as a requirement

From: "Peter Ludemann" <[email-masked]>
Date: Sat, 23 Feb 2002 11:51:56 -0800

This note is about requirements for reliability and underlying transport
protocols and is NOT a critique of
http://www.ietf.org/internet-drafts/draft-gsadasiv-ipfix-proposal-00.txt.
Section 4.2.3 of that document, which describes Collector Failover in
generic terms, leads me to believe that some people in the WG might have
made invalid assumptions about the transport protocol (which is "beyond the
scope of [the] document" (Section 1.1)) ... and these invalid assumptions
lead to omitting important requirements.

TCP's keepalive is not sufficient for detecting loss of connectivity, as
sections 4.2.0 and 4.2.1 suggest (there is no guarantee that a collector
crash will send any kind of signal to the sender, which is why TCP has such
a complex start-up and shut-down; keepalive is not guaranteed to be
implemented and even if implemented its response is measured in hours ...
for more on this, see (at the risk of copyright violation of Stevens'
"TCP/IP Illustrated", Chapter 23)
http://www.comp.nus.edu.sg/~shenru/tcpip/tcp_keep.htm). So, unless we
mandate SCTP for transport, the flow export needs to define its own
equivalent of keepalive and behaviour on failover. I don't think that we can
mandate SCTP because it's not yet widely available.

Therefore, to satisfy the Reliability requirement, the data export protocol
must be able to provide failure detection so that failover can be done
quickly. The data export protocol must also specify the underlying
transports that it is designed to work on.

- peter

----
Peter Ludemann           [email-masked]
Chief Architect          +1.408.330.5732
XACCT Technologies, Inc.
2900 Lakeside Drive      Santa Clara, CA 95054

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Mark Fullmer <[email-masked]>
Date: Sat, 23 Feb 2002 23:54:37 -0500

> a complex start-up and shut-down; keepalive is not guaranteed to be

A few suggestions to help aim at a middle ground between the
lightweight/best effort NetFlow (like) protocol vs. the 
more resource intensive "reliable" drafts/implementations.

  o make the control connection in draft-gsadasiv-ipfix-proposal-00.txt
    required and TCP.  This removes the need for retransmits or
    periodic updates of templates to the collector.  When the collector
    boots up it won't have to wait for templates.  This also can
    stop the router from generating flows to a non existant collector.

  o Implement a keepalive mechanism where the collector will ACK the
    highest sequence number flow record it has processed per (any?)
    "observation domain"   The keepalive can be optional -- ie the
    exporter can send a keepalive of N to inform the collector it
    expects a keepalive every N seconds or 0 to disable it.  Default
    disabled.

  o The exporter will send the collector a message on the control
    connection indicating where to listen for flow records
    (ie what protocol UDP/TCP/SCTP) and addressing information
    (port, IP address).  Must UDP May TCP/SCTP.

  o Change the sequence number in draft-gsadasiv-ipfix-proposal-00.txt
    to number of flows instead of number of packets so that the collector
    can count lost flow records as in NetFlow v5,6,7,8.

  o Ignore the configuration of the exporter issue for now, but implement
    the control connection message format such that it can be
    tackled later, or implemented by a vendor and then submitted for 
    IPFIX v2.  Ie, require the exporter to NAK any messages sent by
    the collector it can not decode.

mark

---

From: "Peter Ludemann" <[email-masked]>
Date: Mon, 25 Feb 2002 00:21:29 -0800

My comments interspersed.

> From: Mark Fullmer [mailto:[email-masked]]
> Sent: Saturday, February 23, 2002 8:55 PM
> To: Peter Ludemann
> Cc: [email-masked]
> Subject: Re: [ipfix-req] Collector Failover as a requirement
>
>

In a normal deployment, with the collector close to the sender, "reliable"
isn't more resource-intensive. And TCP is a reasonable choice; the
latest NFS is done over TCP because it's as efficient as using
UDP and avoids re-inventing wheels for things like congestion control.
When a connection is broken, TCP uses longer and longer intervals when
retransmitting, so the execution overhead remains small.

Having said that, it is impossible to make a 100% reliable system.
If connections go down, eventually there will be no more room inside
the sender to store data, so some data must be discarded. The protocol
must be able to detect this and inform the receiver (when things are
reconnected).

One more thing about unreliable protocols ... they're attractive in theory
but in practice we find ourselves building workarounds to make them
seem reliable (this applies to both NetFlow and SNMP). Any data delivery
method that avoids reliability simply pushes things on to another
part of the system.

>     stop the router from generating flows to a non existant collector.

I don't think that only TCP should be required. I would also allow SCTP
... it's better than TCP for fail-over, but not widely available yet.

>     expects a keepalive every N seconds or 0 to disable it.  Default

This is essentially the way that CRANE works. However, I don't think
that a keepalive gives any advantage ... the sender should just time
out and failover if it doesn't receive a data ACK in time (there's no
need for keepalives when no data are being sent). [Downstream
deduplication will always be needed for handling failures because
ACKs can get lost.]

[I'm currently doing some experiments on the best way to do ACKs to
get the fastest failover; so I reserve the right to change my mind
about heartbeet keepalives.]

>     (port, IP address).  Must UDP May TCP/SCTP.

The initial message can also advertise the protocol version.

I would be opposed to UDP for data delivery. I don't see any
performance advantages and it causes a lot of downstream
processing pain.

>     can count lost flow records as in NetFlow v5,6,7,8.

Access to the packet count is not possible with most TCP implementations
anyway.

>     the collector it can not decode.

I believe that content negotiation is important for reasons of performance
and synchronizing amongst receivers in a fail-over setup. Other in-band
configuration is less important (it can be done via MIBS but is also
easier to define in a generic way (essentially a set of keyword/value
pairs that must be all handled as a single transaction ... like SNMP
SET but with database-style transaction control).

I would like to see in v1:
    - a standard data protocol (with generic configuration) using TCP
      for transport
    - a core set of values and types (the "data model")
and in v2:
    - other transport protocols (such as SCTP)
    - additional standard values and types
    - additional standard kinds of configuration

The v1 protocol must allow this future extensibility.

- peter

----------
Peter Ludemann            +1.408.330.5732
Chief Architect           +1.650.248.3973 (mobile)
XACCT Technologies        [email-masked]
2900 Lakeside Drive       http://www.xacct.com
Santa Clara CA 95054

---

From: Ganesh Sadasivan <[email-masked]>
Date: Mon, 25 Feb 2002 10:30:03 -0800

Hi Peter,

> retransmitting, so the execution overhead remains small.

What is a "normal deployment"? TCP is not as efficient as UDP in terms
of processing requirements (this should be fairly obvious). Well if the
network is reliable enough, I think UDP would work out better.

> method that avoids reliability simply pushes things on to another

I am curious to know the workarounds done on top of Netflow to make it
relaible?

> performance advantages and it causes a lot of downstream

Can you explain a little more here?

> >     can count lost flow records as in NetFlow v5,6,7,8.

There are other means to achive the same in the new protocol. Basically the
flowsets tell the length and using the size of each record in the flowset, one
can calculate the # of records.

> The v1 protocol must allow this future extensibility.

inband config and content negotiation I already replied to your
previous mail. I am not convinced it is worth adding it in this
framework.
Thanks
Ganesh

---

From: [email-masked]
Date: Mon, 25 Feb 2002 10:48:15 -0800 (PST)

Hello all:

The IPFIX Architecture and Data Model version 00 drafts were submitted
on time, early afternoon (PST).  We're waiting for internet-drafts@ietf
to get them published.

Now that we're past the draft submission deadline, we'll certainly keep
working on the drafts.  The latest versions of them will be available
on the IPFIX web site.

Over the next few weeks I believe we should continue to work on
 - expanding/refining the descriptions of the flow attributes
     in the Data Model draft
 - improving the list of protocol selection criteria in the
     Architecture draft
     
On 23 Feb, Peter Ludemann sent a few emails, pointing out the need to
make our arguments very clear on topics like 'how do we want to see
IPFIX exporters and collectors configured?'  I agree that 'not in the
charter' isn't a very strong argument, but the WG is trying hard *not*
to invent a complete new protocol.  So incremental improvements to an
existing protocol can certainly be talked about ...

Cheers, Nevil

-----------------------------------------------------------------------
   Nevil Brownlee                   Director, Technology Development
   Phone: +64 9 373 7599 x8941      ITSS, The University of Auckland
   FAX: +64 9 373 7021      Private Bag 92019, Auckland, New Zealand

---

From: "Kevin Zhang" <[email-masked]>
Date: Mon, 25 Feb 2002 22:28:43 -0500

Hi Mark,

Thanks for some very good ideas you offered, please see my comments inserted.

Kevin

> -----Original Message-----
> From: majordomo listserver [mailto:[email-masked]]On Behalf
> Of Mark Fullmer
> Sent: Saturday, February 23, 2002 11:55 PM
> To: Peter Ludemann
This is a very good idea.  keepalive can be piggybacked through ACKs from collectors.  This will not only serve as indication of collector fault conditions as well as congestion, it can also improve transport reliability.  This should be done on the IPFIX protocol layer whether TCP or UDP is used as the transport protocol.

>     (port, IP address).  Must UDP May TCP/SCTP.
The current IPFIX architecture draft from the design team proposed to use UDP to convey some low level information such as IPFIX version, transport protocol support, and port number.  I think this could achieve this requirment.

>     can count lost flow records as in NetFlow v5,6,7,8.
Simply counting the lost flow records may not be enough as values associated with each record are different (e.g. start of a flow, end of a flow, interim records).  We should really base our design over a reliable transport layer that offers congestion control.  UDP can optionally be used for limited configurations.

>     the collector it can not decode.
I believe required control messages must be included in this version as options. For those exporting devices that can not support them, a simple reply from exporter to collector will signal the limitation of the exporter; then both devices will fall back to a setting that is less demanding on the exporter.

> ��ޖ���[h�f���h���z�ݢ+��+��njwlk/�zZ��y��Ơy��I칻�&ޙ�����j:+v���w���"��"���v�vƲ��칻�&ފ�������,j��܀��b��m����*_�ݢ+��+��n��܆+�

---

From: "Kevin Zhang" <[email-masked]>
Date: Mon, 25 Feb 2002 22:28:48 -0500

Please see my comments inserted.

Thanks,

Kevin

> Of Ganesh Sadasivan
> Sent: Monday, February 25, 2002 1:30 PM
> To: Peter Ludemann
> Cc: Mark Fullmer; [email-masked]
I really doubt we can assume the network is reliable.  The potential deployment of IPFIX systems will involve WAN, and including exporting devices such as probes, mid-boxes etc.  Let's not limit ourselves too much at this time.  Another issue with UDP is that it is not congestion aware, it can interfere with network traffics from other sources.  This is clearly required in our charter.

> I am curious to know the workarounds done on top of Netflow to make it
With TCP and SCTP widely availabe, what's the point to replicate many of their features over UDP. 

> Can you explain a little more here?
Without adding features over UDP, IPFIX over UDP would be unreliable, congestion UN-aware, incapable of supporting redundancy configurations, and more.

---

From: Benoit Claise <[email-masked]>
Date: Wed, 27 Feb 2002 10:14:18 +0100

Nevil,

I'm confused by your email below.

> existing protocol can certainly be talked about ...

And a previous email you sent, which expresses:
+ Configuration: protocol
   ** Out of scope, i.e. WG should not consider this until we have
      the Arch and Data drafts well under way.
   - Randy, 20 Feb: reminder, out of scope

On one side, you wrote: configuration is out of scope
On the other side, you wrote: let's talk about improvements...

Can you please shed some light, I'm confused.

Regards,

---

From: Benoit Claise <[email-masked]>
Date: Wed, 27 Feb 2002 10:50:57 +0100

Mark,

>     stop the router from generating flows to a non existant collector.

This sounds actually like a good trade-off.
Note that this idea is now part of
http://www.ietf.org/internet-drafts/draft-ietf-ipfix-architecture-00.txt

5.3. Collector Crash Detection and Recovery
5.3.1. Simple Export Model
         data stream and control stream: unreliable transport
5.3.2. Export Model with Reliable Control Connection
         data stream: unreliable or reliable transport
         control stream: reliable transport

And also
5.4.Collector Crash Detection and Recovery
5.4.1. Simple Export Model
5.4.2. Export Model with Reliable Control Connection

Regards, Benoit

---

From: [email-masked]
Date: Wed, 27 Feb 2002 09:03:10 -0800 (PST)

> Can you please shed some light, I'm confused.

1) The Arch and Data version 00 drafts are now published (another WG
   Goal met).

2) The charter says we'll "select a protocol," which means we need to
   work on - i.e. discuss further on the list - the selection criteria
   for that protocol.

3) It seems to me that discussing aspects like exporter and collector
   configuration methods is helpful; it would be good to see some
   consensus on this emerge.

4) In selecting a protocol, we'll pick the one which comes closest to
    meeting all the selection criteria.  *If* the chosen one could be
    *slightly* modified or extended to meet all the criteria, we may 
    have to consider including those modifications in the final IPFIX
    Arcitecture document.

The main point of my 'current WG activities' posting was really
"the -00 drafts are out," now we can/should try to reach consensus 
on the issues which got a lot of airing on the mailing list earlier.

Does that help?

Cheers, Nevil

PS: On the question of IPR.  Randy has reiterated the IETF (RFC 2026)
    position on this.  I had assumed that all those with 'candidate 
    protocols' (and published drafts documenting them) had read and
    understood RFC 2026.

---

From: [email-masked]
Date: Wed, 27 Feb 2002 14:00:37 -0500

> Simply counting the lost flow records may not be enough as values associated with each record are different (e.g. start of a flow, end of a flow, interim records).  We should really base our design…

	One note here. Once the exporter runs out of room
	information will be lost. We need to handle/define the
	"lost data" situation in **ANY** solution we define.

> > éi?¨¥¶?s?SÝ¢j'z×hSÜ"±Ç?¹©Ý±¬¡zZb?g¬±¨n?rR{.nÇ+?·¦j)m¢f£¢·hs?ÞµÚ"·¬qçnjwlk+§²æìr¸>z*_<§?ë,j>¡Ü?Èb½èm¶YÿS-âÅÚ"·¬qçnýªÜ?+Þ

---

From: [email-masked]
Date: Wed, 27 Feb 2002 15:18:16 -0500

> Without adding features over UDP, IPFIX over UDP would be unreliable, congestion UN-aware, incapable of supporting redundancy configurations, and more.

	Simply using TCP does not make it reliable. You still
	need application level reliability. Once you do that, UDP
	becomes more reliable becasue re-transmit happens at a
	the application level. However it is still congestion 
	unaware. 

	You can also still do redundancy with a keepalive as has
	already been discussed.

	I don't think UDP can be ruled out so easily.