Subject: [ipfix] Mailing list digest, Friday 8 March

From: [email-masked]
Date: Fri, 8 Mar 2002 19:55:50 -0800 (PST)

IPFIX Co-chairs list of IPFIX issues

Nevil Brownlee & Dave Plonka, Fri 8 Mar 02

++ marks each issue
** WG co-chairs comment
 - Mailing list discussion items

   Contributors names are listed in full following the list of issues

++ Congestion-aware protocols
   ** Need to reach consensus on two major points:
       a. We know we need reliable transport.  Do we
          also need unreliable?
       b. Given consesnus on (a), how do we provide
          congestion awareness?
   - Nevil, 4 Mar: Note setting out IESG / WH chair position on
     this issue.  In brief, if IPFIX doesn't mandate congestion-
     aware transport, the WG RFCs won't get approved.

++ Collector Failover
   ** We need a control session handling keepalives.
      Much of this discussion hinges on the TCP vs UDP debate!
   - Paul, 27 Feb: Need to handle 'lost data' no matter what!
   - Mark, 25 Feb: UDP with retransmits as an alternative to TCP
     or SCTP.  A TCP for keepalives could be enough?
   - Paul/Ganesh, 25 Feb: Not convinced that UDP can be ruled out;
     TCP on its own doesn't provide 'application level reliability.'
   - Peter, 25 Feb: 'Reliable' transport is better and simpler. "Any 
     data delivery method that avoids reliability simply pushes things on
     to another part of the system."  Use SCTP or TCP; data ACKs
     are enough, don;t need keepalives as well.
   - Mark, 23 Feb: Suggestions: use mandatory TCP control connection,
     require collector to send periodic ACKS to exporter.
   - Peter, 23 Feb: TCP keepalive not enough for exporter to detect
     crash of collector.  Need to either use SCTP or have a failure
     detection mechanism.

-------------------------------------------------------------

++ Counter wrapping, long-running flows
   ** Support for both absolute counts (counters) and deltas, no clear
      consensus.
   - Benoit, 22 Feb: Close flow and export it when it nears counter wrap,
     restart as a new flow.
   - Ganesh, 21 Feb: Export flow records when counters go past a threshold,
     Keep IEs for absolute counts or delta counts, whichever is needed.
   - Peter, 21 Feb: In our experience, it's better to output deltas. Absolute 
     values are needed only if you have an unreliable data exporter.
   - KC, 21 Feb: Ability to send a flow record at any interval due to a 
     counter rollover needs to be a selection criteria.
   - Reinaldo, 19 Feb: Could someone elaborate on this?

++ Configuration: protocol
   ** Out of scope, i.e. WG should not consider this until we have
      the Arch and Data drafts well under way.
   - KC, 20 Feb: not doing that, trying to select best of existing protocols
   - Randy, 20 Feb: reminder, out of scope 
   - Reinaldo, 20 Feb, IPFIX config protocol needed, MIB OK as well
   - Fred, 20 Feb: NetFlow v9: good summary posting 
       'bi-directional config' = collector can ask exporter for reqd flows
       Fred doesn't think it would be useful, lots of IOS development,
       implications for reliability, security, etc.
       Config via an 'IPFIX Configuration MIB' would be better.
   - Juergen/Reinaldo/Paul/Benoit/Ram, 14 Feb: 'Not trying to standardise
     meter, therefore don't specify how to configure one.'
   - Mike, 13 Feb: Configuration and MIBs: "A MIB module is a concise guide
     to implementors in their design of a configuration interface (CLI, XML,
     XYZ, whatever) to manage a given techology"
   - Mark, 12 Feb: Comments on exporter-collector session management,
     session IDs.
   - Will, 11 Feb: Doesn't want 'SHOULD support in-band configuration.'

++ Configuration: Content Negotiation
   - Christian/Peter/Juergen, ~19 Feb: Collector should be able to request
     a subset of all available flow attributes, exporter should indicate
     which  attributes it can send.
   - Benoit, 19 Feb: This is pure configuration.  You tell the exporter
     which fields to export by defining a template.

++ Configuration: In-band Configuration
   - Christian/Peter/Juergen: This is the ability for a collector to
     ask an exporter for a particular set of flows

++ Applicability document
   ** Material currently in Arch draft.  Discuss at Minneapolis meeting.
      Charter mentions it to support Architecture draft, so its
      OK to start work on this, but don't let it distract from other
      WG drafts!  Use ipfix-app list alias for discussion.
   - Carter, 20 Feb: OK, but don't go too far off topic
   - KC, 20 Feb: Renaldo, Kevin & Tanja contributed most of this material 
     in Arch, they'd be good editors
   - Tanja, Feb 19: 'Applicability' contents proposal
   - Proposed by KC, support from Benoit

++ Middleboxes document
   *** Not in scope, material currently in Arch draft.
       Do we need a separate document?  Or put it in Applicability draft?

++ Selecting a Flow Info Export protocol
   ** Our charter says we'll select a protocol, however we'll probably
      need to look (small) improvements to an existing protocol
   - 19 Feb: KC had intended to list 'overview' descriptions of existing
     protocols in Arch draft.  Paul thinks this is premature, we should list
     'selection criteria' there instead.  KC agrees, meantime (so as not
     to loose the contributed text from B/G and Kevin) we'll shift it into
     appendices.
   - Randy, 18 Feb: realistically, i think you will have to modify any of
     the existing protocols to meet robust needs.  and yes, i think
     security should be a consideration.  [ note that snmpv3 control and
     tcp export over ipsec may meet some of the more difficult issues you
     raise ]
   - David, 18 Feb: Selection criteria are needed for this!

++ Defining a flow
   - Paul, 19 Feb: Selection process chooses packets to consider,
     flow definition rules determine which unique flows to export,
     i.e. flow definition rules may define sub-flows.  Collector may
     wish to know the selection cfriteria.
   - Reinaldo, 19 Feb: Is a subset of a flow (i.e. one formed by adding
     another filter function) not 'just another' flow?

++ Overload conditions
   - KC/Robert, 13 Feb: Should exporter or collector be allowed to hanle
     overload by changing sampling rate?  Or is it better to loose
     packets instead?

++ Security Considerations
   - Cliff, 12 Feb: text for Arch draft

B/G = Benoit & Ganesh

Benoit Calise, Cisco
Carter Bullard, QOSient
Christian Gilby, Nortel
Cliff Wang, SmartPipes
David Harrington, Enterasys
Fred True, AT&T Research
Ganesh Sadavisan, Cisco
Jean-christophe Martin, Sun
Juergen Quittek, NEC
KC Norseth, Enterasys
Kevin Zhang, Xacct
Man M Li, Nokia
Mark Fullmer, OARnet
Paul Calato, Riverstone
Peter Ludemann, Xacct
Ram Gopal, Nokia
Randy Bush (AD), AT&T
Reinaldo Penno, Nortel
Sebastian Zander, Fraunhofer Institute FOKUS
Simon Leinen, Switch
Tanja Zseby, Fraunhofer Institute FOKUS
Venkatraman G, Infosys Technologies
Will Eatherton, Cisco

++Other comments

Robert Lowe <[email-masked]>, 15 Feb:
  The lurker opinion poll (no vote)... ;-)
    In-band configuration -- not now.
    Negotiation -- limited (but that discussion is promised to follow).
  Benoit, I think this was a very perceptive comment.  It seems we are
  at times confusing the two.

Peter Phaal <[email-masked]>, 13 Feb:
  The IPFIX charter doesn't address the algorithms used by the flow
  collector, but there are system-wide issues that should be explored.
  Design choices made in the IPFIX protocol specification can have a
  fundamental effect on the capabilities of traffic management systems
  collecting IPFIX data (scalability, latency, accuracy, multi-point
  correlation, etc.)

-----------------------------------------------------------------------
   Nevil Brownlee                   Director, Technology Development
   Phone: +64 9 373 7599 x8941      ITSS, The University of Auckland
   FAX: +64 9 373 7021      Private Bag 92019, Auckland, New Zealand

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: "Kevin Zhang" <[email-masked]>
Date: Sun, 10 Mar 2002 23:34:27 -0500

Hi All,

Please see my comments inserted.

Thanks,

Kevin

> -----Original Message-----
> From: majordomo listserver [mailto:[email-masked]]On Behalf
> Of [email-masked]
> Sent: Friday, March 08, 2002 10:56 PM
> To: [email-masked]
> Subject: [ipfix] Mailing list digest, Friday 8 March
> 
> 

I support using a reliable transport protocol (e.g. TCP or SCTP), this takes care of both reliability and congestion awareness requirements outlined in our charter.  

>      crash of collector.  Need to either use SCTP or have a failure

In my mind, IPFIX protocol can be viewed as an application running over a transport protocol (e.g. TCP, SCTP, UDP).  It is a user of transport service provided by the lower layer. Among the transport protocols, the services can be summarized as -
UDP: unreliable, not congestion aware, optional checksum to provide packet integrity check.
TCP: reliable, in-sequence delivery, congestion aware, mandatory checksum to provide packet integrity, lacks support for quick failover/failback.
SCTP: reliable, in-sequence delivery, congestion aware, provide packet integrity check, support quick failover/failback (e.g. using heartbeat messages).

The options in front of us are - 
1) IPFIX over UDP: This requires the IPFIX protocol to replicate many capabilities that can be provided by TCP and SCTP. This may result in a rather complicated IPFIX protocol.   

2) IPFIX over TCP: IPFIX needs to implement IPFIX layer acknowledgment to achieve 'application level reliability', and 'heartbeat' procedures to support fast failover/failback.

3) IPFIX over SCTP: IPFIX needs to implement IPFIX layer acknowledgment to achieve 'application level reliability'.

It appears to me IPFIX over TCP is a good compromise. As we need to have IPFIX layer acknowledgment, 'heartbeat' messages can be piggybacked or sent periodically for detecting collector/link failures. This can also simplify the interaction between the IPFIX and transport layer; as failover detection is handled by IPFIX layer, we do not need to worry about passing signals back and forth between IPFIX and SCTP. 

I did not follow this thread very closely. But if we close flow and restart each time near a counter wrap, are we violating our flow definition already?  I am also not sure about the implications of this choice on applications; it appears there may result in may flows (or sessions) contains identical number of packets. 

>      which fields to export by defining a template.
I support content negotiation, exporter and collectors should at least negotiate a template that governs what are the exported attributes, and the export format. The collector should be allowed to indicate its requirements, and the exporter should try to meet them; though the exporter will ultimately determine what and how to export. 

>      ask an exporter for a particular set of flows
This seems to be the same as content negotiation, I support it.

> ��ޖ���[h�f���h���z�ݢ+��+��njwlk/�zZ��y��Ơy��I칻�&ޙ�����j:+v���w���"��"���v�vƲ��칻�&ފ�������,j��܀��b��m����*_�ݢ+��+��n��܆+�

---

From: Ganesh Sadasivan <[email-masked]>
Date: Mon, 11 Mar 2002 07:22:56 -0800

>      aware transport, the WG RFCs won't get approved.

So the conclusion is to use a congestion aware protocol or
use DCP with external congestion control - correct?

There is a good amount of discussions in
http://ipfix.doit.wisc.edu/archive/0753.html on both Content Negotiation
and In-band Configuration.

---

From: [email-masked]
Date: Mon, 11 Mar 2002 12:12:07 -0500

>      aware transport, the WG RFCs won't get approved.

	I believe there are a large number of real world deployment 
	cases where IPFIX is not going over the Internet. In fact,
	it is likely to be the majority of deployment cases. Is there
	disagreement on this point?

	If not, we ought to decide the best way to solve the problem
	first, then decide what that means for us as an IETF group;
	not the other way around.

---

From: [email-masked]
Date: Mon, 11 Mar 2002 10:11:48 -0800 (PST)

Hi Paul:

> 	not the other way around.

As an IETF WG we're honour-bound to come up with an architecture/
protocol/whatever that's good for the Internet overall, that's why
we have to use congestion-aware transport.  
'Solving the problem,'
in a non-Internet setting, is not

---

From: "Kevin Zhang" <[email-masked]>
Date: Mon, 11 Mar 2002 13:34:55 -0500

Hi Paul,

I have to disagree with you, please see my comments.

Thanks,

Kevin

> Of [email-masked]
> Sent: Monday, March 11, 2002 12:12 PM
> To: [email-masked]
> Cc: [email-masked]
> Subject: Re: [ipfix] Mailing list digest, Friday 8 March
> 
> 
I believe CRANE protocol (http://search.ietf.org/internet-drafts/draft-kzhang-crane-protocol-02.txt) is considered as a candidate for IPFIX.  We have several deployments with tier one carriers, which the exporter and collectors communicate through WAN. Furthermore, potential IPFIX exporting devices may be routers, probes, and mid-boxes as outlined in our charter; I do not agree that the need of high-speed router (may have limited on-device flow processing capability) should over-ride all the other needs.

---

From: [email-masked]
Date: Mon, 11 Mar 2002 13:51:01 -0500

> in a non-Internet setting, is not

	You are missing my point. If the problem space is 95%
	in the non-internet setting and we solve it for the
	internet setting, we solve the wrong problem. What's
	the point of that? 

	We should not ignore our most prevalent deployment
	scenario because it doesn't fit nicely within IETF 
	borders. We should say this is how to "best" solve
	our problem space and then decide what to do next.

	It would certainly be good if we can come up with
	something that works in both settings. I don't
	see why we can't accomplish that.

---

From: [email-masked]
Date: Mon, 11 Mar 2002 13:58:17 -0500

> I believe CRANE protocol (http://search.ietf.org/internet-drafts/draft-kzhang-crane-protocol-02.txt) is considered as a candidate for IPFIX.  We have several deployments with tier one carriers, whi…

	I agree 100%. And the opposite is also true,
	why would we ignore intra-net deployments and only consider
	Inter-net deployments? Especially when Intra-net is more
	prevalent (I didn't say exclusive, just more prevalent).

	BTW - I don't think WAN == Internet.

---

From: "Peter Ludemann" <[email-masked]>
Date: Mon, 11 Mar 2002 16:12:36 -0800

Based on my experience, congestion-awareness is useful even in a LAN. What
happens if the receiver can't keep up with the sender for a short period?
You don't want to be flooding the receiver with packets that it can't
handle.

TCP's capabilities turned out to be quite adequate for the congestion
handling, combined with a record-level ACK for time-out and fail-over to
another receiver.

As I mentioned earlier, in such situations TCP's overheads are barely more
than UDP's; but TCP avoids having to reinvent a congestion-awareness wheel.
And if you don't use a congestion-aware transport, you'll just have to put
it in yourself (based on my experience with testing in the field).

It is an educating experience to deploy a system inside a Tier 1 backbone
network ... packets often take longer/stranger paths than one would expect;
routing tables getting screwed up and connectivity lost; OC3 pipes connected
by a T1; etc.

(Or are you discussing something other than the need for
congestion-awareness?)

- peter

-----
Peter Ludemann            +1.408.330.5732
Chief Architect           +1.650.248.3973 (mobile)
XACCT Technologies        [email-masked]
2900 Lakeside Drive       http://www.xacct.com
Santa Clara CA 95054

> Sent: Monday, March 11, 2002 10:51 AM

---

From: [email-masked]
Date: Mon, 11 Mar 2002 19:48:09 -0500

I have no problem with TCP. In fact, it is my preferred 
choice (LFAP also runs over TCP). I have an issue with basing 
the decision on design criteria that I believe are highly 
suspect. It will only lead to "wrong" choices down the road.

If we say that UDP is out because it is not Inter-net friendly
and most of the deployments are Intra-net, that logic seems
flawed, even if it arrives at the "correct" answer.

Just to push the other side, more comments inline...

> You don't want to be flooding the receiver with packets that it can't

	You lose some records. In a lot of applications, that
	is perfectly acceptable.  Even with TCP, if the Collector
	backs up long enough the exporter will have to drop some
	information. In either case, IPFIX will need to define and code
	for a data loss situation. 

> handling, combined with a record-level ACK for time-out and fail-over to

	Record level ACK's can be done with UDP. In a lot of cases
	there is no congestion issue.

> than UDP's; 

	Maybe someone else can comment on the TCP vs UDP overhead.
	This seems to be the crux of the issue. If TCP and UDP are
	roughly the same in terms of overhead, speed, etc... then
	TCP is a clear winner. If they are not roughly equal, then
	maybe both will be needed.

> but TCP avoids having to reinvent a  wheel.

	Not if you don't congestion-awareness.

---

From: "Peter Ludemann" <[email-masked]>
Date: Tue, 12 Mar 2002 00:39:00 -0800

It seems that there's no firm consensus on reliable/non-reliable
and whether we need congestion awareness. My belief is that
reliability and congestion-awareness cost little and gain a lot;
and that the "benefits" of UDP are marginal at best:
  - load-balancing by multi-cast -- but almost the same thing can
    be done by having multiple output streams with TCP;
  - silent loss of data with UDP when you don't care much
    about the data.
(did I miss any?)

Additional comments inline (I found a few references on efficiency
of TCP vs UDP).

> From: calato [mailto:calato]On Behalf Of [email-masked]
> Sent: Monday, March 11, 2002 4:48 PM
> To: Peter Ludemann
> Cc: [email-masked]; [email-masked]

But with TCP (or some other "reliable" transport), you know which
records you've lost and can compensate in some (e.g., aggregate
the lost values and send them later). With UDP, you don't know, unless
you have ACKs, in which case you're just reinventing a partial TCP
without retransmission and congestion control (and you'd also need to
handle duplicates and out-of-orders).

> 	maybe both will be needed.

I don't have numbers available; but I do notice that NFS Version 3
is over TCP (RFC 1813, 3010). Presumably efficiency isn't a problem.
As a thought experiment, on a local network without congestion,
TCP won't be retransmitting or reordering packets; there'll be
the ACKs coming back (possibly piggybacked on the record-level
ACKs for the data protocol), so at the network level, performance
should be about the same and only slight more CPU usage.

Java is much more efficient with TCP than UDP, by the way.

RFC 3010 references Macklem, Rick, "Lessons Learned Tuning the
4.3BSD Reno Implementation of the NFS Protocol", which I found
at http://sunsite.nstu.nsk.su/sun/inform/whitepapers.html.
Graph 6 shows that TCP is about 20% more expensive in CPU than
UDP (and this paper is over 10 years old, so implementations
have probably improved) -- it's not clear to me whether this
number takes into account the acknowledgment work done outside
the transport with UDP. My interpretation is that on a more
powerful machine than a microVAX, the congestion control of TCP
will more than offset CPU overhead and that TCP is a winner.

Additional references ...
http://www.ietf.org/proceedings/98dec/43rd-ietf-98dec-142.html

   Brent Callaghan discussed the NFS version 4 effort. He recalled
   that NFS was developed for UDP because TCP performance was
   pretty poor in the target environment, Ethernet LANs. NFS mount
   options were added for controlling timeouts and transfer sizes.
   The adaptive timeouts worked okay, but the transfer size adjustment
   didn't. As NFS started being used over WANs, it evolved towards
   also being run over TCP, certainly a win on congested WANs, and
   fitting better with NFSv3's potentially very big reads and writes.
   The NFSv4 Transport Wish List: As fast as UDP on LANs; optimized
   for fast RPC transactions (fast binding, reduced TIME-WAIT);
   integrated record marking.

and this:
http://www.frontiertech.com/supernfs/WHITE-P/client_opt.htm

   Since TCP is a stateful protocol, it transfers a large part
   of the responsibility for guaranteeing the completion of
   file requests from the client to the network, with a
   concomitant increase in reliability and performance. The
   efficient exploitation of this, however, requires careful
   client and server design, for which no standards have yet
   been devised. In addition, for similar reasons, NFS over
   TCP offers superior performance when several routers ("hops")
   lie between the client and server. TCP handles multiple "hopping",
   which is a characteristic of the Internet, much more efficiently.

> 	Not if you don't congestion-awareness.

See the comment above on TCP, as to why congestion-awareness
is a Good Thing on WANs. As to LANs, it's surprising how badly some
sysadmins can be at configuring a network; not to mention the
amusing effect of a hub or switch that develops a "glitch".

> > > From: majordomo listserver
> [mailto:[email-masked]]On Behalf

---

From: Fred True <[email-masked]>
Date: Tue, 12 Mar 2002 04:13:50 -0500 (EST)

> you have ACKs,

You know which records you've lost in UDP if the datagrams contain atomic
serial numbers, as they do in the current Cisco Netflow export format.
This is the way we currently detect and account for loss in our netflow
applications (you know how many flow records you lose, but of course you
do not know anything about what they contained; but you wouldn't with TCP
either).

-fred

---

From: "Peter Ludemann" <[email-masked]>
Date: Tue, 12 Mar 2002 09:34:10 -0800

> do not know anything about what they contained; but you wouldn't with TCP

But only the *sender* knows what was in the lost data (e.g., the total
number
of bytes that it saw); and UDP doesn't communicate that back to the sender.
So, yes, the receiver can know how many records were lost but it has no idea
of what data was lost. With TCP, you know which records you couldn't send,
so you could calculate aggregates of the data that wasn't sent (and send
these
aggregates later, present them in a MIB, etc.)

- peter

---

From: Fred True <[email-masked]>
Date: Tue, 12 Mar 2002 12:55:26 -0500 (EST)

> number of bytes that it saw); and UDP doesn't communicate that back to
> the sender. So, yes, the receiver can know how many records were lost
> but it has no idea of what data was lost. With TCP, you know which
> records you couldn't send, so you could calculate aggregates of the
> data that wasn't sent (and send these aggregates later, present them
> in a MIB, etc.)

True; however Cisco's netflow (roughly) does this too, although it is only
available via CLI.  It keeps the number of bytes, pkts, etc. exported via
netflow in a router-side table, which can be re-set or queried via cli. We
have used this to fill in for loss; however it is somewhat difficult to
manage in tight sync with the exported data stream.  Still, I think the
use of TCP is not the only solution to this problem (and for the sender's
tcp stack to have a hook into secondary aggregation sounds like a burden
that not many vendors are going to want to shoulder).

-fred

---

From: "Will Eatherton" <[email-masked]>
Date: Tue, 12 Mar 2002 11:47:51 -0800

Paul Calato,

I agree with your point below completely.  However, my understanding is that
we have been told that if we don't narrow to congestion-aware/internet
scenario then we should pack up IPFIX and dissolve the group.  I dont like
this stance, but this issue isn't worth losing the benefits of a
standardized IPFIX (even if 95% of it's use will be non-compliant
non-internet use :)

Will

---

From: [email-masked]
Date: Wed, 13 Mar 2002 09:14:18 -0500

> handle duplicates and out-of-orders).

	If you have no application level ACK and you
	loose a TCP connection you still do not know
	which of the last n records sent made it to the 
	other side.

	So then you start putting in application level
	ACKs for TCP, which could also be used for UDP.

>    which is a characteristic of the Internet, much more efficiently.

	Good references. I'll read them over as soon as I can.

---

From: [email-masked]
Date: Wed, 13 Mar 2002 09:17:19 -0500

> aggregates later, present them in a MIB, etc.)

	How do you know what records got lost? Some number of
	records are in the TCP pipe and if you loose the TCP
	connection you don't know which ones made it and which
	ones didn't.

---

From: "Peter Ludemann" <[email-masked]>
Date: Wed, 13 Mar 2002 09:25:20 -0800

I think I'll be the first person in this mailing group to say ...

I was wrong. (That the sender would know what records were lost if TCP is
used. Even SCTP can't give a 100% answer for that.)

But my other arguments for reliable transport still are valid, I think.

Has anybody found other references about CPU cost for using TCP vs UDP? All
I can say is that when I use CRANE for sending 3000 records per second (over
TCP), I can barely see the CPU meter move.

By the way, our company has spent a lot of time and money in making NetFlow
reception as reliable as possible. We did this because *customers* wanted it
and paid us money for it. That's the kind of "rough consensus" that I
believe in. (Running code we've already got.)

- peter

> Sent: Wednesday, March 13, 2002 6:17 AM
> To: Peter Ludemann
> Cc: Fred True; [email-masked]; [email-masked]
> Subject: Re: Congestion awareness, reliability (was: RE: [ipfix] Mailing
> list digest, Friday 8 March)
>
>
> Peter Ludemann wrote:
> >