Subject: RE: [ipfix-eval] Discussion of Candidate Protocols

From: "Peter Ludemann" <[email-masked]>
Date: Tue, 1 Oct 2002 23:34:04 -0700

My comments on Jeff Meyer's comments (hopefully this discussion won't get
indented too many levels by those who rebut me).

    - IPDR  - much richer and more formal extensibility model,
        leveraging industry standard XML-Schema.
        Compactness as good as other candidates (after
        negotiation of "templates"/"descriptors"), simple
        set of operations.

I would argue that a hierarchical format is a Bad Thing. History repeats
itself; and the XML people are finding out what old-timers already knew:
that there were good reasons why relational databases supplanted
hierarchical and network databases on the 1970s ... if anybody wishes me to
bore them with the details, just ask ... or read other people's opinions
such as http://www.dbazine.com/pascal4.html

    - Diameter - less compact due to AVP encoding model.  Extensibility
        model formalized, but not provided as a machine readable
        document.  Overhead of acking makes this seem too heavyweight
        for the requirements as specified.  Finally the requirement
        for a recorder to support the SCTP protocol seems prohibitive,
        since this may require kernel support lacking on several OS's.

    - LFAP - has a limited (numeric id based) extensibility model.
        Protocol is relatively simple.  The encoding is tag/len/value
        meaning it will be less compact than IPDR or NFv9.

Compactness is a Good Thing when sending thousands of records per
second. This is not just an issue of bandwidth; it is also an issue
of processing time -- if you know exactly how a record is laid out,
you can process it faster. In some cases, you can avoid copy operations,
which can be a significant cost at high data rates. I don't see how
tag/len/value encoding can have as high performance as externally
defined formats.

BTW, doesn't the Diameter spec say that it can use UDP, TCP, or SCTP?
(I would love to require SCTP for CRANE; but as it is not wide-spread
yet, we must allow the possibility of TCP for now.)

    - NFv9  - compact, but extensibility model (templates) don't
        provide the set of information which IPDR does.  Relatively
        simple.

    - CRANE - proprietary vendor format.  More complex than is
        needed for IPFIX use case (20 different operation codes).
        Template based (like NFv9), has similar lack of formalism
        in actual definition of extensions.

"Proprietary vendor format" is irrelevant. Netscape, LFAP, NetFlow will
all be released under IETF rules if they are adopted as standards.

The alleged complexity in CRANE is to provide the necessary level of
reliability and flexibility in the templates. One specific design goal
was to allow the exporter to be informed of which fields were needed,
so that it could adjust its processing accordingly (for our PacketSight
probe, we can see roughly 3:1 performance change, depending on the
amount of metrics and attributes that are collected). If these are left
out, the number of operation codes drops by 1/3. Remove the operations
for multiplexing sessions on a connection (which would be unnecessary
if we could mandate SCTP) and we're down to 11 operations. 4 of these
are for flow control; 3 are for status reporting.

Anyway, the number of operations is irrelevant (most of the template
operations have similar content) ... the tricky thing in implementing
CRANE is getting the reliability and fail-over definitions just right.

As to "lack of formalism", which appears to be a sin shared by
all by IPDR ... all I can say is that I have taken enough
graduate-level courses in formal semantics to have become
deeply sceptical of any claim of a "formal model". (Anybody remember
the 2nd Algol-68 Report?) In the end, everything boils down to
agreements about the meanings of words. And there's nothing
stopping future standards that define precisely the meanings
of "upstreamByteCount", "applicationResponseTime", etc. which can
then be used by everybody. (Or some kind of registration process,
such as IANA's.)

- peter

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Michael MacFaden <[email-masked]>
Date: Wed, 2 Oct 2002 00:14:09 -0700

>tag/len/value encoding can have as high performance as externally

Having spent a few years debugging snmp agents/mgmt apps, 
I believe compact formats may imply a tradeoff with 
debugging and diagnosis capability. 

Unless your modified tcpdump is capable of caching the template and
its offsets, one can't simply figure out from a given pdu 
if something is askew.
Detecting encoding errors (off by one, etc) in such protocols
I think will be rather difficult. 

If one reads the LFAP protocol spec, specifically the List IE, 
one can see that even AVPs values can be compacted by 
being able to use repeat counts. BGP over TCP seems to be doing just 
fine scaling with its avp encoding scheme to send thousands of 
route entries.
 
I would really like to see us figure out the overall "bandwidth usage" 
and "cpu/memory usage" in a real network protocol than judging just 
one aspect such as compact encoding.
Factors such as protocol statefullness and cpu time to process
any given PDU come into play and may overshadow any one aspect of
the the initial (useful) criteria posted.

Since all the prospective protocols have existed for some time in 
their current form in real networks, it shouldn't be hard to gather 
some real data. Dave Plonka probably already has some sort of test
bed/sample input :-)

Regards,
Mike MacFaden

---

From: "Peter Ludemann" <[email-masked]>
Date: Wed, 2 Oct 2002 01:38:09 -0700

Michael MacFaden wrote Wednesday, October 02, 2002 12:14 AM

> debugging and diagnosis capability.

Agree ... (optional) MD5 checksums would have saved me a few
grey hairs. One unnamed embedded OS's faulty TCP stacks didn't
would have been detected much faster with this ...

> I think will be rather difficult.

I have a Perl script that you might enjoy. But it only works
with CRANE. :-)
[Hint: do this in two steps: first step extracts records from
the stream and deals with issues such as retransmits; second
step interprets the records]

> fine scaling with its avp encoding scheme to send thousands of

Agreed.

However, we have encountered some situations where saving a single
copy operation per output recordcan make all the difference between
being able to deliver the data and not. It's highly unlikely that
an application will keep data in AVP form, so AVP requires copies.
If the protocol packs the data into something that a C programmer
might do with a "struct", then memory-to-memory copying can be
avoided.

> the the initial (useful) criteria posted.

My experience is that once you go past a few thousand records per
second, life becomes miserable in determining processing bottlenecks.
For example, timeliness of ACKs can be important (especially when
ACKs are constrained by persistency issues). When you get into the
10Ks/sec range, a certain amount of purple magic smoke helps.

> some real data. Dave Plonka probably already has some sort of test

Well, I don't think he has CRANE, unless he has implemented it from
scratch. :-) As things stand, even I can't give you a maximum throughput
number right now for CRANEbecause my testbed is transitioning to a
"new improved" collector. Also, your mileage will vary, depending a
lot on length of records, type of data (string? numbers? and delicate
tuning issues on the TCP stacks. (Please, nobody mention VPNs and
"long fat pipes" and "TCP selective acknowledgment" ...
http://www.erg.abdn.ac.uk/public_html/research/satcom/tcp-evol.html)

- peter

---

From: Sebastian Zander <[email-masked]>
Date: Wed, 02 Oct 2002 11:14:29 +0200

First, for me as an advocate I find it somewhat inappropriate to comment on
other advocate drafts because I am biased of course ;). It would be a good
thing if there are other people not directly releated to the advocates who
have some opinions on the protocols. But this is only my personal feeling.

> indented too many levels by those who rebut me).

Lets find out by adding another level :)

>         since this may require kernel support lacking on several OS's.

The Diameter message definition is of course machine readable. Without
the acking there is no application layer reliability which is IMHO a bad
idea for accounting. Since one important application for IPFIX is accounting
I am wondering how how this can be done reliably without. Maybe there
could be a mode without acking for those who don't need it.

You are right that the SCTP requirement is somewhat prohibitive but I
would love to see SCTP become wide-spead. If SCTP should not make it than
probably Diameter will be run mostly over TCP.

> tag/len/value encoding can have as high performance as externally

Compactness is important but not everything. Human readability is often a nice
feature. HTTP, RTSP, SIP are following this philosophy. And an HTTP server
needs to process as much transactions/s as possible. On the other hand maybe
the IPFIX protocol will be so simple that this is not an issue. However
parsing the records is not the complete protocol. There is other stuff
do to such as security etc.

BTW templates could be used with Diameter as well. Templates and data can be
encoded as compound AVPs avoiding tag/len overhead. Most IPFIX attributes
are not yet defined within Diameter anyway. But again I think the record
format is only a part of the protocol. There is other stuff like the message
exchange, security, vendor extensibility mechanism, capability exchange etc.

> BTW, doesn't the Diameter spec say that it can use UDP, TCP, or SCTP?

It can use TCP, SCTP but not UDP.

> yet, we must allow the possibility of TCP for now.)

Probably this is true for Diameter as well

> then be used by everybody. (Or some kind of registration process,

Cheers,

Sebastian

-- 
Sebastian Zander                         E-mail: [email-masked]
FhI FOKUS / Global Networking (GloNe)    Tel: +49-30-3463-7287
Kaiserin-Augusta-Allee 31                Fax: +49-30-3463-8287
D-10589 Berlin, Germany                  www.fokus.fhg.de/usr/sebastian.zander

---

From: [email-masked]
Date: Wed, 02 Oct 2002 08:53:37 -0400

> might do with a "struct", then memory-to-memory copying can be

	Having done a fair amount of tuning on a collector I find
	it interesting that your performance hinged on a memory
	copy. In my experince performance was limited by how fast the 
	disk writes are.

---

From: [email-masked]
Date: Wed, 02 Oct 2002 08:59:46 -0400

> have some opinions on the protocols. But this is only my personal feeling.

	For the most part I have refrained for the same reason.

	Unfortunately the advocates are likely the ones in the best
	position to comment. So we've taken 5 strong voices out of the
	discussion. At some point we'll we'll need to open the
	flood gates. Maybe after the evals come out.
	
	Paul

---

From: Jeff Meyer <[email-masked]>
Date: Thu, 03 Oct 2002 09:08:12 -0700

Hi,

   In this message I just wanted to respond the the
discussion around "hierarchical formats".  Since
the comment indicates the author has not actually
read IPDR...

   The current version of IPDR does in fact require
that data be delivered in First Normal Form (just
like Relational DB's).

   IPDR specifies a SUBSET of the rich (overly so?)
vocabulary of XML Schema.

   So, if you look at the schema proposed for IPFIX
(in Appendix A of the Advocacy draft), you will see
that all of the values are laid out side by side,
no nested structures, no repeating of fields.

   <complexType name="SimpleV4Flow">
     <extension base="ipdr:IPDR">
       <sequence>
         <element ref="sourceAddress" minOccurs="1"/>
         <element ref="destinationAddress" minOccurs="1"/>
         <element ref="protocolIdentifier" minOccurs="1"/>
         <element ref="sourcePort" minOccurs="1"/>
         <element ref="destinationPort" minOccurs="1"/>
         <element ref="ingressPort" minOccurs="1"/>
         <element ref="egressPort" minOccurs="1"/>
         <element ref="packetCount" minOccurs="1"/>
         <element ref="byteCount" minOccurs="1"/>
         <element ref="classOfService" minOccurs="0"/>
         <element ref="flowLabel" minOccurs="0"/>
         <element ref="flowCreationTime" minOccurs="1"/>
         <element ref="flowEndTime" minOccurs="1"/>
         <element ref="tcpControlBits" minOccurs="0"/>
         <element ref="samplingInterval" minOccurs="0"/>
         <element ref="samplingAlgorithm" minOccurs="0"/>
         <element ref="sourceExporterAddress" minOccurs="1"/>
       </sequence>
     </extension>
   </complexType>

   Also to clarify.  The proposed protocol uses Compact
IPDR format.  The wire protocol IS NOT XML, rather XML
is simply used to model the data which is sent.  So
in the example above, the record representing
a flow would have a descriptor identifying the type
of IPDRRecord being encoded followed by the
integer values one next to the other (i.e. 4 bytes/
field).

   Now this modelling has the benefit that it ALSO
defines an equivalent XML encoding for the same
compact content.  And in addition, since it is FNF,
the definition can also be used to define how this
information is to be stored in a DB.

   So you have one description language, based on
XML Schema, a wire protocol, a binary format,
an XML format and a DB mapper, in a single document.

   Show me how that is done w/ the  other proposed
drafts?

Regards,

   Jeff Meyer

P.S.  I am in the process of defining extensions to
IPDR to enable more structured data.  (i.e. repeating
fields and fields which are not of a primitive type).
However this capability does not appear to be required
for IPFIX.

The bottom line however is that flat data

---

From: Jeff Meyer <[email-masked]>
Date: Thu, 03 Oct 2002 10:52:59 -0700

Hi,

   I didn't complete my closing remark before
hitting send...

    The bottom line however is that flat data
representations (First Normal Form) do have
benefits in the ability to map directly to
a relational DB table, and as such, can simplify the
processing model for operating on large sets
of events.  When possible, mapping recorded usage
information to a flat structure is going to
increase the likelihood that off the shelf tools
can operate on it (think Excel spreadsheet).

-- Jeff

---

From: Michael MacFaden <[email-masked]>
Date: Thu, 3 Oct 2002 12:50:03 -0700

>  Show me how that is done w/ the  other proposed

My main issue with such a system is do we *require*
such complexity if a simple numbering of fields
will do just fine? 

I'd really like to see the IPIFIX protocol do one thing 
and do it very well.

Delivering IP header information, as has been done
for a very long time now with NetFlow and others,
only really needs a clear definition 
of is being collected, how it is collected, 
and how to signal when the collector can't 
collect a paritcular piece of data ala RDBMS 
null indicator.  

The LFAP data spec clearly defines what is being
collected and like NetFlow are purpose built
targeted protocols. They not generic data movers.

This makes them simple as can be, consistent and 
extremely interoperable.

Regards,
Mike MacFaden

---

From: Sebastian Zander <[email-masked]>
Date: Fri, 04 Oct 2002 11:06:24 +0200

Jeff Meyer wrote:

[...]
>   Show me how that is done w/ the  other proposed

Jeff,

its very nice to have these things but I don't think its
a good idea to have them in one document. IMHO the main
focus here is on the protocol. Working on all the things
at once will only slow down the standardization process.
I am also wondering if the IETF would ever standardize
things like binary formats. Probably not.

Furthermore I am not convinced that it is a good idea to
standardize things like DB mapper or binary formats. E.g.
there exist a number of different formats and some people
may want to continue using them. But informational RFCs
could cover all these things.

To summarize my point: the evaluation should focus on the
protocol and its capabilities and not on things storage
formats etc.

BTW Diameter has a proposed XML definition as well
(draft-frascone-aaa-xml-dictionary-00.html). Another proposal
was to use SMIng (draft-schoenw-sming-dia-00.txt).

Cheers,

Sebastian

---

From: "Peter Ludemann" <[email-masked]>
Date: Fri, 4 Oct 2002 09:49:45 -0700

> 	flood gates. Maybe after the evals come out.

I too have restrained, but no more! Diogenes-like, I shall
carry my lantern, searching for Truth. Alas, my Silicon Valley
bathtub carries a million dollar mortgage, so I must work 36 hours
a day at XACCT to keep the bankers away, but that won't bias
my opinions.

I think that my biases should be obvious (I come from a world
where protocols are a means to an end and the end is usually
a big database) but if anybody wants a more extensive list of
my background and biases, I'll be happy to provide them.

- peter ludemann
  principal engineer
  xacct technologies

---

From: Jeff Meyer <[email-masked]>
Date: Fri, 04 Oct 2002 09:57:51 -0700

Hi,

   There seems to be some confusion over the separation
between a formal, machine readable description of
the data model (as IPDR defines), and the requirements put
on an implementation (especially in an IPFIX Middlebox).

   Presumably folks are familiar with SNMP and the
separation between what a MIB defines and how the
protocol operates.

   Basically IPDR's use of XML schema is producing a
MIB for sets of accounting information.

   Why not use MIB's?  Well, SNMP is really geared towards
providing views/access into a current system state.  Historical
information tends to be global or other coarse grain
counters.  Fine grain counters in SNMP are hindered by
SNMP's OID indexing model (i.e. you end up with something
like RMON, a clever use of SNMP, but not much fun to
code to).

   IP Flow information is just a specifically defined
set of metrics which are being collected based on
certain observed network traffic.   It just so happens
that this can produce a huge amount of info, which
often cannot be stored on the device itself for any
period of time.  For this reason, using SNMP for gathering
flow info is probably a bad idea, and I'm glad to
see no one signed up as the SNMPv3 advocate ;-)

   But the essential concept of MIBs, which SMIng is also
looking at revising is important.  The extensibility
model which MIBs provided for SNMP is what has led
to its large scale adoption for many problem domains.

   It just so happens that the profile of flow exporting
is not appropriate for the SNMP domain.

   However, the statement that IP flow is somehow unique
in its data requirements and as such a more generalized
"data mover" is somehow problematic, is just plain wrong.

   As a mediation product vendor I see lots of devices
with lots of vendor specific protocols.  And there
are several which have the same protocol requirements
as IPFIX.  Just different data models.

   Consider a middlebox which looks not only at flows,
but also at protocols above layer 4.  Would it not
make sense for the additional information produced
by this to use the same basic protocol as IPFIX?

   IPFIX does identify extensibility as a requirement,
is there some bounded set of extensions which are
envisioned?

   If not, then a trivial numeric id scheme is a pretty
lousy namespace to work from.  Diameter's vendorId/
avaCode model is not much better, but at least has
some partitioning.

   The only possible unique aspect I could picture is
based on observing NFv2,5,7.  In all of these versions,
all data items were fixed width (typically 32-bit
integers).  So...  If IPFIX wants to say that only
fixed with data values may be sent via the IPFIX
protocol, then I'd say, maybe you could optimize
beyond IPDR for that.

   A minimal implementation of IPDR on a middlebox
would NOT need to have any XML knowledge whatsoever.
As a producer, the system could merely hardcode the
information model produced (or make it configurable).

   The encoding itself follows XDR format, which has
worked alright for NFS and RPC, and is the precursor
to the encoding for DCE and CORBA encoding.  It is
a very simple model.  Read RFC 1832.  An implementation
for what IPDR requires is pretty trivial.  IPDR
members have access to opensource for this in
both Java and C.  And as Stas pointed out in an
earlier memo there are > 10 implementations out
there.

   The other benefits from IPDR which I cited, such
as XML mapping, DB, binary file format, etc. are
just that, additional benefits.  They ARE NOT
REQUIRED to be implemented.

Regards,

   Jeff Meyer

---

From: [email-masked]
Date: Fri, 04 Oct 2002 12:54:47 -0400

> my background and biases, I'll be happy to provide them.

	I think the mortgage information will suffice :-)

---

From: Robert Lowe <[email-masked]>
Date: Fri, 04 Oct 2002 12:56:55 -0500

Jeff Meyer wrote:

Attention: lurker factor in play...

> An implementation
> both Java and C.

The common definition of opensource includes descriptives such
as "publicly-available" and "no-strings-attached".  You seem 
like a pretty good evangelist for IPDR, but once someone starts 
passing the collection plate, using the word "opensource" only 
sullies the good name of the sacred cow of 21st century computing 
with the marketeer's comparison to a members-only reference 
implementation.  Some words don't have increasing degrees of 
comparison.  You can be dead, but you can't be 'deader', and in 
my book, open is open, not open for a fee.  Grammer lesson and 
rant off...

-Robert

---

From: Jeff Meyer <[email-masked]>
Date: Fri, 04 Oct 2002 12:55:14 -0700

Robert,

   Good point.  This is currently a reference implementation.

   Although I've advocated to move it to complete opensource,
e.g. SourceForge, this has not happened, YET.

Regards,

   Jeff Meyer

---

From: Jeff Meyer <[email-masked]>
Date: Fri, 04 Oct 2002 14:43:00 -0700

Sebastian,

   Diameter's XML draft is pretty close to IPDR, thanks
for the pointer.

   The trick used by IPDR, however, is to write directly
using XML-Schema's directives for defining information
models, and reuse XML-Schema's existing data type
set rather than inventing new names.  Diameter uses
the older DTD definition syntax to define a grammar
from scratch.

   The benefit of the IPDR model is that you not only have
a description language (for mapping to various serialization
protocols, e.g. Compact IPDR...), but a validateable XML
representation of that data immediately falls out, if
you are interested in having an XML representation.

   Note that the document itself only contains descriptive
information about the information items (elements and
groups of elements).  How one maps from this information
set to Compact IPDR or DB tables, is established by
convention, documented in a separate RFC/spec.  So, I don't
think your concern about putting too much in the document
is an issue.

 > things like binary formats. Probably not.

   Diameter protocol messages define a binary payload,
as do many other IETF protocols?

Regards,

   Jeff Meyer

---

From: "Peter Ludemann" <[email-masked]>
Date: Sat, 5 Oct 2002 11:55:38 -0700

> on an implementation (especially in an IPFIX Middlebox).

YES ... so, can we all agree that data model issues are not
part of this discussion?

[snip]
 
> "data mover" is somehow problematic, is just plain wrong.

YES!
 
> as IPFIX.  Just different data models.

YES!

The data models need to also be aligned; but just dealing
with all the protocols is a headache, especially with the
unreliable protocols such as NetFlow and SNMP (RADIUS is
an "almost reliable" protocol and therefore isn't quite
so difficult).

[snip]

> ...  If IPFIX wants to say that only

CRANE imposes only the header overhead on each record.
The data are also sent in the "native" form for the
exporter (I'll comment on XDR later).

However, I think that a numeric-only protocol is not
sufficient. As soon as metrics are collected at the
application level, people want attributes, such as
URL path, response status, payload type, etc. which
are either difficult or impossible to encode as number.
 
> information model produced (or make it configurable).

Likewise for CRANE, the box could simply reject all
requests to limit the fields that are exported, by
hardcoding the templates. However, processing the
templates is somewhat easier in CRANE because of their
fixed layout, not requiring an XML engine (yes, I know
that an XML engine can be quite compact; but it's not
common on network elements, at least not now).
 
> to the encoding for DCE and CORBA encoding.

We looked at XDR and rejected it because it can be a
little "heavy" for CPU-constrained devices (e.g., CSU/DSU,
"cable routers", etc.). We prefer exporting in the
"native" format (big- or little-endian) with template
flags to allow decoding at the receiver. Of course,
we don't prevent export in XDR (big-endian) form.

- peter

---

From: Simon Leinen <[email-masked]>
Date: 07 Oct 2002 12:47:04 +0200

> 	Having done a fair amount of tuning on a collector I find it
> 	interesting that your performance hinged on a memory copy. In
> 	my experince performance was limited by how fast the disk
> 	writes are.

Some collectors will write large amounts of data to disk - if this is
the main bottleneck I'd assume that they use the accounting data
without too heavy processing (or they need a better approach to disk
I/O, such as direct I/O, scatter/gather interfaces, faster disks etc.).

Unnecessary copying should be avoided mainly because memory bandwidth
doesn't seem to grow as fast as other performance dimensions.
Personally I think this is more of an issue than for example
byte-order optimizations.

Regards,
-- 
Simon.

---

From: [email-masked]
Date: Mon, 07 Oct 2002 10:34:38 -0400

> YES ... so, can we all agree that data model issues are not

	No. We have no interoperability without a data model.

> > "data mover" is somehow problematic, is just plain wrong.

	No. Generalized data movers are not the problem.
	Defining a protocol that is well defined enough so
	that many device vendors can export data and many
	application vendors can process the data regardless
	of the device is problematic if all you define is a 
	general purpose data mover.

> are either difficult or impossible to encode as number.

	I'll just quote Randy's message...

	>> Please let's try to stick with the core goal of looking at IP
(v4&v6)
	>> packets and the "transport" protocol or next header.
	 >
	 > i think that was the intent of this wg's charter.
	 >
	 > randy, with ad hat on

	Paul

---

From: [email-masked]
Date: Mon, 07 Oct 2002 10:47:10 -0400

> I/O, such as direct I/O, scatter/gather interfaces, faster disks etc.).

	I don't follow. Are you saying I/O isn't typically 
	the bottleneck at the collector?

> byte-order optimizations.

	I think there are lots of other issues that come into
	play before memory copies do. malloc and free are typically
	to big culprits of CPU usage. A design that causes lots
	of task switching is another. Your program needs to be
	pretty finely tuned before memory copy shows up, assuming
	the app is not just plain silly about memory copies.

---

From: "Peter Ludemann" <[email-masked]>
Date: Mon, 7 Oct 2002 15:24:57 -0700

(This is a fairly minor point but I'd just like to have my "last word" :-)

> I/O, such as direct I/O, scatter/gather interfaces, faster disks etc.).

Yes. Especially scatter/gather.

> byte-order optimizations.

If the exporter must do byte-swapping it must do copying. That's why we
avoided XDR.

> 	the app is not just plain silly about memory copies.

Which is why Un*x kernels keep pools of memory, use mbufs, etc. Anybody who
uses malloc/free in high-performance code needs some re-education.

But I'm talking about already highly tuned situations, running on small
boxes (no fans, low power CPUs) ... we've encountered situations where bus
bandwidth is a significant performance issue, and there's no point in
putting in roadblocks (such as potential byte-swapping by forcing use of XDR
or forcing field copies because the output format doesn't fit in an obvious
way to the internal data structures).

Even in high-performance situations, export of measurements is treated as an
extra cost, to be minimised. As I recall, on telephone switches the goal was
for operational measurements to take less than 5% of the system CPU.

- peter

---

From: "Peter Ludemann" <[email-masked]>
Date: Mon, 7 Oct 2002 23:53:25 -0700

This note discusses two things:
 - whether we need a data model to define the protocol
   (I claim that we do not)
 - cost of UDP broadcast vs minimalist reliable transport
   (I claim that there is very little difference)

[snip]
> 	No. We have no interoperability without a data model.

I agree that we need a data model. I don't see why it should affect the
discussion of the export protocol. The only requirements on the export
protocol are that it exhibit the desired reliability and efficiency; and
that it be able to transport all the data types required by the data models.
If we can agree on all the data types, then we can define the protocol; this
does not require defining the entire data model.

[After all, SNMP is defined independently of the various MIBs - which
correspond to the data model]

> 	general purpose data mover.

But deciding on an adequate "generalized data mover" is the point of this
evaluation process, I thought. I expect that there will be multiple data
models, for the various kinds of generalized devices. (e.g., for our probe,
we've defined three generic groupings of data: volume metrics; QoS metrics;
usage metrics and attributes -- similarly, SNMP's enterprise MIBs and
RADIUS's vendor-specific attributes).

There is one place where the nature of the data might intersect with the
protocol -- the high-volume metrics such as exported by NetFlow; and where
the claim is that a reliable protocol is an unnecessary overhead. So, let me
discuss this a bit more.

The argument is that with high volume export it is preferable to do
multi-cast UDP with sequence numbers; data loss can be detected by noting
which sequence numbers are missing ... and reliability can be increased by
having multiple receivers and de-duplicating what they receive.

For such a situation, the reliable protocol's *implementation* would not
keep a queue of data to retransmit on fail-over; it would only keep enough
buffer to deal with TCP's or SCTP's flow control and to handle bursts of
records. ACKs would be used only to notice when data are not received at the
other end and to cause fail-over. TCP "write would block" also causes
fail-over.

I would argue that the cost of exporting this way is very similar to that of
the UDP broadcast. And on the receiving end, it is *much* easier to handle
than a UDP-broadcast, which also needs stronger hardware because of the
higher de-duplication load.

Can be about the same for both:
- data copying (if scatter/gather is used)
- protocol overhead (sequence number, template ID, etc.)

The possible extra cost of the "reliable" export version is:
- timer for noticing when ACK isn't received [trivial cost]
- TCP vs UDP (little difference with modern implementations)
- TCP retransmit with packet loss (typically very low)
- cost of fail-over when no ACK received or write would block

The savings and benefits of the "reliable" version are:
- fewer packet writes (broadcast requires 2x as many packets;
  and TCP can pack more efficiently because records can
  span packets)
- lower network traffic (which adds to reliability)
- smaller machines for receiving
- more accurate estimate of loss due to lack of reliability
- can handle congestion

[As an aside, even with the UDP-broadcast version, the exporter ought to
compute totals for the various metrics and output those from time to time,
to provide a more accurate understanding of data loss. Perhaps such totals
are already available in the MIBs, but there remains the issue of how to
correlate with the sequence numbers of the exported data.]

- peter

---

From: Sebastian Zander <[email-masked]>
Date: Tue, 08 Oct 2002 13:49:32 +0200

Jeff,

> the older DTD definition syntax to define a grammar

Yes I know that schemas have some advantages over DTDs but
one could define XML schema for Diameter as well. My point
is to focus on the protocol and not on the data model
representation/description. But I agree that a data
representation with XML would be nice to have.

>   I don't understand your comment:

Sorry I was not sure what you meant with "binary format"
but now I understand that your protocol supports binary encoding
as well as XML right?

> as do many other IETF protocols?

Diameter can handle binary as well as non binary payload
(UTF).

Cheers,

Sebastian

---

From: [email-masked]
Date: Tue, 08 Oct 2002 09:34:40 -0400

> for operational measurements to take less than 5% of the system CPU.

	All good points. I understand better now where you are
	coming from.

	Paul

---

From: [email-masked]
Date: Tue, 08 Oct 2002 10:40:44 -0400

> does not require defining the entire data model.

	If I don't have a data model, how do I evaluate wether or not
	message ordering is a problem? If I don't have a data model how
	do I know I need a message for start of flow and end of flow versus
	one message that says here is a flow? If I don't have a data model
	how do I know I need a message to synchronize timestamps? If I don't
	have a data model, how do I know I can optimize by not encrypting
messages
	that only contain usage data (e.g. number of bytes, packets, etc...)

	We're not specifying the Bulk Data Export Protocol, we are are
specifying 
	a highly optimized IP Flow Information eXport protocol. The data model
	drives requirements to the protocol, not the other way around.

---

From: Jeff Meyer <[email-masked]>
Date: Tue, 08 Oct 2002 10:12:29 -0700

Sebastian,

   Responses inline...

Regards,

   Jeff Meyer

> representation with XML would be nice to have.

Well, I'd say consider the whole package.  IPDR has the
properties described today.  I'll be writing a larger
response to David Harrington's questions.

If we are just interested in the wireline protocol and
not extensibility, then the Schema part would be irrelevant.

However since extensibility is identified as a requirement,
I would not simply write off the fact that SOME extensibility
model exists.  Rather I would want to evaluate the model
overall.

> but now I understand that your protocol supports binary encoding

Yes.  There is an example in the IPDR base specification
(NDM-U 3.1) which illustrates the XML and equivalent
binary format for a given message.

   http://www.ipdr.org/documents/NDM-U_3.1.pdf

See p. 44 (section 4.3.2.1)

> Diameter can handle binary as well as non binary payload

IPDR also specifies UTF8 as the binary "encoding" model for
strings.  (For US ASCII, UTF8 is equivalent to the ASCII
string itself).

---

From: Jeff Meyer <[email-masked]>
Date: Tue, 08 Oct 2002 10:33:08 -0700

Hi,

   The comment about XDR requiring byte swapping is FUD.

   As far as I can tell, all the candidate protocols
use Network Byte order.  So depending on your system
architecture you'll need to byte swap for all of them.
Nothing unique about XDR.

   If you don't want to do byte swapping try CORBA,
however if you have two machines w/ different
(big-endian, little-endian) architectures talking,
I'm hard pressed to see how one side doesn't have
to byte swap.

Regards,

   Jeff Meyer

---

From: "Peter Ludemann" <[email-masked]>
Date: Tue, 8 Oct 2002 10:50:33 -0700

>    The comment about XDR requiring byte swapping is FUD.

Comparing me with Microsoft Marketing ... I'm flattered. :-)

> Nothing unique about XDR.

Not true for CRANE, which uses network byte order only for the template
negotiation and for the message headers. For the actual data, it uses
whatever the exporting device wants.

> I'm hard pressed to see how one side doesn't have

For CRANE, the *receiver* does the byte swapping, if needed. But the
assumption is that the receiver has more CPU and is not as real-time
constrained. Of course, if you want to do everything in network byte order,
CRANE allows that too (just specify big-endian for the data transmission).

- peter

---

From: Jeff Meyer <[email-masked]>
Date: Tue, 08 Oct 2002 12:10:24 -0700

Peter,

   Thanks for the clarification.

-- Jeff Meyer

---

From: "Peter Ludemann" <[email-masked]>
Date: Tue, 8 Oct 2002 23:20:03 -0700

[email-masked] wrote Tuesday, October 08, 2002 7:41 AM

[snip]
> > protocol are that it exhibit the desired reliability and efficiency;
> > and that it be able to transport all the data types required by the
> > data models. If we can agree on all the data types, then we can
> > define the protocol; this does not require defining the entire data
model.
>      messages that only contain usage data (e.g. number of bytes,
>      packets, etc...)
>
>      specifying a highly optimized IP Flow Information eXport protocol.
>      The data model drives requirements to the protocol, not the other
>      way around.

[snip]

Maybe we're just arguing over terminology, but I'm not sure.

In the database world, the term "data model" usually means:
  - the table fields' meanings (semantics + type + attributes
    such as "not null")
  - constraints amongst records within a table (uniqueness)
  - the relationships amongst tables (foreign keys for joins)

Maybe I missed something in all the emails, but I thought that
the export data was to be one record per "flow". There can be
relationships between flows (e.g., parent/child, such as when
dealing with multi-layer protocols like H.323). These have an
obvious representation in a database. The other obvious
representation is the master/detail style, with a master record
containing summaries and a set of details which reference the
master's unique id.

Suppose we're dealing with a long-running flow, whose output
is split into a number of slices. Then the obvious record
format is something like this:
    flow_id, start_time, end_time, metric1, metric2, ...
and to aggregate all the information for one flow:
    select flow_id,
           min(start_time),
           max(end_time),
           sum(metric1),
           ...
    from data_table
    where flow_id = ...
    group by flow_id
A similar query will get me the master/detail summaries.
I can tell whether a flow is finished or not by the presence
or absence of end_time.

So, I don't understand the problem with beginning and end
of flow that you describe ... each individual record just
contains a slice and the determination of how to process
the record is done downstream [of course, it doesn't have
to be in a database; it could be in a stream processor].
We do this kind of stuff all the time when we deploy our
systems and we haven't seen the need for anything fancier
than fairly straightforward individual records. Often there's
a need to do downstream interpretation of fields (e.g.,
map an IP address to a customer based on information from
a RADIUS server); but it's stuff that a database designer
would consider pretty simple (the volumes of data and the
real-time requirements are what make life interesting).

How does "message to synchronize timestamps" fit into the
data model? The timestamp fields are just values, whose
definition is determined by an agreement between the
exporter and the collector (is it local time or UTC?
accuracy? synchronized with other data sources? etc.)

Encryption ... that should be specified by the templates
or field definitions during the data definition - just
specify whether particular records or fields within
records ought to be encrypted.

So, to me any flow export data model will consist of
individual records, whose fields are all of a set of
pre-defined data types. That is, the export protocol
implies a schema for defining individual data models.
The data model allows multiple record types (templates),
which are multiplexed onto the output and distinguished
by one or more fields in the records (for downstream
processing, you might put the different record types
into different tables).

But I wonder whether I'm missing your point, so please tell
me where my response doesn't answer your questions.

- peter

---

From: [email-masked]
Date: Wed, 09 Oct 2002 10:12:20 -0400

>   - the relationships amongst tables (foreign keys for joins)

	In my mind the data model for a stream does share some
	characteristics with a data model of a database.

	However, and I'm no data modeling expert, it seems to me that
	with a database you have a representation of information
	in an existing data set. With a stream you have a represention
	spread out over time. I don't mean timestamps, I mean the time 
	between delivery of one message and delivery of another.

> the export data was to be one record per "flow". 

	Exactly the point of why we need to have the data model.
	One of the questions answered by that is whether or not
	a flow is represented in a single message or over a set
	of messages.
	
	For example, with long standing flows why do I need to repeat 
	all the flow attributes when I just want to update the byte 
	count? I'm not advocating one position or another in this
	particular issue, I'm just using it to show how the data model	
	affects the protocol.

> There can be

	Again, how it is represented in a database may not
	be the best way to represent it in a stream.

> accuracy? synchronized with other data sources? etc.)

	Lets suppose there is an option, for small devices,
	to report time in terms of sysuptime. The collector
	has 2 choices. 1) It can assume the message was delivered
	relatively quickly and use its current time. 2) If the
	protocol provides a way to obtain the sysuptime to
	current time it can use that info for all subsequent
	messages.

	If I don't have view into the data being sent and
	its meaning, how do I know the protocol needs
	to have support for option #2?

> records ought to be encrypted.

	You seem to peer down into the data being sent
	when it is convenient to do so. 

	How did you know the protocol needed to offer 
	encryption features?

	Why did you allow individual fields to be encrypted?
	Do we really need that level of granularity and
	complexity?

	Why are templates providing encryption information?
	Why not just use IPsec and have no knowledge in
	the protocol at all? 

	General data movers will provide lots of features 
	to fit a wide variety of applications. IPFIX data
	movers will provide features optimized for providing
	IP flow data.

	Paul

---

From: Ganesh Sadasivan <[email-masked]>
Date: Wed, 09 Oct 2002 14:51:18 -0700

Agreed.

>    (I claim that there is very little difference)

What do you mean by "minimalist reliable transport" ?
Since congestion awareness is mandatory, I don't know
why UDP is a topic of discussion here. See my comments inline.

> having multiple receivers and de-duplicating what they receive.

One does not need to multicast UDP to indicate missing reliability.

> other end and to cause fail-over. TCP "write would block" also causes

So records can be lost in both cases. Also in a continuous export (in the case of

high volume export) the rate of ACKs is once every 2 send packets.

> the UDP broadcast.

Are you talking w.r.t the end-point (exporter & collector) or the network itself.

For the former, the processing overhead is much higher in TCP.

> And on the receiving end, it is *much* easier to handle

In a typical datapath the instruction ration of UDP to TCP is
multi-fold. I don't know how you can tell that the protocol
overhead is the same.

>   and TCP can pack more efficiently because records can

As mentioned above if the  absence of reliability is what we want,
then we need not do a UDP broadcast.

> - smaller machines for receiving

what does this mean?
thanks
Ganesh

---

From: "Peter Ludemann" <[email-masked]>
Date: Wed, 9 Oct 2002 23:59:03 -0700

> 	between delivery of one message and delivery of another.

I don't understand the point; if you want time intervals, you can
just subtract the timestamps. This is provided by SQL extensions
from Oracle and Teradata, amongst others; or by use of stored
procedures.

(If you're talking about jitter ... this is a flow-specific value
and would be output as a variance parameter with the flow.)
 
> 	particular issue, I'm just using it to show how the data model	

This sounds like the classic master/detail database design (the name
derives from a typical invoice which has a number of line items
and a summary with the totals, name and address, etc.). The master
and detail records are distinguished by a "type" field which would
allow streaming into different database tables; and the two kinds
of records are tied together by the flow ID.

So, as long as the protocol allows more than one kind of record
to be output, this can be done. No need to duplicate data
unnecessarily.

[snip: my description of how to slice a flow over time]

> 	be the best way to represent it in a stream.

Given that we can need to transmit only the data that are needed,
I don't see the problem. My example had some fields repeated for
every record; but it could have easily been done with the
master/detail style to reduce the amount of data.
 
> 	to have support for option #2?

Why do you need the collector to ask the device about its
notion of the time? You could just as easily define a record
which reports the device's idea of the time. With a reliable
protocol, the information is guaranteed delivered.

Anyway, CRANE allows both possibilities: the STATUS REQ
message from the collector is answered by an arbitrary
record, which could include such information. (That is,
CRANE is designed to work primarily in push mode, but allows
pull when needed ... perhaps "STATUS" is a bit of a misnomer.)

[CRANE also has the device report its idea of boot time as
part of the initial handshake ... that information was intended
for de-duplication in case the device rebooted; but obviously
the boot time can also be used to compute times from sysuptime.
But I'd hardly accept or reject any protocol for the absence
or presence of this feature, as long as there is some method
of distinguishing records across re-boot]

> 	when it is convenient to do so. 

It just seems to me that whoever defines the data is
the best person to decide on encryption needs.
 
> 	How did you know the protocol needed to offer 

I don't understand ... anyway, our general view is that
encryption should be provided by IPsec (the principal
of layering).

My experience with databases is that fine control of access
to fields is difficult to do. David Harrington's comments on
this have merit; but they go far beyond just the data export
and collection - there is also access to data once it's collected.

This certainly gives us something to discuss at future WG meetings. :-)

> 	Do we really need that level of granularity and

CRANE doesn't even have encryption; as mentioned above we
suggest the use of IPsec.

> 	movers will provide features optimized for providing

But all the proposals are extensible to some degree, so they
are general data movers to that extent. I don't see how being
"general" is any kind of disadvantage for a protocol, as long
as it doesn't introduce unnecessary overhead.

- peter

---

From: "Peter Ludemann" <[email-masked]>
Date: Thu, 10 Oct 2002 00:04:34 -0700

Sorry for bringing up the topic of UDP ... I mis-read a thread and thought
that UDP was coming back on the table.

It is not clear to me how any protocol that does not contain
application-level ACKs can be fully reliable, even if layered over TCP or
SCTP. This is because the transport-layer acknowledgment only means that the
data have arrived at the other machine, not that the application has
actually processed it. (There's nothing preventing a TCP or SCTP stack from
allowing application-level acknowledgment; but I'm not aware of any for TCP
and my reading of the API in RFC 2960 indicates that the acknowledgment is
transport-level.)

As to my comment about UDP vs TCP for performance ... this is probably
irrelevant for the current discussion because the proposal is that IPFIX
would use NetFlow over TCP or SCTP. In such a case, a UDP-with-reliability
protocol (which NetFlow isn't; but, for example, NFS is) ends up being about
the same performance as TCP. Clearly, the path length with pure UDP would be
less than for TCP because UDP is just "fire and forget". However, if
broadcast is being used to increase reliability (as Ganesh points out, this
isn't necessary; but if there is no ACK mechanism, then the only choice is
to broadcast to all collectors), then the path length increases. Whether
outputting 2 UDP packets is a longer or shorter instruction path length than
sending a single TCP packet, I don't know.

Slight change of subject: could NetFlow be easily extended to handle data
that isn't fixed-length (e.g., HTTP URLs)? I know that in NetFlow's current
use, there's no need for such attribute export; but there are smart devices
that look deeper into packets for smart switching or firewalling, which can
use such information and might want to export it. [Probes can also export
such information.]

Time for me to shut up and go on vacation.

- peter

> -----Original Message-----
> From: majordomo listserver [mailto:[email-masked]]On Behalf
> Of Ganesh Sadasivan
> Sent: Wednesday, October 09, 2002 2:51 PM
> To: Peter Ludemann
> Cc: [email-masked]
> Subject: Re: [ipfix-eval] RE: Discussion of Candidate Protocols - data
> model; broadcast vs reliable
>
>
>
>
> Peter Ludemann wrote:
>

---

From: [email-masked]
Date: Thu, 10 Oct 2002 09:40:09 -0400

In order to follow one idea from start to end, here are several excerpts
from
our exchange so far...

Peter:
------
> discussion of the export protocol.

Paul
----
>         If I don't have a data model, how do I evaluate whether or not
>         message ordering is a problem?

Peter
----
> the export data was to be one record per "flow".

Paul
----
>         particular issue, I'm just using it to show how the data model  

Peter
-----
> to be output, this can be done. No need to duplicate data

        Now we have a message ordering problem which needs to be
addressed.
        Either sequence numbers need to be introduced into the messages
or
        a reliable transport is needed.

        If IPFIX's data model does not allow the splitting up of
information
        then then there is no ordering problem and no requirement for
sequence
        numbers or a reliable transport (at least not driven by the data
model).

        This is not the only example of the data model imposing
requirements
        on the protocol. But if I can show one, I'm hoping you'll agree
that
        there must be others.

        Paul

        P.S. - you can have the last word when you get back from
vacation :-)

---

From: Vamsidhar Valluri <[email-masked]>
Date: Thu, 10 Oct 2002 11:26:12 -0700 (PDT)

Peter,
  Please see inline

> use such information and might want to export it. [Probes can also export

Example: If we want to send a variable length field we use repeat count

Suppose we want to send string "abc"

Netflow version 9 Template

<----2 bytes ----->
------------------
header...
------------------
Field 1 Type = REPEAT_COUNT
------------------
Field 1 Length = 2 bytes
-----------------
Field 2 Type = CHAR_VARIABLE_LEN
-----------------
Field 2 Length = 1 byte
-----------------
.......

Corresponding Netflow version 9 Data record

<-----2 bytes------>
-------------------
Header .....
-------------------
   3 = ( Next field (Field 2) is repeated 3 times)
-------------------
 a      |   b
-------------------
 c      | ......
-------------------

-vamsi

---

From: Ganesh Sadasivan <[email-masked]>
Date: Thu, 10 Oct 2002 11:54:40 -0700

> > use such information and might want to export it. [Probes can also export

Yes the current requirements for netflow does not need it to send variable
length fields, but it can be easily extended as Vamsi explained below using
a "length" field in data records.
-ganesh

---

From: "Peter Ludemann" <[email-masked]>
Date: Thu, 17 Oct 2002 10:34:37 -0700

Paul:

I'm back from vacation and ready to have my last word on this. :-)

[I wonder whether instead of a week in hotel rooms in Atlanta, we should all
book a week on a sailboat in British Columbia or New Zealand and sort out
our issues whilst cruising down the fjords ... ]

My comments interspersed near the end (thank-you for condensing this
discussion so nicely).
But to summarize them:

   - the protocol needs a unique key for each record, to allow
de-duplication.

   - the data model needs a unique key for each record, to
     allow classic database manipulations.

   - the two keys MAY be the same but in general will be different.

   - the issues of a reliable protocol and the data model are independent.

- peter

> [email-masked] wrote Thursday, October 10, 2002 6:40 AM:
>
>
> from our exchange so far...
>
> Peter:
> ------
>         addressed. Either sequence numbers need to be introduced
>         into the messages or a reliable transport is needed.
>
>         information then then there is no ordering problem and no
>         requirement for sequence numbers or a reliable transport
>         (at least not driven by the data model).

The issues of reliable transport and sequence numbering are independent.

Putting on my database designer hat for a moment ... a classic "database
beginner's mistake" is to not have a 'primary key' for records -- that is a
unique distinguishing identifier for each record. In the case of IPFIX, an
obvious 'primary key' is the sequence number (or, in the case of CRANE, the
exporter's "boot time" + sequence number). This is needed anyway for
deduplication, so it ought to be a basic requirement.

However, the sequence number isn't a particularly good choice because it's
not "natural" for the data exported. A better choice would be a time-stamp
that fits with the data (e.g., time stamp of the last packet seen) +
information that uniquely identifies the flow (e.g., IP-port pairs + time
stamp of first packet seen). The exact nature of this unique key is
determined by the data model and should (in general) *not* be part of the
protocol.

Switching to my protocol-design hat ... even a reliable protocol can still
result in lost records; "reliable" simply means that every attempt is made
to deliver the data and if that fails you know when you've lost data [and,
with a good design, you can have some idea of how much has been lost - but,
as I mentioned in an earlier note
(http://ipfix.doit.wisc.edu/archive/1207.html -- see also
http://ipfix.doit.wisc.edu/archive/1255.html), the data model determines the
lost data information also]. If there is failure in delivering the data, a
fail-over might re-send some data that have already been received (because
the ACKs were lost), so the protocol needs unique identification of records
to allow de-duplication. It might be tempting to use the protocol's unique
identification to estimate data loss; but you'll get better results if the
data loss information is more closely tied to the data model.

>         requirements on the protocol. But if I can show one,
>         I'm hoping you'll agree that there must be others.

I hope you'll agree that the data model does not impose any additional
requirements on the protocol for this situation.

>         P.S. - you can have the last word when you get back from

Done. :-)

- peter

---

From: [email-masked]
Date: Mon, 21 Oct 2002 10:04:50 -0400

You address de-duplication and data loss but some how missed
message ordering which was my only point.

If the data in the messages is independant I can process
immediately. If not, I need to know if they came in order
before I can process the message. In this case nothing
is lost and nothing is resent, the messages just arrived 
out of order.

Paul

---

From: "Peter Ludemann" <[email-masked]>
Date: Mon, 21 Oct 2002 07:20:25 -0700

> is lost and nothing is resent, the messages just arrived

Because TCP or SCTP is used, the data will arrive in exported order (with
possible duplicates).

However, just because the data arrive in order doesn't mean that the
exporter produces them in order. (With our PacketSight probe, the flows can
be exported every "n" minutes, whether they are finished or still active --
the order of export is determined by hash tables in the probe because
sorting into order would be too expensive; further sorting may be needed
downstream to combine multiple partial records for a long flow).

To order the flow data, it is necessary to have some kind of sequence data
in the exported records. This is just part of the data model (you'll need it
in the database) ... for PacketSight, we use the nanosecond timestamp of the
first packet for the flow but you might want to use something different
[this is still not an absolute guarantee of ordering: if two probes are
reading the same data they might order two flows differently because of
microsecond non-determinacy in the handling of packets in the two separate
NIC drivers].

Again, this ordering information doesn't require anything special from the
protocol; it's all in the data model.

- peter

---

From: [email-masked]
Date: Mon, 21 Oct 2002 11:08:16 -0400

So you do require something of the protocol to solve this problem!

Paul

---

From: "Peter Ludemann" <[email-masked]>
Date: Mon, 21 Oct 2002 08:46:19 -0700

> So you do require something of the protocol to solve this problem!

In the sense that the protocol must be reliable, allow deduplication, and
support for certain data types: yes. In the sense that the protocol needs to
be know something of the data model: no.

In my view, the order of arrival of records is not very important because
the records will probably have to be deduplicated, reordered, and aggregated
anyway [but they should at least be "almost" in order, to make the
deduplication and reordering easier]; but the order of data is important in
the sense that if data spans packets then the packets must be received in
order (as with TCP or SCTP but not UDP) -- and this is especially important
with variable-length data (e.g., URLs).

Having said that, CRANE does ensure that the records arrive in the order
that they were sent; and it relies on the sequence numbering of the records
to verify that the protocol is working correctly. Downstream deduplication
can take advantage of this ... but it's not part of the data model as such.

- peter

---

From: [email-masked]
Date: Mon, 21 Oct 2002 12:03:37 -0400

The fact that you meet the requirement does not change the
fact that the data model created the requirement. If the
data model spans messages, the ordering problem is a requirement
on the protocol.

I've tried to narrow the discussion down to one simple example to
show the connection between the data model and protocol requirements.

I've either succeeded or failed. In either case I've said enough.

Paul

---

From: "Tal Givoly" <[email-masked]>
Date: Mon, 21 Oct 2002 11:14:25 -0700

Paul,

Please note that after debating the importance of message ordering, Peter
did mention below that the protocol does ensure in-order arrival of messages
sent. In fact, it even ensures in-order arrival in case of failure of one
(or more) servers/receivers.

So if in-order arrival is a requirement of any particular data modely you
envision, then there is indeeed full support within the CRANE protocol.

I tend to agree with you that there would be cases where in-order arrival
may be dictated by the data model (e.g. change in sampling technique while
retaining the same templates, it is so desired) - and, therefore, these
would be fully supportable.

Tal

-----Original Message-----
From: majordomo listserver [mailto:[email-masked]]On Behalf
Of [email-masked]
Sent: Monday, October 21, 2002 9:04 AM
To: Peter Ludemann
Cc: [email-masked]
Subject: Re: [ipfix-eval] Re: Discussion of Candidate Protocols - data
model; broadcast vs reliable

Paul

--
Help        mailto:[email-masked] and say "help" in message
body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Simon Leinen <[email-masked]>
Date: 24 Oct 2002 23:44:08 +0200

Sorry for the late response - finally catching up after vacation...

>> On Wed, 02 Oct 2002 08:53:37 -0400, [email-masked] said:
[in response to Peter Ludemann's message]
>> >       copy. In my experince performance was limited by how fast
>> >       the disk writes are.
>> 
>> Some collectors will write large amounts of data to disk - if this
>> is the main bottleneck I'd assume that they use the accounting data
>> without too heavy processing (or they need a better approach to
>> disk I/O, such as direct I/O, scatter/gather interfaces, faster
>> disks etc.).

> 	I don't follow. Are you saying I/O isn't typically the
> 	bottleneck at the collector?

That depends on what you consider a typical collector.
The one I've written as part of our accounting system has relatively
large in-core data structures, but writes a relatively small amount of
data to disk.  It performs convoluted application-specific aggregation
that doesn't really lend itself to being done in a meter/exporter in a
router.

>> Unnecessary copying should be avoided mainly because memory
>> bandwidth doesn't seem to grow as fast as other performance
>> dimensions.  Personally I think this is more of an issue than for
>> example byte-order optimizations.

> 	to big culprits of CPU usage.

Then I'm glad I wrote my collector in Lisp, because Lisp doesn't have
either of these (I'm being serious here, the garbage collector even
keeps the working set compact over months).

>       A design that causes lots of task switching is another. Your
> 	program needs to be pretty finely tuned before memory copy
> 	shows up, assuming the app is not just plain silly about
> 	memory copies.
-- 
Simon.