Subject: [ipfix-arch] Re: [ipfix] Re: Information Model]

From: Mark Fullmer <[email-masked]>
Date: Tue, 12 Feb 2002 20:15:24 -0500

On Tue, Feb 12, 2002 at 03:14:11PM -0800, Ganesh Sadasivan wrote:
> >      * Flow Header: Every flow export packet includes a flow header
> >        which carries some general information about the exporter, packet
> >        identification etc.
> >
> > ###
> > Using the export "packet" term too soon, ie if this is a TCP control
> > connection the packet may be partial.
> 
> I do not understand.  This packet is generated at the level of exporter

I'm probably being picky, but the "packet" sent from the exporter
may only have a partial IPFIX header if it's a stream protocol like TCP, ie
the the "export" may be in two packets, or three...PDU is an option.

> >    Example:
> >    The flow records which meets the following selection criteria are
> >
> > ###
> > You're talking about exporting before defining what the export is.  Need
> > to establish relationship between flow state expiration and export first.
> 
> Does export need a more explicit definition than IPFIX?

When the flow is ready for export needs definition, ie with NetFlow this
is when it's expired from the cache.  There may be an implementation that
can be configured to periodically export active flows, or generate an
export at flow state creation.

> > ###
> > Why is the separate control channel optional.  Why not just define it
> > and make it TCP?  It simplifies a bunch of other things later on.
> 
> I agree with you that it simplifies. These are 2 models we are suggesting
> which can be used for 2 different situations.

It would be nice if IPFIX required that the exporter shut down the
data channel upon detection of the collector going away, ie when
the control connection is broken.  With the single control/data channel
and no feedback from the collector this is not possible.

> > The control connection should include a simple authenticator like a shared
> > secret so the collector can have some confidence it's a valid exporter.
> 
> You mean to say something like MD5?

Or just a password type/value passed along when the templates are 
uploaded.  I'm guessing the control connection here will also inform
the collector where/how to listen for the data channel?

> >    Sequence Number:
> >    Exporter generated number which uniquely identifies a packet.  The
> >    sequence number increments for subsequent export packets.  A separate
> >    sequence number is manintained for control and export data packets if
> >    control and data are send in separate packets.
> >
> > ###
> > In the current NetFlow export implementation the sequence number is
> > per flow so the collector can calculate lost flows.
> 
> I thought it was per packet. Am I wrong? If so it is additional 4 byte info
> per record? I thought there was a # of flow records in the header which
> along with the sequence number in the header would give the count.

I wasn't clear.  The sequence number is per packet / per linecard and
incremented by the count of records.  This allows the collector to
calculate the number of lost flow records.  If there are multiple 
record types in the packet as you describe the collector no longer has
the ability to calculate lost flows.  Not incrementing the sequence
number for the other record types obviously negates the ability to
detect lost control information.

This is also why I would like to see the Source ID restricted to
16 bits.  The collector needs to potentially maintain a sequence number
for each source ID for each exporter.  A 16 bit source ID as in
the current NetFlow exports can be realized with just a u_int32[65536],
anything larger and hashing is needed to find the sequence number.

> 
> > With per packet
> > sequence number this functionality is lost.  Mixing in control information
> > complicates it even more (did you lose a control packet or flow packet).
> 
> >    Source ID:
> >    Unique identifier per observation domain.
> >
> > ###
> > There needs to be an exporter ID (IP address) so that replicated IPFX
> > packets retain the exporter.  I'd like to see the Source ID restricted
> > to 16 bits so the collector doesn't need to maintain a hash table to
> > track the sequence numbers per exporter.
> 
> What do you mean by "replicated IPFX packets"?

An exporter can maintain m data exports, where m is probably 1 or 2.  If
there are n active collectors where n > m then something needs to replicate
the data channel.  When the all the collector has to identify the exporter
is the transport address (ie destination IP) then the collectors see the
replicator as the data source.  A common implementation is one collector
performing security/IDS and another doing traffic analysis.

> > Need common terminology again.  Statistics or Accounting?
> 
> Statistics.

Your marketing people disagree :)

> > ###
> > Verbage needs to be added about alignment.  u_int32's on 32 bit
> > boundries, u_in16's on on 16 bit boundries, etc.  This also implies
> > needing a 'NULL' field type for padding.  Need to define byte order
> > of fields.
> >
> > Also the exporter SHOULD provide optimal encoding, ie

[...]

> I guess over netwrok we have to send it as packed without any padding.
> The exporter and collector may be running different processors.

If the data records are sent as words with word alignment and long words with
long word alignment the decode/encode process is going to require
less cycles.  With alignment the data elements can be fetched with
with just an indirect reference, without you'll need to bcopy() them out
first.  Worst case overhead is 3 bytes per flow record.

> > ###
> > s/got//
> Just one place - right?

I think I found one more :)

mark

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/