Subject: Re: [ipfix] current enterprise IE approach is insufficient

From: Brian Trammell <[email-masked]>
Date: Sun, 6 Nov 2005 08:20:55 -0800

All,

I've been kicking this idea around for a little while, and I've got a  
little front matter done for an Internet-Draft on the subject. The  
essential idea is to extend templates to add 32 more bits in two  
fields - storage and semantics - to each field specifier IF the  
enterprise bit is set (it is assumed that implementors understand the  
IANA-assigned types). The new field specifier format for enterprise  
specific IEs looks like the following:

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|1|        IE number            |         Field Length          |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                       Enterprise Number                       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|    Storage    |                   Semantics                   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

The specifics I'd like to discuss with the list before continuing.

These additional fields would define the low-level type and semantics  
of the private enterprise IE. The idea here is _not_ to tell a human  
on the other end of the collector what the IE _really_ means - that's  
outside the scope of template type extension mechanism (and can be  
handled with a single information element anyway; see below) -- it is  
instead to tell an automated collection and storage system what it  
can or cannot do with the IE. In order to replace one return-from- 
IANA problem ("I want a new IPFIX type") with another ("I want a new   
IPFIX semantics codepoint"), these storage and semantics specifiers  
are to be implemented as bit fields. The Storage specifier is 8 bits,  
as follows:

MSb           LSb
+-+-+-+-+-+-+-+-+
| group | scale |
+-+-+-+-+-+-+-+-+

where group is

0   unsigned
1   signed (2's complement)
2   seconds (scaled as SI)
3   address
4   float (IEEE)
5   reserved
6   text array
7   binary array
8-F reserved

and scale is

0     1 bit,  SI unit
1     8 bits, SI deci
2    16 bits, SI centi
3    32 bits, SI milli
4    48 bits, SI micro
5    64 bits, SI nano
6   128 bits
7-F reserved

So, a Boolean is a 1 bit unsigned (0x00), a UTF-8 String is an 8 bit  
text array (0x61), and an EAP-48 (MAC) address is a 48 bit address  
(0x34) (this last is why 48 bits is a valid scale). You can very  
easily extend the existing IPFIX typespace using this system, for  
example to include UTF-16 strings (0x62), EAP-64 addresses (0x35),  
128 bit counters (as has already been suggested) (0x06), and so on.

The Semantics specifier is 24 bits, all simple flags defining how the  
value may be interpreted by a collector and which operations make  
sense at collection and storage time. This is incomplete (as it is  
somewhat more difficult to do correctly than the storage types  
above), but the following bit positions should give you an idea where  
I'm trying to go with this:

Bits
  0 isCounter (value may wrap)
  1 deltaSinceLast (0 value is previous record with same keys)
  2 deltaSinceMessage (0 value is beginning of enclosing IPFIX Message)
  3 isKey (is key field; can be rolled up on aggregation)
  4 arithmetic (Arithmetic operations make sense with this value; it  
is scalar)
  5 logical (Logical operations make sense with this value; it is a  
bitfield)
  6 aggregatable (Operations may be used to roll this value up during  
aggregation)
  ... and so on

I'm a little concerned that 24 bits isn't enough to specify all the  
various things that a collector might want to do - the feedback of  
other collector, storage, and aggregator designers would be much  
appreciated.

Conveying template information to a human (who would then implement  
collection for that template) is somewhat simpler (if a little  
handwavy): add a humanReadableDescription information element, a  
string containing a URL containing a human readable description of  
the template and its information elements. This IE is intended to be  
scoped to a TemplateID in an options data record.

If there is any interest in continuing this work, I can finish the  
draft I have in progress for Dallas - anyone wishing to discuss this  
idea further, I'm in Vancouver all week; just drop me a line.

Regards,

Brian

On Sep 22, 2005, at 8:20 AM, David Moore wrote:

>
> In BOF and small group discussions at FloCon this week, several  
> current
> developers of flow meters/exporters had concerns with the  
> usefulness of
> IPFIX for their data.  Developers of analysis/aggregation/reporting
> tools also had similar concerns about their ability to write generic
> processing tools.  While several concerns were raised, this one seemed
> to be the greatest problem for both adoption and functionality.
>
>
> As a short summary: The current information model approach only  
> provides
> information elements in the templates which combine *both* data type
> (i.e. section 3 of info draft) and specific semantics.  And there  
> is no
> current mechanism to provide the data type for a given information
> element (e.g. a vendor specific one).  Allowing this information to be
> provided either in the data-stream via templates/records or via some
> automatable external lookup (e.g. XDS).  The current situation makes
> writing lots of analysis tools difficult.
>
>
> Personal, real life example of the pain:
>
> For caida network telescope data about DoS backscatter, one natural
> representation of the data would be in flows which contain 3 IP
> addresses as part of the key.  We call these 3 addresses: Attack IP,
> Target IP and Responding IP.  None of these 3 addresses have semantic
> meaning which natural maps to IPFIX's sourceIPv4Address or
> sourceIPv4Address.  However all five have the same data type,
> ipv4Address.
>
> So the current procedure for me to provide this data in an IPFIX  
> format
> would be to create 3 new IE (information elements) identifiers  
> within my
> enterpriseID space.  At first glance, this seems like there is
> extensibility supported in the protocol for me to do what I want.
> However, I will not be able to use any existing analysis tools on this
> data, since they will not recognize these IEs.  As a specific  
> example, I
> would be unable to use nafscii to convert my records to a text format
> without either adding code to nafscii or explicitly adding my IEs to
> nafscii config files in a tool-specific manner.  But nafscii (and a
> large set of other tools) doesn't need to know the semantics of these
> fields to display them.  Many tools only need to know the field's data
> type to work on them.  However, the IPFIX working group hasn't  
> provided
> any mechanism for people to provide the data type of an IE in a
> standardized automatable way.
>
> Having to do some sort of tool-specific configuration (or code
> modification) on every tool just to tell it that X is an  
> ipv4Address and
> Y is a totalCounter in an unsigned64 is a huge amount of work for  
> users
> and developers of vendor specific IEs.
>
> Two possible approaches to get around this would be to have some
> mechanism for including the data type infomation in the stream,  
> perhaps
> via some sort of record which allows mapping <enterpriseID,IE>  
> pairs to
> data type information, or to have some external mechanism for tools to
> be able to obtain the information about IEs for a given enterpriseID.
> One potential advantage of the external approach is that there would
> then be plenty of space to include all of the additional information
> that one would want, such as the name, description, references, etc.
> However, including the information inline, allows more rapid
> processessing of the data, and allows the flow stream (or file) to be
> partially self-documenting (the point of templates in general).
>
> Solving this problem seems very appropriate for the ipfix wg.
>
> -- david
>
>
> --
> Help        mailto:[email-masked] and say "help" in  
> message body
> Unsubscribe mailto:[email-masked] and say
> "unsubscribe ipfix" in message body
> Archive     http://ipfix.doit.wisc.edu/archive/