Subject: [ipfix] IPFIX Concentrator

From: Falko Dressler <[email-masked]>
Date: Tue, 30 Nov 2004 13:54:13 +0100

Dear all,

currently, we are implementing an IPFIX aggregation tool, in IPFIX
terminology a concentrator. Regarding the export of aggregated data,
there are a couple of possible solutions, which (all) require the
modification of IPFIX or the addition of some new propoerties. Before we
come in with a new draft describing the aggragation mechanisms and (our)
final solution, we want to ask this list for suggestions/remarks.

Cheers,
Falko

-- 
Dr.-Ing. Falko Dressler
Computer Networks and Communication Systems
University of Erlangen-Nuremberg, Germany
Phone: +49 9131 85-27914 / Fax: +49 9131 85-27409
EMail: [email-masked] / [email-masked]
WWW: http://www7.informatik.uni-erlangen.de/~dressler/

0 Abstract

Due to the possibly high amount of measurement data exported using IPFIX we
started a flow aggregation project. Aggregation should take place at a
measurement point or on a separate flow concentrator (as mentioned in the
IPFIX documents). Aggregation rules include the merging of flows with similar
properties such as a common source network.
Planning on implementing an IPFIX concentrator we were soon faced with the 
question of how to transmit the aggregated flows' shared properties (e.g.
a Source IP in 192.168.0.0/23) via IPFIX.
As all solutions we could find have their pros and cons and some need 
additions to IPFIX, we decided to post the ones we considered most fit 
to this mailing list for review and discussion.

Different ways of sending these common properties will be presented in
Section 1, while Section 2 deals with Data Types necessary or useful
for doing so. Section 3 contains some thoughts on pros and cons of
the different approaches presented.

1 Communicating common properties

This section presents different ways of communicating field values that are 
constant for all records in a Data Set.

In the examples used we assume our concentrator was given the following 
aggregation rules:

 - aggregate all flows with a source IP in 192.168.0.0/23 separating them
   only by their destination port
 - pass through all other flows

These rules could be expressed as the following table:

Protocol        Src_Net         Dst_Net         Src_Ports       Dst_Ports
-------------------------------------------------------------------------------
any             192.168.0.0/23  any             any             each
each            each            each            each            each

We further assume our concentrator receives the following flow records:

Src_IP          Dst_IP          Src_Port        Dst_Port        Pckts
-------------------------------------------------------------------------------
192.168.0.201   194.95.211.10   64235           80              10
192.168.1.202   194.95.211.11   64236           80              10
192.168.0.203   194.95.211.12   64237           110             10
192.168.2.204   194.95.211.13   64238           80              10
192.168.2.205   194.95.211.14   64239           80              10

1.1 Complete Data Set

The most straightforward approach to communicating common properties is to 
simply transmit them with every flow record.

Given the example rules and flows from above our concentrator would then
output the following Data Set (provided a new portlist field):

Src_IP          Dst_IP          Src_Port        Dest_Port       Pckts
-------------------------------------------------------------------------------
192.168.0.0/23  *               *               80,110          20
192.168.2.204   194.95.211.13   64238           80              10
192.168.2.205   194.95.211.14   64239           80              10

1.2 Association of Data Sets with Option Data Sets

This approach transmits all common properties in one option data record rather 
than in each data record. Now two ways of maintaining an association between 
transmitted data records and their option data records can be thought of:
Only having the option data record refer to the data records (1.2.1) or having 
both refer to each other (1.2.2).

1.2.1 Implicit forward association

This approach uses
 (1) one template for all non-aggregatable flows,
 (2) one template each for all flows matching a given aggregation rule, 
     containing only discriminating properties of these flows, along with
 (3) one (option) template each describing common properties of these flows

(1) will be directly passed through by the concentrator paying attention
to reserve template IDs for (2) and (3). The association between (2) and (3) 
will be maintained by using (2)'s template ID as a scope value in (3).

Given the example rules and flows from above our concentrator would now 
first transmit to the collector one Data Set that contains information about 
all flows that matched no given rule and were thus not aggregatable:

Src_IP          Dst_IP          Src_Port        Dest_Port       Pckts
-----------------------------------------------------------------(Set ID 257)--
192.168.2.204   194.95.211.13   64238           80		10
192.168.2.205   194.95.211.14   64239           80		10

The second Data Set contains all flows that matched our aggregation rule.
Note that besides the packet count only the flows' discriminating property, 
their destination port, is sent:

Dest_Port       Pckts
-----------------------------------------------------------------(Set ID 258)--
80              20
110             10

The final set, an option data set, refers to the Template used to transmit
the aggregated flows and defines the property shared by all flows that use 
it - a source IP in 192.168.0.0/23:

Template        Src_IP
-----------------------------------------------------------------(Set ID 259)--
258             192.168.0.0/23

1.2.2 Explicit association

This approach differs from (1.2.1) only in that now aggregated flow records will
each contain a pointer to the option data set defining their common properties.

Again the first Data Set our concentrator would now transmit to the collector
contains information about all flows that matched no given rule and were thus 
not aggregatable:

Src_IP          Dst_IP          Src_Port        Dest_Port       Pckts
-----------------------------------------------------------------(Set ID 257)--
192.168.2.204   194.95.211.13   64238           80              10
192.168.2.205   194.95.211.14   64239           80              10

The second Data Set contains all flows that matched our aggregation rule.
This time each flow record explicitly points to the Option Template used 
to transmit common properties.

See_also        Dest_Port       Pckts
-----------------------------------------------------------------(Set ID 258)--
259             80              20
259             110             10

Like in 1.2.1 the final set, an option data set, defines the property shared 
by all flows that use Template 258: A source IP in 192.168.0.0/23:

Template        Src_IP
-----------------------------------------------------------------(Set ID 259)--
258             192.168.0.0/23

1.3 Introduction of a new type of Template Set

This approach uses a new type of Template, termed a Data Template.
The Data Template allows the sender to both declare and define common 
properties of Data Sets based on it. The basic format of a Data Template Set 
is the same as that of a Template Set and - for the sake of completeness - is 
shown in Figure A. The format of individual Template definitions, however, 
differs from that of the standard Template and is shown in Figure B.
    
       0                   1                   2                   3  
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1  
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
      |          Set ID = 4           |          Length               |  
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
      |                                                               |  
      |                        Data Template 1                        |  
      |                                                               |  
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
      |                                                               |  
      |                              ...                              |  
      |                                                               |  
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
      |                                                               |  
      |                        Data Template N                        |  
      |                                                               |  
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
      |                          Padding (opt)                        | 
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
    
      Figure A: Template Set Format 

   The Data Template Set field definitions are as follows: 
    
     Set ID 
           A Set ID value of 4 might be used for the Data Template Set. 
 
     Length 
           Total length of this Set in bytes. 

     Padding  
           See the description of a Type 3 Template Set in [PROTO]

       0                   1                   2                   3  
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1  
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
      |  Template ID                  |  Field Count  |  Data Count   |  
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
      |                                                               |  
      |                         Field 1 Type                          |  
      |                                                               |  
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
      |                                                               |  
      |                              ...                              |  
      |                                                               |  
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
      |                                                               |  
      |                         Field N Type                          |  
      |                                                               |  
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
      |                                                               |  
      |                          Data 1 Type                          |  
      |                                                               |  
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
      |                                                               |  
      |                          Data 1 Value                         |  
      |                                                               |  
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
      |                                                               |  
      |                              ...                              |  
      |                                                               |  
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
      |                                                               |  
      |                          Data M Type                          |  
      |                                                               |  
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
      |                                                               |  
      |                          Data M Value                         |  
      |                                                               |  
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
    
      Figure B: Data Template Format
    
   The Data Template field definitions are as follows: 
            
     Template ID 
           See the description of a Type 3 Template Set in [PROTO]
            
     Field Count 
           Number of regular Fields that will be sent in subsequent
           Data Sets using this Template.
            
     Data Count 
           Number of fixed-value Fields that will be sent in this
           Template.

     Field N Type 
           Defined according to Section 7.2 in [PROTO]: Type identifier, 
           Type length and an enterprise number, if needed.

     Data M Type 
           Same as "Field N Type", but used for common properties of all
           Data Sets that use this Template.

     Data M Value 
           Bit representation of a common property as would be 
           transmitted in a Data Set.

Given the example rules and flows from above our concentrator would now use 
one Template, say 257, to send all flows that matched no given rule and were 
thus not aggregatable:

Src_IP          Dst_IP          Src_Port        Dest_Port       Pckts
-----------------------------------------------------------------(Set ID 257)--
192.168.2.204   194.95.211.13   64238           80              10
192.168.2.205   194.95.211.14   64239           80              10

The second Data Set uses a Data Template, say 258, to send all flows that 
matched our aggregation rule.

Dest_Port       Pckts           (Src_IP=192.168.0.0/23)
-----------------------------------------------------------------(Set ID 258)--
80              20
110             10

2 Data Types

2.1 Networks

2.1.1 Different Data Types for IP and Netmask

This is the traditional way of representing a network: IP address and netmask 
are transmitted as two fields:

Src_IP          Src_Mask        Dst_IP          Dst_Mask        Pckts
-------------------------------------------------------------------------------
192.168.0.0     23              0.0.0.0         0               10

2.1.2 IP Networks Data Type

This is an alternative way of representing networks: Both IP address and 
netmask are transmitted in one field with a Networks Data Type.

Networks may be casted down to part of an IP address. 
It might also be used as a variable-length data type as defined in Chapter 11 
of [PROTO], thus representing a list of networks.

Examples for Networks:

Network	Length	Binary_Representation
-------------------------------------------------------------------------------
192.168.0.0/24  3       0xC0A000
192.168.0.1     4       0xC0A00001
192.168.0.0/23  5       0xC0A00000 0x17
192.168.0.{1|3} 10      0xC0A00001 0x20 0xC0A00003 0x20

2.2 Port Ranges

As the current draft contains no data type to transmit portlists or portranges,
this section contains two alternatives for a new data type.

2.2.1 PortRanges Data Type 

This Data Type represents a list of port ranges. Its binary representation
will be an arbitrary number of 4-byte sequences, each consisting of 
a 2-byte start port and a 2-byte end port specification.

PortRanges may be casted down to a single Port or might also be used as a 
variable-length data type as defined in Chapter 11 of [PROTO].

Examples for PortRanges:

Ports           Length  Binary_Representation
-------------------------------------------------------------------------------
80              2       0x0050
1:7             4       0x0001 0x0007
80,  443        8       0x0050 0x0050 0x01BB 0x01BB
1:7, 256:1024   8       0x0001 0x0007 0x0100 0x0400
20,  80, 443    12      0x0014 0x0014 0x0050 0x0050 0x01BB 0x01BB
1:7, 80, 443    12      0x0001 0x0007 0x0050 0x0050 0x01BB 0x01BB

2.2.2 Ports Data Type

The interpretation of this Data Type will depend on its length. It can either 
represent a list of 2-byte port numbers or a list of 4-byte port ranges. 
In order to tell one representation from the other the plain port list will
have an additional zero byte appended.

Ports           Length  Binary_Representation
-------------------------------------------------------------------------------
80              2       0x0050 0x00
80,443          4       0x0050 0x01BB 0x00
1:7             5       0x0001 0x0007
20,80,443       6       0x0014 0x0050 0x01BB 0x00
1:7, 256:1024   9       0x0001 0x0007 0x0100 0x0400
1:7, 80, 443    12      0x0001 0x0007 0x0050 0x0050 0x01BB 0x01BB

3 Thoughts about the different approaches

3.1 Communicating common properties

Approach 1.1 (Complete Data Set) seems least suitable for transmitting 
common properties, both because of the enormous overhead of sending
the same constant values over and over again and because of the need for
a dedicated "any" value for fields like Protocol or Flags.

Aproach 1.2.1 (Implicit Forward Association) appears more promising 
but leaves the receiver in doubt as to whether a Template has any fixed 
values associated with it, long as the Options Data Record was not
received yet and - in case of packet loss over SCTP or UDP - might even 
leave the receiver assuming that there just aren't any fixed values when 
in fact only the Options Data Set got lost.

Approach 1.2.2 (Explicit Association) tries to fill this hole by
embedding a pointer to the Options Data Set in each Data Record,
but thus creates probably unnecessary overhead.

Approach 1.3 (Introduction of a new Template Set) arguably uses the least
overhead to transmit constant values, but at a price: Data Sets basing
on a Data Template Set are illegible for clients implementing older
versions of the IPFIX draft. One, however, has to keep in mind that all
other approaches will also be at most partly understood by clients unaware
of the possibility of aggregated flows.

3.2 Data Types

Approach 2.1.1 (Different Data Types for IP and Netmask) represents
the traditional way of transmitting networks. It can, however,
only represent one single network.

Approach 2.1.2 (IP Networks Data Type) bundles IP and Netmask into
one field and may represent lists of Networks as well as single IPs.
A con of this Data Type is that it is illegible for older clients. These
clients are, however, unaware of concentrated flows and might such 
even produce wrong statistics when fed seperate data fields for IP and
Netmask.

For Ports, other that for networks, a new data type seems to be definitely
needed. Approaches 2.2.1 and 2.2.2 differ only in that the latter may also
represent a port list rather than a list of port ranges, making better
use of space - be it in a Flow Record, in an Option Data Record or in a 
Template. Approach 2.2.2 is, however, somewhat more complex to interpret
and might internally be treated like a list of port ranges anyway.