Subject: [ipfix-arch] Re: IPFIX Architecture

From: Ganesh Sadasivan <[email-masked]>
Date: Wed, 06 Feb 2002 10:32:20 -0800

Hi Kevin,
   I have some comments on the arch spec portion.
   There are between 2 "G:"s.
Thanks
Ganesh

1  Scope 
    
  This memo describes the architecture of an IP Flow Information eXport 
  (IPFIX) system. The IPFIX architecture provides a high level view of 
  key functions, interfaces, and protocols employed by the system, and 
  serves as a guide for design, engineering, and deployment of such 
  systems.   
   
  The IPFIX architecture consists of  
   
  -  A reference model that depicts the key IPFIX entities and 
  interfaces. 
   
  -  Functional descriptions of IPFIX entities. 
   
  -  Detailed descriptions of capabilities in support of IPFIX 
  requirements.  
   
  -  Detailed descriptions of the relationship among IPFIX entities, 
  and definition of the IP flow information export interface that meets 
  the key requirements such as reliable, flexible, and efficient.   
   
  -  An IPFIX protocol architecture that present relationship among 
  various layers, and their communication requirements.   
   
1.1 Specification of Requirements 
 
  In this document, the keywords "MUST", "MUST NOT", "SHOULD", "SHOULD 
  NOT", and "MAY" are to be interpreted as described in BCP 14. These 
  keywords are not case sensitive in this document.    
   
2  Terminology 
   
  IPFIX Session 
   
      An IPFIX Session is a logical connection between an exporter 
      entity and one or multiple collector entity for the purpose of 
      exporting IP flow information. Multiple sessions MAY be 
      maintained concurrently in an exporting device.   
   
  Template 
   
      A Template defines the structure of any types of IP flow 
      information record, and specifies the data type, meaning, and 
      location of the fields in the record.  

3  IPFIX Reference Model 
   
  The following figure illustrates the reference model of an IPFIX 
  system. It identifies key IPFIX entities and interfaces within the 
  system.  
     
                                                                     3 

    +---------------+          +---------------+      
    | +----------+  |          |  +----------+ | 
    | |Meter     |  |          |  |Processor | | 
    | |Entity    |  |          |  |Entity    | | 
    | +----------+  |Exporting |  +----------+ |   
    | +----------+  |Interface |  +----------+ | 
    | |Exporter  |<=|==========|=>|Collector | | 
    | |Entity    |  |          |  |Entity    | | 
    | +----------+  |          |  +----------+ | 
    | Exporting     |          |  Collection   | 
    | Device        |          |  Device       |          
    +---------------+          +---------------+ 
   
  An exporting device is typically a network element (e.g. routers, 
  measurement probes); a collection device may be part of mediation 
  systems, accounting/billing systems, and network management systems 
  to facilitate business and operation support.   
   
3.1 IPFIX Entities 
    
1.1.1   Exporter 
   
  The primary task performed by the Exporter entity is to reliably, 
  timely, and efficiently deliver IP flow information to the collector 
  entity. The following functions are responsibilities of the exporter 
  entity: 
G:
what does it timely mean?
We can compute the UTC time the flow started and ended.   
G:

  -  Participating in a capability negotiation process with the 
  collector entity in order to determine a commonly supported IPFIX 
  exporting mechanism (e.g. IPFIX protocol, and its versions) 

G: 
   Is the intention to make "capability negotiation" a MUST?
   IMHO  the collector should accept whatever is 
   being exporterd by the exporter. In most of the cases (atleast 
   from a router point of view)  the collector is only an 
   intermediate box between the exporter and the final node that
   interprets the export information. The job of the collector
   from an information point of view would just be to grab all
   that the exporter sends.
   From an export protocol perspective, maybe it would make sense to
   do some kind of negotiation. But here also with so much of 
   infrastructure existing already for reliability, security
   etc the question is whether we want to come with a new model
   or just concentrate on flow export.IMO there is no need for
   negotiation.
G: 

  -  Participating in control information exchanges with the collector 
  entity.  The control information exchanges typically occur during the 
  initial establishment of the exporting session, and may include 
  procedures such as exporting session setup, security support, 
  capability negotiation, protocol negotiation, IP flow information 
  structure (template) negotiation, etc. 

G:
The underlying transport or other protocol is something which can be
configured by the sys. admin., whatever the way he wants: 
CLI, SNMP, telnet, ...
G:
   
  -  Establishing, maintaining, and removing the state associated with 
  an exporting session.   
     
  -  Generating, transmitting, and receiving IPFIX messages. The IPFIX 
  messages may include IPFIX control messages and IPFIX user messages. 
   
  -  Inserting IP flow information into IPFIX data messages according 
  to an encoding scheme. To enable efficient data export, a template 
  based encoding should be supported. 

G: 
   Sending exporter configuration is one of the most important control
   information.
G: 
   
  -  Segmenting IP Flow information into multiple IPFIX user messages, 
  attaching required control information such as sequence numbers, 
  error checks, etc. 
   
  -  Releasing the transmitted and acknowledged IP flow information 
  from the memory. 
G: 
   This is an implementation detail. Should be taken care of by the
   transport.
G: 
   
  -  Supporting fail-over and fail-back procedures, and performing load 
  balancing if required.   
G:
Fail-back is not possible on a router -> too much memory needed.
The best we can do is fail-over.
G:

1.1.2   Collector 
   
  The primary task performed by the Collector entity is to receive and 
  acknowledge the IP flow information received from the exporter 
  entity. The following functions are responsibilities of the collector 
  entity: 
   
  -  Participating in the capability negotiation and control 
  information exchanges with the Exporter entity.   
G:
   Same concerns on capability negotiation as above.
G: 
   
  -  Establishing, maintaining, and removing the state associated with 
  an exporting session.   
     
  -  Generating, transmitting, and receiving IPFIX messages.    
   
  -  Receiving IPFIX user messages according to a decoding scheme, 
  checking message integrity, and acknowledging it after it is 
  correctly received or optionally processed and persisted in storage. 
G: 
I feel this should be taken care of by the lower layers.
G: 
   
  -  Re-sequencing the received IP flow information if required. In 
  case a reliable, in-sequence delivery service is provided by lower 
  layer protocol, this function can be omitted. 
G:
If we have the timestamp per flow why is resequencing required
G:

  -  Supporting fail-over and fail-back procedures. 

G: 
This should be an exporter feature and not a collector feature.
Are you suggesting an inter collector communication?
IMO this out of scope for this doc.
G: 
          
1.1.3   Meter 
   
  The primary task performed by the meter entity is to monitor IP 
  flows, and derive IP flow information. How the meter entity measures 
  and derives flow information is outside the scope of the IPFIX 
  architecture, but the attributes used to describe flow information 
  must comply with the IPFIX information model and IP flow definitions.  

G: 
The meter entity should interact with the exporter to export information
as to how the metering is done.
G: 
       
1.1.4   Processor 
   
  The primary task performed by the Processor entity is to process the 
  received IP flow information based upon end application.s 
  requirements. The functionality of the processor is outside the scope 
  of the IPFIX architecture, though the following functions should be 
  considered in designing and deploying an IPFIX system. 

G: 
   At the minimum the processor should be able to understand the
   structure of the exported information.
G: 
    
  -  IP flow information aggregation based upon some aspects of IP 
  flows. 
   
  -  IP flow information filtering based upon some aspects of IP flows.  

                                                                     5 

  -  Query additional information (e.g. address translation, user 
  identification, etc.) related to an IP flow from other systems, to 
  facilitate further processing. 
   
  -  Persistent receive IP flow information   
   
3.2 IPFIX Exporting Interface 
     
  The IPFIX Exporting interface connects an exporter entity with one or 
  multiple collector entities.  It MUST support the Internet Protocol 
  (IP) for data communications.  

G: 
I guess V4 is a MUST, V6, MPLS is optional.
G: 
   
4  IPFIX Deployment Scenarios 
    
  IPFIX exporting devices may vary greatly in their core functionality.  
  At one hand, a core router needs to support high-speed interfaces and 
  traffic throughput; it emphasizes on forwarding IP traffic quickly 
  and may not be able to perform extensive measurement on IP flow. It 
  is also likely to export high volume of IP flow information to 
  external systems.  On the other hand, a measurement probe or an 
  access device is capable of providing more detailed information about 
  IP flows and performing complex on-device processing. Consequently, 
  it may only export low volume of IP flow information. 
   
  Given the diversity of exporting devices, two deployment scenarios 
  are described. These scenarios only serve as examples for IPFIX 
  system design and deployment.  
   
4.1 Exporting Interface over LAN 
   
  The following figure illustrates the scenarios where an exporting 
  device connects to the collecting devices over LAN. 

     +----------+              +----------+    +------------+   
     |Exporting |              |Collection|<==>|            | 
     |Device    |              |Device 1  |    |Applications| 
     +----------+              +----------+    |e.g.        | 
         ^ |                       ^ |         |usage       | 
         | v                       | v         |accounting, | 
     ---------------------------------------   |traffic     | 
                   LAN                         |profiling,  |   
     ---------------------------------------   |traffic     | 
                   ^ |                         |engineering,|     
                   | v                         |QoS         | 
               +----------+                    |Monitoring  | 
               |Collection|<==================>|etc.        | 
               |Device 2  |                    |            | 
               +----------+                    +------------+ 
    
  This scenario is suitable for collecting information from core 
  routers. As the volume of exported data is typically high, LAN can 

                                                                     6 

  provide adequate bandwidth and introduce small latency to meet data 
  exporting requirement. 

G: 
   It is also important that the exporting overheads are minimized
   to handle the high volume. Features like reliability
G: 
   
4.2 Exporting Interface over WAN 
   
  The following figure illustrates the scenario where an exporting 
  device connects to the collecting devices over WAN. 
    
                       +----------+     
                       |Collection|            +------------+ 
                       |Device 1  |<==========>|            | 
                       +----------+            |Applications| 
                          ^                    |e.g.        | 
                          |                    |usage       | 
  +----------+         +--|-------+            |accounting, |   
  |Exporting |<--------|---       |            |traffic     | 
  |Device    |<--------|--------- |            |profiling,  | 
  +----------+         |   WAN  | |            |traffic     |    
                       |        | |            |engineering,| 
                       |        | |            |QoS         | 
                       +--------|-+            |Monitoring  | 
                                |              |etc.        |   
                                v              |            | 
                       +----------+            |            | 
                       |Collection|<==========>|            | 
                       |Device 2  |            +------------+ 
                       +----------+ 
   
  This scenario maybe used to collect information from probes or access 
  routers.  It is critical that a congestion aware transport protocol 
  (e.g. TCP, SCTP) is used. 
      
5  Protocol Architecture 
   
  The following figure illustrates the protocol architecture in 
  supporting of the IPFIX system. 

                                                                     7 

      Exporting Device               Collecting Device   
      +----------------+             +----------------+ 
      |    IPFIX       |             |     IPFIX      | 
      |    User        |             |     User       | 
      +----------------+             +----------------+ 
      |    IPFIX       |             |     IPFIX      | 
      |    Layer       |             |     Layer      | 
      +----------------+             +----------------+ 
      |  Transport     |             | Transport      | 
      |  Layer         |             | Layer          | 
      +----------------+-------------+----------------+ 
      |                   IP Layer                    | 
      +-----------------------------------------------+  
       
  The IFPIX protocol is designed an application running over a 
  transport layer protocol. The transport layer protocol is responsible 
  for delivering IPFIX messages between exporters and collectors.  
    
  As the IPFIX system MUST be reliable, a reliable transport layer 
  protocol (e.g. TCP, SCTP) should be used.  Furthermore, IPFIX layer 

G:
   If the reliability is guarenteed by the network, is there a necessity
   for a reliable transport? As we explained in IETF-52, there are 
   scalability concerns with TCP and making this mandartory would
   exclude the exporters with high volume from being IPFIX compliant.
G: 

  acknowledgement and flow control mechanisms should be supported to 
  ensure flow information is not only correctly received, but also 
  processed and persistent at the collection device.      

G:
   Why all these are mandatory for IPFIX layer. I agree the framework
   should take care of this. But if there is a capability already
   available why re-invent?
G: 
    
6  IPFIX Information Model 
   
  The IPFIX information model is listed in the IPFIX requirement 
  document.  The information model consists of the minimum set of 
  attributes that an IPFIX exporting device should be able to export.  
  In conjunction with IP flow definitions, they provide locally 
  accurate information about a flow.   
   
  To meet application.s requirement may require the collection device 
  to obtain additional information about the flow, e.g. address 
  translation, user identification, etc. Additional flow information 
  may also be required; in this case, equipment vendors may define 
  extensions to the IPFIX information model.  Any extension to the 
  IPFIX information model should be conveyed between end points. 
   
  This section presents a rigorous definition and sufficient statistics 
  for these attributes.  
G:
   I guess this should be maintained in the requirements doc and
   not duplicated. There are lots of comments on this section.
G: 
   
6.1 Attributes related to IP Packet Header 
   
  The following attributes are obtained directly from IP packet header 
  belonging to a flow:  
G: 
What is meant by flow in the above sentence?
G: 

  -  IP Version Number 
G: 
Why? Can't this be derived from address itself?
G: 

  -  Source Address 
  -  Destination Address 
G: 
I suggest we keep separate field types for V4 and V6.
G: 
   There would be more like inner IP, outer IP etc within v4 and v6.
   It is better each of such attribute is enumerated.

  -  Protocol Type 
  -  TOS (for version 4) 
G: 
Why not for V6?
G: 

  -  Flow Label (for version 6) 
  -  DSCP (if DiffServ is supported) 
G: 
This is an incomplete list. What about L4 headers?
G: 
     
                                                                     8 

6.2 Attributes Related to Measurement 
   
  The following attributes relates to measurements and  
  measuring methodology:  
  -  Packet Counter 
  -  Dropped Packet Counter 
  -  Payload Byte Counter 
  -  Timestamp of the First Packet Observed 
  -  Timestamp of the Last Packet Observed 
  -  Timeout Indication 
  -  Sampling Method 
  -  Sampling Parameters 
   
6.3 Attributes Related to Flow Definition 
   
  The following attributes relates to IP flow definitions.
    
  -  Incoming Interface 
  -  Outgoing Interface 
  -  Packet Treatment 
  -  Unique ID of the Observation Point 
  -  Unique ID of the Measuring Device 
    
6.4 Attributes related to Upper Layers 
   
  The following attributes provides information of upper
  layer protocols.  
  -  Source Port Address 
  -  MPLS Label (if MPLS is supported) 

7  IPFIX Message Format 
    
       0                   1                   2                   3 
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
      |  Version      |Message ID(MID)|  Session ID   | Message Flags | 
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
      |                         Message Length                        | 
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
      |                                                               | 
      ~                         Message Payload                       ~ 
      |                                                               | 
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
    
G: 
   The message should have the follwoing
   - sequence number if the collector wishes to sequence the packets.
   - the absolute time at which the message was sent and sysUpTime.
   - an ID to identify the exporter if the collector receives packets
     from multiple exporters. Why does one need a session ID? 
G: 

      Version: 8 bit unsigned integer 
   
         The Version field indicates the IPFIX protocol implementation.  
          
      Message ID (MID): 8 bit unsigned integer 
   
         The Message ID field identifies the type of the message.  

                                                                     9 

      Session ID: 8 bit unsigned char 
       
         The Session ID field identifies the IPFIX exporting session 
         with which the message is associated.  
       
      Message Flags: 8 bit unsigned char 
   
         The Message Flags field can be used to identify options 
         associated with the message.  
   
      Message Length: 32 bit unsigned integer 
                        
         The Message Length field is the total length of the IPFIX 
         message in octet including the header. 
          
      Message Payload: Variable Length 
                        
         It contains the control or user information exchanged between 
         IPFIX end-points (exporter and collector entities). 
   
8  IPFIX Messages  
   
   To be added 
    
9  IPFIX Template 
    
   The IPFIX system MUST support efficient exporting of IP flow 
   information.  This can be achieved by negotiating a set of IPFIX 
   Templates for an IPFIX session before actual IP flow information is 
   exported. 

G: 
   Templates can be exchanged at any time. This may be done
   as a result of a config change in the exporter or it may be a
   periodic update.
G: 

   A template defines the structure of an IPFIX user message 
   payload by describing the data type, meaning, and location of the 
   fields in the payload. By agreeing on IPFIX templates, IPFIX end-
   points understand how to process IPFIX user messages. As a result, 
   an exporting device only needs to export actual flow information 
   without attaching any descriptors of the attributes; this reduces 
   the amount of bytes sent over communication links. 

G:
   I do not understand, how the collector can interpret pure data 
   records without any descriptors as to what template it represents.
G: 
    
   A template is an ordered list of keys. A key specifies an attribute 
   that a meter entity MAY export. The specification MUST consist of 
   the description and the data type of the attribute. (e.g. 'Number of 
   Sent Bytes'  can be a key that is an 32 bit unsigned integer) An 
   exporting device typically defines keys. Based on the IPFIX 
   information model, a set of IPFIX standard keys can be defined.  
    
9.1 Template Negotiation 
    
   During the initial setup of an IPFIX session, template negotiation 
   procedures should be carried out.  It would allow all the IPFIX end-
   points to synchronize on a set of templates that will be used during 
   IP flow information export.  

G: 
As I said above I really do not agree to this model.
G:

G:
In short the fundamental aspects of disagreement are:
- no template/capabilities/etc... negotiation in IPFIX
- no MUST for reliable
- no state information (except based on the transport protocol, if
connection oriented)
- no connection initiated from the collector.
G:
    
10 Capability Negotiation 

                                                                    10 

   To be added 
    
11 Redundancy and Error Handling 
    
   To be added 
    
12 Author's Address 
   
  Questions about this memo can be directed to: 
   
  Kevin Zhang  
  XACCT Technologies, Inc. 
  2900 Lakeside Drive 
  Santa Clara, CA 95054 
  Phone +1 301 992 4697 
  Email: [email-masked]

---

From: [email-masked]
Date: Fri, 08 Feb 2002 10:58:00 -0500

>The best we can do is fail-over.

	Not sure why. I think this is as simple as detecting
	the server is back and droping the existing connection
	in favor of the other.

>IMO this out of scope for this doc.

	Collector to Collector communication is not
	in scope. Exporter to Collector is in scope
	and IMHO we need to define how and when to move
	between Collectors on a failure. 

>   exclude the exporters with high volume from being IPFIX compliant.

	Transport level delivery is not the same as application
	level delivery. At minimum, a top of discussion for IPFIX
	is whether or not we are going to provide application level
	reliability in the protocol as an option.

	The reason you may want this is that only the collector
	can acknowledge when it has actually processed the
	information. For example, it may send an acknowldge when
	it has actually persisted the data.

Ganesh Sadasivan wrote:
>   ------------------------------------------------------------------------
> 
> 
> 1  Scope
> 

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Ganesh Sadasivan <[email-masked]>
Date: Fri, 08 Feb 2002 09:02:12 -0800

Hi Paul,
See inline.

>         the server is back and droping the existing connection

Please ignore the above statement. That was a mistake.

>         between Collectors on a failure.

Agreed. It is specified to some extent in the draft we published.
You can look at it and comment on it.

>         it has actually persisted the data.

As long as this is optional & *not* negotiated, I agree to this.
Thanks
Ganesh