Subject: [ipfix-protocol] common Template format & absolute/delta timestamp

From: "STEPHAN Emile RD-CORE-LAN" <[email-masked]>
Date: Wed, 30 Mar 2005 18:14:55 +0200

Hi

Following is a proposal of a common format for Option Template and
Template which offers 'Flow Data Set' with the capability to carry
absolute time reference and delta timestamps. 

This proposal is fully compatible the semantic and the encoding of
existing templates and data records. An Option Template is defined using
the regular Template format: field types are systematically aligned on
32 bits, so enterprise bit detection is much more easy and fast.

Common format for Option Template and Template:
-----------------------------------------------

Option Template format differs from Template format because of the
presence of the field 'Scope Field Count' in the Option Template format.
So an Option Template may be defined using the Template format if we
define 'Scope Field Count' as a Field Type in the info model. 
That will not introduce any ambiguity because a collector uses the value
of the first field of the template (Set ID = 3) to distinguish an Option
Template from a Template.

Example:

We define the field type 'ScopeFieldNumbers' (e.g. identifier 1034) in
the info model.

The example of the page 58 "16.4.3 Options Template Set using an
Enterprise Specific scope" of protocol-09.txt becomes:
    0                   1                   2                   3  
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1  
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
   |         Set ID = 3            |          Length = 28          |  
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
   |       Template ID 257         |        Field Count = 4        |  
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
   |     ScopeFieldNumbers = 1034  |              1                |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
   |1|    Scope 1 Field Type = 123 |       Enterprise Number     ...
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
   ...                             |    Scope 1 Field Length = 4   |  
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
   |     exportedPacketCount = 41  |       Field Length = 4        |  
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
   |     exportedFlowCount = 42    |       Field Length = 4        |  
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
    
 The format of the Options Data Records do not change  
    
   In this example, we report the following two Options Data Records: 
   Line Card ID             | IPFIX Message   | Exported Flow Records 
   ------------------------------------------------------------------ 
   Line Card 1 (lineCardId=1) | 345             | 10201     
   Line Card 2 (lineCardId=2) | 690             | 20402 
    
   0                   1                   2                   3  
   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1  
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
  |      Set ID = 257             |         Length = 20           |  
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
  |                               1                               | 
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
  |             345               |            10201              | 
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
  |                               2                               |  
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
  |             690               |            20402              |  
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 

Template Set offering absolute time reference and delta timestamp:
-------------------------------------------------------------------

Discussion:
We must preserve the separation of the exporter/sourceID defined in the
architecture: 'Export Time' is the time the message leaves the exporter
and consequently 'Export Time' is set by the exporter which uses its own
reference of time; Data record timestamps, if any, are set by the
'Source ID' which uses its own reference of time.

Allowing the exporter to choose different references of time for 'Export
Time' will lead to many interoperability issues because the message
header doesn't describe which reference of time is used and consequently
collectors will not be able to discover the encoding of the reference of
time. Are we ready to accept to add a new field in the header of the
message?

Allowing the exporter to compute delta timestamp will mix the meter
clock and the exporter clock. That will corrupt timestamps accuracy
because such difference of time will mix the 2 clocks synchronisation
skews and siblings. The consequence is that the best accuracy of the
flow record will be the accuracy of the exporter process even if the
meter accuracy is much more precise. 

I really agree with the motivation and the need: it is a MUST. But we
MUST keep separated the semantic of the message header from the semantic
of templates, moreover from the semantic of individual Field Type. I am
convinced that the only way to address this issue is to provide the
'Observation Domain' with a mechanism to mix absolute times reference
and delta time references in a 'Flow Data Records'. 

Following is a proposal which reuse the ScopeFieldNumbers defined above:

Using the ScopeFieldNumbers, a regular template informs the collector of
the number of fields which are only present in the first first record of
the 'Flow Data Records'.

Example:

We (re)define the field type 'dateTimeMicroSeconds' (e.g. identifier
1052) in the info model.
Using the ScopeFieldNumbers, a regular template informs the collector
that the absolute time dateTimeMicroSeconds is only present in the first
field of the first record of the 'Flow Data Records'.

    0                   1                   2                   3  
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1  
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
   |         Set ID = 0            |          Length = 32          |  
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
   |       Template ID 256         |        Field Count = 6        |  
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
   |     ScopeFieldNumbers = 1034  |              1                |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
   |  dateTimeMicroSeconds = 1052  |       Field Length = 8        |  
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
   |  156(flowStartDeltaUSecondst) |       Field Length = 4        |  
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
   |  157 (flowEndDeltaUSeconds)   |       Field Length = 4        |  
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+    
   |    8 (sourceIPv4Address)      |       Field Length = 4        |  
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
   |  12 (destinationIPv4Address)  |       Field Length = 4        |  
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+    

Following is a flow data set. It contains 3 records. The first one has 5
fields. Others have 4 fields.

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
|          Set ID = 256         |          Length = 60          |  
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+    
|                       4575688568445                           |
absolute
+                                                               + micro
|                       7875224045965                           |
seconds
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
|                            5000                               | delta
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ micro
|                            5500                               | second
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                       192.181.17.7                            |  
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
|                       192.181.17.18                           |  
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
|                            6000                               | delta
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ micro
|                            8000                               | second
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
|                       192.181.17.17                           |  
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
|                       192.181.17.19                           | 
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
|                            6010                               | delta
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ micro
|                            7000                               | second
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                       192.181.17.25                           |  
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 
|                       192.181.17.28                           |  
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Best Regards
Emile

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Benoit Claise <[email-masked]>
Date: Thu, 31 Mar 2005 12:23:18 +0200

Emile,

>Common format for Option Template and Template:
If you have different time on the Observation Domain (source ID), and 
the Exporting Process, a solution is to use the absolute timestamps in 
the data records.

>field of the first record of the 'Flow Data Records'.
I'm not in favor of this approach as you now have dependencies between 
Information Elements.
I mean that the meaning of the I.E. Y will depend on the meaning of I.E. X.
If the collector doesn't have the I.E. X in his information model, you 
can't conclude anything about Y
If the application that uses the data records doesn't have the 
intelligence that there is a relation between those 2 I.E., again this 
is a problem

Regards, Benoit.