Subject: [ipfix-protocol] Re: common Template format & absolute/delta timestamp

From: Benoit Claise <[email-masked]>
Date: Sat, 02 Apr 2005 16:28:28 +0200

Hi Emile,

>Hi Benoit
>
>
>You said "
>  
>
>>If you have different time on the Observation Domain (source ID), and
>>the Exporting Process, a solution is to use the absolute timestamps in
>>the data records.
>>
>>    
>>
>"
>
>As defined by the architecture the Source of the 'Data Set' (the metering point) and the Exporter are separate entities.
>
They could.

> So in most of the cases they have different sources of time (especially if we take in account aggregation points). So, no way, timestamping is a central information element for flows and measurement, and delta timestamping is crucial to save bandwidth.
>  
>
 Flow Record 
 
   A Flow Record contains information about a specific Flow _that was 
   observed at an Observation Point._  ...

The approach in the draft was: this is also for the case for the time: we take the time at the observation point.

Now, 3 different situations:
1. synchronized time for both the Observation Domain and the Exporter
   -> you could use the absolute time Information Elements in the data records
      if you want to save some CPU
   -> you could use the delta time Information Elements in the data records
      if you want to save some bandwidth
2. non synchronized time for both the Observation Domain and the Exporter,
   but very close time (for example: a line card has got 1 sec. extra compared
   to the route processor because it booted a little lit later
   -> you could use the absolute time Information Elements in the data records
      if you want to save some CPU
   -> you could use the delta time Information Elements in the data records
      if you want to save some bandwidth
      In rare condition, you might have a problem if the flow starts just at 
      the 71 minutes boundary.
3. non synchronized time for both the Observation Domain and the Exporter,
   but no very close (for example: longer that 71 minutes). 
   -> you must use the absolute time Information Elements in the data records
   Note: this implies that a mediation device must use the absolute time
         if it doesn't forward directly the data records.

>
>You said "
>  
>
>>I'm not in favor of this approach as you now have dependencies between
>>Information Elements.
>>I mean that the meaning of the I.E. Y will depend on the meaning of I.E. X.
>>If the collector doesn't have the I.E. X in his information model, you
>>can't conclude anything about Y
>>If the application that uses the data records doesn't have the
>>intelligence that there is a relation between those 2 I.E., again this
>>is a problem
>>    
>>
>"
>
>The current delta timestamping introduces much more new dependencies that the one I propose:
>
>Currently values of deltatimestamping IEs are modified by the exporter according to a value of a field of the header. It introduces dependencies between information blocks separated by 3 different levels ('Message header'.'Data flow sets'.'Data flow set'.'IE value') and managed by different entities. 
>  
>
Currently, there is a dependency on the Message Header. All IPFIX 
collecting process MUST anyway be able to understand it to decode the 
data records.
With your proposal, there is a dependency between I.E. I think this is 
worse.

>The current delta timestamping is a new job and a new task for the Exporter. The exporter must be able to identify IEs to re-comptute (So he must know any IE used by the meter: is it its job? what happen is a new one is used by the metering entity?); Moreover as the exporter must re-compute any timestamps there is a risk to overload the exporter (at large computing take time so when the last one will be computed the "export time" will have changed...);
>  
>
Again, if you want to save some CPU on the network element, use the 
absolute timestamps.

>		
>My proposal respects the separation between metering entity and Exporter. IE values are not modified and the dependency is managed by the metering entity inside each 'Data Set'. The Exporter is not impacted. The collector received 'Data Set' including the reference of time of the delta timestamp IEs it carries. 
>  
>
We can achieve that right now with absolute time I.E.

Regards, Benoit.

>The dependency comes only from the delta timestamp requirement we have. 
>
>Regards
>Emile
>
>
>  
>
>>-----Message d'origine-----
>>De : Benoit Claise [mailto:[email-masked]]
>>Envoyé : jeudi 31 mars 2005 12:23
>>À : STEPHAN Emile RD-CORE-LAN
>>Cc : Sebastian Zander; Nevil Brownlee; [email-masked];
>>Juergen Quittek
>>Objet : Re: common Template format & absolute/delta timestamp
>>
>>Emile,
>>
>>    
>>
>>>Hi
>>>
>>>Following is a proposal of a common format for Option Template and
>>>Template which offers 'Flow Data Set' with the capability to carry
>>>absolute time reference and delta timestamps.
>>>
>>>This proposal is fully compatible the semantic and the encoding of
>>>existing templates and data records. An Option Template is defined using
>>>the regular Template format: field types are systematically aligned on
>>>32 bits, so enterprise bit detection is much more easy and fast.
>>>
>>>
>>>Common format for Option Template and Template:
>>>-----------------------------------------------
>>>
>>>Option Template format differs from Template format because of the
>>>presence of the field 'Scope Field Count' in the Option Template format.
>>>So an Option Template may be defined using the Template format if we
>>>define 'Scope Field Count' as a Field Type in the info model.
>>>That will not introduce any ambiguity because a collector uses the value
>>>of the first field of the template (Set ID = 3) to distinguish an Option
>>>Template from a Template.
>>>
>>>Example:
>>>
>>>We define the field type 'ScopeFieldNumbers' (e.g. identifier 1034) in
>>>the info model.
>>>
>>>The example of the page 58 "16.4.3 Options Template Set using an
>>>Enterprise Specific scope" of protocol-09.txt becomes:
>>>   0                   1                   2                   3
>>>   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
>>>  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>>  |         Set ID = 3            |          Length = 28          |
>>>  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>>  |       Template ID 257         |        Field Count = 4        |
>>>  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>>  |     ScopeFieldNumbers = 1034  |              1                |
>>>  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>>  |1|    Scope 1 Field Type = 123 |       Enterprise Number     ...
>>>  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>>  ...                             |    Scope 1 Field Length = 4   |
>>>  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>>  |     exportedPacketCount = 41  |       Field Length = 4        |
>>>  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>>  |     exportedFlowCount = 42    |       Field Length = 4        |
>>>  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>>
>>>The format of the Options Data Records do not change
>>>
>>>  In this example, we report the following two Options Data Records:
>>>  Line Card ID             | IPFIX Message   | Exported Flow Records
>>>  ------------------------------------------------------------------
>>>  Line Card 1 (lineCardId=1) | 345             | 10201
>>>  Line Card 2 (lineCardId=2) | 690             | 20402
>>>
>>>  0                   1                   2                   3
>>>  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
>>> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>> |      Set ID = 257             |         Length = 20           |
>>> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>> |                               1                               |
>>> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>> |             345               |            10201              |
>>> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>> |                               2                               |
>>> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>> |             690               |            20402              |
>>> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>>
>>>
>>>
>>>Template Set offering absolute time reference and delta timestamp:
>>>-------------------------------------------------------------------
>>>
>>>Discussion:
>>>We must preserve the separation of the exporter/sourceID defined in the
>>>architecture: 'Export Time' is the time the message leaves the exporter
>>>and consequently 'Export Time' is set by the exporter which uses its own
>>>reference of time; Data record timestamps, if any, are set by the
>>>'Source ID' which uses its own reference of time.
>>>
>>>
>>>      
>>>
>>If you have different time on the Observation Domain (source ID), and
>>the Exporting Process, a solution is to use the absolute timestamps in
>>the data records.
>>
>>    
>>
>>>Allowing the exporter to choose different references of time for 'Export
>>>Time' will lead to many interoperability issues because the message
>>>header doesn't describe which reference of time is used and consequently
>>>collectors will not be able to discover the encoding of the reference of
>>>time. Are we ready to accept to add a new field in the header of the
>>>message?
>>>
>>>Allowing the exporter to compute delta timestamp will mix the meter
>>>clock and the exporter clock. That will corrupt timestamps accuracy
>>>because such difference of time will mix the 2 clocks synchronisation
>>>skews and siblings. The consequence is that the best accuracy of the
>>>flow record will be the accuracy of the exporter process even if the
>>>meter accuracy is much more precise.
>>>
>>>I really agree with the motivation and the need: it is a MUST. But we
>>>MUST keep separated the semantic of the message header from the semantic
>>>of templates, moreover from the semantic of individual Field Type. I am
>>>convinced that the only way to address this issue is to provide the
>>>'Observation Domain' with a mechanism to mix absolute times reference
>>>and delta time references in a 'Flow Data Records'.
>>>
>>>Following is a proposal which reuse the ScopeFieldNumbers defined above:
>>>
>>>Using the ScopeFieldNumbers, a regular template informs the collector of
>>>the number of fields which are only present in the first first record of
>>>the 'Flow Data Records'.
>>>
>>>
>>>Example:
>>>
>>>We (re)define the field type 'dateTimeMicroSeconds' (e.g. identifier
>>>1052) in the info model.
>>>Using the ScopeFieldNumbers, a regular template informs the collector
>>>that the absolute time dateTimeMicroSeconds is only present in the first
>>>field of the first record of the 'Flow Data Records'.
>>>
>>>
>>>   0                   1                   2                   3
>>>   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
>>>  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>>  |         Set ID = 0            |          Length = 32          |
>>>  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>>  |       Template ID 256         |        Field Count = 6        |
>>>  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>>  |     ScopeFieldNumbers = 1034  |              1                |
>>>  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>>  |  dateTimeMicroSeconds = 1052  |       Field Length = 8        |
>>>  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>>  |  156(flowStartDeltaUSecondst) |       Field Length = 4        |
>>>  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>>  |  157 (flowEndDeltaUSeconds)   |       Field Length = 4        |
>>>  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>>  |    8 (sourceIPv4Address)      |       Field Length = 4        |
>>>  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>>  |  12 (destinationIPv4Address)  |       Field Length = 4        |
>>>  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>>
>>>Following is a flow data set. It contains 3 records. The first one has 5
>>>fields. Others have 4 fields.
>>>
>>>+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>>|          Set ID = 256         |          Length = 60          |
>>>+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>>|                       4575688568445                           |
>>>absolute
>>>+                                                               + micro
>>>|                       7875224045965                           |
>>>seconds
>>>+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>>|                            5000                               | delta
>>>+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ micro
>>>|                            5500                               | second
>>>+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>>|                       192.181.17.7                            |
>>>+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>>|                       192.181.17.18                           |
>>>+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>>|                            6000                               | delta
>>>+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ micro
>>>|                            8000                               | second
>>>+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>>|                       192.181.17.17                           |
>>>+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>>|                       192.181.17.19                           |
>>>+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>>|                            6010                               | delta
>>>+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ micro
>>>|                            7000                               | second
>>>+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>>|                       192.181.17.25                           |
>>>+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>>|                       192.181.17.28                           |
>>>+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>>
>>>
>>>      
>>>
>>I'm not in favor of this approach as you now have dependencies between
>>Information Elements.
>>I mean that the meaning of the I.E. Y will depend on the meaning of I.E. X.
>>If the collector doesn't have the I.E. X in his information model, you
>>can't conclude anything about Y
>>If the application that uses the data records doesn't have the
>>intelligence that there is a relation between those 2 I.E., again this
>>is a problem
>>
>>Regards, Benoit.
>>
>>    
>>
>>>Best Regards
>>>Emile
>>>
>>>
>>>      
>>>