Subject: [IPFIX] observation domain, template, and session: compromise

From: Benoit Claise <[email-masked]>
Date: Wed, 11 Oct 2006 15:08:18 +0200

Dear all,

Trying to address the concerns posted on the mailing recently regarding 
"observation domain, template, and session", Andrew, Paul, and I sat 
down and worked out a compromise. We're proposing to make template IDs 
unique per (Observation Domain ID, Session) pair. And NOT to scope the 
Observation Domain to the communication session between the Exporting 
Process and Collecting Process (as proposed by Andrew initially)
I think that this compromise will make everybody happy as this solution 
is mainly a clarification, as opposed to a protocol design change.

Problem Statement.
If I read the protocol draft now, one interpretation could be: as I see 
that the Template Ids are unique per Observation Domain, and that the 
Observation Domain Id are RECOMMENDED to be unique per IPFIX Device, 
this implies that the several Exporting Processes of the same IPFIX 
Device must synchronize the assignments of their Template Ids (to make 
sure that the Template Id are unique per Observation Domain). This might 
prove difficult/impossible in some cases.
Instead of imposing this, the other solution is to make the template IDs 
unique per (Observation Domain ID, Session) pair.
This makes sense as, logically, the template IDs have only a local 
significance between E.P. and C.P.
Note: I brought up the issue of load-balancing or backup where it would 
be nice to have the same template IDs between two connections. The 
solution in this case is to use a single E.P. when load-balancing or 
backup is required.

Please acknowledge that this compromise is fine with you.

Note: 3 minor changes are required in the information elements 
definitions in [IPFIX-INFO]

Regards, Andrew, Paul, and Benoit.

See below for the proposed changes.

_[IPFIX-PROTO]
__
_NEW:_
_

Transport Session
In SCTP, the transport session is known as the SCTP association, which 
is uniquely identified by the SCTP endpoints [RFC2960]; in TCP, the 
transport session is known as the TCP connection, which is uniquely 
identified by the combination of IP addresses and TCP ports used; In 
UDP, the transport session is known as the UDP session, which is 
uniquely identified by the combination of IP addresses and UDP ports used.

OLD
Template ID
      Each of the newly generated Template Records is given a unique
      Template ID.  This uniqueness is local to the Observation
      Domain that generated the Template ID.  Template IDs 0-255 are
      reserved for Template Sets, Options Template Sets, and other
      reserved Sets yet to be created.  Template IDs of Data Sets
      are numbered from 256 to 65535.  There are no constraints
      regarding the order of the Template ID allocation.

NEW:
Template ID
      Each of the newly generated Template Records is given a unique
      Template ID.  This uniqueness is local to the Transport Session and
     Observation Domain that generated the Template ID.  Template IDs 
0-255 are
      reserved for Template Sets, Options Template Sets, and other
      reserved Sets yet to be created.  Template IDs of Data Sets
      are numbered from 256 to 65535.  There are no constraints
      regarding the order of the Template ID allocation.

OLD (Section 4.4)
     templateId              An identifier of a Template that
                             locally unique to the Exporting
                             Process.  This Information
                             Element MUST be defined as a Scope
                             Field.
NEW
     templateId              An identifier of a Template. This Information
                                   Element MUST be defined as a Scope
                                   Field.

Section 3.1
OLD
Observation Domain ID
A 32-bit identifier of the Observation Domain that is locally unique to 
the Exporting Process.  The Exporting Process uses the Observation 
Domain ID to uniquely identify to the Collecting Process the Observation 
Domain that metered the Flows.  It is RECOMMENDED that this identifier 
is also unique per IPFIX Device.  Collecting Processes SHOULD use the 
combination of the Exporter (exporterIPv4Address, exporterIPv6Address, 
or exportingProcessId) and the Observation Domain ID field to separate 
different export streams originating from the same Exporting Process.  
The Observation Domain ID SHOULD be 0 when no specific Observation 
Domain ID is relevant for the entire IPFIX Message.  For example, when 
exporting the Exporting Process Statistics, or in case of hierarchy of 
Collector when aggregated data records are exported.

NEW
Observation Domain ID
A 32-bit identifier of the Observation Domain that is locally unique to 
the Exporting Process.  The Exporting Process uses the Observation 
Domain ID to uniquely identify to the Collecting Process the Observation 
Domain that metered the Flows.  It is RECOMMENDED that this identifier 
is also unique per IPFIX Device.  Collecting Processes SHOULD use the 
Transport Session and the Observation Domain ID field to separate 
different export streams originating from the same Exporting Process.  
The Observation Domain ID SHOULD be 0 when no specific Observation 
Domain ID is relevant for the entire IPFIX Message.  For example, when 
exporting the Exporting Process Statistics, or in case of hierarchy of 
Collector when aggregated data records are exported.

Section 8, Template Management
OLD
The Exporting Process assigns and maintains the Template IDs for the 
Exporter's Observation Domains. A newly created Template Record is 
assigned an unused Template ID by the Exporting Process.

NEW
The Exporting Process assigns and maintains the Template IDs per SCTP 
association for the Exporter's Observation Domains.  A newly created 
Template Record is assigned an unused Template ID by the Exporting Process.

OLD
A Template ID MUST be unique per Observation Domain. Different 
Observation Domains from the same Exporter may use the same Template ID 
value to refer to different Templates.

NEW
Different Observation Domains from the same SCTP association may use the 
same Template ID value to refer to different Templates.

OLD
When the Exporting Process restarts, due to the SCTP association 
shutdown, all Template assignments are lost and Template IDs MUST be 
re-assigned.

NEW
When the SCTP association shuts down or the Exporting Process restarts, 
all Template assignments are lost and Template IDs MUST be re-assigned.

Section 9, Collecting Process's side
OLD
Template IDs are unique per IPFIX Device and per Observation Domain.  If 
the Collecting Process receives a Template which has already been 
received but which has not previously been withdrawn (i.e. a Template 
Record from the same Exporter Observation Domain with the same Template 
ID), then the Collecting Process MUST shutdown the association.

NEW
Template IDs are unique per SCTP association and per Observation 
Domain.  If the Collecting Process receives a Template which has already 
been received but which has not previously been withdrawn (i.e. a 
Template Record from the same Exporter Observation Domain with the same 
Template ID received on the SCTP association), then the Collecting 
Process MUST shutdown the association.

OLD

If a Collecting Process receives a Template Withdraw Message, the 
Collecting Process MUST delete the corresponding Template Records 
associated with the specific Exporter and specific Observation Domain, 
and stop decoding IPFIX Messages that use the withdrawn Templates.
 
If the Collecting Process receives a Template Withdraw message for a 
Template Record it has not received before, it MUST reset the SCTP 
association, discard the IPFIX Message, and SHOULD log the error as it 
does for malformed IPFIX Messages. 
 
A Collecting Process that receives IPFIX Messages from several 
Observation Domains from the same Exporter MUST be aware that the 
uniqueness of the Template ID is not guaranteed across Observation Domains.

NEW
If a Collecting Process receives a Template Withdraw Message, the 
Collecting Process MUST delete the corresponding Template Records 
associated with the specific SCTP association and specific Observation 
Domain, and stop decoding IPFIX Messages that use the withdrawn Templates.
I
f the Collecting Process receives a Template Withdraw message for a 
Template Record it has not received before on this SCTP assocation, it 
MUST reset the SCTP association, discard the IPFIX Message, and SHOULD 
log the error as it does for malformed IPFIX Messages. 
 
A Collecting Process that receives IPFIX Messages from several 
Observation Domains on the same Transport Session MUST be aware that the 
uniqueness of the Template ID is not guaranteed across Observation Domains.

10.3.7 Collecting Process
OLD
At any given time the Collecting Process SHOULD maintain the following 
for all the current Template Records and Options Template Records: 
<Exporting Process, Observation Domain ID, Template ID, Template 
Definition, Last Received>.

NEW
Template IDs are unique per UDP session and per Observation Domain.  At 
any given time the Collecting Process SHOULD maintain the following for 
all the current Template Records and Options Template Records: <IPFIX 
Device, Exporter source UDP port, Observation Domain ID, Template ID, 
Template Definition, Last Received>.

10.4.2.2 Template Management

Template IDs are unique per TCP connection and per Observation Domain.  
A Collecting Process MUST record all Template and Options Template 
Records for the duration of the connection, as an Exporting Process is 
not required to re-export Template Records.

_[IPFIX-INFO]_

NEW

Transport Session
In SCTP, the transport session is known as the SCTP association, which 
is uniquely identified by the SCTP endpoints [RFC2960]; in TCP, the 
transport session is known as the TCP connection, which is uniquely 
identified by the combination of IP addresses and TCP ports used; In 
UDP, the transport session is known as the UDP session, which is 
uniquely identified by the combination of IP addresses and UDP ports used.

OLD
5.1.8.  templateId

   Description:
      An identifier of a Template that is locally unique within an
      Observation Domain.

NEW
5.1.8.  templateId

   Description:
      An identifier of a Template that is locally unique within an
      Observation Domain and the Transport Session.

OLD
5.1.7.  flowId

   Description:
      An identifier of a Flow that is unique within an Observation
      Domain.  This Information Element can be used to distinguish
      between different Flows if Flow Keys such as IP addresses and port
      numbers are not reported or are reported in separate records.

NEW
5.1.7.  flowId

   Description:
      An identifier of a Flow that is unique within an Observation
      Domain and the Transport Session.  This Information Element can be used to distinguish
      between different Flows if Flow Keys such as IP addresses and port
      numbers are not reported or are reported in separate records.

OLD
5.1.11.  commonPropertiesId

   Description:
      An identifier of a set of common properties that is unique per
      Observation Domain.  Typically, this Information Element is used
      to link to information reported in separate records.

NEW:
5.1.11.  commonPropertiesId

   Description:
      An identifier of a set of common properties that is unique per
      Observation Domain and Transport Session.  Typically, this Information Element is used
      to link to information reported in separate records.

_______________________________________________
IPFIX mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipfix

---

From: Brian Trammell <[email-masked]>
Date: Wed, 11 Oct 2006 08:39:55 -0700

Benoit and all,

Excellent - this looks good to me, and addresses all the concerns I  
had with scoping templates and observation IDs (and as you point out,  
has the added benefit of minimal change to the protocol as it stands).

One point: do we want to be more pedantic about defining a Transport  
Session for UDP? Both the SCTP and TCP Transport Session definitions  
have temporal limits (the Association or the Connection lifetime,  
defined by each transport protocol itself), while UDP does not. I  
would suggest (from a past message to the list):

In UDP, the transport session is known as the UDP session, which is  
uniquely identified by the IP addresses and UDP ports used, within  
the time between the first UDP packet sent by the EP in the session  
and the timeout after the last template retransmission.

Thoughts?

- Brian

On Oct 11, 2006, at 6:08 AM, Benoit Claise wrote:

> Dear all,
>
> Trying to address the concerns posted on the mailing recently  
> regarding "observation domain, template, and session", Andrew,  
> Paul, and I sat down and worked out a compromise. We're proposing  
> to make template IDs unique per (Observation Domain ID, Session)  
> pair. And NOT to scope the Observation Domain to the communication  
> session between the Exporting Process and Collecting Process (as  
> proposed by Andrew initially)
> I think that this compromise will make everybody happy as this  
> solution is mainly a clarification, as opposed to a protocol design  
> change.
>
> Problem Statement.
> If I read the protocol draft now, one interpretation could be: as I  
> see that the Template Ids are unique per Observation Domain, and  
> that the Observation Domain Id are RECOMMENDED to be unique per  
> IPFIX Device, this implies that the several Exporting Processes of  
> the same IPFIX Device must synchronize the assignments of their  
> Template Ids (to make sure that the Template Id are unique per  
> Observation Domain). This might prove difficult/impossible in some  
> cases.
> Instead of imposing this, the other solution is to make the  
> template IDs unique per (Observation Domain ID, Session) pair.
> Note: I brought up the issue of load-balancing or backup where it  
> would be nice to have the same template IDs between two  
> connections. The solution in this case is to use a single E.P. when  
> load-balancing or backup is required.
>
> [IPFIX-PROTO]
> In SCTP, the transport session is known as the SCTP association,  
> which is uniquely identified by the SCTP endpoints [RFC2960]; in  
> TCP, the transport session is known as the TCP connection, which is  
> uniquely identified by the combination of IP addresses and TCP  
> ports used; In UDP, the transport session is known as the UDP  
> session, which is uniquely identified by the combination of IP  
> addresses and UDP ports used.
>
>
>
> OLD
> Template ID
>      Observation Domain that generated the Template ID.  Template  
> IDs 0-255 are
> A 32-bit identifier of the Observation Domain that is locally  
> unique to the Exporting Process.  The Exporting Process uses the  
> Observation Domain ID to uniquely identify to the Collecting  
> Process the Observation Domain that metered the Flows.  It is  
> RECOMMENDED that this identifier is also unique per IPFIX Device.   
> Collecting Processes SHOULD use the combination of the Exporter  
> (exporterIPv4Address, exporterIPv6Address, or exportingProcessId)  
> and the Observation Domain ID field to separate different export  
> streams originating from the same Exporting Process.  The  
> Observation Domain ID SHOULD be 0 when no specific Observation  
> Domain ID is relevant for the entire IPFIX Message.  For example,  
> when exporting the Exporting Process Statistics, or in case of  
> hierarchy of Collector when aggregated data records are exported.
>
> NEW
> Observation Domain ID
> A 32-bit identifier of the Observation Domain that is locally  
> unique to the Exporting Process.  The Exporting Process uses the  
> Observation Domain ID to uniquely identify to the Collecting  
> Process the Observation Domain that metered the Flows.  It is  
> RECOMMENDED that this identifier is also unique per IPFIX Device.   
> Collecting Processes SHOULD use the Transport Session and the  
> Observation Domain ID field to separate different export streams  
> originating from the same Exporting Process.  The Observation  
> Domain ID SHOULD be 0 when no specific Observation Domain ID is  
> relevant for the entire IPFIX Message.  For example, when exporting  
> the Exporting Process Statistics, or in case of hierarchy of  
> The Exporting Process assigns and maintains the Template IDs for  
> the Exporter's Observation Domains. A newly created Template Record  
> is assigned an unused Template ID by the Exporting Process.
>
> NEW
> The Exporting Process assigns and maintains the Template IDs per  
> SCTP association for the Exporter's Observation Domains.  A newly  
> created Template Record is assigned an unused Template ID by the  
> Exporting Process.
>
> OLD
> Observation Domains from the same Exporter may use the same  
> Template ID value to refer to different Templates.
>
> NEW
> Different Observation Domains from the same SCTP association may  
> use the same Template ID value to refer to different Templates.
>
> OLD
> shutdown, all Template assignments are lost and Template IDs MUST  
> be re-assigned.
>
> NEW
> When the SCTP association shuts down or the Exporting Process  
> restarts, all Template assignments are lost and Template IDs MUST  
> be re-assigned.
>
> Template IDs are unique per IPFIX Device and per Observation  
> Domain.  If the Collecting Process receives a Template which has  
> already been received but which has not previously been withdrawn  
> (i.e. a Template Record from the same Exporter Observation Domain  
> with the same Template ID), then the Collecting Process MUST  
> shutdown the association.
>
>
> NEW
> Domain.  If the Collecting Process receives a Template which has  
> already been received but which has not previously been withdrawn  
> (i.e. a Template Record from the same Exporter Observation Domain  
> with the same Template ID received on the SCTP association), then  
> the Collecting Process MUST shutdown the association.
>
> OLD
>
> associated with the specific Exporter and specific Observation  
> If the Collecting Process receives a Template Withdraw message for  
> a Template Record it has not received before, it MUST reset the  
> SCTP association, discard the IPFIX Message, and SHOULD log the  
> error as it does for malformed IPFIX Messages.
>
> associated with the specific SCTP association and specific  
> Observation Domain, and stop decoding IPFIX Messages that use the  
> withdrawn Templates.
> I
> Template Record it has not received before on this SCTP assocation,  
> it MUST reset the SCTP association, discard the IPFIX Message, and  
> SHOULD log the error as it does for malformed IPFIX Messages.
>
> Observation Domains on the same Transport Session MUST be aware  
> that the uniqueness of the Template ID is not guaranteed across  
> Observation Domains.
>
> At any given time the Collecting Process SHOULD maintain the  
> following for all the current Template Records and Options Template  
> Records: <Exporting Process, Observation Domain ID, Template ID,  
> Template IDs are unique per UDP session and per Observation  
> Domain.  At any given time the Collecting Process SHOULD maintain  
> the following for all the current Template Records and Options  
> Template Records: <IPFIX Device, Exporter source UDP port,  
> Observation Domain ID, Template ID, Template Definition, Last  
> Received>.
>
>
> Template IDs are unique per TCP connection and per Observation  
> Domain.  A Collecting Process MUST record all Template and Options  
> Template Records for the duration of the connection, as an  
> Exporting Process is not required to re-export Template Records.
>
>
> [IPFIX-INFO]
> NEWTransport Session
> In SCTP, the transport session is known as the SCTP association,  
> which is uniquely identified by the SCTP endpoints [RFC2960]; in  
> TCP, the transport session is known as the TCP connection, which is  
> uniquely identified by the combination of IP addresses and TCP  
> ports used; In UDP, the transport session is known as the UDP  
> session, which is uniquely identified by the combination of IP  
> addresses and UDP ports used.
> OLD 5.1.8. templateId Description: An identifier of a Template that  
> is locally unique within an Observation Domain. NEW 5.1.8.  
> templateId Description: An identifier of a Template that is locally  
> unique within an Observation Domain and the Transport Session. OLD  
> 5.1.7. flowId Description: An identifier of a Flow that is unique  
> within an Observation Domain. This Information Element can be used  
> to distinguish between different Flows if Flow Keys such as IP  
> addresses and port numbers are not reported or are reported in  
> separate records. NEW 5.1.7. flowId Description: An identifier of a  
> Flow that is unique within an Observation Domain and the Transport  
> Session. This Information Element can be used to distinguish  
> numbers are not reported or are reported in separate records. OLD  
> 5.1.11. commonPropertiesId Description: An identifier of a set of  
> common properties that is unique per Observation Domain. Typically,  
> this Information Element is used to link to information reported in  
> separate records. NEW: 5.1.11. commonPropertiesId Description: An  
> identifier of a set of common properties that is unique per  
> Observation Domain and Transport Session. Typically, this  
> Information Element is used to link to information reported in  
> separate records.

---

From: Brian Trammell <[email-masked]>
Date: Wed, 11 Oct 2006 08:39:55 -0700

Benoit and all,

Thoughts?

- Brian

---

From: Paul Aitken <[email-masked]>
Date: Wed, 11 Oct 2006 17:12:48 +0100

Brian,

> uniquely identified by the IP addresses and UDP ports used, within  the 
> time between the first UDP packet sent by the EP in the session  and the 
> timeout after the last template retransmission.

Seems good to me. But change "last" to "latest" or "most recent".

Someone once asked me how we know it's the "last packet". We don't; we 
mean "last seen so far", aka "latest".

-- 
Paul Aitken
Cisco Systems Ltd, Edinburgh, Scotland.

---

From: Paul Aitken <[email-masked]>
Date: Wed, 11 Oct 2006 17:12:48 +0100

Brian,

---

From: Brian Trammell <[email-masked]>
Date: Wed, 11 Oct 2006 09:14:03 -0700

Paul,

Yep; this is much better.

- Brian

On Oct 11, 2006, at 9:12 AM, Paul Aitken wrote:

> Brian,
>
>> In UDP, the transport session is known as the UDP session, which  
>> is  uniquely identified by the IP addresses and UDP ports used,  
>> within  the time between the first UDP packet sent by the EP in  
>> the session  and the timeout after the last template retransmission.
>
> Someone once asked me how we know it's the "last packet". We don't;  
> we mean "last seen so far", aka "latest".
>
> -- 
> Paul Aitken

---

From: Brian Trammell <[email-masked]>
Date: Wed, 11 Oct 2006 09:14:03 -0700

Paul,

- Brian

---

From: Benoit Claise <[email-masked]>
Date: Wed, 11 Oct 2006 19:04:08 +0200

Brian,
> has the added benefit of minimal change to the protocol as it stands).
Good.
> time between the first UDP packet sent by the EP in the session and 
> the timeout after the last template retransmission.
Which timeout? (options) template retransmission? template lifetime? 3 
times template lifetime?
Where: Exporter and/or Collector?

Regards, Benoit.
>> regarding "observation domain, template, and session", Andrew, Paul, 
>> and I sat down and worked out a compromise. We're proposing to make 
>> template IDs unique per (Observation Domain ID, Session) pair. And 
>> NOT to scope the Observation Domain to the communication session 
>> between the Exporting Process and Collecting Process (as proposed by 
>> Andrew initially)
>> see that the Template Ids are unique per Observation Domain, and that 
>> the Observation Domain Id are RECOMMENDED to be unique per IPFIX 
>> Device, this implies that the several Exporting Processes of the same 
>> IPFIX Device must synchronize the assignments of their Template Ids 
>> (to make sure that the Template Id are unique per Observation 
>> Domain). This might prove difficult/impossible in some cases.
>> Instead of imposing this, the other solution is to make the template 
>> IDs unique per (Observation Domain ID, Session) pair.
>> would be nice to have the same template IDs between two connections. 
>> The solution in this case is to use a single E.P. when load-balancing 
>> or backup is required.
>>
>> which is uniquely identified by the SCTP endpoints [RFC2960]; in TCP, 
>> the transport session is known as the TCP connection, which is 
>> uniquely identified by the combination of IP addresses and TCP ports 
>> used; In UDP, the transport session is known as the UDP session, 
>> which is uniquely identified by the combination of IP addresses and 
>> UDP ports used.
>>
>>
>>
>> OLD
>> Template ID
>> A 32-bit identifier of the Observation Domain that is locally unique 
>> to the Exporting Process.  The Exporting Process uses the Observation 
>> Domain ID to uniquely identify to the Collecting Process the 
>> Observation Domain that metered the Flows.  It is RECOMMENDED that 
>> this identifier is also unique per IPFIX Device.  Collecting 
>> Processes SHOULD use the combination of the Exporter 
>> (exporterIPv4Address, exporterIPv6Address, or exportingProcessId) and 
>> the Observation Domain ID field to separate different export streams 
>> originating from the same Exporting Process.  The Observation Domain 
>> ID SHOULD be 0 when no specific Observation Domain ID is relevant for 
>> the entire IPFIX Message.  For example, when exporting the Exporting 
>> Process Statistics, or in case of hierarchy of Collector when 
>> aggregated data records are exported.
>>
>> NEW
>> Observation Domain ID
>> A 32-bit identifier of the Observation Domain that is locally unique 
>> to the Exporting Process.  The Exporting Process uses the Observation 
>> Domain ID to uniquely identify to the Collecting Process the 
>> Observation Domain that metered the Flows.  It is RECOMMENDED that 
>> this identifier is also unique per IPFIX Device.  Collecting 
>> Processes SHOULD use the Transport Session and the Observation Domain 
>> ID field to separate different export streams originating from the 
>> same Exporting Process.  The Observation Domain ID SHOULD be 0 when 
>> no specific Observation Domain ID is relevant for the entire IPFIX 
>> Message.  For example, when exporting the Exporting Process 
>> Statistics, or in case of hierarchy of Collector when aggregated data 
>> records are exported.
>>
>> Observation Domains from the same Exporter may use the same Template 
>> ID value to refer to different Templates.
>>
>> NEW
>> Different Observation Domains from the same SCTP association may use 
>> the same Template ID value to refer to different Templates.
>>
>> OLD
>> restarts, all Template assignments are lost and Template IDs MUST be 
>> re-assigned.
>>
>> Template IDs are unique per IPFIX Device and per Observation Domain.  
>> If the Collecting Process receives a Template which has already been 
>> Record from the same Exporter Observation Domain with the same 
>> Template ID), then the Collecting Process MUST shutdown the association.
>>
>>
>> NEW
>> with the same Template ID received on the SCTP association), then the 
>> Collecting Process MUST shutdown the association.
>>
>> OLD
>>
>> association, discard the IPFIX Message, and SHOULD log the error as 
>> it does for malformed IPFIX Messages.
>>
>> Observation Domains on the same Transport Session MUST be aware that 
>> the uniqueness of the Template ID is not guaranteed across 
>> Observation Domains.
>>
>> Template IDs are unique per UDP session and per Observation Domain.  
>> Records: <IPFIX Device, Exporter source UDP port, Observation Domain 
>> ID, Template ID, Template Definition, Last Received>.
>>
>>
>> Template Records for the duration of the connection, as an Exporting 
>> Process is not required to re-export Template Records.
>>
>>
>> [IPFIX-INFO]
>> NEWTransport Session
>> which is uniquely identified by the SCTP endpoints [RFC2960]; in TCP, 
>> the transport session is known as the TCP connection, which is 
>> uniquely identified by the combination of IP addresses and TCP ports 
>> used; In UDP, the transport session is known as the UDP session, 
>> which is uniquely identified by the combination of IP addresses and 
>> UDP ports used.
>> is locally unique within an Observation Domain. NEW 5.1.8. templateId 
>> Description: An identifier of a Template that is locally unique 
>> within an Observation Domain and the Transport Session. OLD 5.1.7. 
>> flowId Description: An identifier of a Flow that is unique within an 
>> Observation Domain. This Information Element can be used to 
>> distinguish between different Flows if Flow Keys such as IP addresses 
>> and port numbers are not reported or are reported in separate 
>> records. NEW 5.1.7. flowId Description: An identifier of a Flow that 
>> is unique within an Observation Domain and the Transport Session. 
>> This Information Element can be used to distinguish between different 
>> Flows if Flow Keys such as IP addresses and port numbers are not 
>> reported or are reported in separate records. OLD 5.1.11. 
>> commonPropertiesId Description: An identifier of a set of common 
>> properties that is unique per Observation Domain. Typically, this 
>> Observation Domain and Transport Session. Typically, this Information 
>> Element is used to link to information reported in separate records.

---

From: Benoit Claise <[email-masked]>
Date: Wed, 11 Oct 2006 19:04:08 +0200

Brian,
Good.

Regards, Benoit.

---

From: Andrew Johnson <[email-masked]>
Date: Thu, 12 Oct 2006 09:29:00 +0100

Brian Trammell wrote:
> Benoit and all,
> 
> Excellent - this looks good to me, and addresses all the concerns I had 
> with scoping templates and observation IDs (and as you point out, has 
> the added benefit of minimal change to the protocol as it stands).
> 
> defined by each transport protocol itself), while UDP does not. I would 
> suggest (from a past message to the list):
> 

In UDP the definition associated with a template ID has a certain
lifetime.  The lifetime is different on the the Exporting Process and
the Collecting Process.  I think we can simply extend the principle
to apply to all definitions associated with identifiers that are
scoped to the Transport Session.

I think only one section would need updated.

OLD:
   The Collecting Process MUST associate a lifetime with each Template
   received via UDP.  Templates not refreshed by the Exporting Process
   within the lifetime are expired at the Collecting Process.  If the
   template is not refreshed by the Exporting Process before that
   lifetime has expired, the Collecting Process MUST discard the
   Template and any current and future associated Data Records.  In
   which case, an alarm MUST be logged.
   ...
   The Collecting Process SHOULD accept Data Records without the
   associated Template Record. If the Template Records have not been
   received at the time Data Records are received, the Collecting
   Process SHOULD store the Data Records for a short period of time and
   decode them after the Template Records are received.  The short
   period of time MUST be lower than the Template lifetime.

NEW:
   The Collecting Process MUST associate a lifetime with each Template
   (or another of definition of an identifier considered unique within
   the Transport Session) received via UDP.  Templates (and similar
   definitions) not refreshed by the Exporting Process within the
   lifetime are expired at the Collecting Process.  If the Template
   (or other definition) is not refreshed before that lifetime has
   expired, the Collecting Process MUST discard that definition
   and any current and future associated Data Records.  In which case,
   an alarm MUST be logged.
   ...
   The Collecting Process SHOULD accept Data Records without the
   associated Template Record (or other definitions) required to decode
   the Data Record. If the Template Records (or other definitions such as
   Common Properties) have not been received at the time Data Records
   are received, the Collecting Process SHOULD store the Data Records
   for a short period of time and decode them after the Template Records
   (or other definitions) are received. The short period of time MUST be
   lower than the lifetime of definitions associated with identifiers
   considered unique within the UDP session. 

regards

Andrew 

>> template IDs unique per (Observation Domain ID, Session) pair. And NOT 
>> to scope the Observation Domain to the communication session between 
>> the Exporting Process and Collecting Process (as proposed by Andrew 
>> initially)
>> (to make sure that the Template Id are unique per Observation Domain). 
>> This might prove difficult/impossible in some cases.
>> this identifier is also unique per IPFIX Device.  Collecting Processes 
>> SHOULD use the combination of the Exporter (exporterIPv4Address, 
>> exporterIPv6Address, or exportingProcessId) and the Observation Domain 
>> same Exporting Process.  The Observation Domain ID SHOULD be 0 when no 
>> specific Observation Domain ID is relevant for the entire IPFIX 
>> this identifier is also unique per IPFIX Device.  Collecting Processes 
>> SHOULD use the Transport Session and the Observation Domain ID field 
>> to separate different export streams originating from the same 
>> Exporting Process.  The Observation Domain ID SHOULD be 0 when no 
>> specific Observation Domain ID is relevant for the entire IPFIX 
>> (i.e. a Template Record from the same Exporter Observation Domain with 
>> the same Template ID received on the SCTP association), then the 
>> the uniqueness of the Template ID is not guaranteed across Observation 
>> Domains.
>>
>> <IPFIX Device, Exporter source UDP port, Observation Domain ID, 
>> Template ID, Template Definition, Last Received>.
>>
>>
>> OLD 5.1.8. templateId Description: An identifier of a Template that is 
>> locally unique within an Observation Domain. NEW 5.1.8. templateId 
>> Description: An identifier of a Template that is locally unique within 
>> an Observation Domain and the Transport Session. OLD 5.1.7. flowId 
>> Description: An identifier of a Flow that is unique within an 
>> and port numbers are not reported or are reported in separate records. 
>> NEW 5.1.7. flowId Description: An identifier of a Flow that is unique 
>> within an Observation Domain and the Transport Session. This 
>> Information Element can be used to distinguish between different Flows 
>> if Flow Keys such as IP addresses and port numbers are not reported or 
>> are reported in separate records. OLD 5.1.11. commonPropertiesId 
>> Description: An identifier of a set of common properties that is 
>> unique per Observation Domain. Typically, this Information Element is 
>> used to link to information reported in separate records. NEW: 5.1.11. 
>> properties that is unique per Observation Domain and Transport 
>> Session. Typically, this Information Element is used to link to 
>> information reported in separate records.

---

From: Andrew Johnson <[email-masked]>
Date: Thu, 12 Oct 2006 09:29:00 +0100

OLD:

NEW:

regards

Andrew

---

From: Gerhard Muenz <[email-masked]>
Date: Thu, 12 Oct 2006 10:31:31 +0200

Dear Brian, All,

I also agree with the proposed changes.

> timeout after the last template retransmission.

Fine with me. So using UDP, both templates and obervation domains are
"soft states" with limited lifetime unless they are renewed.

One question which is orthogonal but related:
The proposed IPFIX configuration data model
http://net.informatik.uni-tuebingen.de/fileadmin/RI/members/muenz/documents/draft-muenz-ipfix-configuration-00.txt
covers the case when the observation domain ID is configurable.
Currently, the observation domain ID is a parameter of the observation
point.
What are your opinions? Is this the right place?
Or do you prefer to have an observation domain ID paramater per
exporting process?
Or both (so it can be adapted to the implementation of the device)?

Regards,
Gerhard

---

From: Gerhard Muenz <[email-masked]>
Date: Thu, 12 Oct 2006 10:31:31 +0200

Dear Brian, All,

Regards,
Gerhard

---

From: Benoit Claise <[email-masked]>
Date: Thu, 12 Oct 2006 13:50:22 +0200

Gerhard,
> Or both (so it can be adapted to the implementation of the device)?
My view is that, on a router, the Observation Domain Id should be the 
slot Id (available in the ENTITY-MIB)

Regards, Benoit.

---

From: Benoit Claise <[email-masked]>
Date: Thu, 12 Oct 2006 13:50:22 +0200

Gerhard,

Regards, Benoit.

---

From: Lutz Mark <[email-masked]>
Date: Thu, 12 Oct 2006 16:06:03 +0200

Benoit,

> Please acknowledge that this compromise is fine with you.

perfect.

Best regards,
Lutz

---

From: Lutz Mark <[email-masked]>
Date: Thu, 12 Oct 2006 16:06:03 +0200

Benoit,

perfect.

Best regards,
Lutz

---

From: Brian Trammell <[email-masked]>
Date: Thu, 12 Oct 2006 11:42:10 -0700

Andrew and all,

Looks good to me!

Cheers,

Brian

On Oct 12, 2006, at 1:29 AM, Andrew Johnson wrote:

> Brian Trammell wrote:
>> Benoit and all,
>> Excellent - this looks good to me, and addresses all the concerns  
>> I had with scoping templates and observation IDs (and as you point  
>> out, has the added benefit of minimal change to the protocol as it  
>> stands).
>> One point: do we want to be more pedantic about defining a  
>> Transport Session for UDP? Both the SCTP and TCP Transport Session  
>> definitions have temporal limits (the Association or the  
>> Connection lifetime, defined by each transport protocol itself),  
>> while UDP does not. I would suggest (from a past message to the  
>> list):
>> within the time between the first UDP packet sent by the EP in the  
>> session and the timeout after the last template retransmission.
>> Thoughts?
>
>>> pair. And NOT to scope the Observation Domain to the  
>>> communication session between the Exporting Process and  
>>> Collecting Process (as proposed by Andrew initially)
>>> solution is mainly a clarification, as opposed to a protocol  
>>> design change.
>>>
>>> Problem Statement.
>>> If I read the protocol draft now, one interpretation could be: as  
>>> I see that the Template Ids are unique per Observation Domain,  
>>> and that the Observation Domain Id are RECOMMENDED to be unique  
>>> per IPFIX Device, this implies that the several Exporting  
>>> Processes of the same IPFIX Device must synchronize the  
>>> assignments of their Template Ids (to make sure that the Template  
>>> Id are unique per Observation Domain). This might prove difficult/ 
>>> impossible in some cases.
>>> This makes sense as, logically, the template IDs have only a  
>>> local significance between E.P. and C.P.
>>> connections. The solution in this case is to use a single E.P.  
>>> when load-balancing or backup is required.
>>>
>>> TCP, the transport session is known as the TCP connection, which  
>>> is uniquely identified by the combination of IP addresses and TCP  
>>> RECOMMENDED that this identifier is also unique per IPFIX  
>>> Device.  Collecting Processes SHOULD use the combination of the  
>>> Exporter (exporterIPv4Address, exporterIPv6Address, or  
>>> exportingProcessId) and the Observation Domain ID field to  
>>> separate different export streams originating from the same  
>>> Statistics, or in case of hierarchy of Collector when aggregated  
>>> data records are exported.
>>>
>>> NEW
>>> Observation Domain ID
>>> RECOMMENDED that this identifier is also unique per IPFIX  
>>> Device.  Collecting Processes SHOULD use the Transport Session  
>>> the Exporter's Observation Domains. A newly created Template  
>>> Record is assigned an unused Template ID by the Exporting Process.
>>>
>>> NEW
>>> If the Collecting Process receives a Template Withdraw message  
>>> for a Template Record it has not received before, it MUST reset  
>>> the SCTP association, discard the IPFIX Message, and SHOULD log  
>>> the error as it does for malformed IPFIX Messages.
>>>
>>> f the Collecting Process receives a Template Withdraw message for  
>>> a Template Record it has not received before on this SCTP  
>>> assocation, it MUST reset the SCTP association, discard the IPFIX  
>>> Message, and SHOULD log the error as it does for malformed IPFIX  
>>> Messages.
>>>
>>> following for all the current Template Records and Options  
>>> Template Records: <Exporting Process, Observation Domain ID,  
>>> Domain.  A Collecting Process MUST record all Template and  
>>> Options Template Records for the duration of the connection, as  
>>> an Exporting Process is not required to re-export Template Records.
>>>
>>>
>>> [IPFIX-INFO]
>>> NEWTransport Session
>>> TCP, the transport session is known as the TCP connection, which  
>>> is uniquely identified by the combination of IP addresses and TCP  
>>> OLD 5.1.8. templateId Description: An identifier of a Template  
>>> that is locally unique within an Observation Domain. NEW 5.1.8.  
>>> templateId Description: An identifier of a Template that is  
>>> locally unique within an Observation Domain and the Transport  
>>> Session. OLD 5.1.7. flowId Description: An identifier of a Flow  
>>> that is unique within an Observation Domain. This Information  
>>> Element can be used to distinguish between different Flows if  
>>> Flow Keys such as IP addresses and port numbers are not reported  
>>> or are reported in separate records. NEW 5.1.7. flowId  
>>> Observation Domain and the Transport Session. This Information  
>>> Element can be used to distinguish between different Flows if  
>>> Flow Keys such as IP addresses and port numbers are not reported  
>>> or are reported in separate records. OLD 5.1.11.

---

From: Brian Trammell <[email-masked]>
Date: Thu, 12 Oct 2006 11:42:10 -0700

Andrew and all,

Looks good to me!

Cheers,

Brian