Subject: [ipfix-protocol] PROTO-25, 31, 32, and 33: template management, collecting process and transport

From: Benoit Claise <[email-masked]>
Date: Mon, 29 Nov 2004 16:23:54 +0100

Dear all,

Stewart Bryant, Paul Aitken, and I have been trying to address the
protocol issues # 25, 31, 32, and 33.

     PROTO-25: The section "Template Management" and "The Collecting
    Process's Side" will have to updated according to the transport
    protocol.
    - For example, the point 2 of the section "Template Management".
    Remark: the template management will vary with TCP, SCTP,
    etc...
    Must have both sections updated: transport updated and
    template management sections (BTW, this is the same for the
    failover section).
    - From Deri's draft: On the other hand as a probe can send flows
    to several collectors (e.g. in round-robin or as a reflector) it
    must keep track of per-collector templates transmission. This
    means that if collector X reconnects, the probe must send the
    template only to this collector and not to all collectors.

    PROPOSAL, after IETF60: treat UDP as the exception, in the template
    subsection of the UDP transport protocol.

    PROTO-31 The "Sequence Number" and "Source ID" treatment in case of
    multiple streams in SCTP is not well described. For example, in case
    the Templates are sent to one stream and the flow records to another
    one, what should the "Sequence Number"? What should the collector
    do? See David Moore's post

    PROTO-32 Correct this issue below
    The Collecting Process SHOULD verify that the received IPFIX
    Messages inside one stream do not have differing Source ID values
    and silently discard any data that does NOT match the initial
    value.

    The Exporting Process SHOULD NOT transmit messages inside one
    stream with multiple Source ID values. The correlated Flow Records
    are then treated like a normal export Flow.

    PROTO-33: correct the next paragraphs: silently? reset the
    connection? log an error? should the exporting process be allowed to
    sent multiple Source ID per stream.

    The Collecting Process SHOULD verify that the received IPFIX
    Messages inside one stream do not have differing Source ID
    values and silently discard any data that does NOT match the
    initial value.

    The Exporting Process SHOULD NOT transmit messages inside one
    stream with multiple Source ID values. The correlated Flow
    Records are then treated like a normal export Flow.

Here is some text, to start the discussion
Only the relevant sections have been introduced:
- transport sections (section 5)
  BTW, the TCP sections has been changed much as Simon will do it
- template managment (section 12)
- Collecting process's side (section 13)

BTW, I think we should move the transport protocols mappings section after section 13, because section 12/13 are dedicated to SCTP.
The transport mappings for TCP and UDP are the exceptions, and as a consequence should be handled afterwards.

Feedback?

Regards, Benoit.

             5.               Transport Protocol 

             5.2                TCP 
                 
                This section describes how IPFIX can be transported over 
                TCP [TCP].  

             5.2.4.4   Template Management 
                 
                New Template Records SHOULD be transmitted as soon as 
                they are created on the Metering Process, and preferably 
                before any associated Flow or  Options Data Record is 
                transmitted.  The Collecting Process SHOULD accept Flow 
                and Options Data Records without the associated Template 
                Record.  
                 
                A Collecting Process MUST record all Template and Option 
                Template Records for the duration of the connection, as 
                an Exporting Process is not required to re-export 
                Template Records.  

             5.3                  SCTP 
             
                This section describes how IPFIX can be transported over 
                SCTP [RFC2960] using the PR-SCTP [RFC3758] extension.    

             5.3.4.5 Template Management 
             
                When the transport protocol is SCTP the default Template 
                Management described in Section 12 is used. 

             5.3.5   Collecting Process 
             
                When the transport protocol is SCTP, the default 
                Collector processing described in Section 13 is used.  

             5.4                  UDP 
                
                This section describes how IPFIX can be transported over 
                UDP [RFC768]. 

             5.4.5.1   Template Management 
                 
                When IPFIX uses UDP as the transport protocol, Template 
                Sets and Option Template Sets MUST be re-sent at regular 
                intervals.  The frequency of (Options) Template 
                transmission MUST be configurable.  New Template Records 
                SHOULD be transmitted as soon as they are created, and 
                SHOULD be transmitted before any associated Data  Record 
                is transmitted. 
             
                In the event of configuration changes, the Exporting 
                Process SHOULD send the new template definitions at an 
                accelerated rate.  In such a case, it MAY transmit the 
                changed Template Record(s) and Options Template 
                Record(s), without any data, in advance to help ensure 
                that the Collector will have the correct template 
                information before receiving the first data. 
                 
                If the Option Template scope is defined in another 
                Template, then both Templates SHOULD be sent in the same 
                IPFIX Message. For example: if a Flow Key Option 
                Template (see section 8.3) is sent in an Option 
                Template, then the associated Template SHOULD be sent in 
                the same IPFIX Message. 
                 
                Note that following a configuration change a new 
                Template ID should be used and the old Template ID 
                SHOULD NOT be reused until its lifetime has expired. 
                 
                Template Withdraw Messages SHOULD NOT be sent over UDP. 

             5.4.6   Collecting Process 
                 
                The Collecting Process SHOULD accept Flow and Options 
                Data Records without the associated Template Record. If 
                the Template Records have not been received at the time 
                Flow Data Records (or Options Data Records) are 
                received, the Collecting Process SHOULD store the Flow 
                Data Records (or Options Data Records) for a short 
                period of time and decode them after the (Options) 
                Template Records are received.  The short period of time 
                MUST be lower than the Template lifetime. 
                 
                The lifetime of a template at the Collecting Process is 
                limited to a fixed refresh timeout.  The Collecting 
                Process MUST associate a lifetime with each Template 
                received via UDP.  Templates not refreshed by the 
                Exporting Process within the timeout are expired at the 
                Collecting Process.  If the template is not refreshed by 
                the Exporting Process before that lifetime has expired, 
                the Collecting Process MUST discard the Template, and 
                any current and future associated Flow or Option Data 
                Records.  The Collecting Process MUST NOT decode any 
                further Flow or Option Data Records which are associated 
                with  an that expired Template.  
                  
                At any given time the Collecting Process SHOULD maintain 
                the following for all the current Template Records and 
                Options Template Records: <Exporting Process, 
                Observation Domain, Template ID, Template Definition, 
                Last Received>.   
                 
             12                 Template Management 
             
                This section describes Template management when using 
                SCTP and SCTP-PR as the transport protocol. Any 
                necessary changes to Template management specifically 
                related to TCP or UDP transport protocols are specified 
                in section 5. 
                 
                The Exporting Process assigns and maintains the 
                Template IDs for the Exporter’s Observations Domains. A 
                newly created Template Record is assigned an unused 
                Template ID by the Exporting Process.  
                 
                Templates Sets and Option Template Sets MUST be only 
                sent once on SCTP stream zero with full reliability.  As 
                such, the Collecting Process MUST store the Template 
                Record information for the duration of the association 
                so that it can interpret the corresponding Flow Data 
                Records that are received in subsequent Data Sets. 
                 
                New Template Records SHOULD be transmitted as soon as 
                they are created. The Exporting Process MAY transmit 
                the Template Set and Options Template in advance of any 
                Data Sets that use that (Options) Template ID, to 
                ensure that the Collector has the Template Record 
                before receiving the first Flow or Options Data Record.  
                Flow and Options Data Records that correspond to a 
                Template Record MAY appear in the same and/or 
                subsequent IPFIX Message(s).   
             
                A Template ID MUST be unique per Observation Domain. 
                Different Observation Domains from the same Exporter may 
                use the same Template ID value to refer to different 
                Templates. 
             
                If the measurement parameters change, the Template MUST 
                be withdrawn using a Template Withdraw Message or an 
                unused Template ID MUST be used. Examples of the 
                measurement changes are: a new sampling rate, a new 
                flow expiration process, a new filtering definition, 
                etc. If a Template is changed, a Template Withdraw 
                Message MUST be sent to delete the Template.  
                 
                If the Exporting Process restarts, the SCTP association 
                MUST be shutdown and restarted. When the Exporting 
                Process restarts, all Template assignments are lost and 
                Template IDs MUST be re-assigned. If the Metering 
                Process restarts, the Exporting Process MUST either 
                reuse the previously assigned Template ID for each 
                Template, or it MUST withdraw the previously issued 
                Template IDs by sending Template Withdraw Message(s) 
                before reusing them. EDITOR’S NOTE: A TEMPLATE WITHDRAW 
                MESSAGE THAT WITHDRAW ALL THE TEMPLATE FOR A SPECIFIC 
                SOURCE ID MIGHT BE USEFUL. 
                 
                When the SCTP association restarts, the Exporting 
                Process MUST resend all the Template Records. 
                 
                The Exporting Process MUST NOT transmit IPFIX Messages 
                with more than one Source ID value inside any single 
                stream. 
                 
                More that one (Option) Template Set MAY be sent in an 
                IPFIX Message.  
                 
             13.                 The Collecting Process's Side 
                 
                This section describes the Collecting Process when using 
                SCTP and SCTP-PR as the transport protocol. Any 
                necessary changes to the Collecting Process specifically 
                related to TCP or UDP transport protocols are specified 
                in section 5. 
                 
                The Collecting Process SHOULD listen for a new 
                association request from the Exporting Process.  The 
                Exporting Process will request a number of streams to 
                use for export.  A Collecting Process MUST support at 
                least two inbound streams per association.  An Exporting 
                Process MAY ask for and support more than two streams. 
                 
                The Collecting Process MUST verify that only one Source 
                ID value is used inside each stream. If the Collecting 
                Process detects that more than one Source ID has been 
                received within a stream, it MUST discard the IPFIX 
                Message, reset the SCTP association, and SHOULD log the 
                error 
                 
                If the Collecting Process receives a malformed IPFIX 
                Message, it MUST reset the SCTP association, discard the 
                message, and SHOULD log the error.  
                 
                Templates Sets and Option Template Sets are only sent 
                once.  The Collecting Process MUST store the Template 
                Record information for the duration of the association 
                so that it can interpret the corresponding Flow Data 
                Records that are received in subsequent Data Sets. 
                 
                Template IDs are unique per Exporting Process and per 
                Observation Domain.  If the Collecting Process receives 
                a Template which has already been received but which has 
                not previously been withdrawn (i.e. a Template Record 
                from the same Exporter Observation Domain with the same 
                Template ID), then the Collecting Process MUST shutdown 
                the association. 
                  
                When an SCTP association is closed, the Collecting 
                Process MUST discard all templates received over that 
                association and stop decoding IPFIX Messages that use 
                those templates. 
                 
                The Collecting Process normally receives Template 
                Records from the Exporting Process, before receiving 
                Flow or Options Data Records.  The Flow Data Records (or 
                Options Data Records) are then decoded and stored by the 
                Collector. If the Template Records have not been 
                received at the time Flow Data Records (or Options Data 
                Records) are received, the Collecting Process MAY store 
                the Flow Data Records (or Options Data Records) for a 
                short period of time and decode them after the Template 
                Records are received.  A Collecting Process MUST NOT 
                assume that the Data Set and the associated Template Set 
                (or Options Template Set) are exported in the same IPFIX 
                Message. 
                 
                The Collecting Process MUST note the Field ID of any 
                Information Element that it does not understand and MAY 
                discard that Information Element from the Flow Record.  
                The Collecting Process MUST note the size and position 
                of any Vendor Specified Information Element that it does 
                not understand and discard that Information Element from 
                the Flow Record. 
                 
                More that one (Options) Template Set MAY be received in 
                an IPFIX Message. 

                The Collector MUST accept padding in Flow Data Records, 
                Options Data Records and Template Records. 
                 
                The IPFIX protocol has a Sequence Number field in the 
                Export header which increases with each IPFIX Message.  
                A Collector may detect out of sequence, dropped, or 
                duplicate IPFIX Messages by tracking the Sequence 
                Number.  EDITOR’S NOTE: THIS MIGHT CHANGE IF THE 
                SEQUENCE NUMBER IS PER STREAM AND IF THIS IS THE NUMBER 
                OF FLOW RECORDS. A collector SHOULD provide a logging 
                mechanism for tracking out of sequence IPFIX Messages.  
                Such out of sequence IPFIX Messages may be due to 
                Exporter resource exhaustion where it can not transmit 
                messages at their creation rate, an Exporting Process 
                reset, congestion on the network link between the 
                Exporter and Collector, Collector resource exhaustion 
                where it can not process the IPFIX Messages at their 
                arrival rate, out of order packet reception, duplicate 
                packet reception, or an attacker injecting false 
                messages. 
                 
                If a Collecting Process receives a Template Withdraw 
                Message, the Collecting Process MUST delete the 
                corresponding Template Record associated with the 
                specific Exporter and specific Observation Domain, and 
                stop decoding IPFIX Messages that use those Templates.  
             
                A Collecting Process that receives IPFIX Messages from 
                several Observation Domains from the same Exporter MUST 
                be aware that the uniqueness of the Template ID is not 
                guaranteed across Observation Domains.