Subject: [ipfix-protocol] PROTO-39: Template ID wrap up

From: Benoit Claise <[email-masked]>
Date: Mon, 22 Nov 2004 19:31:08 +0100

Dear all,

I'm trying to find a solution for this issue.
   PROTO-39 : what is happening when we reach the maximum number of 
   Template ID? Wrap around? 

Let me propose some text, to be placed in the section 12 "Template Management"

    In order to delete an allocated Template, the Template MUST be
    withdrawn through the use of a Template Withdraw Message. The
    Template Withdraw Message MUST not be sent until sufficient time has
    elapsed to allow the Collecting Process to receive and process the
    last data record using this Template information.

    The Template ID from a withdrawn Template MUST NOT be reused until
    sufficient time has elapsed to allow for the Collecting Process to
    receive and process the Template withdraw message.

    A Template Withdraw Message is Template Record for that Template ID
    with a Field Count of 0.

Your feedback is welcome.

Regards,  Benoit.

---

From: Benoit Claise <[email-masked]>
Date: Tue, 23 Nov 2004 07:34:29 +0100

Mark,

You forgot that
- with SCTP-PR, we send the templates on the reliable stream and the 
flow data records potentially on a different stream
   Hence some potential re-ordering
- with UDP, some potential re-ordering.

If you simply change the Template definition, you could confuse the 
Collecting Process.
Example: you change the Template definition from bytes count to packets 
count, and the new Template definition arrives after just after the 
first associated flow data record... what will the Collecting Process do 
with the first flow data record??? It will conclude that this is a bytes 
count, and it's not!

Regards, Benoit.

> Hi Benoit,
> I think we do not need a withdraw message in this case. Just sending a 
> new template with the former used ID should be enough.
> The collector can delete the old template after receiving a new
> template with the same ID.
>
> A withdraw message might help the exporter to signal that a specific
> template will not be used anymore. Then the collector would be able to
> free allocated resources.
>
> Regards,
> Lutz

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Lutz Mark <[email-masked]>
Date: Tue, 23 Nov 2004 10:43:31 +0100

> - with UDP, some potential re-ordering.

ok, but we talk in the context of the wrap around of
a 16 bit template id. There will be no re-ordering.

> with the first flow data record??? It will conclude that this is a bytes 

This is a little bit constructed.

ok, the transmission of the withdraw message makes it waterproof.

what about:

     Disused Templates SHOULD be deleted. Prior to reuse a Template ID
     the disused Template MUST be deleted.
     In order to delete an allocated Template, the Template is withdrawn
     through the use of a Template Withdraw Message.

     The Template Withdraw Message MUST not be sent until sufficient time has
     elapsed to allow the Collecting Process to receive and process the
     last data record using this Template information.

Regards,
Lutz

---

From: Benoit Claise <[email-masked]>
Date: Mon, 29 Nov 2004 11:45:12 +0100

Mark,

> a 16 bit template id. There will be no re-ordering.

I expressed myself badly. I wanted to say: packet arrival order issue, 
so the new template definition arriving after the first flow record.
And this influences the fact that we can't just redefine a template 
definition.

>> Example: you change the Template definition from bytes count to 
>> packets count, and the new Template definition arrives after just 
>> after the first associated flow data record... what will the 
>> Collecting Process do with the first flow data record??? It will 
>> conclude that this is a bytes count, and it's not!
>
>

I like your proposal. Augmenting the text with some extra information, I 
propose:

    The Template Withdraw Message MUST not be sent until sufficient time 
has
    elapsed to allow the Collecting Process to receive and process the
    last data record using this Template information.

    A Template Withdraw Message is Template Record for that Template ID
    with a Field Count of 0. The format of the Template Withdrawal Message
    is shown in figure X.

       0                   1                   2                   3   
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |       Set ID = (2 or 3)       |          Length = 8           |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |          Template ID          |        Field Count = 0        |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

         Figure X: Template Withdrawal Message format

   The Set ID field MUST contain the value 2 for Template Set withdrawal, and 
   the value 3 for Options Template Set. 

   Multiple Template ID MAY be withdrawn with a single Template Withdrawal
   Message: in that case, padding MAY be used.

Regards, Benoit.

---

From: Stewart Bryant <[email-masked]>
Date: Mon, 29 Nov 2004 13:10:14 +0000

> And this influences the fact that we can't just redefine a template 

BTW I don't like the use of the term "wrap of the 16 bit template id"
that we (the group, including me) have used from time to time.

They are template identifiers, not template sequence numbers and
may be allocated in any order. The issue is not id wrap, it
is id reuse. Perhaps we need a couple of lines in the spec to
say that they can be allocated in any order?

Stewart

>   The Set ID field MUST contain the value 2 for Template Set withdrawal, 
> and   the value 3 for Options Template Set.

---

From: Benoit Claise <[email-masked]>
Date: Mon, 29 Nov 2004 16:24:34 +0100

Stewart Bryant wrote:

>
>
> Benoit Claise wrote:
>
>> Mark,
>>
>>>
>>>> You forgot that
>>>> - with SCTP-PR, we send the templates on the reliable stream and 
>>>> the flow data records potentially on a different stream
>> I expressed myself badly. I wanted to say: packet arrival order 
>> issue, so the new template definition arriving after the first flow 
>> record.

You're right. Actually the protocol issue should have been "Template ID 
reuse... potentially due Template ID running out of values"
The new I just posted in the email "PROTO-25, 31, 32, and 33: template 
management, collecting process and transport" contains:
The Exporting Process assigns and maintains the
Template IDs for the Exporter’s Observations Domains. A
newly created Template Record is assigned an unused
Template ID by the Exporting Process.

Do we need something else?

Regards, Benoit.

>> I like your proposal. Augmenting the text with some extra 
>> information, I propose:
>>
>> 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 
>> 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>> The Set ID field MUST contain the value 2 for Template Set 
>> withdrawal, and the value 3 for Options Template Set.

---

From: Benoit Claise <[email-masked]>
Date: Mon, 29 Nov 2004 17:24:24 +0100

>> with a Field Count of 0. The format of the Template Withdrawal Message
One more precision.
"The Template Withdraw Message withdraws the Template ID for the Source 
ID specified in the IPFIX Message header."

One more comment.
 From the previous posted called " [ipfix-protocol] PROTO-25, 31, 32, 
and 33: template management, collecting process and transport"

                If the Exporting Process restarts, the SCTP association 
                MUST be shutdown and restarted. When the Exporting 
                Process restarts, all Template assignments are lost and 
                Template IDs MUST be re-assigned. If the Metering 
                Process restarts, the Exporting Process MUST either 
                reuse the previously assigned Template ID for each 
                Template, or it MUST withdraw the previously issued 
                Template IDs by sending Template Withdraw Message(s) 
                before reusing them. EDITOR’S NOTE: A TEMPLATE WITHDRAW 
                MESSAGE THAT WITHDRAW ALL THE TEMPLATE FOR A SPECIFIC 
                SOURCE ID MIGHT BE USEFUL. 

We could use the following to withdraw all Templates for the Source ID 
specified in the IPFIX message header
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 2 | Length = 8 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 2 | Field Count = 0 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

We could use the following to withdraw all Options Templates for the 
Source ID specified in the IPFIX message header

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Set ID = 3 | Length = 8 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Template ID = 3 | Field Count = 0 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Feedback?

Regards, Benoit.