Subject: [ipfix-protocol] minor Template Management MAY/SHOULD question

From: Brian Trammell <[email-masked]>
Date: Fri, 21 Jul 2006 16:55:49 -0400

All,

Paragraph 6 of PROTO-22 section 8 states "The Exporting Process MAY  
transmit the Template Set and Options Template Set in advance of any  
Data Sets that use that (Options) Template ID, to ensure that the  
Collector has the Template Record before receiving the first Data  
Record"... Should this MAY not instead be a SHOULD? We already state  
the Template Sets SHOULD be sent "as soon as they are created", which  
implies they SHOULD be sent before data records anyway.

Regards,

Brian

---

From: Paul Aitken <[email-masked]>
Date: Sat, 22 Jul 2006 08:59:28 +0100

Brian,

> Record"... Should this MAY not instead be a SHOULD?

No, there's another allowance:

    If the Template
    Records have not been received at the time Data Records are
    received, the Collecting Process MAY store the Data Records for a
    short period of time and decode them after the Template Records are
    received.

> We already state  

Define sent?

If all the templates are being exported immediately upon startup, then 
it's possible that the necessary template is stuck in a transmit queue 
while the data record, although transmitted later on a different stream, 
is actually received ahead of the template.

Cheers.
-- 
Paul Aitken
Cisco Systems Ltd, Edinburgh, Scotland.

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Brian Trammell <[email-masked]>
Date: Sat, 22 Jul 2006 13:36:46 -0400

On Jul 22, 2006, at 3:59 AM, Paul Aitken wrote:

> Brian,
>
>> Paragraph 6 of PROTO-22 section 8 states "The Exporting Process  
>> MAY  transmit the Template Set and Options Template Set in advance  
>> of any  Data Sets that use that (Options) Template ID, to ensure  
>> that the  Collector has the Template Record before receiving the  
>> first Data  Record"... Should this MAY not instead be a SHOULD?
>

These would seem to be compatible... The exporting process is of  
course not required to ensure that templates are received and  
processed before the data they describe, as, for example, head of  
line blocking in SCTP stream 0 may cause partially- or un-reliable  
data in other streams to be processed by the collector first.

But we presently have two MAYs here - either the exporter or the  
collector MAY make an effort to ensure that templates are available  
for received data to be initially decoded (keep in mind that the  
template is required even to find record boundaries; cached sets  
without template information MUST be buffered as full byte arrays at  
the collector side). The protocol should make a choice here as to  
which side of the exporter/collector interaction primary  
responsibility for ensuring template availability falls on; one of  
these MAYs should be a SHOULD.

I'm recommending here that we place the burden on the exporter, as  
simply requiring the exporter to make an effort to transmit templates  
before related data seems less onerous than requiring collector  
implementers to have an extremely good reason not to design in set  
caching.

(This came to my attention while looking over the implementation  
guidelines, which as of -02 restates the present exporter-side MAY as  
a SHOULD in section 4.2.1 para 1. I was thinking about the conflict  
between the two documents when I realized I thought the guidelines  
were more correct on this point).

>> We already state  the Template Sets SHOULD be sent "as soon as  
>> they are created", which  implies they SHOULD be sent before data  
>> records anyway.
>
> Define sent?

IPFIX-PROTO -22 section 8 para 6: "New Template Records SHOULD be  
transmitted as soon as they are created," where "created" is not well- 
defined IMO.

Cheers,

Brian

---

From: Andreas Bourges <[email-masked]>
Date: Mon, 24 Jul 2006 11:50:02 +0200

Hi Paul,

...I'm not following the discussions here very closely - so maybe my question 
has already bee asked (sorry if that's the case...) - in any case it might be 
a bit off-topic in this stage of the discussion...

Has anybody thought on storing the template information for a *short* (in the 
range of (seconds) period of time on the collecting process side (on disk for 
example) - to make sure that in the case of a collecting process restart the 
template information would be right in place again? Or is it too risky (since 
the exporting device may undergo a restart at the same and template-id 
assignment would not be consistent any more...).

Would that be allowed?

Cheers,

Andy

On Monday 24 July 2006 11:09, Paul Aitken wrote:
> >>> transmit the Template Set and Options Template Set in advance  of
> >>> any  Data Sets that use that (Options) Template ID, to ensure  that
> >>> the  Collector has the Template Record before receiving the  first
> >>> Data  Record"... Should this MAY not instead be a SHOULD?
> >>
> > These would seem to be compatible... The exporting process is of  course
> > not required to ensure that templates are received and  processed before
> > the data they describe,
>
> How would or could it anyway? There'd need to be an ACK from the CP, but
> IPFIX is an unacknowledged protocol.
>
> > as, for example, head of  line blocking in SCTP
> > stream 0 may cause partially- or un-reliable  data in other streams to
> > be processed by the collector first.
>
> Yep.
>
> > collector MAY make an effort to ensure that templates are available  for
> > received data to be initially decoded
>
> How can the collector ensure anything?
>
> > (keep in mind that the  template
> > is required even to find record boundaries;
>
> Indeed.
>
> > cached sets  without
> > template information MUST be buffered as full byte arrays at  the
> > collector side).
>
> Not so - they could be compressed or even discarded.
>
> > The protocol should make a choice here as to  which
> > side of the exporter/collector interaction primary  responsibility for
> > ensuring template availability falls on; one of  these MAYs should be a
> > SHOULD.
>
> The collector can't ensure templates are available since it's
> essentially a passive device. All it can do is wait until they become
> available, either storing or discarding received records until then.
>
> The exporter may well export templates ahead of data - but the collector
> may not even have been connected, may have been connected but not
> listening, or may have since lost the template information (out of
> memory, restart, software bug).
>
> Ideally the collector would be up and running before the exporter - but
> more typically we need to deal with the situation where you bring a new
> collector online without restarting your exporter(s).
>
> > guidelines, which as of -02 restates the present exporter-side MAY as  a
> > SHOULD in section 4.2.1 para 1.
>
> That'll be the difference between a standard and a guideline then.
>
> > I was thinking about the conflict
> > between the two documents when I realized I thought the guidelines  were
> > more correct on this point).
>
> Regardless, guidelines are not standards.
>
> >>> We already state  the Template Sets SHOULD be sent "as soon as  they
> >>> are created", which  implies they SHOULD be sent before data  records
> >>> anyway.
> >>
> >> Define sent?
> >
> I believe the point is that while data can be buffered at the EP,
> templates should not be buffered or delayed in any way, but exported as
> soon as possible.
>
> Cheers.

- -- 
 Andreas Bourges
 CCIE #6947 (R&S,C&S)
 GSM: +49 178 7072022
 e-mail: [email-masked]
- -----------------------------------
 IsarNet AG
 Terminalstrasse Mitte 18
 85356 München
 Tel.: 07000-isarnet
 Fax : 089-97007-200
 http://www.isarnet.de
- -----------------------------------

---

From: Paul Aitken <[email-masked]>
Date: Mon, 24 Jul 2006 10:09:35 +0100

Brian,

Yep.

Indeed.

Cheers.

---

From: Paul Aitken <[email-masked]>
Date: Mon, 24 Jul 2006 11:15:35 +0100

Andy,

> Has anybody thought on storing the template information for a *short*
> (in the range of (seconds) period of time on the collecting process
> side (on disk for example)

Brian certainly has some ideas on IPFIX storage.

> - to make sure that in the case of a collecting process restart the 
> template information would be right in place again? Or is it too
> risky (since the exporting device may undergo a restart at the same
> and template-id assignment would not be consistent any more...).

It's a bit risky, because (supposing UDP) the CP wouldn't know if
the EP had restarted while the CP was down, so different IDs could now
be in use.

However, a simple device (eg, using UDP export) might always use the
same fixed template IDs, so it might work.

Also, after a reset, the EP may send templates more frequently than the
normal refresh - from IPFIX-PROTO 10.3.6:

    In the event of configuration changes, the Exporting Process SHOULD
    send multiple copies of the new template definitions, in different
    IPFIX Messages, at an accelerated rate.

So it may be possible for the CP to determine that an EP reset did or
did not occur - though it's not a recommended or reliable method!

> Would that be allowed?

Off the top of my head, I can't think of anything expressly forbidding it...