Subject: RE: [ipfix] padding spec

From: "Thomas Dietz" <[email-masked]>
Date: Wed, 24 Aug 2005 09:35:10 +0200

Hi Brian,

Find my comments inline...

Regards,

Thomas

> -----Original Message-----
> From: Brian Trammell [mailto:[email-masked]] 
> Sent: Wednesday, August 24, 2005 6:33 AM
> To: Juergen Quittek
> Cc: Thomas Dietz; Paul Aitken; [email-masked]; 
> [email-masked]
> Subject: Re: [ipfix] padding spec
> 
> Juergen:
> 
> Inline replies, as always. :)
> 
> On Aug 23, 2005, at 12:41 PM, Juergen Quittek wrote:
> 
> > Hi Brian,
> >
> > Thanks for the comments.  Please find replies inline.
> >
> > --On 8/23/2005 5:17 PM +0200 Brian Trammell wrote:
> >
> >>
> >> I'd argue against deprecating padding; specific commentary is  
> >> inline  below.
> >>
> >> On Aug 23, 2005, at 9:50 AM, Thomas Dietz wrote:
> >>
> >>
> >>> Dear all,
> >>>
> >>> Do we really need padding on the wire? Since we have variable
> >>> length IE's
> >>> you cannot rely on the fact that a data value will not cross a 4-,
> >>> 8- or
> >>> whatever byte boundary. You have to walk byte by byte anyway.
> >>>
> >>
> >> Not necessarily - a receiving implementation could verify that a   
> >> given template
> >> contains no variable length IE's and switch to a non- 
> byte-walking  
> >> codepath to
> >> speed up receiving.
> >>
> >
> > I doubt that the achieved speed up justifies the resources 
> required  
> > for
> > implementing and maintaining two parallel code-paths.
> 
> The application I envision here is "recognizing" that a given IPFIX  
> template corresponds to an existing (pre-IPFIX) record format, and  
> passing control to existing software to handle same. You may well be  
> right, though - I've not done the requisite profiling to back up my  
> intuition that this is worth the effort.
> 
> >> A sending implementation  could simply avoid sending templates
> >> with variable length IEs when  not necessary.
> >>
> >
> > A funny thing is that we just added a variable-length IE 
> for padding.
> > It might happen that the only variable-length IE you are using is  
> > the one
> > for padding :-)
> 
> True enough :) ... We do still get to define fixed length  
> paddingOctets octet arrays for padding, though, correct? (I'm  
> primarily interested in using fixed padding for word alignment  
> purposes, especially in using IPFIX templates to describe existing  
> word-aligned record formats).

The only IE for padding I saw in the drafts so far was paddingOctet which
had a fixed length of 1 byte (and was not extendible). So what you describe
here (fixed length IE) was not in any draft. Maybe we can discuss if we want
the fixed length IE or the variable length IE or both IE's for padding. I
personally think only the fixed length IE makes sense at all.

> 
> >>> If the styles of the machines change we need to change padding
> >>> again. Now we
> >>> have 32 or 64 bit machines. Maybe in 2 or 3 years we will have  
> >>> 128 bit
> >>> machines. So you recode your application to have even more padding
> >>> and wast
> >>> bandwidth on the wire.
> >>>
> >>
> >> The protocol doesn't - nor should it - require padding to any given
> >> word boundary. If a sending implementation doesn't want to pad,   
> >> that's fine.
> >> However, the protocol should also not enjoin implementors using  
> >> IPFIX to
> >> communicate between two implementations  they own to exploit word  
> >> alignment
> >> for performance benefit. The burden on receiver 
> implementations to  
> >> support
> >> padding is minimal; I don't see what would be gained by its  
> >> deprecation.
> >>
> >
> > I agree that the burden for the receiver is less.  But a compliant  
> > receiver
> > MUST implement padding because the exporter is free to use it or  
> > not.  And
> > my guess would be that implementing re-alignment at the 
> receiver is  
> > even less
> > burden that implementing padding. Since IEs in a record may be  
> > arbitrarily
> > aligned anyway, I expect minimal advantages from padding of flow  
> > records only,
> > even if you really care so much about performance that aligned or  
> > non-aligned
> > records would make a difference at all.  Therefore, also on the  
> > exporter side
> > I do not see that implementing padding on the exporter could be  
> > worth the effort.
> 
> As above, from the exporter's point of view, it may be worth it if  
> the exporter already has word-aligned padded data to work with.

But you only can add padding at the end of the record!
* So if you pad the data corresponding to a IE (e.g. You have a unsigned16
IE and you pad it to 32 bit) you have to remove this padding anyway if the
IE is not the last in the record.
* If you have padding at the end of the data then it is only one small
operation to remove that padding.

> 
> The real point I'm trying to make here (and one I think has been  
> adequately made by others) is that supporting padding 
> requires little  
> effort, is already in the protocol and information model, and adds  
> some flexibility to the protocol in how information is represented  
> that may incidentally ease implementation.
> 
> Thanks,
> 
> Brian
> 
>

---

From: Juergen Quittek <[email-masked]>
Date: Wed, 24 Aug 2005 11:58:37 +0200

Dear all,

This is a try to summarize the discussion on padding.

Since the last version (-10) of the info model we have an Information Element
for padding of type octetArray.  The protocol allows using encoding it as a
fixed-length array as well as a variable length array.

As a reminder, an IPFIX message is structures as follows:
  - IPFIX message    contains  sets
  - set              contains  records
  - data record      contains  Information Elements (IEs)
  - template record  contains  IE specifiers

Alignment of IEs within a data record

The padding IE gives us flexible means for aligning IEs within a data record.
Aligning within a data record can be useful, because you may very easily
convert internal data structures into flow records at the exporter and
vice versa at the collector.

Alignment of IE specifiers within a template record

We do not have means for aligning IE specifiers within template records,
but there is a limited need for it and IE specifiers are aligned to 32-bit
address boundaries anyway.

Alignment of records within a set

We do not have explicit means for aligning records within a set.  However,
for template records the need is limited and they are aligned to 32-bit
boundaries anyway.  For data records we can use padding IEs at the end
of a the record extending the length of the record to the next alignment
boundary.  Note that this way also the sets are aligned (up to 32-bit
boundaries) within an IPFIX message, because also at the end of the last
records within a set there is a padding IE.

Alignment of sets within an IPFIX message

If records are already aligned within a set by using padding IEs, then
this alignment is probably already achieved.  But for aligning sets within
an IPFIX message, the protocol further means, as described in section 3.3.1
of the protocol document.  This padding mechanism can be applied even if
the records within sets are not aligned.

Conclusion

In my view, the main point of our discussion is whether it makes sense
having this additional padding mechanism.  I see advantages of padding
within a record for aligning internal data structures and flow records.

But I see very little advantage in aligning sets if the sets and their
contained flow records are not aligned also.

Consequently, I suggest removing the padding at the end of a set from
the protocol specification.

Thanks,

    Juergen
-- 
Juergen Quittek        [email-masked]       Tel: +49 6221 90511-15
NEC Europe Ltd.,       Network Laboratories        Fax: +49 6221 90511-55
Kurfuersten-Anlage 36, 69115 Heidelberg, Germany   http://www.netlab.nec.de

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Benoit Claise <[email-masked]>
Date: Wed, 31 Aug 2005 15:56:55 +0200

Juergen and all,

> the protocol specification.

I understand your way of thinking and agree with you when you wrote "But 
I see very little advantage in aligning sets if the sets and their 
contained flow records are not aligned also. "
However, I'm not sure if your conclusion is the right one.
I'm wondering why we should remove some flexibility from the IPFIX 
protocol specifications, some flexibility that requires little effort.
Don't forget that the padding at the end of the Set is optional. See 
section 3.3.1 "The Exporting Process MAY insert some padding octets"
I don't have right now a good example of why this would be needed. But 
maybe in the future?
What about 
http://www.ietf.org/internet-drafts/draft-trammell-ipfix-file-00.txt? 
Would a Set alignment provide some benefits? I'm not sure yet, 
potentially. Why close the door now?

With Paul Aitken, we went through padding again in the protocol draft.
We would like to copy a sentence from [IPFIX-INFO] that should be part 
of [IPFIX-PROTO]

      Padding 
         The Exporting Process MAY insert some padding octets, so that 
         the subsequent Set starts at an aligned boundary.  Padding 
         MUST be composed of zero (0) octets.  The padding length MUST 
         be shorter than any allowable Record in this Set.  
         _If padding of the IPFIX Message is desired in combination
         with very short Records, then the padding Information Element
         'paddingOctets' [IPFI-INFO] can be used for padding Flow Records such 
         that their length is increased to a multiple of 4 or 8 octets._
         Because Template Sets are always 4-octet aligned by definition  
         padding is only needed in case of other alignments e.g. on 8- 
         octet boundaries. 

Further more, we would like to add a clarification in the section 9 "
The Collecting Process's Side"

   The Collector MUST accept padding in Data Records and Template 
   Records.  _The padding size is the Set Length minus the size of 
   the Set Header (4 octets for the Set ID and the Set Length), 
   modulo the Record size deduced from the Template Record._

Any objections?

Regards, Benoit.

---

From: Benoit Claise <[email-masked]>
Date: Wed, 31 Aug 2005 18:00:57 +0200

Dear all,

Please state your point of view, as we're at the end of the last-call.
Should the protocol draft continue to specify the optional padding at 
the end of Set (see section 3.3.1)? Or should this be removed?

Regards, Benoit.

> I understand your way of thinking and agree with you when you wrote 
> "But I see very little advantage in aligning sets if the sets and 
> their contained flow records are not aligned also. "

---

From: Brian Trammell <[email-masked]>
Date: Wed, 31 Aug 2005 13:29:46 -0400

Not required at the exporter, minimal effort required at the  
collector, no document effort required to leave it in - I say we keep  
3.3.1.

- Brian

On Aug 31, 2005, at 12:00 PM, Benoit Claise wrote:

> Should the protocol draft continue to specify the optional padding  
> at the end of Set (see section 3.3.1)? Or should this be removed?
>
> Regards, Benoit.
>
>> Juergen and all,
>>> Dear all,
>>>
>> I understand your way of thinking and agree with you when you  
>> wrote "But I see very little advantage in aligning sets if the  
>> sets and their contained flow records are not aligned also. "
>> Don't forget that the padding at the end of the Set is optional.  
>> See section 3.3.1 "The Exporting Process MAY insert some padding  
>> octets"
>> I don't have right now a good example of why this would be needed.  
>> But maybe in the future?
>> What about http://www.ietf.org/internet-drafts/draft-trammell- 
>> ipfix-file-00.txt? Would a Set alignment provide some benefits?  
>> I'm not sure yet, potentially. Why close the door now?
>>
>>
>> We would like to copy a sentence from [IPFIX-INFO] that should be  
>> part of [IPFIX-PROTO]
>> Padding The Exporting Process MAY insert some padding octets, so  
>> that the subsequent Set starts at an aligned boundary. Padding  
>> MUST be composed of zero (0) octets. The padding length MUST be  
>> shorter than any allowable Record in this Set. If padding of the  
>> IPFIX Message is desired in combination with very short Records,  
>> then the padding Information Element 'paddingOctets' [IPFI-INFO]  
>> can be used for padding Flow Records such that their length is  
>> increased to a multiple of 4 or 8 octets. Because Template Sets  
>> are always 4-octet aligned by definition padding is only needed in  
>> case of other alignments e.g. on 8- octet boundaries. Further  
>> more, we would like to add a clarification in the section 9 " The  
>> Collecting Process's Side" The Collector MUST accept padding in  
>> Data Records and Template Records. The padding size is the Set  
>> Length minus the size of the Set Header (4 octets for the Set ID  
>> and the Set Length), modulo the Record size deduced from the  
>> Template Record.
>> Any objections?
>>
>> Regards, Benoit.
>

---

From: Paul Aitken <[email-masked]>
Date: Wed, 31 Aug 2005 21:47:05 +0100

Brian Trammell wrote:

> Not required at the exporter, minimal effort required at the  collector, 
> no document effort required to leave it in - I say we keep  3.3.1.

I'm very much in agreement.

Padding isn't mandated in the existing drafts; it's purely optional. So 
why go to the trouble of removing it? It makes the protocol less 
flexible and requires more work which delays us even further...

If we removed it then we would have to write some new text mandating 
that the set length MUST be the exact size of the set, no more or less.

And we'd have to decide whether a collector may, should, or must accept 
or reject records in sets with incorrect lengths.

Whereas it's zero-effort, zero-confusion, max-clarity and 
max-flexibility to leave it alone.

Cheers.
-- 
Paul Aitken
Cisco Systems Ltd, Edinburgh, Scotland.

---

From: Juergen Quittek <[email-masked]>
Date: Wed, 31 Aug 2005 23:38:00 +0200

Hi Brian,

> no document effort required to leave it in - I say we keep  3.3.1.

You are right: it does not cost us much keeping it.

You just forgot to mentions that it is of no practical use anymore
since we introduced the padding IE.

I am fine with still keeping it, but only because it is there
and removing it would further delay the standardization process.

We can make a recommendation not to use it when we write the IPFIX
implementation guidelines.

Thanks,

    Juergen

---

From: Paul Aitken <[email-masked]>
Date: Wed, 31 Aug 2005 22:53:52 +0100

Juergen,

> since we introduced the padding IE.

Why do you say that?

I consider that padding with the padding IE is *explicit* padding, 
because you clearly state "I am adding padding here".

Whereas padding at the end of a set by making the set length longer than 
(the set header size) + (the number of records) x (the record size) is 
*implicit* padding, because you don't say anywhere that you're doing 
this; it's just extra, trailing data in the set that the collector has 
to deal with.

We might not see any use for this today, but who's to say what might 
happen in future? eg, if someone invents a transport which is optimised 
for 1000-octet blocks? Then we might be quite glad we had the foresight 
not to disallow this possibility.

> implementation guidelines.

Rather, I would say "to use it appropriately", where "appropriately" is 
defined by the implimentation and customer requirements.

Cheers.

---

From: Juergen Quittek <[email-masked]>
Date: Thu, 01 Sep 2005 00:33:24 +0200

Hi Paul,

>> since we introduced the padding IE.

Because I think so.

> because you clearly state "I am adding padding here".

Yes, and it does not need any special support by the collector,
because you just use an ordinary IE.

> *implicit* padding, because you don't say anywhere that you're doing this;
> it's just extra, trailing data in the set that the collector has to deal with.
>
> We might not see any use for this today,

Thank you for sharing my view that there is
no use case for this way of padding today.

>                                          but who's to say what might

Sorry, I'm not foreseeing it.

> Rather, I would say "to use it appropriately", where "appropriately"
> is defined by the implimentation and customer requirements.

Agreed.

Thanks,

    Juergen

---

From: Paul Aitken <[email-masked]>
Date: Wed, 31 Aug 2005 23:42:52 +0100

Juergen,

> Because I think so.

Indeed, it's the best reason. But then it begs the question "and why do 
you think so?"

> no use case for this way of padding today.

When I say "We", I speak only for my team: it's easier for us not to 
impliment this optional part of the specification.

While it's good that we are also in agreement with you, I cannot speak 
for anyone else - obviously they must do that for themselves - and I 
cannot predict with much accuracy what people may need of the protocol 
tomorrow. So I don't like to place undue restrictions where they are not 
necessary.

> Sorry, I'm not foreseeing it.

I do not see that as any reason to mandate against extra flexibility in 
the protocol.

Cheers.

---

From: Juergen Quittek <[email-masked]>
Date: Thu, 01 Sep 2005 01:42:33 +0200

Hi Paul,

--On 8/31/2005 11:42 PM +0100 Paul Aitken wrote:

> Juergen,
>
>>> Why do you say that?
>>
>> Because I think so.
>
> Indeed, it's the best reason. But then it begs the
> question "and why do you think so?"

I tried to explain it in the first message with subject
'Padding summary'.  But the main reason why I think it is useless
is that I do not see any concrete use case (as you do as well).

> When I say "We", I speak only for my team: it's easier for us not
> to impliment this optional part of the specification.
>
> While it's good that we are also in agreement with you, I cannot
> speak for anyone else - obviously they must do that for themselves
> - and I cannot predict with much accuracy what people may need of
> the protocol tomorrow. So I don't like to place undue
> restrictions where they are not necessary.

I am not arguing for restrictions.
I am arguing that there is a feature in the protocol
that does not have a use case (at least not yet).

> I do not see that as any reason to mandate against extra flexibility in the protocol.

The common procedure for standardizing protocol features is first identifying
requirements based on scenarios and use cases.  Without any use case, I do not
see a requirement for the padding mechanism we have in the protocol document.
Consequently, I do not see any need for standardizing this feature.

Thanks,

    Juergen