Subject: RE: [ipfix] padding spec

From: "Thomas Dietz" <[email-masked]>
Date: Tue, 23 Aug 2005 15:50:25 +0200

Dear all,

Do we really need padding on the wire? Since we have variable length IE's
you cannot rely on the fact that a data value will not cross a 4-, 8- or
whatever byte boundary. You have to walk byte by byte anyway.

If the styles of the machines change we need to change padding again. Now we
have 32 or 64 bit machines. Maybe in 2 or 3 years we will have 128 bit
machines. So you recode your application to have even more padding and wast
bandwidth on the wire.

Beware, I am not talking about any internal representation of data, just
about the part that crosses the wires. That you may copy your 4 byte
integer32 into a 8 bytes of memory is another story.

So I would suggest to keep the padding for now but deprecate the use of any
padding.

Best Regards,

Thomas

-- 
Thomas Dietz
Network Laboratories, NEC Europe Ltd., 69115 Heidelberg, Germany

> -----Original Message-----
> From: majordomo listserver 
> [mailto:[email-masked]] On Behalf Of Paul Aitken
> Sent: Tuesday, August 09, 2005 9:40 AM
> To: Juergen Quittek
> Cc: [email-masked]; [email-masked]
> Subject: Re: [ipfix] padding spec
> 
> Juergen,
> 
> > I am fine with changing the name of IE #220 from paddingOneOctet to
> > paddingOctets.
> 
> Or indeed, just "padding" since other IE's don't generally have their 
> length ("One") or type ("Octet") specified in their name.
> 
> 
> > But how do you want to have your other issue solved: "remove the 
> > restriction".  I guess that you address the data type that currently
> > is 'octet'.  Doe you suggest using 'octetArray'?
> 
> Fine.
> 
> Cheers.
> -- 
> Paul Aitken
> Cisco Systems Ltd, Edinburgh, Scotland.
> 
> --
> Help        mailto:[email-masked] and say "help" 
> in message body
> Unsubscribe mailto:[email-masked] and say
> "unsubscribe ipfix" in message body
> Archive     http://ipfix.doit.wisc.edu/archive/
>

---

From: Juergen Quittek <[email-masked]>
Date: Tue, 23 Aug 2005 17:08:55 +0200

We are still working on the initial version of a standard.
It sounds cumbersome to me to have a deprecated feature in there.
Would it be a lot of effort to remove padding completely from the protocol?

Thanks,

    Juergen
-- 
Juergen Quittek        [email-masked]       Tel: +49 6221 90511-15
NEC Europe Ltd.,       Network Laboratories        Fax: +49 6221 90511-55
Kurfuersten-Anlage 36, 69115 Heidelberg, Germany   http://www.netlab.nec.de

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Brian Trammell <[email-masked]>
Date: Tue, 23 Aug 2005 11:17:40 -0400

I'd argue against deprecating padding; specific commentary is inline  
below.

> whatever byte boundary. You have to walk byte by byte anyway.

Not necessarily - a receiving implementation could verify that a  
given template contains no variable length IE's and switch to a non- 
byte-walking codepath to speed up receiving. A sending implementation  
could simply avoid sending templates with variable length IEs when  
not necessary.

> machines. So you recode your application to have even more padding  

The protocol doesn't - nor should it - require padding to any given  
word boundary. If a sending implementation doesn't want to pad,  
that's fine. However, the protocol should also not enjoin  
implementors using IPFIX to communicate between two implementations  
they own to exploit word alignment for performance benefit. The  
burden on receiver implementations to support padding is minimal; I  
don't see what would be gained by its deprecation.

Regards,

Brian

---

From: Juergen Quittek <[email-masked]>
Date: Tue, 23 Aug 2005 18:41:00 +0200

Hi Brian,

Thanks for the comments.  Please find replies inline.

--On 8/23/2005 5:17 PM +0200 Brian Trammell wrote:
>
> I'd argue against deprecating padding; specific commentary is inline  below.
>
> Not necessarily - a receiving implementation could verify that a  given template
> contains no variable length IE's and switch to a non- byte-walking codepath to
> speed up receiving.

I doubt that the achieved speed up justifies the resources required for
implementing and maintaining two parallel code-paths.

> A sending implementation  could simply avoid sending templates
> with variable length IEs when  not necessary.

A funny thing is that we just added a variable-length IE for padding.
It might happen that the only variable-length IE you are using is the one
for padding :-)

> word boundary. If a sending implementation doesn't want to pad,  that's fine.
> However, the protocol should also not enjoin implementors using IPFIX to
> communicate between two implementations  they own to exploit word alignment
> for performance benefit. The burden on receiver implementations to support
> padding is minimal; I don't see what would be gained by its deprecation.

I agree that the burden for the receiver is less.  But a compliant receiver
MUST implement padding because the exporter is free to use it or not.  And
my guess would be that implementing re-alignment at the receiver is even less
burden that implementing padding.  Since IEs in a record may be arbitrarily
aligned anyway, I expect minimal advantages from padding of flow records only,
even if you really care so much about performance that aligned or non-aligned
records would make a difference at all.  Therefore, also on the exporter side
I do not see that implementing padding on the exporter could be worth the effort.

Thanks,

    Juergen

---

From: Andrew Johnson <[email-masked]>
Date: Tue, 23 Aug 2005 19:24:10 +0100

Hi all

There are two padding ideas here and I'm not entiry sure what is being
reference all the time:
  1. The optional padding at the end of the flowset (to 32-bits)
  2. The padding field within the records

I think Thomas wants to deprecate 1, which I'm not really against,
especially since you can't always pad AND it's optional anyway.
I think the advantage of getting rid of it is that the collector
should handle non-aligned records anyway and it removes the area
of likely bugs where a variable-length field CAN make the records
too small to pad, even if they are never sent that small.  i.e.
exporter pads and the collector assumes it doesn't.

Since IPFIX can be used for a lot of things then I think the padding
field is a good thing.  The padding field is of a selectable size so
you can use it for 64-bit, 128-bit etc in the future.  If someone wants
to export records all nicely aligned using padding then they can.

More inline.

>> contains no variable length IE's and switch to a non- byte-walking codepath to

I think certain senders and recievers can be designed to work together
using IPFIX.  It is certainly possible to optimise this solution with padding
all the fields to certain boundries.

> implementing and maintaining two parallel code-paths.

Maybe not in the general case, but that shouldn't be reason enough to remove
it from the protocol.

> It might happen that the only variable-length IE you are using is the one

I wouldn't call the padding a variable-length IE (although I you could
probably use it that way).  Since you can choose the length in the template
I would call it a selectable-length IE.  This means that that a collector
could have aligned and non-aligned paths for different templates that
might work quite well.

>>> machines. So you recode your application to have even more padding and wast

For the flowset padding we will not be changing the optional alignment from
the 32-bit boundry to a higher boundry.  For the padding withing records then
you could optimise your sender/reciever pair for a certain application.

I don't think there is any requirement that a reciever built to handle a
certain application must handle ALL applications.  If the application you
are building a collector for always pads to an 8-byte boundry and never
uses variable-length IEs, then by all means optimise for that use.

> I do not see that implementing padding on the exporter could be worth 

I beleive you are looking at the case of a general purpose reciever.  If you
have a specific application in mind then I don't see the harm in optimising
for that application, as long as you verify your assumptions and don't
contradict the protocol spec.  I think the case where the sender uses padding
to neatly align all the data is a good example.  You can have an optimal path
for this, but you can verify the aligments from the incoming template before
choosing that path.

Andrew

---

From: Juergen Quittek <[email-masked]>
Date: Tue, 23 Aug 2005 23:13:32 +0200

Andrew,

Thanks for your comments.  Please find replies inline.

> could have aligned and non-aligned paths for different templates that

Based on the interop event we gave the padding IE (#210) the data type octetArray
which definitely is a variable-length IE.

> uses variable-length IEs, then by all means optimise for that use.

In this case, please don't call your collector compliant with the IPFIX
protocol specification.  The exporter would be, though.

> for this, but you can verify the aligments from the incoming template before

Standards are about interoperability between different implementations.
However, if a vendor sells me an IPFIX-compliant receiver, I expect from this
software, that is can parse every IPFIX message that was created by a compliant
IPFIX exporter.

If you build a collector that only works well with a certain exporter,
then it is NOT IPFIX-compliant and you do not need a standard at all.

This is different for the exporter.  Here you can streamline your implementation
and only use a small subset of the IPFIX protocol, for example only fixed-length
data types or no padding at all.

Thanks,

    Juergen

---

From: Juergen Quittek <[email-masked]>
Date: Wed, 24 Aug 2005 03:11:06 +0200

Hi Andrew,

My previous message was not very clear in some points.
Here are the replies to the important points of your message.

>   2. The padding field within the records

You are right.  The discussion was not clear in this point.

> exporter pads and the collector assumes it doesn't.

I have no general problem with both ways of padding, (1) padding at
the end of the data/template set and (2) padding with the padding IE.
But we should not have something in the protocol that we consider to
deprecate already at this stage.

The padding IE (1) is fine for padding if we do not use variable-length IEs.
We can use it such that there is no need for additional padding by other
means at the end of the data set.

If we use variable-length IEs in the flow records, then we also need
variable-length padding.  This is also possible with the padding IE,
but has the problem that it cannot have a length of 0 and that it is
potentially confusing that then the length of the padding IE is given
as the number of padding octets minus one.  But also here, we do not
gain much with (2) padding at the end of the data set.

A disadvantage of (1) is that we cannot use it for template sets.
If an IPFIX message contains a template set followed by data sets,
then we can only make sure that the data set is aligned if we use (2).
But since field specifiers are anyway 32-bit aligned, I do not consider
this a real problem.

> to export records all nicely aligned using padding then they can.

I agree with Thomas and you that the padding IE is useful.  It does
not create additional effort at the collector, since the collector can
treat it like any other IE.

So the only padding we should discuss is (2) padding at the end of the
data/template set.

Thanks,

    Juergen

---

From: Brian Trammell <[email-masked]>
Date: Wed, 24 Aug 2005 00:33:21 -0400

Juergen:

Inline replies, as always. :)

On Aug 23, 2005, at 12:41 PM, Juergen Quittek wrote:

>> given template

The application I envision here is "recognizing" that a given IPFIX  
template corresponds to an existing (pre-IPFIX) record format, and  
passing control to existing software to handle same. You may well be  
right, though - I've not done the requisite profiling to back up my  
intuition that this is worth the effort.

> It might happen that the only variable-length IE you are using is  

True enough :) ... We do still get to define fixed length  
paddingOctets octet arrays for padding, though, correct? (I'm  
primarily interested in using fixed padding for word alignment  
purposes, especially in using IPFIX templates to describe existing  
word-aligned record formats).

>> that's fine.

As above, from the exporter's point of view, it may be worth it if  
the exporter already has word-aligned padded data to work with.

The real point I'm trying to make here (and one I think has been  
adequately made by others) is that supporting padding requires little  
effort, is already in the protocol and information model, and adds  
some flexibility to the protocol in how information is represented  
that may incidentally ease implementation.

Thanks,

Brian