Subject: [PSAMP] PSAMP-INFO: padding of reported packet sections

From: Juergen Quittek <[email-masked]>
Date: Tue, 15 Jul 2008 07:39:54 +0200

Dear all,

When discussing how to solve the issues raised by the AD review of
draft-ietf-psamp-info the authors identified new issues with the
encoding of sections of a raw packet.

I try to summarize the first one in this email.
Paul, please correct me if I misunderstood the points
you made last week:

Currently all Information Elements containing sections (chunks)
of sampled packets (ipHeaderPacketSection, ipPayloadPacketSection,
dataLinkFrameSection, mplsLabelStackSection, mplsPayloadPacketSection)
contain a phrase in their description saying "The data for this
field MUST NOT be padded."

The obvious reason for this statement is that if you use padding
within these sections, then a receiver of the IE might not be able
to distinguish packet data from padding octets.

However, there are also reasons for allowing padding, particularly
if you do not want to use variable length encoding of packet
sections.  You may want to use fixed length encoding matching
the sampling length you apply to the packet or to its payload.
Then padding could be applied in case packets are shorter than
the sampling length.  In this case, the receiver of the section
would also need further information such as the packet length,
payload length, etc. in order to identify which octets belong to
the packet and which are padding, but this information could be
provided by further IEs that are already available from IPFIX-INFO.

Do people in the WG think we should allow padding of reported
packet sections within an IE or shall we rather stick with the
current rule that section IEs MUST NOT be padded?

Thanks,

    Juergen
-- 
Juergen Quittek        [email-masked]        Tel: +49 6221 4342-115
NEC Europe Limited,    Network Laboratories        Fax: +49 6221 4342-155
Kurfuersten-Anlage 36, 69115 Heidelberg, Germany   http://www.netlab.nec.de
Registered Office: NEC House, 1 Victoria Road, London W3 6BL, UK
Registered in England 2832014

_______________________________________________
PSAMP mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/psamp

---

From: Juergen Quittek <[email-masked]>
Date: Tue, 15 Jul 2008 10:22:55 +0200

Dear all,

Here is another issue for PSAMP-INFO:

In the current version of draft-ietf-psamp-info there
are two Information Elements related to link layer frames:
dataLinkFrameSize and dataLinkFrameSection.

I would like to discuss whether it is of advantage to have
generic dataLinkFrame* IEs rather than specific ones.

A generic one is nice, if you are reporting different
types of data link layer frames.

But if you don't know which data link layer was present,
interpretation of the IE becomes tricky.   The most common
case is probably an Ethernet frame.  Also for 802.11 the
concept of a frame is pretty clear.

It gets more difficult if you use MPLS. What is then the
data link frame?  Or for 802.16 (WiMAX, where multiple packets
for multiple receivers may be contained in a single frame (burst).

Things would get much clearer is we used link layer-specific
IEs instead of a dataLinkFrame* IEs, such as ethernetFrameSection,
wlanFrameSection, wimaxBurstSection, etc.

Any opinions on this issue?

Thanks,

    Juergen

---

From: Paul Aitken <[email-masked]>
Date: Tue, 15 Jul 2008 13:16:37 +0100

Juergen,

> current rule that section IEs MUST NOT be padded?

If packet sections must not be padded then either:

1. we mandate that variable length encoding MUST be used, or

2. we accept that templates may be consumed at an alarming rate
    to cope with the various different sized packet sections.

    Indeed, templates may be exhausted exponentially when multiple
    packet section fields are used.

Since I don't want (2) and (1) seems unnecessarily strict, I came to the 
conclusion that we must allow padding.

And as you say yourself, the IEs required to identify padding are 
already available.

Thanks.
-- 
Paul Aitken
Cisco Systems Ltd, Edinburgh, Scotland.
_______________________________________________
PSAMP mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/psamp

---

From: Gerhard Muenz <[email-masked]>
Date: Tue, 15 Jul 2008 17:51:58 +0200

Paul,

>    packet section fields are used.

Template exhaustion is not an implicit consequence of not mandating VLE.
If fixed length encoding is used, the Exporter may define a small number
of Templates with packet section fields of different lengths and choose
the one with the maximum length not exceeding the packet boundary.

Of course, it is possible to define many different Templates which only
differ in the length of the packet section field. But there are other
ways of defining many Templates in a stupid way which lead to Template
exhaustion as well. Therefore, I do not think that Template exhaustion
is a problem which is specific to the padding issue.

> And as you say yourself, the IEs required to identify padding are

My remark is not in favor or against allowing padding. I just do no
think that your argumentation forces us to allow padding.

Regards,
Gerhard

-- 
Dipl.-Ing. Gerhard Münz
Computer Networks and Internet
Wilhelm Schickard Institute for Computer Science
University of Tuebingen
Sand 13 (Room B309), D-72076 Tuebingen, Germany
Phone:  +49 7071 29-70534       Fax: +49 7071 29-5220
E-mail: [email-masked]
WWW:    http://net.informatik.uni-tuebingen.de/~muenz

---

From: Paul Aitken <[email-masked]>
Date: Tue, 15 Jul 2008 17:08:44 +0100

Gerhard,

> the one with the maximum length not exceeding the packet boundary.

Absolutely!

But then some data may be lost from the end of the packet, so we'd have 
a similar issue to when padding is used - ie, the need for additional 
IEs to be exported indicating how much of the packet data was exported.

> is a problem which is specific to the padding issue.

Sure: template exhaustion isn't necessarily caused by careless use of 
packet sections. However, careless use of packet sections will 
definitely lead to template exhaustion.

> think that your argumentation forces us to allow padding.

I'd be quite happy for this text to be removed: "The data for this
field MUST NOT be padded."

Then padding is neither required nor forced. But neither is it precluded!

---

From: Juergen Quittek <[email-masked]>
Date: Wed, 16 Jul 2008 09:56:52 +0200

Speaking as technical contributor:

I fully support Paul's statement that sending some packet data
that is by nature of variable length in fixed length Information
Elements would require a different template for each length of
the information you want to send.  This may lead to an absurd
number of templates to be used for reporting.

And my conclusion is quite simple:
Trying to send something that is by nature of variable length
in fixed length Information Elements is a bad idea.

The alternative that we are discussing is using a single template
with fixed length fields and filling the fixed length field with
padding octets when the actual data is shorter than the fixed length
field. In order to know the actual length of the information, we
send another Information Element that tells us how many octets in
the padded IE are actual data.

This this alternative would probably work well.  It would be a nice
workaround if we did not had variable length encoding.  But we do
have it and it is much than the workaround.

  - Variable length encoding is more efficient for variable length
    packet data than fixed length encoding because
      - you do not fill your Data Record with useless padding octets
      - you do not need to send the length of the encoded packet
        data in a separate Information Element

  - Variable length encoding is more self-contained than padded
    fixed length encoding, because you can store, forward and
    interpret the IE without having to take care of another
    accompanying one that tells you which octets inside are valid

Consequently, I would keep the statement that *Section IEs MUST NOT
be padded.  

However, I would still not suggest to disallow fixed length
encoding of *section IEs because there are cases where you
know that your section always has the same length, for example,
if you report in ipHeaderPacketSection  the first 12 octets of
the IP header.

    Juergen

---

From: Gerhard Muenz <[email-masked]>
Date: Wed, 16 Jul 2008 11:18:48 +0200

Jürgen, Paul,

>     accompanying one that tells you which octets inside are valid

One disadvantage of VLE is that parsing a Data Sets of Records including
variable length fields is much more complex and therefore requires more
processing time at the collector.

For example, in our Data Set parser implementation, we first check if
the length of the Record is known from the Template (i.e., if there is
no VLE). The complexity of the Data Set treatment which comes behind
this IF-statement is significantly different for fixed-length and
variable-length Records because VLE requires a complex calculation of
the offset for each Record.

Therefore, in practice, it can make sense to accept a slightly increased
Record length if this allows the collector to decode Records at a much
higher rate.

> Consequently, I would keep the statement that *Section IEs MUST NOT

The only working alternative would be allowing both options:

1) VLE
2) fixed length encoding with mandatory additional length IE

However, I'm not sure if 2) represents a real advantage compared to
using different Templates with different fixed-length for *Section fields.

Assumed that we are many interested in the export of network and
transport layer information (IP+TCP/UDP/ICMP) and that IP/TCP options
are rare, then only a small number of different lengths would be required:

IP+TCP=20+20=40
IP+UDP=20+8=28
...

For the small number of special cases differing from this scheme, we can
define another template with variable length field.

Regards,
Gerhard

---

From: Paul Aitken <[email-masked]>
Date: Wed, 16 Jul 2008 12:10:11 +0100

Juergen,

In our implementation the user specifies how much data is to be 
captured, eg "capture packet-data length 200".

The length tells us how wide a field to allocate in the cache.

It's important that the cache field is of a fixed size (ie, regardless 
of the packet data) for two reasons:

1. To ensure that any following fields are located at a constant offset.
2. So we can hash on the data in the case that it's a key field.

Obviously we have to pad the cached data if it's less than the specified 
length.

Since we now have a fixed length field (including the padding) it's 
straightforward for us to export it as such - which is why I'd like the 
"MUST NOT be padded" restriction to be removed.

It'd be more complicated to export as a variable-length field since we'd 
also need to cache the length - which we don't necessarily do, unless 
the user has also configured it. And if the user has configured it then 
we'd also export it, which gives them the necessary information to 
decode the fixed length field!

Another view is that the user has asked for 200 bytes, so that's what we 
should provide.

Finally, variable length fields are also not backwards compatible with 
netflow v9 - and we've been striving to ensure as much v9 and IPFIX 
compatibility as possible, to make the transition as simple as possible.

So we'd prefer not to mandate that variable length MUST be used.

Cheers.

---

From: Gerhard Muenz <[email-masked]>
Date: Wed, 16 Jul 2008 13:28:52 +0200

Hi Paul,

> decode the fixed length field!

In the presence of padding, we should mandate (or at least recommend)
that the section length of valid section data is also included in the
Record. Otherwise, there may be problems to correctly interprete the
exported information.

Regards,
Gerhard

---

From: "Thomas Dietz" <[email-masked]>
Date: Wed, 16 Jul 2008 13:32:34 +0200

Hi Paul,

find my comments inline...

-- 
Thomas Dietz                 E-mail: [email-masked]
NEC Europe Ltd.              Phone:  +49 6221 4342-128
NEC Laboratories Europe      Fax:    +49 6221 4342-155
Network Research Division
Kurfuersten-Anlage 36
69115 Heidelberg, Germany    http://www.nw.neclab.eu

NEC Europe Limited           Registered in England 2832014
Registered Office: NEC House, 1 Victoria Road, London W3 6BL

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Paul Aitken
> Sent: Mittwoch, 16. Juli 2008 13:10
> To: Juergen Quittek
> Cc: IETF PSAMP Working Group
> Subject: Re: [PSAMP] PSAMP-INFO: padding of reported packet sections
> 
> Juergen,
> 

First, if you just export the section without the length it is useless for
the user.

If you export the length then you can (with the same effort) export a
variable length field.

Btw. at one time of the processing you must have the length anyway because
you need to know where the padding starts. At that point in time you can
also cache it. It's just so easy to write it at the beginning (or the end
whatever you like better) of the 200 byte buffer and have it at maximum 2
bytes longer if you store it in IPFIX format.

> Another view is that the user has asked for 200 bytes, so that's what

Hey, but if the packet is only 27 bytes a 200 byte information element does
give the same information as a 27 byte information element and wastes
network bandwidth.

> compatibility as possible, to make the transition as simple as

You can keep your internal processing as is. You just need to change the
exporting. If the user wants to have IPFIX and not Netflow v9 then he/she
must have a collector for it. If he/she has a collector it must conform to
the IPFIX protocol, thus it must support variable length information
elements. So I don't see your point here.

Just my point of view,

Best Regards,

Thomas

---

From: "Thomas Dietz" <[email-masked]>
Date: Wed, 16 Jul 2008 13:58:19 +0200

Dear Gehard, all,

> Of Gerhard Muenz
> Sent: Mittwoch, 16. Juli 2008 11:19
> To: Juergen Quittek

>From my point of view the processing rate is only important if we talk about
mediator/concentrators etc. A normal collector can just store the received
records and process them as time allows.

I personally would include a special case here which just imposes a small
impact on processing (at least I believe so): one VLE at the very end of the
record. Then you know in advance at what position of the packet the VLE
length is and can quickly add it to the fixed size and get the position of
the next record.

> define another template with variable length field.

I would call this the ideal solution. Maybe it need a little more
configuration and processing effort on the exporter (because it must decide:
is the length 40 take template a, is it 28 take template b or else take
template c), but I think that is affordable.

Best Regards,

Thomas

---

From: Paul Aitken <[email-masked]>
Date: Wed, 16 Jul 2008 13:37:37 +0100

Gerhard,

> In the presence of padding, we should mandate (or at least recommend)

I agree with "at least recommend".

We cannot know what the user wants to do with the data, so we should not 
try to guess.

> Record. Otherwise, there may be problems to correctly interprete the

The length can be calculated from existing IEs, eg ipHeaderLength, 
ipPayloadLength and ipTotalLength.

So it's not necessary to export a further IE to indicate the amount of 
"valid" data in the section.

---

From: Paul Aitken <[email-masked]>
Date: Wed, 16 Jul 2008 13:42:20 +0100

Thomas,

> I personally would include a special case here which just imposes a
> small impact on processing (at least I believe so): one VLE at the
> very end of the record.

There may be multiple VLEs.

> Then you know in advance at what position of the packet the VLE 
> length is and can quickly add it to the fixed size and get the
> position of the next record.

Sounds a bit like draft-irino-ipfix-ie-order?

---

From: Gerhard Muenz <[email-masked]>
Date: Wed, 16 Jul 2008 15:01:29 +0200

Hi Thomas, Paul,

Paul Aitken wrote:
>> Then you know in advance at what position of the packet the VLE length
>> is and can quickly add it to the fixed size and get the

Indeed. However, I have not yet understood why VLE fields at the end of
the Record are advantageous.

If the goal is to calculate the Record boundary and if there is only one
VLE field in the Record, the parser can jump to its position and extract
the variable length field. However, the VLE field does not have to be
the last field in the Record in order to do so.

More generally, the number of iterations in the VLE parser depends on
the number of VLE fields in the Record. Therefore, fewer VLE fields
result in a faster parsing and Record boundary determination.

Regards,
Gerhard

---

From: Juergen Quittek <[email-masked]>
Date: Wed, 16 Jul 2008 15:39:41 +0200

Paul,

> "valid" data in the section.

It is definitely necessary to export at least one further IE.
The further IEs would contain packet length or the header length
in one IE and the payload length in another one.

But it is indeed not necessary to define further IEs for this purpose.
We have them already.

    Juergen

---

From: "Thomas Dietz" <[email-masked]>
Date: Wed, 16 Jul 2008 15:49:19 +0200

Hi Gerhard,

> From: Gerhard Muenz [mailto:[email-masked]]
> Sent: Mittwoch, 16. Juli 2008 15:01
> To: Paul Aitken
> Cc: Thomas Dietz; Juergen Quittek; IETF PSAMP Working Group
> Subject: Re: [Sender: [email-masked]] Re: [PSAMP] PSAMP-INFO:
> padding of reported packet sections
> 
> 
> Hi Thomas, Paul,
> 
> Paul Aitken wrote:
> > Thomas,
> >
> length

The VLE does not need to be at the end. But if it is at the end then the
offsets for all other fields (counted from the start of the record) always
remain constant and the elements can be read quickly. You then need only to
add the length of the (last) VLE field to the known constant length and you
know the start of the next record.

> result in a faster parsing and Record boundary determination.

That's of course true, but I would expect to have only 1 VLE most of the
time. Then being smart with decoding this special case where the single VLE
is at the end may speed up processing. Nevertheless I do not expect
performance issues with new powerful collectors/mediators/concentrators
anyway. They must anyway have CPU power to do the concentrator/mediator
work.

Regards,

Thomas

---

From: "Thomas Dietz" <[email-masked]>
Date: Wed, 16 Jul 2008 15:54:12 +0200

Hi Paul,

> From: Paul Aitken [mailto:[email-masked]]
> Sent: Mittwoch, 16. Juli 2008 14:42
> To: Thomas Dietz
> Cc: Gerhard Muenz; Juergen Quittek; IETF PSAMP Working Group

True, but that's why I called this special case. Btw. I wouldn't expect more
than one VLE in a record in the majority of cases anyway.

> Sounds a bit like draft-irino-ipfix-ie-order?

Well, yes. If you are that concerned about performance... But I expect the
device that needs does mediation/concentration anyway to be powerful because
of the processing it needs to do for the concentration. Then decoding VLEs
is not an issue (otherwise decoding IP packets which are by nature variable
length would also be a problem).

Best Regards,

Thomas

---

From: Paul Aitken <[email-masked]>
Date: Wed, 16 Jul 2008 15:18:19 +0100

Juergen,

> In the current version of draft-ietf-psamp-info there are two
> Information Elements related to link layer frames: dataLinkFrameSize
> and dataLinkFrameSection.
> 
> I would like to discuss whether it is of advantage to have generic
> dataLinkFrame* IEs rather than specific ones.
> 
> A generic one is nice, if you are reporting different types of data
> link layer frames.
> 
> interpretation of the IE becomes tricky.

We definitely need a layer 2 protocol IE to allow the layer 2 frame to
be understood.

> The most common case is probably an Ethernet frame.  Also for 802.11
> the concept of a frame is pretty clear.
> 
> It gets more difficult if you use MPLS. What is then the data link
> frame?  Or for 802.16 (WiMAX, where multiple packets for multiple
> receivers may be contained in a single frame (burst).
> 
> Things would get much clearer is we used link layer-specific IEs
> instead of a dataLinkFrame* IEs, such as ethernetFrameSection, 

Instead of IEs specific to each link layer, we should have a generic
section field and extract specific and useful fields into their own IEs.

Exactly as we do for L3, in fact.

---

From: Paul Aitken <[email-masked]>
Date: Wed, 16 Jul 2008 15:20:14 +0100

Thomas,

> Btw. I wouldn't expect more than one VLE in a record in the majority 
> of cases anyway.

That very much depends on the monitor and exporter.

We can have no expectations when the user can configure whatever fields
they like.

---

From: Juergen Quittek <[email-masked]>
Date: Wed, 16 Jul 2008 19:03:57 +0200

Hi Paul,

> We definitely need a layer 2 protocol IE to allow the layer 2 frame to

I don't think we have them yet.

Is there a need for more IEs?

    Juergen

---

From: Paul Aitken <[email-masked]>
Date: Wed, 16 Jul 2008 19:24:12 +0100

Juergen,

> I don't think we have them yet.

Correct.

> Is there a need for more IEs?

At least the L2 protocol. Else, how would you interpret the frame?

We could also define individual L2 fields - though I would see that as a 
separate work; no need to stall any other work in progress.

Cheers.

---

From: Juergen Quittek <[email-masked]>
Date: Thu, 17 Jul 2008 09:00:00 +0200

Paul,

> separate work; no need to stall any other work in progress.

I would create EthernetFrameSize, EthernetFrameSection, wlanFrameSize,
wlanFrameSection, wimaxBurstSize, wimaxBurstSection, mplspacketsize (if
we don't have it already), etc.

These would all be self-contained and none of them would need any
further IEs for interpreting their content.

Cheers,

    Juergen

---

From: Paul Aitken <[email-masked]>
Date: Thu, 17 Jul 2008 09:50:26 +0100

Juergen,

> we don't have it already), etc.

That's exactly what I *wouldn't* do.

> further IEs for interpreting their content.

By the same token we should throw away all the L3 fields and retain just 
the IPv4packetSection and IPv6packetSection - since these are both 
"self-contained and none of them would need any further IEs for 
interpreting their content".

Clearly it makes no sense at all.

As I recall, the previous IPFIX charter said something about defining 
all the fields that are required for the information model. I guess 
there was an assumption that this wouldn't be a minimalistic set of 
packet/frame sections.

---

From: Juergen Quittek <[email-masked]>
Date: Fri, 18 Jul 2008 16:33:50 +0200

Hi Paul,

> Clearly it makes no sense at all.

I think it makes a lot of sense.

All of the Information Elements we standardized so far are
self-contained.   Packet sections with an unspecified number of
padding octets and linkLayerFrameSections for an unspecified link
layer technology would be the first ones that are not self-contained.
In order to read the full content you need to have another IE that
tells you the the use link layer technology or length of the padding,
respectively.

I would not be happy with introducing such a new class of IEs that
are only readable in the presence of others.
 
> there was an assumption that this wouldn't be a minimalistic set of

Definitely not. We have different IEs for v4 and v6. There is no general
problem with having different IEs for Ethernet and WLAN.

    Juergen

---

From: Benoit Claise <[email-masked]>
Date: Wed, 30 Jul 2008 17:40:13 +0100

Dear all,

I propose to remove that dataLinkFrameSection I.E.
We inserted that one to be complete.
When the specific need will be required, then a request to IANA will 
have to be done. And I think that there is a high chance that L2 
specific section (ethernet, dot1Q, etc..) will be requested, which will 
solve the current issue of a vague definition.

Regards, Benoit

---

From: Paul Aitken <[email-masked]>
Date: Wed, 30 Jul 2008 17:48:31 +0100

Dear All,

After discussing this issue with Juergen, I agree that the right 
solution for IPFIX is for packet sections not to be padded.

eg, variable length encoding can be used.

Unfortunately this isn't possible for netflow v9 export :-(