Subject: RE: [ipfix] Variant Field Types.

From: "MEYER,JEFFREY D (HP-Cupertino,ex1)" <[email-masked]>
Date: Tue, 30 Sep 2003 16:20:07 -0400

Ganesh,
 
  I'm not clear on how this benefits the collector.  Tal and myself produce
different implementations of carrier deployed products which collect from
100's of devices and dozens of protocols, perhaps our experience here is
worth something, and it tells us variant is undesireable on the collector...
 
  Regarding your other point:
 
One way to solve this problem is if the collector assumes the largest size
for 
computations?
 
  This was my point in the e-mail to Stewart.  Have the information model be
EXPLICIT about what is the largest size which is being modeled for a data
item and declare it.  Allow the exporter to declare a smaller size in the
template, if the exporter knows that an individual record will always be
accomodated (based on its implementation) in the smaller size.
 
  So for instance this might say that byteCount should be declared in the
information model as a 64-bit int, while AS is probably sufficient at
32-bits.
 
  This accomodates the:
      exporter - can use smaller size if known
      n/w - can reduce BW if know a smaller size is used
      info model - explicit is better than "vague"
      collector - addresses persistence and aggregation issues
 
  Can we call this a compromise?
 
Regards,
 
  Jeff Meyer

-----Original Message-----
From: Ganesh Sadasivan [mailto:[email-masked]]
Sent: Tuesday, September 30, 2003 10:45 AM
To: MEYER,JEFFREY D (HP-Cupertino,ex1)
Cc: [email-masked]
Subject: Re: [ipfix] Variant Field Types.

MEYER,JEFFREY D (HP-Cupertino,ex1) wrote:

Hi,
 
  Although the use of variant types may seem convenient from the perspective
of an
exporter, I think the various consumers of the data would benefit from
sticking with
an explicit type system.

It would benefit the exporter, the n/w, flexibility of interpretation of
fields in the 
information model  and the collector.

  Consider operations which involve summing fields, storing them to a DB,
etc.

One way to solve this problem is if the collector assumes the largest size
for 
computations?

  Explicit typing ensures that these consuming applications can handle these
values
appropriately and also makes the information modeller think about how these 
types are to be used.
 
  The examples you cite:
 
- ingressPort 
- egressPort 
- packetCount 
- byteCount 
- sourceAS 
- destinationAS
- nextHopAS 
- droppedPacketCount
- samplingInterval
- droppedByteCount
 
  fall into a few categories, identifiers (such as ports and AS's), counts
and intervals.
 
  For the identifiers, I believe the size is fixed by the relevant RFC's
(i.e. why would I 
have a 1 byte Port?).  For counts, the use of 32 vs. 64 bits is a valid
question, however
these can be handled by making them explicit (i.e. the information model
says if
I get this value it is a 32-bit quantity or a 64-bit quantity and the code
will address
it accordingly).  For intervals, the resolution would determine which size
is most
appropriate.

For AS the size could be 2 or 4 bytes.
You may prefer to use a 1 byte port if you are a small system as against a 4
byte for 
a gigantic system. Both have their own implementation reasons.

  The option always exist to define variations on these data items
explicitly.  E.g.
byteCount32 and byteCount64.

Yes. But is'nt this not more of an overhead to introduce new field type as
the requirements
come in and then make sure that it gets deployed in exporter and collector?

  If there is a need to vary the sizes for various fields I'd much rather
see the rationale
for each discrete sizing made explicit vs. forcing all the downstream
consumers of this \
information have to deal with special cases and ambiguity.

This is not a special case handling as you can see that the list above
covers about 
40% of the fields defined today in the information model. It is way to
handle 
ambiguity arising out of implementation differences.

Thanks
Ganesh

Regards,
 
  Jeff Meyer

-----Original Message-----
From: Ganesh Sadasivan [ mailto:[email-masked]
<mailto:[email-masked]> ]
Sent: Monday, September 29, 2003 6:00 PM
To: [email-masked] <mailto:[email-masked]> 
Subject: [ipfix] Variant Field Types.

Hi,

The information model <draft-ietf-ipfix-info-01.txt> only

has types for fixed length objects and variable length 

string like objects.

There are a number of information element types (listed below)

that would be better defined in terms of fixed-size run-time

specified objects. The type variation comes due to difference

in the size of the objects used because of an implementation,

as demanded by the network or otherwise.

The way to make this work is to use the template

length field to determine the exact sub-type:

The definitions of a variant field type can be stated as:

type varUnsignedInt

    if (element.length == 1)

         type = unsignedByte

    else if (element.length <= 2)

         type = unsignedShort

    else if (element.length <= 4)

         type = unsignedInt

    else if (element.length <= 8)

         type = unsignedLong

   else type = undefined

type varInt

    if (element.length == 1)

         type = byte

    else if (element.length <= 2)

         type = short

    else if (element.length <= 4)

         type = int

    else if (element.length <= 8)

         type = long

   else type = undefined

The candidates that fit into the description above are (I may

have missed some also):

- ingressPort 

- egressPort 

- packetCount 

- byteCount 

- sourceAS 

- destinationAS

- nextHopAS 

- droppedPacketCount

- samplingInterval

- droppedByteCount

Please comment.

Thanks

Ganesh

---

From: Ganesh Sadasivan <[email-masked]>
Date: Tue, 30 Sep 2003 15:49:58 -0700

Jeff,

MEYER,JEFFREY D (HP-Cupertino,ex1) wrote:

> Ganesh,
>  
>   I'm not clear on how this benefits the collector.  Tal and myself 
> produce different implementations of carrier deployed products which 
> collect from 100's of devices and dozens of protocols, perhaps our 
> experience here is worth something, and it tells us variant is 
> undesireable on the collector...
>  

New collector versions need not be deployed if the field size is changed 
from the exporter.
This is true as far the portion which tries to decode the packet 
containing the flow records.
Maybe I do not understand all the collector implementation. But what I 
infer from your
arguments is that having an explicit type helps the collector in making 
computations on the
the received data easier.
But are'nt we dealing with encoding->exporting->decoding data defined by 
the information
model here?

> One way to solve this problem is if the collector assumes the largest 
> size for
> computations?
>  
>   This was my point in the e-mail to Stewart.  Have the information 
> model be EXPLICIT about what is the largest size which is being 
> modeled for a data item and declare it.  Allow the exporter to declare 
> a smaller size in the template, if the exporter knows that an 
> individual record will always be accomodated (based on its 
> implementation) in the smaller size.
>  
>   So for instance this might say that byteCount should be declared in 
> the information model as a 64-bit int, while AS is probably sufficient 
> at 32-bits.
>  

>       info model - explicit is better than "vague"

How about  defining the information model of the relevant fields with 
varsize
and mentioning that the collector should use explicit type 'X' for a given
varsize field. Otherwise vendors who implement exporters would be left in
dark.

>   Can we call this a compromise?

Let's wait to hear from some more people.

Thanks
Ganesh

>     *From:* Ganesh Sadasivan [mailto:[email-masked]]
>     *Sent:* Tuesday, September 30, 2003 10:45 AM
>     *To:* MEYER,JEFFREY D (HP-Cupertino,ex1)
>     *Cc:* [email-masked]
>     *Subject:* Re: [ipfix] Variant Field Types.
>
>
>
>>       Although the use of variant types may seem convenient from the
>>     perspective of an
>>     exporter, I think the various consumers of the data would benefit
>>     from sticking with
>>     an explicit type system.
>
>     It would benefit the exporter, the n/w, flexibility of
>     interpretation of fields in the
>>       Consider operations which involve summing fields, storing them
>>     to a DB, etc.
>
>     One way to solve this problem is if the collector assumes the
>     largest size for
>     computations?
>
>>      
>>       Explicit typing ensures that these consuming applications can
>>     handle these values
>>       fall into a few categories, identifiers (such as ports and
>>     AS's), counts and intervals.
>>      
>>       For the identifiers, I believe the size is fixed by the
>>     relevant RFC's (i.e. why would I
>>     have a 1 byte Port?).  For counts, the use of 32 vs. 64 bits is a
>>     valid question, however
>>     these can be handled by making them explicit (i.e. the
>>     information model says if
>>     I get this value it is a 32-bit quantity or a 64-bit quantity and
>>     the code will address
>>     it accordingly).  For intervals, the resolution would determine
>>     which size is most
>>     appropriate.
>>      
>
>     You may prefer to use a 1 byte port if you are a small system as
>     against a 4 byte for
>>       The option always exist to define variations on these data
>>     items explicitly.  E.g.
>     Yes. But is'nt this not more of an overhead to introduce new field
>     type as the requirements
>>       If there is a need to vary the sizes for various fields I'd
>>     much rather see the rationale
>>     for each discrete sizing made explicit vs. forcing all the
>>     downstream consumers of this \
>     This is not a special case handling as you can see that the list
>     above covers about
>     40% of the fields defined today in the information model. It is
>     way to handle
>>         *From:* Ganesh Sadasivan [mailto:[email-masked]]
>>         *Sent:* Monday, September 29, 2003 6:00 PM
>>         *To:* [email-masked]
>>         *Subject:* [ipfix] Variant Field Types.
>>
>>Hi,
>>

---

From: "Carter Bullard" <[email-masked]>
Date: Tue, 30 Sep 2003 20:10:06 -0400

Gentle people,
   It is terribly important that IPFIX support the
transport of vendor specific data.  Because of this,
I expect IPFIX to support variant field types.  
One could argue that variant field types are simply the
implementation of a very large number of explicit
types (4 bit int, 8 bit int, 16 bit int, 4-bit src
address, 4 bit protocol field, etc....).

   I support the notion of variant field types because
it is the only mechanism for improving transport efficiency,
which I believe should be the primary design goal for IPFIX.
I'm not convinced that collector complexity should have
any impact on IPFIX's data model.  Let IPFIX be efficient,
expect the collectors do what is needed to support it.

Carter

-----Original Message-----
From: majordomo listserver [mailto:[email-masked]] On Behalf Of
Ganesh Sadasivan
Sent: Tuesday, September 30, 2003 6:50 PM
To: MEYER,JEFFREY D (HP-Cupertino,ex1)
Cc: [email-masked]
Subject: Re: [ipfix] Variant Field Types.

Jeff,

Ganesh,

New collector versions need not be deployed if the field size is changed
from the exporter.
This is true as far the portion which tries to decode the packet containing
the flow records.
Maybe I do not understand all the collector implementation. But what I infer
from your 
arguments is that having an explicit type helps the collector in making
computations on the 
the received data easier.
But are'nt we dealing with encoding->exporting->decoding data defined by the
information
model here? 

  This accomodates the:

Thanks
Ganesh

Regards,

  Jeff Meyer
-----Original Message-----
From: Ganesh Sadasivan [mailto:[email-masked]]
Sent: Tuesday, September 30, 2003 10:45 AM
To: MEYER,JEFFREY D (HP-Cupertino,ex1)
Cc: [email-masked]
Subject: Re: [ipfix] Variant Field Types.

Hi,

  The examples you cite:

- ingressPort 
- egressPort 
- packetCount 
- byteCount 
- sourceAS 
- destinationAS
- nextHopAS 
- droppedPacketCount
- samplingInterval
- droppedByteCount

Thanks
Ganesh

Regards,

  Jeff Meyer

-----Original Message-----
From: Ganesh Sadasivan [mailto:[email-masked]]
Sent: Monday, September 29, 2003 6:00 PM
To: [email-masked]
Subject: [ipfix] Variant Field Types.

Hi,

type varUnsignedInt
    if (element.length == 1)
         type = unsignedByte

type varInt
    if (element.length == 1)
         type = byte

Please comment.

Thanks
Ganesh

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/