Subject: [ipfix] Variant Field Types.

From: Ganesh Sadasivan <[email-masked]>
Date: Mon, 29 Sep 2003 17:59:56 -0700

Hi,

The information model <draft-ietf-ipfix-info-01.txt> only
has types for fixed length objects and variable length 
string like objects.

There are a number of information element types (listed below)
that would be better defined in terms of fixed-size run-time
specified objects. The type variation comes due to difference
in the size of the objects used because of an implementation,
as demanded by the network or otherwise.

The way to make this work is to use the template
length field to determine the exact sub-type:

The definitions of a variant field type can be stated as:

type varUnsignedInt
    if (element.length == 1)
         type = unsignedByte
    else if (element.length <= 2)
         type = unsignedShort
    else if (element.length <= 4)
         type = unsignedInt
    else if (element.length <= 8)
         type = unsignedLong
   else type = undefined

type varInt
    if (element.length == 1)
         type = byte
    else if (element.length <= 2)
         type = short
    else if (element.length <= 4)
         type = int
    else if (element.length <= 8)
         type = long
   else type = undefined

The candidates that fit into the description above are (I may
have missed some also):
- ingressPort 
- egressPort 
- packetCount 
- byteCount 
- sourceAS 
- destinationAS
- nextHopAS 
- droppedPacketCount
- samplingInterval
- droppedByteCount

Please comment.

Thanks
Ganesh

---

From: "MEYER,JEFFREY D (HP-Cupertino,ex1)" <[email-masked]>
Date: Tue, 30 Sep 2003 13:33:35 -0400

Hi,

  One possible way to address Stewart's concern, which seems 
primarily around exporters not wanting to fill in 64-bit fields,
is to separate the concern of the information model and the
encoding.

  Specifically, since the current protocol specification 
explicitly states lengths used in templates, rather than the
type, an exporter could chose to send quantities in 4 bytes
or 2 bytes or whatever.  The consumer would know that the
quantity was actually targeted at some specified size and
process it accordingly.

  Obviously this only works for "downsizing" data items, not
upsizing them.  But since the arguments seem around "constrained"
systems, then downsizing would appear to be the use case.

  Changes in things like AS number from 16 to 32-bytes are
changes in the information model, I'd prefer to see these
reflected with explicit updates.

  If explicit types are good enough for C, Java, and DB's I
imagine they're good enough for IPFIX.

-- Jeff

> -----Original Message-----
> From: Stewart Bryant [mailto:[email-masked]]
> Sent: Tuesday, September 30, 2003 10:06 AM
> To: MEYER,JEFFREY D (HP-Cupertino,ex1)
> Cc: [email-masked]
> Subject: Re: [ipfix] Variant Field Types.
> 
> 
> 
> 
> MEYER,JEFFREY D (HP-Cupertino,ex1) wrote:
> > Hi,
> >  
> >   Although the use of variant types may seem convenient from the 
> > perspective of an
> > exporter, I think the various consumers of the data would 
> benefit from 
> > sticking with
> > an explicit type system.
> >  
> 
> Variant types allow better use of resources and the ability to upgrade
> fields without the need to firstly get IETF approval, and secondly
> synchronise deployment of the exporter and collector upgrade.
> 
> Anything that counts is a good example. To cope with the high end
> systems we really need to make them all 64bits. That means that
> tiny system that are memory and export limited and which may never
> wrap a 16 or 32 bit counter need to devote resources to 64 bits.
> 
> If we were designing this say 5 years ago we would probably not have
> considered 32 bit AS numbers, now we need to. There are bound to be
> other IE that need to be upgraded, and for each and every one we need
> to synchronise both exporter and collector.
> 
> By introducing this flexibility we get greater scope for how we handle
> these upgrades.
> 
> >   Consider operations which involve summing fields, storing 
> them to a 
> > DB, etc.
> >  
> 
> The collector storage is less likely to be the long pole in the tent
> than the exporter or the transmission bandwidth to the collector. The
> collector can therefore more readily be implemented to accept the
> largest reasonable value than can the exporter.
> 
> >   Explicit typing ensures that these consuming applications 
> can handle 
> > these values
> > appropriately and also makes the information modeller think 
> about how these
> > types are to be used.
> 
> But they have to think worst case during deployment, and my contention
> is that this places undue stress on the exporter and on the network
> bandwidth.
> 
> >  
> >   The examples you cite:
> >  
> > - ingressPort
> > - egressPort
> > - packetCount
> > - byteCount
> > - sourceAS
> > - destinationAS
> > - nextHopAS
> > - droppedPacketCount
> > - samplingInterval
> > - droppedByteCount
> >  
> >   fall into a few categories, identifiers (such as ports and AS's), 
> > counts and intervals.
> >  
> >   For the identifiers, I believe the size is fixed by the 
> relevant RFC's 
> > (i.e. why would I
> > have a 1 byte Port?).  
> 
> Clearly but not long ago you would have said "why would I 
> have a 4 byte
> AS number"
> 
> Also you are thinking in terms of the "examples" we have in 
> the current
> RFCs. We need to design this both for future RFCs and for 
> vendor specific
> IEs.
> 
> > For counts, the use of 32 vs. 64 bits is a valid 
> > question, however
> > these can be handled by making them explicit (i.e. the 
> information model 
> > says if
> > I get this value it is a 32-bit quantity or a 64-bit 
> quantity and the 
> > code will address
> > it accordingly).  For intervals, the resolution would 
> determine which 
> > size is most
> > appropriate.
> 
> But this then makes it harder for tiny systems or cramps 
> large systems.
> 
> >  
> >   The option always exist to define variations on these data items 
> > explicitly.  E.g.
> > byteCount32 and byteCount64.
> 
> Thay leads to two (or more) varients that the collector has 
> to support.
> The alternative is for the collector to support the larger, 
> and the exporter
> to support what ever size it wants. An alternative way to 
> think of this
> is as a data compression technology.
> 
> >  
> >   If there is a need to vary the sizes for various fields I'd much 
> > rather see the rationale
> > for each discrete sizing made explicit vs. forcing all the 
> downstream 
> > consumers of this \
> > information have to deal with special cases and ambiguity.
> >  
> 
> That requires too much (unnecessary) synchronisation between 
> exporter and collector.
> 
> Regards
> 
> Stewart
> 
> >  
> > Regards,
> >  
> >   Jeff Meyer
> > 
> >  
> > 
> >     -----Original Message-----
> >     From: Ganesh Sadasivan [mailto:[email-masked]]
> >     Sent: Monday, September 29, 2003 6:00 PM
> >     To: [email-masked]
> >     Subject: [ipfix] Variant Field Types.
> > 
> > Hi,
> > 

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: "MEYER,JEFFREY D (HP-Cupertino,ex1)" <[email-masked]>
Date: Tue, 30 Sep 2003 17:37:38 -0700

Carter,

  IPFIX supports vendor specific data.  The information model
is extensible.  So you're covered on that point.

  Regarding transport efficiency, the proposed compromise was
that one explicitly indicate the maximal size of an information
item in the model, but allow the template to indicate that fewer 
bytes are used in the record.

  Keep in mind that the protocol enables two entities to 
communicate, an exporter and a collector.  Implementation 
feasability and practicality on both ends make for something
that will be adopted.

  From an implementation perspective, the exporters are likely
to be more purpose built in their behavior, while the collectors
will likely be more general purpose to support a variety of 
exporters.  As such the exporters will likely have a fixed 
information model which they simple declare, whereas the collectors
will likely require configuration for given devices.

  So from a collector perspective, the machine readable and unambiguous 
information model is important, and as proposed does not constrain the 
exporter.

Regards,

  Jeff Meyer

> From: Carter Bullard [mailto:[email-masked]]
> Sent: Tuesday, September 30, 2003 5:10 PM
> To: 'Ganesh Sadasivan'; 'MEYER,JEFFREY D (HP-Cupertino,ex1)'
> Subject: RE: [ipfix] Variant Field Types.
> 
> 
> Gentle people,
>    It is terribly important that IPFIX support the
> transport of vendor specific data.  Because of this,
> I expect IPFIX to support variant field types.  
> One could argue that variant field types are simply the
> implementation of a very large number of explicit
> types (4 bit int, 8 bit int, 16 bit int, 4-bit src
> address, 4 bit protocol field, etc....).
> 
>    I support the notion of variant field types because
> it is the only mechanism for improving transport efficiency,
> which I believe should be the primary design goal for IPFIX.
> I'm not convinced that collector complexity should have
> any impact on IPFIX's data model.  Let IPFIX be efficient,
> expect the collectors do what is needed to support it.
> 
> Carter
> 
> 
> 
> 
> 
> From: majordomo listserver 
> [mailto:[email-masked]] On Behalf Of
> Ganesh Sadasivan
> Sent: Tuesday, September 30, 2003 6:50 PM
> Jeff,
> Ganesh,
>   I'm not clear on how this benefits the collector.  Tal and 
> myself produce
> different implementations of carrier deployed products which 
> collect from
> 100's of devices and dozens of protocols, perhaps our 
> experience here is
> worth something, and it tells us variant is undesireable on 
> the collector...
> 
> New collector versions need not be deployed if the field size 
> is changed
> from the exporter.
> This is true as far the portion which tries to decode the 
> packet containing
> the flow records.
> Maybe I do not understand all the collector implementation. 
> But what I infer
> from your 
> arguments is that having an explicit type helps the collector 
> in making
> computations on the 
> the received data easier.
> But are'nt we dealing with encoding->exporting->decoding data 
> defined by the
> information
> model here? 
> 
> 
>   Regarding your other point:
> 
> One way to solve this problem is if the collector assumes the 
> largest size
> for 
> computations?
> 
>   This was my point in the e-mail to Stewart.  Have the 
> information model be
> EXPLICIT about what is the largest size which is being 
> modeled for a data
> item and declare it.  Allow the exporter to declare a smaller 
> size in the
> template, if the exporter knows that an individual record 
> will always be
> accomodated (based on its implementation) in the smaller size.
> 
>   So for instance this might say that byteCount should be 
> declared in the
> information model as a 64-bit int, while AS is probably sufficient at
> 32-bits.
> 
> 
> 
>   This accomodates the:
>       exporter - can use smaller size if known
>       n/w - can reduce BW if know a smaller size is used
>       info model - explicit is better than "vague"
> How about  defining the information model of the relevant fields with
> varsize
> and mentioning that the collector should use explicit type 
> 'X' for a given 
> varsize field. Otherwise vendors who implement exporters 
> would be left in
> dark.
> 
>       collector - addresses persistence and aggregation issues
> 
>   Can we call this a compromise?
> Let's wait to hear from some more people.
> 
> Thanks
> Ganesh
> 
> 
> 
> Regards,
> 
>   Jeff Meyer
> Sent: Tuesday, September 30, 2003 10:45 AM
>   Although the use of variant types may seem convenient from 
> the perspective
> of an
> exporter, I think the various consumers of the data would benefit from
> sticking with
> an explicit type system.
> It would benefit the exporter, the n/w, flexibility of 
> interpretation of
> fields in the 
> information model  and the collector.
> 
> 
> them to a DB,
> etc.
> One way to solve this problem is if the collector assumes the 
> largest size
> for 
> computations?
> 
> 
> can handle these
> values
>   fall into a few categories, identifiers (such as ports and 
> AS's), counts
> and intervals.
> 
> have a 1 byte Port?).  For counts, the use of 32 vs. 64 bits 
> is a valid
> question, however
> I get this value it is a 32-bit quantity or a 64-bit quantity 
> and the code
> will address
> determine which size
> is most
> For AS the size could be 2 or 4 bytes.
> You may prefer to use a 1 byte port if you are a small system 
> as against a 4
> byte for 
> a gigantic system. Both have their own implementation reasons.
> 
> Yes. But is'nt this not more of an overhead to introduce new 
> field type as
> the requirements
> come in and then make sure that it gets deployed in exporter 
> and collector?
> 
> 
>   If there is a need to vary the sizes for various fields I'd 
> much rather
> see the rationale
> for each discrete sizing made explicit vs. forcing all the downstream
> consumers of this \
> This is not a special case handling as you can see that the list above
> covers about 
> 40% of the fields defined today in the information model. It is way to
> handle 
> ambiguity arising out of implementation differences.
> 
> Thanks
> Ganesh
> 
> Regards,
> 
>   Jeff Meyer
> 
> 
>