Subject: RE: [ipfix-req] list of issues - Reinaldo - 7 (or Carter -1)

From: "Reinaldo Penno" <[email-masked]>
Date: Thu, 24 Oct 2002 14:05:03 -0700

I can think of several reasons (inline with Carter's or Tal's), let me you
know what you think.

1 - First of all, most IPfix applications (not the collector, mind you) read
data in special format  (mrtg, rrdtool, security tools, etc). So we need a
conversion from the format the collector saves to disk to the format the
application uses anyway. 

So when say "how you build useful applications...?", my answer is that we
are not standardizing in which format flow records are saved to disk and
applications need data in their own format anyway.   

2 - IPfix applications are not going to sweep the world right after the
standard is approved. Actually some of them might never get updated. This
provides a way for current application to continue to use old formated data,
which came in in a blob, although the export protocol used is IPfix.

3 - IMO makes extension of the data model more flexible. If I want to export
all HTTP or SDP headers, without having to ask for dozens of new IDs for
each different line in the header or combinations thereof, I can to that.
If I want to combine several different pieces of information (not necessary
proprietary information) under one field, instead of sending in different
ones, I can do that. For example, in our old discussion about NAT...If I'm
doing twice NAT, I might want to put all information about before and after
NAT in one field/blob, instead of dealing with several fields and associated
overhead.

regards,

Reinaldo 

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]
> Sent: Thursday, October 24, 2002 7:18 AM
> To: Penno, Reinaldo [BL60:0430:EXCH]
> Cc: Juergen Quittek; [email-masked]
> Subject: Re: [ipfix-req] list of issues - Reinaldo - 7 (or Carter -1)
> 
> 
> > Reinaldo Penno wrote:
> > 
> > I think Carter's proposal (about opaque blobs) should be included on
> > the requirements draft.
> > 
> 
> 	
> 	Carter seems to be proposing using opaque blobs to send 
> native netflow
> 	records or native IPDR records or native vendor xyz records.
> 
> 	Someone needs to explain to me why we want to do that.
> 
> 	How do you build any useful applications when opaque 
> blobs are sent?
> 	Yes you can send them, yes you can pull the string from 
> the message
> 	but then what? Most likely you end up with proprietary 
> information
> 	being sent and proprietary applications making use of 
> it. The fact
> 	that some standard was used to deliver it doesn't help 
> in any way.
> 	Might was well use raw tcp and a well-known port number.
> 
> 	What am I missing?
> 
> 	Paul
> 
> 
> > I'm reposting it here since although we discussed on the eval list,
> > IMO we should put the text formally on the requiments draft.
> > 
> > regards,
> > 
> > Reinaldo
> > 
> > > -----Original Message-----
> > > From: Carter Bullard [mailto:[email-masked]]
> > > Sent: Tuesday, October 22, 2002 8:17 AM
> > > To: 'Juergen Quittek'; [email-masked]
> > > Subject: RE: [ipfix-eval] Juergen's conclusions
> > >
> > >
> > > Hey Juergen,
> > > >
> > > > Juergen's Conclusion
> > > >
> > > [snip]
> > > >
> > > > Considering all this I would recommend to go for NetFlow v9.
> > > > If this is not accepted because NetFlow is considered as too
> > > > simple or lacking functionality, I would recommend using IDPR.
> > > >
> > >
> > > While I like the IPDR data representation strategy and formats,
> > > I see IPDR as a data model, providing a format for communicating
> > > flow activity reports that a probe can generate.  But I don't
> > > see how IPDR is a transport protocol.
> > >
> > > My opinion is that IPFIX MUST be able to transport IPDR data,
> > > it MUST also be able to transport native NetFlow v[1-9] data
> > > as well.  That to me indicates that the appropriate data model
> > > is either a superset of all the candidates, so that IPFIX can
> > > handle all the data types, or the transport protocol supports
> > > an opaque data object, so that vendors can transport their
> > > native records as blobs.  NetFlow doesn't support either
> > > strategy.  IPDR may be able to be the superset, but its doesn't
> > > support an opaque blob.
> > >
> > > While vendors will assure us that their methods can be extended,
> > > I won't feel comfortable making a choice until the vendors
> > > demonstrate how their methods can handle the requirements.
> > >
> > >
> > > Carter
> > >
> > > Carter Bullard
> > > QoSient, LLC
> > > 300 E. 56th Street, Suite 18K
> > > New York, New York  10022
> > >
> > > [email-masked]
> > > Phone +1 212 588-9133
> > > Fax   +1 212 588-9134
> > > http://qosient.com
>

---

From: Jeff Meyer <[email-masked]>
Date: Thu, 24 Oct 2002 14:55:49 -0700

Hi,

   I agree that opaque blobs are useful, but of the use
cases cited by Reinaldo, I agree with #3, I think #1
is interesting, but addressed directly by IPDR, and
#2 doesn't seem to really be a solution.

   A few additional notes inline...

Regards,

   Jeff Meyer

P.S.

This can be done by the IPDR Streaming protocol
today.  The group decided that the standard XML-Schema
data type named "base64Binary" would be used to represent
data items of this form in service definitions.  It will
map to the existing Compact IPDR type of "BYTE_ARRAY",
which has a type code of 7.

Reinaldo Penno wrote:

> I can think of several reasons (inline with Carter's or Tal's), let me 
> you know what you think.
> 
> 
> 1 - First of all, most IPfix applications (not the collector, mind you) 
> read data in special format  (mrtg, rrdtool, security tools, etc). So we 
> need a conversion from the format the collector saves to disk to the 
> format the application uses anyway.
> 
> So when say "how you build useful applications...?", my answer is that 
> we are not standardizing in which format flow records are saved to disk 
> and applications need data in their own format anyway.  

IPDR's serialization format defines two ways you could store
this information directly to disk, either XML or binary.

Not part of the IPFIX requirement but a side benefit of not
creating "yet another format".

> standard is approved. Actually some of them might never get updated. 
> This provides a way for current application to continue to use old 
> formated data, which came in in a blob, although the export protocol 
> used is IPfix.

I'd prefer my old application to just talk the old protocol,
i.e. NetflowV2-8.  Not to have these packets further wrapped.

As a special case of #3 below this might be interesting, but
in general I don't see it.

> 
> 3 - IMO makes extension of the data model more flexible. If I want to 
> export all HTTP or SDP headers, without having to ask for dozens of new 
> IDs for each different line in the header or combinations thereof, I can 
> to that.  If I want to combine several different pieces of information 
> (not necessary proprietary information) under one field, instead of 
> sending in different ones, I can do that. For example, in our old 
> discussion about NAT...If I'm doing twice NAT, I might want to put all 
> information about before and after NAT in one field/blob, instead of 
> dealing with several fields and associated overhead.

Header exporting or other snippets of packets is a good use
case for supporting a blob.

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Michael MacFaden <[email-masked]>
Date: Thu, 24 Oct 2002 15:17:22 -0700

>#2 doesn't seem to really be a solution.

I believe supporting blobs is counter productive. 

As I have stated before, I would hope that IPFIX
does not become generic data mover protocol. 
If our data model can not represent what is to be sent, 
then that data should be transferred over some other generic protocol 
such as http, ftp or scp.

My reasoning for this is from previous experience with SNMP deployments.
In SNMP there is the OCTET STRING datatype.
It often gets abused to provide a way to move blobs since
you don't need to train an engineer on the protocol, 
it's data model and how to version the thing. 

Take the Apple Airport (karlbridge) wireless bridge for example. 
It uses SNMP and OCTET STRING blobs for configuration.
Users have to spend too time reverse engineering how use the thing
with standard tools http://www.icir.org/fenner/airport/ thanks to blobs.

LFAPv5 allows blobs since you can identify it with
an OID and OCTET STRING type. SNMP limits the blob size to < 64k
but we do not specify the actual max blob size accpted which could
lead to interperability problems between a collector(C) and exporter(E).

Regards,
Mike MacFAden

---

From: Chris Elliott <[email-masked]>
Date: Thu, 24 Oct 2002 18:25:50 -0400 (EDT)

> I believe supporting blobs is counter productive.

I strongly agree. Opaque data types or structures are evil.

> such as http, ftp or scp.

Agreed.

> it's data model and how to version the thing.

And there used to be an OPAQUE data type in SNMP. It was eliminated early
on, for the same reason that we shouldn't do opaque blobs with IPFIX.

> lead to interperability problems between a collector(C) and exporter(E).

Chris Elliott  CCIE# 2013       |         |
Customer Diagnostic Engineer   |||       |||
RTP, NC, USA                  |||||     |||||
919-392-2146              .:|||||||||:|||||||||:.
[email-masked]        c i s c o S y s t e m s

---

From: "Tal Givoly" <[email-masked]>
Date: Thu, 24 Oct 2002 16:01:24 -0700

All,

To ensure my previous posting is unambiguous - I believe BLOBs are required.
Not for the reasons of encapsulating any complete record structure or
encapsulated data type, but for extensibility allowing standard or
vendor-specific extensions that have data of the following nature:

- Any string (is support of a variable-length String data type a
requirement?)
- Strings to include header exporting
- Packet portions to include unparsed packet headers
- Arrays of partial routing information (AS paths)
- Digital signatures (for various applications, I can see this as
potentially useful)

Most of the protocols being evaluated do support BLOBs already (CRANE,
Diameter, IPDR and LFAP). One should also consider why these protocols do
so. In all those protocols it wasn't introduced to support opaque
encapsulated data but rather attributes for which finite list of
single-attribute field types may not have been sufficient (such as above
examples).

I completely appreciate the opinions that BLOBs are "evil", however this
should be a guideline in defining standard templates of attributes,
extensions, etc. For instance, we may also add that if a finite, known set,
of fields are to be used, they are not to be stored as BLOBs, or some other
phrasing to attempt avoiding the possibility of abusing the BLOBs to pack
proprietary data.

In practicality, rather than restrict the possible evolution of the
protocol, it seems to me to make sense to require capability to support this
requirement.

Tal