Subject: RE: [ipfix-eval] RE: Discussion of Candidate Protocols - data mod el; broadcast vs reliable

From: "Harrington, David" <[email-masked]>
Date: Tue, 8 Oct 2002 14:23:04 -0400

Hi,

I stand corrected. Please ignore the word "again" in my first sentence.

Dbh

> -----Original Message-----
> From: Jeff Meyer [mailto:[email-masked]] 
> Sent: Tuesday, October 08, 2002 1:49 PM
> To: Harrington, David
> Cc: [email-masked]
> Subject: Re: [ipfix-eval] RE: Discussion of Candidate 
> Protocols - data mod el; broadcast vs reliable
> 
> 
> David,
> 
>    I believe that two separate folks "invoked SNMP", first
> myself, Jeff Meyer (for IPDR), and then Peter Ludemann
> (for XACCT).
> 
> -- Jeff
> 
> Harrington, David wrote:
> 
> > Hi,
> > 
> > You again have invoked SNMP design to justify your opinion. 
> But I think 
> > you overestimate how much the data model and export protocol are 
> > separated in the design of SNMP.
> > 
> > The SMI and the Elements of Procedure define the bindings 
> between the 
> > protocol and the mibs. As you point out, the export 
> protocol needs to be 
> > able to transport the allowed data types. But it needs to 
> do more than that.
> > 
> > The SNMP protocol is also bound to the actions that can be 
> performed on 
> > the data (get/set/trap) and the types of errors that can 
> occur during 
> > processing. Error codes are defined relative to the 
> actions, and are 
> > deliberately NOT defined relative to what is being managed, 
> i.e. there 
> > are no mib-specific error codes. That helped make SNMP simple.
> > 
> > SNMPv3 adds security to SNMP. Security should be an 
> important part of 
> > IPFIX as well, since the exported data is often used for 
> billing. If the 
> > export protocol were totally separate from the data, then 
> SNMPv3 might 
> > have just used IPSec to secure the protocol. But an 
> important part of 
> > SNMP security  is to secure the DATA, not just the message.
> > 
> > To secure the DATA, it is important to understand and to be able to 
> > specify who can do what operations to which subset of data. 
> In IPFIX 
> > terms it will be important to control who is allowed to be 
> the target of 
> > exported data, and to define which exported data they can 
> receive. It is 
> > necessary to have a clearly defined mechanism for 
> addressing the data 
> > elements (the mib objects in SNMP) so they can be 
> referenced in messages 
> > and/or security mechanisms. You cannot just allow any type 
> of data to be 
> > passed in a generic bucket, because then you cannot apply access 
> > controls to that data.
> > 
> > Therefore, it is important to define the actions, the 
> errors, the data 
> > types, the addressing namespace, and the procedures to be 
> followed by 
> > senders and recievers as part of defining the export protocol. To 
> > understand further some of the elements that may need to be defined 
> > based on the SNMP model, I suggest you read the SMING WG 
> requirements 
> > document (RFC3216).
> > 
> > dbh
> > 
> >  > -----Original Message-----
> >  > From: Peter Ludemann [mailto:[email-masked]]
> >  > Sent: Tuesday, October 08, 2002 2:53 AM
> >  > To: [email-masked]
> >  > Subject: [ipfix-eval] RE: Discussion of Candidate Protocols -
> >  > data model; broadcast vs reliable
> >  >
> >  >
> >  > This note discusses two things:
> >  >  - whether we need a data model to define the protocol
> >  >    (I claim that we do not)
> >  >  - cost of UDP broadcast vs minimalist reliable transport
> >  >    (I claim that there is very little difference)
> >  >
> >  > [email-masked] wrote Monday, October 07, 2002 7:35 AM:
> >  >
> >  > > Peter Ludemann wrote:
> >  > [snip]
> >  > > > YES ... so, can we all agree that data model issues are not
> >  > > > part of this discussion?
> >  > >
> >  > >     No. We have no interoperability without a data model.
> >  >
> >  > I agree that we need a data model. I don't see why it should
> >  > affect the
> >  > discussion of the export protocol. The only requirements 
> on the export
> >  > protocol are that it exhibit the desired reliability and
> >  > efficiency; and
> >  > that it be able to transport all the data types required by
> >  > the data models.
> >  > If we can agree on all the data types, then we can define the
> >  > protocol; this
> >  > does not require defining the entire data model.
> >  >
> >  > [After all, SNMP is defined independently of the various 
> MIBs - which
> >  > correspond to the data model]
> >  >
> >  > >     No. Generalized data movers are not the problem.
> >  > >     Defining a protocol that is well defined enough so
> >  > >     that many device vendors can export data and many
> >  > >     application vendors can process the data regardless
> >  > >     of the device is problematic if all you define is a
> >  > >     general purpose data mover.
> >  >
> >  > But deciding on an adequate "generalized data mover" is the
> >  > point of this
> >  > evaluation process, I thought. I expect that there will be
> >  > multiple data
> >  > models, for the various kinds of generalized devices. (e.g.,
> >  > for our probe,
> >  > we've defined three generic groupings of data: volume
> >  > metrics; QoS metrics;
> >  > usage metrics and attributes -- similarly, SNMP's 
> enterprise MIBs and
> >  > RADIUS's vendor-specific attributes).
> >  >
> >  >
> >  > There is one place where the nature of the data might
> >  > intersect with the
> >  > protocol -- the high-volume metrics such as exported by
> >  > NetFlow; and where
> >  > the claim is that a reliable protocol is an unnecessary
> >  > overhead. So, let me
> >  > discuss this a bit more.
> >  >
> >  > The argument is that with high volume export it is 
> preferable to do
> >  > multi-cast UDP with sequence numbers; data loss can be
> >  > detected by noting
> >  > which sequence numbers are missing ... and reliability can be
> >  > increased by
> >  > having multiple receivers and de-duplicating what they receive.
> >  >
> >  > For such a situation, the reliable protocol's
> >  > *implementation* would not
> >  > keep a queue of data to retransmit on fail-over; it would
> >  > only keep enough
> >  > buffer to deal with TCP's or SCTP's flow control and to
> >  > handle bursts of
> >  > records. ACKs would be used only to notice when data are not
> >  > received at the
> >  > other end and to cause fail-over. TCP "write would 
> block" also causes
> >  > fail-over.
> >  >
> >  > I would argue that the cost of exporting this way is very
> >  > similar to that of
> >  > the UDP broadcast. And on the receiving end, it is *much*
> >  > easier to handle
> >  > than a UDP-broadcast, which also needs stronger hardware
> >  > because of the
> >  > higher de-duplication load.
> >  >
> >  > Can be about the same for both:
> >  > - data copying (if scatter/gather is used)
> >  > - protocol overhead (sequence number, template ID, etc.)
> >  >
> >  > The possible extra cost of the "reliable" export version is:
> >  > - timer for noticing when ACK isn't received [trivial cost]
> >  > - TCP vs UDP (little difference with modern implementations)
> >  > - TCP retransmit with packet loss (typically very low)
> >  > - cost of fail-over when no ACK received or write would block
> >  >
> >  > The savings and benefits of the "reliable" version are:
> >  > - fewer packet writes (broadcast requires 2x as many packets;
> >  >   and TCP can pack more efficiently because records can
> >  >   span packets)
> >  > - lower network traffic (which adds to reliability)
> >  > - smaller machines for receiving
> >  > - more accurate estimate of loss due to lack of reliability
> >  > - can handle congestion
> >  >
> >  > [As an aside, even with the UDP-broadcast version, the
> >  > exporter ought to
> >  > compute totals for the various metrics and output those from
> >  > time to time,
> >  > to provide a more accurate understanding of data loss.
> >  > Perhaps such totals
> >  > are already available in the MIBs, but there remains the
> >  > issue of how to
> >  > correlate with the sequence numbers of the exported data.]
> >  >
> >  > - peter
> >  >
> >  >
> >  > --
> >  > Help        mailto:[email-masked] and say "help"
> >  > in message body
> >  > Unsubscribe mailto:[email-masked] and say
> >  > "unsubscribe ipfix" in message body
> >  > Archive     http://ipfix.doit.wisc.edu/archive/
> >  >
> > 
> 
> 
>