Subject: RE: [ipfix-eval] RE: Discussion of Candidate Protocols - data mod el; broadcast vs reliable

From: "Harrington, David" <[email-masked]>
Date: Tue, 8 Oct 2002 11:36:17 -0400

Hi,

You again have invoked SNMP design to justify your opinion. But I think you
overestimate how much the data model and export protocol are separated in
the design of SNMP.

The SMI and the Elements of Procedure define the bindings between the
protocol and the mibs. As you point out, the export protocol needs to be
able to transport the allowed data types. But it needs to do more than that.

The SNMP protocol is also bound to the actions that can be performed on the
data (get/set/trap) and the types of errors that can occur during
processing. Error codes are defined relative to the actions, and are
deliberately NOT defined relative to what is being managed, i.e. there are
no mib-specific error codes. That helped make SNMP simple.

SNMPv3 adds security to SNMP. Security should be an important part of IPFIX
as well, since the exported data is often used for billing. If the export
protocol were totally separate from the data, then SNMPv3 might have just
used IPSec to secure the protocol. But an important part of SNMP security
is to secure the DATA, not just the message. 

To secure the DATA, it is important to understand and to be able to specify
who can do what operations to which subset of data. In IPFIX terms it will
be important to control who is allowed to be the target of exported data,
and to define which exported data they can receive. It is necessary to have
a clearly defined mechanism for addressing the data elements (the mib
objects in SNMP) so they can be referenced in messages and/or security
mechanisms. You cannot just allow any type of data to be passed in a generic
bucket, because then you cannot apply access controls to that data.

Therefore, it is important to define the actions, the errors, the data
types, the addressing namespace, and the procedures to be followed by
senders and recievers as part of defining the export protocol. To understand
further some of the elements that may need to be defined based on the SNMP
model, I suggest you read the SMING WG requirements document (RFC3216).

dbh

> -----Original Message-----
> From: Peter Ludemann [mailto:[email-masked]] 
> Sent: Tuesday, October 08, 2002 2:53 AM
> To: [email-masked]
> Subject: [ipfix-eval] RE: Discussion of Candidate Protocols - 
> data model; broadcast vs reliable
> 
> 
> This note discusses two things:
>  - whether we need a data model to define the protocol
>    (I claim that we do not)
>  - cost of UDP broadcast vs minimalist reliable transport
>    (I claim that there is very little difference)
> 
> [email-masked] wrote Monday, October 07, 2002 7:35 AM:
> 
> > Peter Ludemann wrote:
> [snip]
> > > YES ... so, can we all agree that data model issues are not
> > > part of this discussion?
> >
> > 	No. We have no interoperability without a data model.
> 
> I agree that we need a data model. I don't see why it should 
> affect the
> discussion of the export protocol. The only requirements on the export
> protocol are that it exhibit the desired reliability and 
> efficiency; and
> that it be able to transport all the data types required by 
> the data models.
> If we can agree on all the data types, then we can define the 
> protocol; this
> does not require defining the entire data model.
> 
> [After all, SNMP is defined independently of the various MIBs - which
> correspond to the data model]
> 
> > 	No. Generalized data movers are not the problem.
> > 	Defining a protocol that is well defined enough so
> > 	that many device vendors can export data and many
> > 	application vendors can process the data regardless
> > 	of the device is problematic if all you define is a
> > 	general purpose data mover.
> 
> But deciding on an adequate "generalized data mover" is the 
> point of this
> evaluation process, I thought. I expect that there will be 
> multiple data
> models, for the various kinds of generalized devices. (e.g., 
> for our probe,
> we've defined three generic groupings of data: volume 
> metrics; QoS metrics;
> usage metrics and attributes -- similarly, SNMP's enterprise MIBs and
> RADIUS's vendor-specific attributes).
> 
> 
> There is one place where the nature of the data might 
> intersect with the
> protocol -- the high-volume metrics such as exported by 
> NetFlow; and where
> the claim is that a reliable protocol is an unnecessary 
> overhead. So, let me
> discuss this a bit more.
> 
> The argument is that with high volume export it is preferable to do
> multi-cast UDP with sequence numbers; data loss can be 
> detected by noting
> which sequence numbers are missing ... and reliability can be 
> increased by
> having multiple receivers and de-duplicating what they receive.
> 
> For such a situation, the reliable protocol's 
> *implementation* would not
> keep a queue of data to retransmit on fail-over; it would 
> only keep enough
> buffer to deal with TCP's or SCTP's flow control and to 
> handle bursts of
> records. ACKs would be used only to notice when data are not 
> received at the
> other end and to cause fail-over. TCP "write would block" also causes
> fail-over.
> 
> I would argue that the cost of exporting this way is very 
> similar to that of
> the UDP broadcast. And on the receiving end, it is *much* 
> easier to handle
> than a UDP-broadcast, which also needs stronger hardware 
> because of the
> higher de-duplication load.
> 
> Can be about the same for both:
> - data copying (if scatter/gather is used)
> - protocol overhead (sequence number, template ID, etc.)
> 
> The possible extra cost of the "reliable" export version is:
> - timer for noticing when ACK isn't received [trivial cost]
> - TCP vs UDP (little difference with modern implementations)
> - TCP retransmit with packet loss (typically very low)
> - cost of fail-over when no ACK received or write would block
> 
> The savings and benefits of the "reliable" version are:
> - fewer packet writes (broadcast requires 2x as many packets;
>   and TCP can pack more efficiently because records can
>   span packets)
> - lower network traffic (which adds to reliability)
> - smaller machines for receiving
> - more accurate estimate of loss due to lack of reliability
> - can handle congestion
> 
> [As an aside, even with the UDP-broadcast version, the 
> exporter ought to
> compute totals for the various metrics and output those from 
> time to time,
> to provide a more accurate understanding of data loss. 
> Perhaps such totals
> are already available in the MIBs, but there remains the 
> issue of how to
> correlate with the sequence numbers of the exported data.]
> 
> - peter
> 
> 
> --
> Help        mailto:[email-masked] and say "help" 
> in message body
> Unsubscribe mailto:[email-masked] and say
> "unsubscribe ipfix" in message body
> Archive     http://ipfix.doit.wisc.edu/archive/
>

---

From: Jeff Meyer <[email-masked]>
Date: Tue, 08 Oct 2002 10:49:08 -0700

David,

   I believe that two separate folks "invoked SNMP", first
myself, Jeff Meyer (for IPDR), and then Peter Ludemann
(for XACCT).

-- Jeff

Harrington, David wrote:

> Hi,
> 
> You again have invoked SNMP design to justify your opinion. But I think 
> you overestimate how much the data model and export protocol are 
> separated in the design of SNMP.
> 
> The SNMP protocol is also bound to the actions that can be performed on 
> the data (get/set/trap) and the types of errors that can occur during 
> deliberately NOT defined relative to what is being managed, i.e. there 
> are no mib-specific error codes. That helped make SNMP simple.
> 
> SNMPv3 adds security to SNMP. Security should be an important part of 
> IPFIX as well, since the exported data is often used for billing. If the 
> export protocol were totally separate from the data, then SNMPv3 might 
> have just used IPSec to secure the protocol. But an important part of 
> SNMP security  is to secure the DATA, not just the message.
> 
> To secure the DATA, it is important to understand and to be able to 
> specify who can do what operations to which subset of data. In IPFIX 
> terms it will be important to control who is allowed to be the target of 
> exported data, and to define which exported data they can receive. It is 
> necessary to have a clearly defined mechanism for addressing the data 
> elements (the mib objects in SNMP) so they can be referenced in messages 
> and/or security mechanisms. You cannot just allow any type of data to be 
> passed in a generic bucket, because then you cannot apply access 
> controls to that data.
> 
> senders and recievers as part of defining the export protocol. To 
> understand further some of the elements that may need to be defined 
> based on the SNMP model, I suggest you read the SMING WG requirements 
> document (RFC3216).
> 
> dbh
> 

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Benoit Claise <[email-masked]>
Date: Wed, 09 Oct 2002 15:03:11 +0200

David,

> Hi,
>
> You again have invoked SNMP design to justify your opinion. But I 
> think you overestimate how much the data model and export protocol are 
> protocol and the mibs. As you point out, the export protocol needs to 
> be able to transport the allowed data types. But it needs to do more 
> than that.
>
> The SNMP protocol is also bound to the actions that can be performed 
> on the data (get/set/trap) and the types of errors that can occur 
> during processing. Error codes are defined relative to the actions, 
> and are deliberately NOT defined relative to what is being managed, 
> i.e. there are no mib-specific error codes. That helped make SNMP simple.
>
> IPFIX as well, since the exported data is often used for billing. If 
> the export protocol were totally separate from the data, then SNMPv3 
> might have just used IPSec to secure the protocol. But an important 
> part of SNMP security  is to secure the DATA, not just the message.
>
> specify who can do what operations to which subset of data.
>
Do you think this is really important?

> In IPFIX terms it will be important to control who is allowed to be 
> the target of exported data, and to define which exported data they 
> can receive. It is necessary to have a clearly defined mechanism for 
> addressing the data elements (the mib objects in SNMP) so they can be 
> referenced in messages and/or security mechanisms. You cannot just 
> allow any type of data to be passed in a generic bucket, because then 
> you cannot apply access controls to that data.
>
How is this issue addressed now? If you have the router password, you're 
supposed to be the admin, so you can configure what you want (read: 
export any flow data type to any collector). If we had a configuration 
MIB, you can do what you want with community string.
Why would you need a higher level of configuration control in the IPFIX 
protocol itself?

So to come back to your sentence: "To secure the DATA, it is important 
to understand and to be able to specify who can do what operations to 
which subset of data."
The only operations I can think of is: export
How you would do it practically? Associate a list of possible targets 
next to the each data types?

I'm not convinced this feature would be useful.

Regards, Benoit.