Subject: RE: in band configuration? RE: [ipfix-req] Proposed changes to d raft-ietf-ipfix-reqs-00.txt

From: Benoit Claise <[email-masked]>
Date: Tue, 19 Feb 2002 11:46:26 +0100 (CET)

Chris,

> 
> Content Negotiation
>   I agree that the negotiation of the set of attributes that a collector is
> interested in has a lot of value.  Depending on the application, a collector
> may be interested in a small or large set of attributes.  If the list of
> requested attributes can be advertised to the mesuring device (which can
> indicate which will be sent) then, IF the device does not ignore the request
> it can minimize the processing and export data volume required.  I.e. no
> need to send extraneous attributes that the collector will ignore.

I don't call this above "content negotiation" but pure configuration.
You tell the exporter which fields to export by defining a template.

Regards, Benoit.

> 
> In-band Configuration
>   The ability to perform the configuration in-band (done in a lightweight
> manner) will have payoffs in large network deployments with mixed devices
> from various vendors.  It will be a challenge to deal with configuration of
> existing implementations which can be difficult to add in-band config to
> (they may need to be configured manually on the box) but believe it is
> worthwile to have a framework for performing this config in-band for devices
> in a common way since we are trying to collect a common set of information
> across the devices.
> 
> 
> Regards,
> Chris
> -----Original Message-----
> From: Peter Ludemann [mailto:[email-masked]]
> Sent: Saturday, February 16, 2002 10:02 AM
> Subject: RE: in band configuration? RE: [ipfix-req] Proposed changes to
> draft-ietf-ipfix-reqs-00.txt
> 
> Juergen:
> 
> Thank-you for pointing out the apparent self-contradiction in my argument.
> I neglected to complete me argument.
> 
> I'm glad that everyone agrees on the need for self-describing data. It
> wasn't clear from all the emails that were flying back and forth. :-)
> 
> I'd like to address the two other issues that are not yet agreed to:
>  - content negotiation (which extends content advertisement)
>  - configuration of processing (sampling rates, ports, etc.)
> 
> Some devices are very simple and have no need for any kind of content
> negotiation. Others (such as our PacketSight probe, which can have over 200
> fields) are quite complex and can benefit greatly from content negotiation.
> 
> The capability to respond to a content request is NOT optional. Every
> request MUST be answered; but the answer can be to reject the request, to
> honour it, or to partially honour it. For example:
> 
>     exporter advertises fields A,B,C,D
>     receiver requests only fields A,D
>     exporter responds with:
>       fields A,B,C,D -- if it ignores the request
>       fields A,D     -- if it completely honours the request
>       fields A,C,D   -- if it partially honours the request
> 
> One of our concerns with content negotiation was in minimising the
> development work for putting the transport protocol onto a network element
> (our target is 2 weeks or less). If the device already has the data
> available in a particular form, we want to be able to export it as-is. For
> example, the exporting device determines whether big-endian or little-endian
> numbers are used; the receiver translates as necessary. This has the
> advantages:
>    - simpler implementation on the exporting device
>    - minimal performance impact on the exporting device
>    - minimal space requirements on the exporting device
> Of course, if the exporting device wants to switch the data from
> little-endian to big-endian (network byte order), we aren't going to prevent
> it. But we cater to the exporter's needs, not the receiver's.
> 
> If the exporting device can take advantage of sending fewer fields, it can;
> but if simplicity of implementation or performance concerns would make it
> better to ignore content requests, that's also OK.
> 
> Moving on to the second point:
>    But as 'in-band configuration' i understand the more
>    basic configuration of meter and exporter including which ports
>    to observe, which filtering rules to apply, which flow keys to use,
>    which sampling rate to choose, etc.
> 
> I would like this kind of configuration also; but I think it is less
> necessary than content negotiation. Content information must be communicated
> amongst all the receivers (in a fail-over configuration), so simply using
> MIBs or a CLI won't work very well. However, the issues of determining
> sampling rate, observed ports, etc. do not need to be distributed in the
> same way (if needed, they can be exported as just another kind of record
> from the network device).
> 
> I think that this kind of configuration will always remain "machine
> specific". But it can fit into a very simple structure, something like a
> list of attribute/value pairs being sent from the receiving device and a
> accepted/rejected message being sent back [the configuration must do
> multiple items together because there can be interactions amongst them].
> This is similar to the way configuration is done using a MIB, except that it
> would get around some of the problems with MIB configuration (interactions
> amongst attributes, confirmation of result, multiple tools needed to do
> configuration, etc.).
> 
> To sum up: we need a framework for all the configuration, without requiring
> agreement on the content (which we might never be able to, especially with
> backwards support for existing equipment). This is similar to the
> self-describing data delivery ... the requirement is that data be
> self-describing and that it contain type information, not that we all agree
> to a specific data model (there eventually will be standard data models, of
> course). Similarly, we should agree to a method of in-band content
> negotiation and processing configuration.
> 
> 
> - peter
> 
> ----
> Peter Ludemann           [email-masked]
> Chief Architect          +1.408.330.5732
> XACCT Technologies, Inc.
> 2900 Lakeside Drive      Santa Clara, CA 95054
> 
> -----Original Message-----
> From: majordomo listserver [mailto:[email-masked]]On Behalf
> Of Juergen Quittek
> Sent: Saturday, February 16, 2002 7:22 AM
> To: Peter Ludemann; Reinaldo Penno; [email-masked]; Benoit
> Claise
> Cc: [email-masked]; [email-masked];
> [email-masked]; Kevin Zhang
> Subject: RE: in band configuration? RE: [ipfix-req] Proposed changes to
> draft-ietf-ipfix-reqs-00.txt
> 
> 
> Hi Peter,
> 
> --On 15 February 2002 08:53 -0800 Peter Ludemann <[email-masked]>
> wrote:
> 
> > Juergen Quittek wrote Thursday, February 14, 2002 2:30 PM:
> >
> >> So what is your point? Do you want to have in-band configuration
> >> for the exported attributes/IEs in the requirements document?
> >
> > Yes. The CRANE protocol solves real problems that we've encountered; and
> we
> > have a number of partners who are using it (sorry, I can't announce these
> > right now).
> >
> >> What would be the application requiring this? Or from which
> >> general requirement would you derive this one?
> >
> > My answer is a little long. Our requirements go beyond just delivering the
> > kind of data that IPFIX is discussing; but I think that they are very
> > applicable to IPFIX.
> >
> > If all you care about is delivering one set of values which you'll never
> > change, then there's no need for any in-band configuration. (This would be
> > true only if you think that the data architecture folks will agree on a
> > specification with no SHOULDs in the document ...)
> >
> > If you anticipate change, then at minimum you need to specify the version
> of
> > the data that you're delivering. A more general solution is to use
> > self-describing data.  XML would be fine but it's much too verbose for
> > delivering thousands of records per second from a small network device.
> >
> > So, the minimal requirement is to have some way for the device to
> advertise
> > what it has.  I don't see a need for all of XML's nesting capabilities; a
> > simple 2-level scheme (type of record + fields) should suffice.
> 
> As far, as I followed the ongoing difscussion, (almost) everyone agrees
> to this.
> 
> > The receiver could quietly take this advertisement and process it. But our
> > experience in "IP data mediation" is that people usually only want
> specific
> > fields; and they typically want to post-process them. Examples of such
> > post-processing: associating values with IP addresses (e.g., reversing
> NAT,
> > mapping IP to customer), aggregation, filtering, conditional splitting,
> etc.
> > plus a host of database-oriented processing, such as looking for error
> > patterns, for people who hog bandwidth, etc.
> >
> > If the sending device can be configured by a MIB or CLI, the receiver can
> > adapt to what it advertises. But there are good reasons for such
> > configuration to be done by response from the receiver:
> >   - the needed fields can be derived from the post-processing
> configuration,
> >     so they naturally reside on the receiver, not the sender
> >   - there can be tens or even thousands of sending devices, so
> >     centralized configuration is desirable (especially if they can't all
> >     have their versions changed simultaneously)
> >   - there needs to be a way of coordinating amongst multiple receivers,
> >     for fail-over deployments
> > (these aren't hypothetical requirements: we have customers who are looking
> > at deploying thousands of sending devices; and who require fail-over)
> >
> > Once you decide that the sender should advertises its data, it doesn't
> take
> > much more work to allow the receiver to respond with a list of fields that
> > it does or doesn't want.  The sender can just mark the fields it won't
> send.
> > The advantages of this are:
> >   - bandwidth saving (probably not very important)
> >   - computation saving on the sender (this can be significant
> >     in our experience)
> 
> I think we are discussing two different issues. Your suggestion
> covers - as far as I have understood so far - the negotiation
> between sender an receiver about individual attributes/fields/IEs
> to be sent. But as 'in-band configuration' i understand the more
> basic configuration of meter and exporter including which ports
> to observe, which filtering rules to apply, which flow keys to use,
> which sampling rate to choose, etc.
> 
> But coming back to your point: If you say the exporter may ignore it,
> why then making it a requirement? Requirements should be essentially
> needed and not just nice to have, but opyionally ignored at any time.
> 
> I think your point can be a valuable contribution to the architecture,
> but following your argument it does not belong into the requirement
> docvument.
> 
>     Juergen
> --
> Juergen Quittek     [email-masked]     Tel: +49 6221 90511-15
> NEC Europe Ltd.,    Network Laboratories     Fax: +49 6221 90511-55
> Adenauerplatz 6, 69115 Heidelberg, Germany   http://www.ccrle.nec.de
> 
> >
> > Our proposal makes it OPTIONAL for the sender to respond to the
> > configuration request.  The sender can ignore the configuration request
> and
> > still send all the fields -- the receiver will just have to accept them
> (and
> > throw away what it doesn't want).  But if the sender can take advantage of
> > the configuration request (e.g., turning off expensive computations), then
> > it has a simple mechanism for doing so.
> >
> >
> > ----
> >
> > Peter Ludemann
> > Chief Architect, XACCT
> > +1.408.330.5732
> >

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: "Peter Ludemann" <[email-masked]>
Date: Sat, 23 Feb 2002 12:01:12 -0800

Benoit Claise [mailto:[email-masked]] wrote Tuesday, February 19, 2002

> You tell the exporter which fields to export by defining a template.

So, shall we have an argument about terminology? That's avoiding the point.

It's more important that we discuss the actual requirements. So far, I've
seen arguments in favour of both content negotiation and in-band
configuration but the only arguments I've seen against these seem to be "I
don't like it" or "it's not in the charter". If I missed more substantive
arguments, please send me a reference ... it would be nice to have something
a bit more persuasive.

Benoit's statement begs the question: how is the template defined? If it's
done out-of-band, that requires a separate element management system (unless
you like entering CLI commands or SNMP values by hand). It makes a
tightly-coupled sender-receiver difficult to handle (I'll give an example
later).

Both content negotiation and in-band configuration should be required but in
the trivial sense that the device must respond to the request for content
change or configuration but the response can be "I refuse to change the
content" or "your configuration request is rejected". This is easy to
implement; and it allows other devices to do the content negotiation or
in-band configuration that they want. [Similarly, I would like a MIB to be
required, at minimum to get configuration, performance, and other "out of
band" data; and it could also (optionally) be used to set configuration
values.]

If we don't put these features into the standard, some people will add them
anyway and we won't have a standard any more.

Sometimes configuration at the device (or via an element management system)
is the best way to do things. But sometimes the receiver must be involved;
or the sender and receiver need to be configured together. In-band content
negotiation and configuration simplify this.

Here is one scenario. Consider collecting layer 3/4 data (e.g., collecting
details of individual FTP or HTTP connections) exporting the sessions in
"slices" (e.g., giving the progress every 30 minutes). From this, a nice
histogram of protocol usage by time of day can be generated.

But now we want more details around the busy times (e.g., 5 minutes slices)
but less details at other times [to save space in the database]. This can
easily be done via in-band configuration -- after all, the collector is
connected directly to the data receiving and processing system; but would be
much more complex using an element management system, which isn't designed
for changing things so dynamically. Going further  with this example, we
want do dynamically drill-down to find busy URLs and FTP sites but not to do
this all the time because of performance issues and because of data space
considerations. This could be done without in-band configuration, but it
would be much nastier. (This example is taken from experience with a data
collection system that was inserting over 100 million records a day into a
database ... clearly, being able to limit the amount of data is very
important and can have significant impact on the cost of the deployment.)

In this scenario, we sometimes want to do in-band configuration and
sometimes want to do other configuration (so, there is something to Benoit's
remark). As I've mentioned elsewhere, I think that data negotiation is the
more important of the two, because of the need to coordinate template
information amongst multiple (failover) receivers; but in-band configuration
is also useful and allows a simpler deployment.

(Again, based on our experience, ease-of-deployment is a very important
consideration, unless you enjoy long meetings in large airless rooms with
representatives from a dozen different departments.)

In general, I believe that "perfection is reached, not when there is no
longer anything to add, but when there is no longer anything to take away"
(Antoine de Saint-Exupery). But in-band content negotiation and
configuration are not very complex (we've implemented them, anyway), are
sufficiently useful to be included, and would lead to "standard extension"
if we don't include them.

- peter

----

---

From: Ganesh Sadasivan <[email-masked]>
Date: Mon, 25 Feb 2002 10:12:59 -0800

Hi Peter,

> arguments, please send me a reference ... it would be nice to have something

May be you missed my response to one of the e-mails in this thread.
The main reason I see is complexity. Just taking a few cases..
1.  Defining an observation point : Eg. The collector now has to say let the set
of
       interfaces {a,b,..} belong to this observation point. So now the collector
needs
       to know what kind of interfaces these are, their characteristics, whether
they
       are groupable etc. All these rules have to be fed into the collector and
these mostly
       would be vendor dependent.Boxes from  different vendors exhibit different
       characteristics for these groupings.
   2. Defining a metering process: Metering process can be defined at different
levels
       at the time of observing the packets or after aggregating. There are a
wide variety
       of metering mechanisms that are adopted today in terms of metering. In
many cases
       the sequences in which metering is done is very important especially in
the hardware
       switching platforms. Is the collector expected to know all these intricate
details of a
       platform and send an in-band configuration message? Or are the existing
products
       to be re-engineered to suit IPFIX protocol?
   3. If there are 'n' exporters trying to export to a collector, choosing the
least common
      fields among all the exporters to define a common template of export from
the
      collector is not an easy task. Say the collector decides on a template,
common to all
      exporters and now a new exporter joins which cannot support this set what
is the
      collector's policy now? The choice of policies  varies greatly with
applications
      that finally use this data. This has to be coupled with the understanding
of how
      metering is done at the exporter.
 4. In the routers, whose main job is to route the packets with performance and
     forwarding features as the top priority,  making decisions as to which stack

     to setup (TCP/SCTP) for export, what security protocol should be used,
     dynamically decide what  templates/fields to accept/reject, dynamically
modify
     the export timers, export rate, start peeking deeper into a packet which may

     affect the performance, though ideally achievable is too much an overhead.
     I tend to beleive that each service provider would be interested in defining

     these decisions for the routers in their own way and generalising any such
     behavior may not add much value, other than adding more complexity. It is
    fairly trivial still to use expect scripts and cron jobs to achieve the
same;)

Thanks
Ganesh