Subject: RE: [ipfix] Multiple levels of Reliability and Directionality

From: "Carter Bullard" <[email-masked]>
Date: Mon, 18 Nov 2002 14:26:52 -0500

Gentle people,
   It may be of interest for the group to look at
RFC-3340 which deals with the issues of generic application
transport and capability negotiation, as well as reliability.
http://www.ietf.org/rfc/rfc3340.txt?number=3340.

Doesn't seem that hard.  When did IPFIX drop capability
negotiation as a requirement?

Carter

Carter Bullard
QoSient, LLC
300 E. 56th Street
Suite 18K
New York, New York 10022

+1 212 588-9133 Phone
+1 212 588-9134 Fax

> -----Original Message-----
> From: majordomo listserver 
> [mailto:[email-masked]] On Behalf Of Ganesh Sadasivan
> Sent: Monday, November 18, 2002 1:53 PM
> To: Tal Givoly
> Cc: 'ipfixx'
> Subject: Re: [ipfix] Multiple levels of Reliability and Directionality
> 
> 
> Tal,
> 
>    There has been discussion at length in the mailing list 
> and IETF session as to why
>    we should refrain from  going with capability negotiation 
> in IPFIX protocol. See
>    my response to Peter on this topic:
>    http://ipfx.doit.wisc.edu/list/ipfix/archive/0753.html
>    Though in theory this idea sounds fine the complexity of 
> this feature when trying
>    to generalize for a framework like IPFIX is high. This was 
> one of the main reasons for
>    dropping this feature off the IPFIX protocol.
> 
> Thanks
> Ganesh
> 
> Tal Givoly wrote:
> 
> > Ganesh,
> >
> > There is a middle ground here: multiple LEVELS/DEGREES of 
> reliability 
> > which are different modes of operation of the protocol. 
> Depending on 
> > the exporter and collector set of capabilities, they can 
> negotiate or 
> > predefine that they will operate in a specific mode. This way, some 
> > devices, under some circumstances (for instance, high-speed flow 
> > export) may not support the reliability capabilities. While 
> at other 
> > situations, they may support reliable exports due to the 
> lower volume, 
> > high-value aggregate information they are exporting. If 
> footprint of 
> > implementation within device is extremely constrained, they 
> may have a 
> > subset of the capabilities altogether.
> >
> > This can be done without undue encumbrance on the device while 
> > permitting cost-effective deployments for applications 
> requiring more 
> > reliability.
> >
> > This is also one of the reasons minimal, exporter-overridden, 
> > negotiation capabilities are very useful. That would allow seamless 
> > interoperability between collectors and exporters with 
> varying degrees 
> > of capabilities and interests in data.
> >
> > Note that this is exactly the way faxes/modems knew how to 
> > interoperate and spawned a huge industry as a result of such 
> > standardization leading to seamless interopability.
> >
> > Tal
> >
> > -----Original Message-----
> > From: Ganesh Sadasivan [mailto:[email-masked]]
> > Sent: Monday, November 18, 2002 6:32 AM
> > To: Tal Givoly
> > Cc: 'ipfixx'
> > Subject: Re: [ipfix] Multiple levels of Reliability and 
> Directionality
> >
> > Tal,
> >
> > Tal Givoly wrote:
> >
> > > Ganesh,
> > >
> > > Obviously, this doesn't do justice to the requirements of many
> > applications.
> > > There is no dispute that there is a set of applications 
> that would 
> > > be satisfied with level 3 or 6. However, flow export is 
> also being 
> > > used, and there is a desire to expand its usage, for applications 
> > > mentioned in the charter that require that require higher 
> levels of 
> > > reliability. Most of
> > our
> >
> > By doing this we are penalising the large set of existing  of 
> > application using flow export were requirement is just a 3 
> or 6. BTW 
> > when I refer to 6, I meant "ACK mechanism which covers like 
> a summary 
> > of lost packets" and not "application level ACK mechanism".
> > The number of apps that are and would be using this export 
> mechanism is
> > large and
> >
> > their requirements are widely varying. Rather than addressing in 
> > detail the need for each of the current set of application, it is 
> > better to keep a simple infrastructure
> > and let the applications with more strict requirements 
> build their own
> > mechanism
> > on top of this infrastructure.
> >
> > Thanks
> > Ganesh
> >
> > > many customers are seeking better interfaces down-to-the-device 
> > > level. I
> > can
> > > refer you to many use cases and justifications of this 
> position that 
> > > appeared in many previous posts.
> >
> > >
> > >
> > > Tal
> > >
> > > -----Original Message-----
> > > From: Ganesh Sadasivan [mailto:[email-masked]]
> > > Sent: Friday, November 15, 2002 11:58 AM
> > > To: [email-masked]
> > > Cc: Benoit Claise; Tal Givoly; Jeff Meyer; [email-masked]; 
> > > 'ipfixx'
> > > Subject: Re: [ipfix] Multiple levels of Reliability and 
> Directionality
> > >
> > > [email-masked] wrote:
> > >
> > > > Benoit Claise wrote:
> > > > >
> > > > > Paul,
> > > > >
> > > > > First of all, let me take back your initial email, 
> for clarity 
> > > > > purposes
> > > > >
> > > > >         High   - Little to no data loss
> > > > >         Medium - Minimal data loss is accepted
> > > > >         Low    - Data sent is best effort
> > > > >
> > > > > The six combinations are shown below...
> > > > >
> > > > >                | Uni  |  Bi
> > > > >         -------+------+---------
> > > > >         High   |  1   |    4
> > > > >         -------+------+--------
> > > > >         Medium |  2   |    5
> > > > >         -------+------+--------
> > > > >         Low    |  3   |    6
> > > > >
> > > > > I think (IMHO) the most likely combinations are...
> > > > >
> > > > >         3. Low reliability and uni-directional. This 
> will be used
> > > > >            where the device and collector are in 
> close proximity,
> > > > >            there is high volume and some data loss is 
> acceptable for
> > > > >            the given application. Applications such as 
> > > > > Attack/Intrusion
> > > > >            Detection Traffic Profiling, etc...
> > > > >
> > > > >         5. Bi-directional medium reliability. This 
> will be used
> > > > >            where high volume and increased 
> reliability are needed.
> > > > >            Applications
> > > > >            such as Usage-based Accounting where 95th 
> percentile 
> > > > > billing
> > > > >            model is used. Or even Traffic Engineering 
> where network
> > > > >            policy decisions are based on the data collected.
> > > > >
> > > > >         4. Bi-directional high reliability would be 
> used where the
> > > > >            Usage-based Accounting application has 
> strict requirements
> > > > >            on data loss.
> > > > >
> > > > > I agree on 3, this is the minimum.
> > > > >
> > > > > I agree on 5 but I fear that the application level 
> ACK will not 
> > > > > always be a possible solution from a router point of 
> view. What 
> > > > > if the router exports so much flow records that it 
> can't cache 
> > > > > them while waiting for the ACK. So basically, 
> resending the flow 
> > > > > records is not always possible. So, if you conclude that the 
> > > > > requirement for 5 is the bidirectionality with 
> application level 
> > > > > ACK, then this requirement is not always possible!
> > > >
> > > >         Agreed. In very high volume deployments I would
> > > >         use option #3.
> > > >
> > > > > IMHO, 5 could alos be
> > > > >         2 with best-effort
> > > > >         or 2 with a carefully designed network
> > > >
> > > >         If the customer can design their accounting network
> > > >         in such away. But I'm not sure we want to require that.
> > > >
> > > >         Lets assume that aggregation is happening at
> > > >         a reasonably high level (which drops the flow rate
> > > >         to a manageable level). But the consequence is that
> > > >         one lost record can be a huge chunk of missing data.
> > > >
> > > >         In this case reliability is the issue not the data
> > > >         rate. I'm advocating a reliability option (note 
> I said option)
> > > >         as a way to solve this.
> > > >
> > > >
> > > > >
> > > > > Regarding 4, Randy replied already
> > > > >
> > > > >      > Somehow, the requirements discussion suffers 
> from a split:
> > > > >      > There are requirements resulting from the intention of
> > > > >      > standardzing a highly reliable metering standard for
> > > > >      precise
> > > > >      > accounting (and billing). Differing from these are
> > > > >      requirements
> > > > >      > for a simpler metering standard more oriented at what I
> > > > >      would
> > > > >      > call "typical Internet reliability".
> > > > >
> > > > >      <ad> the original goal, which the iesg thinks 
> was chartered,
> > > > >      is the
> > > > >      latter.
> > > > >
> > > >
> > > >         There was a lot of push back by the group on 
> that statement.
> > > >         I asked for clarification but have not received any.
> > >
> > > I feel the base line IPFIX protocol does require just 3 
> or 6. This 
> > > would provide the simplest and the most efficient means for all 
> > > applications. For those applications like billing with more 
> > > stringent requirements, why can't extensions be built on 
> top of the 
> > > base protocol as a higher level application? By doing this we can 
> > > cover the entire matrix. There is a vast majority of apps 
> that need 
> > > just the 3. and building a multi-level reliable scheme into the 
> > > basre IPFIX protocol would result in a less efficient use of the 
> > > protocol for apps that really don't care for this.
> > >
> > > Thanks
> > > Ganesh
> > >
> > > >
> > > >
> > > >         Paul
> > > >
> > > > > > Let me see if we all agree on a couple things...
> > > > > >
> > > > > >         1. There is no such thing as partially 
> uni-directional.
> > > > > >         2. IPFIX ought to support a uni-directional mode of 
> > > > > > operation.
> > > > > >
> > > > > Agreed.
> > > > >
> > > > > Regards, Benoit.
> > > > >
> > > > > >
> > > > > > Paul
> > > > > >
> > > > > > Tal Givoly wrote:
> > > > > >
> > > > > >
> > > > > >> Jeff,
> > > > > >>
> > > > > >> I agree with you completely here - the protocol 
> should focus 
> > > > > >> on a primarily uni-directional feed from network/service 
> > > > > >> elements (metering+exporting
> > > > > >> process) to collection process. Adding 
> capabilities (such as
> > > > > >> parameter
> > > > > >> configuration, provisioning of other sorts, etc.) here is a
> > > > > >> slippery slope.
> > > > > >>
> > > > > >> I would add, however, that the bi-directionality is needed 
> > > > > >> both for assurance that messages are delivered as well as, 
> > > > > >> IMHO, version, capability
> > > > > >> and template negotiation (which can be simplified 
> as needed by
> > > > > >> device).
> > > > > >> Version and capability negotiation allows for 
> forward/backward
> > > > > >> interoperability and compatibility. Template 
> negotiation allows
> > > > > >> for more
> > > > > >> efficient application-aware communication. As 
> Peter explained
> > > > > >> earlier
> > > > > >> (http://ipfix.doit.wisc.edu/archive/1381.html - 
> scenario #1 or
> > > > > >> #2), it is
> > > > > >> possible to have all these capabilities and have 
> no impact on the
> > > > > >> metering/export processes complexity - the 
> complexity can be,
> > > > > >> optionally,
> > > > > >> added to the collection process (off the device) 
> in order to allow
> > > > > >> for
> > > > > >> interoperability with multiple concurrent
> > > > > >> versions/capabilities/templates in
> > > > > >> the devices.
> > > > > >>
> > > > > >> Essentially, I believe we achieved consensus regarding the 
> > > > > >> need for template capabilities to allow separation of the 
> > > > > >> data model from the protocol.
> > > > > >> Version/capability negotiation are very useful for 
> any protocol in
> > > > > >> order to
> > > > > >> increase its applicability and useful lifespan.
> > > > > >>
> > > > > >> Tal
> > > > > >>
> > > > > >> -----Original Message-----
> > > > > >> From: Jeff Meyer [mailto:[email-masked]]
> > > > > >> Sent: Monday, November 11, 2002 9:39 AM
> > > > > >> To: [email-masked]
> > > > > >> Cc: Tal Givoly; [email-masked]; 'ipfixx'
> > > > > >> Subject: Re: [ipfix] Multiple levels of Reliability and 
> > > > > >> Directionality
> > > > > >>
> > > > > >> OK, then based on this minimal definition of 
> bi-directional, 
> > > > > >> I'm in complete agreement.
> > > > > >>
> > > > > >> It's the sliperry slope of other things one might want to 
> > > > > >> exchange once this two way channel is open, which concerns 
> > > > > >> me.  There are lots of things one might want to 
> exchange, but 
> > > > > >> many of these are probably better addressed by reusing 
> > > > > >> existing protocols, vs. stuffing into the IPFIX protocol.
> > > > > >>
> > > > > >> For example, controlling metering process is 
> something which 
> > > > > >> can be separated out from IPFIX.  For example by 
> using SNMP 
> > > > > >> and a MeterMIB.
> > > > > >>
> > > > > >> Just trying to apply the KISS (Keep it simple stupid) 
> > > > > >> principle.
> > > > > >>
> > > > > >> Regards,
> > > > > >>
> > > > > >>    Jeff Meyer
> > > > > >>    hp
> > > > > >>
> > > > > >> [email-masked] wrote:
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> > Jeff Meyer wrote:
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> >>  Hi,
> > > > > >> >>
> > > > > >> >>    I agree with the aspect of varied reliability.
> > > > > >> >>
> > > > > >> >>    I am less clear on the need for bi-directional  
> > > > > >> >> behavior.  The three scenarios you describe I  
> agree with, 
> > > > > >> >> but I'm not sure how the 2nd and  3rd scenario as 
> > > > > >> >> described have any dependence  on bi-directional aside 
> > > > > >> >> from the some level of  application layer ack.
> > > > > >> >>
> > > > > >> >>
> > > > > >> >>
> > > > > >> >>
> > > > > >> >       An application layer ACK all by itself makes the
> > > > > >> >       protocol bi-directional.
> > > > > >> >
> > > > > >> >       Scenario #1 does not require the sender to 
> also be a 
> > > > > >> > receiver.
> > > > > >> >
> > > > > >> >       Scenarios #2 & #3 do require the sender to 
> also be a
> > > > > >> >       receiver (even if only for ACK's).
> > > > > >> >
> > > > > >> >       Whether or not we decide to add other features in
> > > > > >> >       scenarios #2 & #3 (such as field 
> negotiation, capability
> > > > > >> >       negotiation, etc...) is up to the WG. But 
> they could
> > > > > >> >       not be added to scenario #1.
> > > > > >> >
> > > > > >> >       Paul
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> >>  Regards,
> > > > > >> >>
> > > > > >> >>    Jeff Meyer
> > > > > >> >>
> > > > > >> >>  Tal Givoly wrote:
> > > > > >> >>
> > > > > >> >>
> > > > > >> >>
> > > > > >> >>
> > > > > >> >> > Paul,
> > > > > >> >> >
> > > > > >> >> > I agree completely - that captures the essense of the 
> > > > > >> >> > requirements. It allows balancing different 
> > > > > >> >> > application's need for different levels of
> > > > > >> >> > reliability.
> > > > > >> >> >
> > > > > >> >> > Tal
> > > > > >> >> >
> > > > > >> >> > -----Original Message-----
> > > > > >> >> > From: majordomo listserver [ 
> > > > > >> >> > mailto:[email-masked]]On Behalf 
> Of Carter 
> > > > > >> >> > Bullard
> > > > > >> >> > Sent: Thursday, November 07, 2002 8:40 AM
> > > > > >> >> > To: [email-masked]; 'ipfixx'
> > > > > >> >> > Subject: RE: [ipfix] Multiple levels of 
> Reliability and 
> > > > > >> >> > Directionality
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >> > Hey Paul,
> > > > > >> >> >  My 2 cents worth sez, absolutely yes!
> > > > > >> >> > Carter
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >> >>  -----Original Message-----
> > > > > >> >> >>  From: majordomo listserver  
> > > > > >> >> >> [mailto:[email-masked]] On Behalf Of  
> > > > > >> >> >> [email-masked]
> > > > > >> >> >>  Sent: Thursday, November 07, 2002 9:46 AM
> > > > > >> >> >>  To: ipfixx
> > > > > >> >> >>  Subject: [ipfix] Multiple levels of 
> Reliability and  
> > > > > >> >> >> Directionality
> > > > > >> >> >>
> > > > > >> >> >>
> > > > > >> >> >>
> > > > > >> >> >>  Does IPFIX need to support multiple levels of 
> > > > > >> >> >> reliability  and directionality?
> > > > > >> >> >>
> > > > > >> >> >>  The applications IPFIX needs to support are quite 
> > > > > >> >> >> varied  and have differing needs in terms of 
> > > > > >> >> >> reliability. Devices  also have varying 
> needs in terms 
> > > > > >> >> >> of directionality.
> > > > > >> >> >>
> > > > > >> >> >>  I would like the WG to consider requirements on the 
> > > > > >> >> >> protocol  to support a tiered approach meet these 
> > > > > >> >> >> needs.
> > > > > >> >> >>
> > > > > >> >> >>  Direction is either bi-directional or 
> uni-directional. 
> > > > > >> >> >> I've  split  reliability into 3 categories...
> > > > > >> >> >>
> > > > > >> >> >>        High   - Little to no data loss
> > > > > >> >> >>        Medium - Minimal data loss is accepted
> > > > > >> >> >>        Low    - Data sent is best effort
> > > > > >> >> >>
> > > > > >> >> >>
> > > > > >> >> >>  The six combinations are shown below...
> > > > > >> >> >>
> > > > > >> >> >>
> > > > > >> >> >>               | Uni  |  Bi
> > > > > >> >> >>        -------+------+---------
> > > > > >> >> >>        High   |  1   |    4
> > > > > >> >> >>        -------+------+--------
> > > > > >> >> >>        Medium |  2   |    5
> > > > > >> >> >>        -------+------+--------
> > > > > >> >> >>        Low    |  3   |    6
> > > > > >> >> >>
> > > > > >> >> >>
> > > > > >> >> >>
> > > > > >> >> >>  I think (IMHO) the most likely combinations are...
> > > > > >> >> >>
> > > > > >> >> >>        3. Low reliability and uni-directional. This 
> > > > > >> >> >> will be  used
> > > > > >> >> >>           where the device and collector are 
> in close  
> > > > > >> >> >> proximity,
> > > > > >> >> >>           there is high volume and some data 
> loss is  
> > > > > >> >> >> acceptable for
> > > > > >> >> >>           the given application. 
> Applications such as  
> > > > > >> >> >> Attack/Intrusion Detection
> > > > > >> >> >>           Traffic Profiling, etc...
> > > > > >> >> >>
> > > > > >> >> >>        5. Bi-directional medium reliability. 
> This will 
> > > > > >> >> >> be  used
> > > > > >> >> >>           where high volume and increased 
> reliability 
> > > > > >> >> >> are  needed. Applications
> > > > > >> >> >>           such as Usage-based Accounting where 95th  
> > > > > >> >> >> percentile billing
> > > > > >> >> >>           model is used. Or even Traffic Engineering 
> > > > > >> >> >> where  network policy
> > > > > >> >> >>           decisions are based on the data collected.
> > > > > >> >> >>
> > > > > >> >> >>        4. Bi-directional high reliability 
> would be used 
> > > > > >> >> >> where  the
> > > > > >> >> >>           Usage-based Accounting application 
> has strict  
> > > > > >> >> >> requirements
> > > > > >> >> >>           on data loss.
> > > > > >> >> >>
> > > > > >> >> >>  Does IPFIX need to support multiple levels of 
> > > > > >> >> >> reliability  and directionality?
> > > > > >> >> >>
> > > > > >> >> >>  Paul
> > > > > >> >> >>
> > > > > >> >> >>  --
> > > > > >> >> >>  Help        
> mailto:[email-masked] and say
> > > > > >> >> >>  "help"
> > > > > >> >> >>  in message body
> > > > > >> >> >>  Unsubscribe 
> mailto:[email-masked] and say  
> > > > > >> >> >> "unsubscribe ipfix" in message body
> > > > > >> >> >>  Archive     http://ipfix.doit.wisc.edu/archive/
> > > > > >> >> >>
> > > > > >> >> >>
> > > > > >> >> >>
> > > > > >> >> >>
> > > > > >> >> >>
> > > > > >> >> > --
> > > > > >> >> > Help        
> mailto:[email-masked] and say "help"
> > > > > >> >> > in message
> > > > > >> >> > body
> > > > > >> >> > Unsubscribe 
> mailto:[email-masked] and say 
> > > > > >> >> > "unsubscribe ipfix" in message body
> > > > > >> >> > Archive     http://ipfix.doit.wisc.edu/archive/
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >> > --
> > > > > >> >> > Help        
> mailto:[email-masked] and say "help"
> > > > > >> >> > in message
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> body
> > > > > >>
> > > > > >>
> > > > > >> >> > Unsubscribe 
> mailto:[email-masked] and say 
> > > > > >> >> > 
> "unsubscribe ipfix" in message body
> > > > > >> >> > Archive     http://ipfix.doit.wisc.edu/archive/
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >> >
> > > > > > --
> > > > > > Help        mailto:[email-masked] and 
> say "help" in
> > > > > > message body
> > > > > > Unsubscribe mailto:[email-masked] and say 
> > > > > > "unsubscribe ipfix" in message body
> > > > > > Archive     http://ipfix.doit.wisc.edu/archive/
> > > > > >
> > > > > >
> > > >
> > > > --
> > > > Help        mailto:[email-masked] and say 
> "help" in message
> > > body
> > > > Unsubscribe mailto:[email-masked] and say 
> > > > "unsubscribe ipfix" in message body
> > > > Archive     http://ipfix.doit.wisc.edu/archive/
> > >
> > > --
> > > Help        mailto:[email-masked] and say 
> "help" in message
> > body
> > > Unsubscribe mailto:[email-masked] and say 
> "unsubscribe 
> > > ipfix" in message body
> > > Archive     http://ipfix.doit.wisc.edu/archive/
> 
> 
> --
> Help        mailto:[email-masked] and say "help" 
> in message body
> Unsubscribe mailto:[email-masked] and say 
> "unsubscribe ipfix" in message body
> Archive     http://ipfix.doit.wisc.edu/archive/
> 

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Dave Plonka <[email-masked]>
Date: Mon, 18 Nov 2002 13:49:23 -0600

Hi Carter,

> negotiation as a requirement?

I don't believe IPFIX ever had capability negotiation as a requirement.

Quoting from "http://ipfix.doit.wisc.edu/IETF53/minutes.txt":

   Regarding capability negotiation, consensus amongst attendees (by a
   hum) was that neither exporter configuration nor capabilities
   negotiation between collectors and exporters should be addressed by
   IPFIX.  The chairs will follow-up with a call on the mailing list on
   this topic.

I was remiss in not following up beyond posting the minutes to the list
for review, but AFAIK the issue has not been contentious given our
"standardize existing practice" mantra and that negotiation hasn't been
substantively mentioned since before IETF53, when Juergen posted his
concerns about negotiation feature creep:

   http://ipfix.doit.wisc.edu/archive/0461.html

to which there were no follow-ups.

Dave

-- 
[email-masked]  http://net.doit.wisc.edu/~plonka  ARS:N9HZF  Madison, WI