Subject: Re: [ipfix] IPFIX-architecture Section 5

From: "K.C. Norseth" <[email-masked]>
Date: Mon, 6 May 2002 08:42:36 -0600

HI all,

----- Original Message -----
From: "kevin.zhang" <[email-masked]>
To: "Benoit Claise" <[email-masked]>
Cc: <[email-masked]>
Sent: Tuesday, April 30, 2002 12:00 PM
Subject: RE: [ipfix] IPFIX-architecture Section 5

| Hi Benoit,
|
| Please see my comments.
|
| > -----Original Message-----
| > From: Benoit Claise [mailto:[email-masked]]
| > Sent: Tuesday, April 30, 2002 11:17 AM
| > To: [email-masked]
| > Cc: [email-masked]
| > Subject: Re: [ipfix] IPFIX-architecture Section 5
| >
| >
| > Hi Kevin,
| >
| >
| > > Hi All,
| > >
| > > I have the following comments regarding section 5.1.2 and
| > section 5.1.4 in draft-ietf-ipfix-architecture-01.txt.
| > >
| > > I quote from our charter, the IPFIX system MUST -
| > > " Ensure that the flow export system is reliable in that it will
| > >    minimize the likelihood of flow data being lost due to resource
| > >    constraints in the exporter or receiver and to accurately report
| > >    such loss if it occurs."
| > >
| > > As minimizing the flow data loss is a primary goal, I believe
| > the functionality specified at the export process and collector
| > is not sufficient.  I thus propose the following modifications -
| > >
| > > -------------------------
| > > Original -
| > > 5.1.2. IPFIX Protocol on IPFIX Device (At Export Process)
| > >
| > >      1. Ability to detect loss of connectivity with the collector and
| > >         trigger the appropriate action (eg. a switch over to an
| > >         alternate collector.)
| > >      2. Optionally export flow records to multiple collectors.
| > >      3. Monitor control information/flow record export, detect any
| > >         overload in the process of exporting.
| > > Modifications -
| > > 5.1.2.  IPFIX Device (at Export Process) Functionality
| > >
| > > 1.  Ability to detect loss of connectivity with the collector
| > and trigger the appropriate actions (e.g. a switch over to an
| > alternate collector)
| > > 2.  Optionally export flow records to multiple collectors.
| > > 3.  Optionally re-transmit flow records upon indications of
| > flow record losses from the collector.
| >
| > I don't like too much the term "re-transmit".
| > What if you don't have the flow records anymore?
| >
| The IPFIX protocol should support application level reliability as user
desires.  This is
| why I added "re-transmission" as an optional exporter functionality.  My
understanding
|of the IPFIX protocol is that we need to provide adequate mechansims that
would
|allow different user requirements (e.g. reliability, redundancy,
performance related
|considerations) being addressed. Whether these capabilities are invoked by
the IPFIX
|users (applications using IPFIX protocol)would be implementation specific.

I think the key word is optionally.  In the case of an overload though,
this usually means there is too much information and storing it on the
optional device may be prohibitive.  In the case where we have the collector
on the device, this could be OK.

Of course, looking intothe future, as devices end up having more storage
capability and such, this may be more feasible.

|
|
| > >
| > > 4.  Monitor control information from the collector and export
| > process, detect any overload conditions in the export process
| > and/or the collector.
| >
| > "Monitor control information from the collector and export process".
| > I think this is confusing. We really want to monitor 2 things:
| > - the control information
| > - the export process
| I agree, this is acutally what I meant. How about changing it to "Monitor
control
|  information from the collector, monitor export process, detect ..."
|
| > And not the control information from the export process, as we
| > could understand it from your sentence.
| >
|
|
| > I disagree with "the overload conditions on the collector", we
| > are concentrating on the exporter
|
| IPFIX protocol is split between exporter and collectors.  If the collector
can indicate
| the overload condition, the exporter should detect it and take actions
just as
|overloading at the export process.  We are addressing an IPFIX system, and
try to
|"Minimize" losses as charter outlined. Detection of collector overloading
is one way of
|minimizing losses, as there's no point to sending to a overloaded
collector, where data is
|dropped; instead, fail-over to a redundant collector would be a much better
approach.
|
|
| > >From the requirements draft:
| > 2.6. Collector
| >
| >    The collector receives flow records from one or more export
| >    processes.  The collector might process or store received flow
| >    record, but these actions are out of the scope of this document.
| >
| > >
| > > ------------------------------------
| > >
| > > ------------------------------------
| > > Original -
| > > 5.1.4. IPFIX Protocol on Collector
| > >
| > >      1. Monitor the reception of export packets from IPFIX device.
| > >      2. Optionally notify the status and overload conditions to the
| > >         IPFIX device.
| > >
| > > Modifications -
| > > 5.1.4. Collector Functionality
| > > 1.  Receive and decode the flow records from the IPFIX devices.
| > > 2.  Ability to indicate flow record losses to the exporting
| > IPFIX device and/or IPFIX users.
| >
| > Agreed.
| >
| > Regards, Benoit
| >
| > >
| > > 3.  Optionally notify the status and overload conditions to the
| > IPFIX device.

How would you propose this to happen?

K.C.

| > > -----------------------------------------
| > >
| > > Thanks,
| > >
| > > Kevin
|

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Benoit Claise <[email-masked]>
Date: Mon, 06 May 2002 16:45:38 +0200

Hi Kevin,

>
>
> > -----Original Message-----
> > From: Benoit Claise [mailto:[email-masked]]
> > Sent: Tuesday, April 30, 2002 11:17 AM
> > To: [email-masked]
> > Cc: [email-masked]
> > Subject: Re: [ipfix] IPFIX-architecture Section 5
> >
> >
> > Hi Kevin,
> >
> >
> > > Hi All,
> > >
> > > I have the following comments regarding section 5.1.2 and
> > section 5.1.4 in draft-ietf-ipfix-architecture-01.txt.
> > >
> > > I quote from our charter, the IPFIX system MUST -
> > > " Ensure that the flow export system is reliable in that it will
> > >    minimize the likelihood of flow data being lost due to resource
> > >    constraints in the exporter or receiver and to accurately report
> > >    such loss if it occurs."
> > >
> > > As minimizing the flow data loss is a primary goal, I believe
> > the functionality specified at the export process and collector
> > is not sufficient.  I thus propose the following modifications -
> > >
> > > -------------------------
> > > Original -
> > > 5.1.2. IPFIX Protocol on IPFIX Device (At Export Process)
> > >
> > >      1. Ability to detect loss of connectivity with the collector and
> > >         trigger the appropriate action (eg. a switch over to an
> > >         alternate collector.)
> > >      2. Optionally export flow records to multiple collectors.
> > >      3. Monitor control information/flow record export, detect any
> > >         overload in the process of exporting.
> > > Modifications -
> > > 5.1.2.  IPFIX Device (at Export Process) Functionality
> > >
> > > 1.  Ability to detect loss of connectivity with the collector
> > and trigger the appropriate actions (e.g. a switch over to an
> > alternate collector)
> > > 2.  Optionally export flow records to multiple collectors.
> > > 3.  Optionally re-transmit flow records upon indications of
> > flow record losses from the collector.
> >
> > I don't like too much the term "re-transmit".
> > What if you don't have the flow records anymore?
> >
> The IPFIX protocol should support application level reliability as user desires.  This is why I added "re-transmission" as an optional exporter functionality.  My understanding of the IPFIX protocol is that we need to provide adequate mechansims that would allow different user requirements (e.g. reliability, redundancy, performance related considerations) being addressed. Whether these capabilities are invoked by the IPFIX users (applications using IPFIX protocol)would be implementation specific.

Then I would simply write:
Optionally re-transmit lost flow records, if possible

>
>
> > >
> > > 4.  Monitor control information from the collector and export
> > process, detect any overload conditions in the export process
> > and/or the collector.
> >
> > "Monitor control information from the collector and export process".
> > I think this is confusing. We really want to monitor 2 things:
> > - the control information
> > - the export process
> I agree, this is acutally what I meant. How about changing it to "Monitor control information from the collector, monitor export process, detect ..."

So at the end, you want to change
from:
     3. Monitor control information/flow record export, detect ....
to:
     3. Monitor control information from the collector, monitor export process, detect ...

This is almost the same. I just think that "Monitor control information" (without from the collector) is more generic.

>
>
> > And not the control information from the export process, as we
> > could understand it from your sentence.
> >
>
> > I disagree with "the overload conditions on the collector", we
> > are concentrating on the exporter
>
> IPFIX protocol is split between exporter and collectors.  If the collector can indicate the overload condition, the exporter should detect it and take actions just as overloading at the export process.  We are addressing an IPFIX system, and try to "Minimize" losses as charter outlined. Detection of collector overloading is one way of minimizing losses, as there's no point to sending to a overloaded collector, where data is dropped; instead, fail-over to a redundant collector would be a much better approach.

Right, but this is part of the "monitoring the control information".
Seeing the requirements definition of the collector "these actions are out of the scope of this document.", I think that we don't want to impose anything requirements to the collector.

To summarize, I would just take back the original sentence:
      3. Monitor control information and flow record export, detect any
         overload in the process of exporting.

Regards, Benoit

>
>
> > >From the requirements draft:
> > 2.6. Collector
> >
> >    The collector receives flow records from one or more export
> >    processes.  The collector might process or store received flow
> >    record, but these actions are out of the scope of this document.
> >
> > >
> > > ------------------------------------
> > >
> > > ------------------------------------
> > > Original -
> > > 5.1.4. IPFIX Protocol on Collector
> > >
> > >      1. Monitor the reception of export packets from IPFIX device.
> > >      2. Optionally notify the status and overload conditions to the
> > >         IPFIX device.
> > >
> > > Modifications -
> > > 5.1.4. Collector Functionality
> > > 1.  Receive and decode the flow records from the IPFIX devices.
> > > 2.  Ability to indicate flow record losses to the exporting
> > IPFIX device and/or IPFIX users.
> >
> > Agreed.
> >
> > Regards, Benoit
> >
> > >
> > > 3.  Optionally notify the status and overload conditions to the
> > IPFIX device.
> > > -----------------------------------------
> > >
> > > Thanks,
> > >
> > > Kevin
> > > ÿñÞ–™šŠ[hþf£¢·hšçzßÝ¢+ÿÂ+ýçnjwlk/ázZÿŠyž²Æ
> > yºÉIì¹»®&Þ™¨¥¶æj:+v‰¨þwýÚ"·ü"±ÏÞvæ§vÆ²þéì¹»®&ÞŠ—âÇø§™ë,j›¡Ü€
> > Èb½èm¶Ÿÿþ*_‹Ý¢+ÿÂ+ýçnýªÜ†+Þ

---

From: "kevin.zhang" <[email-masked]>
Date: Mon, 6 May 2002 11:28:24 -0400

Hi Benoit,

> Sent: Monday, May 06, 2002 10:46 AM
> > The IPFIX protocol should support application level reliability 
> as user desires.  This is why I added "re-transmission" as an 
> optional exporter functionality.  My understanding of the IPFIX 
> protocol is that we need to provide adequate mechansims that 
> would allow different user requirements (e.g. reliability, 
> redundancy, performance related considerations) being addressed. 
> Whether these capabilities are invoked by the IPFIX users 
> (applications using IPFIX protocol)would be implementation specific.
> 

This sounds good to me, let's incorporate the change if nobody rejects it.

> > I agree, this is acutally what I meant. How about changing it 
> to "Monitor control information from the collector, monitor 
> export process, detect ..."
> 
> This is almost the same. I just think that "Monitor control 
> information" (without from the collector) is more generic.
> 
As the term "control information" is specific in the IPFIX context, that is why we should explicitly indicate its origin.  Other status information may come from other processes residing on the IPFIX exporting device, of which export process is of our interests here. 

> > IPFIX protocol is split between exporter and collectors.  If 
> the collector can indicate the overload condition, the exporter 
> should detect it and take actions just as overloading at the 
> export process.  We are addressing an IPFIX system, and try to 
> "Minimize" losses as charter outlined. Detection of collector 
> overloading is one way of minimizing losses, as there's no point 
> to sending to a overloaded collector, where data is dropped; 
> instead, fail-over to a redundant collector would be a much 
> better approach.
> 
> Seeing the requirements definition of the collector "these 
> actions are out of the scope of this document.", I think that we 
> don't want to impose anything requirements to the collector.
> 
I believe we meant that how the collector processes flow information is out of the scope. The collector's functionality to support protocol is within the scope. The bottom line is that the IPFIX protocol MUST provide mechanisms (or capabilities) to support the highly reliable flow exporting, that is reliable (application level ack) and robust (failover, failback). Whether the IPFIX end points want to invoke these capabilities is implementation specific and depending on the applications using the IPFIX system.

> > > Èb½èm¶Ÿÿþ*_‹Ý¢+ÿÂ+ýçnýªÜ†+ÞÿñÞ–™šŠ[hþf£¢·hšçzßÝ¢+ÿÂ+ýçnjwlk/ázZÿŠyž²Æ yºÉIì¹»®&Þ™¨¥¶æj:+v‰¨þwýÚ"·ü"±ÏÞvæ§vÆ²þéì¹»®&ÞŠ—âÇø§™ë,j›¡Ü€Èb½èm¶Ÿÿþ*_‹Ý¢+ÿÂ+ýçnýªÜ†+Þ

---

From: Benoit Claise <[email-masked]>
Date: Wed, 08 May 2002 11:16:14 +0200

Hi Kevin,

>>>>>     1. Ability to detect loss of connectivity with the 
>>>>>          
>>>>>
>>collector and
>>    
>>
No. If you look at the definition from the architecture draft, the 
control information is exported. So we don't speak of control 
information from other processes.

* Control Stream, Data Stream:
The information that needs to be exported from the IPFIX device
can be classified into the following categories:

Norseth & Sadasivan [Page 6]

Internet Draft draft-ietf-ipfix-architecture-01.txt February 2002

- Control Information :
This includes the flow type definition, selection criteria
for packets within the flow. This is also called as Control
Stream. This stream carries all the information for the
end-points to understand the IPFIX protocol and specifically
for the receiver to understand and interpret the data stream
send by the sender.

- Flow record :
This includes data records corresponding to the information
on various observed flows at each of the observation
point.This is also called as Data Stream.

>I believe we meant that how the collector processes flow information is out of the scope. The collector's functionality to support protocol is within the scope. The bottom line is that the IPFIX pro…
Again, I disagree with the MUST.
If the charter was saying: "Ensure that the flow export system is 
reliable", then I would agree.
But the charter says: "Ensure that the flow export system is reliable in 
that it will minimize the likelihood of flow data being lost due to 
resource constraints
in the exporter or receiver and to accurately report such loss if it 
occurs."

Regards, Benoit

>>>>>ÿñÞ-^(TM)s(S([hþf£¢·hs(?çzßÝ¢+ÿÂ+ýçnjwlk/ázZÿS(yz(²Æ
>>>>>          
>>>>>
>>>>yºÉIì¹»®&Þ^(TM)¨¥¶?æj:+v0/00¨þwýÚ"·ü"±ÏÞvæ§vÆ²þéì¹»®&ÞS(--âÇø§^(TM)ë,j>¡ÜEUR
>>>>Èb½èm¶Y"ÿþ*_<Ý¢+ÿÂ+ýçnýªÜ++Þ
>>>>        
>>>>

---

From: "kevin.zhang" <[email-masked]>
Date: Wed, 8 May 2002 10:40:18 -0400

Hi Benoit,

> Sent: Wednesday, May 08, 2002 5:16 AM
> >As the term "control information" is specific in the IPFIX 
> context, that is why we should explicitly indicate its origin.  
> Other status information may come from other processes residing 
> on the IPFIX exporting device, of which export process is of our 
> interests here. 
> >
I don't agree that the control information can only be exported from an exportor to the collector. I raised this point many times on the mailing list and at the last meeting, and I believe certain agreement on this had achieved.  I had another email proposing changes to this definition.  Please see my email re. Terminology - Control Stream, Data Stream.  I looked many standard protocols, and have not been able to find any uni-directional restrictions on control information exchanges.  As we are using a connection-oriented transport protocol (TCP, SCTP), it is reasonable to assume that IPFIX is connection oriented, and the control information exchange helps to setup the connection/session.

> > 
> >I believe we meant that how the collector processes flow 
> information is out of the scope. The collector's functionality to 
> support protocol is within the scope. The bottom line is that the 
> IPFIX protocol MUST provide mechanisms (or capabilities) to 
> support the highly reliable flow exporting, that is reliable 
> (application level ack) and robust (failover, failback). Whether 
> the IPFIX end points want to invoke these capabilities is 
> implementation specific and depending on the applications using 
> the IPFIX system.
> >

Minimizing the likelihood of flow data loss is a typical objective for any systems.  This means given the constraints of the system (e.g. exporter processing and memory resource), the loss should be minimized.  If there's no such constraint, no data loss should be achieved.  

We should not design a protocol that intrinsically can not ensure reliability even when exporter/collector resources are not a issue.

> ÿñÞ–™šŠ[hþf£¢·hšçzßÝ¢+ÿÂ+ýçnjwlk/ázZÿŠyž²Æ yºÉIì¹»®&Þ™¨¥¶æj:+v‰¨þwýÚ"·ü"±ÏÞvæ§vÆ²þéì¹»®&ÞŠ—âÇø§™ë,j›¡Ü€Èb½èm¶Ÿÿþ*_‹Ý¢+ÿÂ+ýçnýªÜ†+Þ

---

From: Benoit Claise <[email-masked]>
Date: Wed, 08 May 2002 17:02:15 +0200

Hi Kevin,

>>>This sounds good to me, let's incorporate the change if nobody 
>>>      
>>>
>>rejects it.
>>    
>>
>>> 
>>>
>>>      
>>>
>>>>>     
>>>>>
>>>>>          
>>>>>
I was not discussing the direction of the control information.
I was discussing the fact that the control information is between the 
exporter and the collector. And not between the export process on the 
exporter and any other processes on the exporter. I was referring to 
your sentence (that I disagree with):

"Other status information may come from other processes residing 
on the IPFIX exporting device, of which export process is of our 
interests here."

Regards, Benoit

>>>>>>yºÉIì¹»®&Þ^(TM)¨¥¶?æj:+v0/00¨þwýÚ"·ü"±ÏÞvæ§vÆ²þéì¹»®&ÞS(--
>>>>>>            
>>>>>>
>>âÇø§^(TM)ë,j>¡ÜEUR
>>    
>>