Subject: RE: [ipfix] IPFIX followup charter

From: "Tanja Zseby" <[email-masked]>
Date: Thu, 2 Feb 2006 11:27:15 +0100

Hi Jürgen and all,

I agree that due to the high amount of drafts we need to decide on the most important ones for the first round. Nevertheless, I think we should consider to include the bi-flow draft already now in the new charter. 
The idea for such a draft originated at the last FloCon workshop where a lot of people had doubts about IPFIX because it currently lacks information on how to do bi-flow matching/reporting. Many of the participants have written their own programs in the past to allow bi-flow analysis from NetFlow data. So as a result of discussions during the workshop, Brian and Elisa decided to work on such a draft. I agree that there is not yet very much content in the current version of the draft but since this request came from a larger community I consider it as important. 

Brian, Elisa maybe you can comment on this? Do you share my viewpoint? How would you prioritize your work in IPFIX (since you are also co-authors in other drafts)? 

Best regards,
Tanja 

-----Original Message-----
From: majordomo listserver [mailto:[email-masked]] On Behalf Of Juergen Quittek
Sent: Tuesday, January 31, 2006 11:42 PM
To: [email-masked]
Subject: [ipfix] IPFIX followup charter

Dear all,

As you know, the IPFIX working group has completed its charter by submitting all planned documents (with a delay of several years) to the IESG for publication as RFC.  Also the PSAMP WG will reach this status soon.

Building on the results of these WGs, there are a lot of related ongoing activities that are producing Internet drafts related to IPFIX.  Several of them have already been presented at recent IPFIX and PSAMP sessions.  But working on them is not covered by the IPFIX or PSAMP charter.  If we want to continue this IPFIX-related work, we need a new charter that gives it a home at the IETF.

The text below lists and discusses related work and suggests a charter for a follow-up WG.  It is the output of several discussions  with Tanja, Benoit, and Nevil.

The proposed charter is very short-lived and includes only the three most mature work items out of a longer list of candidates.  The basic idea is completing this charter within less than a year and then re-chartering to cover further work items that have progressed until then.  This lean work model with short-lived charters allows the group to focus on a limited number of issues and is preferable to long-lived WGs working on many issues in parallel.  (It is highly recommended by the IESG.)

Please have a look at it and state whether or not you think it makes sense to have an IPFIX follow-up WG.  Also please read the proposed charter carefully and express your objections, concerns, comments, requests for modifications, etc.

The plan is to elaborate the new charter proposal on this list and submit an agreed version to our area directors soon.  The deadline for requesting BoF sessions at the next IETF meeting in Dallas is February 13.

Thanks,

    Juergen

=======================================================
Why do we need to continue the work of IPFIX and PSAMP?
=======================================================

IPFIX has completed its charter and PSAMP will do so very soon.  Still, there are a lot of ongoing activities in the community of these two WGs:

1. Flow aggregation
   draft-dressler-ipfix-aggregation-01.txt

2. reducing redundancy in IPFIX and PSAMP reports
   draft-boschi-export-perpktinfo-00.txt

3. IPFIX implementation guidelines
   draft-boschi-ipfix-implementation-guidelines-00.txt

4. Path-coupled meter configuration
   draft-fessi-nsis-m-nslp-framework-02.txt
   draft-dressler-nsis-metering-nslp-03.txt
   (currently under discussion in the NSIS WG, but not covered
   by the NSIS charter. It is a candidate work item for NSIS
   re-chartering, but the NSIS WG asks if it would not be better
   covered by IPFIX)

5. IPFIX reliability
   draft-bclaise-ipfix-reliability-00.txt

6. Reporting bi-directional flows with IPFIX
   draft-boschi-ipfix-biflow-01.txt

7. a format for storing IPFIX records
   draft-trammell-ipfix-file-00.txt

8. IPFIX MIB module
   no I-D yet, but two teams working on it independently.

9. Common IPFIX templates
   draft-stephan-isp-templates-01.txt

10. Reliable server pooling for IPFIX
    draft-coene-rserpool-applic-ipfix-01.txt

11. Flow sampling
    draft-molina-flow-selection-00.txt (expired)

Did I miss something?

1.-4. and 9. have already been discussed at past IPFIX or PSAMP sessions.

This list shows two things.
  - There is a community interested in IPFIX that is not too small.
  - This community and is willing to further work on issues IPFIX-related
    issues in the IETF.
This is a very good starting point for a charter discussion.

============================
Which work items are suited?
============================

Not all of the issues listed above are at a stage, where they should be considered as a WG work item, but 1.-4. are quite well developed and 5.
and 6. are candidates.  Since 4. is a candidate for NSIS re-chartering, I dropped it from the following considerations.

Each of 1.-3. has been presented at IPFIX or PSAMP sessions already two times with some discussion on the suggested approach.  For all I sense an agreement in the IPFIX WG (at least no objections so far) that these issues are relevant work and potential WG work items (to be confirmed on the list, of course).

  - The flow aggregation work is rather mature.  Actually this draft covers
    something that is missing in IPFIX: How to tell the collector that the
    metering probe does not have the standard (Netflow default) configuration,
    but filters and aggregates certain flows.
    There are some terminology problems and a set of technical issues to be
    solved, but there is not problem with the general direction and the chosen
    approach.  Still, thorough reviews are missing as well as a discussion on
    how to fit it well into the IPFIX architecture.

  - Reducing redundancy in IPFIX and PSAMP reports is an issue that was
    received very well by both WGs when past versions of the IDs were presented.
    It is considered a useful method of applying IPFIX efficiently.
    Still, the current drafts were considered as to specific.  They apply
    the optimization to packet reports only. At the last PSAMP meeting it was
    noted that the method should be generalized such that it can be applied to
    all redundant IPFIX transmissions.  This generalization needs to be done,
    but the way to go is clear and basically agreed on.

  - The implementation guidelines are considered the most important work item
    by many WG members (including myself).  Many people are currently implementing
    IPFIX and several recommendations were identified at the first IPFIX interop
    (next one is scheduled for end of February).  The sooner this document is
    available, the more will help improving ongoing implementations.
    My problem with this item is that the current individual draft is in a bad
    shape.  Therefore, the milestones for this item are later than for the
    others.

The two weaker candidates for WG items are IPFIX reliability and reporting of bidirectional flows.  Both have been requested on the IPFIX mailing list several time in the past years, but we could not agree on making them part of the basic IPFIX standard.
But as add-ons, that integrate well with the standard, they can be considered, particularly since I heard about operator requests for both of them.
A problem of these issues is that so far they have not been presented at IPFIX or PSAMP sessions and there has not been a discussion yet on the approaches followed by the existing drafts.  Therefore, these two are not included in the draft proposal below.

=================================
Charter Proposal:
Efficient Use of IPFIX (USEIPFIX)
=================================

The IPFIX working group has specified the IPFIX protocol for exporting flow records. The PSAMP working group has specified the usage of the IPFIX protocol for exporting packet records. With both specifications available, several implementers have started building applications using the IPFIX protocol.

At a first interoperability testing event, several IPFIX protocol implementations were tested. The experiences made at this event were fed back to IPFIX protocol specification, particularly for removing ambiguities.  In addition, several lessons were learned about how to implement and use IPFIX correctly and efficiently.  The exchange among different implementers further led to new ideas for advanced usage of IPFIX.  Many of these ideas are currently documented in individual Internet drafts.

The goal of the USEIPFIX working group is producing best current practice and guideline documents concerning implementation, application and usage of the IPFIX protocol.

Out of scope are modifications of the core IPFIX and PSAMP protocol specifications.  In scope is the definition of new IPFIX and PSAMP information elements within the documents produced by the USEIPFIX WG.

Specific Goals of the USEIPFIX WG are:

o Developing guidelines for implementers based on experiences
  gained individually by implementers and jointly at interoperability
  testing events.  The guidelines will give recommendations for
  integrating IPFIX observation points, measurement processes, and
  exporting processes into the packet flow at different kinds of IPFIX
  devices.  They will make suggestions for efficient implementation of
  the IPFIX protocol features and identify parts of the IPFIX
  specification that have already been misunderstood by several
  implementers.  For some implementation choices that the protocol
  specification leaves to the implementer, the guidelines will discuss
  advantages and disadvantages of the different choices.

o Developing methods and means for an efficient use of the IPFIX
  protocol that reduces redundancy in flow reports.  The basic idea
  to be followed is very simple.  For multiple flow records that all
  report the same value in one or more of the contained IPFIX
  information elements, these values are removed from the flow records
  and instead reported once for all in a separate record.  Such an
  approach integrates very well with the IPFIX protocol and only needs
  few means for expressing the relationship between flow records and
  corresponding separate records.

o Develop a method for flow aggregation reducing the amount of
  measurement data exchanged between IPFIX exporters and IPFIX
  collectors.  Using aggregation techniques, measurement information of
  multiple similar flows is aggregated into few meta-flow records.
  Applied aggregation rules need to be communicate.

o Investigate further ways of efficiently using the IPFIX protocols
  including but not limited to
    - providing reliability for IPFIX transmissions,
    - reporting bi-directional flows,
    - path-coupled configuration of IPFIX devices,
    - reporting the configuration of IPFIX devices,
    - flow sampling at IPFIX devices,
    - storing IPFIX flow records and packet records.
  These issues are not current work items of the USEIPFIX WG but are
  evaluated as candidates for potential future work items.

Milestones:

Mar 2006  Initial version of flow aggregation methods Mar 2006  Initial version of reducing redundandy in IPFIX records Mar 2006  IPFIX and PSAMP interoperability testing event (not a real WG milestone?) Apr 2006  Initial version of implementation guidelines Jul 2006  Submit flow aggregation methods to IESG Sep 2006  Submit reducing redundancy in IPFIX records to IESG Sep 2006  Submit implementation guidelines to IESG

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say "unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Brian Trammell <[email-masked]>
Date: Thu, 2 Feb 2006 12:46:14 -0500

Tanja,

As co-author of the biflow draft, of course I agree with you that it  
should be included in this charter. There has not been much  
discussion among the wider group on the biflow draft because there  
has not been a full working group meeting since the effort was  
initiated. That said, in one-on-one meetings in Vancouver, we did  
recieve some very good feedback and are incorporating it into a -02  
draft that will be available well before Dallas for discussion.

There's not a whole lot of content in -01 (or the forthcoming -02 for  
that matter) mainly because single-record biflow export is a  
relatively simple idea with a relatively simple design. I personally  
don't anticipate that adding the biflow draft to the proposed short- 
term charter will have a significant impact on the amount of time  
required to see it to completion.

Given that many of us in the security-oriented flow analysis  
community, where the question "did anyone answer?" often separates  
mere concern from actionability, consider biflow export support to be  
quite important, failure to improve upon the relatively inefficient  
current methods for biflow export in IPFIX may hinder its adoption as  
anything other than a slightly more complicated NetFlow V9: another  
data source to read from and translated to one of a number of  
proprietary collection protocols as near to the sensing edge as  
possible. Delaying official WG action on this until July 2007, as I  
believe leaving biflow support off the next charter and allowing for  
one IETF meeting cycle of slack in the schedule implies, won't help  
IPFIX adoption within this community. By that time, CERT/NetSA's own  
biflow export system will have been in production use for a year,  
albeit with CERT private enterprise IEs standing in for official  
reverse-direction value fields.

It would instead be preferable to continue this work within the  
working group without delay.

Regards,

Brian

On Feb 2, 2006, at 5:27 AM, Tanja Zseby wrote:

> Hi Jürgen and all,
>
> I agree that due to the high amount of drafts we need to decide on  
> the most important ones for the first round. Nevertheless, I think  
> we should consider to include the bi-flow draft already now in the  
> new charter.
> The idea for such a draft originated at the last FloCon workshop  
> where a lot of people had doubts about IPFIX because it currently  
> lacks information on how to do bi-flow matching/reporting. Many of  
> the participants have written their own programs in the past to  
> allow bi-flow analysis from NetFlow data. So as a result of  
> discussions during the workshop, Brian and Elisa decided to work on  
> such a draft. I agree that there is not yet very much content in  
> the current version of the draft but since this request came from a  
> larger community I consider it as important.
>
> Brian, Elisa maybe you can comment on this? Do you share my  
> viewpoint? How would you prioritize your work in IPFIX (since you  
> are also co-authors in other drafts)?
>
> Best regards,
> Tanja
>
> As you know, the IPFIX working group has completed its charter by  
> submitting all planned documents (with a delay of several years) to  
> the IESG for publication as RFC.  Also the PSAMP WG will reach this  
> status soon.
>
> Building on the results of these WGs, there are a lot of related  
> ongoing activities that are producing Internet drafts related to  
> IPFIX.  Several of them have already been presented at recent IPFIX  
> and PSAMP sessions.  But working on them is not covered by the  
> IPFIX or PSAMP charter.  If we want to continue this IPFIX-related  
> work, we need a new charter that gives it a home at the IETF.
>
> The text below lists and discusses related work and suggests a  
> charter for a follow-up WG.  It is the output of several  
> discussions  with Tanja, Benoit, and Nevil.
>
> The proposed charter is very short-lived and includes only the  
> three most mature work items out of a longer list of candidates.   
> The basic idea is completing this charter within less than a year  
> and then re-chartering to cover further work items that have  
> progressed until then.  This lean work model with short-lived  
> charters allows the group to focus on a limited number of issues  
> and is preferable to long-lived WGs working on many issues in  
> parallel.  (It is highly recommended by the IESG.)
>
> Please have a look at it and state whether or not you think it  
> makes sense to have an IPFIX follow-up WG.  Also please read the  
> proposed charter carefully and express your objections, concerns,  
> comments, requests for modifications, etc.
>
> The plan is to elaborate the new charter proposal on this list and  
> submit an agreed version to our area directors soon.  The deadline  
> for requesting BoF sessions at the next IETF meeting in Dallas is  
> February 13.
>
> Thanks,
>
>     Juergen
>
>
> IPFIX has completed its charter and PSAMP will do so very soon.   
> Still, there are a lot of ongoing activities in the community of  
> these two WGs:
>
> 1. Flow aggregation
> Not all of the issues listed above are at a stage, where they  
> should be considered as a WG work item, but 1.-4. are quite well  
> developed and 5.
> and 6. are candidates.  Since 4. is a candidate for NSIS re- 
> chartering, I dropped it from the following considerations.
>
> Each of 1.-3. has been presented at IPFIX or PSAMP sessions already  
> two times with some discussion on the suggested approach.  For all  
> I sense an agreement in the IPFIX WG (at least no objections so  
> far) that these issues are relevant work and potential WG work  
> items (to be confirmed on the list, of course).
>
> The two weaker candidates for WG items are IPFIX reliability and  
> reporting of bidirectional flows.  Both have been requested on the  
> IPFIX mailing list several time in the past years, but we could not  
> agree on making them part of the basic IPFIX standard.
> But as add-ons, that integrate well with the standard, they can be  
> considered, particularly since I heard about operator requests for  
> both of them.
> A problem of these issues is that so far they have not been  
> presented at IPFIX or PSAMP sessions and there has not been a  
> discussion yet on the approaches followed by the existing drafts.   
> Therefore, these two are not included in the draft proposal below.
>
>
> The IPFIX working group has specified the IPFIX protocol for  
> exporting flow records. The PSAMP working group has specified the  
> usage of the IPFIX protocol for exporting packet records. With both  
> specifications available, several implementers have started  
> building applications using the IPFIX protocol.
>
> At a first interoperability testing event, several IPFIX protocol  
> implementations were tested. The experiences made at this event  
> were fed back to IPFIX protocol specification, particularly for  
> removing ambiguities.  In addition, several lessons were learned  
> about how to implement and use IPFIX correctly and efficiently.   
> The exchange among different implementers further led to new ideas  
> for advanced usage of IPFIX.  Many of these ideas are currently  
> documented in individual Internet drafts.
>
> The goal of the USEIPFIX working group is producing best current  
> practice and guideline documents concerning implementation,  
> application and usage of the IPFIX protocol.
>
> Out of scope are modifications of the core IPFIX and PSAMP protocol  
> specifications.  In scope is the definition of new IPFIX and PSAMP  
> information elements within the documents produced by the USEIPFIX WG.
>
> Mar 2006  Initial version of flow aggregation methods Mar 2006   
> Initial version of reducing redundandy in IPFIX records Mar 2006   
> IPFIX and PSAMP interoperability testing event (not a real WG  
> milestone?) Apr 2006  Initial version of implementation guidelines  
> Jul 2006  Submit flow aggregation methods to IESG Sep 2006  Submit  
> reducing redundancy in IPFIX records to IESG Sep 2006  Submit  
> implementation guidelines to IESG
>
>
> --

---

From: Carter Bullard <[email-masked]>
Date: Thu, 2 Feb 2006 14:06:38 -0500

Gentle people,
My two cents worth.  I do believe that the biggest problems for IPFIX  
adoption are
first, its requirement for SCTP and second its shortcomings in  
extensibility, which
include (but are not limited to) issues in support for vendor  
specific information
elements.

I would be interested in using IPFIX if it could support multicast  
transport, and if
it could support middlebox recognition and processing of vendor  
specific proprietary
information elements in an efficient/utilitarian way.     If these  
topics were in the
next IPFIX WG effort, then that would help a great deal!!!!

But, I would be more interested in supporting an effort to describe  
an IPFIX file
format (exchange doesn't happen just on the wire you know).   While  
the IETF
would probably not support a working group for a file format, any  
future IPFIX
WG should include in its set of goals an informational RFC that  
describes issues for
persistent IPFIX data.

Any idea if you guys are going to have a formal meeting in Dallas?

Carter

On Feb 2, 2006, at 12:46 PM, Brian Trammell wrote:

> Tanja,
>
> As co-author of the biflow draft, of course I agree with you that  
> it should be included in this charter. There has not been much  
> There's not a whole lot of content in -01 (or the forthcoming -02  
> for that matter) mainly because single-record biflow export is a  
> relatively simple idea with a relatively simple design. I  
> personally don't anticipate that adding the biflow draft to the  
> proposed short-term charter will have a significant impact on the  
> amount of time required to see it to completion.
>
> mere concern from actionability, consider biflow export support to  
> be quite important, failure to improve upon the relatively  
> inefficient current methods for biflow export in IPFIX may hinder  
> its adoption as anything other than a slightly more complicated  
> NetFlow V9: another data source to read from and translated to one  
> of a number of proprietary collection protocols as near to the  
> sensing edge as possible. Delaying official WG action on this until  
> July 2007, as I believe leaving biflow support off the next charter  
> and allowing for one IETF meeting cycle of slack in the schedule  
> implies, won't help IPFIX adoption within this community. By that  
> time, CERT/NetSA's own biflow export system will have been in  
> production use for a year, albeit with CERT private enterprise IEs  
> standing in for official reverse-direction value fields.
>
>> discussions during the workshop, Brian and Elisa decided to work  
>> on such a draft. I agree that there is not yet very much content  
>> in the current version of the draft but since this request came  
>> from a larger community I consider it as important.
>>
>> submitting all planned documents (with a delay of several years)  
>> to the IESG for publication as RFC.  Also the PSAMP WG will reach  
>> this status soon.
>>
>> IPFIX.  Several of them have already been presented at recent  
>> IPFIX and PSAMP sessions.  But working on them is not covered by  
>> the IPFIX or PSAMP charter.  If we want to continue this IPFIX- 
>> related work, we need a new charter that gives it a home at the IETF.
>>
>> Each of 1.-3. has been presented at IPFIX or PSAMP sessions  
>> already two times with some discussion on the suggested approach.   
>> For all I sense an agreement in the IPFIX WG (at least no  
>> objections so far) that these issues are relevant work and  
>> potential WG work items (to be confirmed on the list, of course).
>>
>> IPFIX mailing list several time in the past years, but we could  
>> not agree on making them part of the basic IPFIX standard.
>> usage of the IPFIX protocol for exporting packet records. With  
>> both specifications available, several implementers have started  
>> Out of scope are modifications of the core IPFIX and PSAMP  
>> protocol specifications.  In scope is the definition of new IPFIX  
>> and PSAMP information elements within the documents produced by  
>> the USEIPFIX WG.
>>

---

From: Elisa Boschi <[email-masked]>
Date: Fri, 03 Feb 2006 09:16:25 +0100

Dear all,

I agree with Tanja and Brian and think that the biflow draft should be
included already now in the new charter.  Reporting bidirectional flows
has been requested by a large community and therefore I too consider it
important

As Brian said,  the reason why we didn't present the draft at the IETF
is that there was no IPFIX meeting scheduled in Vancouver. Still, we
received comments and feedback and are incorporating them in the -02
version. We will definitely request a slot to present this draft at the
upcoming IPFIX (or USEIPFIX) meeting in Dallas.

Apart from this, I agree with Jurgen's proposed charter and milestones.

Regards,
Elisa

Brian Trammell wrote:

>> such a draft. I agree that there is not yet very much content in  the
>> current version of the draft but since this request came from a 
>> and PSAMP sessions.  But working on them is not covered by the  IPFIX
>> or PSAMP charter.  If we want to continue this IPFIX-related  work,
>> we need a new charter that gives it a home at the IETF.
>>
>> The proposed charter is very short-lived and includes only the  three
>> most mature work items out of a longer list of candidates.   The
>> basic idea is completing this charter within less than a year  and
>> then re-chartering to cover further work items that have  progressed
>> until then.  This lean work model with short-lived  charters allows
>> the group to focus on a limited number of issues  and is preferable
>> to long-lived WGs working on many issues in  parallel.  (It is highly
>> recommended by the IESG.)
>>
>> Please have a look at it and state whether or not you think it  makes
>> sense to have an IPFIX follow-up WG.  Also please read the  proposed
>> charter carefully and express your objections, concerns,  comments,
>> requests for modifications, etc.
>>
>> Not all of the issues listed above are at a stage, where they  should
>> be considered as a WG work item, but 1.-4. are quite well  developed
>> and 5.
>> two times with some discussion on the suggested approach.  For all  I
>> sense an agreement in the IPFIX WG (at least no objections so  far)
>> that these issues are relevant work and potential WG work  items (to
>> be confirmed on the list, of course).
>>
>> specifications available, several implementers have started  building
>> applications using the IPFIX protocol.
>>
>> implementations were tested. The experiences made at this event  were
>> fed back to IPFIX protocol specification, particularly for  removing
>> ambiguities.  In addition, several lessons were learned  about how to
>> implement and use IPFIX correctly and efficiently.   The exchange
>> among different implementers further led to new ideas  for advanced
>> usage of IPFIX.  Many of these ideas are currently  documented in
>> individual Internet drafts.
>>

---

From: Juergen Quittek <[email-masked]>
Date: Fri, 03 Feb 2006 15:58:33 +0100

Hi Carter,

--On 2/2/06 2:06 PM -0500 Carter Bullard wrote:

> Gentle people,
> My two cents worth.  I do believe that the biggest problems for IPFIX  adoption are
> first, its requirement for SCTP and second its shortcomings in  extensibility, which
> include (but are not limited to) issues in support for vendor  specific information
> elements.
>
> I would be interested in using IPFIX if it could support multicast  transport, and if
> it could support middlebox recognition and processing of vendor  specific proprietary
> information elements in an efficient/utilitarian way.     If these  topics were in the
> But, I would be more interested in supporting an effort to describe  an IPFIX file
> format (exchange doesn't happen just on the wire you know).   While  the IETF
> would probably not support a working group for a file format, any  future IPFIX
> WG should include in its set of goals an informational RFC that  describes issues for
> persistent IPFIX data.

I also think this would be a useful piece to have.
But so far, nobody has come up with an initial suggestion.
The way I would like to see the new charter is that it is short lived
and updated frequently.  We could discuss in our meetings already upcoming
work items that are submitted as individual drafts and potential candidates
to be covered by the next charter update.

> Any idea if you guys are going to have a formal meeting in Dallas?

Yes, we will definitely meet, just the way how we organize it is not clear.

Formally, the best way would be having a BoF session.  But if this approach
fails, we can go on like last time and host discussions on these work items
at IPFIX or PSAMP sessions.

Thanks,

    Juergen

---

From: Brian Trammell <[email-masked]>
Date: Fri, 3 Feb 2006 10:00:47 -0500

> at IPFIX or PSAMP sessions.

My preference would be for a USEIPFIX BoF - it would help having  
everyone in the room to gauge WG interest in the length and scope of  
the first follow-on charter.