Subject: [IPFIX] draft-ietf-ipfix-mediators-problem-statement-03 -> terminology, summary?

From: Benoit Claise <[email-masked]>
Date: Fri, 26 Jun 2009 18:25:04 +0200

Dear all,

I'm trying to summarize the long email thread. Not easy, there are some 
diverging opinions. I hope I haven't forgotten anything
Below is the terminology proposal, based on Brian's proposal, improved 
by Gerhard, and everybody.

2.  Terminology and Definitions

   The terms in this section are in line with those in the IPFIX
   Protocol specifications [RFC5101] and the PSAMP specification
   document [RFC5476].  The terms Observation Point, Observation Domain,
   Flow Key, Flow Record, Data Record, Exporting Process, Exporter,
   IPFIX Device, Collecting Process, Collector, IPFIX Message, Metering
   Process, Transport Session, Information Element, and Template
   Withdrawal Message, are defined in the IPFIX protocol specifications
   [RFC5101].  The terms Packet Report, Sampling, Filtering, PSAMP
   Device, and Configured Selection Fraction are defined in the PSAMP
   specification document [RFC5476].  Furthermore, new terminology to be
   used in the context of IPFIX Mediation is defined in this section.
   All these terms have an initial capital letter in this document.

   In this document, we use the generic term "Data Records" for IPFIX
   Flow Records, PSAMP Packet Reports, Data Records defined by Options
   Templates, unless an explicit distinction is required.

   Original Exporter

      Original Exporter is an IPFIX Device that hosts the Observation
      Points where the metered IP packets are observed.

   IPFIX Mediation   

      IPFIX Mediation describes the manipulation and conversion of records
      for subsequent export using IPFIX, by applying mediation functions
      within one or more Intermediate Processes to a stream of records in
      an IPFIX Mediator.
 
   The following terms are used in this document to describe the
   architectural entities used by IPFIX Mediation.

   Intermediate Process

      An Intermediate Process takes a sequence of records from a Collecting
      Process, Metering Process, IPFIX File Reader, or another Intermediate
      Process within a Mediator; performs some transformation on these 
records
      based upon the content of the records themselves, state kept across
      multiple records, configuration parameters, or other data; and passes
      a sequence of transformed records on to an Exporting Process, 
IPFIX File
      Writer, or another Intermediate Process within a Mediator. This 
document
      describes specific Intermediate Processes below used in the 
elaboration
      of the problem statement; however, this is not an exhaustive list.

   Intermediate Aggregation Process

     An Intermediate Aggregation Process is an Intermediate Process that
     aggregates records based upon a set of Flow Keys, or functions
     applied to fields from the record (e.g. binning, subnet aggregation).

   Intermediate Correlation Process

      An Intermediate Correlation Process is an Intermediate Process
      which adds information to records noting correlations among them,
      or generates new records with correlated data from multiple
      records (e.g. the production of bidirectional flow records from
      unidirectional flow records).

   Intermediate Selection Process

      An Intermediate Selection Process is an Intermediate Process that
      selects records from a sequence based upon criteria evaluated
      record values, and passes only those records that match the
      criteria (e.g. filtering only records from a given network to a
      given Collector).

   Intermediate Anonymisation Process

      An Intermediate Anonymisation Process is an Intermediate Process
      that transforms records in order to anonymise them, to protect the
      identity of the entities described by the records (e.g. by
      applying prefix-preserving pseudonymisation of IP addresses).

   IPFIX Mediator

      An IPFIX Mediator is an IPFIX Device that provides mediation
      capabilities by receiving records from some data source, hosting
      zero or more Intermediate Processes to transform those records,
      and exporting those records in IPFIX Messages via an Exporting
      Process.  In the common case, a Mediator receives records from a
      Collecting Process, but could also receive records from data
      sources not encoded using IPFIX, e.g., in the case of NetFlow V9
      protocol translation.  Specific types of Mediators are defined
      below; some of these reference original mediation capabilities
      defined in earlier IPFIX documens before the elaboration of the
      Mediator problem statement.

   Original Exporter

      Original Exporter is an IPFIX Device that hosts the Observation
      Points where the metered IP packets are observed.

   IPFIX Proxy

      An IPFIX Proxy is an IPFIX Mediator that relays incoming IPFIX
      Messages or messages in other protocols to one or more Collectors.
      It can provide transport protocol mediation and re-encoding.

   IPFIX Concentrator

      An IPFIX Concentrator is an IPFIX Mediator that recieves data from
      one or more Exporters and sends them to a single Collector,
      optionally transforming the records using zero or more
      Intermediate Processes on the way.

   IPFIX Distributor

      An IPFIX Distributor is an IPFIX Mediator that recieves data from
      one or more Exporters and sends them to one or more Collectors,
      deciding which Collector(s) to send each record to based upon the
      decision of an Intermediate Selection Process.

   IPFIX Masquerading Proxy

      An IPFIX Masquerading Proxy is an IPFIX Mediator that recieves
      data from one or more Exporters and sends them to a single
      Collector, using one or more Intermediate Aggregation Functions
      and Intermediate Anonymisation Functions to screen out parts of
      records according to configured policies, in order to protect the
      privacy of the network's end users or senstive data of the
      exporting organization.

I think that we agree that the same terminology sections in both the 
problem statement and framework.
Personally, I'm not sure (yet) if we need the Intermediate 
Aggregation/Correlation/Selection/Anonynisation Process.
However, if we do, my proposal is to omit this list from the problem 
statement, and to simply put them in the framework.

Please let us know, so that we can move forward with the drafts, both 
the problem statements and the framework.

Regards, Benoit.

---

From: Falko Dressler <[email-masked]>
Date: Sat, 27 Jun 2009 14:35:36 +0200

Dear Benoit,

many thanks for this summary. I think that covers the discussion quite well.

> the problem statements and the framework.

Actually, I do think that these intermediate processes are very helpful 
to finally define more complex mediation systems. However, I agree to 
completely move them to the framework draft.

Best regards,
Falko

-- 
Dr. Falko Dressler
Computer Networks and Communication Systems
University of Erlangen, Germany
Phone: +49 9131 85-27914 / Fax: +49 9131 85-27409
mailto:[email-masked]
http://www7.informatik.uni-erlangen.de/~dressler/

---

From: Gerhard Muenz <[email-masked]>
Date: Sat, 27 Jun 2009 15:53:19 +0200

Hi Benoit,

Thanks for the wrap up. I'm fine with this terminology.

Just for clarification:
An Original Exporter hosting an Intermediate Process is not an IPFIX
Mediator according to your definition, right?

s/recieve/receive

Regards,
Gerhard

Benoit Claise wrote:
>      a sequence of transformed records on to an Exporting Process, IPFIX
> File
> _______________________________________________
> IPFIX mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ipfix

-- 
Dipl.-Ing. Gerhard Münz
Chair for Network Architectures and Services (I8)
Department of Informatics
Technische Universität München
Boltzmannstr. 3, 85748 Garching bei München, Germany
Phone:  +49 89 289-18008       Fax: +49 89 289-18033
E-mail: [email-masked]    WWW: http://www.net.in.tum.de/~muenz

---

From: Atsushi Kobayashi <[email-masked]>
Date: Mon, 29 Jun 2009 10:53:19 +0900

Hi Falko and all,

On Sat, 27 Jun 2009 14:35:36 +0200

> completely move them to the framework draft.

I agree to move them to the framework draft.

If we omit the Intermediate Aggregation/Correlation/Selection/Anonynisation
Process, we need to modify the IPFIX Distributor/Masquerading Proxy referring
to the Intermediate FOO Process .

    IPFIX Distributor

       An IPFIX Distributor is an IPFIX Mediator that recieves data from
       one or more Exporters and sends them to one or more Collectors,
       deciding which Collector(s) to send each record to based upon the
       decision of an Intermediate Process.
                      ~~~~~~~~~~~~~~~~~~~~~
    IPFIX Masquerading Proxy

       An IPFIX Masquerading Proxy is an IPFIX Mediator that receives
       data from one or more Exporters and sends them to a single
       Collector, using one or more records aggregation functions
                                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
       and anonymisation functions to screen out parts of
           ~~~~~~~~~~~~~~~~~~~~~~~
       records according to configured policies, in order to protect the
       privacy of the network's end users or senstive data of the
       exporting organization.

Is it OK?

Regards,
Atsushi

> Falko

--- 
Atsushi KOBAYASHI  <[email-masked]>
NTT Information Sharing Platform Lab.
tel:+81-(0)422-59-3978 fax:+81-(0)422-59-5637

---

From: Brian Trammell <[email-masked]>
Date: Mon, 29 Jun 2009 08:30:55 +0200

Hi all,

(returned from holidays, still catching up on the thread, but have the  
following quick point to make...)

actually, there's something weird here, in that we switch from  
Intermediate Foo Processes to Intermediate Foo Functions when talking  
about the Masquerading Proxy. We should stick with Processes (since  
we're defining them), I think. At this level, then, I would suggest  
the following:

    IPFIX Masquerading Proxy

       Collector, using one or more Intermediate Processes to screen  
out parts of
                                    ~~~~~~~~~~~~~~~~~~~~~~

Cheers,

Brian

---

From: Atsushi Kobayashi <[email-masked]>
Date: Mon, 29 Jun 2009 16:47:35 +0900

Hi Gerhard and all,

The definition implies that the Original Exporter hosting IP is not
IPFIX Mediator. Because it mentions that the IPFIX Mediator receives
records from some data source.

To clarify, do we need to modify as follows?

   IPFIX Mediator

      capabilities by receiving records from some exporting device, hosting
                                             ^^^^^^^^^^^^^^^^^^^^^

Regards,
Atsushi

On Sat, 27 Jun 2009 15:53:19 +0200
Gerhard Muenz <[email-masked]> wrote:

> Dipl.-Ing. Gerhard M
> Technische Universitchen
> Boltzmannstr. 3, 85748 Garching bei Mchen, Germany

---

---

From: Gerhard Muenz <[email-masked]>
Date: Mon, 29 Jun 2009 10:05:37 +0200

Hi Atsushi,

> records from some data source.

This is how I understood the terminology.
My question was if this is intended.

>       capabilities by receiving records from some exporting device, hosting

No.

If you want to clarify something, you could add a sentence to the IP
definition, for example:

Intermediate Process
  ...
  Typically, an Intermediate Process is hosted by an IPFIX Mediator.
  Alternatively, an Intermediate Process may be hosted by an Original
  Exporter.

Gerhard

---

From: Atsushi Kobayashi <[email-masked]>
Date: Mon, 29 Jun 2009 17:05:06 +0900

Hi Brian and all,

Your suggestion is fine for me.
Thus, just Intermediate FOO Processes can move to framework draft.

Regards,
Atsushi

On Mon, 29 Jun 2009 08:30:55 +0200

---

---

From: Atsushi Kobayashi <[email-masked]>
Date: Mon, 29 Jun 2009 17:54:00 +0900

Hi Gerhard and all,

Yes, it is more important that Original Exporter can host Intermediate
Process.

Your suggestion is fine for me. However, Intermediate Process definition
already uses term "IPFIX Mediator" in previous paragraph. There is
little bit strange if just your proposal is put in it.

I would like to modify it as follows.

   Intermediate Process

      An Intermediate Process takes a sequence of records from a Collecting
      Process, Metering Process, IPFIX File Reader, or another Intermediate
      Process within a same device; performs some transformation on these records
                      ^^^^^^^^^^^^
      based upon the content of the records themselves, state kept across
      multiple records, configuration parameters, or other data; and passes
      a sequence of transformed records on to an Exporting Process, IPFIX File
      Writer, or another Intermediate Process within a same device. 
                                                     ^^^^^^^^^^^^^
      Typically, an Intermediate Process is hosted by an IPFIX Mediator.
      Alternatively, an Intermediate Process may be hosted by an Original
      Exporter.

Does it make sense?

Regards,
Atsushi

On Mon, 29 Jun 2009 10:05:37 +0200

---

---

From: Gerhard Muenz <[email-masked]>
Date: Mon, 29 Jun 2009 11:39:46 +0200

Hi Atsushi,

> little bit strange if just your proposal is put in it.

Ah, you are right.

>       Process within a same device; performs some transformation on these records

What about removing those parts of the sentence:

Intermediate Process
  An Intermediate Process takes a sequence of records from a Collecting
  Process, Metering Process, IPFIX File Reader, or another Intermediate
  Process; performs some transformation on these records based upon the
  content of the records themselves, state kept across multiple records,
  configuration parameters, or other data; and passes a sequence of
  transformed records on to an Exporting Process, IPFIX File Writer, or
  another Intermediate Proces.
  Typically, an Intermediate Process is hosted by an IPFIX Mediator.
  Alternatively, an Intermediate Process may be hosted by an Original
  Exporter.

Regards,
Gerhard

---

From: Atsushi Kobayashi <[email-masked]>
Date: Mon, 29 Jun 2009 18:40:42 +0900

Hi Gerhard, and all,

It's fine for me. 
Thanks.

Atsushi

On Mon, 29 Jun 2009 11:39:46 +0200

---