Subject: Re: [IPFIX] draft-ietf-ipfix-mediators-problem-statement-03 -> terminology, summary?

From: Atsushi Kobayashi <[email-masked]>
Date: Tue, 07 Jul 2009 01:04:35 +0900

Dear Brian, Gerhard, Benoit and all,

While writing next version IPFIX problem statement and framework draft,
I found the following discrepancy between IPFIX Mediation and IPFIX
Mediator. the term IPFIX Mediation implies that IPFIX Mediator hosts one or
more Intermediate Processes.

   IPFIX Mediation

      IPFIX Mediation describes the manipulation and conversion of
      records for subsequent export using IPFIX, by applying mediation
      functions within one or more Intermediate Processes to a stream of
                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      records in an IPFIX Mediator.

   IPFIX Mediator

      An IPFIX Mediator is an IPFIX Device that provides mediation
      capabilities by receiving records from some data source, hosting
      zero or more Intermediate Processes to transform those records,
      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      and exporting those records in IPFIX Messages via an Exporting
      Process.  In the common case, a Mediator receives records from a
      Collecting Process, but could also receive records from data
      sources not encoded using IPFIX, e.g., in the case of NetFlow V9
      protocol translation.

I'd like to propose the new term IPFIX Mediation as follows.

   IPFIX Mediation

      IPFIX Mediation describes the manipulation and conversion of
      records or IPFIX Messages for subsequent export using IPFIX, by
      applying mediation functions within one or more Intermediate
      Processes to a stream of records, or applying transport based 
      mediation functions to IPFIX Messages.

Does it make sense?

Regards,
Atsushi

On Fri, 26 Jun 2009 18:25:04 +0200
Benoit Claise <[email-masked]> wrote:

> Dear all,
> 
> I'm trying to summarize the long email thread. Not easy, there are some 
> diverging opinions. I hope I haven't forgotten anything
> Below is the terminology proposal, based on Brian's proposal, improved 
> by Gerhard, and everybody.
> 
> 2.  Terminology and Definitions
> 
>    The terms in this section are in line with those in the IPFIX
>    Protocol specifications [RFC5101] and the PSAMP specification
>    document [RFC5476].  The terms Observation Point, Observation Domain,
>    Flow Key, Flow Record, Data Record, Exporting Process, Exporter,
>    IPFIX Device, Collecting Process, Collector, IPFIX Message, Metering
>    Process, Transport Session, Information Element, and Template
>    Withdrawal Message, are defined in the IPFIX protocol specifications
>    [RFC5101].  The terms Packet Report, Sampling, Filtering, PSAMP
>    Device, and Configured Selection Fraction are defined in the PSAMP
>    specification document [RFC5476].  Furthermore, new terminology to be
>    used in the context of IPFIX Mediation is defined in this section.
>    All these terms have an initial capital letter in this document.
> 
>    In this document, we use the generic term "Data Records" for IPFIX
>    Flow Records, PSAMP Packet Reports, Data Records defined by Options
>    Templates, unless an explicit distinction is required.
> 
>    Original Exporter
> 
>       Original Exporter is an IPFIX Device that hosts the Observation
>       Points where the metered IP packets are observed.
> 
>    IPFIX Mediation   
> 
>       IPFIX Mediation describes the manipulation and conversion of records
>       for subsequent export using IPFIX, by applying mediation functions
>       within one or more Intermediate Processes to a stream of records in
>       an IPFIX Mediator.
>  
>    The following terms are used in this document to describe the
>    architectural entities used by IPFIX Mediation.
> 
>    Intermediate Process
> 
>       An Intermediate Process takes a sequence of records from a Collecting
>       Process, Metering Process, IPFIX File Reader, or another Intermediate
>       Process within a Mediator; performs some transformation on these 
> records
>       based upon the content of the records themselves, state kept across
>       multiple records, configuration parameters, or other data; and passes
>       a sequence of transformed records on to an Exporting Process, 
> IPFIX File
>       Writer, or another Intermediate Process within a Mediator. This 
> document
>       describes specific Intermediate Processes below used in the 
> elaboration
>       of the problem statement; however, this is not an exhaustive list.
> 
>    Intermediate Aggregation Process
> 
>      An Intermediate Aggregation Process is an Intermediate Process that
>      aggregates records based upon a set of Flow Keys, or functions
>      applied to fields from the record (e.g. binning, subnet aggregation).
> 
>    Intermediate Correlation Process
> 
>       An Intermediate Correlation Process is an Intermediate Process
>       which adds information to records noting correlations among them,
>       or generates new records with correlated data from multiple
>       records (e.g. the production of bidirectional flow records from
>       unidirectional flow records).
> 
>    Intermediate Selection Process
> 
>       An Intermediate Selection Process is an Intermediate Process that
>       selects records from a sequence based upon criteria evaluated
>       record values, and passes only those records that match the
>       criteria (e.g. filtering only records from a given network to a
>       given Collector).
> 
>    Intermediate Anonymisation Process
> 
>       An Intermediate Anonymisation Process is an Intermediate Process
>       that transforms records in order to anonymise them, to protect the
>       identity of the entities described by the records (e.g. by
>       applying prefix-preserving pseudonymisation of IP addresses).
> 
>    IPFIX Mediator
> 
>       An IPFIX Mediator is an IPFIX Device that provides mediation
>       capabilities by receiving records from some data source, hosting
>       zero or more Intermediate Processes to transform those records,
>       and exporting those records in IPFIX Messages via an Exporting
>       Process.  In the common case, a Mediator receives records from a
>       Collecting Process, but could also receive records from data
>       sources not encoded using IPFIX, e.g., in the case of NetFlow V9
>       protocol translation.  Specific types of Mediators are defined
>       below; some of these reference original mediation capabilities
>       defined in earlier IPFIX documens before the elaboration of the
>       Mediator problem statement.
> 
>    Original Exporter
> 
>       Original Exporter is an IPFIX Device that hosts the Observation
>       Points where the metered IP packets are observed.
> 
>    IPFIX Proxy
> 
>       An IPFIX Proxy is an IPFIX Mediator that relays incoming IPFIX
>       Messages or messages in other protocols to one or more Collectors.
>       It can provide transport protocol mediation and re-encoding.
> 
>    IPFIX Concentrator
> 
>       An IPFIX Concentrator is an IPFIX Mediator that recieves data from
>       one or more Exporters and sends them to a single Collector,
>       optionally transforming the records using zero or more
>       Intermediate Processes on the way.
> 
>    IPFIX Distributor
> 
>       An IPFIX Distributor is an IPFIX Mediator that recieves data from
>       one or more Exporters and sends them to one or more Collectors,
>       deciding which Collector(s) to send each record to based upon the
>       decision of an Intermediate Selection Process.
> 
>    IPFIX Masquerading Proxy
> 
>       An IPFIX Masquerading Proxy is an IPFIX Mediator that recieves
>       data from one or more Exporters and sends them to a single
>       Collector, using one or more Intermediate Aggregation Functions
>       and Intermediate Anonymisation Functions to screen out parts of
>       records according to configured policies, in order to protect the
>       privacy of the network's end users or senstive data of the
>       exporting organization.
> 
> 
> I think that we agree that the same terminology sections in both the 
> problem statement and framework.
> Personally, I'm not sure (yet) if we need the Intermediate 
> Aggregation/Correlation/Selection/Anonynisation Process.
> However, if we do, my proposal is to omit this list from the problem 
> statement, and to simply put them in the framework.
> 
> Please let us know, so that we can move forward with the drafts, both 
> the problem statements and the framework.
> 
> Regards, Benoit.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> _______________________________________________
> IPFIX mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ipfix

--- 
Atsushi KOBAYASHI  <[email-masked]>
NTT Information Sharing Platform Lab.
tel:+81-(0)422-59-3978 fax:+81-(0)422-59-5637

---

From: Brian Trammell <[email-masked]>
Date: Mon, 6 Jul 2009 18:40:22 +0200

hi Kobayashi-san,

see inline, as per the protocol. ;)

>      functions within one or more Intermediate Processes to a stream  

Almost. Mediation may actually consist of transport mediation only,  
so, zero intermediate processes. "Mediation" as a defined term (if we  
even need it) probably doesn't need to talk about processes anyway, so  
how about:

      IPFIX Mediation is the manipulation and conversion of
      records or IPFIX Messages for subsequent export using IPFIX,
      by applying mediation functions to a stream of records, and/or
      applying transport based mediation functions to IPFIX Messages.

Regards,

Brian

---

From: Gerhard Muenz <[email-masked]>
Date: Mon, 06 Jul 2009 21:07:32 +0200

Atsushi,

What is the motivation for this change?
Is it the use case of translating NetFlow into IPFIX?

I assume that in most of the cases this translation cannot be done by
just copying Records from a NetFlow Packet into an IPFIX Message.
Typically, some small changes to the Template and Data Records are
required (e.g., conversion of IE IDs), which would be done by an
Intermediate Process. You can call it Intermediate Conversion Process.

Having an Intermediate Process, it fits into the current definition of
IPFIX Mediation :)

Regards,
Gerhard

-- 
Dipl.-Ing. Gerhard Münz
Chair for Network Architectures and Services (I8)
Department of Informatics
Technische Universität München
Boltzmannstr. 3, 85748 Garching bei München, Germany
Phone:  +49 89 289-18008       Fax: +49 89 289-18033
E-mail: [email-masked]    WWW: http://www.net.in.tum.de/~muenz

---

From: Atsushi Kobayashi <[email-masked]>
Date: Tue, 07 Jul 2009 07:44:52 +0900

Hi Brian,

On Mon, 6 Jul 2009 18:40:22 +0200

>       applying transport based mediation functions to IPFIX Messages.

Fine. I agree.

Regards,
Atsushi

---

---

From: Atsushi Kobayashi <[email-masked]>
Date: Tue, 07 Jul 2009 07:45:47 +0900

Hi Gerhard,

> Having an Intermediate Process, it fits into the current definition of

I would like to discuss your points.
We can choose the two ways:

1) Transport mediation does not need Intermediate Process, content
mediation needs one or more Intermediate Process.

We can consider the following IPFIX-related Devices in RFC3917 as IPFIX
Mediation.

       +---+     +---+     +---+
       | E-+->   | E-+->   | E-+------------->---+
       | | |     | | |     | | | +---+         +-+-----+
       +-+-+     | M |     | M | | E-+------->-+-C-M-E-+->
         |       | | |     | | | | | | +---+   +-+-----+
       +-+-+     +-+-+     | O | | M | | E-+->---+
       | | |       |       +---+ | | | | | |
       | M |     +-+-+           | O | | M |
       | | |     | | |           +---+ | | |           +-----+
       | O |     | O |                 | O |        ->-+-C-E-+->
       +---+     +---+                 +---+           +-----+

      Protocol   Remote             Concentrator        Proxy
      Converter  Observation

                   Figure 2: IPFIX-related Devices

In case of conversion NetFlow to IPFIX, the protocol converter hosts a
Exporting process only.

2) All of IPFIX Mediation has one or more Intermediate Process whether
transport mediation or content mediation.

In case of protocol converter, IPFIX Mediation has Intermediate
Conversion Process as you suggested. And in case of IPFIX Proxy, we have
to consider some Intermediate Foo Process. Although the definition is
not consistent with RFC3917, it is not important in it.

Gerhard, which one do you prefer? 

Actually, I thought same thing that the protocol converter needs Intermediate
Process converting IE and/or creating subsidiary information (e.g. Flow
Key map and sampling parameter) in form of Data Records. But, it can
done in Exporting Process forming IPFIX Message, too.

While reading twice in mailing list, I assumed that we will adopt case
1 through long discussion. But, I don't adhere case 1, I would like to
get the consensus to progress to next stage.

Regards,
Atsushi

On Mon, 06 Jul 2009 21:07:32 +0200

---

From: Gerhard Muenz <[email-masked]>
Date: Tue, 07 Jul 2009 12:22:21 +0200

Hi Atsushi,

Current consensus is:

   IPFIX Mediation

   IPFIX Mediator

You and me agree that this fits for conversion of NetFlow to IPFIX.
So, where is the problem?

You can forget about the details in the figure of RFC3917. If you
believe in them, there would not be any Intermediate Process at all,
just Metering Processes!

> done in Exporting Process forming IPFIX Message, too.

The Exporting Process forms IPFIX Messages, but would not convert any
Templates or Records.

> get the consensus to progress to next stage.

As far as I see, current consensus are the definitions above. If you
propose a change, I do not see that we have already reached consensus.

BTW, I assume that an IP implementing a conversion function is needed
anyway. For example, if you want to aggregate records from different
NetFlow Versions, I assume that you first convert all of them to IPFIX
Data Records, such that the Intermediate Aggregation Process only has to
process IPFIX records. At least, this is how we will implement it.

Regards,
Gerhard

---

From: Atsushi Kobayashi <[email-masked]>
Date: Wed, 08 Jul 2009 08:14:34 +0900

Hi Gerhard,

On Tue, 07 Jul 2009 12:22:21 +0200

> So, where is the problem?

IPFIX Mediation covers the manipulation of records, and needs at least
single Intermediate Process. On the other hand, IPFIX Mediator doesnot
need Intermediate Process while manipulation of IPFIX Messages.

In the definition, the protocol conversion from NetFlow to IPFIX can
be executed in IPFIX Mediator, but it does not seem IPFIX Mediation, and
also changing transport protocol carrying IPFIX Messages (e.g. from UDP
to SCTP) can be executed in IPFIX Mediator, but it does not seem IPFIX
Mediation.

Because RFC3917 already described that the above functions can be done by
Exporting Process and/or Collecting Process only, such as protocol
converter, and proxy. These functions do not seem to need Intermediate
Process.

Or do you mean that the above functions need another Intermediate
Process? One is Intermediate Conversion Process? Which Intermediate
Process do change transport protocol?

Could you please elaborate your idea?

BTW, Brian proposed the term IPFIX Mediation, as follows. Where is
the problem in your idea?

    IPFIX Mediation

I think the current discussion is the level of framework rather than
problem statement. The above term is fine for us regardless of the
result of the discussion.

Regards,
Atsushi

---

From: Gerhard Muenz <[email-masked]>
Date: Wed, 08 Jul 2009 14:12:44 +0200

Hi Atsushi,

Atsushi Kobayashi wrote:
> be executed in IPFIX Mediator, but it does not seem IPFIX Mediation, 

IMO, this is IPFIX Mediation because an Intermediate Process converts
NetFlow records into IPFIX records.

> and

Yes, in this corner case, you do not really need an Intermediate
Process. However, you could also say that you have a transparent
Intermediate Process which does not do anything to the stream of records.

> Could you please elaborate your idea?

Current terminology is:

IPFIX Mediation:
	...one or more Intermediate Processes...
IPFIX Mediator:
	...zero or more Intermediate Processes...

Benoit chose the different numbers because IPFIX Proxy without
Intermediate Process should be covered by IPFIX Mediator. That's ok with me.

If you want to equalize the specified numbers of IPs, you can either
change the definition of IPFIX Mediation (as you propose):

IPFIX Mediation:
	... *zero* or more Intermediate Processes...
	(or omit mentioning IPs)
IPFIX Mediator:
	...zero or more Intermediate Processes...

Or you change the definition of IPFIX Mediator:

IPFIX Mediation:
	...one or more Intermediate Processes...
IPFIX Mediator:
	... *one* or more Intermediate Processes...

If you want to change the definitions, I prefer the second option over
the first one. However, I would not object to the first one if you think
that this makes more sense.

>       applying transport based mediation functions to IPFIX Messages.

The old definition was simple and clear. The new definition causes
confusion because it introduces two types of mediation functions:
- "mediation functions"
- "transport based mediation functions"
You would have to explain the difference and probably introduce two more
terms like "Record Mediation Function" and "Transport Based Mediation
Function".

Regards,
Gerhard

---

From: Atsushi Kobayashi <[email-masked]>
Date: Fri, 10 Jul 2009 14:54:55 +0900

Hi Gerhard,

I will change thread. 
It is consensus issue. I will ask IPFIX members.

Regards,
Atsushi

On Wed, 08 Jul 2009 14:12:44 +0200

---