Subject: [IPFIX] transport mediation in IPFIX Mediation?

From: Atsushi Kobayashi <[email-masked]>
Date: Fri, 10 Jul 2009 14:53:39 +0900

Dear all,

I change the thread. I would like to ask IPFIX members.

I and Gerhald are discussing terms IPFIX Mediation/IPFIX Mediaor.
These current terms are:

   IPFIX Mediation

      IPFIX Mediation describes the manipulation and conversion of
      records for subsequent export using IPFIX, by applying mediation
      functions within one or more Intermediate Processes to a stream of
      records in an IPFIX Mediator.

   IPFIX Mediator

      An IPFIX Mediator is an IPFIX Device that provides mediation
      capabilities by receiving records from some data source, hosting
      zero or more Intermediate Processes to transform those records,
      and exporting those records in IPFIX Messages via an Exporting
      Process.  In the common case, a Mediator receives records from a
      Collecting Process, but could also receive records from data
      sources not encoded using IPFIX, e.g., in the case of NetFlow V9
      protocol translation.

Question:

Does IPFIX Mediation or an IPFIX Mediator cover the transport mediation,
e.g. from IPFIX-over-UDP to IPFIX-over-SCTP ?

According to above terms, we can choose in two ways:

a) An IPFIX Mediator covers the transport mediation, and IPFIX Mediation
does not, if an Intermediate Process does not cover the transport
mediation.

b) Both of an IPFIX Mediator and IPFIX Mediation cover the transport
mediation, if an Intermediate Process covers the transport mediation.

First of all, I believe the IPFIX Mediation covering transport mediation
is already consensus regardless of whether an Intermediate Process
covers transport mediation or not, because I presented the following
slide in IETF73rd and there is no objection so far.
https://www.ietf.org/proceedings/08nov/slides/ipfix-1.pdf

Thus, I agree the following new term.

   IPFIX Mediation

      IPFIX Mediation is the manipulation and conversion of
      records or IPFIX Messages for subsequent export using IPFIX,
      by applying mediation functions to a stream of records, and/or
      applying transport based mediation functions to IPFIX Messages.

Let me know your opinions. I will write down this feedback in new
version drafts according to consensus as far as possible. Its deadline
next monday is approaching.

BTW, I think that whether an Intermediate Process covers transport
mediation or not is the framework issue. Simply, even when relaying
IPFIX Messages, I think that we needs some function that handles
incoming session status and IPFIX Withdrawal Messages. Thus, it may need
Intermediate Process.

Regards,
Atsushi

--- 
Atsushi KOBAYASHI  <[email-masked]>
NTT Information Sharing Platform Lab.
tel:+81-(0)422-59-3978 fax:+81-(0)422-59-5637

---

From: Gerhard Muenz <[email-masked]>
Date: Fri, 10 Jul 2009 13:08:36 +0200

Dear Atsushi,

> next monday is approaching.

I still do not think that this is not a good definition. Especially, I
would object to the formulation "transport based mediation functions".
It is not clear what "transport based" is supposed to be.

What about the following:

   IPFIX Mediation

      IPFIX Mediation is the manipulation and conversion of
      records or IPFIX Messages for subsequent export using IPFIX,
      by applying mediation functions to a stream of records, and/or
      applying transport protocol mediation.

Note that this corresponds much better to what is written for IPFIX Proxy:

  IPFIX Proxy

     An IPFIX Proxy is an IPFIX Mediator that relays incoming IPFIX
     Messages or messages in other protocols to one or more Collectors.
     It can provide transport protocol mediation and re-encoding.
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Regards,
Gerhard

> _______________________________________________
> IPFIX mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ipfix

-- 
Dipl.-Ing. Gerhard Münz
Chair for Network Architectures and Services (I8)
Department of Informatics
Technische Universität München
Boltzmannstr. 3, 85748 Garching bei München, Germany
Phone:  +49 89 289-18008       Fax: +49 89 289-18033
E-mail: [email-masked]    WWW: http://www.net.in.tum.de/~muenz

---

From: Brian Trammell <[email-masked]>
Date: Fri, 10 Jul 2009 13:15:17 +0200

hi Gerhard, Atsushi, everyone,

We seem to be getting there...

IPFIX Mediation is, in my view, an activity that takes traffic _data_  
from somewhere (not traffic off the wire; that's an MP's job), _may_  
change its content if necessary for the application, and sends it  
somewhere else _via IPFIX_. A Mediator is something that mediates.  
First we make sure the definitions express this; they seem to. Then  
we can get down to details. Which I will. Inline. :)

> It is not clear what "transport based" is supposed to be.

Agreed, and I'd prefer not to define them, because they're only used  
for this definition.

>      It can provide transport protocol mediation and re-encoding.

Agreed... "transport protocol mediation" is better here.

Regards,

Brian

---

From: "Juergen Quittek" <[email-masked]>
Date: Fri, 10 Jul 2009 13:55:03 +0200

Hi all,

Writing as technical contributor, I am not too happy about including
transport in the mediation definition.

First, I think that mediation rather is a process that affects the
IPFIX records and not the transport. But I know that there are many
definitions for general mediation in use and not all are in line
with this. However, if using different transport requires modification
of IPFIX records, then we can talk about mediation, not because
transport is changed, but because IPFIX records are changed.

Second, using the word "transport" in the definition may be ambiguous.
For example, if there is a UDP-based tunnel on the the path of an IPFIX
export. Would then the tunnel endpoints be mediators?
Probably, we could exclude such cases by a more complicated statement,
but I would prefer to keep it simple: mediation is affecting the IPFIX
records and not how records are transmitted.

Third, how would we see an exporter offering different transport
capabilities at the same time, for example, export to collector 1
over SNMP and to collector 2 over TCP. Would it then be correct to
state that it contains a mediation function for putting its internal
representation of flow records onto different transport protocols?

Thanks,

    Juergen

---

From: Atsushi Kobayashi <[email-masked]>
Date: Mon, 13 Jul 2009 17:29:35 +0900

Hi Gerhard, Brian, Juergen, and all,

I recognize that you (Juergen, Brian, Gerhard) seem have a same opinion
that IPFIX Mediation doesn't include transport mediation.
I would like to clarify your point.

Ok, I agree the document does not use ambiguous word "transport".
Ok, I agree that IPFIX Mediation does not cover the transport mediation
relaying IPFIX Messages directly with not changing whole of them.

What about manipulating IPFIX Messages? 

In case of changing Observation Domain ID in IPFIX Messages, is it
covered in IPFIX Mediation? I think Yes.

Observation Domain ID and Export Time are subsidiary information of Data
Record. Maybe they seem be included in general word "record".
Thus, the manipulating IPFIX Messages is covered in IPFIX Mediator. Does
it make sense?

I think terms IPFIX Mediation/Mediator are as follows.

    IPFIX Mediation

        IPFIX Mediation describes the manipulation and conversion of
        records for subsequent export using IPFIX, by applying mediation
        functions to a stream of records.

In my opinion, IPFIX Mediation is executed in Original Exporter as well.
Thus, basically, IPFIX Mediation manipulates records. 

   IPFIX Mediator

      Collecting Process but could also receive records from data

The above term is kept in current one.
In my opinion, IPFIX Mediator is separate box. Thus, IPFIX Mediator can
manipulate records. and IPFIX Messages as well. It is reason for IPFIX
Mediator hosting zero Intermediate Process.

Regards,
Atsushi

On Fri, 10 Jul 2009 13:55:03 +0200

---

---

From: "Juergen Quittek" <[email-masked]>
Date: Mon, 13 Jul 2009 14:27:54 +0200

Dear Kobayashi-san,

> covered in IPFIX Mediation? I think Yes.

Agreed.

    Juergen

---

From: Nishida Haruhiko <[email-masked]>
Date: Tue, 14 Jul 2009 10:48:53 +0900

all,

I agree to exclude transport mediation from our scope. 
Juregen's comment on transport medation form UDP to TCP 
(or something else) is a good example. I have no intention to
call it a "Mediator".

One case I worried was a "Distributor", which receives (in the simplest 
case) one IPFIX stream and export it to a number of collectors, 
without any mediation on records. By definition, it is not a "Mediator" 
if we exclude "transport mediation".

As Atsushi pointed out that if the device changes "observation 
domain ID" or filters some records, it is a "Mediator". And I could 
implement filtering function "configurable", means I can configure 
"just pass through" filter, then it is not a "Mediator"...

I wonder whether there is a better solution, but I think I could
survive.  A device which could be configured as a "Mediator"
should follow Mediator standards anyway.

-- Haru Nishida

---

From: Gerhard Muenz <[email-masked]>
Date: Tue, 14 Jul 2009 12:15:00 +0200

Hi,

In the MIB and in the configuration data model, we have the
exportMemberType parameter. If an Exporting Process is configured with
multiple export destinations, this parameter determines if the data is
exported in parallel or in a load balancing fashion. In my opinion, this
kind of load balancing is performed statistically without considering
the message content.

As an IPFIX Distributor is supposed to make more sophisticated decision
regarding where to export a record, it is IPFIX Mediation requiring an
Intermediate Process. At least, that's how I would interpret it.

Gerhard

---

From: Brian Trammell <[email-masked]>
Date: Tue, 14 Jul 2009 12:24:16 +0200

Gerhard,

I agree fully in this use case... however, this implies that an  
Intermediate Process is not just a potentially stateful record shovel;  
it also receives control inputs from upstream processes (e.g., the  
identity of the exporter from which records were received), and passes  
control outputs to downstream processes (in the distributor case,  
which exporter to send to).

Cheers,

Brian

On Jul 14, 2009, at 12:15 PM, Gerhard Muenz wrote:

>>>>>> Note that this corresponds much better to what is written for  
>>>>>> IPFIX
>>>>>> Proxy:
>>>>>>
>>>>>> IPFIX Proxy
>>>>>>

---

From: Benoit Claise <[email-masked]>
Date: Fri, 24 Jul 2009 16:06:59 +0200

Kobayashi-san,
> relaying IPFIX Messages directly with not changing whole of them.
Does it imply that we should change the abstract?

Abstract

   Flow-based measurement is a popular method for various network
   monitoring usages.  The sharing of flow-based information for
   monitoring applications having different requirements raises some
   open issues in terms of measurement system scalability, flow-based
   measurement flexibility, and export reliability that IPFIX Mediation
   may help resolve.  _IPFIX Mediation covers two classes of mediation:
   content mediation for traffic data and transport mediation for
   transport protocols. _ This document describes the IPFIX Mediation
   applicability examples, along with some problems that network
   administrators have been facing.

Regards, Benoit.

---

From: Cristian Morariu <[email-masked]>
Date: Fri, 24 Jul 2009 13:06:16 +0200

Hi,

there is an editing error in 
draft-ietf-ipfix-mediators-problem-statement-04 in the IPFIX 
Concentrator definition:

   IPFIX Concentrator

      An IPFIX Distributor is an IPFIX Mediator that receives data from
               ^^^^^^^^^^^
      one or more Exporters and sends them to one or more Collectors,
      deciding which Collector(s) to send each record to based upon the
      decision of an Intermediate Process.

   IPFIX Distributor

      An IPFIX Distributor is an IPFIX Mediator that receives data from
      one or more Exporters and sends them to one or more Collectors,
      deciding which Collector(s) to send each record to based upon the
      decision of an Intermediate Process.

Version 3 of the draft reads fine that section.

Regards,
Cristian Morariu

---

From: Atsushi Kobayashi <[email-masked]>
Date: Sun, 26 Jul 2009 22:46:39 +0900

Dear Cristian,

oops, I will correct it. Thank you!

Regards,
Atsushi

On Fri, 24 Jul 2009 13:06:16 +0200

---

---

From: Juergen Quittek <[email-masked]>
Date: Mon, 27 Jul 2009 11:01:11 +0200

On 24.07.2009 at 16:06 Benoit Claise wrote:

>    may help resolve.  IPFIX Mediation covers two classes of mediation:
>    transport protocols.  This document describes the IPFIX Mediation
Yes.

     Juergen