Subject: [IPFIX] draft-ietf-ipfix-mediators-problem-statement-03 -> terminology

From: Atsushi Kobayashi <[email-masked]>
Date: Thu, 18 Jun 2009 17:52:33 +0900

Hi Benoit, Gerhard, Brian, and all,

In terminology section, there are main two discussion points.

1) the problem statement includes the more specific terms (IPFIX
Concentrator, Proxy, Masq.Proxy, Distributor), or not?

Brian, Gerhalz points out these terms seems restrict the work at framework,
it is better the problem statement uses the more general terms. Benoit
say it does not prefer that the discussion for the terminologies are
postponed.

I agree the Benoit. Although I can use the general terms in problem
statement, using the terms enhances the images of the IPFIX mediation
examples. 

So, I would like to keep these terms in problem statement.

2) the more specific terms (IPFIX Concentrator, Proxy, Masq.Proxy,
Distributor) indicate the type of device, or a set of the functions.

Gerhalz points out these terms are already perceived as type of
devices, for example IPFIX Concentrator in RFC3917.
Same questions arose from some NTT co-workers. Benoit say that the
"device" implies a physical entity and restricts the future functions.

I am worried that the terms become more complex meaning after thinking
the future functions. I think that even if it is "device", it does not seem
restrict the location in which the future functions applied.

Because we can say that IPFIX Mediator does not implicitly require a
separate physical entity.

   IPFIX Mediator

      An IPFIX Mediator is an IPFIX Device that implements one or more
      IPFIX Mediation capabilities.  Examples are devices such as
      routers, switches, network management systems (NMS), etc.

Certainly, a IPFIX Mediator can have a set of the functions hosted in
both IPFIX Proxy and IPFIX Concentrator. In that case, the Mediator
can become IPFIX Proxy on one situation, and it can also become IPFIX
Concentrator on other situations.

Thus, I proposed here.

   IPFIX Proxy

      An IPFIX Proxy is a type of IPFIX Mediator that relays incoming
      Transport Sessions to one or multiple Collectors.

Regards,
Atsushi

--- 
Atsushi KOBAYASHI  <[email-masked]>
NTT Information Sharing Platform Lab.
tel:+81-(0)422-59-3978 fax:+81-(0)422-59-5637

---

From: Gerhard Muenz <[email-masked]>
Date: Fri, 19 Jun 2009 10:20:22 +0200

Hi Atsushi,

>       Transport Sessions to one or multiple Collectors.

This approach is ok for me. So, IPFIX Concentrator|Proxy|... describe
specific flavors of IPFIX Mediator devices.

Gerhard

---

From: Brian Trammell <[email-masked]>
Date: Fri, 19 Jun 2009 10:56:32 +0200

> So, I would like to keep these terms in problem statement.

Agreed...

> restrict the location in which the future functions applied.

Device, in English usage, clearly means "box"; and further, from 5470,  
requires export:

An IPFIX Device hosts at least one Exporting Process. It may host  
further Exporting Processes and arbitrary numbers of Observation  
Points and Metering Processes.

I'm okay with a Mediator being a Device. I think, though, that when  
composing complex mediation functions, we should compose them of  
functions within a single Mediator, and not, for instance, compose  
things out of sets of Concentrators and Proxies... We shouldn't  
confuse the device with what it does, and we should be talking in  
terms of actions now.

> Concentrator on other situations.

Of course. The point is that a Mediator is a generalization of all of  
the other previously envisioned IPFIX "transformation" devices/ 
processes.

>      Transport Sessions to one or multiple Collectors.

Okay...

I propose the following unified terminology section, following the  
thread to date:

IPFIX Mediation describes the manipulation and conversion of records  
exported using IPFIX, by applying mediation functions within one or  
more Intermediate Processes in an IPFIX Mediator. The following terms  
are used in this document to describe the architectural entities used  
by IPFIX Mediation.

Intemediate Process

An Intermediate Process takes a sequence of records from a Collecting  
Process, IPFIX File Reader, or another Intermediate Process within a  
Mediator; performs some transformation on these records based upon the  
content of the records themselves, state kept across multiple records,  
configuration parameters, or other data; and passes a sequence of  
transformed records on to an Exporting Process, IPFIX File Writer, or  
another Intermediate Process within a Mediator. This document  
describes specific Intermediate Processes below used in the  
elaboration of the problem statement; however, this is not an  
exhaustive list.

Intermediate Aggregation Process

An Intermediate Aggregation Process is an Intermediate Process that  
aggregates records based upon a new set of Flow Key fields, or  
functions applied to Flow Key fields (e.g. binning, subnet aggregation).

Intermediate Correlation Process

An Intermediate Correlation Process is an Intermediate Process which  
adds information to records noting correlations among them, or  
generates new records with correlated data from multiple records (e.g.  
the production of bidirectional flow records from unidirectional flow  
records).

Intermediate Selection Process

An Intermediate Selection Process is an Intermediate Process that  
selects records from a sequence based upon criteria evaluated record  
values, and passes only those records that match the criteria (e.g.  
filtering only records from a given network to a given Collector).

Intermediate Anonymisation Process

An Intermediate Anonymisation Process is an Intermediate Process that  
transforms records in order to anonymise them, to protect the identity  
of the entities described by the records (e.g. by applying prefix- 
preserving pseudonymisation of IP addresses).

IPFIX Mediator

An IPFIX Mediator is an IPFIX Device that provides mediation  
capabilities by receiving records from some data source, hosting zero  
or more Intermediate Processes to transform those records, and  
exporting those records in IPFIX Messages via an Exporting Process. In  
the common case, a Mediator receives records from a Collecting  
Process, but could also receive records from data sources not encoded  
using IPFIX, e.g., in the case of NetFlow V9 protocol translation.  
Specific types of Mediators are defined below; some of these reference  
original mediation capabilities defined in earlier IPFIX documens  
before the elaboration of the Mediator problem statement.

IPFIX Proxy

An IPFIX Proxy is an IPFIX Mediator that relays incoming
IPFIX Messages or messages in other protocols to one or more  
Collectors. It can provide transport protocol mediation and re-encoding.

IPFIX Concentrator

An IPFIX Concentrator is an IPFIX Mediator that recieves data from one  
or more Exporters and sends them to a single Collector, optionally  
transforming the records using zero or more Intermediate Processes on  
the way.

IPFIX Distributor

An IPFIX Distributor is an IPFIX Mediator that recieves data from one  
or more Exporters and sends them to one or more Collectors, deciding  
which Collector(s) to send each record to based upon the decision of  
an Intermediate Selection Process.

IPFIX Masquerading Proxy

An IPFIX Masquerading Proxy is an IPFIX Mediator that recieves data  
from one or more Exporters and sends them to a single Collector, using  
one or more Intermediate Aggregation Functions and Intermediate  
Anonymisation Functions to screen out parts of records according to  
configured policies, in order to protect the privacy of the network's  
end users or senstive data of the exporting organization.

Best regards,

Brian

---

From: Christoph Sommer <[email-masked]>
Date: Fri, 19 Jun 2009 12:05:54 +0200

Brian, all,

Brian Trammell wrote:
> exported using IPFIX, by applying mediation functions within one or more
> Intermediate Processes in an IPFIX Mediator. The following terms are
> used in this document to describe the architectural entities used by
> IPFIX Mediation.
> 
> Intemediate Process
> 
> another Intermediate Process within a Mediator. This document describes
> specific Intermediate Processes below used in the elaboration of the
> problem statement; however, this is not an exhaustive list.
> 
> aggregates records based upon a new set of Flow Key fields, or functions
> applied to Flow Key fields (e.g. binning, subnet aggregation).
> 
> adds information to records noting correlations among them, or generates
> new records with correlated data from multiple records (e.g. the
> production of bidirectional flow records from unidirectional flow records).
> 
> of the entities described by the records (e.g. by applying
> prefix-preserving pseudonymisation of IP addresses).
> 
> IPFIX Mediator
> 
> capabilities by receiving records from some data source, hosting zero or
> more Intermediate Processes to transform those records, and exporting
> those records in IPFIX Messages via an Exporting Process. In the common
> case, a Mediator receives records from a Collecting Process, but could
> also receive records from data sources not encoded using IPFIX, e.g., in
> the case of NetFlow V9 protocol translation. Specific types of Mediators
> are defined below; some of these reference original mediation
> capabilities defined in earlier IPFIX documens before the elaboration of
> the Mediator problem statement.
> 
> IPFIX Proxy
> 
> IPFIX Messages or messages in other protocols to one or more Collectors.
> It can provide transport protocol mediation and re-encoding.
> 
> IPFIX Concentrator
> 
> An IPFIX Distributor is an IPFIX Mediator that recieves data from one or
> more Exporters and sends them to one or more Collectors, deciding which
> Collector(s) to send each record to based upon the decision of an
> Intermediate Selection Process.
> 
> IPFIX Masquerading Proxy
> 
> An IPFIX Masquerading Proxy is an IPFIX Mediator that recieves data from
> one or more Exporters and sends them to a single Collector, using one or
> more Intermediate Aggregation Functions and Intermediate Anonymisation
> Functions to screen out parts of records according to configured
> policies, in order to protect the privacy of the network's end users or
> senstive data of the exporting organization.

I like this way of defining the processes and devices mentioned in the
draft!

One minor comment: Maybe the draft could also stress that (like the list
of processes) the list of devices is non-exhaustive.

Cheers,

  Christoph

-- 
Christoph Sommer
Computer Networks and Communication Systems
University of Erlangen-Nuremberg, Germany
Phone: +49 9131 85-27993 / Fax: +49 9131 85-27409
http://www7.informatik.uni-erlangen.de/~sommer/

---

From: Gerhard Muenz <[email-masked]>
Date: Fri, 19 Jun 2009 22:10:28 +0200

Brian,

I'm not sure if we need all this information in the terminology section.
Maybe, there is a better place them, either in another section of the
draft or in the framework draft.

BTW, there are several problems with your definitions. See inline.

Regards,
Gerhard

> exported using IPFIX, by applying mediation functions within one or more

"exported using IPFIX" should be removed, because they could have been
exported using Netflow.

> Process, IPFIX File Reader, or another Intermediate Process within a

The Metering Process is missing as a source of records. I wonder if we
have to specify the sources at all.

> another Intermediate Process within a Mediator. This document describes

Again, do we have to specify the sinks here?

> applied to Flow Key fields (e.g. binning, subnet aggregation).

It is also possible to aggregate PSAMP Packet Reports that do not have
Flow Keys! So, we cannot assume that the original records have flow key
fields.

> case, a Mediator receives records from a Collecting Process, but could

What about records from a Metering Process?

> _______________________________________________
> IPFIX mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ipfix

-- 
Dipl.-Ing. Gerhard Münz
Chair for Network Architectures and Services (I8)
Department of Informatics
Technische Universität München
Boltzmannstr. 3, 85748 Garching bei München, Germany
Phone:  +49 89 289-18008       Fax: +49 89 289-18033
E-mail: [email-masked]    WWW: http://www.net.in.tum.de/~muenz

---

From: Brian Trammell <[email-masked]>
Date: Mon, 22 Jun 2009 12:56:50 +0200

Gerhard,

Inline.

> draft or in the framework draft.

This is a different argument, which we seem to be converging in the  
direction of defining everything consistently twice, and allowing the  
problem statement to use the "real" terminology in its example. There  
are of course valid reasons not to do this, as well. I'm not sold  
either way, I'm just trying to follow consensus; if this is what we  
want in the terminology, then this is what it should look like.

I'm not at all certain that the Intermediate FOO Process definitions  
are necessary, but they do underscore that _devices_ we seem to have  
already defined are really just hosts for processes, and that  
processes are the important part.

> BTW, there are several problems with your definitions. See inline.

Thanks for the comments; comments thereon below. :)

>> more

Negative, although I agree the phrasing is confusing here... What I  
meant was, the _output_ of mediation is necessarily and by definition  
IPFIX. In full, the correction should read as follows:

IPFIX Mediation describes the manipulation, conversion, and export of  
records via IPFIX, by applying mediation functions within one or more  
Intermediate Processes to a stream of records in an IPFIX Mediator.  
The following terms are used in this document to describe the  
architectural entities used by IPFIX Mediation.

> have to specify the sources at all.

Point.

>> describes

I think so; I've tried constructing this definition without source/ 
sink specifications and it sounds so general that it doesn't actually  
say anything at all.

>> problem statement; however, this is not an exhaustive list.

Corrected:

Intermediate Process

An Intermediate Process takes a sequence of records from a Collecting
Process, Metering Process, IPFIX File Reader, or another Intermediate  
Process within a Mediator; performs some transformation on these  
records based upon the
content of the records themselves, state kept across multiple records,
configuration parameters, or other data; and passes a sequence of
transformed records on to an Exporting Process, IPFIX File Writer, or
another Intermediate Process within a Mediator. This document describes
specific Intermediate Processes below used in the elaboration of the
problem statement; however, this is not an exhaustive list.

>> functions

Point, however, aggregation by necessity requires some "keys" on which  
to aggregate. So we can use lowercase "keys" (which I acknowledge  
risks confusing people... are keys Flow Keys?, but it's better than  
nothing.)

An Intermediate Aggregation Process is an Intermediate Process that
aggregates records based upon a set of key fields, or functions
applied to fields from the record (e.g. binning, subnet aggregation).

>> generates

(see above re: maybe we don't need these here.)

> What about records from a Metering Process?

Does a Mediator actually receive records directly from an MP, or is an  
MP->IP->CP box something different? I don't have an opinion here  
(well, I do, but it's a mild one)... it just seems like something that  
hasn't been thought out yet, and should be.

>> Collectors.
>> from

---

From: Gerhard Muenz <[email-masked]>
Date: Mon, 22 Jun 2009 13:52:15 +0200

Brian,

Comments inline.

Brian Trammell wrote:
> are of course valid reasons not to do this, as well. I'm not sold either
> way, I'm just trying to follow consensus; if this is what we want in the
> terminology, then this is what it should look like.
> 
> I'm not at all certain that the Intermediate FOO Process definitions are
> necessary, but they do underscore that _devices_ we seem to have already
> defined are really just hosts for processes, and that processes are the
> important part.
> 
> Negative, although I agree the phrasing is confusing here... 

Indeed. I (mis)understood the phrase in a way that other people could to
as well ;)

> What I

Agreed.

> Intermediate Processes to a stream of records in an IPFIX Mediator. The
> following terms are used in this document to describe the architectural
> entities used by IPFIX Mediation.

I do not think that IPFIX Mediation "describes" the export of records.
I would change your original text in something like:

"IPFIX Mediation describes the manipulation and conversion of records
for subsequent export using IPFIX, by applying mediation functions ..."

> I think so; I've tried constructing this definition without source/sink
> specifications and it sounds so general that it doesn't actually say
> anything at all.
> 
> Process within a Mediator; performs some transformation on these records
> based upon the

Nice.

> to aggregate. So we can use lowercase "keys" (which I acknowledge risks
> confusing people... are keys Flow Keys?, but it's better than nothing.)
> 

In the first paraphrase of the original text, there is no problem with
"Flow Keys" as they may come from the configuration of the IP:

An Intermediate Aggregation Process is an Intermediate Process that
aggregates records based upon a set of Flow Keys, or functions
applied to fields from the record (e.g. binning, subnet aggregation).

> MP->IP->CP box something different? I don't have an opinion here (well,

Do you mean MP->IP->EP ?

> I do, but it's a mild one)... it just seems like something that hasn't
> been thought out yet, and should be.

Hm, I think we have already thought about it.
According to the definition, an IPFIX Mediator includes at least one IP
and one EP (because it is an IPFIX Device). So, MP->IP->EP is an IPFIX
Mediator by definition.

Gerhard

---

From: Atsushi Kobayashi <[email-masked]>
Date: Tue, 23 Jun 2009 02:39:12 +0900

Hi Gerhard, Brian,

On Mon, 22 Jun 2009 13:52:15 +0200

> Indeed. I (mis)understood the phrase in a way that other people could to
Me, too. :)

> > entities used by IPFIX Mediation.

Brian, let me know your definition in detail.
Your definition looks like IPFIX Mediation is only implemented in IPFIX
Mediator. IPFIX Mediation, that is Intermediate Processes, can be hosted
in any device. 

And IPFIX Mediation definition refers to IPFIX Mediator, and IPFIX
Mediator definition refers to mediation function. It seems a little bit
strange for me.

Or can you remove "in an IPFIX Mediator"? 

> for subsequent export using IPFIX, by applying mediation functions ..."

Gerhard, the above definition say that finally IPFIX exporting after
executing IPFIX Mediation is implemented. Is it right? If Intermediate
Process is hosted in NMS or Collector, this device would not export
IPFIX.

My original definition is as follows.

   IPFIX Mediation

      IPFIX Mediation is a generic function that covers the manipulation
      of IPFIX Flow Records, PSAMP Packet Reports, or entire IPFIX
      Messages, or their transmission.  The IPFIX Mediation offers one
      or multiple of the following capabilities:

I would like to edit as follows.

"IPFIX Mediation describes the manipulation and conversion of records or IPFIX
Message, by applying..."

> > configuration parameters, or other data; and passes a sequence of

Configuration parameter is good idea.

> applied to fields from the record (e.g. binning, subnet aggregation).

Um.., problem statement and framework proposes "spatial composition". In
that case, Exporter IP address, or Observation Domain ID, or some
identifier based on observation information becomes key fields, thus,
original Flow Records does not includes Flow Key fields. I am not sure
the definition can cover the spatial composition.

> and one EP (because it is an IPFIX Device). So, MP->IP->EP is an IPFIX

I agreed.

Regards,
Atsushi

---

From: Gerhard Muenz <[email-masked]>
Date: Mon, 22 Jun 2009 20:08:58 +0200

Hi Atsushi,

> Process is hosted in NMS or Collector, this device would not export

I'm not sure if we should consider IPFIX Mediation hosted in NMS aka
Collectors that do not reexport the data afterwards.

Of course, NMS/Collectors typically analyze the received data, and the
analysis functions can be the same as in an IP. However, we do not need
to standardize what happens here because the data is not exported any more.

I do not think that we should standardize how IPFIX data is treated
beyond the Collector.

> "IPFIX Mediation describes the manipulation and conversion of records or IPFIX

If this is consensus, I would not oppose it.

However, if you look at this:

IPFIX messages--> CP --records--> IP --records--> EP --IPFIX messages-->

then, the IP does not see/treat any IPFIX messages, just records.

> the definition can cover the spatial composition.

I do not understand where the problem should be.

In my opinion, the output of the aggregation process is a stream of Flow
Records. These Flow Records have Flow Keys. These Flow Keys do not have
to be the same as those potentially included in the records that entered
the aggregation process.

My definition does not mention, where the Flow Keys come from. I would
suggest that they typically come from configuration.

Regards,
Gerhard

---

From: Atsushi Kobayashi <[email-masked]>
Date: Tue, 23 Jun 2009 03:41:15 +0900

Brian, Gerhard, Benoit and all,

Let me know your opinion.

On Mon, 22 Jun 2009 12:56:50 +0200

> want in the terminology, then this is what it should look like.
Thank you, Brian. Yes, I'm just trying to follow consensus.

> processes are the important part.

First of all, IMO, there is a better place that IPFIX XXXtor is introduced
in problem statement, because first reader feels easily the image of
implementation analysis. And the instances of Mediation functions are
shown as Intermedeate Process in framework draft. Framework draft
describes the components, or some functions in detail.
Thus, I thought that IPFIX XXXtors are defined in Problem Statement, 
IPFIX Intermedeate FOO Processes are defined in Framework. 
But, regarding Brian's proposal that IPFIX XXXtor refers to Intermedeate
FOO Process, I am not sure where is a better place for them yet. It is
hard choice.

We can choose three types.

a) problem statement and framework describes whole terminologies, twice.
In case of Brian's terminologies, we describes what is IPFIX Mediation,
Mediator, and Intermediate Process in problem statement. It seems cover
the framework description.
  
b) problem statement does not define terminologies, and then it
elaborates about applicable examples without capital letter.

c) problem statement defines some terms, framework defines some terms.
I am not sure. One examples are that Problem statement defines IPFIX
Mediation and Mediator. Framework defines others.

What do you think about that?

Regards,
Atsushi

---

From: Atsushi Kobayashi <[email-masked]>
Date: Tue, 23 Jun 2009 09:51:29 +0900

Hi Gerhard, Benoit, Falko, Emile, Christoph, and Nishida-san,

On Mon, 22 Jun 2009 20:08:58 +0200

> I do not think that we should standardize how IPFIX data is treated

I wonder we try to standardize IPFIX aggregation and anonymization. These
functions can be applied to NMS/Collector as well as IPFIX Mediators.
Maybe, one example of IPFIX aggregation and anonymization would be Mediator. 
It seems same thing for me.

Even if the standardization of IPFIX Mediation covers only IPFIX
Mediator, the IPFIX Mediation function can be applied to any devices,
thus it should not restrict in its terminologies.

I and co-authors decided the IPFIX Mediator terminologies as follows.

   IPFIX Mediator

      An IPFIX Mediator is an IPFIX Device that implements one or more
      IPFIX Mediation capabilities.  Examples are devices such as
      routers, switches, network management systems (NMS), etc.
                         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

But, I don't have strong opinion about it. I will follow the consensus
including co-authors.

> then, the IP does not see/treat any IPFIX messages, just records.

In simple case, transport protocol is changed from UDP to SCTP. It is
one-to-one transmission of incoming/outgoing IPFIX session.
In that case, IP does not need to manipulate the records.

> suggest that they typically come from configuration.

OK. I understand.

Regards,
Atsushi

> Dipl.-Ing. Gerhard M〓z
> Technische Universit舩 M〓chen
> Boltzmannstr. 3, 85748 Garching bei M〓chen, Germany

---

---

From: Falko Dressler <[email-masked]>
Date: Tue, 23 Jun 2009 08:23:14 +0200

Kobayashi-san, all,

On 23.06.2009 02:51, Atsushi Kobayashi wrote:
...
>>> Mediator. IPFIX Mediation, that is Intermediate Processes, can be hosted
...
>        routers, switches, network management systems (NMS), etc.

Well, back to the original meaning of IPFIX Mediation, I still prefer
the "manipulation" of data (or more precisely, received IPFIX records).
Thus, functionality on the receiving NMS system might be described by
means of IPFIX mediation, but, of course, there is no need to follow
standardized rules here.

As for the suggested definition of an IPFIX Mediator, I would recommend
to drop the sentence "Examples are ... NMS..." This might be misleading.

Best regards,
Falko

-- 
Dr. Falko Dressler
Computer Networks and Communication Systems
University of Erlangen, Germany
Phone: +49 9131 85-27914 / Fax: +49 9131 85-27409
mailto:[email-masked]
http://www7.informatik.uni-erlangen.de/~dressler/

---

From: Gerhard Muenz <[email-masked]>
Date: Tue, 23 Jun 2009 13:07:22 +0200

Atsushi,

>       routers, switches, network management systems (NMS), etc.

RFC5101 defines an IPFIX Device "an IPFIX Device hosts at least one
Exporting Process." If you define an IPFIX Mediator is an IPFIX Device,
there always is an Exporting Process in the same device.

If you understand NMS as a Collector which does not reexport the data,
then this is not an IPFIX Mediator by definition.

> In that case, IP does not need to manipulate the records.

In my opinion, a change of the transport protocol below IPFIX does not
require any IP:

 CP --> EP

This setup is shown in RFC3917, Figure 2.

Gerhard

>> Dipl.-Ing. Gerhard M?z
>> Technische Universit? M?chen
>> Boltzmannstr. 3, 85748 Garching bei M?chen, Germany