Subject: Re: [IPFIX] NetFlow v9 to IPFIX conversion

From: Gerhard Muenz <[email-masked]>
Date: Thu, 02 Apr 2015 22:36:39 +0200

Hi Petr,

I do not understand the use case of the new PEN that you suggest.

It does not make sense to register a PEN for an ID space that is not 
centrally managed in a kind of registry because uniqueness of ID usage 
must be ensured. Usually, the owner of a PEN maintains such a registry.

Is there a registry that lists all vendor specific Netflow 9 Field Types?
If not, how can you be sure that there are no collisions in the ID space?

I think that you need to find and use the PENs of the vendors that have 
defined the additional Netflow 9 Field Types IEs, regardless of whether 
the IDs are below or above 2^15. Unfortunately, there is no PEN for 
general experimental use (at least I have not found any) that you could 
use as a fallback if you do not find an appropriate vendor PEN.

If you want to use your own non-standard IEs, then you should use the 
PEN of your organization:

8057
   CESNET
     CESNET masters team
       masters&cesnet.cz

Maybe, you can also use this PEN to map Field Types for which you do not 
find a vendor.

Regards,
Gerhard

On 26.03.2015 08:14, Petr Velan wrote:
> Hi Andrew, all,
>
> thank you for your explanation regarding nprobe.
>
> However, we also need a fallback for unknown exporters with IEs > 
> 2^15. The generic requests for PENs need organization name, contact 
> name and email address. I can try to request the PEN for NetFlow v9 
> compatibility myself, but I'd like it to be more public. Therefore, I 
> suggest to complete the request with something like:
> *Organization Name*: NetFlow v9 to IPFIX
> *Contact Name*: IPFIX WG
> *Contact E-Mail: [email-masked] <mailto:[email-masked]>
>
> This is just a first proposal to get things moving, please add your 
> thoughts. Once the PEN is granted, we can move forward and explain its 
> purpose in a short RFC.
>
> Petr
>
> On Tue, Jan 6, 2015 at 9:07 PM, Andrew Feren <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Hi Petr,
>
>     On 01/06/2015 07:03 AM, Petr Velan wrote:
>>     Hello all,
>>
>>     I'm not sure whether this is the right place to ask, but we
>>     encountered following problem when converting NetFlow v9 messages
>>     to IPFIX.
>>
>>     Some vendors (I've heard of ntop) are using elements IDs large
>>     than 32767 in NetFlow v9. When converting messages with these
>>     elements to IPFIX, they are considered to be Enterprise Numbers.
>>     To generate proper IPFIX message, we need to do one of the following:
>>     a) Generate a list of the elements and map them to PEN of the
>>     correct vendor. However, this would result in an attempt to cover
>>     all possible elements that anybody used in NetFlow v9. Moreover,
>>     we would still have to somehow handle the cases where the element
>>     is unknown
>     This should help with ntop/nprobe
>
>     Recent versions of nprobe (since version 5.5.5 I think) all use
>     the following mapping.
>
>     PEN = 35632 and IPFIXID = (v9ID - 57472)
>
>     For example, one v9 IE that nprobe exports is MYSQL_SERVER_VERSION
>     57667.  The IPFIX equivalent would be
>     MYSQL_SERVER_VERSION(35632/195).
>
>     The nprobe docs have a complete list.
>
>     Older versions of nprobe (pre ~2010) use IEs not in RFC 3954, but
>     later allocated in IANA.  There is no good way to convert those v9
>     exports to IPFIX.
>
>     -Andrew
>
>
>>     b) Request a PEN for NetFlow compatibility and just add this PEN
>>     for every element that has ID larger than 32767.
>>
>>     Personally, I believe that the b) is more general and
>>     error-prone. Do you think, that it would be possible to dedicate
>>     whole PEN to this cause?
>>
>>     Thank you for any opinions,
>>
>>     Petr Velan
>>
>>
>>
>>     _______________________________________________
>>     IPFIX mailing list
>>     [email-masked]  <mailto:[email-masked]>
>>     https://www.ietf.org/mailman/listinfo/ipfix
>
>
>
>
> _______________________________________________
> IPFIX mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ipfix

---

From: Petr Velan <[email-masked]>
Date: Thu, 16 Apr 2015 10:09:27 +0200

Hi Gerhard,

the problem we are facing here is that the NetFlow v9 ID space is not
centrally managed, therefore we cannot hope to cover all possible IDs and
vendors, since they can change. Moreover, as you say, the owner of a PEN
maintains his ID registry, therefore we should not use someones PEN just
because he defined the nf9 elements.

What I propose is similar to RFC 5612. We should define an NetFlow to IPFIX
conversion PEN which could be used by anyone for mapping nf9 elements with
ID > 2^15. The ID of that element would not be changed, only the PEN would
be added, which, as an added benefit, would be fairly easy to implement in
IPFIX mediators. No central management is necessary, just as you need to
know the nf9 elements that you use, you would have to know the semantics
from the appropriate vendor. The collisions cannot be avoided, however,
they are not managed in nf9 either.

As you say, there is currently no PEN for general experimental use that
could be used as a fallback. I think that creating a specific PEN for this
task is the right solution.

Best regards,
Petr

On Thu, Apr 2, 2015 at 10:36 PM, Gerhard Muenz <[email-masked]> wrote:

> centrally managed in a kind of registry because uniqueness of ID usage must
> be ensured. Usually, the owner of a PEN maintains such a registry.
>
> defined the additional Netflow 9 Field Types IEs, regardless of whether the
> IDs are below or above 2^15. Unfortunately, there is no PEN for general
> experimental use (at least I have not found any) that you could use as a
> fallback if you do not find an appropriate vendor PEN.
>
> If you want to use your own non-standard IEs, then you should use the PEN
> of your organization:
>
> 8057
>   CESNET
>     CESNET masters team
>       masters&cesnet.cz
>
> However, we also need a fallback for unknown exporters with IEs > 2^15.
> The generic requests for PENs need organization name, contact name and
> email address. I can try to request the PEN for NetFlow v9 compatibility
> myself, but I'd like it to be more public. Therefore, I suggest to complete
> the request with something like:
> *Contact E-Mail: [email-masked]
>
> On Tue, Jan 6, 2015 at 9:07 PM, Andrew Feren <[email-masked]> wrote:
>
>>  Hi Petr,
>>
>>  I'm not sure whether this is the right place to ask, but we encountered
>> following problem when converting NetFlow v9 messages to IPFIX.
>>
>>  Some vendors (I've heard of ntop) are using elements IDs large than
>> 32767 in NetFlow v9. When converting messages with these elements to IPFIX,
>> they are considered to be Enterprise Numbers. To generate proper IPFIX
>> message, we need to do one of the following:
>>  a) Generate a list of the elements and map them to PEN of the correct
>> vendor. However, this would result in an attempt to cover all possible
>> elements that anybody used in NetFlow v9. Moreover, we would still have to
>> somehow handle the cases where the element is unknown
>>
>> Recent versions of nprobe (since version 5.5.5 I think) all use the
>> following mapping.
>>
>> Older versions of nprobe (pre ~2010) use IEs not in RFC 3954, but later
>> allocated in IANA.  There is no good way to convert those v9 exports to
>> IPFIX.
>>
>> -Andrew
>>
>>
>>    b) Request a PEN for NetFlow compatibility and just add this PEN for
>> every element that has ID larger than 32767.
>>
>>  Personally, I believe that the b) is more general and error-prone. Do
>> you think, that it would be possible to dedicate whole PEN to this cause?
>>
>> IPFIX mailing [email-masked]://www.ietf.org/mailman/listinfo/ipfix
>>
>>
>>
>
>
> IPFIX mailing [email-masked]://www.ietf.org/mailman/listinfo/ipfix
>
>
>

---

From: Paul Aitken <[email-masked]>
Date: Thu, 16 Apr 2015 10:54:30 +0100

Petr,

> centrally managed

NetFlow v9 is a cisco protocol, and the IDs are allocated by Cisco's 
netflow police. Contact [email-masked].

Anyone using NFv9 IDs without contacting them is asking for trouble (ie, 
ID collisions).

> therefore we cannot hope to cover all possible IDs and vendors, since 
> they can change. Moreover, as you say, the owner of a PEN maintains 
> his ID registry, therefore we should not use someones PEN just because 
> he defined the nf9 elements.

Equally, nobody should be self-defining NFv9 elements without reference 
to Cisco, because they've no idea what IDs Cisco (or others) might use 
in future.

> What I propose is similar to RFC 5612. We should define an NetFlow to 
> IPFIX conversion PEN which could be used by anyone for mapping nf9 
> elements with ID > 2^15. The ID of that element would not be changed, 
> only the PEN would be added, which, as an added benefit, would be 
> fairly easy to implement in IPFIX mediators. No central management is 
> necessary, just as you need to know the nf9 elements that you use, you 
> would have to know the semantics from the appropriate vendor. The 
> collisions cannot be avoided, however, they are not managed in nf9 either.
>
> As you say, there is currently no PEN for general experimental use 
> that could be used as a fallback.

Even if there was, you shouldn't use such a PEN in a released product 
because it could conflict with someone else's usage.

P.

> I think that creating a specific PEN for this task is the right solution.
>
> Best regards,
> Petr
>
> On Thu, Apr 2, 2015 at 10:36 PM, Gerhard Muenz <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>
>     Hi Petr,
>
>     It does not make sense to register a PEN for an ID space that is
>     not centrally managed in a kind of registry because uniqueness of
>     ID usage must be ensured. Usually, the owner of a PEN maintains
>     such a registry.
>
>     I think that you need to find and use the PENs of the vendors that
>     have defined the additional Netflow 9 Field Types IEs, regardless
>     of whether the IDs are below or above 2^15. Unfortunately, there
>     is no PEN for general experimental use (at least I have not found
>     any) that you could use as a fallback if you do not find an
>     appropriate vendor PEN.
>
>     If you want to use your own non-standard IEs, then you should use
>     the PEN of your organization:
>
>     8057
>       CESNET
>         CESNET masters team
>           masters&cesnet.cz <http://cesnet.cz>
>
>     Maybe, you can also use this PEN to map Field Types for which you
>     do not find a vendor.
>
>     Regards,
>     Gerhard
>
>
>
>>     2^15. The generic requests for PENs need organization name,
>>     contact name and email address. I can try to request the PEN for
>>     NetFlow v9 compatibility myself, but I'd like it to be more
>>     public. Therefore, I suggest to complete the request with
>>     something like:
>>     This is just a first proposal to get things moving, please add
>>     your thoughts. Once the PEN is granted, we can move forward and
>>     explain its purpose in a short RFC.
>>
>>     Petr
>>
>>>         encountered following problem when converting NetFlow v9
>>>         messages to IPFIX.
>>>
>>>         Some vendors (I've heard of ntop) are using elements IDs
>>>         large than 32767 in NetFlow v9. When converting messages
>>>         with these elements to IPFIX, they are considered to be
>>>         Enterprise Numbers. To generate proper IPFIX message, we
>>>         need to do one of the following:
>>>         a) Generate a list of the elements and map them to PEN of
>>>         the correct vendor. However, this would result in an attempt
>>>         to cover all possible elements that anybody used in NetFlow
>>>         v9. Moreover, we would still have to somehow handle the
>>>         cases where the element is unknown
>>         Recent versions of nprobe (since version 5.5.5 I think) all
>>         use the following mapping.
>>
>>         For example, one v9 IE that nprobe exports is
>>         MYSQL_SERVER_VERSION 57667. The IPFIX equivalent would be
>>         Older versions of nprobe (pre ~2010) use IEs not in RFC 3954,
>>         but later allocated in IANA.  There is no good way to convert
>>         those v9 exports to IPFIX.
>>
>>         -Andrew
>>
>>
>>>         b) Request a PEN for NetFlow compatibility and just add this
>>>         PEN for every element that has ID larger than 32767.
>>>
>>>         error-prone. Do you think, that it would be possible to
>>>         dedicate whole PEN to this cause?
>>>