Subject: [IPFIX] NetFlow v9 to IPFIX conversion

From: Petr Velan <[email-masked]>
Date: Tue, 6 Jan 2015 13:03:29 +0100

Hello all,

I'm not sure whether this is the right place to ask, but we encountered
following problem when converting NetFlow v9 messages to IPFIX.

Some vendors (I've heard of ntop) are using elements IDs large than 32767
in NetFlow v9. When converting messages with these elements to IPFIX, they
are considered to be Enterprise Numbers. To generate proper IPFIX message,
we need to do one of the following:
a) Generate a list of the elements and map them to PEN of the correct
vendor. However, this would result in an attempt to cover all possible
elements that anybody used in NetFlow v9. Moreover, we would still have to
somehow handle the cases where the element is unknown
b) Request a PEN for NetFlow compatibility and just add this PEN for every
element that has ID larger than 32767.

Personally, I believe that the b) is more general and error-prone. Do you
think, that it would be possible to dedicate whole PEN to this cause?

Thank you for any opinions,

Petr Velan

---

From: Paul Aitken <[email-masked]>
Date: Tue, 6 Jan 2015 14:52:42 +0100

Petr, please see replies inline:

From: IPFIX [mailto:[email-masked]] On Behalf Of Petr Velan
Sent: 06 January 2015 12:03
To: [email-masked]
Subject: [IPFIX] NetFlow v9 to IPFIX conversion

Hello all,
I'm not sure whether this is the right place to ask, but we encountered following problem when converting NetFlow v9 messages to IPFIX.
Some vendors (I've heard of ntop) are using elements IDs large than 32767 in NetFlow v9.

[P] That’s not invalid in NFv9. RFC 3954 allows a 16-bit “Field Type” ID.

When converting messages with these elements to IPFIX, they are considered to be Enterprise Numbers. To generate proper IPFIX message, we need to do one of the following:
a) Generate a list of the elements and map them to PEN of the correct vendor. However, this would result in an attempt to cover all possible elements that anybody used in NetFlow v9. Moreover, we would still have to somehow handle the cases where the element is unknown

[P] When I was netflow–police at cisco, the guys at Plixer helped to identify non-cisco NFv9 exporters, and I allocated blocks of 500 or 1000 IDs for each them in the 50,000 – 60,000 range – effectively blacklisting those IDs so that cisco wouldn’t create duplicate IDs.  At that time there were just 5 such blocks recognized by Cisco. If we can identify a PEN for each block then it’d be simple to write a decoder. Unfortunately this method doesn’t extend to recognizing future allocations, or export IDs that weren’t known to us at the time.

BTW, you’re also supposing that the IDs < 32768 are identical and will remain so in future.

b) Request a PEN for NetFlow compatibility and just add this PEN for every element that has ID larger than 32767.
[P] It’d be more accurate to use Cisco’s PEN for this ;-)  However in principal it’s a NFv9-in-IPFIX indicator.

Personally, I believe that the b) is more general and error-prone. Do you think, that it would be possible to dedicate whole PEN to this cause?
[P] I believe you could convince IANA to allocate an ID. However I’m not yet convinced that it’s a good idea.
P.

Petr Velan

---

From: Brian Trammell <[email-masked]>
Date: Tue, 6 Jan 2015 15:26:14 +0100

hi Paul, Petr,

So unless I misunderstand here, what we're talking about is third party vendors who have more or less NF9-compatible exporters that are squatting on chunks of the NF9 IE number space (i.e., there are no Cisco-defined IEs for NetFlow 9 which are > 2^15).

So the first question is why aren't they using IPFIX in the first place? But alas...

It doesn't seem right to me to use Cisco's PEN for this, since even though Cisco has reserved the squatted blocks they know about to prevent interoperability problems, there is no guarantee they know about all the blocks that exist, and it's kind of unfair to ask one company to burn PEN space to correct the mistakes of others. 

I'd be in favor of asking IANA for a new PEN for this, since it really is a protocol-level incompatibility between NetFlow 9 and IPFIX, and it would be a relatively quick AD-sponsored or opsawg draft to do. (I do think it needs an RFC, so that the PEN registry can point at something to say "what is this?"; see e.g. PEN 29305 for RFC 5103). 

Yes, this has the problem that squatting blocks might collide with each other. I'm perfectly okay with that. As a matter of policy, we should not sanction the de facto creation of codepoints in public or semipublic registries through squatting, and while building a registry of squatted blocks to appropriate PENs is a more *elegant* solution, it seems like way too much effort for the community to expend to come up with a nice fix to a messy problem caused by the lazy. 

Cheers,

Brian

> On 06 Jan 2015, at 14:52, Paul Aitken <[email-masked]> wrote:
> 
> _______________________________________________
> IPFIX mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ipfix

---

From: Paul Aitken <[email-masked]>
Date: Tue, 6 Jan 2015 16:31:28 +0100

Brian,

> -----Original Message-----
> From: Brian Trammell [mailto:[email-masked]]
> Sent: 06 January 2015 14:26
> To: Paul Aitken
> Cc: Petr Velan; [email-masked]
> Subject: Re: [IPFIX] NetFlow v9 to IPFIX conversion
> 
> hi Paul, Petr,
> 
> So unless I misunderstand here, what we're talking about is third party vendors
> who have more or less NF9-compatible exporters that are squatting on chunks
> of the NF9 IE number space

Correct - though "squatting" might not be quite the right word. These IDs were allocated with cisco's knowledge in order to prevent NFv9 ID clashes.

> (i.e., there are no Cisco-defined IEs for NetFlow 9 which are > 2^15).

I believe there are many Cisco IEs > 32768.

> So the first question is why aren't they using IPFIX in the first place? But alas...

Doubtless it was easier to use some NFv9 IDs than to upgrade exporters and collectors to IPFIX.

> It doesn't seem right to me to use Cisco's PEN for this, since even though Cisco
> has reserved the squatted blocks they know about to prevent interoperability
> problems, there is no guarantee they know about all the blocks that exist, and
> it's kind of unfair to ask one company to burn PEN space to correct the mistakes
> of others.

+1

> I'd be in favor of asking IANA for a new PEN for this, since it really is a protocol-
> level incompatibility between NetFlow 9 and IPFIX, and it would be a relatively
> quick AD-sponsored or opsawg draft to do. (I do think it needs an RFC, so that
> the PEN registry can point at something to say "what is this?"; see e.g. PEN
> 29305 for RFC 5103).
> 
> Yes, this has the problem that squatting blocks might collide with each other. I'm
> perfectly okay with that. As a matter of policy, we should not sanction the de
> facto creation of codepoints in public or semipublic registries through squatting,
> and while building a registry of squatted blocks to appropriate PENs is a more
> *elegant* solution, it seems like way too much effort for the community to
> expend to come up with a nice fix to a messy problem caused by the lazy.

As far as I know, that registry currently has just 5 non-cisco entries, so it wouldn't be hard to code.

However as you say there could be other ID conflicts which aren't known about today, and a new PEN would remove the guesswork.

P.

> > When converting messages with these elements to IPFIX, they are considered
> to be Enterprise Numbers. To generate proper IPFIX message, we need to do
> one of the following:
> > a) Generate a list of the elements and map them to PEN of the correct vendor.
> However, this would result in an attempt to cover all possible elements that
> anybody used in NetFlow v9. Moreover, we would still have to somehow handle
> the cases where the element is unknown
> >
> > [P] When I was netflow–police at cisco, the guys at Plixer helped to identify
> non-cisco NFv9 exporters, and I allocated blocks of 500 or 1000 IDs for each
> them in the 50,000 – 60,000 range – effectively blacklisting those IDs so that
> cisco wouldn’t create duplicate IDs.  At that time there were just 5 such blocks
> recognized by Cisco. If we can identify a PEN for each block then it’d be simple
> to write a decoder. Unfortunately this method doesn’t extend to recognizing
> future allocations, or export IDs that weren’t known to us at the time.
> >
> > Personally, I believe that the b) is more general and error-prone. Do you think,
> that it would be possible to dedicate whole PEN to this cause?
> >

---

From: Brian Trammell <[email-masked]>
Date: Tue, 6 Jan 2015 16:33:47 +0100

hi Paul,

> On 06 Jan 2015, at 16:31, Paul Aitken <[email-masked]> wrote:
> 
> Brian,
> 
> 

Ah, okay. Then, on review... well, while maybe I'd tone down the cynicism a little bit, everything I say below about my opinion stands. :)

Cheers,

Brian

---

From: Andrew Feren <[email-masked]>
Date: Tue, 6 Jan 2015 14:17:29 -0500

Hi Brian,

> So unless I misunderstand here, what we're talking about is third party vendors who have more or less NF9-compatible exporters that are squatting on chunks of the NF9 IE number space (i.e., there a…
Not exactly.  There are Cisco defined IEs >2^15.  They just don't happen
to be in the range 50,000 - 60,000.

> So the first question is why aren't they using IPFIX in the first place? But alas...
Mostly this is a transitional tactic. 
> It doesn't seem right to me to use Cisco's PEN for this, since even though Cisco has reserved the squatted blocks they know about to prevent interoperability problems, there is no guarantee they kn…
I sort of agree.  In the absence of squatters using Cisco's PEN is the
right thing to do.  From what I see in the wild using Cisco's PEN will,
in practice, be right far more than wrong.  Given that there are
squatters it seems polite to not use Cisco's PEN space unless one is
reasonably certain that it is a Cisco v9 export that is being mapped.

> I'd be in favor of asking IANA for a new PEN for this, since it really is a protocol-level incompatibility between NetFlow 9 and IPFIX, and it would be a relatively quick AD-sponsored or opsawg dra…
I'm OK with a new PEN and RFC to explain it.  At the very least it
provides a path forward forward for mediators that doesn't result in
collisions in Cisco's PEN space. 

Is this a problem for anything other than RFC 7119 mediators?
> Yes, this has the problem that squatting blocks might collide with each other. I'm perfectly okay with that. As a matter of policy, we should not sanction the de facto creation of codepoints in pub…
Instead of the community; each mediator can build their own list of
elegant mappings and fall back to using the new PEN when a better
mapping can't be determined.  OK.

-Andrew

---

From: Andrew Feren <[email-masked]>
Date: Tue, 6 Jan 2015 15:07:51 -0500

Hi Petr,

On 01/06/2015 07:03 AM, Petr Velan wrote:
> Hello all,
>
> I'm not sure whether this is the right place to ask, but we
> encountered following problem when converting NetFlow v9 messages to
> IPFIX.
>
> Some vendors (I've heard of ntop) are using elements IDs large than
> 32767 in NetFlow v9. When converting messages with these elements to
> IPFIX, they are considered to be Enterprise Numbers. To generate
> proper IPFIX message, we need to do one of the following:
> elements that anybody used in NetFlow v9. Moreover, we would still
> have to somehow handle the cases where the element is unknown
This should help with ntop/nprobe

Recent versions of nprobe (since version 5.5.5 I think) all use the
following mapping.

PEN = 35632 and IPFIXID = (v9ID - 57472)

For example, one v9 IE that nprobe exports is MYSQL_SERVER_VERSION
57667.  The IPFIX equivalent would be
MYSQL_SERVER_VERSION(35632/195).

The nprobe docs have a complete list.

Older versions of nprobe (pre ~2010) use IEs not in RFC 3954, but later
allocated in IANA.  There is no good way to convert those v9 exports to
IPFIX.

-Andrew

> b) Request a PEN for NetFlow compatibility and just add this PEN for
> every element that has ID larger than 32767.
>
> Personally, I believe that the b) is more general and error-prone. Do
> you think, that it would be possible to dedicate whole PEN to this cause?
>