Subject: [IPFIX] Maji - an open source IPFIX meter implementation

From: Shane Alcock <[email-masked]>
Date: Mon, 16 Feb 2009 15:31:34 +1300

Hi all,

I'm pleased to announce that maji, an open source implementation of an 
IPFIX meter developed at the University of Waikato, has finally been 
released.

Some of the major features include:
    * Read packets from a variety of sources including PCAP interfaces, 
trace files and DAG capture cards
    * Support for over 50 different standard information elements 
defined in RFC 5101
    * Define your own data and option templates, rather than being 
limited to fixed templates
    * Export IPFIX messages via SCTP, TCP or UDP, or export IPFIX flow 
records directly to an SQLite database

One of the design goals for maji was to create a framework that could be 
utilised by researchers to test new concepts and ideas for IPFIX without 
having to implement the entire protocol themselves. To this end, maji 
can be easily extended to include new information elements, export 
formats or special templates. A series of developer notes are included 
as part of the documentation that should aid anyone looking to build 
upon our code.

Further details and a link to download maji can be found at 
http://research.wand.net.nz/software/maji.php

We appreciate any feedback you might have about maji. Please direct any 
questions, comments or bug reports to myself or feel free to file a 
ticket in our bug tracker at http://www.wand.net.nz/trac/traceflow_ipfix

Shane Alcock
Research Assistant
WAND Network Research Group
University of Waikato
Hamilton, New Zealand
Email: [email-masked]

---

From: Gerhard Muenz <[email-masked]>
Date: Mon, 23 Feb 2009 17:03:36 +0100

Hi Shane, all,

I had a quick lood at the code of Maji because I was interested how you
configure your templates.

Please tell me if the following is correct:
It seems that you configure the templates exactly as they will be
exported. The definition of the templates implicitly configures the
metering process. Packets that contain all fields of a given template
are aggregated into a corresponding flow record. On the other hand, if
any field of the template is missing, the packet is not accounted.
Hence, the configuration of the template implicitly defines a packet filter.

This is a feasible approach. BTW, we did the same in our monitoring
probe implementation Vermont.

However, in the course of standardizing the IPFIX configuration model,
we got to a different solution where we configure packet filters and
"cache layouts" of the metering process. The cache layout defines a
superset of all fields that are contained in the resulting records if
they are available in the incoming packets. Hence, we do not drop a
packet or flow if a field is missing. Instead, it is accounted in a
record that does not include this field. As a consequence, one cache
layout definition may result in multiple templates that have to be
generated on demand by the exporting process.

The idea is to clearly separate metering and export functions in the
configuration model. Also, the configuration of cache layouts instead of
templates is closer to Cisco's current version of Flexible Netflow.
As a downside, the exporting process must support dynamic template
generation, which can be quite complex to implement.

After long discussions with Paul, I was convinced that configuring the
metering process through the template is putting the cart before the horse.
However, I also think that the configuration of a fixed set of fields
per record (as in Maji and Vermont) has its merits because it easy to
implement.

If there is support from your side (and maybe others), we can extend the
configuration model such that the configuration of such IPFIX
implementations is supported as well.

Regards,
Gerhard

Shane Alcock wrote:
>    * Support for over 50 different standard information elements defined
> in RFC 5101
> _______________________________________________
> IPFIX mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ipfix

-- 
Dipl.-Ing. Gerhard Münz
Chair for Network Architectures and Services (I8)
Department of Informatics
Technische Universität München
Boltzmannstr. 3, 85748 Garching bei München, Germany
Phone:  +49 89 289-18008       Fax: +49 89 289-18033
E-mail: [email-masked]    WWW: http://www.net.in.tum.de/~muenz

---

From: Paul Aitken <[email-masked]>
Date: Mon, 23 Feb 2009 21:55:08 +0000

Gerhard, Shane,

> metering process through the template is putting the cart before the horse.

A support question I received some years ago asked how to reconfigure 
our NFv9 templates - because they thought this would change what was 
being monitored. It wouldn't; that's never been our model. Export has 
always followed from monitoring and not the other way around.

> implementations is supported as well.

The existing IPFIX-CONFIG design already allows for this, if you 
consider that the cache is being configured (rather than the exporter, 
as in Maji and Vermont) and that the exporter matches with it 1:1.

P.

-- 
Paul Aitken
Cisco Systems Ltd, Edinburgh, Scotland.

---

From: Shane Alcock <[email-masked]>
Date: Tue, 24 Feb 2009 11:52:54 +1300

Gerhard, all.

Thanks for taking the time to look over our efforts! Further responses 
in-line.

> Hence, the configuration of the template implicitly defines a packet filter.
This is true to an extent. If any field that is part of the flow key for 
a template is missing, then the packet will be essentially ignored. If 
the field is not part of the flow key, a "default" value is provided for 
that field, usually either 0 or MAXINT (depending on whether zero is a 
valid value for the field or not). This is far from an ideal approach, 
as it would be very easy for a collector or analysis program to 
misinterpret the defaults as genuine measurements.
> generation, which can be quite complex to implement.
This would alleviate the problems with the "default" values and an 
implementation of this approach may make it into a subsequent version of 
maji (at least for non-key elements) :)
> implementations is supported as well.
Maji has always been developed primarily as a tool to enable and aid 
network measurement research. The design has been based around the idea 
of placing control over what gets metered entirely into the hands of the 
user. The template-driven configuration approach works really well in 
this regard and I think it is important for it to at least be 
acknowledged as valid within the configuration model, possibly depending 
on the circumstances in which the meter is operating.

Shane

---

From: Gerhard Muenz <[email-masked]>
Date: Tue, 24 Feb 2009 10:32:06 +0100

Hi Shane,

Please find replies inline.

> misinterpret the defaults as genuine measurements.

Ok, so you distinguish between missing flow key fields and non-key fields.

What do you mean by "depending on whether zero is valid or not"?
Do you use zero if it is a valid value? Or is the default value chosen
to be invalid so that the collector can recognize it as such?

Anyway, records are generated that contain fields that contain wrong or
invalid values. Netflow used to do similar things. Often, the collector
can recognize such invalid fields, for example if a TCP flags field
appears in a UDP flow.

However, I do not think that the export of invalid field values is
conform to the IPFIX standard. If we wanted to do so, we would need to
define which values to be considered as invalid for every Information
Element. This is not done in RFC5102.

> maji (at least for non-key elements) :)

Ok :) but if restricted to non-key elements, you still implicitly drop
packets that do not have all flow key fields.

> on the circumstances in which the meter is operating.

Thanks for your feedback. I'll propose a solution in an answer to Paul's
reply in the parallel thread.

Gerhard

---

From: Gerhard Muenz <[email-masked]>
Date: Tue, 24 Feb 2009 10:51:30 +0100

Paul,

> as in Maji and Vermont) and that the exporter matches with it 1:1.

In fact, it is not possible to describe a configuration that results in
a 1:1 mapping of Cache Layout and Template with the existing IPFIX-CONFIG.

The current rules guarantee that a Cache accounts every incoming packet
in a single Flow Record. At the end, this may result in Flow Records
with different combination of fields defined in the Cache Layout. For
each of these combinations, the Exporting Process will define a
different Template.

The "Cache" of Maji and Vermont only accounts packets that contain a
certain set of fields. As a result, it generates Flow Records with one
fixed combination of fields. To achieve this, it implicitly filters out
packets that miss any of these fields specified in the Cache Layout.
With the current configuration model, it is not possible to configure
such a filter mechansim that only accepts packets from which certain
fields can be extracted.

To solve this, we could add a parameter that specifies how packets with
missing fields are treated by the Cache, with settings such as "accept"
and "ignore".

I'm open to better solutions :)

Gerhard

---

From: Paul Aitken <[email-masked]>
Date: Tue, 24 Feb 2009 11:19:26 +0000

Gerhard,

> a 1:1 mapping of Cache Layout and Template with the existing IPFIX-CONFIG.

I'm surprised. I must re-review the draft.

> with different combination of fields defined in the Cache Layout.

Because some fields could not be observed.

> For each of these combinations, the Exporting Process will define a
> different Template.

For simplicity - and especially when a large number of fields might lead 
to a combinatorial explosion in the number of templates - it'd be good 
if we could export a single template describing all the fields in the 
cache, and either set the unobserved fields to an out-of-band value or 
export a bitfield indicating which particular fields had been observed.

> missing fields are treated by the Cache, with settings such as "accept"

This reminds me of an ACL. It seems to be an important and necessary 
filter, complimentary to the existing filters. eg, if you only want to 
account TCP traffic, or only IPv6, or only traffic with a specific IPv4 
option or IPv6 extension header. Or again, if you specifically want to 
rule out certain subsets of traffic.

P.

---

From: Gerhard Muenz <[email-masked]>
Date: Tue, 24 Feb 2009 13:13:44 +0100

Paul,

> export a bitfield indicating which particular fields had been observed.

Out-of-band values do not exist for many Information Elements. And if
they exist, IPFIX currently does not specify how to interpret them.

The bitfield seems to be a feasible solution that can be easily
standardized.

However, we first need a consensus in the WG that invalid field values
in a Record are generally acceptable.

> rule out certain subsets of traffic.

You are right, the filter could be a new type of Selector that is
implemented in front of the cache:

            +----+    +-------+
packets --->| SP |--->| Cache |---> records
            +----+    +-------+

In this case, the SP must be configured with the same fields as the Cache!

A shortcut would be the additional parameter that I mentioned before:

            +-----------------------------------------+
packets --->| Cache                                   |---> records
            | ignorePacketsWithMissingFields = "true" |
            +-----------------------------------------+

(I havn't found a better parameter name, yet.)

Gerhard

---

From: Paul Aitken <[email-masked]>
Date: Tue, 24 Feb 2009 12:54:46 +0000

Gerhard,

> they exist, IPFIX currently does not specify how to interpret them.

Exactly.

> The bitfield seems to be a feasible solution that can be easily

Good, we agree.

I'm thinking of something similar to the flowKeyIndicator.

> in a Record are generally acceptable.

Sure. There are advantages in several situations:

1) where the exporting or collecting process/device is too simple (lack 
of memory, CPU, time etc) to be able to make/store/decode all the 
necessary templates.

2) where an excessive combination of fields might make the number of 
templates unmanageable.

> implemented in front of the cache:

Yes. However, an implementation of the configuration model could easily 
take care of that - perhaps even implementing it as you show below.

> A shortcut would be the additional parameter that I mentioned before:

This allows modeling of "ignore non-TCP traffic" (ie, "monitor only TCP 
traffic") by putting TCP specific fields in the cache. However, it 
doesn't allow the opposite, ie "monitor non-TCP traffic" - so it's not 
quite as flexible as having a pre-filtering SP.

Cheers,
P.

---

From: Andrew Johnson <[email-masked]>
Date: Tue, 24 Feb 2009 13:10:34 +0000

Hi Gerhard

Gerhard Muenz wrote:
[SNIP]
> I'm open to better solutions :)

It seems to me that explicitly configuring the filter would be more helpful.

In [PSAMP-PROTO], section 6.5.2 there are details on how to export the 
config of various types of filtering and sampling mechanisms. 
Specifically 6.5.2.5 details how filters are configured.  This could be 
used as the basis for an extension to ipfix-configuration that would 
specify how to configure filtering.

Cheers

Andrew

---

From: Brian Trammell <[email-masked]>
Date: Tue, 24 Feb 2009 14:24:30 +0100

Hi Andrew, all,

I'd tend to agree with Andrew on this point; specifying implicit  
filtering through template specification seems to be a _very_  
implementation-specific sort of thing, which should to the extent  
possible be avoided in the configuration draft.

Cheers,

Brian

On Feb 24, 2009, at 2:10 PM, Andrew Johnson wrote:

> In [PSAMP-PROTO], section 6.5.2 there are details on how to export  
> the config of various types of filtering and sampling mechanisms.  
> Specifically 6.5.2.5 details how filters are configured.  This could  
> be used as the basis for an extension to ipfix-configuration that  
> would specify how to configure filtering.
>
> Cheers
>
> Andrew
>

---

From: Gerhard Muenz <[email-masked]>
Date: Tue, 24 Feb 2009 14:37:01 +0100

Brian, all,

I understand what you mean, yet you use the wrong terminology ;)

Templates are currently not configured, and we do not plan to do so.

What we do configure is the Cache, which requires that we have a common
understand how a Cache functions.

Andrew is completely right that we need to report any kind of filtering
(whether implicit or explicit) to the Collector. This should be done in
the PSAMP way.

I agree with both of you that modeling the filter explicitly is more
consistent with the current separation of packet selection and flow
generation functions.

So, what is the right way to standardize a new Selector?
Is the configuration draft the right place to do this?
Same question for the bitfield Information Element Paul suggested?

Cheers,
Gerhard

---

From: Andrew Johnson <[email-masked]>
Date: Tue, 24 Feb 2009 13:48:08 +0000

Hi Gerhard

You can register a new bitfield with IANA without it being a part of any 
draft.  You can then reference it from your draft.

Why do you want a new Selector?  Can you not use the Selector that is 
defined in [PSAMP-PROTO]?

I had hoped that the configuration model would be extended to include 
the PSAMP configuration.  I thought that would most likely be done with 
a new draft that built upon your current one, but perhaps you could see 
if there are elements that are worth bringing in today.

Cheers

Andrew

---

From: Gerhard Muenz <[email-masked]>
Date: Tue, 24 Feb 2009 15:02:09 +0100

Hi Andrew,

> defined in [PSAMP-PROTO]?

PSAMP specifies "property match filtering", but that is not what we want
to do. Or do you know how to deploy this filter for our purpose?

> the PSAMP configuration.  

It already does. The configuration of those Selectors that are specified
in the PSAMP documents has been included since the beginning.

Cheers,
Gerhard

> I thought that would most likely be done with

---

From: Andrew Johnson <[email-masked]>
Date: Tue, 24 Feb 2009 14:48:13 +0000

> to do. Or do you know how to deploy this filter for our purpose?

Well, I'm not sure yet that it'll cover all fields, but it seems that it 
you can do a lot with it.  For example, if you have TCP flags in your 
cache you could filter on the IP protocol matching TCP.

Looking again, you seem to have have a start, stop and mask value for 
each filter.  So what would you expect the behaviour to be of a filter 
that demands a TCP flags value of 0 when faced with a UDP packet?

My instinct is that it should reject the UDP packet because it cannot 
provide a TCP flags value of 0.  An alternate filter could be set up to 
accept UDP packets.

If we agree on that point, then how about a TCP flags filter that 
filtered on TCP flags being in the range 0x00 to 0xFF.  It would accept 
all values for the TCP flags field, but reject packets that didn't have 
the field.

> in the PSAMP documents has been included since the beginning.

Great.  I remembered some parts of it were in, and our discussions on 
selector chaining, but had forgotten how much your draft covers.

Cheers

Andrew

---

From: Gerhard Muenz <[email-masked]>
Date: Tue, 24 Feb 2009 16:19:40 +0100

Hi Andrew,

> cache you could filter on the IP protocol matching TCP.

Concluding that one field is present because of the value of another
field is a work-around that does not work in general.

> that demands a TCP flags value of 0 when faced with a UDP packet?

I do not know. Is it defined somewhere?

> all values for the TCP flags field, but reject packets that didn't have

Then, we could place a property match filter in front of the Cache that
matches all fields of the Cache Layout with mask=0x00 (or start=0 and
end=infinity) to ensure that all fields are present.

However, I'm a little bit confused about the following paragraph in
PSAMP-TECH:

6.1 Property Match Filtering

    ....

    A packet is selected if Field=Value. Masks and ranges are only
    supported to the extent to which [RFC5102] allows them e.g. by
    providing explicit fields like the netmasks for source and
    destination addresses.

Also, PSAMP-PROTOCOL does not allow to export a value range [start,
stop] in an Option :(

Note that the configuration data model is based in PSAMP-MIB where
"start", "stop" and "mask" are mentioned as parameters of property match
filtering. It seems that these parameters are outdated?

Regards,
Gerhard

---

From: Andrew Johnson <[email-masked]>
Date: Tue, 24 Feb 2009 15:45:52 +0000

Hi Gerhard

> field is a work-around that does not work in general.

I agree, it's only useful for fields taken from the packet.

> I do not know. Is it defined somewhere?

Not as far as I know, however if you're matching on and IPv4 address of 
192.168.0.1, I would not expect all IPv6 traffic to pass the filter.

Exactly.

> Also, PSAMP-PROTOCOL does not allow to export a value range [start,

Yes, I know.  It's very difficult to export min/max/mask using IPFIX 
without defining new Information Elements for each field you want to 
filter on.

> filtering. It seems that these parameters are outdated?

When we added the Filtering Interpretation Report in version 02 we 
didn't have min/max/mask either.  There's just no easy way to export 
this sort of think in IPFIX without defining three additional I.E.s per 
field.  Unless anyone can think of something?

Cheers

Andrew

---

From: Shane Alcock <[email-masked]>
Date: Wed, 25 Feb 2009 11:06:34 +1300

Gerhard,

Responses are once again in-line.

> to be invalid so that the collector can recognize it as such?

For each information element implemented within maji, I tried to select 
a default value that was highly unlikely to occur in "reality". For most 
elements, zero worked very well for this purpose. For example, the ipTTL 
field is set to zero if there is no valid IP header in the packet. 
However, zero is a valid ICMP type, so for the icmpTypeIPv4 element, I 
instead use 255 as the default value for all non-ICMP packets.

> Element. This is not done in RFC5102.

As I had read it, the standard did not define what exactly to do in the 
case where an element could not be determined from the packet (and the 
configuration draft had not been written at the time). I had the problem 
that I had to put some value into the flow record to conform to my 
templates. In the end, I selected what I thought were the best default 
values for each individual element. The aim was to at least give the 
collector some hope of recognising the value as invalid or undefined.

I don't advocate this as the best solution, nor do I suggest that anyone 
start formally defining such values in the standard - especially as 
there will be elements for which all of the possible values are equally 
valid.

But I do think that there needs to be a nice way to indicate that 
certain fields in a flow record are undefined.  Preferably one that 
isn't reliant on my better judgment! :)

> packets that do not have all flow key fields.

Correct. In the case of maji, the template also defines what a flow is - 
if I can't extract all the key elements, how can I determine the flow 
that the packet belongs to? If a packet can't be matched to a flow, I 
have no real choice but to ignore it.

Shane

---

From: Gerhard Muenz <[email-masked]>
Date: Wed, 25 Feb 2009 09:24:01 +0100

Hi Andrew,

> 192.168.0.1, I would not expect all IPv6 traffic to pass the filter.

Is it useful to add a note to PSAMP-TECH?

> without defining new Information Elements for each field you want to

Hm, I do not think that defining min/max/mask IEs for every existing IE
is needed. I'm sure that a better solution can be found.

> field.  Unless anyone can think of something?

Yes:
http://tools.ietf.org/html/draft-sommer-ipfix-mediator-ext-01

In my opinion, the current property match filtering specified in PSAMP
is too restrictive because it only covers simplistic fiters that are
rarely used in practice. Maybe it should be removed from the documents
until a better solution is found.

Cheers,
Gerhard

---

From: Andrew Johnson <[email-masked]>
Date: Wed, 25 Feb 2009 15:06:06 +0000

> Is it useful to add a note to PSAMP-TECH?

Probably, but maybe too late now.

> is needed. I'm sure that a better solution can be found.

Perhaps, I'd certainly like to think so, but so far I've not seen any real
contenders.  I can't think of anything that doesn't need data types and/or
deviate a lot from how we use IPFIX today.

> http://tools.ietf.org/html/draft-sommer-ipfix-mediator-ext-01

I don't see how this helps, other than the ability to exclude certain
values.  I didn't see any indication of how to handle multiple uses of
the same I.E., but presumably they follow the same rule as for different
I.E.s and are added with an AND operation, and are therefore
automatically conflicting.

For example, how would you represent a match on the TCP syn flag?  With
min/max/flags we could have 0x02, 0x02, 0x02.  Easy.

In the Filtering Interpretation Report you could use multiple inclusion
filters / selection paths:

  OP1, TCP flags = 0x02
  OP1, TCP flags = 0x03
  OP1, TCP flags = 0x06
  OP1, TCP flags = 0x07
  ...
  OP1, TCP flags = 0x1A
  OP1, TCP flags = 0x1B
  OP1, TCP flags = 0x1E
  OP1, TCP flags = 0x1F

Yuck.

How would this be done with
http://tools.ietf.org/html/draft-sommer-ipfix-mediator-ext-01?

> until a better solution is found.

Well, a future draft could extend the filtering mechanisms, but I don't
think we should discard what we have today.  A new selection type can be
added later when we have a way of representing more complex solutions.

Cheers

Andrew

---

From: Gerhard Muenz <[email-masked]>
Date: Thu, 26 Feb 2009 09:37:58 +0100

Hi Andrew,

> automatically conflicting.

The abstract data type portRanges has been conceived to report a list of
port numbers. This ADT can be used to define an IE
destinationTransportPortList which can be used in the Filtering
Interpretation Report we have today.

I admit that an ADT portRange is very restrictive. Maybe, it is better
to standardize an ADT listUnsigned16 which works like "orderedList" in
the draft. This ADT can be used to express a list of values for any
unsigned16-field.

You are right, this approach requires new IEs, but only one new IE for
any existing one where it makes sense. Instead of assigning new IE Ids,
we could think of a trick using a specific enterprise number (like in
biflow) to extend the ADT of existing IEs to list-type ADTs.

> filters / selection paths:

Can you give me an example of all destination ports except port 80 ;)

> http://tools.ietf.org/html/draft-sommer-ipfix-mediator-ext-01?

We have not thought of TCP flags, yet.

First, we could define an ADT listUnsigned8. Then you can
describe all the values from above with a single IE tcpControlBitsList.

If you prefer min/max/mask, an ADT minMaxMaskUnsigned8 could be defined
with a length of 3 octets.

> added later when we have a way of representing more complex solutions.

Do you think that it is possible to standardize such ADTs that follow
this "any value of"-semantic? Or would it be a solution that deviates "a
lot from how we use IPFIX today" - as you call it?

Cheers,
Gerhard