Subject: Multiple identical Information Elements in a template

From: Benoit Claise <[email-masked]>
Date: Wed, 07 Dec 2005 18:40:07 +0100

Dear all,

During the last PSAMP IETF meeting in Vancouver, we discussed the issue 
of multiple identical Information Elements in a template.

First of all, [IPFIX-PROTO] doesn't address the issue.
However, we do realize that multiple similar I.E.s are possible in PSAMP
Example 1: in a composite selector, we must export all selector IDs
    [PSAMP-PROTO] specifies:

   If the packets are selected by a Composite Selector, the Associations 
   ID field is composed of several Primitive Selectors. In such a case, 
   the Associations Report Interpretation MUST contain the list of all 
   the Primitive Selector IDs in the Associations ID.

Example 2: a composite selector composed of two hash functions, we might want to export both hash values in the same record.
Example 3: a composite selector, we must export the input sequence number of the primitive selector.
    [PSAMP-PROTO] specifies:
   For each selected packet, the Packet Report MUST contain the 
   following information: 
   ...
   - the input sequence number(s) of any Selectors that acted on the 
   packet   

There are actually 3 solutions to the problem. I classified them in 
order of my preference
_Solution 1:_
We try to fix [IPFIX-PROTO]. I think that this is the only right 
solution. If IPFIX is used to export other information (IPPM? NSIS?), 
there is a big chance that we will face this issue again.
Let me try to propose some text for this in a next email.

_Solution 2:_
We overload the I.E.s like we did with the MPLS label: 
mplsLabelStackEntry2, mplsLabelStackEntry3, mplsLabelStackEntry4, etc...
So we would have selectorId1, selectorId2, selectorId3, selectorId4, etc...
The advantage is that we don't modify [IPFIX-PROTO]. The disadvantage is 
that we overload the information model.
How many do we need? Which do we need now, as opposed to the future?

_Solution 3:_
For each occurrence of a PSAMP I.E. that might be duplicated in a PSAMP 
record, we specify the rule in the [PSAMP-PROTO].
For example, [PSAMP-PROTO] specifies:

   If the packets are selected by a Composite Selector, the Associations 
   ID field is composed of several Primitive Selectors. In such a case, 
   the Associations Report Interpretation MUST contain the list of all 
   the Primitive Selector IDs in the Associations ID.  If multiple 
   Selectors are contained in the Associations Report Interpretation, 
   the Selectors ID MUST be identified in the order they are used. 

The advantage is that we don't have to change [IPFIX-PROTO].
The disadvantage is that we put some more PSAMP rules on the top of IPFIX.
What now if IPFIX is used by another protocol (example: NSIS) that 
requires the extra set of PSAMP rules? Shall we say that the we use the 
PSAMP protocol and not the IPFIX protocol? This doesn't work too well. I 
think that we should keep the IPFIX rules in [IPFIX-PROTO]

Feedback?

Regards, Benoit.

---

From: Benoit Claise <[email-masked]>
Date: Thu, 08 Dec 2005 15:34:00 +0100

Replying to myself with some proposed text for the solution 1...
> Dear all,
>
> During the last PSAMP IETF meeting in Vancouver, we discussed the 
> issue of multiple identical Information Elements in a template.
>
In section 8 "Template Management", after the following paragraph...

   If a specific Information Element is required by a Template but is 
   not available in observed packets, the Exporting Process MAY choose 
   to export Flow Records without this Information Element in a Data 
   Record defined by a new Template. 

I would add:

   If an Information Element is required more than once in Template,
   the different occurrences of this Information Element SHOULD follow
   the logical order of their treatments by the Metering Process.
   For example, if a selected packet goes through two hash functions,
   and if the two hash values are sent within a single template, the 
   first occurrence of the hash value should belong to the first hash
   function in the Metering Process. For example, when exporting the 
   two source IP addresses of an IPv4 in IPv4 packets, the first 
   sourceIPv4Address Information Element occurrence should be the IPv4
   address of the outer header, while the second occurrence should be 
   the inner header one.

In section 9 "The Collecting Process's Side", at the very end, I would add:

     The Collector MUST support the use of Templates containing multiple 
occurrences of
     the similar Information Elements.

Note: this text should solve the section 3.1.2.1 
"Multiple IE of same field type" open issue in draft-boschi-ipfix-implementation-guidelines-00.txt 

Regards, Benoit.
> The advantage is that we don't modify [IPFIX-PROTO]. The disadvantage 
> is that we overload the information model.
> For each occurrence of a PSAMP I.E. that might be duplicated in a 
> PSAMP record, we specify the rule in the [PSAMP-PROTO].
> requires the extra set of PSAMP rules? Shall we say that the we use 
> the PSAMP protocol and not the IPFIX protocol? This doesn't work too 
> well. I think that we should keep the IPFIX rules in [IPFIX-PROTO]
>
> Feedback?
>
> Regards, Benoit.
>
>

---

From: Sven Anderson <[email-masked]>
Date: Fri, 09 Dec 2005 17:45:38 +0100

Dear Benoit and all,

>      the similar Information Elements.

Isn't that quite exactly what I proposed in my mail of 2005/08/04? It 
was a long mail I admit, so maybe nobody read it. ;-) That mail is 
attached, if somebody now is interested to read it.

But doesn't make this all the post-/pre- I.E. discussion obsolete? 
Wouldn't it be enough just to export two I.E. of the same type? Of 
course, with the post-I.E. you have more semantics. But wouldn't it be 
better to give this semantics by other means?

But there are certain amiguities. For instance in your example (IP in 
IP), if you also export the packet size (for single packet reports) or 
in octet counters in general, is it corresponding to the outer or the 
inner packet? That's why I also proposed a kind of separator I.E. with 
length 0, which separates different groups of I.E. in the template.  And 
in each group, every I.E. MUST appear only one time. That way you always 
know which I.E. belong together.

Template example:
<sourceIPv4Address>
<octetDeltaCount>
*<newObservationGroup>* (pseudo I.E. with length 0)
<sourceIPv4Address>

That way it's clear, that the counter corresponds to the first packet 
state size, which in the IP in IP scenario is the outer packet size.

Jurgen stated to my proposal, that you try to avoid more complex 
concepts if possible. But as it seems, it's getting more complex anyway.

Cheers,

Sven

-- 
Sven Anderson
Institute for Informatics - http://www.ifi.informatik.uni-goettingen.de
Georg-August-Universitaet Goettingen
Lotzestr. 16-18, 37083 Goettingen, Germany

Subject: [ipfix-info] Proposal for IFPIX observations at middleboxes
From: Sven Anderson <[email-masked]>
To: [email-masked]
Date: Thu, 04 Aug 2005 19:05:42 +0200

Hi all,

I will try to explain again my idea for reporting flows from middleboxes
as clear and short(?) as possible:

When IP packets travel through a middlebox, their properties may change.
Examples would be TTL, IP adresses (NAT), DSCP or even fragmentation or
replication (multicasting). That means, that if an Observation Domain
includes several Observation Points, and the same packet passes more than
one of these Observation Points, some properties can be ambiguous in this
Observation Domain.

You could solve this by saying, that every propery of a flow-report
has to derive from the same observation point. But this would be
restrictive. Sometimes it is desirable, that you can classify a single
Flow by properties, that derive from _different_ Observation Points. For
example a flow-definition that includes the source IP address before and
after a NAT, or the DSCP at three different Observation Points (before
ingress linecard, after ingress linecard, after egress linecard).

To realise this, you need a way to use certain properties more than one
time in a flow-template, to correspond to the different states at the
different Observation Points. One way to do this, is the introduction of
additional post- I.E.s, which then correspond to the first and the last
Observation Point in the Observation Domain. But this is a restriction to
two special Observation Points, and the second example from above is not
solvable with this approach.

A more general solution would be to allow the use of the same I.E. more
than one time in the same template. The order of the multiple I.E.s
corresponds to the different observations as the packet travels through
the middlebox. The problem here is, that you cannot tell then, to which of
the different observations the _singular_ I.E.s belong to.

To solve this, you need a possibility to build goups of I.E.s in the
Template: The I.E. values in one "Observation Group" always derive from
the same Observation Point (for each packet!). The Observation Groups are
ordered accordingly to the sequence of Observation Points the packet is
passing. Of course there are also fields which don't depend on the
Observation Point and can be reported in any Observation Group, like
ingressInterface (not egress!), as it is always the same, no matter where
in the middlebox the packet is observed. (You may say, that these fields
_have_ to be reported in the first group then, if this is an advantage.)

Important to note is, that packets of a Flow defined by (for example) two
Observation Groups don't necessarily pass the same sequence of two
Observation Points. They just share the same Flow poperties at the their
first and second Observation Points respectively. To make sure, that all
packets passed the same sequence of Observation Points you have to include
an (yet to be defined) I.E. "observationPointID" as a Flow Key in each
Observation Group.

BTW.: It's possible that a Flow has the same observationPointID in
different Observation Groups. For example if your Flow contains two
Observation Groups, corresponding to Observation Points at the ingress and
egress interface, you could have Flows where ingress and egress interface
and therefore the observationPointID is the same (e.g. after some NAT).

The next question is: what is a good way to define these groups? Here are
just two ideas:

- my first idea, which I descibed in an former mail, was to define
"Combined Templates", which are a ordered list of other Templates. Each
Template in the Combined Template represents an Observation Group. The
reports for Combined Templates are just normal reports with all the Fields
of all the listed Templates concatenated. The disadvantage is, that a
change of IPFIX-PROTO is necessary.

- another idea is to define a special separator I.E. with length 0, like
"newObservationGroup". This pseudo-field separates the Fields in the
Template into different Observation Groups. In the reports they don't
appear, but the collector has the template and knows where to split. Here
no change in IPFIX-PROTO is necessary. (Jeroen Massar had a very similar idea)

I think this concept is a superset of the proposals of Jürgen and Benoit.
Instead of using post-I.E.s Benoit could use Templates like this (using
the second grouping-mechanism):

<sourceIPv4Address>
[...different Fields...]
<octetDeltaCount>
<DSCP>
*<newObservationGroup>*
<DSCP>

The DSCP field in the second ObservationGroup is the same as his postDSCP
field.

Jürgen would do it maybe like this:

<destinationIPv4Address>
*<newObservationGroup>*
<sourceIPv4Address>
<destinationIPv4Address>
[...different Fields...]
<octetDeltaCount>

instead of using an preDestinationIPv4Address Field.

Why do I like this concept:

- it includes all the possibilities of the other solutions.

- you can have as many Observation Groups as you want. (I have an
VPN-Relay here, where I will need 3 Observation Groups, which is not
possible with the pre-/post- solutions.)

- it is always clear, which fields derive from the same Observation Points
(same packet state). This is especially true for the counter Fields! You
can have even the same counter in different Observation Groups, if you
expect the packet size to be changed.

- you don't need any post-/pre- variants of the I.E.s You just need the
newObservationGroup I.E.. The observationPointID is a good idea anyway, I
think.

- you don't need complicated semantics, you just report an ordered
sequence of packet states. (And that's all you know, in fact.)

I'm not really sure, but I think, that the in- and out- counters also get
obsolete then. Wouldn't it be the same as having a counter in the first
and last Observation Group?

A side note: I think a packet-altering instance - like a NAT process -
which is reporting information to the metering process, should always be
seen as _two_ observation points, one before and one after the change.

Anyway, this is my "work in progress" idea. I hope I could present this
quite complex object in an understandable manner. Now tell me, why it is a
bad idea. :-)

Cheers,

Sven

-- 
Sven Anderson
Institute for Informatics - http://www.ifi.informatik.uni-goettingen.de
Georg-August-Universitaet Goettingen
Lotzestr. 16-18, 37083 Goettingen, Germany

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Andrew Johnson <[email-masked]>
Date: Fri, 16 Dec 2005 16:07:58 +0000

Hello Sven

Sven Anderson wrote:
>>      The Collector MUST support the use of Templates containing multiple occurrences of

I would say that the problem your proposal is trying to solve and the problem
that this proposal is trying to solve are subtely different.

You are trying to solve the issue of building a flow record with element
captured from different observation points.  The problem with using the
same I.E.s for the same field but seen in different places is that you
can't tell which entry in the record applies to which field.  Even adding
an order dependence doesn't solve this issue because there is no
requirement to put all versions of the element in the record.  This is
detailed in your proposal but your solution is quite a complex addition
to the protocol.

There is a seperate problem when it comes to having the same element
be applicable more than once at a single observation point.  Actually
there are two similar problems:

1. The same element applies more than once but you want to be able to
   selectively report only some of them.  i.e. MPLS labels.  In this
   case we have to use seperate I.E.s

2. The same element applies more than once but you need to report them
   all.

This is the issue Benoit's proposal is an attempt to solve.  For example
a classification routines may classify a packet as classes 1, 10 and 15
similtaneously because these classes have independent properties.  In
this example order may not seem important, but if you want to match the
classes with some statistics about each class then making it order
dependent allows the order of each to be alligned.

Andrew

> Jürgen stated to my proposal, that you try to avoid more complex 
> ------------------------------------------------------------------------
> 
> Subject:
> [ipfix-info] Proposal for IFPIX observations at middleboxes
> From:
> Sven Anderson <[email-masked]>
> Date:

--
to unsubscribe send a message to [email-masked] with
the word 'unsubscribe' in a single line as the message text body.
archive: <http://ops.ietf.org/lists/psamp/>

---

From: Benoit Claise <[email-masked]>
Date: Mon, 19 Dec 2005 15:58:00 +0100

Sven,

[sorry for the delay in replying to you]

Even if I agree that we will certainly have to solve the issue of 
exporting structured records in clean way in a future IPFIX version, 
even if I agree that your proposal is powerful, I was aiming for a quick 
editorial change in [IPFIX-PROTO]. Let's not forget that the IPFIX 
drafts are right now under IESG review.

Regards, Benoit.

> length 0, which separates different groups of I.E. in the template.  
> And in each group, every I.E. MUST appear only one time. That way you 
> always know which I.E. belong together.
>
> Template example:
> <sourceIPv4Address>
> <octetDeltaCount>