Subject: [IPFIX] Observation Domains, Templates, and Session

From: Andrew Johnson <[email-masked]>
Date: Thu, 05 Oct 2006 19:05:23 +0100

Hello all

With regard to the recent discussion on the mailing list titled:
  "Re: [IPFIX] Re: [ipfix-protocol] IPFIX-INFO last call comment:
   Observation Domains, Templates, and Sessions"

The discussion doesn't seem to be going anywhere and I hope this
email will define the heart of the problem sufficiently that we
can all jump on board and find a solution.  I realise this is
quite a long email but it's written in logical sections, each one
providing more details.  You can stop reading as soon as you
agree with me :-)

regards

Andrew

THE PROBLEM
===========
It is agreed that with the present definition of the IPFIX protocol
the Collecting Process cannot reliably differentiate between
different Exporting Processes on a single IPFIX device.  This is a
problem because the Observation Domain ID is considered "unique per
Exporting Process".

Now if the Collector cannot differentiate between Exporting
Processes, how does it correctly scope this Identifier?  In
short, it can't.  We MUST change the scoping of this identifier
to something the CP can differentiate.

Re-scoping OD ID requires care because it is included in the IPFIX
header and several other IEs are scoped per Observation Domain:
  templateId,
  flowId,
  observationPointId and
  commonPropertiesId.

WHAT WE HAVE TODAY
==================
Currently the Observation Domain is considered unique per Exporting
Process and the Exporting Process ID is considered unique per IPFIX
device.  Scope has to be inherited, i.e. a template ID is really
unique per Observation Domain, per Exporting Process, per IPFIX
Device.

Currently we have:
  IPFIX Device -> Exporting Process -> Observation Domain -> Template

This means that each Exporting process has a set of templates and
other identifiers per Observation Domain.  This is great because then
the different parts of a distributed Exporter simply need to pick a
different OD ID and then they can use templates etc. as required.
If you look at the history of the OD ID (sourceID) you can see this
was the original purpose (see below).

Alas, we can't have Exporting Process as the scope so what are the
alternatives?

PROPOSED SOLUTION
=================
Scope the Observation Domain to the communication session between the
EP and the CP.  This is very similar to scoping it to the Exporting
Process however if an Exporting Process has a session restart then
it would have to resend the definitions of all its templates and
other identifiers.

Well originally, we thought that the Collecting Process would
differentiate between Exporting Processes via the session and
the protocol already says that the templates must be resent on a
TCP or SCTP session restart.

Now that we realise that a session can restart even if the Exporting
Process doesn't we have to be explicit and make sure we resend all the
identifiers scoped to the session for all transport protocols,
including UDP.

SO WHAT'S THE DOWNSIDE?
=======================
Well the humble sourceID has now evolved into an identifier of the
Observation Domain and there are some who feel that that is a really
useful thing to know.  The thing is we don't actually ever know the
Observation Domain, we know the Observation Domain ID, but it is
never defined.

If we look at the history of the Observation Domain ID (sourceID)
we can see it was never intended as anything other than something
to scope other fields.  Recent editing changes have evolved how we
have come to think of it.

HISTORY OF THE OBSERVATION DOMAIN ID
====================================
It began life as the source ID.  It was used, as in v9, by an
Exporting Process to select the scope of information from different
line cards in a distributed system.  By using a different source ID
per line card, each line card could use its own set of templates and
other identifiers.  From IPFIX-INFO 7 to 11:
  5.1.9.  sourceId
     Description:
        An identifier of an Observation Domain that is locally unique to
        an Exporting Process.  Typically, this Information Element is used
        for limiting the scope of other Information Elements.

Then we changed it to be called the Observation Domain ID.  After all,
it is an identifier of the Observation Domain, right?  Well it is, but
it's primary function is still to make a distributed Exporting Process
easier to manage by allowing different line cards to have their own set
of templates and identifiers.  From IPFIX-INFO 12:
  5.1.9.  observationDomainId
     Description:
        An identifier of an Observation Domain that is unique per
        Exporting Process.

And now in from IPFIX-INFO 13:
  5.1.9.  observationDomainId
     Description:
        An identifier of an Observation Domain that is locally unique to
        an Exporting Process.  The Exporting Process uses the Observation
        Domain ID to uniquely identify to the Collecting Process the
        Observation Domain where Flows were metered.

Now these are the definition of the Observation Domain ID Information
Element.  Note that the Observation Domain ID is included in the header
of every IPFIX message, and this is why it is so important.

_______________________________________________
IPFIX mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipfix

---

From: Sven Anderson <[email-masked]>
Date: Fri, 06 Oct 2006 17:07:03 +0200

Hi all,

> providing more details.  You can stop reading as soon as you

if you are interested in a bit more abstract, top-down approach thoughts
from an outsider viewpoint: I think you guys _have_ to use a session ID
for scoping.

(Be careful, the following might be confusing to some people. ;-) )

To avoid collisions the collector needs to globally identify the template
of each record in some way. So either you send the whole cascade of
identifiers with each record, which together form a globally unique
identifier for the template, or - to save space in the headers - you use a
temporary identifier, which is unique for both the sender and receiver and
which is mapped to the global template identifier or at least a part of
it. For the second option you need a handshake, which is not possible with
the IPFIX protocol, as it is designed as a one-way protocol. So the only
handshake that happens is on the transport layer for session management.
Only this session identifier you can use as a replacement for the part of
the identifier cascade, which doesn't change during one session, in order
to save space in the headers.

So you would probably, if I understand it correctly, map the session ID to
{IPFIX Device ID, Exporting Process ID} on the collectors side and scope
the OD ID to the session ID. If you expect that the OD ID is most of the
time the same during one session, you could even map the session ID to
{IPFIX Device ID, Exporting Process ID, OD ID} and scope the template ID
directly to the session ID, then you could even remove the OD ID from the
headers. Of course with the down side, that you have to open several
sessions for several OD IDs. But probably changing the IPFIX protocol is
not an option at this time anyway. ;-)

Regards,

Sven

-- 
Sven Anderson
Institute for Informatics - http://www.ifi.informatik.uni-goettingen.de
Georg-August-Universitaet Goettingen
Lotzestr. 16-18, 37083 Goettingen, Germany

---

From: Sven Anderson <[email-masked]>
Date: Fri, 06 Oct 2006 17:07:03 +0200

Hi all,

Regards,

Sven

---

From: Andrew Johnson <[email-masked]>
Date: Fri, 06 Oct 2006 17:04:55 +0100

Sven Anderson wrote:
> the OD ID to the session ID.

Almost.  The Collecting Process can't identify the Exporting Process.  I'm
proposing that the session identifier won't map directly to anything, but
instead provide scope as a replacement for the Exporting Process ID.

regards

Andrew

> If you expect that the OD ID is most of the

---

From: Andrew Johnson <[email-masked]>
Date: Fri, 06 Oct 2006 17:04:55 +0100

regards

Andrew

---

From: Lutz Mark <[email-masked]>
Date: Mon, 09 Oct 2006 16:29:29 +0200

Andrew,

> THE PROBLEM

yes.

> problem because the Observation Domain ID is considered "unique per

I don't follow you here.

You can have the observation domain unique per Exporting Process.
It is enough to limit the validity of TemplateIDs and 
CommonPropertiesIDs to the respective communication sessions.

Then the Collecting Process does not need any knowledge
about the originating Exporting Process.

Or do I miss something?

Best regards,
Lutz

---

From: Lutz Mark <[email-masked]>
Date: Mon, 09 Oct 2006 16:29:29 +0200

Andrew,

yes.

I don't follow you here.

Or do I miss something?

Best regards,
Lutz