Subject: [ipfix] flowId Information Element

From: Juergen Quittek <[email-masked]>
Date: Thu, 20 Apr 2006 11:57:53 +0200

Dear all,

When Elisa and her co-authors wrote the draft on per-packet
information export, they requested the IPFIX Information Element
#148 flowId.  Now that their draft has evolved towards reducing
redundancy, their requirements have changed and the requested
Information Element is called commonPropertiesId.  Does anyone see
a need for still keeping IE flowId?

If not, I will replace it with commonPropertiesId as suggested below.

OLD
5.1.7.  flowId

   Description:
      An identifier of a Flow that is unique within an Observation
      Domain.  This Information Element can be used to distinguish
      between different Flows if Flow Keys such as IP addresses and port
      numbers are not reported or are reported in separate records.
   Abstract Data Type: unsigned32
   Data Type Semantics: identifier
   ElementId: 148
   Status: current

NEW
5.1.7.  commonPropertiesId

   Description:
      An identifier of a set of common properties that is unique within
      an Observation Domain.  This ID can be used to link to information
      reported in separate records.
   Abstract Data Type: unsigned64
   Data Type Semantics: identifier
   ElementId: 148
   Status: current

Thanks,

    Juergen

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Paul Aitken <[email-masked]>
Date: Thu, 20 Apr 2006 11:52:58 +0100

Juergen,

> If not, I will replace it with commonPropertiesId as suggested below.

Seems to me that commonPropertiesId is just one use of flowId, and there 
might be other uses - other situations where it's useful for each flow 
to have a unique ID.

So IPFIX should specify the generic flowId element, and the PP/RR drafts 
should specify the more specific use of the flowId element as a 
commonPropertiesId.

-- 
Paul Aitken
Cisco Systems Ltd, Edinburgh, Scotland.

---

From: Elisa Boschi <[email-masked]>
Date: Thu, 20 Apr 2006 13:11:47 +0200

Paul,

>
> Seems to me that commonPropertiesId is just one use of flowId, and
> there might be other uses - other situations where it's useful for
> each flow to have a unique ID.
>
> So IPFIX should specify the generic flowId element, and the PP/RR
> drafts should specify the more specific use of the flowId element as a
> commonPropertiesId.

actually commonPropertiesID is more generic than flowID. To me, it is
the flowID IE to be a special case of the commonPropertiesID IE...

cheers,
Elisa

---

From: Lutz Mark <[email-masked]>
Date: Thu, 20 Apr 2006 13:20:52 +0200

> might be other uses - other situations where it's useful for each flow 

yes, let's keep the flowID IE.

Best regards,
Lutz

---

From: Gerhard Muenz <[email-masked]>
Date: Thu, 20 Apr 2006 13:29:12 +0200

Paul, Jürgen,

I assume Elisa and Lutz know better but in my opinion, the
commonPropertiesId does not relate to a single flow. Instead it is
common to multiple flows. Thus, the term flowId would be inappropriate.

In my opinion, it's the other way round: flowId is a special case of
commonPropertiesId.

Ok, I just saw Elisa's mail... ;)

Gerhard

-- 
Dipl.-Ing. Gerhard Münz
Computer Networks and Internet
Wilhelm Schickard Institute for Computer Science
University of Tuebingen
Auf der Morgenstelle 10C 9P16, D-72076 Tuebingen, Germany
Phone: +49 7071 29-70534 / Fax: +49 7071 29-5220
EMail: [email-masked]
WWW:   http://net.informatik.uni-tuebingen.de/~muenz

---

From: Brian Trammell <[email-masked]>
Date: Thu, 20 Apr 2006 09:34:05 -0400

Paul and Juergen,

flowId and commonPropertiesId are two logically separate information  
elements.

flowId is specifically bound to the definition of an IPFIX Flow, and  
may be useful in certain situations. Off the top of my head I can  
think of at least one: if a flow requires a record to describe it  
larger than the IPFIX message MTU, the Flow could be described by two  
Flow Records linked by flowId. commonPropertiesId would not be  
entirely appropriate in this situation, as it does not carry the  
additional semantic "these records describe the same Flow".

commonPropertiesId is, on the other hand, a generic IE useful in many  
more situations; indeed, any situation where one needs a key to  
reference an options record from multiple data records. It is  
completely "virtual"; it's part of a mechanism, not a description of  
an object.

They are different enough in meaning, and each useful enough on their  
own, that I would suggest keeping flowId as is, and assigning a new  
IE number for commonPropertiesId.

Regards,

Brian

On Apr 20, 2006, at 6:52 AM, Paul Aitken wrote:

> drafts should specify the more specific use of the flowId element  
> as a commonPropertiesId.
>
> -- 
> Paul Aitken

---

From: Juergen Quittek <[email-masked]>
Date: Thu, 20 Apr 2006 17:29:47 +0200

OK, let's keep the flowId.

But them please check again its specification below.
The size of the commonPropertiesId has a size of 64 bit
Is 32 bit fine for the flowId?  Why would we need less here?
Is the description still OK?

Thanks,

    Juergen

5.1.7.  flowId

   Description:

5.1.8.  commonPropertiesId

   Description:
   ElementId: XXX
   Status: current

--On 20.04.2006 15:55 Uhr +0100 Andrew Johnson wrote:

> Brian Trammell wrote:
>> flowId is specifically bound to the definition of an IPFIX Flow, and may
>> be useful in certain situations. Off the top of my head I can think of
>> at least one: if a flow requires a record to describe it larger than the
>> IPFIX message MTU, the Flow could be described by two Flow Records
>> linked by flowId. commonPropertiesId would not be entirely appropriate
>> in this situation, as it does not carry the additional semantic "these
>> records describe the same Flow".
>>
>> reference an options record from multiple data records. It is completely
>> "virtual"; it's part of a mechanism, not a description of an object.
>>
>> own, that I would suggest keeping flowId as is, and assigning a new IE
>> number for commonPropertiesId.
>
> I agree with this.  I think we want to keep the flowId, although no-one
> has advertised that they are using it yet.
>
> On a slight tangent, from IPFIX-PROTO:
>   "All packets belonging to a particular Flow have a set of common properties."
>
> I still think commonPropertiesId is the right name for this field, but it
> should be made clear that the commonProperties are properties common between
> flows, usually a sub-set of the commonProperties which define a flow.
>
> Andrew
>
>
>>>> # 148 flowId.  Now that their draft has evolved towards reducing
>>> --Paul Aitken

---

From: Elisa Boschi <[email-masked]>
Date: Thu, 20 Apr 2006 17:49:21 +0200

> Is 32 bit fine for the flowId?  Why would we need less here?

I propose unsigned64 for both...

cheers,
Elisa

---

From: Andrew Johnson <[email-masked]>
Date: Thu, 20 Apr 2006 15:55:18 +0100

Andrew

---

From: Paul Aitken <[email-masked]>
Date: Thu, 20 Apr 2006 15:35:00 +0100

Brian,

> flowId and commonPropertiesId are two logically separate information  

Having thought about it some more, I agree.

> number for commonPropertiesId.

I agree.

---

From: Paul Aitken <[email-masked]>
Date: Thu, 20 Apr 2006 18:13:46 +0100

>I propose unsigned64 for both...

With reduced size encoding, it's better to specify the largest base type 
that will ever be required.

Implimentations are free to reduce the size from 64 bits if that's all 
they need.

---

From: Carter Bullard <[email-masked]>
Date: Sun, 23 Apr 2006 16:09:25 -0400

Argus has had a flow-id for 12 years now, so ..., I'd say its being  
used.
And it should be a separate entity.  Argus uses it for correlation when
a specific flow activity is reported in multiple output records.

Understand that the entire concept of flow reporting is that the flow- 
key
identifies the set of common properties, but, it seems attractive to  
think
that there other non-key related common properties.   We've wrestled
with this concept in argus for about 10 years, encapsulation encodings,
common payload characteristics, etc ....  and pretty much abandoned
the idea, as it was nebulous and confusing.  Without specific scoping
rules, it doesn't make any sense.

Carter

On Apr 20, 2006, at 10:55 AM, Andrew Johnson wrote:

> Brian Trammell wrote:
>> Paul and Juergen,
>> flowId and commonPropertiesId are two logically separate  
>> information elements.
>> flowId is specifically bound to the definition of an IPFIX Flow,  
>> and may be useful in certain situations. Off the top of my head I  
>> can think of at least one: if a flow requires a record to describe  
>> it larger than the IPFIX message MTU, the Flow could be described  
>> by two Flow Records linked by flowId. commonPropertiesId would not  
>> be entirely appropriate in this situation, as it does not carry  
>> the additional semantic "these records describe the same Flow".
>> commonPropertiesId is, on the other hand, a generic IE useful in  
>> many more situations; indeed, any situation where one needs a key  
>> to reference an options record from multiple data records. It is  
>> completely "virtual"; it's part of a mechanism, not a description  
>> of an object.
>> They are different enough in meaning, and each useful enough on  
>> their own, that I would suggest keeping flowId as is, and  
>> assigning a new IE number for commonPropertiesId.
>
>>> Seems to me that commonPropertiesId is just one use of flowId,  
>>> and there might be other uses - other situations where it's  
>>> useful for each flow to have a unique ID.
>>>

Carter Bullard
CEO/President
QoSient, LLC
150 E. 57th Street Suite 12D
New York, New York 10022

+1 212 588-9133 Phone
+1 212 588-9134 Fax

---

From: Juergen Quittek <[email-masked]>
Date: Mon, 24 Apr 2006 07:34:29 +0200

Hi Carter,

Thank you for your comments.

The question to be solved is not whether or not we need
something like a flow ID, the question was whether or not
we need two of them:
  - the IE we call flowID and
  - the new more general IE that we call commonPropertiesId.

Do you have any advice on this issue?

Thanks,

    Juergen

--On 23.04.2006 16:09 Uhr -0400 Carter Bullard wrote:

> Argus has had a flow-id for 12 years now, so ..., I'd say its being  used.
> Understand that the entire concept of flow reporting is that the flow- key
> identifies the set of common properties, but, it seems attractive to  think

---

From: Carter Bullard <[email-masked]>
Date: Mon, 24 Apr 2006 11:38:18 -0400

Well, I did.   There is an important need for a flow id, but there is
very limited use for a commonPropertiesId.    Its an attractive
concept, and one that works great, when scoping rules are very
strict, say in a persistent data file, but, in all, when building a
compression algorithm for an infinite stream of data, this type of
technique is unworkable and unscalable, or rather doesn't
compress very well (in some cases increases the number of
bytes).

There are a large number of scenarios where any combination of
objects that you would want to compress become huge and
unmanageable, such as compressing IPv6 addresses.   Obviously
there are more IPv6 addresses than commonPropertiesIds and template
ids combined, so what do you do when some one scans your IPv6 address
space and they've sent the 2^64+1 IPv6 address, in say less than 120
seconds?   Well you'll have to invalidate the commonPropertiesIds,  
somehow,
or just not compress any more, and of course, if the commonPropertiesIds
are all used up with single packet object identifiers, you'll make  
the system
much worse than better.  In order to make this scheme work in all  
situations
you have to build a very complex mechanism.

But the biggest issue is that this is a data compression strategy, and
the literature has many many many papers on what the issues are
in compressing infinite streams.   This type of strategy (persistent  
token
compression) has its problems in processing complexity, token
identification and scoping and performance when dealing with infinite
streams and the input is arbitrary.   Hard to say that the input is not
arbitrary.

Carter

Carter Bullard
CEO/President
QoSient, LLC

+1 212 588-9133 Phone
+1 212 588-9134 Fax

---

From: Brian Trammell <[email-masked]>
Date: Mon, 24 Apr 2006 15:23:15 -0400

Carter,

Agreed fully on flowId - I'd missed the Argus-like case where each  
Flow may be represented by multiple records emitted to meet reporting  
delay constraints with delta counters, where flowId can be used to  
assist re-aggregation at the collector side.

The commonPropertiesId IE (herein cpId) evolves very clearly from the  
scope mechanism defined by the IPFIX protocol. Each of the existing  
scope identifiers is bound to a specific data model or architecture  
object; cpId simply provides a method for implementing one-to-many  
relationships among records without overloading the usage of one of  
the "bound" scopes; specifically, we as a working group seem  
frequently to be tempted to abuse templateId this way...

The various complications regarding scope lifetime are not unique to  
cpId; indeed, they _already exist_ in the protocol, with templates,  
with observation domains, with flowIds, with identifiers for the  
various processes within the architecture, and so on. Collectors and  
exporters are already required to manage scopes (as an aside, Andrew  
Johnson and I are presently working on a draft to address this  
complexity in scope management).

Yes, cpId can be used as a record-level compression mechanism, and  
yes, as such it's not especially efficient with respect to degree of  
compression. The issue with compression is that stream compression  
would break the separability of records within sets, sets within  
messages, and so on; and in general, efficient block compression  
requires blocks larger than the MTU we see in packet- or segment- 
sized IPFIX messages. When using cpId to implement compression, one  
makes an explicit tradeoff between efficiency and separability -- not  
to mention, that a message stream using cpId is IPFIX-compliant,  
while there is no support for general-purpose compression of message  
streams in the present revision of the protocol.

cpId can also be used as a general purpose linking mechanism, which  
may be useful in situations when both the collector and exporter have  
multi-table internal data models.

Were the introduction of cpId to introduce new scope management  
problems, I'd be wary of its inclusion in the information model. But  
the complexity is already there; we might as well make full use of  
what we're paying for.

Regards,

Brian

On Apr 24, 2006, at 11:38 AM, Carter Bullard wrote:

> in compressing infinite streams.   This type of strategy  
> (persistent token
>>>> I agree with this.  I think we want to keep the flowId, although  
>>>> no-
>>>> one