Subject: Re: [IPFIX] Implementation Guidelines // SCTP

From: Benoit Claise <[email-masked]>
Date: Wed, 23 May 2007 11:51:47 +0200

Gerhard, all,

What you're proposing is something that Cisco has been thinking about 
for more than 6 months now.
Actually we filled in a patent on that one. Because you proposed the 
idea, we worked on the creation of a new draft
http://www.ietf.org/internet-drafts/draft-claise-ipfix-export-per-sctp-stream-00.txt 
fully explaining the idea
We also posted an IPR: see 
https://datatracker.ietf.org/public/ipr_list.cgi.
The title of the IPR disclosure is "Cisco's Statement about IPR claimed 
in draft-claise-ipfix-export-per-sctp-stream-00.txt."
Alternatively, directly look up 
http://www1.ietf.org/ietf/IPR/cisco-ipr-draft-claise-ipfix-export-per-sctp-stream-00.txt

I didn't want to answer this email before the draft and the IPR were 
posted.

Note that a new version of this draft will follow, with the 
functionality to add streams in SCTP, when available in the SCTP reset 
draft 
(http://www.ietf.org/internet-drafts/draft-stewart-sctpstrrst-04.txt).
Once done, my personal view is that it would be a good candidate for the 
new charter discussion in Chicago

Regards, Benoit.
> Hi Elisa, Benoit,
>
> Boschi, Elisa wrote:
>   
>>> 6.1:
>>> The citation from protocol-24 has changed !!! Now it is:
>>>
>>>    Template Sets and Option Template Sets SHOULD be sent on the SCTP
>>>    stream zero. Template Sets and Option Template Sets MUST be sent
>>>    reliably.
>>>
>>> I propose to emphasize more on this important implementation aspect since
>>> the reliable transport of Templates is one of the main advantages of SCTP!
>>> There is a side-effect:
>>>    "A simple system might use just one data stream for all the exported
>>>    data."
>>> This implies that the all IPFIX messages are sent over one reliable
>>>       
>> stream!
>>     
>> 6.1 has changed quite a bit and the new version should address your
>> concerns... more about this in the next days
>>     
>
> What is the reason for separating Templates from the corresponding Data
> Sets and for sending them over a separate stream (zero)?
>
> I've been talking to Michael Tüxen, who was working on PR-SCTP. It seems
> that the sentence
>
>     Template Sets and Option Template Sets SHOULD be sent on the SCTP
>     stream zero.
>
> from the protocol draft is a relict from very early days of PR-SCTP when
> reliability options were assumed to be a per-stream setting. Actually,
> it is a per-message property, i.e. in the same stream, one message can
> be sent fully reliable and another one partially reliable. The effect of
> a stream is that the message order is preserved and that head-of-line
> blocking at the receiver does not exist between different streams.
>
> Therefore, it seems to be more appropriate to have a separate stream for
> each Template, and to send the Template as well as the corresponding
> Data Sets over this stream. This ensures that the Template arrives
> before the Data Sets. Templates would be sent in reliable messages, Data
> Sets in partially reliable messages.
>
> Am I missing the point here?
>
> Regards,
> Gerhard
>
>   
> ------------------------------------------------------------------------
>
> _______________________________________________
> IPFIX mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/ipfix
>   

_______________________________________________
IPFIX mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipfix

---

From: Benoit Claise <[email-masked]>
Date: Wed, 23 May 2007 11:51:47 +0200

Gerhard, all,

Regards, Benoit.

---

From: [email-masked]
Date: Wed, 23 May 2007 10:07:45 +0000

Sounds like using TCP isn't such a bad idea after all!!!!!!!

Carter

Carter Bullard
QoSient LLC
150 E. 57th Street Suite 12D
New York, New York 10022
+1 212 588-9133 Phone
+1 212 588-9134 Fax  

-----Original Message-----
From: Benoit Claise <[email-masked]>
Date: Wed, 23 May 2007 11:51:47 
To:Gerhard Muenz <[email-masked]>
Cc:"Peter Lei \(peterlei\)" <[email-masked]>, [email-masked]
Subject: Re: [IPFIX] Implementation Guidelines // SCTP

Gerhard, all,
 
 What you're proposing is something that Cisco has been thinking about for more than 6 months now.
 Actually we filled in a patent on that one. Because you proposed the idea, we worked on the creation of a new draft
 http://www.ietf.org/internet-drafts/draft-claise-ipfix-export-per-sctp-stream-00.txt: <http://www.ietf.org/internet-drafts/draft-claise-ipfix-export-per-sctp-stream-00.txt> fully explaining the idea
 We also posted an IPR: see https://datatracker.ietf.org/public/ipr_list.cgi: <https://datatracker.ietf.org/public/ipr_list.cgi> . 
 The title of the IPR disclosure is "Cisco's Statement about IPR claimed in draft-claise-ipfix-export-per-sctp-stream-00.txt."
 Alternatively, directly look up http://www1.ietf.org/ietf/IPR/cisco-ipr-draft-claise-ipfix-export-per-sctp-stream-00.txt: <http://www1.ietf.org/ietf/IPR/cisco-ipr-draft-claise-ipfix-export-per-sctp-stream-00.txt> 
 
 I didn't want to answer this email before the draft and the IPR were posted. 
 
 Note that a new version of this draft will follow, with the functionality to add streams in SCTP, when available in the SCTP reset draft (http://www.ietf.org/internet-drafts/draft-stewart-sctpstrrst-04.txt: <http://www.ietf.org/internet-drafts/draft-stewart-sctpstrrst-04.txt> ). 
 Once done, my personal view is that it would be a good candidate for the new charter discussion in Chicago
 
 Regards, Benoit.
 Hi Elisa, Benoit, Boschi, Elisa wrote: 6.1: The citation from protocol-24 has changed !!! Now it is: Template Sets and Option Template Sets SHOULD be sent on the SCTP stream zero. Template Sets and Option Template Sets MUST be sent reliably. I propose to emphasize more on this important implementation aspect since the reliable transport of Templates is one of the main advantages of SCTP! There is a side-effect: "A simple system might use just one data stream for all the exported data." This implies that the all IPFIX messages are sent over one reliable stream! 6.1 has changed quite a bit and the new version should address your concerns... more about this in the next days What is the reason for separating Templates from the corresponding Data Sets and for sending them over a separate stream (zero)? I've been talking to Michael Tüxen, who was working on PR-SCTP. It seems that the sentence Template Sets and Option Template Sets SHOULD be sent on the SCTP stream zero. from the protocol draft is a relict from very early days of PR-SCTP when reliability options were assumed to be a per-stream setting. Actually, it is a per-message property, i.e. in the same stream, one message can be sent fully reliable and another one partially reliable. The effect of a stream is that the message order is preserved and that head-of-line blocking at the receiver does not exist between different streams. Therefore, it seems to be more appropriate to have a separate stream for each Template, and to send the Template as well as the corresponding Data Sets over this stream. This ensures that the Template arrives before the Data Sets. Templates would be sent in reliable messages, Data Sets in partially reliable messages. Am I missing the point here? Regards, Gerhard 
----------------
 _______________________________________________ IPFIX mailing list [email-masked]: <mailto:[email-masked]> https://www1.ietf.org/mailman/listinfo/ipfix: <https://www1.ietf.org/mailman/listinfo/ipfix> 
 _______________________________________________
IPFIX mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipfix

---

From: [email-masked]
Date: Wed, 23 May 2007 10:07:45 +0000

Carter

Carter Bullard
QoSient LLC

-----Original Message-----
From: Benoit Claise <[email-masked]>
Date: Wed, 23 May 2007 11:51:47 

Gerhard, all,

 
 
 Regards, Benoit.

---

From: Jeroen Massar <[email-masked]>
Date: Wed, 23 May 2007 13:19:29 +0100

[..]
>  Alternatively, directly look up http://www1.ietf.org/ietf/IPR/cisco-ipr-draft-claise-ipfix-export-per-sctp-stream-00.txt: <http://www1.ietf.org/ietf/IPR/cisco-ipr-draft-claise-ipfix-export-per-sct…

(The above long formatted lines are not mine, 72 is a nice limit FYI)

Sorry to be blunt, but what exactly is the point again of 'opening up
NetFlow' if there is going to be IPR stuff being smacked upon IPFIX and
thus encumbering it?

And no the "we don't sue when you don't sue" sillyness is not 'open'.

If there is IPR on this part of IPFIX then it MUST never become part of
the standard of IPFIX that is to be the IETF version of NetFlow.

The "Disclosure"
http://www1.ietf.org/ietf/IPR/cisco-ipr-draft-claise-ipfix-export-per-sctp-stream-00.txt
doesn't contain *ANY* details on what exactly is IPR'd and what not. So
do we just have to assume that everything in that draft is encumbered?

I also have to note that the existing IPFIX capable collectors actually
can't care so much about what exactly gets delivered and how and in
which stream exactly the IPFIX packets are being sent, I am very sure
that the collectors that already exist are automatically prior art to
what is described in the draft document, therefor nullifying at least
half of the draft.

I am sincerely wondering _why_ Cisco is first going the long route of
trying to get an IETF, and thus widely open accepted, standard form of
NetFlow, thus making all kinds of vendors, who are still reluctant to
use NetFlow because of IPR nonsense, happy and then suddenly, after a
long time of getting people to accept it as a standard means, still yet
again go the IPR route to go back to square zero: no vendor will accept
IPFIX, and there will be various different kinds of "NetFlow" for doing
traffic, and other kinds of, metering.

Please, please, please, come up with something which actually is
original, get a patent for that and make your lawyers happy sueing each
other over those things. Don't go back to encumber something that is
actually starting to get adopted by a lot of vendors.

Greets,
 Jeroen

 (before anybody screams murder, of course like anything IETFish
  should be, this is my private & personal opinion bla bla...)

---

From: Jeroen Massar <[email-masked]>
Date: Wed, 23 May 2007 13:19:29 +0100

[..]

The "Disclosure"

Greets,
 Jeroen

---

From: Gerhard Muenz <[email-masked]>
Date: Wed, 23 May 2007 16:49:18 +0200

Benoit, Elisa, all,

Apparently, my assumption was right:
There are no good reasons at all for sending all Templates over the same
stream 0, as specified in ipfix-protocol.

I made this comment about the proposed usage of SCTP for IPFIX exactly
two months ago. So thank you for the late answer. Yet, my comment
concretely refered to the protocol draft and the implementation
guidelines, and I do not understand why we need to postpone this issue
to future WG activities.

What I've heard, few to no people actually implemented IPFIX over
PR-SCTP using multiple streams. At least, this was not tested at the
interop, was it? This may explain why nobody recognized the uselessness
of this detail in the IPFIX protocol specs earlier.

Instead of standardizing something useless now and waiting for an
improvement later, I suggest the following quite small changes:

1. Remove the stream 0 related parts from the protocol draft, thus
avoiding to specify something useless or confusing.

2. Incorporate draft-claise-ipfix-export-per-sctp-stream-00 as
recommendations into the implementation guidelines.

Regards,
Gerhard

Benoit Claise wrote:
>> [email-masked] <mailto:[email-masked]>

-- 
Dipl.-Ing. Gerhard Münz
Computer Networks and Internet
Wilhelm Schickard Institute for Computer Science
University of Tuebingen
Sand 13 (Room B309), D-72076 Tuebingen, Germany
Phone:  +49 7071 29-70534       Fax: +49 7071 29-5220
E-mail: [email-masked]
WWW:    http://net.informatik.uni-tuebingen.de/~muenz

---

From: Gerhard Muenz <[email-masked]>
Date: Wed, 23 May 2007 16:49:18 +0200

Benoit, Elisa, all,

Regards,
Gerhard

---

From: Benoit Claise <[email-masked]>
Date: Thu, 24 May 2007 11:07:24 +0200

Gerhard,
> stream 0, as specified in ipfix-protocol.
You got it right in your initial email: "is a relict from very early 
days of PR-SCTP when reliability options were assumed to be a per-stream 
setting"
> PR-SCTP using multiple streams. 
Using multiple separate streams for the data... This is my understanding
> At least, this was not tested at the
> interop, was it? 
This is also my understanding.
> This may explain why nobody recognized the uselessness
Yep.
> avoiding to specify something useless or confusing.
The latest draft mentions "Template Sets and Option Template Sets SHOULD 
be sent on the SCTP stream zero"
It was a MUST in [IPFIX-PROTO] in the past, and was changed.
 From RFC2119

    3. SHOULD   This word, or the adjective "RECOMMENDED", mean that there
       may exist valid reasons in particular circumstances to ignore a
       particular item, but the full implications must be understood and
       carefully weighed before choosing a different course.

So it is really a big problem?

> recommendations into the implementation guidelines.
IMHO, there is still one piece missing in the picture.
Sending one template per stream is useful but how do you know how many 
streams you need when you start up the SCTP association?
What if you need more streams, and don't want to shut down the SCTP 
association to increase the number?
This is what I meant by "Note that a new version of this draft will 
follow, with the functionality to add streams in SCTP, when available in 
the SCTP reset draft"

Regards, Benoit.

---

From: Benoit Claise <[email-masked]>
Date: Thu, 24 May 2007 11:07:24 +0200

Gerhard,
Yep.

Regards, Benoit.

---

From: Eliot Lear <[email-masked]>
Date: Thu, 24 May 2007 14:42:40 +0200

[Not speaking in any way for Cisco on this issue...]

Simon Josefsson wrote:
>
> Such clauses may be acceptable when we know what the patent is, and what
> it covers, but this refer to unpublished patent applications.  This
> particular license also explicitly enable Cisco to collect retroactive
> royalties against anyone who sues Cisco over _any_ patent.
>   
> I don't believe these are acceptable terms for a Standards Track
> document.
>   

Just imagine the number of lawsuits that would exist if everyone did 
things this way.  Stallman should be jealous.

---

From: Eliot Lear <[email-masked]>
Date: Thu, 24 May 2007 14:42:40 +0200

---

From: Brian Trammell <[email-masked]>
Date: Fri, 25 May 2007 11:21:43 -0400

Benoit, all,

I have reviewed draft-claise-ipfix-export-per-sctp-stream-00 (herein
per-stream), and have the following comments. I'll restrict myself to
the content of the document itself, and the implications of this
approach to the IPFIX protocol and the future work of the IPFIX Working
Group.

First, in general, it seems to me like the per-stream approach
(restricting each template to a single stream) has a single IPFIX
deployment pattern in mind: multiple "applications" running over one or
more Observation Domains on a single Exporting Process, such that each
"application" uses a single template. I'm inferring from the importance
placed, for instance, on per-template loss rates that templates are used
as a proxy for "applications". If this is the case, the approach does
handle that deployment pattern quite well.

However, there exist other possible (and indeed, actual) deployment
patterns. The one example with which I'm most familiar is our own YAF
flow meter (http://tools.netsa.cert.org/yaf) which implements a single
application in a single Observation Domain on a single Exporting Process
and uses multiple templates for export efficiency. The per-stream
approach is not at all appropriate for this deployment pattern; opening
one stream for UDP uniflows and one for TCP biflows doesn't buy us much,
and we don't really care about per-template loss rates in this case.

I'm concerned that per-stream updates the protocol document before the
protocol's even out of the RFC Editor queue, and does so in an
incompatible way, additionally contradicting the implementation
guidelines (which have not yet even been submitted to the IESG), AND
does so in a way that supports only a single deployment pattern.
Adopting this document on the standards track such that it 1. fixes
stream selection rules and 2. mandates a particular deployment pattern
can only serve to confuse IPFIX implementors, potentially leading them
to believe that this is the only deployment pattern supported by the
IPFIX protocol, which will severely limit the protocol's adoption.

I agree that the stream selection rules in the IPFIX Protocol are
currently broken. Indeed, I and many others attempted to change them,
but the attempt was too late in the process to be effective (a MUST
became a SHOULD, but even that is in my opinion too strong). It does not
appear that we as a working group or a community really understand the
best way to use SCTP streams. Per-stream is one way, that would appear
to work very well in certain circumstances. I'm sure there are others.
If we're going to update the protocol, instead of replacing one set of
restrictions with another, why not relax the restrictions to allow
interoperable experimentation with stream selection rules?

We could easily adopt a short errata draft, much like RFC 3309 which
corrected a similar issue in the SCTP protocol itself, that relaxes all
the stream selection rules, allowing an Exporting Process to export
Templates and Data on any streams as appropriate for its own application
requirements. Elisa and I have written such a draft, which has been
submitted as draft-trammell-ipfix-sctp-change-00 and will be available
shortly. The draft is quite brief, four and a half pages exclusive of
boilerplate, and could be adopted and submitted to correct the stream
selection problem rapidly.

Per-stream could then be expressed in terms of the new unrestrictive
rules, as a more-restrictive specialization of the stream selection
rules useful in the specific deployment pattern which it addresses. As a
working group, we can then consider the per-stream draft solely on the
technical merits of the approach, separate from the stream selection
problem.

Regards,

Brian

---

From: Brian Trammell <[email-masked]>
Date: Fri, 25 May 2007 11:21:43 -0400

Benoit, all,

Regards,

Brian

---

From: Gerhard Muenz <[email-masked]>
Date: Fri, 25 May 2007 23:33:52 +0200

Brian,

Thank you for your effort. Your suggestion is completely inline with
what I wished in my earlier mail :)
http://www1.ietf.org/mail-archive/web/ipfix/current/msg03976.html

"1. Remove the stream 0 related parts from the protocol draft, thus
avoiding to specify something useless or confusing."

Which means, remove all stream-related restrictions and recommendations
in the protocol specs, and then maybe mention some hints about different
possibilities how to use streams in the implementations guidelines (or
separate drafts, if you do not want to have it in there).

Have a nice weekend,
Gerhard

---

From: Gerhard Muenz <[email-masked]>
Date: Fri, 25 May 2007 23:33:52 +0200

Brian,

Have a nice weekend,
Gerhard

---

From: Andrew Johnson <[email-masked]>
Date: Wed, 30 May 2007 13:51:48 +0100

Hi Brian

The per-stream draft specifies one possible implementation of the rules
given in the protocol.  While it does go against some of the
recommendations (SHOULDs), I wouldn't say it was incompatible with the
protocol as it stands today and is not intended to update the protocol.

The intended deployment is, as you say, for multiple applications using
a single exporting process.  If the collector knows that the per-stream
rules are being used then it can infer additional information when
packets are lost, as well as benefit from the template/data ordering.

If your deployment pattern doesn't benefit from these additional rules,
then you are free to be compliant with the protocol, but not use the
per-stream rules.

regards

Andrew

---

From: Andrew Johnson <[email-masked]>
Date: Wed, 30 May 2007 13:51:48 +0100

Hi Brian

regards

Andrew

---

From: Benoit Claise <[email-masked]>
Date: Wed, 30 May 2007 15:51:02 +0200

Brian, Andrew,

I agree with Andrew. I see the per-sctp-stream draft an en extension to 
the current IPFIX protocol.
Note that extension will fully make sense with the possible addition of 
streams within an existing SCTP association.

Now, I understand Brian/Gehrard 's suggestions to simplify the IPFIX 
protocol, and I'm not opposed to it.
Nevertheless, I'm concerned about this last minute quite significant 
change. Have we spent enough time thinking about the consequences: 
Template Withdraw Messages, Options Template, etc... By giving the full 
flexibility, is there enough guidelines for the different cases?

My view is that a RFC with errata is confusing. But, if the protocol is 
changed, it will delay further the RFC publication.
Note: the IPFIX WG has taken too long to produce RFCs (all the 
references amongst IPFIX and PSAMP documents is one of the factor) This 
is good in the sense we could have multiple interops to test the 
protocol, but this leads into this complex situation where we always to 
improve the documents. If we had more classical WG ;), I guess that we 
would have the Proposed Standard by now, and that those changes would go 
in the Draft Standard

Note: even the protocol is changed, I think the per-stream draft still 
makes sense, as Andrew wrote:

    If the collector knows that the per-stream
    rules are being used then it can infer additional information when
    packets are lost, as well as benefit from the template/data ordering

Regards, Benoit.

---

From: Benoit Claise <[email-masked]>
Date: Wed, 30 May 2007 15:51:02 +0200

Brian, Andrew,

Regards, Benoit.

---

From: "Gerhard Muenz" <[email-masked]>
Date: Wed, 30 May 2007 21:49:52 +0200 (CEST)

Dear all,

As I see we reached a consensus on what we want, and the current
discussion is more about how this can be achieved :)

Just one more argument in favor of a change of the protocol which shows
that it is not only about a "SHOULD". The current draft/upcoming RFC says:

   An Exporting Process MUST request at least two outbound streams per
   association.  The first stream (referred to as stream zero in the
   rest of this document), is used to send the Template Set and the
   Options Template Set.  Data Sets MUST NOT be sent on stream zero.

If you want to use streams as specified in
draft-claise-ipfix-export-per-sctp-stream-00, you still must open a stream
zero that you probably won't use at all.

Regards,
Gerhard

On Wed, May 30, 2007 15:51, Benoit Claise said:
>>>>> I've been talking to Michael TÃ¼xen, who was working on PR-SCTP. It
>>>>> seems
>>>>> that the sentence
>>>>>

---

From: "Gerhard Muenz" <[email-masked]>
Date: Wed, 30 May 2007 21:49:52 +0200 (CEST)

Dear all,

Regards,
Gerhard

---

From: Andrew Johnson <[email-masked]>
Date: Wed, 30 May 2007 21:30:28 +0100

On Wed, 2007-05-30 at 21:49 +0200, Gerhard Muenz wrote:
> that it is not only about a "SHOULD".

What do you mean?  The protocol has a number of requirements that I
think we're agreed aren't strictly necessary.

>  The current draft/upcoming RFC says:
> 

Indeed, per-stream only uses stream 0 for the withdraw all templates
message.  If you run out of streams we specify stream 1 as a default for
all other template/data sets and to use stream >= 2 for each
application.

Ideally we'd use stream 0 as the default, but as you point out, the
protocol says that we can't send data sets on stream 0.

Cheers

Andrew

---

From: Andrew Johnson <[email-masked]>
Date: Wed, 30 May 2007 21:30:28 +0100

Cheers

Andrew