Subject: [ipfix-protocol] PROTO-48: sequence number improvement - any objection?

From: Benoit Claise <[email-masked]>
Date: Mon, 14 Feb 2005 17:35:09 +0100

Dear all,

Regarding this open issue in the IPFIX protocol draft, I'm wondering if 
someone objects this proposal.

PROTO-48: Should the "sequence number" be improved to contains the 
number of flow records, like proposed in 
http://ipfix.doit.wisc.edu/archive/2587.html?

If someone does, this is the right time to discuss it!
Otherwise, it will change in the next version of the draft.

Regards, Benoit.

---

From: Robert Lowe <[email-masked]>
Date: Tue, 15 Feb 2005 11:51:11 -0600

Benoit Claise wrote:

Benoit,

> Otherwise, it will change in the next version of the draft.

The idea is nice, but what does it really do?  If an IPFIX message
contains sets other than data sets (some of which may be periodically
resent, and thus not permanently lost), then what do you know about how
much has been permanently lost?  I assume this is only for detection
and notification that data has really been lost.  Or are you suggesting
that since the ratio of non-data sets to data sets is so low as to make
that distinction unimportant, and this is just a simple measure of
congestion and/or reliability of the transport?

-Robert

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Benoit Claise <[email-masked]>
Date: Thu, 17 Feb 2005 21:41:05 +0100

Robert,

> Benoit Claise wrote:
>
> Benoit,
>
>> Regarding this open issue in the IPFIX protocol draft, I'm wondering 
>> if someone objects this proposal.
>>

First of all, let's say that this improvement is mainly for SCTP, the MUST implement transport protocol
Let's however analyze the 3 transport protocols.

1. In case of SCTP, we have in the protocol draft:

    An Exporting Process MUST request at least two outbound streams per
    association. The first stream (referred to as stream zero in the
    rest of this document), is used to send the Template Set and the
    Options Template Set. Stream zero MUST be fully reliable. Data
    Sets MUST NOT be sent on stream zero.

So we don't have the problem of mixed non-data sets and data sets with SCTP, and retransmission of non-data sets.
Creating multiple streams, one for each template ID, would return the number of records per template. 

BTW, "loss of flow records during the data transfer" is requirement from RFC 3917:

    6.3.2. Reliability

    Loss of flow records during the data transfer from the exporting
    process to the collecting process must be indicated at the collecting
    process. This indication must allow the collecting process to gauge
    the number of flow records lost.  

2. TCP is reliable protocol and will keep retransmitting. 
So, even if we have the issue of "mixed non-data sets and data sets", this is not important!

3. UDP. I think that your comments were mainly about UDP, right?
As you observed (as specified in section 13.3.6 of[IPFIX-PROTO], we might have the issue of 
"mixed non-data sets and data sets" in the same IPFIX Message.
Basically, there are 2 different cases:
3.1 We loose one IPFIX Message with "mixed non-data sets and data sets", then the collectors knows 
from the gap in data records sequence that there is something wrong (at least one IPFIX Message lost).
You are right: "since the ratio of non-data sets to data sets is so low as to make
that distinction unimportant, and this is just a simple measure of congestion and/or reliability of the transport"
3.2 We loose all the IPFIX Messages that contain the template and options template Set, by a strange coincidence (a kind of payload inspection access-list).
This is the only serious case where I see a potential drawback compared to the current situation. 

However, in both case 3.1 and 3.2, the collector might still monitor the condition of not refreshed templates.
>From [IPFIX-PROTO]:

    Templates not refreshed by the Exporting Process within the timeout
    are expired at
    the Collecting Process. If the template is not refreshed by the
    Exporting Process before that lifetime has expired, the Collecting
    Process MUST discard the Template, and any current and future
    associated Flow or Option Data Records.

We could complete the sentence "and an alarm MAY be logged"

Regards, Benoit.

---

From: Robert Lowe <[email-masked]>
Date: Fri, 18 Feb 2005 15:56:00 -0600

Benoit Claise wrote:

Hi Benoit,

> Creating multiple streams, one for each template ID, would return the number of records per template. 

Yes, and the collecting process would know not only that something is missing,
but of what type.

>     the number of flow records lost.  

Perhaps it's a bit unfortunate that the definition of "flow record" is not
completely precise in RFC3917, but I infer from the context of [IPFIX-PROTO]
that it really means flow data record (or perhaps includes options data
record, thereby covering the possibilities for a data set).  This now means,
in the strictest sense, that this requirement cannot be met with UDP, unless
of course, gauge means only an estimate.  ;-)  Perhaps a caveat is in order?

> 3. UDP. I think that your comments were mainly about UDP, right?

Sure, but don't take that to mean anything other than a quest for
accuracy in the document.

> We could complete the sentence "and an alarm MAY be logged"

I would agree that something should be added (perhaps stronger
than MAY) if all that's necessary for an attacker to force the
discarding of flow [data] records is to inhibit the  arrival of
the template set.

-Robert

---

From: Stewart Bryant <[email-masked]>
Date: Fri, 18 Feb 2005 23:47:13 +0000

Robert Lowe wrote:

>> So we don't have the problem of mixed non-data sets and data sets with 
>> SCTP, and retransmission of non-data sets.

With SCTP the templates are sent over the a reliable channel so they
will not get lost, or at least if they do, there will be a transport
reset to tell you.

> of course, gauge means only an estimate.  ;-)  Perhaps a caveat is in 

Sure, but in the UDP specific section, as this is a UDP specific issue.

> 
>> 2. TCP is reliable protocol and will keep retransmitting. So, even if 
>> we have the issue of "mixed non-data sets and data sets", this is not 
>> important!
>>

> 
>> As you observed (as specified in section 13.3.6 of[IPFIX-PROTO], we 
>> might have the issue of "mixed non-data sets and data sets" in the 
>> same IPFIX Message.
>> 3.1 We loose one IPFIX Message with "mixed non-data sets and data 
>> sets", then the collectors knows from the gap in data records sequence 
>> that there is something wrong (at least one IPFIX Message lost).
>> congestion and/or reliability of the transport"
>> 3.2 We loose all the IPFIX Messages that contain the template and 
>> options template Set, by a strange coincidence (a kind of payload 
>> inspection access-list).
>> However, in both case 3.1 and 3.2, the collector might still monitor 
>> the condition of not refreshed templates.
>>
>>> From [IPFIX-PROTO]:
>>
>>

MUST sounds OK.

Stewart

---

From: Robert Lowe <[email-masked]>
Date: Fri, 18 Feb 2005 18:27:36 -0600

Stewart Bryant wrote:

> 
> 
> Robert Lowe wrote:
> 
>>
>>
>> Benoit Claise wrote:
>>
>> Hi Benoit,
>>
>>>>> Regarding this open issue in the IPFIX protocol draft, I'm 
>>>>> wondering if someone objects this proposal.
>>>>>
>>> So we don't have the problem of mixed non-data sets and data sets 
>>> with SCTP, and retransmission of non-data sets.

Yes, but as an added advantage, the sequence number for the non-reliable
streams, which are for data sets, takes on more meaning, i.e. N flow
data records were lost vs. N records were lost, some of which may have
been flow data records.  This is good.  :-)

> Sure, but in the UDP specific section, as this is a UDP specific issue.

Perfect.

>>> sets", then the collectors knows from the gap in data records 
>>> sequence that there is something wrong (at least one IPFIX Message 
>>> lost).

Fine by me.

-Robert