Subject: [IPFIX] RFC5101: time first flow dropped and time last flow dropped

From: Benoit Claise <[email-masked]>
Date: Tue, 20 Sep 2011 16:19:24 +0200

Dear all,

[this email addresses draft-claise-ipfix-protocol-rfc5101bis-00.txt,  TO 
DO -> "time first flow dropped" and "time last flow dropped" 
inconsistency.  See the discussion on the list.]

Paul Aitken discovered this issue.,

 From [RFC5101], section 4.3. The Exporting Process Reliability 
Statistics Option Template

time first flow dropped
                         The timestamp of the first _Flow _was dropped by
                         the Metering Process.  For this timestamp, any
                         of the "flowStart" timestamp Information
                         Elements flowStartMilliseconds,
                         flowStartMicroseconds, flowStartNanoseconds, and
                         flowStartDeltaMicroseconds can be used.

time last flow dropped
                         The timestamp of the last _IP packet_ that was
                         ignored by the Metering Process.  For this
                         timestamp, any of the "flowEnd" timestamp
                         Information Elements flowEndMilliseconds,
                         flowEndMicroseconds, flowEndNanoseconds, and
                         flowEndDeltaMicroseconds can be used.

Firstly, these definitions are inconsistent since the names and the 
first definition say "flow" while the second definition says "IP 
packet". Obviously "IP packet" != "flow" :-o

Secondly, "The timestamp of the first Flow was dropped by the Metering 
Process." is bad English: at least it's missing "that".

Thirdly, the section 4.2 doesn't take into account a metering process 
id. Indeed, what if we have multiple caches on the exporter?

Here is a proposal for 4.2 and 4.3. The underlined parts are new

      4.2. The Metering Process Reliability Statistics Option Template

    The Metering Process Reliability Option Template specifies the
    structure of a Data Record for reporting lack of reliability in the
    Metering Process.  It SHOULD contain the following Information
    Elements that are defined in [RFC5102  <http://tools.ietf.org/html/rfc5102>]:

    observationDomainId
                            An identifier of an Observation Domain that
                            is locally unique to the Exporting Process.
                            This Information Element MUST be defined as a
                            Scope Field.

_    meteringProcessId (*)
                            The identifier of the Metering Process for
                            which lack of reliability is reported.  There
                            This Information Element MUST be defined as a
                            Scope Field._

    ignoredPacketTotalCount
                            The total number of IP packets that the
                            Metering Process did not process.

    ignoredOctetTotalCount
                            The total number of octets in observed IP
                            packets that the Metering Process did not
                            process.

    time first_packet_ignored
                            The timestamp of the first IP packet that was
                            ignored by the Metering Process.  For this
                            timestamp, any of the "flowStart" timestamp
                            Information Elements flowStartMilliseconds,
                            flowStartMicroseconds, flowStartNanoseconds,
                            and flowStartDeltaMicroseconds can be used.

    time last_packet_ignored
                            The timestamp of the last IP packet that was
                            ignored by the Metering Process.  For this
                            timestamp, any of the "flowEnd" timestamp
                            Information Elements flowEndMilliseconds,
                            flowEndMicroseconds, flowEndNanoseconds, and
                            flowEndDeltaMicroseconds can be used.

      4.3. The Exporting Process Reliability Statistics Option Template

    The Exporting Process Reliability Option Template specifies the
    structure of a Data Record for reporting lack of reliability in the
    Exporting process.  It SHOULD contain the following Information
    Elements that are defined in [RFC5102  <http://tools.ietf.org/html/rfc5102>]:

    Exporting Process ID
                         The identifier of the Exporting Process for
                         which lack of reliability is reported.  There
                         are three Information Elements specified in
                         [RFC5102  <http://tools.ietf.org/html/rfc5102>] that can be used for this purpose:
                         exporterIPv4Address, exporterIPv6Address, or
                         exportingProcessId.  This Information Element
                         MUST be defined as a Scope Field.

    notSentFlowTotalCount
                         The total number of Flows that were generated by
                         the Metering Process and dropped by the Metering
                         Process or by the Exporting Process instead of
                         being sent to the Collecting Process.

    notSentPacketTotalCount
                         The total number of packets in Flow Records that
                         were generated by the Metering Process and
                         dropped by the Metering Process or by the
                         Exporting Process instead of being sent to the
                         Collecting Process.

    notSentOctetTotalCount
                         The total number of octets in packets in Flow
                         Records that were generated by the Metering
                         Process and dropped by the Metering Process or
                         by the Exporting Process instead of being sent
                         to the Collecting Process.

    time first flow dropped
   _                       The time at which the first Flow Record was dropped by
                         the__Exporting Process._   For this timestamp, any
                         of the "flowStart" timestamp Information
                         Elements flowStartMilliseconds,
                         flowStartMicroseconds, flowStartNanoseconds, and
                         flowStartDeltaMicroseconds can be used.

    time last flow dropped
                         _The time at which the last Flow Record was
                         dropped by the Exporting Process_.  For this
                         timestamp, any of the "flowEnd" timestamp
                         Information Elements flowEndMilliseconds,
                         flowEndMicroseconds, flowEndNanoseconds, and
                         flowEndDeltaMicroseconds can be used.

    The Exporting Process SHOULD export the Data Record specified by the
    Exporting Process Reliability Statistics Option Template on a regular
    basis or based on some export policy.  This periodicity or export
    policy SHOULD be configurable.

(*) regarding the meteringProcessId, we were hesitating between the 
meteringProcessId and the cache id
1. there is a meteringProcessId in IPFIX IANA while there is no cache id
2. if the IPFIX exporter is configured from [IPFIX-CONF], what should be 
the value in meteringProcessId?
     [IPFIX-CONF] doesn't mention the meteringProcessId, but a cache name.
     From figure 1 and 2, it seems that there is a one to one matching 
between the cache and the metering process.
     If this is the case, a solution could be to add a meteringProcess 
inside the cache in the figure 12 in [IPFIX-CONF] below

      4.3. Cache Class

     +-----------------------------------+
     | Cache                             |
     +-----------------------------------+        1 +------------------+
     | name                              |<>--------| immediateCache/  |
     | dataRecords {readOnly}            |          | timeoutCache/    |
     | cacheDiscontinuityTime {readOnly} |          | naturalCache/    |
     |                                   |          | permanentCache   |
     |                                   |          +------------------+
     |                                   |
     |                                   |     0..* +------------------+
     |                                   |--------->| ExportingProcess |
     +-----------------------------------+          +------------------+

                           Figure 12: Cache class

What do you think? Do you see another way?

Regards, Paul & Benoit.

---

From: Andrew Feren <[email-masked]>
Date: Tue, 20 Sep 2011 15:54:58 -0400

On 09/20/2011 10:19 AM, Benoit Claise wrote:
> Dear all,
>
> [this email addresses draft-claise-ipfix-protocol-rfc5101bis-00.txt,  
> TO DO -> "time first flow dropped" and "time last flow dropped" 
[ snip ]
>                             This Information Element MUST be defined as a

There is an extra "There" there.  :-)

[ snip ]

> 2. if the IPFIX exporter is configured from [IPFIX-CONF], what should 
> be the value in meteringProcessId?

meteringProcessId makes sense to me since you are exporting "Metering 
Process Reliability Statistics".

If you want to send a cache id as meteringProcessId (and your cache ids 
are "unique per IPFIX Device") I don't see any harm.  At least as long 
as the 1:1 relationship between metering process and cache holds.  The 
IDs aren't meaningful to me as anything other than an identifier.

-Andrew

---

From: Paul Aitken <[email-masked]>
Date: Tue, 20 Sep 2011 21:32:21 +0100

Benoit,

>     [IPFIX-CONF] doesn't mention the meteringProcessId, but a cache name.

IPFIX-CONF (or any configurer) wouldn't set the process ID; it's read-only.

The configurer would create the process (technically, add the config 
which causes the process to be created by the OS), then could be told 
(or could discover) what the process ID is.
The cache name could be used to relate the given config to the process ID.

> inside the cache in the figure 12 in [IPFIX-CONF] below

Do you mean, a "meteringProcessId" (IANA #144) as a read-only property?

And also an "exportingProcessId" (IANA #145) as a read-only property in 
the ExportingProcess?

Those seem useful.

P.

---

From: Benoit Claise <[email-masked]>
Date: Tue, 20 Sep 2011 22:56:31 +0200

Andrew,

Thanks for your comments.
See in line.
> There is an extra "There" there.  :-)
Thanks
> If you want to send a cache id as meteringProcessId (and your cache 
> ids are "unique per IPFIX Device") I don't see any harm.  At least as 
> long as the 1:1 relationship between metering process and cache holds. 
That's something I was unable to double check in [IPFIX-CONF] or 
RFC5470, even if I believe it's a right assumption.
Gerhard?

Regards, Benoit.

> The IDs aren't meaningful to me as anything other than an identifier.
>
> -Andrew
>
>
>
> _______________________________________________
> IPFIX mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ipfix

---

From: Benoit Claise <[email-masked]>
Date: Tue, 20 Sep 2011 22:57:51 +0200

> The cache name could be used to relate the given config to the process 
Agreed.
> And also an "exportingProcessId" (IANA #145) as a read-only property 
> in the ExportingProcess?
That would solve the issue.

Regards, Benoit.

---

From: Gerhard Muenz <[email-masked]>
Date: Wed, 21 Sep 2011 21:22:35 +0200

Hi Benoit,

See comments inline.

>                             flowEndDeltaMicroseconds can be used.

I do not find these IEs in RFC5102.

>                          flowEndDeltaMicroseconds can be used.

Again, I do not find these IEs in RFC5102.

> 1. there is a meteringProcessId in IPFIX IANA while there is no cache id

meteringProcessId should do the job.

> inside the cache in the figure 12 in [IPFIX-CONF] below

See reply to Pauls mail.

Regards,
Gerhard

---

From: Gerhard Muenz <[email-masked]>
Date: Wed, 21 Sep 2011 21:46:18 +0200

Dear all,

> The cache name could be used to relate the given config to the process ID.

Correct. In general, IDs cannot be considered to be configurable.

> And also an "exportingProcessId" (IANA #145) as a read-only property in

I'm not sure about the one-to-one relationship between Metering Process 
and Cache. On purpose, IPFIX-CONF does not make a statement because it 
is not an architectural document. A Metering Process may have multiple 
Caches. But I do not think that a Cache would belong to multiple 
Metering Process. So, there is only one meteringProcessId which is 
associated with a Cache.

We do not have selectorId, exportingProcessId, and meteringProcessId in 
the configuration data model. If we think about adding 
meteringProcessId, it may be worth thinking about the other IDs as well.

I do not see a problem if these are defined as state parameters (e.g. 
read-only).

selectorId is a bit different because the scope is OD, not IPFIX device. 
Therefore, the same ID value may appear for different Selectors of the 
same device.

Gerhard

---

From: Paul Aitken <[email-masked]>
Date: Wed, 21 Sep 2011 22:18:27 +0100

Gerhard,

> I'm not sure about the one-to-one relationship between Metering 
> Process and Cache. On purpose, IPFIX-CONF does not make a statement 
> because it is not an architectural document. A Metering Process may 
> have multiple Caches. But I do not think that a Cache would belong to 
> multiple Metering Process. So, there is only one meteringProcessId 
> which is associated with a Cache.

Can you see anywhere in the IPFIX documents which shows the relationship 
between a Metering Process and cache(s) ?

I don't see "cache" in RFC5101 (IPFIX PROTO) or RFC5470 (IPFIX ARCH).

However, Figures 1 and 2 of IPFIX CONF show a Metering Process with only 
one cache.

>
> We do not have selectorId, exportingProcessId, and meteringProcessId 
> in the configuration data model. If we think about adding 

Definitely.

Thanks,
P.

> selectorId is a bit different because the scope is OD, not IPFIX 
> device. Therefore, the same ID value may appear for different 
> Selectors of the same device.
>
> Gerhard

---

From: Benoit Claise <[email-masked]>
Date: Thu, 22 Sep 2011 00:58:37 +0200

Hi Gerhard,
> I do not see a problem if these are defined as state parameters (e.g. 
Great.
I propose to add them in the AUTH48 process.

Regards, Benoit.

---

From: Benoit Claise <[email-masked]>
Date: Thu, 22 Sep 2011 01:11:23 +0200

Hi Gerhard,

Thanks for your reply.
See in line.
>> Secondly, "The timestamp of the first Flow was dropped by the 
>> Metering Process." is bad English: at least it's missing "that".
>>
Which ones?  flowEndMilliseconds, flowEndMicroseconds, 
flowEndNanoseconds, and flowEndDeltaMicroseconds
There are in http://www.iana.org/assignments/ipfix/ipfix.xml
However, re-reading

    time last_packet_ignored

And http://www.iana.org/assignments/ipfix/ipfix.xml for the "flowEnd" 
timestamp IE,
I now believe that we should be using this series of IEs

    322    observationTimeSeconds
    323    observationTimeMilliseconds
    324    observationTimeMicroseconds
    325    observationTimeNanoseconds

> Again, I do not find these IEs in RFC5102.
Same remark, I believe we should be using for the two previous IEs

    322    observationTimeSeconds
    323    observationTimeMilliseconds
    324    observationTimeMicroseconds
    325    observationTimeNanoseconds

Regards, Benoit.

---

From: Andrew Feren <[email-masked]>
Date: Wed, 21 Sep 2011 22:49:36 -0400

Hi Benoit,

I had the same thought about using

The problem is that you need two timestamps in the same record (first
and last).  How does that work with the above?

-Andrew

On 09/21/2011 07:11 PM, Benoit Claise wrote:
>>> [this email addresses
>>> draft-claise-ipfix-protocol-rfc5101bis-00.txt,  TO DO -> "time first
>>> flow dropped" and "time last flow dropped" inconsistency.  See the
>>> discussion on the list.]
>>>
>>> Thirdly, the section 4.2 doesn't take into account a metering
>>> process id. Indeed, what if we have multiple caches on the exporter?
>>>
>>>    time first _packet _ignored
>>>    time last _packet _ignored
>    time first _packet _ignored
>    time last _packet _ignored
>>>                         the __Exporting Process._  For this timestamp, any
>>> 2. if the IPFIX exporter is configured from [IPFIX-CONF], what
>>> should be the value in meteringProcessId?
>>>     From figure 1 and 2, it seems that there is a one to one
>>> matching between the cache and the metering process.
>>>     If this is the case, a solution could be to add a
>>> meteringProcess inside the cache in the figure 12 in [IPFIX-CONF] below
>>
>> See reply to Pauls mail.
>>
>> Regards,
>> Gerhard
>>
>>>
>>>       4.3. Cache Class
>>>
>>>

---

From: Benoit Claise <[email-masked]>
Date: Thu, 22 Sep 2011 09:54:57 +0200

Hi Andrew,
> and last).  How does that work with the above?
Excellent question. We debated this question with Paul Aitken recently.
Let me rephrase the question slightly differently: how does the 
collector determine that this is a "The Metering Process Reliability 
Statistics Options Template", and understands the semantic of the two 
counters?
Well. Two solutions:
1. we duplicate all the IEs to have the exact correct semantic.
     
ObservationTime[packet|flow][first|last][Seconds|Milliseconds|Microseconds|Nanoseconds] 
... and potentially some more such as pre|post
     This brings multiple new dimensions to IE and will lead to an 
explosion of IEs if we want to cover all the case
2. the collector tests the possible combination of IE in the Options 
Template Record.
     For example, if the collector sees such an Options Template Record, 
it knows that this a "The Metering Process Reliability Statistics 
Options Template"

   observationDomainId (scope)
   meteringProcessId (scope)_
_   ignoredPacketTotalCount
   ignoredOctetTotalCount
   observationTimeSeconds | observationTimeMilliseconds | observationTimeMicroseconds | observationTimeNanoseconds
   observationTimeSeconds | observationTimeMilliseconds | observationTimeMicroseconds | observationTimeNanoseconds

While the solution 2 is not perfect, this was the chosen track.

Does it make sense?

Regards, Benoit
>>>> draft-claise-ipfix-protocol-rfc5101bis-00.txt,  TO DO -> "time 
>>>> first flow dropped" and "time last flow dropped" inconsistency.  
>>>> See the discussion on the list.]
>>>>

---

From: Gerhard Muenz <[email-masked]>
Date: Thu, 22 Sep 2011 14:20:02 +0200

Hi Benoit,

You are right. I was confused because "time first packet ignored" is 
not an IE name.

Regards,
Gerhard

On Thu, 22 Sep 2011 01:11:23 +0200, Benoit Claise wrote:
>>> The timestamp of the first Flow was dropped by
>>> Firstly, these definitions are inconsistent since the names and
>>> the first definition say "flow" while the second definition says
>>> "IP packet". Obviously "IP packet" != "flow" :-o
>>>
>>> 4.2. THE METERING PROCESS RELIABILITY STATISTICS OPTION TEMPLATE
>>>
>>> Elements that are defined in [RFC5102 [1]]:
>>>
>>> observationDomainId
>>> meteringProcessId (*)
>>> time first packet ignored
>>> time last packet ignored
>  There are in http://www.iana.org/assignments/ipfix/ipfix.xml [2]
>  However, re-reading
>
>  time first packet ignored
>  time last packet ignored
>  And http://www.iana.org/assignments/ipfix/ipfix.xml [3] for the
> "flowEnd" timestamp IE,
> 4.3. THE EXPORTING PROCESS RELIABILITY STATISTICS OPTION TEMPLATE
>
>  Elements that are defined in [RFC5102 [4]]:
>
>  Exporting Process ID
>  [RFC5102 [5]] that can be used for this purpose:
>   The time at which the first Flow Record was dropped by
>  the Exporting Process. For this timestamp, any
>  > flowStartDeltaMicroseconds can be used. time last flow dropped The
>> time at which the last Flow Record was dropped by the Exporting
>> Process. For this timestamp, any of the "flowEnd" timestamp
>> Information Elements flowEndMilliseconds, flowEndMicroseconds,
>> flowEndNanoseconds, and flowEndDeltaMicroseconds can be used.
>> 4.3. CACHE CLASS
>> +-----------------------------------+ | Cache |
>> +-----------------------------------+ 1 +------------------+ | name
>> |--------| immediateCache/ | | dataRecords {readOnly} | |
>> timeoutCache/ | | cacheDiscontinuityTime {readOnly} | |
>> naturalCache/ | | | | permanentCache | | | +------------------+ | |
>> |
>  | 0..* +------------------+ | |--------->| ExportingProcess |
> +-----------------------------------+ +-----------------
>
>> Figure 12: Cache class What do you think? Do you see another way?
>>
>> Regards, Paul & Benoit.
> br>
>
> [email-masked] [6]
> https://www.ietf.org/mailman/listinfo/ipfix [7]
>
>>
>
>
> Links:
> ------
> [1] http://tools.ietf.org/html/rfc5102
> [2] http://www.iana.org/assignments/ipfix/ipfix.xml
> [3] http://www.iana.org/assignments/ipfix/ipfix.xml
> [4] http://tools.ietf.org/html/rfc5102
> [5] http://tools.ietf.org/html/rfc5102
> [6] mailto:[email-masked]
> [7] https://www.ietf.org/mailman/listinfo/ipfix

---

From: Andrew Feren <[email-masked]>
Date: Fri, 23 Sep 2011 17:07:04 -0400

Yes, this makes sense.

I gave this some thought and I don't have a better suggestion.  I also 
agree that even though the flow[Start|End]* collection of IEs 
conveniently has one IE for both first and last, observationTime* is the 
more correct choice here.

-Andrew

On 09/22/2011 03:54 AM, Benoit Claise wrote:
>     For example, if the collector sees such an Options Template 
> Record, it knows that this a "The Metering Process Reliability 
> Statistics Options Template"
>>> And http://www.iana.org/assignments/ipfix/ipfix.xml for the 
>>> "flowEnd" timestamp IE,
>>>>> (*) regarding the meteringProcessId, we were hesitating between 
>>>>> the meteringProcessId and the cache id

---

From: Brian Trammell <[email-masked]>
Date: Tue, 27 Sep 2011 18:15:33 +0200

Greetings, all,

This is a little bit of an abuse of the ordering of IEs, in my opinion, and of the general rule that only IEs, and the scope in which they are defined, carry semantics. Even for templates which are defined within Standards Track RFCs, the records defined by the template should be decipherable without reference to the RFC. Put another way, we should not define magical templates, and this template as proposed looks pretty magical.

Without reading the text of this thread, and following basically everything we've ever said about multiple IEs in a record, I would interpret a record matching this template as follows:

For the metering process A and observation domain B:
N packets were ignored; these contained a total of M octets.
The first observation at this metering process was at time T0.
The second observation at this metering process was at time T1.

This is quite far from the stated intention. I would strongly suggest that we not so lightly give up on "templates should be unambiguously understandable" requirement, even if the exception itself lives in the core protocol document.

I would suggest, if you want to do a multiple IE solution here, you define something like missingObservationWindowEdge, and have two of them:

  observationDomainId {scope}
  meteringProcessId {scope}
  ignoredPacketTotalCount
  ignoredOctetTotalCount
  missingObservationWindowEdge
  missingObservationWindowEdge

This has the advantage that you can have a long basicList of window edges in order to define multiple missing observation windows; each edge pair is a loss window. 

Seconds suffice for these, I think. You might want to do one in milliseconds. But if you have the resources to calculate your packet loss window to nanosecond precision without having the resources to not lose packets, I humbly submit you've picked the wrong engineering tradeoffs. :)

Best regards,

Brian

On Sep 23, 2011, at 11:07 PM, Andrew Feren wrote:

> Yes, this makes sense.
> 
> I gave this some thought and I don't have a better suggestion.  I also agree that even though the flow[Start|End]* collection of IEs conveniently has one IE for both first and last, observationTime* is the more correct choice here.
> 
> -Andrew
> 
>>> The problem is that you need two timestamps in the same record (first and last).  How does that work with the above?
>> Let me rephrase the question slightly differently: how does the collector determine that this is a "The Metering Process Reliability Statistics Options Template", and understands the semantic of the two counters?
>> Well. Two solutions:
>>     ObservationTime[packet|flow][first|last][Seconds|Milliseconds|Microseconds|Nanoseconds] ... and potentially some more such as pre|post
>>     This brings multiple new dimensions to IE and will lead to an explosion of IEs if we want to cover all the case
>> 2. the collector tests the possible combination of IE in the Options Template Record.
>>     For example, if the collector sees such an Options Template Record, it knows that this a "The Metering Process Reliability Statistics Options Template"
>>   meteringProcessId (scope)
>> 
>>   ignoredPacketTotalCount
>>   ignoredOctetTotalCount
>>>>>> [this email addresses draft-claise-ipfix-protocol-rfc5101bis-00.txt,  TO DO -> "time first flow dropped" and "time last flow dropped" inconsistency.  See the discussion on the list.]
>>>>>> 
>>>>>> From [RFC5101], section 4.3. The Exporting Process Reliability Statistics Option Template
>>>>>> 
>>>>>> time first flow dropped 
>>>>>> Firstly, these definitions are inconsistent since the names and the first definition say "flow" while the second definition says "IP packet". Obviously "IP packet" != "flow" :-o 
>>>>>> 
>>>>>> Secondly, "The timestamp of the first Flow was dropped by the Metering Process." is bad English: at least it's missing "that". 
>>>>>> 
>>>>>> Thirdly, the section 4.2 doesn't take into account a metering process id. Indeed, what if we have multiple caches on the exporter?
>>>>>> 
>>>>>>    Elements that are defined in [
>>>>>> RFC5102
>>>>>> ]:
>>>>>> 
>>>>>>    observationDomainId
>>>>>>    time first 
>>>>>> packet 
>>>>>> ignored
>>>>>>    time last 
>>>>>> packet 
>>>>>> ignored
>>>> Which ones?  flowEndMilliseconds, flowEndMicroseconds, flowEndNanoseconds, and flowEndDeltaMicroseconds
>>>>    time first packet 
>>>> ignored
>>>>    time last 
>>>> packet 
>>>> ignored
>>>> And http://www.iana.org/assignments/ipfix/ipfix.xml for the "flowEnd" timestamp IE,
>>>>>>    Elements that are defined in [
>>>>>> RFC5102
>>>>>> ]:
>>>>>> 
>>>>>>    Exporting Process ID
>>>>>>                         [
>>>>>> RFC5102
>>>>>> ] that can be used for this purpose:
>>>>>>                         the 
>>>>>> Exporting Process.
>>>>>>   For this timestamp, any
>>>>>> The time at which the last Flow Record was
>>>>>>                         dropped by the Exporting Process
>>>>>> .  For this
>>>>>> (*) regarding the meteringProcessId, we were hesitating between the meteringProcessId and the cache id
>>>>>> 2. if the IPFIX exporter is configured from [IPFIX-CONF], what should be the value in meteringProcessId?
>>>>>>     From figure 1 and 2, it seems that there is a one to one matching between the cache and the metering process.
>>>>>>     If this is the case, a solution could be to add a meteringProcess inside the cache in the figure 12 in [IPFIX-CONF] below
>>>>> 
>>>>> See reply to Pauls mail.
>>>>> 
>>>>> Regards,
>>>>> Gerhard
>>>>> 
>>>>>> 4.3.  Cache Class
>>>>>> 
>>>>>>

---

From: Andrew Feren <[email-masked]>
Date: Tue, 27 Sep 2011 13:30:03 -0400

Hi Brian,

Well stated.

Additional comments inline

> This is a little bit of an abuse of the ordering of IEs, in my opinion, and of the general rule that only IEs, and the scope in which they are defined, carry semantics. Even for templates which are…
Yes, a black magic sort of magical.  We have been discussing this 
proposed change in the office and coming to the same conclusion.
> The second observation at this metering process was at time T1.
This was exactly how I interpreted this template until I read the 
updated draft and saw the bit about "the Collecting Process MUST 
determine which is the oldest and the most recent timestamp in order the 
determine the right semantic ...."

> This has the advantage that you can have a long basicList of window edges in order to define multiple missing observation windows; each edge pair is a loss window.
I like this suggestion, and I have a few additional thoughts.

Perhaps the basic list of edge times should either be limited to only 
one pair or the ignored*TotalCount IEs needs to be in a basiclist with N 
entries too.

How about observationWindowEdge instead (no "missing") to be more 
generally useful.

I think a list of time edges also works neatly when sending this 
template at regular intervals (which SHOULD be done).  When the 
ignored*TotalCount IEs are zero, an empty list makes sense for edge times.

> Seconds suffice for these, I think. You might want to do one in milliseconds. But if you have the resources to calculate your packet loss window to nanosecond precision without having the resources…
Unless window edge isn't defined specifically for "missing" values.  
Then the higher precision timestamps might have some use.

-Andrew
>>>>>>> Regards, Paul&  Benoit.

---

From: Paul Aitken <[email-masked]>
Date: Tue, 27 Sep 2011 21:25:20 +0100

Andrew, Brian,

How do you know that the received template is a MPRSO?
     - must the fields be exactly in the specified order and no other 
combination in order to be a valid MPRSO?
     - how do you know which of the observationTime* elements (or 
missingObservationWindowEdge elements) is the first and which is the last?
         - only from their position in the template (so, field order and 
position are crucial)
         - by comparing them (so field order and position aren't important)

Suppose a mediator adds, removes, or re-orders the fields.
     - does this invalidate the MPRSO?
     - how does a mediator know not to do this?
     - how does the collector cope if a mediator does?

The change to "missingObservationWindowEdge" is simply a rename of 
observationTime. I don't see that it adds anything.

A long basicList isn't the right solution because
     - various list lengths mean there are multiple possible templates 
making the detection of the MPRSO a little more difficult
        ie, it's no longer a fixed template, nor a combination of a few 
possibilities, but a great many different possible templates.
     - there are two different semantics for the start and end of the 
window, so it's actually an ordered list of pairs ie a subTemplateList.

Lists can have zero elements. However, if you want to report that 
nothing was lost, the ignored*Count = 0 report should indicate the start 
and end of the 100% period. A list length of zero doesn't tell when 
stuff was or was not lost - so it's of limited usefulness, unless 
perhaps as a total figure? Again, see questions above about what is / is 
not a valid MPRSO.

Thanks,
P.

On 27/09/11 18:30, Andrew Feren wrote:
>> This is a little bit of an abuse of the ordering of IEs, in my 
>> opinion, and of the general rule that only IEs, and the scope in 
>> which they are defined, carry semantics. Even for templates which are 
>> defined within Standards Track RFCs, the records defined by the 
>> template should be decipherable without reference to the RFC. Put 
>> another way, we should not define magical templates, and this 
>> template as proposed looks pretty magical.
>> Without reading the text of this thread, and following basically 
>> everything we've ever said about multiple IEs in a record, I would 
>> interpret a record matching this template as follows:
>>
> determine which is the oldest and the most recent timestamp in order 
> the determine the right semantic ...."
>
>> This is quite far from the stated intention. I would strongly suggest 
>> that we not so lightly give up on "templates should be unambiguously 
>> understandable" requirement, even if the exception itself lives in 
>> the core protocol document.
>>
>> I would suggest, if you want to do a multiple IE solution here, you 
>> define something like missingObservationWindowEdge, and have two of 
>> them:
>>
>> This has the advantage that you can have a long basicList of window 
>> edges in order to define multiple missing observation windows; each 
>> edge pair is a loss window.
> one pair or the ignored*TotalCount IEs needs to be in a basiclist with 
> N entries too.
>
>> Seconds suffice for these, I think. You might want to do one in 
>> milliseconds. But if you have the resources to calculate your packet 
>> loss window to nanosecond precision without having the resources to 
>> not lose packets, I humbly submit you've picked the wrong engineering 
>> tradeoffs. :)
>>> I gave this some thought and I don't have a better suggestion.  I 
>>> also agree that even though the flow[Start|End]* collection of IEs 
>>> conveniently has one IE for both first and last, observationTime* is 
>>> the more correct choice here.
>>>
>>> -Andrew
>>>
>>>>> The problem is that you need two timestamps in the same record 
>>>>> (first and last).  How does that work with the above?
>>>> collector determine that this is a "The Metering Process 
>>>> Reliability Statistics Options Template", and understands the 
>>>> semantic of the two counters?
>>>> Well. Two solutions:
>>>>    observationTimeSeconds | observationTimeMilliseconds | 
>>>> observationTimeMicroseconds | observationTimeNanoseconds
>>>>    observationTimeSeconds | observationTimeMilliseconds | 
>>>> observationTimeMicroseconds | observationTimeNanoseconds
>>>>
>>>>
>>>>>>>> first flow dropped" and "time last flow dropped" 
>>>>>>>> the first definition say "flow" while the second definition 
>>>>>>>> says "IP packet". Obviously "IP packet" != "flow" :-o
>>>>>>>>
>>>>>>>>      |                                   |          | 
>>>>>>>> permanentCache   |
>>>>>>>> +------------------+
>>>>>>>> +------------------+

---

From: Brian Trammell <[email-masked]>
Date: Tue, 27 Sep 2011 23:13:51 +0200

Hi, Paul, all,

I suppose I should state for the sake of completeness that this is the first I've ever actually _thought_ about the MPRSO. I find the whole use case for it rather dubious -- it presumes an MP with somehow trustworthy knowledge about packet loss, which is never the case, and relying on such information at the collector side for anything other than entertainment purposes is fraught with peril -- so I saw the MAY on section 4 in 5101, was thankful it wasn't a MUST I had to fight against and/or simply ignore, and never looked back.

That said, it does make sense to fix, since it's there.

> How do you know that the received template is a MPRSO?

Does it matter? Seriously, what is the difference between "an MPRSO" and "a template containing information about lost packets for a given time interval, along with maybe some other information"? Do we actually want to require an MP to keep time interval information for loss? Wouldn't those resources be better deployed not losing packets in the first place? So let's say I want to report missing packets, but don't have information about when I lost them. Do I have to throw a timestamp in there anyway in order for it to be "an MPRSO"?

I think these are arguments against having the timestamps at all (or to include them, with a further instruction that they are optional).

>    - must the fields be exactly in the specified order and no other combination in order to be a valid MPRSO?

This isn't a valid template recognition strategy in IPFIX. At least, it hasn't been to date, and it's a bad idea to make it so. IE-Doctors contains language meant to instruct IPFIX application authors to refrain from specifying such magic templates.

>    - how do you know which of the observationTime* elements (or missingObservationWindowEdge elements) is the first and which is the last?
>        - only from their position in the template (so, field order and position are crucial)

Time intervals can be treated as a special case, here, since time can generally be assumed to go in only one direction. "The time interval from A to B" is completely equivalent to "the time interval from B to A", so one could argue order doesn't matter. (We don't argue that anywhere in IPFIX, to be true. But what do I do when I get a flow with an endTime before the startTime?)

>    - how does the collector cope if a mediator does?

These questions seem to apply only to the case where MPRSO is magical. I'm trying to define it such that it isn't.

> The change to "missingObservationWindowEdge" is simply a rename of observationTime. I don't see that it adds anything.

observationTime already has a meaning. "This Information Element specifies the absolute time in seconds of an observation." In other words, "I saw the thing I'm talking about in this record at the specified time". It has never to date been specified in such a way that it means "I saw the thing I'm talking about in this record at the specified time, unless the record contains two observationTimes, in which case I saw the thing in the time interval between those two things".

missingObservationWindowEdge means "there is missing information within an interval described between this IE and the other identical IE in the record". If we keep the MPRSO defined as it is (and fix it only by dropping in , missingObservationWindowStartSeconds and missingObservationWindowEndSeconds, since it's more IPFIXish, but there was concern voiced in this thread about IE space explosion, so I figured we could exploit the nature of time and cut down on the IEs required by half.

The advantage of a new IE is that it isn't overloaded, but more importantly that it does not lead to an ambiguous interpretation of the entire template at the collector.

>    - various list lengths mean there are multiple possible templates making the detection of the MPRSO a little more difficult
>       ie, it's no longer a fixed template, nor a combination of a few possibilities, but a great many different possible templates.
>    - there are two different semantics for the start and end of the window, so it's actually an ordered list of pairs ie a subTemplateList.
> Lists can have zero elements. However, if you want to report that nothing was lost, the ignored*Count = 0 report should indicate the start and end of the 100% period. A list length of zero doesn't tell when stuff was or was not lost - so it's of limited usefulness, unless perhaps as a total figure? Again, see questions above about what is / is not a valid MPRSO.

I withdraw my basicList comment; it was essentially just "thinking aloud", and I didn't actually intend to include a mention of it in the 5101bis MPRSO. 1. It doesn't really add anything to the original intent of the MPRSO, and 2. adding it in 5101bis would cause 5101bis to have a normative dependency on 6313, which is a terrible idea.

Cheers,

Brian

>> Yes, a black magic sort of magical.  We have been discussing this proposed change in the office and coming to the same conclusion.
>> This was exactly how I interpreted this template until I read the updated draft and saw the bit about "the Collecting Process MUST determine which is the oldest and the most recent timestamp in order the determine the right semantic ...."
>> 
>> Perhaps the basic list of edge times should either be limited to only one pair or the ignored*TotalCount IEs needs to be in a basiclist with N entries too.
>> 
>> How about observationWindowEdge instead (no "missing") to be more generally useful.
>> 
>> I think a list of time edges also works neatly when sending this template at regular intervals (which SHOULD be done).  When the ignored*TotalCount IEs are zero, an empty list makes sense for edge times.
>> 
>> 
>> Unless window edge isn't defined specifically for "missing" values.  Then the higher precision timestamps might have some use.
>> 
>> -Andrew
>>> 
>>> Best regards,
>>> 
>>> Brian
>>>

---

From: Paul Aitken <[email-masked]>
Date: Tue, 27 Sep 2011 23:42:07 +0100

Brian,

> I think these are arguments against having the timestamps at all (or to include them, with a further instruction that they are optional).

When a MP knows that it failed to meter some packets, it must have a way 
of reporting this which the CP can understand correctly. So it matters 
in that the CP has to understand the implicit "lost traffic" semantic 
and potentially take action, perhaps involving reports or alarm bells.

> This isn't a valid template recognition strategy in IPFIX. At least, it hasn't been to date, and it's a bad idea to make it so. IE-Doctors contains language meant to instruct IPFIX application auth…

Great; I'm pleased to hear it. Magic templates lead to inflexible 
collectors, which in turn lead to bug reports against the EP?!

> Time intervals can be treated as a special case, here, since time can generally be assumed to go in only one direction. "The time interval from A to B" is completely equivalent to "the time interva…

When the timestamp wraps. The time between 11pm and 1am is 2 hours, 
while the time between 1am and 11pm is 22 hours.

> These questions seem to apply only to the case where MPRSO is magical. I'm trying to define it such that it isn't.

I'm trying to get to the core of what constitues a valid, recognisable 
MPRSO. Put another way, what's the closest thing to an MPRSO that isn't 
an MPRSO?

If I use ignoredPacketTotalCount and ignoredOctetTotalCount in another 
template, does it become an MPRSO?

If we ignore what is and what isn't an MPRSO for the moment, how exactly 
should a MP tell a CP about the traffic it knows it didn't meter?

> observationTime already has a meaning. "This Information Element specifies the absolute time in seconds of an observation." In other words, "I saw the thing I'm talking about in this record at the…

Then you're saying that it's invalid to have two different 
observationTime values in a flow record, because you can't observe an 
event at two different times.

Thinking about it some more, I suppose this reports multiple instances 
of the observation with an AND semantic: I saw this at T1 AND at T2 
(since other semantics don't make sense).
Think if these were interfaces: the meaning would be "observed at int1 
and int2", rather than "observed from int1 to int2".
So I've convinced myself to agree that 2 x observationTime don't make sense.

> The advantage of a new IE is that it isn't overloaded, but more importantly that it does not lead to an ambiguous interpretation of the entire template at the collector.

Unfortunately WindowEdge is ambiguous in the wrap case, whereas start 
and end aren't.

> I withdraw my basicList comment; it was essentially just "thinking aloud", and I didn't actually intend to include a mention of it in the 5101bis MPRSO. 1. It doesn't really add anything to the ori…

+1. I already discussed with Benoit about using an ordered list or STL 
of observationTime, and discounted it for this reason.

P.

---

From: Brian Trammell <[email-masked]>
Date: Wed, 28 Sep 2011 08:03:00 +0200

On Sep 28, 2011, at 12:42 AM, Paul Aitken wrote:

> When a MP knows that it failed to meter some packets, it must have a way of reporting this which the CP can understand correctly. So it matters in that the CP has to understand the implicit "lost traffic" semantic and potentially take action, perhaps involving reports or alarm bells.

That's an argument for the following template:

Which, in my opinion, is a pretty good starting point.

You can add ignoredOctetTotalCount if you actually know it. Calculating it at libpcap-based MPs like YAF, for example, would involve a little work and generate a bad number: you only get a guess from the kernel how many packets it dropped and no information about how big they are. You can infer octet loss by looking at TCP sequence numbers but now we're back to clear overengineering I think. Maybe on ASIC or FPGA based measurement systems you get good total counters and can measure loss subtractively... But I personally wouldn't waste the real estate on it unless it was a hard requirement.

Again, if you know the timestamps, you can add them. But for the general case as I see it, your fidelity per IE for the template in 5101 is as follows:

perfect:                observationDomainId {scope}
 (i hope you know this) meteringProcessId {scope}

reasonably good guess:  ignoredPacketTotalCount

poor to fair guess:     ignoredOctetTotalCount
 (dep. on HW support)

poor to good guess:     a start timestamp of indeterminate IE
 (dep. on HW support)   an end timestamp of indeterminate IE

> Great; I'm pleased to hear it. Magic templates lead to inflexible collectors, which in turn lead to bug reports against the EP?!
> 
> 
> When the timestamp wraps. The time between 11pm and 1am is 2 hours, while the time between 1am and 11pm is 22 hours.

We don't have any timestamps that only carry time information. They've all got implied dates (either based on the 1970 epoch for the UNIX-like timestamps, or on either the NTP or the UNIX epoch for the 1305 timestamps (we'll need to fix this in 5101bis), and there is no specified wrapping behavior as yet. You only hit this problem in 2038, then, and when you do, you've got bigger problems.

> I'm trying to get to the core of what constitues a valid, recognisable MPRSO. Put another way, what's the closest thing to an MPRSO that isn't an MPRSO?

I'm saying that it doesn't matter. Sure, 5101bis should give recommendations for the template an exporting process uses for this information, but any time a CP sees a template that's scoped to something that ID's an MP that contains loss counter IEs, it's an MPRSO. 

> If I use ignoredPacketTotalCount and ignoredOctetTotalCount in another template, does it become an MPRSO?

Depends. Are you binding them to a scope that identifies a metering process? You could also export them in a flow or "flexible flow"/flow aggregate, at which point you're giving very detailed error information. And the collector can infer whatever it wants about total reliability based on _that_...

> If we ignore what is and what isn't an MPRSO for the moment, how exactly should a MP tell a CP about the traffic it knows it didn't meter?

Using something much like the MPRSO as defined by 5101... I'm just trying to lead us away from thinking of a "bright line" between MPRSO and not-MPRSO, because 

> Then you're saying that it's invalid to have two different observationTime values in a flow record, because you can't observe an event at two different times.

Well.... yes, this is a bit ambiguous.

> Thinking about it some more, I suppose this reports multiple instances of the observation with an AND semantic: I saw this at T1 AND at T2 (since other semantics don't make sense).
> Think if these were interfaces: the meaning would be "observed at int1 and int2", rather than "observed from int1 to int2".

(I'm trying to think of cases where it would be, but I don't think any are illustrative for this discussion...)

> Unfortunately WindowEdge is ambiguous in the wrap case, whereas start and end aren't.

(see above on wrapping)

> +1. I already discussed with Benoit about using an ordered list or STL of observationTime, and discounted it for this reason.

Yay! At least we emphatically agree on _something_. ;)

Cheers,

Brian

---

From: Benoit Claise <[email-masked]>
Date: Wed, 28 Sep 2011 17:59:38 +0200

Brian,
> Hi, Paul, all,
>
> I suppose I should state for the sake of completeness that this is the first I've ever actually _thought_ about the MPRSO. I find the whole use case for it rather dubious -- it presumes an MP with somehow trustworthy knowledge about packet loss, which is never the case,
Well, I can give you an example of Cisco devices, with metering in 
hardware, that displays the number of packets that were not taken into 
account, due to excessive cache collisions.

Regards, Benoit.
>   and relying on such information at the collector side for anything other than entertainment purposes is fraught with peril -- so I saw the MAY on section 4 in 5101, was thankful it wasn't a MUST I had to fight against and/or simply ignore, and never looked back.
>

---

From: Benoit Claise <[email-masked]>
Date: Wed, 28 Sep 2011 18:06:47 +0200

Dear all,

Let me restate what I put in my initial email:

    Well. Two solutions:

And I'm not sure what's the conclusion of this email thread is...

Regards, Benoit.

---

From: Brian Trammell <[email-masked]>
Date: Wed, 28 Sep 2011 18:31:32 +0200

hi, Benoit,

okay, cool... and this hardware never loses a packet for any other reason? (and it also keeps octet counts for the missing packets? and usable timestamps? very neat. I'd buy one if I were in the market. :)

there are lots of ways to count single-point loss. I'm not convinced that any of them count what they pretend to. this is my point here: yes, it is useful to detect whether the mp knows it's dropping or not. the order of magnitude might be useful in some cases. but real lost packet counts should always be taken with a half ton of salt at the CP. 

all of this is kind of a sideline, though. the standing question is: given you want an MPRSO, how do you do it; on which, see next message...

cheers, 

Brian

Sent from my iPhone

On 28.09.2011, at 17:59, Benoit Claise <[email-masked]> wrote:

> Well, I can give you an example of Cisco devices, with metering in hardware, that displays the number of packets that were not taken into account, due to excessive cache collisions.
> 
> Regards, Benoit.

---

From: Brian Trammell <[email-masked]>
Date: Wed, 28 Sep 2011 18:47:36 +0200

Hi, Benoit, all,

Conclusion (from my side): neither.

2. relies on magical templates, which I regard as basically poisonous. Being able to know what a template represents unambiguously is one of the key strengths of IPFIX.

1. is half-magical, so half poisonous. The timestamps carry no semantics identifying that they have to do with a window during which loss occurred. We could I suppose change the definition of ignored*count to say "if present, timestamps in the same record identify the outer bounding window of the loss event(s)" or similar.

I suggest 3. drop the timestamps completely, or 4. define loss window timestamps. 

Cheers, 

Brian

Sent from my iPhone

---

From: Benoit Claise <[email-masked]>
Date: Thu, 29 Sep 2011 11:59:51 +0200

On 28/09/2011 18:47, Brian Trammell wrote:
> 2. relies on magical templates, which I regard as basically poisonous. 
> Being able to know what a template represents unambiguously is one of 
> the key strengths of IPFIX.
Agreed.
>
> 1. is half-magical, so half poisonous.
And again agreed.

Out of the bad solutions 1 and 2, we have to chose one.
> The timestamps carry no semantics identifying that they have to do 
> with a window during which loss occurred. We could I suppose change 
> the definition of ignored*count to say "if present, timestamps in the 
> same record identify the outer bounding window of the loss event(s)" 
> or similar.
>
> I suggest 3. drop the timestamps completely,
Sure, the IPFIX specifications will look fine, but we're postponing the 
problems.
> or 4. define loss window timestamps.
Proposal 5.  Keep for a single point in time observation

      Have to new series of observation times

    .           observationStartTimeSeconds
    .           observationStartTimeMilliseconds
    .           observationStartTimeMicroseconds
    .           observationStartTimeNanoseconds

    .           observationEndTimeSeconds
    .           observationEndTimeMilliseconds
    .           observationEndTimeMicroseconds
    .           observationEndTimeNanoseconds

Regards, Benoit.

>
> Cheers,
>
> Brian
>
> Sent from my iPhone
>
> On 28.09.2011, at 18:06, Benoit Claise <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>> Dear all,
>>
>>>>>>>>>>>> There are inhttp://www.iana.org/assignments/ipfix/ipfix.xml
>>>>>>>>>>>> However, re-reading
>>>>>>>>>>>>     time first packet
>>>>>>>>>>>> ignored
>>>>>>>>>>>> Andhttp://www.iana.org/assignments/ipfix/ipfix.xml  for the "flowEnd" timestamp IE,
>>>>>>>>>>>>>> [email-masked]  <mailto:[email-masked]>
>>>>>>>>>>>> [email-masked]  <mailto:[email-masked]>
>>>>>>>>>>> [email-masked]  <mailto:[email-masked]>
>>>>>>>>> [email-masked]  <mailto:[email-masked]>
>>>>>>> [email-masked]  <mailto:[email-masked]>
>>> [email-masked]  <mailto:[email-masked]>

---

From: Andrew Feren <[email-masked]>
Date: Thu, 29 Sep 2011 09:13:54 -0400

Hi, Benoit and list,

On 09/29/2011 05:59 AM, Benoit Claise wrote:
>> 2. relies on magical templates, which I regard as basically 
>> poisonous. Being able to know what a template represents 
>> unambiguously is one of the key strengths of IPFIX.
> Agreed.
>>
> Sure, the IPFIX specifications will look fine, but we're postponing 
> the problems.

Benoit could you elaborate on what problems we are postponing?  This is 
not clear to me.

>     325    observationTimeNanoseconds
Seems reasonable.

The more I have looked at this template the less clear I have been on 
the reason for the two timestamps.  On my initial read I thought they 
bounded a window during which the most recent set of errors occurred.  
However, as I read more (and reread) it occurred to me that the counters 
are both total counters.  This seems to imply that the first timestamp 
is the first ignored[Packet|Octet] observed  since metering process 
start.  Resending first ignored over and over again seems to have little 
if any benefit.

There is, however, probably some detail(s) that I have not fully grokked.

Thanks for any clarification,
-Andrew

>>>         This brings multiple new dimensions to IE and will lead to
>>>     an explosion of IEs if we want to cover all the case