Subject: RE: [ipfix-arch] counter wrapping

From: "Norseth, KC" <[email-masked]>
Date: Wed, 27 Feb 2002 08:51:45 -0500

|-----Original Message-----
|From: Benoit Claise [mailto:[email-masked]] 
|Sent: Wednesday, February 27, 2002 3:32 AM
|To: Calato, Paul
|Cc: [email-masked]; 
[email-masked]; [email-masked]
|Subject: Re: [ipfix-arch] counter wrapping
|
|
|Paul,
|
|
|> Thought I would reply direct.
|>
|> I would check on that. Since the hardware
|> is the one that increments the counter as
|> packets are going through, the only way to
|> read the count and then set it to zero as
|> one operation would be to stop the hardware
|> from updating the counter while the read-set-to-zero operation is 
|> happening.
|>
|> So either the netflow hardware really supports that
|> operation or a tradeoff was made to leave the issue as is.
|
|I called our specialist. Without going into the implementation 
|details, his answer is: Our h/w does not loose any data when 
|it happens. I'd think it's a bug if some h/w does it.
|
|Here is my view on the absolute/delta counters, as I wrote in 
|a previous email. But I would like to expand on absolute/delta 
|counters.
|
|If you think of keeping an absolute counter value, this is a 
|counter you must maintain for all active flows. What should be 
|the size? You have no idea initially ... a 32 or 64 bit 
|counter? So you must have a 64 bit counter for every single 
|flow. This implies a lot of memory (if not done in hardware) 
|and it implies an extra 32 bits sent to the collector for 
|every flow records!
|
|If you think of sending delta values, it's delta compared to 
|what? To the flow aging time, right? Side question: Does it 
|mean that you must have an absolute counter as well?
|
|So why don't we forget about absolute or delta counters but 
|speak about the counters of the flow  live time.

You cannot just forget absolute or delta counters.  It is obviously an
issue.  It is also not clear in current documentation of flow accounting
protocols, and implementation is difficult when you have to guess what is
meant. We need to be clear in what we mean.

 you were to assume 30 minute intervals, that is asking for trouble.  In the
case of things like intrusion detection, you have to shorten the interval
time down to 1 to 5 minutes.

If a flow is exporting about 5m of packets every 5 minutes and the flow
lasts 20 minutes, and we had an interval set of 5 minutes, we would see the
byte count during export as 5m, 10m, 15m, 20m,...  For this to be meainngful
data to a an network analyst, he would want to see that there are 5m bytes
going on during this time.  Someone needs to give valid representation of
this information, either the collector or the exporter.  It is a lot easier
for the exporter to provide this information.  Having the exporter do it
also makes sense in the case of collector failover nbeacuse the new
collector does not know the history of the flow and would assume tnat when
it sees 15m, it really means 15m, not 5 since the last time the exporter
sent the flow.

|For long run flows, you can just export the flow records with 
|the accounting information seen during the flow live time. 
|Knowing a long run flow is actually a serie of shorter flows 
|with the same characteristics.
|
|You must export the flow records because:
| - you faced the long aging time timeout (for example: 30 
|minutes), then the live time of the flow is 30 minutes. We 
|export the counters for that specific period of 30 minutes.
| - the 32 bits counter is about to wrap. Then you consider the 
|flow to be ended (expired) sooner and export the accounting 
|data for that period, whatever the period is.
|
|Now, because packets with the same characterics continue to 
|arrive, a new flow with the same characterics will be directly 
|created. Obviously with a counter starting at 0. That way, the 
|collector won't be confused at all and you have an easier 
|implementation.
|
|
|The explanation above was this idea when Ganesh and I wrote: 
|http://www.ietf.org/internet-drafts/draft-ietf-ipfix-data-00.txt
|
|5.4. Flow Expiration
|
|      ...
|
|       3. For long aging flows, the exporter SHOULD export the flow
|         records on regular basis, in order to:
|
|         a.  report the flow records periodic accounting information
|             to the collector
|         b.  avoid counter wrapping
|
|          THERE IS A SMALL FORMATTING PROBLEM. THERE SHOULD BE 
|A NEW LINE
|          This activity timeout SHOULD be configurable
|
|      4. If the exporter experiences resources constraints, a flow
|         MAY be prematurely expired (example: memory)
|
|Regards, Benoit
|
|>
|>
|> It would be good to know as it affects how to best
|> accomplish intermediate updates should we go that way.
|>
|> Paul
|>
|> Benoit Claise wrote:
|> >
|> > Paul,
|> >
|> > [email-masked] wrote:
|> >
|> > > Benoit Claise wrote:
|> > > >
|> > > > Nevil,
|> > > >
|> > > > [email-masked] wrote:
|> > > >
|> > > > > Hi Reinaldo:
|> > > > >
|> > > > > > b. avoid counter wrapping."
|> > > > > >
|> > > > > > This activity SHOULD be configurable. Btw, can someone 
|> > > > > > elaborate on the counter wrapping stuff?
|> > > > > >
|> > > > > > It is very simple to send so much data though the 
|line on a 
|> > > > > > flow that the byte counter rolls over.  This is 
|very typical 
|> > > > > > in 1 gig and 10g  ports. counter 32 is not large enough 
|> > > > > > field size for this.
|> > > > > >
|> > > > > > yes, that's what I tought. So the solution is just to 
|> > > > > > augment th efield to a 64 counter. It seems that the 
|> > > > > > peirodic reporting is completely orthogonal to 
|this specific 
|> > > > > > issue.
|> > > > >
|> > > > > You've covered the 'counter wrapping' problem - a flow 
|> > > > > information meter (whatever that turns out to be) needs 
|> > > > > counters wide enough not to wrap too often, and that becomes 
|> > > > > more of a problem at high link speeds. In RTFM, all 
|counts are 
|> > > > > kept in 64-bit unsigned counters so that they don't wrap too 
|> > > > > often.
|> > > > >
|> > > > > Of more interest, I don't remember seeing any IPFIX 
|discussion 
|> > > > > concerning just how information about long-running 
|flows is to 
|> > > > > be exported.  In particular, I believe that counters should 
|> > > > > never be reset.
|> > > >
|> > > > Why?
|> > > > For long run flows, you can just export the flow records with 
|> > > > the acconting information seen during that interval. 
|If you must 
|> > > > export the flow records because:
|> > > >  - you faced the long aging time timeout (for example: 30 
|> > > > minutes)
|> > > >  - the counters is about to wrap.
|> > > > then you consider the flow to be ended (expired) and 
|export the accounting data
|> > > > for that period.
|> > > > Now a new flow with the same characterics will be 
|directly created. Not a problem.
|> > > > And you start the accounting from 0 for that "new" flow.
|> > >
|> > >         Most hardware can't do a get and set in one
|> > >         operation. So in between reading the count
|> > >         and setting it to zero you may loose some
|> > >         of the count.
|> >
|> > I'm not too sure about the details of most hardware, but 
|this is the 
|> > way it works with netflow.
|> >
|> > Regards, Benoit
|> >
|> > >
|> > >
|> > > > The collector won't be confused at all and don't need to 
|> > > > remember any values.
|> > > >
|> > > > Regards, Benoit
|> > > >
|> > > > >
|> > > > > That would mean that successive exports of the same 
|flow would 
|> > > > > have increasing counter values, a collector would need to 
|> > > > > remember the previous value and subtract it.  Furthermore, 
|> > > > > provided unsigned arithmetic is used, the difference is 
|> > > > > correct even if the counter has wrapped.  Provided, 
|of course 
|> > > > > that the counter has only wrapped once!
|> > > > >
|> > > > > Cheers, Nevil
|> > > > >
|> > > > > 
|-----------------------------------------------------------------------
|> > > > >    Nevil Brownlee                   Director, 
|Technology Development
|> > > > >    Phone: +64 9 373 7599 x8941      ITSS, The 
|University of Auckland
|> > > > >    FAX: +64 9 373 7021      Private Bag 92019, 
|Auckland, New Zealand
|> > > > >
|> > > > > --
|> > > > > Help        mailto:[email-masked] and 
|say "help" in message body
|> > > > > Unsubscribe mailto:[email-masked] and say 
|> > > > > "unsubscribe ipfix" in message body
|> > > > > Archive     http://ipfix.doit.wisc.edu/archive/
|> > > >
|> > > > --
|> > > > Help        mailto:[email-masked] and say 
|"help" in message body
|> > > > Unsubscribe mailto:[email-masked] and say 
|> > > > "unsubscribe ipfix" in message body
|> > > > Archive     http://ipfix.doit.wisc.edu/archive/
|
|
|--
|Help        mailto:[email-masked] and say "help" 
|in message body
|Unsubscribe mailto:[email-masked] and say 
|"unsubscribe ipfix" in message body
|Archive     http://ipfix.doit.wisc.edu/archive/
|

---

From: Benoit Claise <[email-masked]>
Date: Wed, 27 Feb 2002 15:22:56 +0100

KC,

|To: Calato, Paul

|

|

|Paul,

|

|

|>

|> from updating the counter while the read-set-to-zero
operation is

|> happening.

|>

|> operation or a tradeoff was made to leave the issue
as is.

|

|details, his answer is: Our h/w does not loose any data
when

|

|Here is my view on the absolute/delta counters, as I
wrote in

|counters.

|

|If you think of keeping an absolute counter value, this
is a

|counter you must maintain for all active flows. What
should be

|the size? You have no idea initially ... a 32 or 64
bit

|counter? So you must have a 64 bit counter for every
single

|and it implies an extra 32 bits sent to the collector
for

|every flow records!

|

|If you think of sending delta values, it's delta compared
to

|what? To the flow aging time, right? Side question:
Does it

|

|So why don't we forget about absolute or delta counters
but

You cannot just forget absolute or delta counters.
It is obviously an issue.  It is also not clear in current documentation
of flow accounting protocols, and implementation is difficult when you
have to guess what is meant. We need to be clear in what we mean.

you were to assume 30 minute intervals, that is
asking for trouble.  In the case of things like intrusion detection,
you have to shorten the interval time down to 1 to 5 minutes.

5.4

section 3

This activity timeout SHOULD be configurable

If a flow is exporting about 5m of packets every 5 minutes
and the flow lasts 20 minutes, and we had an interval set of 5 minutes,
we would see the byte count during export as 5m, 10m, 15m, 20m,...

Call it delta if you want to, but makes sense to export is:

timestamps flow start     timestamps flow stop
counter

------------------------------------------------------------------------------------------

t1 (minute 0)
t2 (5th minute)
5m

t3 (5th minute)
t4 (10th minute)
5m

t5 (10th minute)
t6 (15th minute)
5m

t7 (20th minute)
t7 (20th minute)
5m

Then the collector doesn't have to do diff to find out meaningfull values,
to find out what happened in the previous 5 minutes

For this to be meainngful data to a
an network analyst, he would want to see that there are 5m bytes going
on during this time.  Someone needs to give valid representation of
this information, either the collector or the exporter.

It is a lot easier for the exporter
to provide this information.  Having the exporter do it also makes
sense in the case of collector failover nbeacuse the new collector does
not know the history of the flow and would assume tnat when it sees 15m,
it really means 15m, not 5 since the last time the exporter sent the flow.

Some more arguments not to report 5m, 10m, 15m, 20m,.., or do I miss something?

Regards, Benoit

|For long run flows, you can just export the flow records
with

|the accounting information seen during the flow live
time.

|Knowing a long run flow is actually a serie of shorter
flows

|

| - you faced the long aging time timeout (for example:
30

|minutes), then the live time of the flow is 30 minutes.
We

| - the 32 bits counter is about to wrap. Then you consider
the

|

|Now, because packets with the same characterics continue
to

|arrive, a new flow with the same characterics will be
directly

|created. Obviously with a counter starting at 0. That
way, the

|implementation.

|

|

|The explanation above was this idea when Ganesh and
I wrote:

|

|5.4. Flow Expiration

|

|      ...

|

|       3. For long aging
flows, the exporter SHOULD export the flow

|         records
on regular basis, in order to:

|

|         a.
report the flow records periodic accounting information

|
to the collector

|         b.
avoid counter wrapping

|

|
THERE IS A SMALL FORMATTING PROBLEM. THERE SHOULD BE

|A NEW LINE

|
This activity timeout SHOULD be configurable

|

|      4. If the exporter experiences
resources constraints, a flow

|         MAY
be prematurely expired (example: memory)

|

|Regards, Benoit

|

|>

|>

|> accomplish intermediate updates should we go that
way.

|>

|> Paul

|>

|> Benoit Claise wrote:

|> >

|> > Paul,

|> >

|> >

|> > > >

|> > > > Nevil,

|> > > >

|> > > >

|> > > > > Hi Reinaldo:

|> > > > >

|> > > > > >

|> > > > > > This activity SHOULD be configurable. Btw,
can someone

|> > > > > >

|> > > > > > It is very simple to send so much data though
the

|line on a

|> > > > > > flow that the byte counter rolls over.
This is

|very typical

|> > > > > > in 1 gig and 10g  ports. counter 32
is not large enough

|> > > > > >

|> > > > > > yes, that's what I tought. So the solution
is just to

|> > > > > > augment th efield to a 64 counter. It seems
that the

|> > > > > > peirodic reporting is completely orthogonal
to

|this specific

|> > > > > > issue.

|> > > > >

|> > > > > You've covered the 'counter wrapping' problem
- a flow

|> > > > > information meter (whatever that turns out
to be) needs

|> > > > > counters wide enough not to wrap too often,
and that becomes

|> > > > > more of a problem at high link speeds. In
RTFM, all

|counts are

|> > > > > kept in 64-bit unsigned counters so that they
don't wrap too

|> > > > > often.

|> > > > >

|> > > > > Of more interest, I don't remember seeing
any IPFIX

|discussion

|flows is to

|> > > > > be exported.  In particular, I believe
that counters should

|> > > >

|> > > > Why?

|> > > > For long run flows, you can just export the
flow records with

|If you must

|> > > >  - you faced the long aging time timeout
(for example: 30

|> > > > minutes)

|> > > > then you consider the flow to be ended (expired)
and

|> > > > Now a new flow with the same characterics will
be

|> > > > And you start the accounting from 0 for that
"new" flow.

|> > >

|> > >
Most hardware can't do a get and set in one

|> > >
operation. So in between reading the count

|> > >
and setting it to zero you may loose some

|> > >
of the count.

|> >

|> > I'm not too sure about the details of most hardware,
but

|this is the

|> >

|> > Regards, Benoit

|> >

|> > >

|> > >

|> > > > The collector won't be confused at all and don't
need to

|> > > >

|> > > > Regards, Benoit

|> > > >

|> > > > >

|> > > > > That would mean that successive exports of
the same

|flow would

|> > > > > have increasing counter values, a collector
would need to

|> > > > > remember the previous value and subtract it.
Furthermore,

|> > > > > provided unsigned arithmetic is used, the
difference is

|> > > > > correct even if the counter has wrapped.
Provided,

|of course

|> > > > >

|> > > > > Cheers, Nevil

|> > > > >

|> > > > >

|> > > > >    Nevil Brownlee
Director,

|Technology Development

|> > > > >    Phone: +64 9 373 7599 x8941
ITSS, The

|University of Auckland

|> > > > >    FAX: +64 9 373 7021
Private Bag 92019,

|Auckland, New Zealand

|> > > > >

|> > > > > --

|> > > > > Help
mailto:[email-masked]
and

|> > > > > Unsubscribe mailto:[email-masked]
and say

|> > > >

|> > > > --

|> > > > Help
mailto:[email-masked]
and say

|"help" in message body

|> > > > Unsubscribe mailto:[email-masked]
and say

|

|

|--

|Help        mailto:[email-masked]
and say "help"

|in message body

|Unsubscribe mailto:[email-masked]
and say

|

--
Help mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive http://ipfix.doit.wisc.edu/archive/

---

From: [email-masked]
Date: Wed, 27 Feb 2002 16:06:12 -0500

Benoit Claise wrote:
> > You cannot just forget absolute or delta counters.  It is obviously
> > an issue.  It is also not clear in current documentation of flow
> > accounting protocols, and implementation is difficult when you have
> > to guess what is meant. We need to be clear in what we mean.
> >
> >  you were to assume 30 minute intervals, that is asking for
> > trouble.  In the case of things like intrusion detection, you have
> > to shorten the interval time down to 1 to 5 minutes.
> 
> 5.4
>     section 3
> > If a flow is exporting about 5m of packets every 5 minutes and the
> > flow lasts 20 minutes, and we had an interval set of 5 minutes, we
> > would see the byte count during export as 5m, 10m, 15m, 20m,...
> 
>               t1 (minute 0)                 t2 (5th
> minute)                   5m
>               t3 (5th minute)              t4 (10th
> minute)                 5m
>               t5 (10th minute)            t6 (15th
> minute)                 5m
>               t7 (20th minute)            t7 (20th
> minute)                 5m
> 
> Then the collector doesn't have to do diff to find out meaningfull
> values, to find out what happened in the previous 5 minutes
> 
> 

	Agreed. But if the device cannot do the deltas in hardware
	then it is forced to get extra memory for EACH flow so
	it can do the deltas. For situations where the flow
	key is granular I would rather have the Collector keep
	the extra memory per flow since. 

	But you do have an issue on failover. However,
	this may be mitigated by the fact that low granularity flows
	be definition will have less data lost. perhaps an
	acceptable tradeoff is this high volume case. The Collectors
	can of course also solve the problem, but that is out of scope. 

	When the Key is coarse, then I would want to use
	delta counts which help ensure no data loss on
	failover. The volume is also likely to be less
	so if the hardware doesn't support it, a shadow
	software count may be acceptable.

	By having both, the implementor can make the choice.
	
	BTW - I'm all for limiting the number of options where
	possible. But **if** strong, real world arguments can 
	be made for both cumulative and deltas then we should 
	not force one or the other.

> > For this to be meainngful data to a an network analyst, he would
> > want to see that there are 5m bytes going on during this time.
> > Someone needs to give valid representation of this information,
> > either the collector or the exporter.
> 
> > It is a lot easier for the exporter to provide this information.
> > Having the exporter do it also makes sense in the case of collector
> > failover nbeacuse the new collector does not know the history of the
> > flow and would assume tnat when it sees 15m, it really means 15m,
> > not 5 since the last time the exporter sent the flow.
> 
> > -----------------------------------------------------------------------
> >