Subject: RE: [ippm] draft-shalunov-ippm-reporting-00.txt - availability

From: "Geib, Ruediger" <[email-masked]>
Date: Wed, 3 May 2006 09:39:55 +0200

Stas,

I've added an example with my last response to Henk, hopefully 
clarifying the long term / ten second issue you raise here.

Regards,

Rudiger

[snip]

|> One of the reasons to define two-states like this is to distinguish
|> time intervals with failures from "normal" performance when preparing
|> long-term summaries.  Then users know how much time their A-to-B
|> avail/connect-ability was present (say 99.9% of time) and what the
|> overall loss performance was during the fraction of time they could
|> use their service (they don't particularly care about 
|> performance while the service was broken, or to have that 
|> performance affect the long-term report).
|
|With a default measurement interval duration of 10 seconds, the
|metrics the draft discusses are hardly long-term.  Description of
|long-term network properties is interesting, too, but let's take it
|one step at a time: first define how to turn a 100-packet sample into
|human-readable measurements, then take it from there...

_______________________________________________
ippm mailing list
[email-masked] 
https://www1.ietf.org/mailman/listinfo/ippm

---

From: "Schmoll, Carsten" <[email-masked]>
Date: Wed, 3 May 2006 10:51:29 +0200

> -----Original Message-----
> From: Henk Uijterwaal [mailto:[email-masked]] 
> Sent: Wednesday, May 03, 2006 8:20 AM

> >+  how many packets must be sent in each window for valid decision
> 
> That is a valid question and probably something to include in the
> draft.  (If N out of M packets arrive, how confident can I be that
> the loss is below X%.)  Any statistician on the list who can
> provide text?

I'm not a statistician but I did the math once. I'm not sure
we should start to include probability calculations into the 
draft though, since that is another quite wide field...

To be *able* to check probaility of a threshold violation, if desired,
I would engourage to:

* report the number of probe packets n used together with the report 
  (would this be needed *per* QoS value?)

* and add some text to the draft like e.g.:

"Note that the reported delay and loss values have a limited accuracy
due
to the sampling-like nature of the QoS measurements generating them.

It is possible to estimate the probability of loss/delay not exceeding a
certain threshold from the loss/delay values and the number of probe 
packets used."

Regards,
Carsten.

---

From: "Geib, Ruediger" <[email-masked]>
Date: Wed, 3 May 2006 09:49:50 +0200

Carsten,

while I understand your interest and intention in adding 
"control plane" information to an 
avaialbility/connectivity metric, I side with Stas in 
rejecting the idea for now. Let's deal with the relatively 
simple and basic IP connectivty and performance first.

DHCP, DNS, RADIUS.....may each define their own connectivity 
and performance metrics. The IPPM charter however doesn't 
seem to include more then basic data delivery by now. If 
interest is there, it may be enhanced. I personally prefer 
the current chartered tasks to be done prior to this.

Regards

Rudiger

|Still I think connectivity deserves an extra report entry,
|since it might be "FALSE" for different reasons which cannot
|be derived from delay etc. For a user the reasons are of interest.
|
|Think about connectivity *to the Internet* as a desired metric;
|this can be FALSE due to a lot of things, e.g.
|the 1st L2 link being down, DHCP, DNS, routing, firewall, or
|gateway failure.
|
|Could the metric "connectivity to the Internet" be something more
|than binary and could it give a user a value stating either
|"OK", or some value indicating ''how far'' connectivity is from being
|OK?
|
|If connectivity is just a binary metric then I think it could 
|be derived
|
|from the other reported metrics (if we just talk about Layer3
|connectivity)
|as long as the destination parameter is the same.
|
|What do you think about a finer granularity for connectivity?

---

From: "Geib, Ruediger" <[email-masked]>
Date: Thu, 4 May 2006 10:02:16 +0200

|From: stanislav shalunov 
|
|
|
Henk|> (If N out of M packets arrive, how confident can I be that the loss
|> is below X%.)
|
Stas| To be able to say things like that, you need an assumption of an
|underlying distribution and knowledge of the underlying distribution
|(and a few other things).  I would much prefer the statistics to be
|empirical properties of the data set.  In that case, when the network
|properties change, the statistics can stay the same.
|
|Even if we somehow miraculously agree today that packet delay
|singletons are i.i.d. deviates and the underlying distribution is a
|power law with given parameter relationships, such agreement or its
|implications are unlikely to hold up very long.  Better to make no
|assumptions when we can avoid it.

I agree with Stas' assessment. 

Regards,

Rudiger

---

From: "Geib, Ruediger" <[email-masked]>
Date: Fri, 5 May 2006 08:23:35 +0200

Stas, Al,

good conclusion, I wanted to add a similar request. Doing a 
generic differentiation between short term reports and 
temporal aggregation of metrics is allright and both 
documents may clarify what they are about in their scope.

Regards,

Rudiger

|> I think the most important clarification you can make at this point
|> is to add a Scope section to the draft.
|
|You're quite right: the scope of the current document is not clear.  I
|see the source of confusion now.  I'll work something out and include
|it in the next version.

---

From: "Schmoll, Carsten" <[email-masked]>
Date: Mon, 8 May 2006 12:33:25 +0200

> Sent: Friday, May 05, 2006 4:05 PM
> To: Geib, Ruediger
> Cc: [email-masked]
> Subject: RE: [ippm] draft-shalunov-ippm-reporting-00.txt - 
> availability
> 
> 
> I understand that but that wasn't the question.  I asked if 98.3% is
> consistent with the claim of 99.9%.  It is a similar problem to: if I
> flip a coin a lot of times, I get heads half the time, tails the other
> half.  I now flip the coin only 10 times and get 6 heads and 4tails.
> Is the coin biased or is this due to the limited size of my sample.

The answer is: you absolutely cannot tell with 10 coin flips.

In general you would do a Hypothesis test to check this to answer
the question: Is the deviation 6:4 (instead of the expected 5:5) 
a *significant* deviation considering 10 coin flips?

High Significance of an event means: It is very *unlikely* to occur by 
chance. In the above case a 6:4 is nothing special, since only 24.6%
of all results using 10 coin flips result in a 5:5 result and 6:4 + 4:6
results make up for 41% of all results (based on Binomial distribution).
So even the chance for something more unbalances than 6:4 as a result 
is still larger than 34%!

An event is usually said to be significant if the probability of getting
it via chance alone is less than 5%.

Hope this helped a litte,
Carsten.

---

From: "Geib, Ruediger" <[email-masked]>
Date: Wed, 10 May 2006 12:04:51 +0200

Henk,

H>| Example: I want that packets sent by my VoIP phone arrive
|>| about 99.9% of the time, or roughly 3599 seconds every hour. 
|>| I measure by sending probe packets at a rate of 1/minute.
|>| In a particular hour, 1 packet gets lost, or 1/60.  Is 
|>| that consistent with 99.9% or not?
|>
R>  No, 98,3% within one hour. Your resolution can't be better 
|>  than 1,7% here.
|
H |I understand that but that wasn't the question.  I asked if 98.3% is
|consistent with the claim of 99.9%.  It is a similar problem to: if I
|flip a coin a lot of times, I get heads half the time, tails the other
|half.  I now flip the coin only 10 times and get 6 heads and 4tails.
|Is the coin biased or is this due to the limited size of my sample.
|
|In general, I don't want to fill my network with probe packets, so the
|frequency of probe packets may be a lot lower and I still want to say
|something about the network.

In the case of your coin experiment, 50% (5/5) would have been a valid 
result of your experiment and you can calculate an expecation of 
this outcome. Your experiment has an error, but it is relatively 
small.

If you send less than 1000 probes, 99.9% free of loss isn't possible 
as a result. You can estimate if 99.9% free of loss can be proved 
by your experiment, but you will have a considerable error.

R>Your proposal to work with a loss rate instead of consecutive 
|>losses is allright to me. The fundamental mechanisms of an 
|>availability/connectivity measurement must be agreed.
|>
|>The data amaount of a report may be reduced (that's why I proposed 15
|>minutes). 10 second reports may not be desireable. One second reports
|>certainly aren't.
|
H|I would not report the data for each 10 second interval to the user,
|just count the number of such intervals in a longer period and report
|that.

Yes, that's reasonable and sound.

Regards, 

Rudiger