Subject: RE: [ippm] draft-shalunov-ippm-reporting-00.txt - availability

From: "Geib, Ruediger" <[email-masked]>
Date: Tue, 2 May 2006 15:53:00 +0200

Hi Al,

|Users experience network performance in at least two states:
|available or "broken" (unavailable) where typically some
|sort of failure or misconfiguration cannot be restored automatically
|and requires human intervention.
|
|The tricky part is defining the performance levels that
|constitute the transitions between the two states,
|and keep them application-independent for simplicity.

Exactly.

|Despite the difficulties, Availability is probably one of
|the most important network metrics, as far as real users
|are concerned.

Yes, it is part of many product descriptions and SLAs.

Regards, 

Rudiger

|At 08:39 AM 5/2/2006, Geib, Ruediger wrote:
|>Stas,
|>
|>Real time and leased line services are characterised by an 
|>"availability". I think availability is a useful and common 
|>means to characterise these services. Today, there's only a 
|>meausurement principle for IP service availability. There's 
|>no commonly implemented IP service availability measurement 
|>(RFC2678 on connectivity proposes some metrics which may be 
|>useful).
|>
|>The idea is that you define a service availability over time. If the 
|>service is available, the metrics you've listed below are applied to 
|>characterise the service. If the service is not available, 
|>then no further performance characterisation is required.
|>
|>As more real time services start to be migrated to the 
|>Internet, measuring IP transport availability may be of interest.
|>
|>Regards,
|>
|>Rudiger
|>
|>
|>Carsten|> * additionally "availability" could be one metric
|>|>   (probably with param = (list of) server(s) or network)
|>
|>Stas|Why aren't existing metrics enough?  When no packets
|>|come through, you'll see
|>|
|>|delay = +infinity
|>|loss = 100%
|>|jitter = undefined
|>|duplication = 0%
|>|reordering = 0%
|>|
|>|Is that not enough?
|>
|>_______________________________________________
|>ippm mailing list
|>[email-masked]
|>https://www1.ietf.org/mailman/listinfo/ippm
|
|
|_______________________________________________
|ippm mailing list
[email-masked] 
|https://www1.ietf.org/mailman/listinfo/ippm
|

_______________________________________________
ippm mailing list
[email-masked] 
https://www1.ietf.org/mailman/listinfo/ippm

---

From: Henk Uijterwaal <[email-masked]>
Date: Tue, 02 May 2006 17:37:19 +0200

Al, Ruediger,

>Yes, it is part of many product descriptions and SLAs.

But don't they (implicitly) define "available" as some mixture of a
number of parameters all being below some threshold?  Isn't it sufficient
to report the numbers, then leave the math to the application?

Henk

------------------------------------------------------------------------------
Henk Uijterwaal                           Email: henk.uijterwaal(at)ripe.net
RIPE Network Coordination Centre          http://www.amsterdamned.org/~henk
P.O.Box 10096          Singel 258         Phone: +31.20.5354414
1001 EB Amsterdam      1016 AB Amsterdam  Fax: +31.20.5354445
The Netherlands        The Netherlands    Mobile: +31.6.55861746
------------------------------------------------------------------------------

1160438400. Watch this space...

---

From: Al Morton <[email-masked]>
Date: Tue, 02 May 2006 15:36:37 -0400

>to report the numbers, then leave the math to the application?

Henk,

In my experience, they explicitly define availability as a
threshold on loss (and that makes it a difficult topic for
us in IPPM to take-up -- defining good/bad is out-of-scope).

But they also define much more:

+  the length of the time window of evaluation

+  whether the window is sliding or fixed

+  how many packets must be sent in each window for valid decision

+  other conditions that affect the reliability of the testing

and many of these details could be standardized.

One of the reasons to define two-states like this is to distinguish
time intervals with failures from "normal" performance when preparing
long-term summaries.  Then users know how much time their A-to-B
avail/connect-ability was present (say 99.9% of time) and what the
overall loss performance was during the fraction of time they could
use their service (they don't particularly care about performance while
the service was broken, or to have that performance affect the long-term
report).

just my 2 cents,
Al

---

From: stanislav shalunov <[email-masked]>
Date: 02 May 2006 16:23:04 -0400

> +  other conditions that affect the reliability of the testing

These things have nothing specific for availability/connectivity
metric.

The draft attempts to set the default values for all relevant
parameters.  Please point out the gaps so I can fill them.

> the service was broken, or to have that performance affect the long-term

With a default measurement interval duration of 10 seconds, the
metrics the draft discusses are hardly long-term.  Description of
long-term network properties is interesting, too, but let's take it
one step at a time: first define how to turn a 100-packet sample into
human-readable measurements, then take it from there...

-- 
Stanislav Shalunov		http://www.internet2.edu/~shalunov/

This message is designed to be viewed at sea level.

---

From: Henk Uijterwaal <[email-masked]>
Date: Wed, 03 May 2006 08:19:44 +0200

Al,

>us in IPPM to take-up -- defining good/bad is out-of-scope).

OK, this is a specific case of "some set of parameters all being below
a certain value".

>+  whether the window is sliding or fixed

Isn't this setting "good or bad"?

>+  how many packets must be sent in each window for valid decision

That is a valid question and probably something to include in the
draft.  (If N out of M packets arrive, how confident can I be that
the loss is below X%.)  Any statistician on the list who can
provide text?

>+  other conditions that affect the reliability of the testing

I think it'd be good to list them (and even throw out the ones that
are not relevant).

>long-term summaries.

I agree with all of this, but I still don't see why availability
has to be in this draft.  If we send X packets/unit of time and
report loss, then the user can do the math for his version of
availability himself using the algorithm he likes.

Henk

---

From: Al Morton <[email-masked]>
Date: Wed, 03 May 2006 09:38:50 -0400

>Isn't this setting "good or bad"?

I don't think so.  It's setting the methods of the measurement
that would serve as a practical approximation to assess
the ideal notion of availabil/continu-ity.

So, maybe this topic is a separate effort?
I was thinking along the same lines on a test stream as
Ruediger suggested:
>Reporting Intervall:     15 min (900 sec)
>Measurement Intervall:   1 sec between probe packets
>Availability definition: 10 consecutive packets lost make any such
>                          ten second interval unavailable

in general, that this test stream may be less dense than
a stream that characterizes key performance parameters.
It would be possible to sub-sample the measurement using a dense
stream, of course, but there is value in having a method
that specifies the "minimum" packet rate to assess availabil/continu-ity.

I was confused by this (from version 02)

>4.1.  One-Way Active Measurement
>
>    The default duration of the measurement interval is 10 milliseconds.
>                                                           ^^^^^^^^^^^^
>    The default sending schedule is a Poisson stream.
>
>    The default sending rate is 10 packets/second on average.

Hopefully this is fixed in 03. Supposed to be 10 seconds, right?

But even then I'm struggling with this statement from Section 2:
>    The set of metrics of this document is meant for human consumption.
>    It must therefore be small.  Anything greater than half-dozen numbers
>    is certainly too confusing.

And the set of metrics is presented to a human every 10 seconds.
That seems like a lot of numbers, more all the time...

So I guess the real intent here is to "roll-up" these 10 second reports
into another report over a longer time frame that is human consumable.
(several folks have said, "take the numbers and do the math", so there's
more work to make a report human-consumable)

But now we're getting into Temporal Aggregation:
  -  we've already chartered that work in IPPM
  -  at least, it adds a requirement here that the metrics be "aggregatable"

(wishing for more time on this, many more comments to make)
Al

---

From: stanislav shalunov <[email-masked]>
Date: 03 May 2006 12:17:11 -0400

> Hopefully this is fixed in 03. Supposed to be 10 seconds, right?

Yes, as you may have noticed, the typo is fixed in 03.

> That seems like a lot of numbers, more all the time...

The set can be derived from a run of a ping-like tool.  In that case,
it's just one set.  If an application such as a game or a VoIP phone
produces it, it just shows the statistics for the last 10 seconds (as
numbers, dial positions, whatever), continually updating them.

> more work to make a report human-consumable)

No.  That's not an intent and, I think, should be separate.

This message is designed to be viewed upside down.

---

From: stanislav shalunov <[email-masked]>
Date: 03 May 2006 12:24:21 -0400

Henk Uijterwaal <[email-masked]> writes:

> (If N out of M packets arrive, how confident can I be that the loss
> is below X%.)

To be able to say things like that, you need an assumption of an
underlying distribution and knowledge of the underlying distribution
(and a few other things).  I would much prefer the statistics to be
empirical properties of the data set.  In that case, when the network
properties change, the statistics can stay the same.

Even if we somehow miraculously agree today that packet delay
singletons are i.i.d. deviates and the underlying distribution is a
power law with given parameter relationships, such agreement or its
implications are unlikely to hold up very long.  Better to make no
assumptions when we can avoid it.

This message is designed to be viewed at room temperature.