Subject: RE: [ippm] draft-shalunov-ippm-reporting-00.txt

From: "Schmoll, Carsten" <[email-masked]>
Date: Thu, 20 Apr 2006 19:59:06 +0200

Dear Stanislav,

I read your draft and I think it is going in the 
right direction. Some suggestions for the next 
version from my point of view are:

* Motivation: should state some application areas, i.e.
  "why would a human user want to get those values" - e.g.
  'to check that IP telephony would be feasable with available QoS'

* please don't be offended, but I disagree on the traffic 
  statistics you plan to apply to the traffic metrics.
  To me they seem to be quite unusual. see below:

* delay: I would suggest to use mean or a high percentile as
  I have never seen median delay to be used to report QoS

* loss: just a typo = per cent -> percent

* jitter: as far as I know the two common defintions are 
  (high) percentile minus min delay across some interval
  (ITU approach) or min/max of IP delay variation, taken from
  the series of differences of consecutuve OWD values (IETF approach)

* duplication: just lacks definition of a default timeout value

* additionally I think two things will need to be added to the draft:
  a) mentioning about the size of the interval about which all
     those metrics are obtained and the fact whether this will be 
     a sliding window or fixed time slots, e.g. new values each 10sec
  b) I'd personally like to see some "packet filter" added to the
     reporting record. Data might relate only to a fraction of all
     traffic, e.g. only to UDP or only to traffic from videos.fun.org

* additionally "availability" could be one metric
  (probably with param = (list of) server(s) or network)

* a small final note (not specific to the draft): when applying the 
  ITU definition for jitter, then high percentile of delay and jitter
  are quite correlated, and just differ by the interval's min-delay.
  I cannot say if/how they correlate when the IETF definition (IPDV) is
used.

* one question from my side: how would you define/report the measurement
parameters?
  (i.e. the src/dst/network/path to which the reported delay/jitter etc.
applies)

With best regards,
Carsten.

> -----Original Message-----
> From: stanislav shalunov [mailto:[email-masked]] 
> Sent: Monday, April 10, 2006 4:16 AM
> To: [email-masked]
> Subject: [ippm] draft-shalunov-ippm-reporting-00.txt
> 
> I did not see this announcement on the IPPM list.
> 
> [email-masked] writes:
> 
> > Network measurement has many purposes.  Often, the ultimate purpose
> > is to report a concise set of metrics describing a 
> network's state to
> > an end user.  The aim of this document is to define a small set of
> > metrics that are robust, easy to understand, orthogonal, relevant,
> > and easy to compute.
> > 
> > A URL for this Internet-Draft is:
> > 
> http://www.ietf.org/internet-drafts/draft-shalunov-ippm-report
ing-00.txt
> 
> -- 
> Stanislav Shalunov		http://www.internet2.edu/~shalunov/
> 
> Al your Qaeda are belong to US.
> 
> _______________________________________________
> ippm mailing list
> [email-masked] 
> https://www1.ietf.org/mailman/listinfo/ippm
> 
> 

_______________________________________________
ippm mailing list
[email-masked] 
https://www1.ietf.org/mailman/listinfo/ippm

---

From: Henk Uijterwaal <[email-masked]>
Date: Fri, 21 Apr 2006 10:51:03 +0200

Hi all,

>   'to check that IP telephony would be feasable with available QoS'

I agree with this.

>   I have never seen median delay to be used to report QoS

We have had this discussion before and consensus seems that median (or
percentiles in general) seem the most appropriate way to report delays.
We should probably point to this discussion.

Carsten: If median cannot be used, then please explain why.

>* duplication: just lacks definition of a default timeout value

I think we should define one time-out value for delay, loss and
duplication.

>      traffic, e.g. only to UDP or only to traffic from videos.fun.org

The way I understood the set, is that this refers to 1 pair of
source-destination, that is, you get 5 numbers for traffic between src1
and dst1, then 5 other numbers for src1 to dst2.

>   (probably with param = (list of) server(s) or network)

There is an availability metric but I don't think anybody has implemented
it.  I don't see the need in this set of 5 either, loss and delay already
tell you that the dst can see the src.  "availability" would just confirm
that.

Henk

------------------------------------------------------------------------------
Henk Uijterwaal                           Email: henk.uijterwaal(at)ripe.net
RIPE Network Coordination Centre          http://www.amsterdamned.org/~henk
P.O.Box 10096          Singel 258         Phone: +31.20.5354414
1001 EB Amsterdam      1016 AB Amsterdam  Fax: +31.20.5354445
The Netherlands        The Netherlands    Mobile: +31.6.55861746
------------------------------------------------------------------------------

1160438400. Watch this space...

---

From: stanislav shalunov <[email-masked]>
Date: 25 Apr 2006 18:23:27 -0400

> version from my point of view are:

Carsten,

Thank you for the thoughtful review.  In my response, unless specified
otherwise, unidiff format means changes I made to the document to
accommodate your concern.

>   'to check that IP telephony would be feasable with available QoS'

@@ -68,7 +68,28 @@
 Such a set would enable different tools to produce results that can be
 compared against each other.</t>
 
-<t>The set is meant for human consumption.  It must therefore be
+<t>Existing tools already report statistic about the network.  This is
+done for varying reasons: network testing tools, such as the ping
+program available in UNIX-derived operating systems as well as in
+Microsoft Windows, report statistics with no knowledge of why the user
+is running the program; networked games might report statistics of the
+network connection to the server so users can better understand why
+they get the results they get (e.g., if something is slow, is this
+because of the network or the CPU?), so they can compare their
+statistics to those of others (``you're not lagged any more than I
+am'') or perhaps so that users can decide whether they need to upgrade
+the connection to their home; IP telephony hardware and software might
+report the statistics for similar reasons.  While existing tools
+report statistics all right, the particular set of metrics they choose
+is ad hoc; some metrics are not statistically robust, some are not
+relevant, and some are not easy to understand; more important than
+specific shortcomings, however, is the incompatibility: even if the
+sets of metrics were perfect, they would still be all different, and,
+therefore, metrics reported by different tools would not be
+comparable.</t>
+
+<t>The set of metrics of this document is meant for human consumption.
+It must therefore be
 small.  Anything greater than half-dozen numbers is certainly too
 confusing.</t>

>   I have never seen median delay to be used to report QoS

Henk already responded to this.

Mean of delay is undefined, infinite, or unknown (depending on how you
look at it) for most samples.  Reporting invalid statistics can't be a
good strategy.  The only thing mean has going for it is that it's
trivial to compute.  Luckily, percentiles can be computed
approximately with good efficiency, too.  In addition, mean, even when
defined, is not robust.

I fail to see advantage of high percentile over median.  A high
percentile will be more frequently infinite, is less robust, is harder
to understand, and, in general, has no advantages in relevancy,
orthogonality, or ease of computation [1].

> * loss: just a typo = per cent -> percent

Actually, it was not a typo, but if it causes confusion, then

-<t>Loss is the fraction, expressed in per cent, of packets that did
+<t>Loss is the fraction, expressed as a percentage, of packets that did

>   the series of differences of consecutuve OWD values (IETF approach)

The definition of the document fits into the very general IPDV
framework (with an appropriate selection function, if I remember IPDV
terminology right).

The use of 0th percentile (minimum) in inter-percentile spread is,
indeed, appealing.  However, it is appealing for reasons that might
turn out to be temporary: namely, the main reason to use a min is that
it the maximum likelihood estimator of propagation delay in network
models where propagation delay is fixed and queuing delay is
distributed so that there's significant clumping at 0 (ddf(x) -->
+infinity as x --> +0 is a necessary condition, but almost certainly
not sufficient; shifted exponential distribution works as an example
where minimum, is, indeed, the maximum likelihood estimate of the
shift parameter).  Maximum likelihood estimators are asymptotically
consistent, and this makes minimum attractive.  However, two
considerations made me reject it:

(1) While one end of the spread starts to mean something, the other is
    just as arbitrary.  Using maximum as the other end is clearly a
    poor choice, so symmetry must be broken.  This makes the metric
    harder to understand.

(2) In today's world, indeed, the minimum delay is a decent
    propagation delay estimate.  We don't know whether it'll remain
    this way.  For example, a world with most traffic going between
    high-speed wireless points (e.g., where a ubiquitous mode of
    connectivity is through a low-orbit satellites) has little use for
    minimum delay.  In such a world minimum breaks.  On the other
    hand, interquartile spread is a well-understood metric that is
    robust without strong assumptions about the nature of the
    distribution.

> * duplication: just lacks definition of a default timeout value

 <t>Duplication is the fraction of packets for which more than a single
-copy of the packet was received within the timeout period, expressed
+copy of the packet was received within the timeout period (same
+timeout as in the definition of loss), expressed
 in percentage points.</t>

>      traffic, e.g. only to UDP or only to traffic from videos.fun.org

+<section title="Sample Source">
+
+<t><xref target="sec-metrics"/> describes the metrics to compute on a
+sample of measurements.  The source of the sample in not discussed
+there, and, indeed, the metrics discussed (delay, loss, etc.) are
+simply estimators that could be applied to any sample whatsoever.  For
+the names of the estimators to be applicable, of course, the
+measurements need to come from a packet delivery network.</t>
+
+<t>The data in the samples for the set of metrics discussed in this
+document can come from the following sources: one-way active
+measurement, round-trip measurement, and passive measurement.  There
+infrequently is a choice between active and passive measurement, as,
+typically, only one is available; consequently, no preference is given
+to one over the other.  In cases where clocks can be expected to be
+synchronized, in general, one-way measurements are preferred over
+round-trip measurements (as one-way measurements are more
+informative).  When one-way measurements cannot be obtained, or when
+clocks cannot be expected to be synchronized, round-trip measurement
+MAY be used.</t>
+
+<section title="One-Way Active Measurement" anchor="sec-one-way">
+
+<t>The default duration of the measurement interval is 10 milliseconds.</t>
+
+<t>The default sending schedule is a Poisson stream.</t>
+
+<t>The default sending rate is 10 packets/second on average.  When
+randomized schedules, such as a Poisson stream, are used, the rate
+MUST be set with the distribution parameter(s).</t>
+
+<t>The default packet size is the minimum necessary for the
+measurement.</t>
+
+<t>Values other than the default ones MAY be used; if they are used,
+their use, and specific values used, MUST be reported.</t>
+
+<t>A one-way active measurement is characterized by the source IP
+address, the destination IP address, and the time when measurement was
+taken.  For the time, the middle of the measurement interval MUST be
+reported.</t>
+
+</section>
 
+<section title="Round-Trip Active Measurement">
+
+<t>The same default parameters and characterization apply to
+round-trip measurement as to one-way measurement (<xref
+target="sec-one-way"/>).</t>
+
+</section>
+
+<section title="Passive Measurement">
+
+<t>Passive measurement use whatever data it is natural to use.  For
+example, an IP telephony application or a networked game would use the
+data that it sends.  An analysis of performance of a link might use
+all the packets that traversed the link in the measurement interval.
+An analysis of performance of an Internet service provider's network
+might use all the packets that traversed the network in the
+measurement interval.  An analysis of performance of a specific
+service from the point of view of a given site might use an
+appropriate filter to select only the relevant packets.</t>
+
+<t>The same default duration applies to passive measurement as to
+one-way active measurement (<xref target="sec-one-way"/>).</t>
+
+<t>When the passive measurement data is reported in real time, a
+sliding window SHOULD be used as a measurement period, so that recent
+data become more quickly reflected.</t>
+
+</section>
+
+</section>

>   (probably with param = (list of) server(s) or network)

Why aren't existing metrics enough?  When no packets come through,
you'll see

delay = +infinity
loss = 100%
jitter = undefined
duplication = 0%
reordering = 0%

Is that not enough?

>   I cannot say if/how they correlate when the IETF definition (IPDV) is

The median and the interquartile spread are sufficiently orthogonal
and widely used by statisticians.

>   (i.e. the src/dst/network/path to which the reported delay/jitter etc.

I believe this is covered now.

Thanks again,                                           --Stanislav

P.S. I'll send out a version of the draft with these changes rolled in.

[1] Well, OK, the 100th percentile (maximum) *is* easier to compute,
but it is clearly unusable otherwise, as it has all the deficiencies
of mean, only to a much larger extent.

This message is designed to be viewed in boustrophedon.

---

From: stanislav shalunov <[email-masked]>
Date: 25 Apr 2006 18:26:15 -0400

> >   'to check that IP telephony would be feasable with available QoS'

Addressed.

> Carsten: If median cannot be used, then please explain why.

Agreed.

> I think we should define one time-out value for delay, loss and

I believe this is what we have now.

> and dst1, then 5 other numbers for src1 to dst2.

This is addressed now.  (The applicability, as written, is more
general than that.)

> tell you that the dst can see the src.  "availability" would just confirm

Agreed.

This message is designed to be viewed with 0.06479891g of NaCl.

---

From: Al Morton <[email-masked]>
Date: Wed, 26 Apr 2006 11:44:36 -0400

At 06:26 PM 4/25/2006, stanislav shalunov wrote:
> > > [Carsten wrote:]
> > [Henk wrote:]

Can someone provide the pointer to this discussion?

thx,
Al

---

From: stanislav shalunov <[email-masked]>
Date: 27 Apr 2006 00:27:02 -0400

> Can someone provide the pointer to this discussion?

I can't seem to find the pointers, but the fact that mean of observed
values is not the best estimator has been rehashed a few times.  One
notable outcome is the prominent absence of mean delay from RFC2679.
I'll write up the usual boring reasons why mean delay is an invalid
metric, and why mean of the observed singleton delays is not very
useful so that the writeup is more permanent than the back-and-forth
of the mailing list.

A fool's brain digests philosophy into folly, science into superstition,
and art into pedantry.  Hence University education.       -- G. B. Shaw

---

From: stanislav shalunov <[email-masked]>
Date: 27 Apr 2006 14:57:48 -0400

> useful so that the writeup is more permanent than the back-and-forth

As promised,
http://www.internet2.edu/~shalunov/writing/mean-delay-considered-harmful.html

Democracy is a form of government that substitutes election by the
incompetent many for appointment by the corrupt few.  -- G. B. Shaw