Subject: [ippm] Comments on draft-ietf-ippm-framework-compagg-04.txt [2]

From: "Alan Clark" <[email-masked]>
Date: Sat, 28 Jul 2007 12:38:01 -0400

Al
 
Outlook decided to send the email before I'd finished it!!
 
The draft has some good definitions of metrics - however to make it a little
more generic and hence capable of wider application it may be useful to make
some minor changes. There is a risk that the draft defines some terminology
within the context of IPPM and packet metrics however the terminology has
broader applicability - hence terms can get "baked in" and referenced in a
broader ranger of applications than those considered during the definition
process.
 
(i) Composite Metric definition - 1
There are many cases in the industry where the term Composite Metric is
used.  One common case is where a metric is constructed from several
dissimilar metrics - for example a simple case is the combined loss/discard
rate in RFC3611, which is not the sum of two packet loss rates but the sum
of a packet loss rate with a jitter buffer packet discard rate.  There are
many more complex cases in common usage.  The text at the present seems to
be more oriented towards composition of identical metrics - either in space
or time.
 
Recommendation: add text to state that composite metrics may be constructed
from either identically defined metrics or from dissimilar metrics.
 
(ii) An "Index"
There are common cases when an artifical index is created - often from a
complex function of some metrics.  The value range of the index typically
has no relationship to any observable parameter - and is selected for
convenience, the function use to calculate the index is then determined.
Obvious telecom examples are MOS scores and R factors. 
 
Recommendation:  Add
3.n Index
An Index is a composite metric for which the output value range has been
selected for convenience or clarity, and the behavior of which is selected
to support ease of understanding.  The composition function for an index is
often developed after the index range and index behavior have been
determined.  Examples include MOS scores and R factors.

(iii) Models and metrics
Many metrics are either implicitly or explicitly the result of assuming a
model of the process being measured, and then estimating the parameters of
the model.  Simple examples include "average packet loss rate", which seems
to assume independent loss probability, RFC3550's PPDV, which seems to
assume alternately early (or on-time) and late packets, and consecutive loss
metrics.  IMHO it is better to explicitly state what model is being used, as
that brings out the assumptions being made and allows their validity to be
questioned.  For example, one of the most common means of reporting
performance is to measure the average of some value during a time interval -
in many cases this is done simply because it is a common thing to do and the
process of analyzing the behavior of the parameter being measured is
skipped.   
 
Models can also be very useful in understanding system behavior and
identifying particularly useful metrics.  I've used the term "intermediate
model" to mean a model that is (a) representative of some key aspect of
performance for an/the application that is using the service/ system being
measured and (b) which is capable of accepting as input data extracted from
the service/ system being measured.  In some senses an intermediate model is
a composition function - however the function has been designed to be
"tuned' for some class of applications.  One example of this is the 4-state
Markov Model used in RFC3611, which captures information on sparse bursts of
lost or discarded packets, another is the MAPDV2 jitter metric, which is
based on a simplified representation of jitter buffer behavior.
 
Recommendation:
 
Add to Section 5. text to indicate that the model (statistical) of the
process being measured and the validity of this assumption should be
explicitly stated
 
Add;
 
Section 4.n Composition Functions based on Intermediate Models
An Intermediate Model is a composite function that is representative of some
key aspect of performance for a class of applications and which is capable
of accepting as input metrics from the system being measured.  
 
Regards
 
Alan
 
_______________________________________________
ippm mailing list
[email-masked] 
https://www1.ietf.org/mailman/listinfo/ippm