Subject: [PMOL] Comments on G.1010 for framework metrics draft

From: "Loki Jorgenson" <[email-masked]>
Date: Mon, 6 Apr 2009 09:36:01 -0700

In San Francisco at 74, Al Morton responded to my concerns about the
ambiguity of terms in the framework draft.   Specifically, they included
"application performance" as they pertained to QoS (better defined) and
QoE (somewhat defined).  He suggested that a reference ITU-T G.1010
(2001) might suffice.  I said I would review and comment.
 
Having reviewed G.1010, I suggest that a reference to G.1010 should
appear in the framework draft.  It provides a useful list of categories
that at least suggests how application performance relates to network
QoS.  
 
That said, I still see a clear need for something like a framework
definition for application performance (quality of application) and how
it relates to network QoS.  I won't address that here.
 
In particular, I suggest that section 3.2 be revised somewhat to refer
directly to "application performance" as it related to QoS.  Suggested
changes below (in CAPS) in -02 version of text (with additional
references suggested to G.107 and P.800 for illustration):
 
3.2.  Definitions of a Metric
   A metric is a measure of an observable behavior of an application,
   protocol or other system.  The definition of a metric often assumes
   some implicit or explicit underlying statistical process, and a
   metric is an estimate of a parameter of this process.  If the assumed
   statistical process closely models the behavior of the system then
   the metric is "better" in the sense that it more accurately
   characterizes the state or behavior of the system.

   A metric should serve some defined purpose.  This may include the
   measurement of capacity, quantifying how bad some problem is,
   measurement of service level, problem diagnosis or location and other
   such uses.  A metric may also be an input to some other process, for
   example the computation of a composite metric or a model or
   simulation of a system AS MIGHT BE EMPLOYED TO ASSESS THE 
   PERFORMANCE OF A SPECIFIC APPLICATION - FOR EXAMPLE THE EMODEL FOR
VOIP (G107).
   Tests of the "usefulness" of a metric include:

      (i) the degree to which its absence would cause significant loss
      of information on the behavior or state OR PERFORMANCE of the
application or
      system being measured

      (ii) the correlation between the metric and NETWORK QoS [G1000],
APPLICATION-SPECIFIC
      PERFORMANCE (G1010), OR experience delivered to the user (person
or other application) (P800)

      (iii) the degree to which the metric is able to support the
      identification and location of problems affecting service quality.

   For example, consider a distributed application operating over a
   network connection that is subject to packet loss.  A Packet Loss
   Rate (PLR) metric is defined as the mean packet loss rate over some
   time period.  If the application performs poorly over network
   connections with high packet loss rate and always performs well when
   the packet loss rate is zero then the PLR metric is useful to some
   degree.  Some applications are sensitive to short periods of high
   loss (bursty loss) and are relatively insensitive to isolated packet
   loss events; for this type of application there would be very weak
   correlation between PLR and application performance.  A "better"
   metric would consider both the packet loss rate and the distribution
   of loss events.  If application performance is degraded when the PLR
   exceeds some rate then a useful metric may be a measure of the
   duration and frequency of periods during which the PLR exceeds that
   rate.

Loki Jorgenson
Chief Scientist
Apparent Networks
The Hudson House
Suite 400 - 321 Water Street
Vancouver, BC, Canada, V6B 1B8

e   [email-masked]
t   604 433 2333 ext 105
f   604 433 2311
m   604 250-4642
w   www.ApparentNetworks.com