Subject: [PMOL] comments on Framework Metrics draft

From: "Loki Jorgenson" <[email-masked]>
Date: Wed, 4 Mar 2009 09:03:31 -0800

All - I am aware that the WG last call closed January 2, 2009 for the
Framework for Performance Metrics Development draft (v0.1).   I am not
sure of the utility of any comments past the expiry time.  I have not
seen any mention of it being submitted.
 
Regardless, on a recent re-reading, I noted some additional concerns.
I am posting them in brief for consideration:
 
-------------------------------------
 
Quality of Service/Experience:
 
There are a variety of references within the document to "quality of
service" (QoS) and several more for "quality of experience" (QoE).
Further, there are references to "application performance".  None of
these have been defined and therefore have been accepted with denotative
meaning.  However, these words have broader connotative meaning within
the field and it would be difficult to interpret certain passages
without additional reference.
 
>>> I would suggest that they should be more precisely defined.  I can
provide some input to this if it is seen as desirable.
 
Related point - there are repeated references to "application
performance" as well as references to "application performance models"
(ex. 3.5.3).  These seem distinct and yet related to QoS and QoE.  Some
pundits (including myself) have begun referring to "quality of
application" (QoA) to distinguish between the effects attributed to the
network layers (QoS), the impacts of the application design and
implementation (e.g. codec), and the experience of the end-user (QoE)
which derives from other influences besides QoS and QoA.   QoA seems
most pertinent here insofar as PMOL aims to capture aspects of what may
be considered application design (choice of buffer, codec, protocol).
 
>>> I would suggest considering the addition of QoA and using it as part
of the definition set above.  Again, I can provide some input if
desirable.
 
Related point - there is a general suggestion that a proposed metric
should be correlated against one of "quality of service", "quality of
experience", or perhaps "application performance/quality of application"
(see 3.2 (ii)) as a "test of usefulness".   Similarly, 3.4.2 (ii) refers
to "how this relates to the performance of the system being measured".
Also, 4.2 refers to metric assessment on the basis of "correlation with
application performance/user experience".
 
However there are no further directions on how that correlation is done,
what the criteria are, or how correlation should be described within a
PE entity definition.  For example using the language referred to above,
it may be that a given metric is considered identical to QoE (e.g. MOS)
and thus directly comparable to end-user experience - correlation would
then be exact with respect to the G.107 implementation and correlate
approximately with standard means for generating user opinion scores.
Alternately, a given metric may be considered only partially related
(e.g. rate of discards given a fixed buffer size) and could be described
as having some well-defined correlation to QoE or a QoE-specific metric
(perhaps simply a graph of discards to MOS).
 
>>> I would recommend a section in the informative part of the metric
definition describing any anticipated relationships or correlations to a
quality assessment, and, where possible, how to confirm that
correlation.
 
3.3.1 - the phrase "each of which has a greater time span that the
original metrics" causes some confusion on first read.  Upon reflection,
and by virtue of the example provided, the meaning of the qualifier can
be determined correctly.  However, it can be incorrectly read that (each
or all of)  the summarized metrics span a greater time than the source
metrics that generated the summary - which of course they don't - at
best they span the same overall period.  What is intended was something
like "each individual metric value within the summarized set spans a
period of time greater than or equal to each metric value from the
original set".  However, that still may not be the case where the
periods for each metric value are unequal within the original set and/or
where the summarized values may aggregate those original values
unequally.
 
3.3.2 - The reference to an index would benefit from an example.  An
obvious one would be MOS or R-value from the Emodel.
 
3.4.2 (iii) Measurement Method - The second line suggests "It is
important to also define exception cases...".   I suggest that this
should be replaced with "This SHOULD also define exception cases and how
these are handled".
 
3.4.2 (iv) Units of Measurement - "must" --> "MUST"
 
3.4.2 (v) Measurement Timing - "must" --> "MUST"
 
3.4.3 (i) Implementation - "may" --> "MAY"
 
3.4.3 (iv) Reporting Model - the initial statement should rephrased as a
"SHOULD" statement.  Such as "The Reporting Model definition SHOULD
describe any relationship(s) between the metric and the reporting model"
 
3.4.4. The "Verification" section described in 3.4.3 (ii) is missing.
 
3.4.4  The layout could be improved for readability - the "Normative"
and "Informative" headers appear as peers of the other aspects.
 
3.4.5 Examples - the "Verification" section appears named as
"Conformance Testing" - this should be made consistent with the balance
of the document.
 
3.5.3 Relationship .... - as mentioned above, this section seems
centrally important and yet under-represented in the definition of a PE.
Simply listing this as a "dependency" seems inconsistent with the other
statements throughout the document.  I would recommend that QoS/QoE be
more explicitly defined, with appropriate examples, possibly with
addition of QoA (if there is support for this), and some sketch at least
of how correlation may be established or can be described.

Loki Jorgenson
Chief Scientist
Apparent Networks
The Hudson House
Suite 400 - 321 Water Street
Vancouver, BC, Canada, V6B 1B8

e   [email-masked]
t   604 433 2333 ext 105
f   604 433 2311
m   604 250-4642
w   www.ApparentNetworks.com