Subject: [PMOL] Fwd: comments on Framework Metrics draft

From: Al Morton <[email-masked]>
Date: Wed, 04 Mar 2009 22:34:09 -0500

Forwarded on behalf of Loki,

Subject: comments on Framework
Metrics draft

Date: Wed, 4 Mar 2009 13:26:09 -0800

X-MS-Has-Attach:

X-MS-TNEF-Correlator:

thread-topic: comments on Framework Metrics draft

thread-index: Acmc6yDA5SVndh2KQbGG9V2tieZapgAJIcGQ

From: "Loki Jorgenson"
<[email-masked]>

To: <[email-masked]>

Cc: "Alan Clark" <[email-masked]>, "Al
Morton" <[email-masked]>

X-Brightmail-Tracker: AAAAAA==

All - I am aware that the WG last
call closed January 2, 2009 for the Framework for Performance Metrics
Development draft (v0.1).   I am not sure of the utility of any
comments past the expiry time.  I have not seen any mention of it
being submitted.

Regardless, on a recent re-reading,
I noted some additional concerns.   I am posting them in brief
for consideration:

-------------------------------------

Quality of Service/Experience:

There are a variety of references
within the document to "quality of service" (QoS) and several
more for "quality of experience" (QoE).  Further, there
are references to "application performance".  None of
these have been defined and therefore have been accepted with denotative
meaning.  However, these words have broader connotative meaning
within the field and it would be difficult to interpret certain passages
without additional reference.

>>> I would suggest that
they should be more precisely defined.  I can provide some input to
this if it is seen as desirable.

Related point - there are repeated
references to "application performance" as well as references
to "application performance models" (ex. 3.5.3).  These
seem distinct and yet related to QoS and QoE.  Some pundits
(including myself) have begun referring to "quality of
application" (QoA) to distinguish between the effects attributed to
the network layers (QoS), the impacts of the application design and
implementation (e.g. codec), and the experience of the end-user (QoE)
which derives from other influences besides QoS and QoA.   QoA
seems most pertinent here insofar as PMOL aims to capture aspects of what
may be considered application design (choice of buffer, codec,
protocol).

>>> I would suggest
considering the addition of QoA and using it as part of the definition
set above.  Again, I can provide some input if desirable.

Related point - there is a general
suggestion that a proposed metric should be correlated against one of
"quality of service", "quality of experience", or
perhaps "application performance/quality of application" (see
3.2 (ii)) as a "test of usefulness".   Similarly,
3.4.2 (ii) refers to "how this relates to the performance of the
system being measured".  Also, 4.2 refers to metric assessment
on the basis of "correlation with application performance/user
experience".

However there are no further
directions on how that correlation is done, what the criteria are, or how
correlation should be described within a PE entity definition.  For
example using the language referred to above, it may be that a given
metric is considered identical to QoE (e.g. MOS) and thus directly
comparable to end-user experience - correlation would then be exact with
respect to the G.107 implementation and correlate approximately with
standard means for generating user opinion scores.  Alternately, a
given metric may be considered only partially related (e.g. rate of
discards given a fixed buffer size) and could be described as having some
well-defined correlation to QoE or a QoE-specific metric (perhaps simply
a graph of discards to MOS).

>>> I would recommend a
section in the informative part of the metric definition describing any
anticipated relationships or correlations to a quality assessment, and,
where possible, how to confirm that correlation.

3.3.1 - the phrase "each of
which has a greater time span that the original metrics" causes some
confusion on first read.  Upon reflection, and by virtue of the
example provided, the meaning of the qualifier can be determined
correctly.  However, it can be incorrectly read that (each or all
of)  the summarized metrics span a greater time than the source
metrics that generated the summary - which of course they don't - at best
they span the same overall period.  What is intended was something
like "each individual metric value within the summarized set spans a
period of time greater than or equal to each metric value from the
original set".  However, that still may not be the case where
the periods for each metric value are unequal within the original set
and/or where the summarized values may aggregate those original values
unequally.

3.3.2 - The reference to an index
would benefit from an example.  An obvious one would be MOS or
R-value from the Emodel.

3.4.2 (iii) Measurement Method - The
second line suggests "It is important to also define exception
cases...".   I suggest that this should be replaced with
"This SHOULD also define exception cases and how these are
handled".

3.4.2 (iv) Units of Measurement -
"must" --> "MUST"

3.4.2 (v) Measurement Timing -
"must" --> "MUST"

3.4.3 (i) Implementation -
"may" --> "MAY"

3.4.3 (iv) Reporting Model - the
initial statement should rephrased as a "SHOULD"
statement.  Such as "The Reporting Model definition SHOULD
describe any relationship(s) between the metric and the reporting
model"

3.4.4. The "Verification"
section described in 3.4.3 (ii) is missing.

3.4.4  The layout could be
improved for readability - the "Normative" and
"Informative" headers appear as peers of the other
aspects.

3.4.5 Examples - the
"Verification" section appears named as "Conformance
Testing" - this should be made consistent with the balance of the
document.

3.5.3 Relationship .... - as
mentioned above, this section seems centrally important and yet
under-represented in the definition of a PE.  Simply listing this as
a "dependency" seems inconsistent with the other statements
throughout the document.  I would recommend that QoS/QoE be more
explicitly defined, with appropriate examples, possibly with addition of
QoA (if there is support for this), and some sketch at least of how
correlation may be established or can be described.

Loki Jorgenson

Chief Scientist

Apparent Networks

The Hudson House

Suite 400 - 321 Water Street

Vancouver, BC, Canada, V6B 1B8

e   [email-masked]

t   604 433 2333 ext 105

f   604 433 2311

m   604 250-4642

w

www.ApparentNetworks.com