
Message-Id: <200911301443.nAUEhYTO032280@alpd052.aldc.att.com>
Date: Mon, 30 Nov 2009 09:42:45 -0500
To: Benoit Claise <bclaise@cisco.com>, pmol@ietf.org
From: Al Morton <acmorton@att.com>
In-Reply-To: <4AE5ADD7.7020304@cisco.com>
References: <4AE5ADD7.7020304@cisco.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"; format=flowed
Subject: Re: [PMOL] [Fwd: New Version Notification for draft-ietf-pmol-metrics-framework-03]
Precedence: list

At 09:10 AM 10/26/2009, Benoit Claise wrote:
>I have integrated many suggested improvements compared to the 
>previous version (which expired).
>As you can see there is a TO DO list, and some EDITOR'S NOTES 
>throughout the document.
>All comments, feedbacks, improvements, text are welcome.

Hi Benoit,

First and foremost, thanks for taking-on the editing role so the
WG can finish this draft.

I have a few comments to share on the latest version,
(as a participant).
Al

>1.  TO DO
>
>    o  Multiple EDITOR'S NOTES throughout the document
>
>    o  Should we refer to ITU G1010 for application performance metrics?

G.1010 mentions only the most basic performance metrics
at the User-layer: Delay, Delay Variation, and Information Loss.
It goes on to categorize various applications in terms of their
needs for Delay and Information loss, as approximate numerical
requirements. In this framework, I don't think the scope includes
the User layer, so referencing G.1010 might cause some confusion.

However, there might be some specific context where a ref in
appropriate...


>    o  Do we want a definition for Performance Metric Entity?

I think it makes sense to say what the entity evolved into, IMO.
I described this as a way to proceed when we met briefly in Stockholm,
but let me see if this sounds reasonable to the wider community.

It seems to me that the simplest way to have such an entity is
to view it as a performance community, composed of folks who
work in IPPM, BMWG, and PMOL (while it lasts). The organizers are
the chairs of all the WGs, and when a request for review comes
to one or more of the WGs/Chairs, we jointly find one or more
people to provide a review & comments. The PMOL Framework Document
should be consulted as part of the review, similar to the way
that http://tools.ietf.org/html/rfc5706 guides OPS and Management
Directorate Reviews.

S 2.1
...
>    The IETF is also actively involved in the development of reliable
>    transport protocols which would affect the relationship between IP
>    performance and application performance.
>
>    EDITOR'S NOTE: I'm not sure what the previous sentence refers to?

TCP is the most obvious example, but FECFRAME and RMD are other
relevant IETF wgs.

>    Thus there is a gap in the currently chartered coverage of IETF WGs:
>    development of performance metrics for non-IP-layer protocols that
>    can be used to characterize performance on live networks.
>
>    EDITOR'S NOTE: must extend on the "non-IP-layer".  Could be above L4
>    such as voice specific metrics, but also L2 such as (G)MPLS

Yes, could simply say;
... performance metrics for protocols above and below the IP-layer that ...

>3.1.  Quality of Service
>
>    The Quality of Service (QoS) is defined similarly to the ITU "QoS
>    experienced/perceived by customer/user (QoSE)" E800 [E800], i.e.:
>    Totality of characteristics of a telecommunications service that bear
>    on its ability to satisfy stated and implied needs of the user of the
>    service.
The last sentence above is also a quote, isn't it?

>3.3.  Quality of Experience
>
>    The Quality of Experience (QoE) is defined similarly to the ITU "QoS
>    experienced/perceived by customer/user (QoSE)" E800 [E800], i.e.: a
>    statement expressing the level of quality that customers/users
>    believe they have experienced.
>
>    NOTE 1 - The level of QoS experienced and/or perceived by the
>    customer/user may be expressed by an opinion rating.
>
>    NOTE 2 - QoSE has two main man components: quantitative and
>    qualitative.  The quantitative component can be influenced by the
>    complete end-to-end system effects (network infrastructure).
>
>    NOTE 3 - The qualitative component can be influenced by user
>    expectations, ambient conditions, psychological factors, application
>    context, etc.
>
>    NOTE 4 - QoSE may also be considered as QoSD received and interpreted
>    by a user with the pertinent qualitative factors influencing his/her
>    perception of the service.

SG 2 chose to use an odd acronym for QoE, that was
QoSE for Quality of Service - Experienced.  I recommend that we adapt these
definitions to the more widely accepted acronym, QoE.
So,  s/QoSE/QoE/ above.

Also, QoSD is Quality of Service - Delivered, so I suggest to
change Note 4 to
    NOTE 4 - QoSE may also be considered as QoS delivered, received, 
and interpreted
    by a user with the pertinent qualitative factors influencing his/her
    perception of the service.

>4.  Purpose and Scope
>
>    The purpose of this document is to define a framework and a process
>    for developing performance metrics for IP-based applications that
>    operate over reliable or datagram transport protocols, and that can
>    be used to characterize traffic on live networks and services.  As
>    such, this document will not define any performance metrics.

Here's another place where we can make the above and below IP-layer
aspect clear:

    The purpose of this document is to define a framework and a process
*  for developing performance metrics for protocols above and below the
*  IP-layer (such as IP-based applications that
*  operate over reliable or datagram transport protocols), that can
    be used to characterize traffic on live networks and services.  As
    such, this document will not define any performance metrics.

>5.  QoS versus Application Performance Metrics versus QoE
I suggest:
5.  Relationships between QoS, Application Performance Metrics, and QoE

Some more suggested edits below (as replacement for the text above):
>    QoS deals with the network and protocol,

Network QoS deals with network and networking protocol performance,

>while QoE deals with the
>    notion of a user

...deals with the assessment of a user's experience

>  in a context of a task or a service.

>As a
>    consequence, QoE leads to the notion of Application Performance
>    Metrics.

As a result, the topic of Application metrics includes the
opportunities to quantify performance at layers between IP and the User.

>For example, QoS performance metrics contain the one-way
>    delay and the delay variation RFC 5481 [RFC5481] and the Mean Opinion
>    Score (MOS) P.800 [P.800] can be modelled and calculated as an
>    Application Performance Metric for multimedia applications.

For example, Network QoS metrics (packet loss, delay, and delay variation)
can be used to estimate application performance metrics (de-jitter 
buffer size and RTP-layer packet loss), then combined with other 
known aspects of
a VoIP application (such as codec type) to estimate a Mean Opinion Score
(MOS).

>However,
>    the MOS for a particular user in the specific context such as a

However, the QoE for a particular VoIP user depends on the specific context,
such as a

>    conference call, an IPTV or an emergency call are different QoE's.

casual conversation, a business conference call, or an emergency call.

>    Finally, QoS and Application Performance Metrics are quantitative,
>    while QoE is qualitative.
(add, not replace)
Also, Network QoS and Application performance metrics can be directly or
indirectly evident to the user, while QoE is directly evident.


>6.3.1.  Composed Metrics
...
>    EDITOR'S NOTE: review draft-ietf-ippm-framework-compagg-08.txt and
>    determine is something should be added in this section
I think the key definitions are already included.


6.4.2
>    (iii) Collection Method
>
>    EDITOR'S NOTE: remove "measurement" in "measurement method" as this
>    this method can be measured, estimated or computed".  Looking for a
>    generic term -> collection method?  Do we want to change from
>    measurement to collection all over?

I think "Method of Measurement or Calculation" will work best here.
Estimation is nominally a calculation. This section needs to prescribe
the method in a general flow, or as a series of operations or steps.
"Collection" can easily be confused with the operation to access
the results of measurement, and that's not what we want...

>7.  Performance Metric Development Process

I suggest to add:
This process is intended to add additional considerations to the
processes for adopting new work as described in RFC 2026 and RFC 2418.