Subject: [PMOL] [ippm] WGLC on pmol-metrics-framework-01
From: "Fardid, Reza" <[email-masked]>
To: <[email-masked]>
Date: Tue, 16 Dec 2008 11:37:24 -0800
List-Id: Performance Metrics at Other Layers <pmol.ietf.org>

Hi,

I have a number of comments on the PMOL framework draft:

1.1 Since RAQMON is mentioned, adding an Informative reference to its
framework is appropriate.

3.3.2 Since Index is a derived metric, it is best to specify how its
components contribute to its measurement beyond a function definition.
An index composed of N orthogonal metrics all of which are measurable
has more "usefulness" than an index whose measurement is based on a
small subset of these N metrics.

3.4.2 (vi) Does measurement point define the measurement domain? If not,
then measurement domain (intra-, inter-) is another factor to consider.
Security restrictions may prevent inter-domain measurement of certain
metrics.

3.4.3 Are both active and passive measurement types within scope? If so,
then measurement type is another factor to consider.

3.5 Some metrics and their measurements may be affected by middleboxes.
I can think of the following:

3.5.3 Middlebox presence

Presence of a middlebox, e.g., proxy, NAT, redirect server, session
border controller (SBC), and application layer gateway (ALG) may add
variability to or restrict the scope of measurements of a metric.  For
example, an SBC that does not process RTP loopback packets may block or
locally terminate this traffic rather then pass it through to its
target.

Thanks,
Reza Fardid

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Al Morton
Sent: Tuesday, December 02, 2008 7:11 AM
To: [email-masked]
Cc: [email-masked]; [email-masked]
Subject: [ippm] WGLC on pmol-metrics-framework-01

PMOL WG,
cc IPPM and BMWG WGs,

This message begins a WG Last Call on the following draft:

        Title           : Framework for Performance Metric Development
	Author(s)       : A. Clark
	Filename        : draft-ietf-pmol-metrics-framework-01.txt
	Pages           : 14
	Date            : 2008-11-02

This memo describes a framework and guidelines for the development of
performance metrics that are beyond the scope of existing working
group charters in the IETF.  In this version, the memo refers to a
Performance Metrics Entity, or PM Entity, which may in future be a
working group or directorate or a combination of these two.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-pmol-metrics-framework-01
.txt

There has been discussion of this draft for over a year,
mostly on the PMOL list and at meetings.  The Author believes he
has addressed all comments to date.

Please weigh-in on whether or not this Draft should be forwarded
to the Area Directors for publication as a Standards Track RFC.
Send your comments to the PMOL list or the co-chairs.

The Last Call will be open till January 2, 2009.

thanks for your review and comments,
Al Morton
co-chair, PMOL WG

_______________________________________________
ippm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ippm
_______________________________________________
PMOL mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/pmol

Content-class: urn:content-classes:message
Date: Mon, 15 Dec 2008 15:48:45 -0800
From: "Fardid, Reza" <[email-masked]>
To: "Al Morton" <[email-masked]>, <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: [PMOL] [ippm] WGLC on pmol-metrics-framework-01
Sender: [email-masked]

Hi,

I have a number of comments on the PMOL framework draft:

1.1 Since RAQMON is mentioned, adding an Informative reference to its
framework is appropriate.

3.3.2 Since Index is a derived metric, it is best to specify how its
components contribute to its measurement beyond a function definition.
An index composed of N orthogonal metrics all of which are measurable
has more "usefulness" than an index whose measurement is based on a
small subset of these N metrics.

3.4.2 (vi) Does measurement point define the measurement domain? If not,
then measurement domain (intra-, inter-) is another factor to consider.
Security restrictions may prevent inter-domain measurement of certain
metrics.

3.4.3 Are both active and passive measurement types within scope? If so,
then measurement type is another factor to consider.

3.5 Some metrics and their measurements may be affected by middleboxes.
I can think of the following:

3.5.3 Middlebox presence

Presence of a middlebox, e.g., proxy, NAT, redirect server, session
border controller (SBC), and application layer gateway (ALG) may add
variability to or restrict the scope of measurements of a metric.  For
example, an SBC that does not process RTP loopback packets may block or
locally terminate this traffic rather then pass it through to its
target.

Thanks,
Reza Fardid

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Al Morton
Sent: Tuesday, December 02, 2008 7:11 AM
To: [email-masked]
Cc: [email-masked]; [email-masked]
Subject: [ippm] WGLC on pmol-metrics-framework-01

PMOL WG,
cc IPPM and BMWG WGs,

This message begins a WG Last Call on the following draft:

        Title           : Framework for Performance Metric Development
	Author(s)       : A. Clark
	Filename        : draft-ietf-pmol-metrics-framework-01.txt
	Pages           : 14
	Date            : 2008-11-02

This memo describes a framework and guidelines for the development of
performance metrics that are beyond the scope of existing working
group charters in the IETF.  In this version, the memo refers to a
Performance Metrics Entity, or PM Entity, which may in future be a
working group or directorate or a combination of these two.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-pmol-metrics-framework-01
.txt

There has been discussion of this draft for over a year,
mostly on the PMOL list and at meetings.  The Author believes he
has addressed all comments to date.

Please weigh-in on whether or not this Draft should be forwarded
to the Area Directors for publication as a Standards Track RFC.
Send your comments to the PMOL list or the co-chairs.

The Last Call will be open till January 2, 2009.

thanks for your review and comments,
Al Morton
co-chair, PMOL WG

_______________________________________________
ippm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ippm
_______________________________________________
PMOL mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/pmol

Date: Fri, 12 Dec 2008 01:31:29 -0500
To: [email-masked]
From: Al Morton <[email-masked]>
Subject: [PMOL] Fwd: RE:  WGLC on pmol-metrics-framework-01
Sender: [email-masked]

Forwarded at Loki's request...

>X-VirusChecked: Checked
>X-Env-Sender: [email-masked]
>X-Msg-Ref: server-13.tower-121.messagelabs.com!1228755277!24935304!1
>X-StarScan-Version: 5.5.12.14.2; banners=-,-,-
>X-Originating-IP: [209.139.228.52]
>X-SpamReason: No, hits=0.5 required=7.0 tests=BODY_RANDOM_LONG
>X-AuditID: ac108108-00000e4c00000774-d5-493d514d1d00
>Subject: RE: [PMOL] WGLC on pmol-metrics-framework-01
>Date: Mon, 8 Dec 2008 08:54:35 -0800
>X-MS-Has-Attach:
>X-MS-TNEF-Correlator:
>thread-topic: [PMOL] WGLC on pmol-metrics-framework-01
>thread-index: AclUuJeWbGY6AyloRw+sWjN4K8ltvABneGJgAL/HSyA=
>From: "Loki Jorgenson" <[email-masked]>
>To: "Al Morton" <[email-masked]>
>X-Brightmail-Tracker: AAAAAA==
>
>Al - any reason why my post to PMOL did not make it onto the list?
>
>Loki Jorgenson
>Apparent Networks
>t   604 433 2333 ext 105
>m   604 250-4642
>
>-----Original Message-----
>From: Loki Jorgenson
>Sent: Thursday, December 04, 2008 16:49
>To: [email-masked]
>Cc: Alan Clark
>Subject: RE: [PMOL] WGLC on pmol-metrics-framework-01
>
>Pardon my late comments as I have been unavailable for PMOL
>participation as of late.
>
>One request for clarification as well as some tidying of prose.
>
>1) Is there any value in defining the relationship of this framework to
>active or passive measurement methodologies?
>
>Originally I had thought not.  It appeared that this document was
>intended to be high-level and agnostic to active or passive
>methodologies, and therefore said nothing differentiate between the two.
>
>
>However, there is a reference in Section 6 (Security Considerations)
>that states that "security considerations that apply to any active
>measurements of live networks are relevant here" with a pointer to
>IPPM-developed RFC4656, OWAMP.  IPPM thus far limits itself to active
>methodologies.  This very mildly suggests that this framework is
>specific to active techniques.
>
>In Purpose and Scope, the reference is "for... metrics... that can be
>used to characterize traffic on live networks and services" - which
>sounds distinctly passive, insofar as "traffic" suggests applications
>and not active probing.
>
>And, various cross-references to other IPPM drafts and RFCs have been
>directly incorporated (e.g. compagg in at least Section 3.3).  However,
>examples to the Telchemy burst-loss algorithm and RFC 4546 may be
>applied to either passive or active.
>
>Further, if passive is included, then security considerations specific
>to passive should apply in Section 6 as well (e.g. anonymization).  I
>don't have a handy sample reference for that.
>
>Not to make more of it than necessary, but maybe it is worth addressing
>explicitly.  Perhaps a sentence or two within the Purpose and Scope
>section declaring its applicability with regard to common measurement
>methodologies.
>
>2) Awkward sentence:
>
>Section 3.4.2 (v)
>
>"Short intervals or frequent sampling provides a richer source of
>information that can be helpful in assessing application performance
>however can lead to excessive measurement data."
>
>It is comprehensible but somewhat awkward.  Suggest a couple of changes:
>
>"A SHORT SAMPLING INTERVAL or frequent sampling provides a richer source
>of information that MAY be helpful in assessing application performance
>BUT MAY ALSO lead to excessive measurement data."
>
>(Capitals indicate changes, not use of capitals in the document - e.g.
>MAY here is not the usual IETF directive convention)
>
>3) Ambiguous sentence:
>
>Section 3.4.2 (v)
>
>"Long measurement or sampling intervals reduce that amount of reported
>and collected data however may be insufficient to truly understand
>application performance or service quality if this varies with time."
>
>Semantically challenged although more or less readable.  Suggest:
>
>"Long measurement or sampling intervals reduce THE amount of reported
>and collected data SUCH THAT IT may be insufficient to ___ understand
>application performance or service quality INSOFAR AS THE MEASURED
>QUANTITY MAY vary SIGNIFICANTLY with time."
>
>4) Awkward sentence:
>
>Section 3.4.3 (iv)
>
>"For example, if the metric is a short term running average packet delay
>variation (e.g.  PPDV as defined in RFC3550) however this value is
>reported at intervals of 6-10 seconds this results in a sampling model
>which may have limited accuracy if packet delay variation is
>non-stationary."
>
>A very long sentence with missing or awkward conjunction.  Suggest:
>
>"For example, if the metric is a short term running average packet delay
>variation (e.g.  PPDV as defined in RFC3550) AND (YET) this value is
>reported at intervals of 6-10 seconds, THE RESULTING MEASUREMENT may
>have limited accuracy WHEN packet delay variation is non-stationary."
>
>5) Typographic error:
>
>Section 3.4.5 (Measurement Timing:)
>
>"... due to time-of-day load network load changes."
>
>Suggest:
>
>"... due to time-of-day changes in network load."
>
>
>6) Inconsistent notation:
>
>Section 3.5.1
>
>"... a 10 percent variation ..." (two instances)
>
>Could using number word and "percent" for all instances or numeric value
>and '%'.  Suggest:
>
>"... a 10% variation ..." (two instances)
>
>
>End of Comments
>
>------------------------------------------------------------------------
>-------------
>
>Message: 1
>Date: Tue, 02 Dec 2008 10:11:00 -0500
>From: Al Morton <[email-masked]>
>Subject: [PMOL] WGLC on pmol-metrics-framework-01
>To: [email-masked]
>Cc: [email-masked], [email-masked]
>
>PMOL WG,
>cc IPPM and BMWG WGs,
>
>This message begins a WG Last Call on the following draft:
>
>         Title           : Framework for Performance Metric Development
>         Author(s)       : A. Clark
>         Filename        : draft-ietf-pmol-metrics-framework-01.txt
>         Pages           : 14
>         Date            : 2008-11-02
>
>

_______________________________________________
PMOL mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/pmol

Date: Thu, 11 Dec 2008 11:23:27 +0100
From: Henk Uijterwaal <[email-masked]>
To: Al Morton <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked]
Subject: Re: [PMOL] [ippm] WGLC on pmol-metrics-framework-01
Sender: [email-masked]

Hi all,

Detail (page 7):

         (b) the speech level of a telephone signal is commonly expressed
          in dBm0.  If the user is presented with:

                   Speech level = -7 dBm0

          this is not intuitively understandable, unless the user is a
          telephony expert.  If the metric definition explains that the
          typical range is -18 to -28 dBm0, a value higher than -18
          means the signal may be too high (loud) and less than -28
          means that the signal may be too low (quiet), it is much
          easier to interpret the metric.

If the typical range is -18 to -28, then I still have a problem to
interpret -7 (outside the range).   I'd also add that this is a
logarithmic scale.

Other than that, yes, I've read this and read it again, it still looks
fine.

Henk

-- 
------------------------------------------------------------------------------
Henk Uijterwaal                           Email: henk.uijterwaal(at)ripe.net
RIPE Network Coordination Centre          http://www.amsterdamned.org/~henk
P.O.Box 10096          Singel 258         Phone: +31.20.5354414
1001 EB Amsterdam      1016 AB Amsterdam  Fax: +31.20.5354445
The Netherlands        The Netherlands    Mobile: +31.6.55861746
------------------------------------------------------------------------------

Ceterum censeo Asplain esse delendam  (Cato & Henk)
_______________________________________________
PMOL mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/pmol

Date: Sun, 07 Dec 2008 09:00:54 -0500
To: [email-masked]
From: Al Morton <[email-masked]>
Subject: [PMOL] Draft minutes from IETF-73 pmol session
Sender: [email-masked]

PMOL WG,

the draft minutes are available here:
http://www.ietf.org/proceedings/08nov/minutes/pmol.html

Thanks to Juergen Quittek for taking notes at the meeting!

One key Action Item for the entire working group is to
continue the discussion of the "Future of PMOL". See the
last page or so of the minutes and add your thoughts on
the list.

corrections, questions, or comments to the list please,
by December 19th, 2008.

Al
pmol co-chair

_______________________________________________
PMOL mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/pmol

Date: Tue, 02 Dec 2008 10:11:00 -0500
To: [email-masked]
From: Al Morton <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: [PMOL] WGLC on pmol-metrics-framework-01
Sender: [email-masked]

PMOL WG,
cc IPPM and BMWG WGs,

This message begins a WG Last Call on the following draft:

        Title           : Framework for Performance Metric Development
	Author(s)       : A. Clark
	Filename        : draft-ietf-pmol-metrics-framework-01.txt
	Pages           : 14
	Date            : 2008-11-02

This memo describes a framework and guidelines for the development of
performance metrics that are beyond the scope of existing working
group charters in the IETF.  In this version, the memo refers to a
Performance Metrics Entity, or PM Entity, which may in future be a
working group or directorate or a combination of these two.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-pmol-metrics-framework-01.txt

There has been discussion of this draft for over a year,
mostly on the PMOL list and at meetings.  The Author believes he
has addressed all comments to date.

Please weigh-in on whether or not this Draft should be forwarded
to the Area Directors for publication as a Standards Track RFC.
Send your comments to the PMOL list or the co-chairs.

The Last Call will be open till January 2, 2009.

thanks for your review and comments,
Al Morton
co-chair, PMOL WG

_______________________________________________
PMOL mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/pmol

Content-class: urn:content-classes:message
Date: Mon, 1 Dec 2008 10:55:37 -0500
From: "Christou, Christos [USA]" <[email-masked]>
To: <[email-masked]>
Subject: [PMOL] application performance metrics
Sender: [email-masked]

This is a multi-part message in MIME format.

--===============1588372743==
Content-class: urn:content-classes:message
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C953CD.40FF1CD1"

This is a multi-part message in MIME format.

------_=_NextPart_001_01C953CD.40FF1CD1
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hello,
=20
Having reviewed the work of this working group and listening to the
presentation at the Ops Area open meeting in Minneapolis, I would like
to understand from WG participants whether there is interest in
expanding the work of this WG to define performance metrics for
different application types? =20
=20
In discussing this with Al and others, I wanted to see if participants
think this is needed, if it is too difficult, out of scope for the IETF,
if the appropriate metrics have already been defined, etc.  For example,
ITU-T SG 12 has definied G.1080 and G.1081 to define different "QoE"
metrics--is this sufficient? =20
=20
It seems like there has been much work over the years on how to measure
MOS.  However, what is the parameter that is used for other
applications?  If this has previously been defined, please do include a
pointer...
=20
Thanks--just wanted to gauge the list's thoughts on this,
=20
Chris

------_=_NextPart_001_01C953CD.40FF1CD1
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=3DContent-Type content=3D"text/html; =
charset=3Dus-ascii">
<META content=3D"MSHTML 6.00.2900.3429" name=3DGENERATOR></HEAD>
<BODY>
<DIV>
<DIV><SPAN class=3D730173919-19112008><FONT face=3DArial=20
size=3D2>Hello,</FONT></SPAN></DIV>
<DIV><SPAN class=3D730173919-19112008><FONT face=3DArial=20
size=3D2></FONT></SPAN>&nbsp;</DIV>
<DIV><SPAN class=3D730173919-19112008><FONT face=3DArial size=3D2>Having =
reviewed the=20
work of this working group and listening to the presentation at the Ops =
Area=20
open meeting&nbsp;<SPAN class=3D918425415-01122008>in =
Minneapolis</SPAN>, I would=20
like to understand from WG participants whether there is interest in =
expanding=20
the work of this WG to define performance metrics for different =
application=20
types?&nbsp; </FONT></SPAN></DIV>
<DIV><SPAN class=3D730173919-19112008><FONT face=3DArial=20
size=3D2></FONT></SPAN>&nbsp;</DIV>
<DIV><SPAN class=3D730173919-19112008><FONT face=3DArial size=3D2>In =
discussing this=20
with Al and others, I wanted to see if participants think this is =
needed, if it=20
is too difficult, out of scope for the IETF, if the appropriate metrics =
have=20
already been defined, etc.&nbsp; For example, ITU-T SG 12 has definied =
G.1080=20
and G.1081 to define different "QoE" metrics--is this sufficient?&nbsp;=20
</FONT></SPAN></DIV>
<DIV><SPAN class=3D730173919-19112008><FONT face=3DArial=20
size=3D2></FONT></SPAN>&nbsp;</DIV>
<DIV><SPAN class=3D730173919-19112008><FONT face=3DArial =
size=3D2>It&nbsp;seems like=20
there has been much work over the years on how to measure MOS.&nbsp; =
However,=20
what is the parameter that is used for other applications?&nbsp; If this =
has=20
previously been defined, please do include a =
pointer...</FONT></SPAN></DIV>
<DIV><SPAN class=3D730173919-19112008><FONT face=3DArial=20
size=3D2></FONT></SPAN>&nbsp;</DIV>
<DIV><SPAN class=3D730173919-19112008><FONT face=3DArial =
size=3D2>Thanks--just wanted=20
to gauge the list<SPAN class=3D918425415-01122008>'</SPAN>s thoughts on=20
this,</FONT></SPAN></DIV>
<DIV><SPAN class=3D730173919-19112008><FONT face=3DArial=20
size=3D2></FONT></SPAN>&nbsp;</DIV>
<DIV><SPAN class=3D730173919-19112008><FONT face=3DArial=20
size=3D2>Chris</FONT></SPAN></DIV></DIV></BODY></HTML>

------_=_NextPart_001_01C953CD.40FF1CD1--

--===============1588372743==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
PMOL mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/pmol

--===============1588372743==--