Subject: [PMOL] Re: A question about [Fwd: WG Review: Performance Metrics at Other Layers (pmol)]

From: Sam Hartman <[email-masked]>
Date: Mon, 29 Oct 2007 16:24:02 -0400

>>>>> "Leslie" == Leslie Daigle <[email-masked]> writes:

I doubt I'll use the output in security protocols.

    Leslie> Leslie.

    Leslie> -------- Original Message -------- Subject: WG Review:
    Leslie> Performance Metrics at Other Layers (pmol) Date: Mon, 22
    Leslie> Oct 2007 14:15:02 -0400 From: IESG Secretary
    Leslie> <[email-masked]> Reply-To: [email-masked] To:
    Leslie> [email-masked]

    Leslie> A new IETF working group has been proposed in the
    Leslie> Operations and Management Area.  The IESG has not made any
    Leslie> determination as yet.  The following draft charter was
    Leslie> submitted, and is provided for informational purposes
    Leslie> only.  Please send your comments to the IESG mailing list
    Leslie> ([email-masked]) by October 29.

    Leslie> +++

    Leslie> Performance Metrics at Other Layers (pmol)
    Leslie> ==============================================

    Leslie> Current Status: Proposed Working Group

    Leslie> WG Chairs: TBD

    Leslie> Operations and Management Area: Dan Romascanu
    Leslie> <[email-masked]> Ronald Bonica <[email-masked]>

    Leslie> Description:

    Leslie> The successful implementation and operation of IP based
    Leslie> applications often depends on some underlying performance
    Leslie> measurement infrastructure that helps service operators or
    Leslie> network managers to recognize when performance is
    Leslie> unsatisfactory and identify problems affecting service
    Leslie> quality. Standardized performance metrics add the
    Leslie> desirable features of consistent implementation,
    Leslie> interpretation, no comparison.

    Leslie> The IETF has two Working Groups dedicated to the
    Leslie> development of performance metrics however each has strict
    Leslie> limitations in their charters:

    Leslie> - The Benchmarking Methodology WG has addressed a range of
    Leslie> networking technologies and protocols in their long
    Leslie> history (such as IEEE 802.3, ATM, Frame Relay, and Routing
    Leslie> Protocols), but the charter strictly limits their
    Leslie> Performance characterizations to the laboratory
    Leslie> environment.

    Leslie> - The IP Performance Metrics WG has the mandate to develop
    Leslie> metrics applicable to the performance of Internet data
    Leslie> delivery, but it is specifically prohibited from
    Leslie> developing metrics that characterize traffic (such as a
    Leslie> VoIP stream).

    Leslie> The IETF also has current and completed activities related
    Leslie> to the reporting of application performance metrics
    Leslie> (e.g. RAQMON and RTCP XR) and is also actively involved in
    Leslie> the development of reliable transport protocols which
    Leslie> would affect the relationship between IP performance and
    Leslie> application performance.

    Leslie> Thus there is a gap in the currently chartered coverage of
    Leslie> IETF WGs: development of performance metrics for IP-based
    Leslie> protocols and applications that operate over UDP, TCP,
    Leslie> SCTP, DCCP, Forward Error Correction (FEC) and other
    Leslie> robust transport protocols, and that can be used to
    Leslie> characterize traffic on live networks.

    Leslie> The working group will focus on the completion of two
    Leslie> RFCs:

    Leslie> 1. A PMOL framework and guidelines memo that will describe
    Leslie> the necessary elements of performance metrics of protocols
    Leslie> and applications transported over IETF-specified protocols
    Leslie> (such as the formal definition, purpose, and units of
    Leslie> measure) and the various types of metrics that
    Leslie> characterize traffic on live networks (such as metrics
    Leslie> derived from other metrics, possibly on lower layers). The
    Leslie> framework will also address the need to specify the
    Leslie> intended audience and the motivation for the performance
    Leslie> metrics. There will also be guidelines for a performance
    Leslie> metric development process that includes entry criteria
    Leslie> for new proposals (how a proposal might be evaluated for
    Leslie> possible endorsement by a protocol development working
    Leslie> group), and how an successful proposal will be developed
    Leslie> by PMOL WG in cooperation with a protocol development WG.

    Leslie> 2. A proof-of-concept RFC defining performance metrics for
    Leslie> SIP, based on draft-malas-performance-metrics. This memo
    Leslie> would serve as an example of the framework and the PMOL
    Leslie> development process in the IETF.

    Leslie> Discussion of new work proposals is strongly discouraged
    Leslie> under the initial charter of the PMOL WG, except to advise
    Leslie> a protocol development WG when they are evaluating a new
    Leslie> work proposal for related performance metrics.

    Leslie> The PMOL WG will also be guided by a document describing
    Leslie> how memos defining performance metrics are intended to
    Leslie> advance along the IETF Standards track
    Leslie> (draft-bradner-metricstest).

    Leslie> PMOL WG will take advantage of expertise and seek to avoid
    Leslie> overlap with other standards development organizations,
    Leslie> such as ETSI STQ, ITU-T SG 12, ATIS IIF, ATIS PRQC, and
    Leslie> others.

    Leslie> Milestones

    Leslie> June 08 SIP Performance Metrics Draft to IESG Review for
    Leslie> consideration as Proposed Standard

    Leslie> Sept 08 PMOL Framework and Guidelines Draft to IESG Review
    Leslie> for consideration as BCP

    Leslie> _______________________________________________
    Leslie> IETF-Announce mailing list [email-masked]
    Leslie> https://www1.ietf.org/mailman/listinfo/ietf-announce

    Leslie> --

    Leslie> -------------------------------------------------------------------
    Leslie> "Reality: Yours to discover."  -- ThinkingCat Leslie
    Leslie> Daigle [email-masked]
    Leslie> -------------------------------------------------------------------

_______________________________________________
PMOL mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/pmol

---

From: Leslie Daigle <[email-masked]>
Date: Mon, 29 Oct 2007 16:04:17 -0400

At the Application Performance Metrics BoF in
Chicago, there was a lot of discussion about whether
or how to best capture expertise in the area of performance
metrics for application to IETF protocols.

There seemed to be 2 separate questions on the table:

	1/ how, and
	2/ whether

to generalize a performance metric framework for IETF
protocols.

The proposed PMOL WG addresses the first question -- how
to do it.

On the second point -- the question is really about whether
the IETF community as a whole supports/values performance
metrics and will invest the effort in using any general
framework for existing/new protocols.

I believe the folks that joined the PMOL mailing list after
the APM BoF are assuming that support is there.

I suggest that now is an excellent time for folks _not_
involved in the PMOL effort, but who are working on
protocols that could/should make use of its output, to
voice some opinions on whether or not this approach
is helpful/will be useful to them in future protocols.

Leslie.

-------- Original Message --------
Subject: WG Review: Performance Metrics at Other Layers (pmol)
Date: Mon, 22 Oct 2007 14:15:02 -0400
From: IESG Secretary <[email-masked]>
Reply-To: [email-masked]
To: [email-masked]

A new IETF working group has been proposed in the Operations and
Management Area.  The IESG has not made any determination as yet.
The following draft charter was submitted, and is provided for
informational purposes only.  Please send your comments to the IESG
mailing list ([email-masked]) by October 29.

+++

Performance Metrics at Other Layers (pmol)
==============================================

Current Status: Proposed Working Group

WG Chairs:
TBD

Operations and Management Area:
Dan Romascanu <[email-masked]>
Ronald Bonica <[email-masked]>

Description:

The successful implementation and operation of IP based applications
often depends on some underlying performance measurement infrastructure
that helps service operators or network managers to recognize when
performance is unsatisfactory and identify problems affecting service
quality. Standardized performance metrics add the desirable features of
consistent implementation, interpretation, no comparison.

The IETF has two Working Groups dedicated to the development of
performance metrics however each has strict limitations in their
charters:

- The Benchmarking Methodology WG has addressed a range of networking
technologies and protocols in their long history (such as IEEE 802.3,
ATM, Frame Relay, and Routing Protocols), but the charter strictly
limits their Performance characterizations to the laboratory
environment.

- The IP Performance Metrics WG has the mandate to develop metrics
applicable to the performance of Internet data delivery, but it is
specifically prohibited from developing metrics that characterize
traffic (such as a VoIP stream).

The IETF also has current and completed activities related to the
reporting of application performance metrics (e.g. RAQMON and RTCP XR)
and is also actively involved in the development of reliable transport
protocols which would affect the relationship between IP performance and
application performance.

Thus there is a gap in the currently chartered coverage of IETF WGs:
development of performance metrics for IP-based protocols and
applications that operate over UDP, TCP, SCTP, DCCP, Forward Error
Correction (FEC) and other robust transport protocols, and that can be
used to characterize traffic on live networks.

The working group will focus on the completion of two RFCs:

1. A PMOL framework and guidelines memo that will describe the necessary
elements of performance metrics of protocols and applications
transported over IETF-specified protocols (such as the formal
definition, purpose, and units of measure) and the various types of
metrics that characterize traffic on live networks (such as metrics
derived from other metrics, possibly on lower layers). The framework
will also address the need to specify the intended audience and the
motivation for the performance metrics. There will also be guidelines
for a performance metric development process that includes entry
criteria for new proposals (how a proposal might be evaluated for
possible endorsement by a protocol development working group), and how
an successful proposal will be developed by PMOL WG in cooperation with
a protocol development WG.

2. A proof-of-concept RFC defining performance metrics for SIP, based on
draft-malas-performance-metrics. This memo would serve as an example of
the framework and the PMOL development process in the IETF.

Discussion of new work proposals is strongly discouraged under the
initial charter of the PMOL WG, except to advise a protocol development
WG when they are evaluating a new work proposal for related performance
metrics.

The PMOL WG will also be guided by a document describing how memos
defining performance metrics are intended to advance along the IETF
Standards track (draft-bradner-metricstest).

PMOL WG will take advantage of expertise and seek to avoid overlap with
other standards development organizations, such as ETSI STQ, ITU-T SG
12, ATIS IIF, ATIS PRQC, and others.

Milestones

June 08 SIP Performance Metrics Draft to IESG Review for consideration
as Proposed Standard

Sept 08 PMOL Framework and Guidelines Draft to IESG Review for
consideration as BCP

_______________________________________________
IETF-Announce mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ietf-announce

-- 

-------------------------------------------------------------------
"Reality:
      Yours to discover."
                                 -- ThinkingCat
Leslie Daigle
[email-masked]
-------------------------------------------------------------------

---

From: Yangwoo Ko <[email-masked]>
Date: Tue, 30 Oct 2007 12:06:45 +0900

Since application developers have developed many tricks to smooth out 
some peculiarities of network-based protocols (e.g. jitter, delay, drops 
and so on), it is very often hard to related the performance 
measured/perceived at application layer with those of underlying layers. 
If this WG provides guidelines (or at least hints) to get over this 
vagueness, I will be very happy.

Regards