Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1Impuv-0002hg-40; Tue, 30 Oct 2007 08:09:57 -0400 Received: from pmol by megatron.ietf.org with local (Exim 4.43) id 1ImhRb-0003sb-22 for pmol-confirm+ok@megatron.ietf.org; Mon, 29 Oct 2007 23:07:07 -0400 Received: from [10.90.34.44] (helo=chiedprmail1.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1ImhRa-0003i7-Cw for pmol@ietf.org; Mon, 29 Oct 2007 23:07:06 -0400 Received: from sniper.icu.ac.kr ([210.107.128.51]) by chiedprmail1.ietf.org with esmtp (Exim 4.43) id 1ImhRX-0000fN-8G for pmol@ietf.org; Mon, 29 Oct 2007 23:07:04 -0400 Received: (snipe 19899 invoked by uid 0); 30 Oct 2007 12:06:56 +0900 Received: from newcat@icu.ac.kr with Spamsniper 2.96.00 (Processed in 1.074600 secs); Received: from unknown (HELO ?210.107.250.146?) (Z???own@210.107.250.146) by unknown with SMTP; 30 Oct 2007 12:06:55 +0900 X-SNIPER-SENDERIP: 210.107.250.146 X-SNIPER-MAILFROM: newcat@icu.ac.kr X-SNIPER-RCPTTO: leslie@thinkingcat.com, ietf@ietf.org, pmol@ietf.org, iesg@ietf.org, yangwooko@gmail.com Message-ID: <47269FC5.5030802@icu.ac.kr> Date: Tue, 30 Oct 2007 12:06:45 +0900 From: Yangwoo Ko User-Agent: Thunderbird 2.0.0.6 (Windows/20070728) MIME-Version: 1.0 To: Leslie Daigle References: <47263CC1.8000104@thinkingcat.com> In-Reply-To: <47263CC1.8000104@thinkingcat.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.0 (/) X-Scan-Signature: 6ba8aaf827dcb437101951262f69b3de X-Mailman-Approved-At: Tue, 30 Oct 2007 08:09:55 -0400 Cc: IESG , ietf@ietf.org, pmol@ietf.org Subject: [PMOL] Re: A question about [Fwd: WG Review: Performance Metrics at Other Layers (pmol)] X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: pmol-bounces@ietf.org Since application developers have developed many tricks to smooth out some peculiarities of network-based protocols (e.g. jitter, delay, drops and so on), it is very often hard to related the performance measured/perceived at application layer with those of underlying layers. If this WG provides guidelines (or at least hints) to get over this vagueness, I will be very happy. Regards Leslie Daigle wrote: > > > At the Application Performance Metrics BoF in > Chicago, there was a lot of discussion about whether > or how to best capture expertise in the area of performance > metrics for application to IETF protocols. > > There seemed to be 2 separate questions on the table: > > 1/ how, and > 2/ whether > > to generalize a performance metric framework for IETF > protocols. > > > The proposed PMOL WG addresses the first question -- how > to do it. > > On the second point -- the question is really about whether > the IETF community as a whole supports/values performance > metrics and will invest the effort in using any general > framework for existing/new protocols. > > I believe the folks that joined the PMOL mailing list after > the APM BoF are assuming that support is there. > > I suggest that now is an excellent time for folks _not_ > involved in the PMOL effort, but who are working on > protocols that could/should make use of its output, to > voice some opinions on whether or not this approach > is helpful/will be useful to them in future protocols. > > > Leslie. > > > -------- Original Message -------- > Subject: WG Review: Performance Metrics at Other Layers (pmol) > Date: Mon, 22 Oct 2007 14:15:02 -0400 > From: IESG Secretary > Reply-To: iesg@ietf.org > To: ietf-announce@ietf.org > > A new IETF working group has been proposed in the Operations and > Management Area. The IESG has not made any determination as yet. > The following draft charter was submitted, and is provided for > informational purposes only. Please send your comments to the IESG > mailing list (iesg@ietf.org) by October 29. > > +++ > > Performance Metrics at Other Layers (pmol) > ============================================== > > Current Status: Proposed Working Group > > WG Chairs: > TBD > > Operations and Management Area: > Dan Romascanu > Ronald Bonica > > Description: > > The successful implementation and operation of IP based applications > often depends on some underlying performance measurement infrastructure > that helps service operators or network managers to recognize when > performance is unsatisfactory and identify problems affecting service > quality. Standardized performance metrics add the desirable features of > consistent implementation, interpretation, no comparison. > > The IETF has two Working Groups dedicated to the development of > performance metrics however each has strict limitations in their > charters: > > - The Benchmarking Methodology WG has addressed a range of networking > technologies and protocols in their long history (such as IEEE 802.3, > ATM, Frame Relay, and Routing Protocols), but the charter strictly > limits their Performance characterizations to the laboratory > environment. > > - The IP Performance Metrics WG has the mandate to develop metrics > applicable to the performance of Internet data delivery, but it is > specifically prohibited from developing metrics that characterize > traffic (such as a VoIP stream). > > The IETF also has current and completed activities related to the > reporting of application performance metrics (e.g. RAQMON and RTCP XR) > and is also actively involved in the development of reliable transport > protocols which would affect the relationship between IP performance and > application performance. > > Thus there is a gap in the currently chartered coverage of IETF WGs: > development of performance metrics for IP-based protocols and > applications that operate over UDP, TCP, SCTP, DCCP, Forward Error > Correction (FEC) and other robust transport protocols, and that can be > used to characterize traffic on live networks. > > The working group will focus on the completion of two RFCs: > > 1. A PMOL framework and guidelines memo that will describe the necessary > elements of performance metrics of protocols and applications > transported over IETF-specified protocols (such as the formal > definition, purpose, and units of measure) and the various types of > metrics that characterize traffic on live networks (such as metrics > derived from other metrics, possibly on lower layers). The framework > will also address the need to specify the intended audience and the > motivation for the performance metrics. There will also be guidelines > for a performance metric development process that includes entry > criteria for new proposals (how a proposal might be evaluated for > possible endorsement by a protocol development working group), and how > an successful proposal will be developed by PMOL WG in cooperation with > a protocol development WG. > > 2. A proof-of-concept RFC defining performance metrics for SIP, based on > draft-malas-performance-metrics. This memo would serve as an example of > the framework and the PMOL development process in the IETF. > > Discussion of new work proposals is strongly discouraged under the > initial charter of the PMOL WG, except to advise a protocol development > WG when they are evaluating a new work proposal for related performance > metrics. > > The PMOL WG will also be guided by a document describing how memos > defining performance metrics are intended to advance along the IETF > Standards track (draft-bradner-metricstest). > > PMOL WG will take advantage of expertise and seek to avoid overlap with > other standards development organizations, such as ETSI STQ, ITU-T SG > 12, ATIS IIF, ATIS PRQC, and others. > > Milestones > > June 08 SIP Performance Metrics Draft to IESG Review for consideration > as Proposed Standard > > Sept 08 PMOL Framework and Guidelines Draft to IESG Review for > consideration as BCP > > _______________________________________________ > IETF-Announce mailing list > IETF-Announce@ietf.org > https://www1.ietf.org/mailman/listinfo/ietf-announce > _______________________________________________ PMOL mailing list PMOL@ietf.org https://www1.ietf.org/mailman/listinfo/pmol Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1Imcul-00028Z-90; Mon, 29 Oct 2007 18:16:55 -0400 Received: from pmol by megatron.ietf.org with local (Exim 4.43) id 1Imb9a-0004Gm-3W for pmol-confirm+ok@megatron.ietf.org; Mon, 29 Oct 2007 16:24:06 -0400 Received: from [10.90.34.44] (helo=chiedprmail1.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1Imb9Z-0004GW-P0; Mon, 29 Oct 2007 16:24:05 -0400 Received: from dhcp-18-188-10-61.dyn.mit.edu ([18.188.10.61] helo=carter-zimmerman.suchdamage.org) by chiedprmail1.ietf.org with esmtp (Exim 4.43) id 1Imb9X-0004nA-1T; Mon, 29 Oct 2007 16:24:03 -0400 Received: by carter-zimmerman.suchdamage.org (Postfix, from userid 8042) id 0674B4A45; Mon, 29 Oct 2007 16:24:02 -0400 (EDT) From: Sam Hartman To: Leslie Daigle References: <47263CC1.8000104@thinkingcat.com> Date: Mon, 29 Oct 2007 16:24:02 -0400 In-Reply-To: <47263CC1.8000104@thinkingcat.com> (Leslie Daigle's message of "Mon, 29 Oct 2007 16:04:17 -0400") Message-ID: User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/21.4 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Spam-Score: 0.1 (/) X-Scan-Signature: 6640e3bbe8a4d70c4469bcdcbbf0921d X-Mailman-Approved-At: Mon, 29 Oct 2007 18:16:54 -0400 Cc: IESG , ietf@ietf.org, pmol@ietf.org Subject: [PMOL] Re: A question about [Fwd: WG Review: Performance Metrics at Other Layers (pmol)] X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: pmol-bounces@ietf.org >>>>> "Leslie" == Leslie Daigle writes: I doubt I'll use the output in security protocols. Leslie> Leslie. Leslie> -------- Original Message -------- Subject: WG Review: Leslie> Performance Metrics at Other Layers (pmol) Date: Mon, 22 Leslie> Oct 2007 14:15:02 -0400 From: IESG Secretary Leslie> Reply-To: iesg@ietf.org To: Leslie> ietf-announce@ietf.org Leslie> A new IETF working group has been proposed in the Leslie> Operations and Management Area. The IESG has not made any Leslie> determination as yet. The following draft charter was Leslie> submitted, and is provided for informational purposes Leslie> only. Please send your comments to the IESG mailing list Leslie> (iesg@ietf.org) by October 29. Leslie> +++ Leslie> Performance Metrics at Other Layers (pmol) Leslie> ============================================== Leslie> Current Status: Proposed Working Group Leslie> WG Chairs: TBD Leslie> Operations and Management Area: Dan Romascanu Leslie> Ronald Bonica Leslie> Description: Leslie> The successful implementation and operation of IP based Leslie> applications often depends on some underlying performance Leslie> measurement infrastructure that helps service operators or Leslie> network managers to recognize when performance is Leslie> unsatisfactory and identify problems affecting service Leslie> quality. Standardized performance metrics add the Leslie> desirable features of consistent implementation, Leslie> interpretation, no comparison. Leslie> The IETF has two Working Groups dedicated to the Leslie> development of performance metrics however each has strict Leslie> limitations in their charters: Leslie> - The Benchmarking Methodology WG has addressed a range of Leslie> networking technologies and protocols in their long Leslie> history (such as IEEE 802.3, ATM, Frame Relay, and Routing Leslie> Protocols), but the charter strictly limits their Leslie> Performance characterizations to the laboratory Leslie> environment. Leslie> - The IP Performance Metrics WG has the mandate to develop Leslie> metrics applicable to the performance of Internet data Leslie> delivery, but it is specifically prohibited from Leslie> developing metrics that characterize traffic (such as a Leslie> VoIP stream). Leslie> The IETF also has current and completed activities related Leslie> to the reporting of application performance metrics Leslie> (e.g. RAQMON and RTCP XR) and is also actively involved in Leslie> the development of reliable transport protocols which Leslie> would affect the relationship between IP performance and Leslie> application performance. Leslie> Thus there is a gap in the currently chartered coverage of Leslie> IETF WGs: development of performance metrics for IP-based Leslie> protocols and applications that operate over UDP, TCP, Leslie> SCTP, DCCP, Forward Error Correction (FEC) and other Leslie> robust transport protocols, and that can be used to Leslie> characterize traffic on live networks. Leslie> The working group will focus on the completion of two Leslie> RFCs: Leslie> 1. A PMOL framework and guidelines memo that will describe Leslie> the necessary elements of performance metrics of protocols Leslie> and applications transported over IETF-specified protocols Leslie> (such as the formal definition, purpose, and units of Leslie> measure) and the various types of metrics that Leslie> characterize traffic on live networks (such as metrics Leslie> derived from other metrics, possibly on lower layers). The Leslie> framework will also address the need to specify the Leslie> intended audience and the motivation for the performance Leslie> metrics. There will also be guidelines for a performance Leslie> metric development process that includes entry criteria Leslie> for new proposals (how a proposal might be evaluated for Leslie> possible endorsement by a protocol development working Leslie> group), and how an successful proposal will be developed Leslie> by PMOL WG in cooperation with a protocol development WG. Leslie> 2. A proof-of-concept RFC defining performance metrics for Leslie> SIP, based on draft-malas-performance-metrics. This memo Leslie> would serve as an example of the framework and the PMOL Leslie> development process in the IETF. Leslie> Discussion of new work proposals is strongly discouraged Leslie> under the initial charter of the PMOL WG, except to advise Leslie> a protocol development WG when they are evaluating a new Leslie> work proposal for related performance metrics. Leslie> The PMOL WG will also be guided by a document describing Leslie> how memos defining performance metrics are intended to Leslie> advance along the IETF Standards track Leslie> (draft-bradner-metricstest). Leslie> PMOL WG will take advantage of expertise and seek to avoid Leslie> overlap with other standards development organizations, Leslie> such as ETSI STQ, ITU-T SG 12, ATIS IIF, ATIS PRQC, and Leslie> others. Leslie> Milestones Leslie> June 08 SIP Performance Metrics Draft to IESG Review for Leslie> consideration as Proposed Standard Leslie> Sept 08 PMOL Framework and Guidelines Draft to IESG Review Leslie> for consideration as BCP Leslie> _______________________________________________ Leslie> IETF-Announce mailing list IETF-Announce@ietf.org Leslie> https://www1.ietf.org/mailman/listinfo/ietf-announce Leslie> -- Leslie> ------------------------------------------------------------------- Leslie> "Reality: Yours to discover." -- ThinkingCat Leslie Leslie> Daigle leslie@thinkingcat.com Leslie> ------------------------------------------------------------------- _______________________________________________ PMOL mailing list PMOL@ietf.org https://www1.ietf.org/mailman/listinfo/pmol Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1Imcul-00028S-6A; Mon, 29 Oct 2007 18:16:55 -0400 Received: from pmol by megatron.ietf.org with local (Exim 4.43) id 1Imasn-00013P-TJ for pmol-confirm+ok@megatron.ietf.org; Mon, 29 Oct 2007 16:06:45 -0400 Received: from [10.90.34.44] (helo=chiedprmail1.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1Imasi-0000yY-Ap; Mon, 29 Oct 2007 16:06:40 -0400 Received: from zeke.blacka.com ([69.31.8.124] helo=zeke.ecotroph.net) by chiedprmail1.ietf.org with esmtp (Exim 4.43) id 1Imash-0003D9-Kh; Mon, 29 Oct 2007 16:06:40 -0400 Received: from beethoven.local ([::ffff:209.183.196.229]) (AUTH: PLAIN leslie, SSL: TLSv1/SSLv3,256bits,AES256-SHA) by zeke.ecotroph.net with esmtp; Mon, 29 Oct 2007 16:01:25 -0400 id 015880FB.47263C15.00005BB7 Message-ID: <47263CC1.8000104@thinkingcat.com> Date: Mon, 29 Oct 2007 16:04:17 -0400 From: Leslie Daigle User-Agent: Thunderbird 2.0.0.6 (Macintosh/20070728) MIME-Version: 1.0 To: ietf@ietf.org Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.0 (/) X-Scan-Signature: 3971661e40967acfc35f708dd5f33760 X-Mailman-Approved-At: Mon, 29 Oct 2007 18:16:54 -0400 Cc: pmol@ietf.org, IESG Subject: [PMOL] A question about [Fwd: WG Review: Performance Metrics at Other Layers (pmol)] X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: pmol-bounces@ietf.org At the Application Performance Metrics BoF in Chicago, there was a lot of discussion about whether or how to best capture expertise in the area of performance metrics for application to IETF protocols. There seemed to be 2 separate questions on the table: 1/ how, and 2/ whether to generalize a performance metric framework for IETF protocols. The proposed PMOL WG addresses the first question -- how to do it. On the second point -- the question is really about whether the IETF community as a whole supports/values performance metrics and will invest the effort in using any general framework for existing/new protocols. I believe the folks that joined the PMOL mailing list after the APM BoF are assuming that support is there. I suggest that now is an excellent time for folks _not_ involved in the PMOL effort, but who are working on protocols that could/should make use of its output, to voice some opinions on whether or not this approach is helpful/will be useful to them in future protocols. Leslie. -------- Original Message -------- Subject: WG Review: Performance Metrics at Other Layers (pmol) Date: Mon, 22 Oct 2007 14:15:02 -0400 From: IESG Secretary Reply-To: iesg@ietf.org To: ietf-announce@ietf.org A new IETF working group has been proposed in the Operations and Management Area. The IESG has not made any determination as yet. The following draft charter was submitted, and is provided for informational purposes only. Please send your comments to the IESG mailing list (iesg@ietf.org) by October 29. +++ Performance Metrics at Other Layers (pmol) ============================================== Current Status: Proposed Working Group WG Chairs: TBD Operations and Management Area: Dan Romascanu Ronald Bonica Description: The successful implementation and operation of IP based applications often depends on some underlying performance measurement infrastructure that helps service operators or network managers to recognize when performance is unsatisfactory and identify problems affecting service quality. Standardized performance metrics add the desirable features of consistent implementation, interpretation, no comparison. The IETF has two Working Groups dedicated to the development of performance metrics however each has strict limitations in their charters: - The Benchmarking Methodology WG has addressed a range of networking technologies and protocols in their long history (such as IEEE 802.3, ATM, Frame Relay, and Routing Protocols), but the charter strictly limits their Performance characterizations to the laboratory environment. - The IP Performance Metrics WG has the mandate to develop metrics applicable to the performance of Internet data delivery, but it is specifically prohibited from developing metrics that characterize traffic (such as a VoIP stream). The IETF also has current and completed activities related to the reporting of application performance metrics (e.g. RAQMON and RTCP XR) and is also actively involved in the development of reliable transport protocols which would affect the relationship between IP performance and application performance. Thus there is a gap in the currently chartered coverage of IETF WGs: development of performance metrics for IP-based protocols and applications that operate over UDP, TCP, SCTP, DCCP, Forward Error Correction (FEC) and other robust transport protocols, and that can be used to characterize traffic on live networks. The working group will focus on the completion of two RFCs: 1. A PMOL framework and guidelines memo that will describe the necessary elements of performance metrics of protocols and applications transported over IETF-specified protocols (such as the formal definition, purpose, and units of measure) and the various types of metrics that characterize traffic on live networks (such as metrics derived from other metrics, possibly on lower layers). The framework will also address the need to specify the intended audience and the motivation for the performance metrics. There will also be guidelines for a performance metric development process that includes entry criteria for new proposals (how a proposal might be evaluated for possible endorsement by a protocol development working group), and how an successful proposal will be developed by PMOL WG in cooperation with a protocol development WG. 2. A proof-of-concept RFC defining performance metrics for SIP, based on draft-malas-performance-metrics. This memo would serve as an example of the framework and the PMOL development process in the IETF. Discussion of new work proposals is strongly discouraged under the initial charter of the PMOL WG, except to advise a protocol development WG when they are evaluating a new work proposal for related performance metrics. The PMOL WG will also be guided by a document describing how memos defining performance metrics are intended to advance along the IETF Standards track (draft-bradner-metricstest). PMOL WG will take advantage of expertise and seek to avoid overlap with other standards development organizations, such as ETSI STQ, ITU-T SG 12, ATIS IIF, ATIS PRQC, and others. Milestones June 08 SIP Performance Metrics Draft to IESG Review for consideration as Proposed Standard Sept 08 PMOL Framework and Guidelines Draft to IESG Review for consideration as BCP _______________________________________________ IETF-Announce mailing list IETF-Announce@ietf.org https://www1.ietf.org/mailman/listinfo/ietf-announce -- ------------------------------------------------------------------- "Reality: Yours to discover." -- ThinkingCat Leslie Daigle leslie@thinkingcat.com ------------------------------------------------------------------- _______________________________________________ PMOL mailing list PMOL@ietf.org https://www1.ietf.org/mailman/listinfo/pmol Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1Il7zj-0003YB-Go; Thu, 25 Oct 2007 15:03:51 -0400 Received: from pmol by megatron.ietf.org with local (Exim 4.43) id 1Il7zi-0003OJ-Hl for pmol-confirm+ok@megatron.ietf.org; Thu, 25 Oct 2007 15:03:50 -0400 Received: from [10.90.34.44] (helo=chiedprmail1.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1Il7zc-0002cq-Og for pmol@ietf.org; Thu, 25 Oct 2007 15:03:44 -0400 Received: from [209.139.228.52] (helo=JSRVR18.jaalam.net) by chiedprmail1.ietf.org with esmtp (Exim 4.43) id 1Il7zW-0002jK-Rz for pmol@ietf.org; Thu, 25 Oct 2007 15:03:39 -0400 X-AuditID: ac108108-00000ac8000007ec-1d-4720e8875069 Received: from jsrvr8.jaalam.net ([172.16.128.105] RDNS failed) by JSRVR18.jaalam.net with Microsoft SMTPSVC(6.0.3790.1830); Thu, 25 Oct 2007 12:03:35 -0700 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Subject: RE: [PMOL] Re: draft-morton-perf-metrics-framework-00 Date: Thu, 25 Oct 2007 12:03:35 -0700 Message-ID: In-Reply-To: <471FAC92.9010706@gmail.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [PMOL] Re: draft-morton-perf-metrics-framework-00 Thread-Index: AcgWya8n81hVzNGAQx2XFYGlbo6DbgAWke7g References: <471E74AC.90209@gmail.com><200710240230.l9O2U0BN015298@mlph070.sfdc.sbc.com> <471FAC92.9010706@gmail.com> From: "Loki Jorgenson" To: "Brian E Carpenter" , "Aamer Akhter (aakhter)" X-Brightmail-Tracker: AAAAAA== X-Spam-Score: 0.1 (/) X-Scan-Signature: 92df29fa99cf13e554b84c8374345c17 Cc: Nevil Brownlee , pmol@ietf.org X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: pmol-bounces@ietf.org Brian/Aamer "Unreliable" more likely reflects ignorance on our part, more than any fundamental incoherence between upper and lower layer behaviors. I would posit that there exists a coupling between any two layers/metrics that can be discovered and described. The work of PMOL could include the definition of such couplings. Optimally, there would be an abstracted interface between layers that is parameterized in terms of existing well-defined quantities - for example, IPDV or One-way Loss Pattern. Thus expressions of higher level metrics (take MOS derived from E-model as an exemplar) can be mapped explicitly (if not simply) back to underlying packet metrics. Then the relationship between metrics such as associated variances are explicitly defined by statistical means and not simply a subject of off-hand and potentially arbitrary thresholds for "reliability". Mind, those coupling description may be quite complex in how they related and compound underlying metrics. This might also provide an explicit framework for application or middleware behaviors to be expressed as translation functions within the coupling context. Accounting for adaptive behaviors for "network aware" applications will otherwise be very challenging. Of course, I may be just being wishful that such a thing is possible. It certainly seems non-trivial. Loki Jorgenson Apparent Networks t 604 433 2333 ext 105 m 604 250-4642 -----Original Message----- From: Brian E Carpenter [mailto:brian.e.carpenter@gmail.com]=20 Sent: Wednesday, October 24, 2007 1:36 PM To: Aamer Akhter (aakhter) Cc: Nevil Brownlee; pmol@ietf.org Subject: Re: [PMOL] Re: draft-morton-perf-metrics-framework-00 On 2007-10-25 01:22, Aamer Akhter (aakhter) wrote: > Al, et al, >=20 >>> When metric Foobar is measured, metrics Barfoo and Foofoo must >>> be measured simultaneously between the same endpoints. >>> Results for Foobar must be accompanied by the results for Barfoo >>> and Foofoo. If the observed variance of Barfoo or Foofoo exceeds >>> , the results for Foobar must be labelled as 'unreliable'. >>> >=20 > This makes sense.=20 >=20 > I wonder however, about the cases where the application (or L7 entity etc) that is aware of the underlying disruptive transport and is working around it.=20 >=20 > Example: the work of digital fountain. I think in these cases, the metrics for the underlying would be even more important/relevant, but perhaps the final result shouldn't be labeled unreliable. >=20 > Thoughts? My thought is that the variance in the "upper" metric will be very hard to understand if there is a large variance in the "lower" metrics. There seem likely to be non-linear effects at work. If the upper metric is being used only as an observation (e.g. to check SLA conformance) that is one thing, but if it's being used to make any kind of prediction or to analyze the behaviour of the upper layer protocol, I think describing it as unreliable is justified. Brian _______________________________________________ PMOL mailing list PMOL@ietf.org https://www1.ietf.org/mailman/listinfo/pmol Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1IkvPc-0006Q9-KJ; Thu, 25 Oct 2007 01:37:44 -0400 Received: from pmol by megatron.ietf.org with local (Exim 4.43) id 1Ikmx9-00037C-Ns for pmol-confirm+ok@megatron.ietf.org; Wed, 24 Oct 2007 16:35:47 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1Ikmx8-000374-VW for pmol@ietf.org; Wed, 24 Oct 2007 16:35:46 -0400 Received: from rv-out-0910.google.com ([209.85.198.186]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1Ikmx2-0005EK-PS for pmol@ietf.org; Wed, 24 Oct 2007 16:35:46 -0400 Received: by rv-out-0910.google.com with SMTP id l15so250727rvb for ; Wed, 24 Oct 2007 13:35:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:organization:user-agent:mime-version:to:cc:subject:references:in-reply-to:content-type:content-transfer-encoding; bh=sFowU6yL9BKJ7B3R5T42mFcD4fhWVX3XJRrOc5suzUc=; b=LPIsB5GIdHmaXBNZNh0zzfwYPKMGOAJ+3Tcbb9WUlNMxBTH+NJ0Tt+K/KTv6sSE9yHV3OVE/8irA3vWEX5W1FVYC2o8HuXSguaxtLl6q/Q3nlty8CIRbdQJ/kvwsvRw7SBCVboLyTAUoCM8RYydXWJMq7h9ksu6DC4libTsJ8lU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:organization:user-agent:mime-version:to:cc:subject:references:in-reply-to:content-type:content-transfer-encoding; b=tLc+X0oBpN8fTdG6689GnGq9CsGlcV64E4BJ3OsVH8Zf/6Wt9u2RsnEC+SVo0CvXM8BmScrb6yp2kwoWK3NL3oMg9bJuvo1/xd7pTOM1POn75VyiqDe/w1SyfRnksaQQPPoGBmW/oAso04BFJuzYaxzh/WpLSGJ63acohsGEgCA= Received: by 10.141.172.6 with SMTP id z6mr532215rvo.1193258134099; Wed, 24 Oct 2007 13:35:34 -0700 (PDT) Received: from ?130.216.38.124? ( [130.216.38.124]) by mx.google.com with ESMTPS id f42sm2379578rvb.2007.10.24.13.35.32 (version=SSLv3 cipher=RC4-MD5); Wed, 24 Oct 2007 13:35:33 -0700 (PDT) Message-ID: <471FAC92.9010706@gmail.com> Date: Thu, 25 Oct 2007 09:35:30 +1300 From: Brian E Carpenter Organization: University of Auckland User-Agent: Thunderbird 2.0.0.6 (Windows/20070728) MIME-Version: 1.0 To: "Aamer Akhter (aakhter)" Subject: Re: [PMOL] Re: draft-morton-perf-metrics-framework-00 References: <471E74AC.90209@gmail.com> <200710240230.l9O2U0BN015298@mlph070.sfdc.sbc.com> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.0 (/) X-Scan-Signature: 2409bba43e9c8d580670fda8b695204a X-Mailman-Approved-At: Thu, 25 Oct 2007 01:37:43 -0400 Cc: Nevil Brownlee , pmol@ietf.org X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: pmol-bounces@ietf.org On 2007-10-25 01:22, Aamer Akhter (aakhter) wrote: > Al, et al, > >>> When metric Foobar is measured, metrics Barfoo and Foofoo must >>> be measured simultaneously between the same endpoints. >>> Results for Foobar must be accompanied by the results for Barfoo >>> and Foofoo. If the observed variance of Barfoo or Foofoo exceeds >>> , the results for Foobar must be labelled as 'unreliable'. >>> > > This makes sense. > > I wonder however, about the cases where the application (or L7 entity etc) that is aware of the underlying disruptive transport and is working around it. > > Example: the work of digital fountain. I think in these cases, the metrics for the underlying would be even more important/relevant, but perhaps the final result shouldn't be labeled unreliable. > > Thoughts? My thought is that the variance in the "upper" metric will be very hard to understand if there is a large variance in the "lower" metrics. There seem likely to be non-linear effects at work. If the upper metric is being used only as an observation (e.g. to check SLA conformance) that is one thing, but if it's being used to make any kind of prediction or to analyze the behaviour of the upper layer protocol, I think describing it as unreliable is justified. Brian _______________________________________________ PMOL mailing list PMOL@ietf.org https://www1.ietf.org/mailman/listinfo/pmol Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1IknpJ-0004Xg-6S; Wed, 24 Oct 2007 17:31:45 -0400 Received: from pmol by megatron.ietf.org with local (Exim 4.43) id 1IknpH-0004XQ-Rf for pmol-confirm+ok@megatron.ietf.org; Wed, 24 Oct 2007 17:31:43 -0400 Received: from [10.90.34.44] (helo=chiedprmail1.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1IknpG-0004WL-U5; Wed, 24 Oct 2007 17:31:42 -0400 Received: from mail146.messagelabs.com ([216.82.245.131]) by chiedprmail1.ietf.org with esmtp (Exim 4.43) id 1IknpG-0002Xa-IO; Wed, 24 Oct 2007 17:31:42 -0400 X-VirusChecked: Checked X-Env-Sender: acmorton@att.com X-Msg-Ref: server-3.tower-146.messagelabs.com!1193261500!9870074!1 X-StarScan-Version: 5.5.12.14.2; banners=-,-,- X-Originating-IP: [144.160.128.149] Received: (qmail 3107 invoked from network); 24 Oct 2007 21:31:41 -0000 Received: from sbcsmtp9.sbc.com (HELO flph024.enaf.ffdc.sbc.com) (144.160.128.149) by server-3.tower-146.messagelabs.com with AES256-SHA encrypted SMTP; 24 Oct 2007 21:31:41 -0000 Received: from enaf.ffdc.sbc.com (localhost.localdomain [127.0.0.1]) by flph024.enaf.ffdc.sbc.com (8.14.0/8.14.0) with ESMTP id l9OLVd4m009627; Wed, 24 Oct 2007 14:31:40 -0700 Received: from flph023.ffdc.sbc.com (flph023.ffdc.sbc.com [150.234.117.36]) by flph024.enaf.ffdc.sbc.com (8.14.0/8.14.0) with ESMTP id l9OLVaBQ009599; Wed, 24 Oct 2007 14:31:36 -0700 Received: from ffdc.sbc.com (localhost.localdomain [127.0.0.1]) by flph023.ffdc.sbc.com (8.14.0/8.14.0) with ESMTP id l9OLVaqA023935; Wed, 24 Oct 2007 14:31:36 -0700 Received: from maillennium.att.com (dns.maillennium.att.com [135.25.114.99]) by flph023.ffdc.sbc.com (8.14.0/8.14.0) with ESMTP id l9OLVTV2023804; Wed, 24 Oct 2007 14:31:29 -0700 Message-Id: <200710242131.l9OLVTV2023804@flph023.ffdc.sbc.com> Received: from acmt.att.com (acmt.mt.att.com[135.16.251.73](misconfigured sender)) by maillennium.att.com (mailgw1) with SMTP id <20071024213128gw10010gite>; Wed, 24 Oct 2007 21:31:28 +0000 X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Wed, 24 Oct 2007 17:29:48 -0400 To: "MAK, Leen \(Leen\)" From: Al Morton In-Reply-To: References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Spam-Score: 1.5 (+) X-Scan-Signature: 21c69d3cfc2dd19218717dbe1d974352 Cc: pmol@ietf.org, ops-area@ietf.org Subject: [PMOL] Re: [OPS-AREA] RE: WG Review: Performance Metrics at Other Layers (pmol) X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: pmol-bounces@ietf.org At 09:50 AM 10/24/2007, MAK, Leen \(Leen\) wrote: >...The word "also" suggests that this task is understood as a kind of >secondary task, next to the task to "describe the necessary elements of >performance metrics of protocols and applications ". > >I would like to suggest that the specification of intended audience and >motivation should be a primary task.... Yes, a clarification for both the charter and the memo itself. Al (pmol@ietf.org is the best place for further discussion) At 09:50 AM 10/24/2007, MAK, Leen \(Leen\) wrote: >All, > >I read in the draft charter: "The framework will also address the need >to specify the intended audience and the motivation for the performance >metrics." > >The word "also" suggests that this task is understood as a kind of >secondary task, next to the task to "describe the necessary elements of >performance metrics of protocols and applications ". > >I would like to suggest that the specification of intended audience and >motivation should be a primary task. >If working bottom-up from the consideration of protocols and their >properties, it is very easy and very tempting to define all sorts of >nice and fancy parameters and metrics. However, collecting those >parameters and storing them and processing them does not come for free. > >In order to limit the set of metrics to what is really useful, the >approach should be the other way. The first question to be answered >should be: which metrics are needed by the network operators to properly >maintain the network? > >Regards, > >Leen Mak. > _______________________________________________ PMOL mailing list PMOL@ietf.org https://www1.ietf.org/mailman/listinfo/pmol Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1IkfFm-0006Yj-5U; Wed, 24 Oct 2007 08:22:30 -0400 Received: from pmol by megatron.ietf.org with local (Exim 4.43) id 1IkfFl-0006YS-3e for pmol-confirm+ok@megatron.ietf.org; Wed, 24 Oct 2007 08:22:29 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1IkfFk-0006Xz-Gr for pmol@ietf.org; Wed, 24 Oct 2007 08:22:28 -0400 Received: from rtp-iport-2.cisco.com ([64.102.122.149]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1IkfFe-0002mN-B0 for pmol@ietf.org; Wed, 24 Oct 2007 08:22:28 -0400 X-IronPort-AV: E=Sophos;i="4.21,324,1188792000"; d="scan'208";a="135551909" Received: from rtp-dkim-2.cisco.com ([64.102.121.159]) by rtp-iport-2.cisco.com with ESMTP; 24 Oct 2007 08:22:16 -0400 Received: from rtp-core-1.cisco.com (rtp-core-1.cisco.com [64.102.124.12]) by rtp-dkim-2.cisco.com (8.12.11/8.12.11) with ESMTP id l9OCMHJM012784; Wed, 24 Oct 2007 08:22:17 -0400 Received: from xbh-rtp-201.amer.cisco.com (xbh-rtp-201.cisco.com [64.102.31.12]) by rtp-core-1.cisco.com (8.12.10/8.12.6) with ESMTP id l9OCM593024666; Wed, 24 Oct 2007 12:22:16 GMT Received: from xmb-rtp-201.amer.cisco.com ([64.102.31.13]) by xbh-rtp-201.amer.cisco.com with Microsoft SMTPSVC(6.0.3790.1830); Wed, 24 Oct 2007 08:22:13 -0400 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Subject: RE: [PMOL] Re: draft-morton-perf-metrics-framework-00 Date: Wed, 24 Oct 2007 08:22:10 -0400 Message-ID: In-Reply-To: <200710240230.l9O2U0BN015298@mlph070.sfdc.sbc.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [PMOL] Re: draft-morton-perf-metrics-framework-00 Thread-Index: AcgV5h1mEtMcCgprSd6a1kMQWnWmbQAUbEeQ References: <471E74AC.90209@gmail.com> <200710240230.l9O2U0BN015298@mlph070.sfdc.sbc.com> From: "Aamer Akhter (aakhter)" To: "Al Morton" , "Brian E Carpenter" X-OriginalArrivalTime: 24 Oct 2007 12:22:13.0799 (UTC) FILETIME=[822C7370:01C81638] X-TM-AS-Product-Ver: SMEX-8.0.0.1181-5.000.1023-15502.002 X-TM-AS-Result: No--34.621600-8.000000-31 X-TM-AS-User-Approved-Sender: No X-TM-AS-User-Blocked-Sender: No DKIM-Signature: v=0.5; a=rsa-sha256; q=dns/txt; l=3067; t=1193228537; x=1194092537; c=relaxed/simple; s=rtpdkim2001; h=Content-Type:From:Subject:Content-Transfer-Encoding:MIME-Version; d=cisco.com; i=aakhter@cisco.com; z=From:=20=22Aamer=20Akhter=20(aakhter)=22=20 |Subject:=20RE=3A=20[PMOL]=20Re=3A=20draft-morton-perf-metrics-framework- 00 |Sender:=20 |To:=20=22Al=20Morton=22=20, =0A=20=20=20=20=20=20=20=20 =22Brian=20E=20Carpenter=22=20; bh=B42LNUl8WB8Kvax4R6pkM6DIcQ7H1DmlCJRCYDTq0Y0=; b=mg6/PbLNx8R9jbRxceCNGUHqj8IcGi6UJbDf327rNTgh3qm6/O+heIAqzo7m0jOt8GjQahmp AT4+f8zi4EGyvc8S4xuo0k+x5VdS9YwF/EgZM3Sv3UbIILU07PQmiWsI; Authentication-Results: rtp-dkim-2; header.From=aakhter@cisco.com; dkim=pass ( sig from cisco.com/rtpdkim2001 verified; ); X-Spam-Score: -4.0 (----) X-Scan-Signature: c3a18ef96977fc9bcc21a621cbf1174b Cc: Nevil Brownlee , pmol@ietf.org X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: pmol-bounces@ietf.org Al, et al, > > > > When metric Foobar is measured, metrics Barfoo and Foofoo must > > be measured simultaneously between the same endpoints. > > Results for Foobar must be accompanied by the results for Barfoo > > and Foofoo. If the observed variance of Barfoo or Foofoo exceeds > > , the results for Foobar must be labelled as 'unreliable'. > > This makes sense.=20 I wonder however, about the cases where the application (or L7 entity = etc) that is aware of the underlying disruptive transport and is working = around it.=20 Example: the work of digital fountain. I think in these cases, the = metrics for the underlying would be even more important/relevant, but = perhaps the final result shouldn't be labeled unreliable. Thoughts? --=20 Aamer Akhter / aa@cisco.com Ent & Commercial Systems, cisco Systems > -----Original Message----- > From: Al Morton [mailto:acmorton@att.com] > Sent: Tuesday, October 23, 2007 10:28 PM > To: Brian E Carpenter > Cc: Nevil Brownlee; pmol@ietf.org > Subject: [PMOL] Re: draft-morton-perf-metrics-framework-00 >=20 > Hi Brian, >=20 > The draft is intended for discussion on the list. >=20 > You've raised a worthwhile consideration for the draft, so > I have cc'd the pmol list for further discussion. >=20 > regards, > Al >=20 > At 06:24 PM 10/23/2007, Brian E Carpenter wrote: > >Al, > > > >I'm not sure where the draft is being discussed, so please feel > >free to add in any appropriate cc's, including your co-author, > >whose address is missing in the draft. > > > >This comment was triggered by seeing the draft charter for > >pmol, but I think it really belongs in the document and not in > >the charter. > > > >It's fairly clear for the traditional ippm/bmwg metrics what > >is meant by "all other things being equal." Once you start > >discussing metrics for the application level, as in pmol, > >that's much less clear. I believe that any definition of > >an upper layer metric should include a discussion of the extent > >to which lower layer performance must be held stable for the > >upper layer metric to be of value, and in fact what lower layer > >metrics MUST be measured simultaneously and what bounds on > >those metrics must be met. I'd envisage statements of the type > > > > When metric Foobar is measured, metrics Barfoo and Foofoo must > > be measured simultaneously between the same endpoints. > > Results for Foobar must be accompanied by the results for Barfoo > > and Foofoo. If the observed variance of Barfoo or Foofoo exceeds > > , the results for Foobar must be labelled as 'unreliable'. > > > >As a one word summary, this can be called 'dependencies', but > >I think the framework needs to be explicit about it. > > > > Brian > >-- > >Regards > > Brian Carpenter > > University of Auckland > > >=20 >=20 >=20 > _______________________________________________ > PMOL mailing list > PMOL@ietf.org > https://www1.ietf.org/mailman/listinfo/pmol _______________________________________________ PMOL mailing list PMOL@ietf.org https://www1.ietf.org/mailman/listinfo/pmol Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1IkW0w-0000Ka-Gh; Tue, 23 Oct 2007 22:30:34 -0400 Received: from pmol by megatron.ietf.org with local (Exim 4.43) id 1IkW0v-0000Jk-Cf for pmol-confirm+ok@megatron.ietf.org; Tue, 23 Oct 2007 22:30:33 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1IkW0u-0000Jc-IW for pmol@ietf.org; Tue, 23 Oct 2007 22:30:32 -0400 Received: from mail146.messagelabs.com ([216.82.245.131]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1IkW0o-0000tn-4y for pmol@ietf.org; Tue, 23 Oct 2007 22:30:32 -0400 X-VirusChecked: Checked X-Env-Sender: acmorton@att.com X-Msg-Ref: server-3.tower-146.messagelabs.com!1193193010!9796287!1 X-StarScan-Version: 5.5.12.14.2; banners=-,-,- X-Originating-IP: [144.160.20.53] Received: (qmail 1519 invoked from network); 24 Oct 2007 02:30:10 -0000 Received: from sbcsmtp6.sbc.com (HELO mlph073.enaf.sfdc.sbc.com) (144.160.20.53) by server-3.tower-146.messagelabs.com with AES256-SHA encrypted SMTP; 24 Oct 2007 02:30:10 -0000 Received: from enaf.sfdc.sbc.com (localhost.localdomain [127.0.0.1]) by mlph073.enaf.sfdc.sbc.com (8.14.0/8.14.0) with ESMTP id l9O2UAJS023598 for ; Tue, 23 Oct 2007 22:30:10 -0400 Received: from mlph070.sfdc.sbc.com (mlph070.sfdc.sbc.com [144.155.224.139]) by mlph073.enaf.sfdc.sbc.com (8.14.0/8.14.0) with ESMTP id l9O2U9Eu023593 for ; Tue, 23 Oct 2007 22:30:09 -0400 Received: from sfdc.sbc.com (localhost.localdomain [127.0.0.1]) by mlph070.sfdc.sbc.com (8.14.0/8.14.0) with ESMTP id l9O2U9un015514 for ; Tue, 23 Oct 2007 22:30:09 -0400 Received: from maillennium.att.com (mailgw1.maillennium.att.com [135.25.114.99]) by mlph070.sfdc.sbc.com (8.14.0/8.14.0) with ESMTP id l9O2U0BN015298 for ; Tue, 23 Oct 2007 22:30:06 -0400 Message-Id: <200710240230.l9O2U0BN015298@mlph070.sfdc.sbc.com> Received: from acmt.att.com (unknown[135.210.81.187](misconfigured sender)) by maillennium.att.com (mailgw1) with SMTP id <20071024023000gw10010g7ee>; Wed, 24 Oct 2007 02:30:00 +0000 X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Tue, 23 Oct 2007 22:28:20 -0400 To: Brian E Carpenter From: Al Morton In-Reply-To: <471E74AC.90209@gmail.com> References: <471E74AC.90209@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Spam-Score: 1.5 (+) X-Scan-Signature: c1c65599517f9ac32519d043c37c5336 Cc: Nevil Brownlee , pmol@ietf.org Subject: [PMOL] Re: draft-morton-perf-metrics-framework-00 X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: pmol-bounces@ietf.org Hi Brian, The draft is intended for discussion on the list. You've raised a worthwhile consideration for the draft, so I have cc'd the pmol list for further discussion. regards, Al At 06:24 PM 10/23/2007, Brian E Carpenter wrote: >Al, > >I'm not sure where the draft is being discussed, so please feel >free to add in any appropriate cc's, including your co-author, >whose address is missing in the draft. > >This comment was triggered by seeing the draft charter for >pmol, but I think it really belongs in the document and not in >the charter. > >It's fairly clear for the traditional ippm/bmwg metrics what >is meant by "all other things being equal." Once you start >discussing metrics for the application level, as in pmol, >that's much less clear. I believe that any definition of >an upper layer metric should include a discussion of the extent >to which lower layer performance must be held stable for the >upper layer metric to be of value, and in fact what lower layer >metrics MUST be measured simultaneously and what bounds on >those metrics must be met. I'd envisage statements of the type > > When metric Foobar is measured, metrics Barfoo and Foofoo must > be measured simultaneously between the same endpoints. > Results for Foobar must be accompanied by the results for Barfoo > and Foofoo. If the observed variance of Barfoo or Foofoo exceeds > , the results for Foobar must be labelled as 'unreliable'. > >As a one word summary, this can be called 'dependencies', but >I think the framework needs to be explicit about it. > > Brian >-- >Regards > Brian Carpenter > University of Auckland > _______________________________________________ PMOL mailing list PMOL@ietf.org https://www1.ietf.org/mailman/listinfo/pmol Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1Ik2Ey-0003qw-Jv; Mon, 22 Oct 2007 14:43:04 -0400 Received: from pmol by megatron.ietf.org with local (Exim 4.43) id 1Ik2Ex-0003pL-8W for pmol-confirm+ok@megatron.ietf.org; Mon, 22 Oct 2007 14:43:03 -0400 Received: from [10.90.34.44] (helo=chiedprmail1.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1Ik2El-0003l4-SR; Mon, 22 Oct 2007 14:42:51 -0400 Received: from de307622-de-outbound.net.avaya.com ([198.152.71.100]) by chiedprmail1.ietf.org with esmtp (Exim 4.43) id 1Ik2El-0000K0-3z; Mon, 22 Oct 2007 14:42:51 -0400 X-IronPort-AV: E=Sophos;i="4.21,312,1188792000"; d="scan'208";a="67769875" Received: from unknown (HELO 307622ANEX5.global.avaya.com) ([135.64.140.14]) by de307622-de-outbound.net.avaya.com with ESMTP; 22 Oct 2007 14:42:49 -0400 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Mon, 22 Oct 2007 20:42:07 +0200 Message-ID: X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: WG Review: Performance Metrics at Other Layers (pmol) Thread-Index: AcgU1+vTdjWi6NJdT7i92QOkA3WAqQAA0cow From: "Romascanu, Dan (Dan)" To: , , "IETF IPPM WG" , X-Spam-Score: 0.0 (/) X-Scan-Signature: a7d2e37451f7f22841e3b6f40c67db0f Cc: Subject: [PMOL] FW: WG Review: Performance Metrics at Other Layers (pmol) X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: pmol-bounces@ietf.org =20 =20 -----Original Message----- From: IESG Secretary [mailto:iesg-secretary@ietf.org]=20 Sent: Monday, October 22, 2007 8:15 PM To: ietf-announce@ietf.org Subject: WG Review: Performance Metrics at Other Layers (pmol)=20 A new IETF working group has been proposed in the Operations and Management Area. The IESG has not made any determination as yet. =20 The following draft charter was submitted, and is provided for informational purposes only. Please send your comments to the IESG mailing list (iesg@ietf.org) by October 29. +++ Performance Metrics at Other Layers (pmol) =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Current Status: Proposed Working Group WG Chairs: TBD Operations and Management Area: Dan Romascanu Ronald Bonica Description: The successful implementation and operation of IP based applications often depends on some underlying performance measurement infrastructure that helps service operators or network managers to recognize when performance is unsatisfactory and identify problems affecting service quality. Standardized performance metrics add the desirable features of consistent implementation, interpretation, no comparison. The IETF has two Working Groups dedicated to the development of performance metrics however each has strict limitations in their charters: - The Benchmarking Methodology WG has addressed a range of networking technologies and protocols in their long history (such as IEEE 802.3, ATM, Frame Relay, and Routing Protocols), but the charter strictly limits their Performance characterizations to the laboratory environment. - The IP Performance Metrics WG has the mandate to develop metrics applicable to the performance of Internet data delivery, but it is specifically prohibited from developing metrics that characterize traffic (such as a VoIP stream). The IETF also has current and completed activities related to the reporting of application performance metrics (e.g. RAQMON and RTCP XR) and is also actively involved in the development of reliable transport protocols which would affect the relationship between IP performance and application performance. Thus there is a gap in the currently chartered coverage of IETF WGs: development of performance metrics for IP-based protocols and applications that operate over UDP, TCP, SCTP, DCCP, Forward Error Correction (FEC) and other robust transport protocols, and that can be used to characterize traffic on live networks. The working group will focus on the completion of two RFCs: 1. A PMOL framework and guidelines memo that will describe the necessary elements of performance metrics of protocols and applications transported over IETF-specified protocols (such as the formal definition, purpose, and units of measure) and the various types of metrics that characterize traffic on live networks (such as metrics derived from other metrics, possibly on lower layers). The framework will also address the need to specify the intended audience and the motivation for the performance metrics. There will also be guidelines for a performance metric development process that includes entry criteria for new proposals (how a proposal might be evaluated for possible endorsement by a protocol development working group), and how an successful proposal will be developed by PMOL WG in cooperation with a protocol development WG. 2. A proof-of-concept RFC defining performance metrics for SIP, based on draft-malas-performance-metrics. This memo would serve as an example of the framework and the PMOL development process in the IETF. Discussion of new work proposals is strongly discouraged under the initial charter of the PMOL WG, except to advise a protocol development WG when they are evaluating a new work proposal for related performance metrics. The PMOL WG will also be guided by a document describing how memos defining performance metrics are intended to advance along the IETF Standards track (draft-bradner-metricstest). PMOL WG will take advantage of expertise and seek to avoid overlap with other standards development organizations, such as ETSI STQ, ITU-T SG 12, ATIS IIF, ATIS PRQC, and others. Milestones June 08 SIP Performance Metrics Draft to IESG Review for consideration as Proposed Standard Sept 08 PMOL Framework and Guidelines Draft to IESG Review for consideration as BCP _______________________________________________ PMOL mailing list PMOL@ietf.org https://www1.ietf.org/mailman/listinfo/pmol Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1IjbjI-0001Ut-F7; Sun, 21 Oct 2007 10:24:36 -0400 Received: from pmol by megatron.ietf.org with local (Exim 4.43) id 1IjbjG-0001TW-1S for pmol-confirm+ok@megatron.ietf.org; Sun, 21 Oct 2007 10:24:34 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1Ijbj1-0001Gf-JS for pmol@ietf.org; Sun, 21 Oct 2007 10:24:19 -0400 Received: from de307622-de-outbound.net.avaya.com ([198.152.71.100]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1Ijbiv-0000aX-CE for pmol@ietf.org; Sun, 21 Oct 2007 10:24:19 -0400 X-IronPort-AV: E=Sophos;i="4.21,306,1188792000"; d="url'?txt'208?scan'208,208";a="67479639" Received: from unknown (HELO 307622ANEX5.global.avaya.com) ([135.64.140.14]) by de307622-de-outbound.net.avaya.com with ESMTP; 21 Oct 2007 10:24:05 -0400 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----_=_NextPart_001_01C813ED.F150BC75" Date: Sun, 21 Oct 2007 16:23:23 +0200 Message-ID: X-MS-Has-Attach: yes X-MS-TNEF-Correlator: Thread-Topic: I-D Action:draft-morton-perf-metrics-framework-00.txt Thread-Index: AcgQWbDYN+CC1t3IR6OsDnkxLdkovwDlAjvQ From: "Romascanu, Dan (Dan)" To: X-Spam-Score: 0.0 (/) X-Scan-Signature: 2086112c730e13d5955355df27e3074b Subject: [PMOL] FW: I-D Action:draft-morton-perf-metrics-framework-00.txt X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: pmol-bounces@ietf.org This is a multi-part message in MIME format. ------_=_NextPart_001_01C813ED.F150BC75 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable =20 For those of you who may have missed the announcement - the initial pmol individual submission.=20 Also, you may see in the coming days the IETF Last Call for the WG charter.=20 Dan =20 -----Original Message----- From: Internet-Drafts@ietf.org [mailto:Internet-Drafts@ietf.org]=20 Sent: Wednesday, October 17, 2007 2:50 AM To: i-d-announce@ietf.org Subject: I-D Action:draft-morton-perf-metrics-framework-00.txt=20 A New Internet-Draft is available from the on-line Internet-Drafts directories. Title : Framework for Performance Metric Development Author(s) : A. Morton Filename : draft-morton-perf-metrics-framework-00.txt Pages : 8 Date : 2007-10-16 This memo describes a framework and guidelines for the development of performance metrics that are beyond the scope of existing working group charters in the IETF. In this version, the memo refers to a Performance Metrics Entity, or PM Entity, which may in future be a working group or directorate or a combination of these two.Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-morton-perf-metrics-framework- 00.txt To remove yourself from the I-D Announcement list, send a message to i-d-announce-request@ietf.org with the word unsubscribe in the body of the message. You can also visit https://www1.ietf.org/mailman/listinfo/I-D-announce to change your subscription settings. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-morton-perf-metrics-framework-00.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-morton-perf-metrics-framework-00.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. ------_=_NextPart_001_01C813ED.F150BC75 Content-Type: application/octet-stream; name="ATT1012450.TXT" Content-Transfer-Encoding: base64 Content-Description: ATT1012450.TXT Content-Disposition: attachment; filename="ATT1012450.TXT" Q29udGVudC1UeXBlOiBNZXNzYWdlL0V4dGVybmFsLWJvZHk7IGFjY2Vzcy10eXBlPSJtYWlsLXNl cnZlciI7DQoJc2VydmVyPSJtYWlsc2VydkBpZXRmLm9yZyINCg0KQ29udGVudC1UeXBlOiB0ZXh0 L3BsYWluDQpDb250ZW50LUlEOiA8MjAwNy0xMC0xNjIwNDMxNy5JLURAaWV0Zi5vcmc+DQoNCkVO Q09ESU5HIG1pbWUNCkZJTEUgL2ludGVybmV0LWRyYWZ0cy9kcmFmdC1tb3J0b24tcGVyZi1tZXRy aWNzLWZyYW1ld29yay0wMC50eHQNCg== ------_=_NextPart_001_01C813ED.F150BC75 Content-Type: application/octet-stream; name="draft-morton-perf-metrics-framework-00.URL" Content-Transfer-Encoding: base64 Content-Description: draft-morton-perf-metrics-framework-00.URL Content-Disposition: attachment; filename="draft-morton-perf-metrics-framework-00.URL" W0ludGVybmV0U2hvcnRjdXRdDQpVUkw9ZnRwOi8vZnRwLmlldGYub3JnL2ludGVybmV0LWRyYWZ0 cy9kcmFmdC1tb3J0b24tcGVyZi1tZXRyaWNzLWZyYW1ld29yay0wMC50eHQNCg== ------_=_NextPart_001_01C813ED.F150BC75 Content-Type: text/plain; name="ATT1012451.txt" Content-Transfer-Encoding: base64 Content-Description: ATT1012451.txt Content-Disposition: attachment; filename="ATT1012451.txt" X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18NCkktRC1Bbm5v dW5jZSBtYWlsaW5nIGxpc3QNCkktRC1Bbm5vdW5jZUBpZXRmLm9yZw0KaHR0cHM6Ly93d3cxLmll dGYub3JnL21haWxtYW4vbGlzdGluZm8vaS1kLWFubm91bmNlDQo= ------_=_NextPart_001_01C813ED.F150BC75 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ PMOL mailing list PMOL@ietf.org https://www1.ietf.org/mailman/listinfo/pmol ------_=_NextPart_001_01C813ED.F150BC75-- Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1IhysK-0001RL-US; Tue, 16 Oct 2007 22:43:13 -0400 Received: from pmol by megatron.ietf.org with local (Exim 4.43) id 1IhysK-0001RE-2y for pmol-confirm+ok@megatron.ietf.org; Tue, 16 Oct 2007 22:43:12 -0400 Received: from [10.90.34.44] (helo=chiedprmail1.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1IhysJ-0001R2-4f for pmol@ietf.org; Tue, 16 Oct 2007 22:43:11 -0400 Received: from mail120.messagelabs.com ([216.82.250.83]) by chiedprmail1.ietf.org with esmtp (Exim 4.43) id 1IhysI-0003cl-Hx for pmol@ietf.org; Tue, 16 Oct 2007 22:43:11 -0400 X-VirusChecked: Checked X-Env-Sender: acmorton@att.com X-Msg-Ref: server-2.tower-120.messagelabs.com!1192588989!17722544!1 X-StarScan-Version: 5.5.12.14.2; banners=-,-,- X-Originating-IP: [144.160.128.149] Received: (qmail 23280 invoked from network); 17 Oct 2007 02:43:09 -0000 Received: from sbcsmtp9.sbc.com (HELO flph024.enaf.ffdc.sbc.com) (144.160.128.149) by server-2.tower-120.messagelabs.com with AES256-SHA encrypted SMTP; 17 Oct 2007 02:43:09 -0000 Received: from enaf.ffdc.sbc.com (localhost.localdomain [127.0.0.1]) by flph024.enaf.ffdc.sbc.com (8.14.0/8.14.0) with ESMTP id l9H2h8Px006031 for ; Tue, 16 Oct 2007 19:43:09 -0700 Received: from flph023.ffdc.sbc.com (flph023.ffdc.sbc.com [150.234.117.36]) by flph024.enaf.ffdc.sbc.com (8.14.0/8.14.0) with ESMTP id l9H2h7MP006025 for ; Tue, 16 Oct 2007 19:43:07 -0700 Received: from ffdc.sbc.com (localhost.localdomain [127.0.0.1]) by flph023.ffdc.sbc.com (8.14.0/8.14.0) with ESMTP id l9H2h7ad015212 for ; Tue, 16 Oct 2007 19:43:07 -0700 Received: from maillennium.att.com (dns.maillennium.att.com [135.25.114.99]) by flph023.ffdc.sbc.com (8.14.0/8.14.0) with ESMTP id l9H2ghYC015028 for ; Tue, 16 Oct 2007 19:43:00 -0700 Message-Id: <200710170243.l9H2ghYC015028@flph023.ffdc.sbc.com> Received: from acmt.att.com (unknown[135.210.105.107](misconfigured sender)) by maillennium.att.com (mailgw1) with SMTP id <20071017024243gw10010g62e>; Wed, 17 Oct 2007 02:42:43 +0000 X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Tue, 16 Oct 2007 22:41:15 -0400 To: pmol@ietf.org From: Al Morton Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Spam-Score: 1.5 (+) X-Scan-Signature: fb6060cb60c0cea16e3f7219e40a0a81 Subject: [PMOL] Fwd: I-D Action:draft-morton-perf-metrics-framework-00.txt X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: pmol-bounces@ietf.org >A New Internet-Draft is available from the on-line Internet-Drafts >directories. > > Title : Framework for Performance Metric Development > Author(s) : A. Morton, A. Clark > Filename : draft-morton-perf-metrics-framework-00.txt > Pages : 8 > Date : 2007-10-16 > >This memo describes a framework and guidelines for the development of >performance metrics that are beyond the scope of existing working >group charters in the IETF. In this version, the memo refers to a >Performance Metrics Entity, or PM Entity, which may in future be a >working group or directorate or a combination of these two.Requirements >Language > >The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", >"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this >document are to be interpreted as described in RFC 2119 [RFC2119]. > >A URL for this Internet-Draft is: >http://www.ietf.org/internet-drafts/draft-morton-perf-metrics-framework-00.txt > >To remove yourself from the I-D Announcement list, send a message to >i-d-announce-request@ietf.org with the word unsubscribe in the body of >the message. >You can also visit https://www1.ietf.org/mailman/listinfo/I-D-announce >to change your subscription settings. > >Internet-Drafts are also available by anonymous FTP. Login with the >username "anonymous" and a password of your e-mail address. After >logging in, type "cd internet-drafts" and then > "get draft-morton-perf-metrics-framework-00.txt". > >A list of Internet-Drafts directories can be found in >http://www.ietf.org/shadow.html >or ftp://ftp.ietf.org/ietf/1shadow-sites.txt > >Internet-Drafts can also be obtained by e-mail. > >Send a message to: > mailserv@ietf.org. >In the body type: > "FILE /internet-drafts/draft-morton-perf-metrics-framework-00.txt". > >NOTE: The mail server at ietf.org can return the document in > MIME-encoded form by using the "mpack" utility. To use this > feature, insert the command "ENCODING mime" before the "FILE" > command. To decode the response(s), you will need "munpack" or > a MIME-compliant mail reader. Different MIME-compliant mail readers > exhibit different behavior, especially when dealing with > "multipart" MIME messages (i.e. documents which have been split > up into multiple messages), so check your local documentation on > how to manipulate these messages. > >Below is the data which will enable a MIME compliant mail reader >implementation to automatically retrieve the ASCII version of the >Internet-Draft. > >Content-Type: text/plain >Content-ID: <2007-10-16204317.I-D@ietf.org> > >ENCODING mime >FILE /internet-drafts/draft-morton-perf-metrics-framework-00.txt > > > >_______________________________________________ >I-D-Announce mailing list >I-D-Announce@ietf.org >https://www1.ietf.org/mailman/listinfo/i-d-announce _______________________________________________ PMOL mailing list PMOL@ietf.org https://www1.ietf.org/mailman/listinfo/pmol Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1Ihrhu-0002J9-Ed; Tue, 16 Oct 2007 15:03:58 -0400 Received: from pmol by megatron.ietf.org with local (Exim 4.43) id 1Ihrhs-0002GY-E7 for pmol-confirm+ok@megatron.ietf.org; Tue, 16 Oct 2007 15:03:56 -0400 Received: from [10.90.34.44] (helo=chiedprmail1.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1Ihrhf-0001wl-Id for pmol@ietf.org; Tue, 16 Oct 2007 15:03:43 -0400 Received: from mail120.messagelabs.com ([216.82.250.83]) by chiedprmail1.ietf.org with esmtp (Exim 4.43) id 1IhrhZ-0001yj-N3 for pmol@ietf.org; Tue, 16 Oct 2007 15:03:38 -0400 X-VirusChecked: Checked X-Env-Sender: acmorton@att.com X-Msg-Ref: server-5.tower-120.messagelabs.com!1192561415!26201233!1 X-StarScan-Version: 5.5.12.14.2; banners=-,-,- X-Originating-IP: [144.160.128.149] Received: (qmail 10141 invoked from network); 16 Oct 2007 19:03:35 -0000 Received: from sbcsmtp9.sbc.com (HELO flph024.enaf.ffdc.sbc.com) (144.160.128.149) by server-5.tower-120.messagelabs.com with AES256-SHA encrypted SMTP; 16 Oct 2007 19:03:35 -0000 Received: from enaf.ffdc.sbc.com (localhost.localdomain [127.0.0.1]) by flph024.enaf.ffdc.sbc.com (8.14.0/8.14.0) with ESMTP id l9GJ3ZJF016058 for ; Tue, 16 Oct 2007 12:03:35 -0700 Received: from flph023.ffdc.sbc.com (flph023.ffdc.sbc.com [150.234.117.36]) by flph024.enaf.ffdc.sbc.com (8.14.0/8.14.0) with ESMTP id l9GJ3OkE015955 for ; Tue, 16 Oct 2007 12:03:28 -0700 Received: from ffdc.sbc.com (localhost.localdomain [127.0.0.1]) by flph023.ffdc.sbc.com (8.14.0/8.14.0) with ESMTP id l9GJ3O3D013408 for ; Tue, 16 Oct 2007 12:03:24 -0700 Received: from maillennium.att.com (dns.maillennium.att.com [135.25.114.99]) by flph023.ffdc.sbc.com (8.14.0/8.14.0) with ESMTP id l9GJ3JKa013328 for ; Tue, 16 Oct 2007 12:03:20 -0700 Message-Id: <200710161903.l9GJ3JKa013328@flph023.ffdc.sbc.com> Received: from acmt.att.com (acmt.mt.att.com[135.16.251.73](misconfigured sender)) by maillennium.att.com (mailgw1) with SMTP id <20071016190319gw10010g14e>; Tue, 16 Oct 2007 19:03:19 +0000 X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Tue, 16 Oct 2007 15:01:56 -0400 To: Lars Eggert , pmol@ietf.org From: Al Morton Subject: Re: [PMOL] Suggested updates to proposed charter wording In-Reply-To: <2F377D6F-C725-4F63-A01D-33386F8256A2@nokia.com> References: <2F377D6F-C725-4F63-A01D-33386F8256A2@nokia.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Spam-Score: 1.5 (+) X-Scan-Signature: 5ebbf074524e58e662bc8209a6235027 Cc: X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: pmol-bounces@ietf.org Hi Lars, We haven't ignored your comment, but it seems I haven't done enough to address it. I'm struggling with how much detail to add in the charter, and hope some discussion will help refine what's needed. When you said, referring to the guidelines and framework deliverable: >Some of the content of this proposed WG deliverable is what we >typically want to see already during the BOF proposal phase: > >Specifically, the "motivation for work", "how that work fills a >need and any gap in IETF-chartered work", "motivation for the >performance metrics" seem pretty relevant already when asking for a >BOF. Our intent was to document aspects of the original BOF proposal, which motivated the work area and identified gaps in IETF coverage, as an Intro to the guidelines and framework memo. The proposed PMOL charter just gives a short summary of the BOF proposal text. Maybe we should add more background in the charter? from the BoF minutes: >The BOF introduced the IETF community to the possibility of a >generalized activity to define standardized performance metrics. The >existence of a growing list of Internet-Drafts on performance metrics (with community interest in development, but in un-chartered areas) >supports the need for this activity, and the majority of people >present supported the proposition that IETF should be working in >these areas. No one objected to any of these proposals. However, >many people from beyond the performance community sought >clarifications about the scope of the activity. The proposal was >deliberately made broad to cover the wide variety of performance >work that is currently excluded by the IPPM and BMWG charters. or maybe we shouldn't mention the Intro material at all? Last paragraph of your comment: >Likewise, information on "the various types of metrics that may be >prepared in this work" seem important to define the scope for the >planned WG. "Entry criteria for new proposals" is also something >that should be in the proposed charter already. The scope of the WG is very limited, and it allows development of only the two deliverables described: the guidelines and framework memo (a process for developing performance metrics that don't fall within other charters and their frameworks) and the SIP metrics memo. The PMOL WG must develop the process and entry criteria for *additional* proposals, and it will only work on additional deliverables if explicitly chartered to do so. A key aspect of the process is that most proposals will be vetted in a protocol development WG before reaching PMOL WG. (maybe that paragraph should be part of the charter)... We're certainly trying to do something unusual in my short IETF experience, and appreciate further suggestions or questions to help clarify what the proposed PMOL can do. regards, Al At 01:51 PM 10/15/2007, Lars Eggert wrote: >Nothing in the new charter addresses my comment on the previous version: > >On 2007-9-26, at 3:09, ext Lars Eggert wrote: >>On 2007-9-25, at 21:22, ext Alan Clark wrote: >>>1. A PMOL framework and guidelines memo that outlines the motivation >>> for work to define performance metrics for applications >>>transported >>> over IETF-specified protocols, how that work fills a need and >>>any gap >>> in IETF-chartered work. The framework will describe the necessary >>> elements of performance metric drafts and the various types of >>>metrics >>> that may be prepared in this work. The framework will also >>>address the >>> need to specify the intended audience and the motivation for the >>> performance metrics. There will also be guidelines for a >>>performance >>> metric development process that includes entry criteria for >>> new proposals (how a proposal might be evaluated for possible >>> endorsement by a protocol development working group), and how a >>> successful proposal will be developed by PMOL WG in >>>cooperation with a >>> protocol development WG. >> >>Some of the content of this proposed WG deliverable is what we >>typically want to see already during the BOF proposal phase: >> >>Specifically, the "motivation for work", "how that work fills a >>need and any gap in IETF-chartered work", "motivation for the >>performance metrics" seem pretty relevant already when asking for a >>BOF. >> >>Likewise, information on "the various types of metrics that may be >>prepared in this work" seem important to define the scope for the >>planned WG. "Entry criteria for new proposals" is also something >>that should be in the proposed charter already. >> >>Lars > > > >_______________________________________________ >PMOL mailing list >PMOL@ietf.org >https://www1.ietf.org/mailman/listinfo/pmol _______________________________________________ PMOL mailing list PMOL@ietf.org https://www1.ietf.org/mailman/listinfo/pmol Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1IhU6G-0000h3-6Y; Mon, 15 Oct 2007 13:51:32 -0400 Received: from pmol by megatron.ietf.org with local (Exim 4.43) id 1IhU6E-0000cm-4A for pmol-confirm+ok@megatron.ietf.org; Mon, 15 Oct 2007 13:51:30 -0400 Received: from [10.90.34.44] (helo=chiedprmail1.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1IhU6D-0000a8-9P for pmol@ietf.org; Mon, 15 Oct 2007 13:51:29 -0400 Received: from smtp.nokia.com ([131.228.20.173] helo=mgw-ext14.nokia.com) by chiedprmail1.ietf.org with esmtp (Exim 4.43) id 1IhU6C-00088u-Ha for pmol@ietf.org; Mon, 15 Oct 2007 13:51:29 -0400 Received: from esebh108.NOE.Nokia.com (esebh108.ntc.nokia.com [172.21.143.145]) by mgw-ext14.nokia.com (Switch-3.2.5/Switch-3.2.5) with ESMTP id l9FHpJjE018198; Mon, 15 Oct 2007 20:51:26 +0300 Received: from esebh103.NOE.Nokia.com ([172.21.143.33]) by esebh108.NOE.Nokia.com with Microsoft SMTPSVC(6.0.3790.1830); Mon, 15 Oct 2007 20:51:23 +0300 Received: from mgw-int02.ntc.nokia.com ([172.21.143.97]) by esebh103.NOE.Nokia.com over TLS secured channel with Microsoft SMTPSVC(6.0.3790.1830); Mon, 15 Oct 2007 20:51:21 +0300 Received: from [192.35.165.6] (daec-linuxvpn059137.americas.nokia.com [10.241.59.137]) by mgw-int02.ntc.nokia.com (Switch-3.2.5/Switch-3.2.5) with ESMTP id l9FHpEwe007918; Mon, 15 Oct 2007 20:51:19 +0300 Mime-Version: 1.0 (Apple Message framework v752.3) In-Reply-To: References: Message-Id: <2F377D6F-C725-4F63-A01D-33386F8256A2@nokia.com> From: Lars Eggert Subject: Re: [PMOL] Suggested updates to proposed charter wording Date: Mon, 15 Oct 2007 11:51:12 -0600 To: ext Alan Clark , pmol@ietf.org X-Mailer: Apple Mail (2.752.3) X-OriginalArrivalTime: 15 Oct 2007 17:51:21.0720 (UTC) FILETIME=[FF224F80:01C80F53] X-Nokia-AV: Clean X-Spam-Score: 0.0 (/) X-Scan-Signature: 5011df3e2a27abcc044eaa15befcaa87 Cc: X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============1545476074==" Errors-To: pmol-bounces@ietf.org --===============1545476074== Content-Type: multipart/signed; micalg=sha1; boundary=Apple-Mail-5--584892479; protocol="application/pkcs7-signature" --Apple-Mail-5--584892479 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Nothing in the new charter addresses my comment on the previous version: On 2007-9-26, at 3:09, ext Lars Eggert wrote: > On 2007-9-25, at 21:22, ext Alan Clark wrote: >> 1. A PMOL framework and guidelines memo that outlines the motivation >> for work to define performance metrics for applications >> transported >> over IETF-specified protocols, how that work fills a need and >> any gap >> in IETF-chartered work. The framework will describe the necessary >> elements of performance metric drafts and the various types of >> metrics >> that may be prepared in this work. The framework will also >> address the >> need to specify the intended audience and the motivation for the >> performance metrics. There will also be guidelines for a >> performance >> metric development process that includes entry criteria for >> new proposals (how a proposal might be evaluated for possible >> endorsement by a protocol development working group), and how a >> successful proposal will be developed by PMOL WG in >> cooperation with a >> protocol development WG. > > Some of the content of this proposed WG deliverable is what we > typically want to see already during the BOF proposal phase: > > Specifically, the "motivation for work", "how that work fills a > need and any gap in IETF-chartered work", "motivation for the > performance metrics" seem pretty relevant already when asking for a > BOF. > > Likewise, information on "the various types of metrics that may be > prepared in this work" seem important to define the scope for the > planned WG. "Entry criteria for new proposals" is also something > that should be in the proposed charter already. > > Lars --Apple-Mail-5--584892479 Content-Transfer-Encoding: base64 Content-Type: application/pkcs7-signature; name=smime.p7s Content-Disposition: attachment; filename=smime.p7s MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIGQDCCAvkw ggJioAMCAQICEGtxkLFHQu1g67hlmYfwPuIwDQYJKoZIhvcNAQEFBQAwYjELMAkGA1UEBhMCWkEx JTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQ ZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBMB4XDTA3MDEwNDE2MTE1NFoXDTA4MDEwNDE2MTE1 NFowXDEPMA0GA1UEBBMGRWdnZXJ0MQ0wCwYDVQQqEwRMYXJzMRQwEgYDVQQDEwtMYXJzIEVnZ2Vy dDEkMCIGCSqGSIb3DQEJARYVbGFycy5lZ2dlcnRAbm9raWEuY29tMIIBIjANBgkqhkiG9w0BAQEF AAOCAQ8AMIIBCgKCAQEA26hjntVPZMiwh4d8tyuk9KiucvG92BVUGArk5zO1jhmq7tLFgU1mqcIg VGumfQqjkAzIuh83kxIiqBrB05Wvxp85apwt4sUCvzMe8mQWzZZZYp0rBzwpOj+VC5pxsMRYtYW+ BaTFBEmvFr7D7C7roZUk0lDppS5SZFs4SQbEe9ykB9aeUf1iTiw2+ikyP2+JEYto14WYCoxFWbms FbQ1uaaK+yn1WH2YHhyMfi0IOxuT0jtbsngjHJMpWIqq334zBhj+rPSUug47YBHr41FCDMHbbbjv WbyTyDYneOW3WY8LY8bGWVlAknQGB7lgduShe6e0RI/7SL/VE6X27lM9LwIDAQABozIwMDAgBgNV HREEGTAXgRVsYXJzLmVnZ2VydEBub2tpYS5jb20wDAYDVR0TAQH/BAIwADANBgkqhkiG9w0BAQUF AAOBgQCx3nxxTg/WowobVTRXBiXUEEZj4LBiunWbuAnR0UbJIgijvq3JbiJvZo2gUGiW9LPaHSBz 4oxT1VR/S/6O0a4e87oV5cOQm6ReB68o9rDfUzxHTwoCfv2wwMwBrXyzQMjyQK4Lt8siV41gmZpm rSXMhjTJdd9uYYNoZKQLq+VM+TCCAz8wggKooAMCAQICAQ0wDQYJKoZIhvcNAQEFBQAwgdExCzAJ BgNVBAYTAlpBMRUwEwYDVQQIEwxXZXN0ZXJuIENhcGUxEjAQBgNVBAcTCUNhcGUgVG93bjEaMBgG A1UEChMRVGhhd3RlIENvbnN1bHRpbmcxKDAmBgNVBAsTH0NlcnRpZmljYXRpb24gU2VydmljZXMg RGl2aXNpb24xJDAiBgNVBAMTG1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBDQTErMCkGCSqGSIb3 DQEJARYccGVyc29uYWwtZnJlZW1haWxAdGhhd3RlLmNvbTAeFw0wMzA3MTcwMDAwMDBaFw0xMzA3 MTYyMzU5NTlaMGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29uc3VsdGluZyAoUHR5 KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNzdWluZyBDQTCBnzAN BgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEAxKY8VXNV+065yplaHmjAdQRwnd/p/6Me7L3N9VvyGna9 fww6YfK/Uc4B1OVQCjDXAmNaLIkVcI7dyfArhVqqP3FWy688Cwfn8R+RNiQqE88r1fOCdz0Dviv+ uxg+B79AgAJk16emu59l0cUqVIUPSAR/p7bRPGEEQB5kGXJgt/sCAwEAAaOBlDCBkTASBgNVHRMB Af8ECDAGAQH/AgEAMEMGA1UdHwQ8MDowOKA2oDSGMmh0dHA6Ly9jcmwudGhhd3RlLmNvbS9UaGF3 dGVQZXJzb25hbEZyZWVtYWlsQ0EuY3JsMAsGA1UdDwQEAwIBBjApBgNVHREEIjAgpB4wHDEaMBgG A1UEAxMRUHJpdmF0ZUxhYmVsMi0xMzgwDQYJKoZIhvcNAQEFBQADgYEASIzRUIPqCy7MDaNmrGcP f6+svsIXoUOWlJ1/TCG4+DYfqi2fNi/A9BxQIJNwPP2t4WFiw9k6GX6EsZkbAMUaC4J0niVQlGLH 2ydxVyWN3amcOY6MIE9lX5Xa9/eH1sYITq726jTlEBpbNU1341YheILcIRk13iSx0x1G/11fZU8x ggMQMIIDDAIBATB2MGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29uc3VsdGluZyAo UHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNzdWluZyBDQQIQ a3GQsUdC7WDruGWZh/A+4jAJBgUrDgMCGgUAoIIBbzAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcB MBwGCSqGSIb3DQEJBTEPFw0wNzEwMTUxNzUxMTJaMCMGCSqGSIb3DQEJBDEWBBQVyeUSNeesHBAG yBWH2JgntqYyojCBhQYJKwYBBAGCNxAEMXgwdjBiMQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhh d3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVt YWlsIElzc3VpbmcgQ0ECEGtxkLFHQu1g67hlmYfwPuIwgYcGCyqGSIb3DQEJEAILMXigdjBiMQsw CQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UE AxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0ECEGtxkLFHQu1g67hlmYfwPuIw DQYJKoZIhvcNAQEBBQAEggEAVNzaCyowrWQUxGyfwunWV04CZdKgQWLtJx1uyGsqmfaPuJybYzZb ABvVcSMphlFQcW4YP6+skvErkuUE0YZJA8POzBl8knqeYyUVJ93C3aMmbNgbOZN5+hygbeOWEUUX iwQFEskBMIR7cKpHDUmN4/U+OZjUcsL3hEpTuXjmgbtbYgJizIe1AhgtkkN/VWnC2ykGerk+X+h6 JLDBA6k8QCe3eHf4PiWfVQ9DjXCsSFtMLao9xTkrDkdGlHR4cJNuO8dhmf6pH1lo/kYnxfa84AIF 7bvO5tU6ErPo6xg1RlsloQXuB/UR4m+PVoLUmw4gkoyfuMFKPgIkhMPAb+1EFQAAAAAAAA== --Apple-Mail-5--584892479-- --===============1545476074== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ PMOL mailing list PMOL@ietf.org https://www1.ietf.org/mailman/listinfo/pmol --===============1545476074==-- Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1IhPIE-0000wk-A4; Mon, 15 Oct 2007 08:43:34 -0400 Received: from pmol by megatron.ietf.org with local (Exim 4.43) id 1IhPIC-0000t0-On for pmol-confirm+ok@megatron.ietf.org; Mon, 15 Oct 2007 08:43:32 -0400 Received: from [10.90.34.44] (helo=chiedprmail1.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1IhPIB-0000rt-RW for pmol@ietf.org; Mon, 15 Oct 2007 08:43:31 -0400 Received: from de307622-de-outbound.net.avaya.com ([198.152.71.100]) by chiedprmail1.ietf.org with esmtp (Exim 4.43) id 1IhPIB-0001eI-2E for pmol@ietf.org; Mon, 15 Oct 2007 08:43:31 -0400 X-IronPort-AV: E=Sophos;i="4.21,277,1188792000"; d="scan'208";a="65894564" Received: from unknown (HELO 307622ANEX5.global.avaya.com) ([135.64.140.14]) by de307622-de-outbound.net.avaya.com with ESMTP; 15 Oct 2007 08:43:27 -0400 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Subject: RE: [PMOL] Revised Charter Date: Mon, 15 Oct 2007 14:42:47 +0200 Message-ID: In-Reply-To: <200710151233.l9FCXkVA025804@flph023.ffdc.sbc.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [PMOL] Revised Charter Thread-Index: AcgPKCdRgU0/hWpoRuqvj5KGZALKTQAAJQuA References: <200710151233.l9FCXkVA025804@flph023.ffdc.sbc.com> From: "Romascanu, Dan (Dan)" To: "Al Morton" , X-Spam-Score: 0.0 (/) X-Scan-Signature: ff03b0075c3fc728d7d60a15b4ee1ad2 Cc: X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: pmol-bounces@ietf.org I would add to the milestones that the framework document will be considered for BCP and the SIP Performance metrics document as standards-track.=20 Dan =20 =20 > -----Original Message----- > From: Al Morton [mailto:acmorton@att.com]=20 > Sent: Monday, October 15, 2007 2:32 PM > To: pmol@ietf.org > Subject: [PMOL] Revised Charter >=20 > Folks, > here's the latest version of the charter text, addressing=20 > most, if not all, comments to date. > Al >=20 > Proposed Charter (0.3) >=20 >=20 > Performance Metrics at Other Layers WG (PMOL) >=20 > The successful implementation and operation of IP based=20 > applications often depends on some underlying performance=20 > measurement infrastructure that helps service operators or=20 > network managers to recognize when performance is=20 > unsatisfactory and identify problems affecting service=20 > quality. Standardized performance metrics add the desirable=20 > features of consistent implementation, interpretation, nd comparison. >=20 > The IETF has two Working Groups dedicated to the development=20 > of performance metrics however each has strict limitations in=20 > their charters: >=20 > - The Benchmarking Methodology WG has addressed a range of=20 > networking technologies > and protocols in their long history (such as IEEE 802.3,=20 > ATM, Frame Relay, and > Routing Protocols), but the charter strictly limits=20 > their performance > characterizations to the laboratory environment. >=20 > - The IP Performance Metrics WG has the mandate to develop=20 > metrics applicable > to the performance of Internet data delivery, but it is=20 > specifically prohibited > from developing metrics that characterize traffic (such=20 > as a VoIP stream). >=20 > The IETF also has current and completed activities related to=20 > the reporting of application performance metrics (e.g. RAQMON=20 > and RTCP XR) and is also actively involved in the development=20 > of reliable transport protocols which would affect the=20 > relationship between IP performance and application performance. >=20 > Thus there is a gap in the currently chartered coverage of IETF WGs: > development of performance metrics for IP-based applications=20 > that operate over UDP, TCP, SCTP, DCCP, Forward Error=20 > Correction (FEC) and other robust transport protocols, and=20 > that can be used to characterize traffic on live networks. >=20 > The working group will focus on the completion of two RFCs: >=20 > 1. A PMOL framework and guidelines memo that documents the motivation > for work to define performance metrics for applications=20 > transported > over IETF-specified protocols, and how that work fills a=20 > need and a gap > in IETF-chartered work (motivation and gap=20 > identification having been part > of the original BOF proposal). The framework will=20 > describe the necessary > elements of performance metric drafts (such as the=20 > formal definition, > purpose, and units of measure) and the various types of metrics > that may be prepared in this work (such as metrics=20 > derived from other > metrics, possibly on lower layers). The framework will=20 > also address the > need to specify the intended audience and the motivation for the > performance metrics. There will also be guidelines for a=20 > performance > metric development process that includes entry criteria for > new proposals (how a proposal might be evaluated for possible > endorsement by a protocol development working group), and how a > successful proposal will be developed by PMOL WG in=20 > cooperation with a > protocol development WG. >=20 > 2. A proof-of-concept RFC defining performance metrics for=20 > SIP, based on > draft-malas-performance-metrics. This memo would serve=20 > as an example of > the framework and the PMOL development process in the IETF. >=20 > Discussion of new work proposals is strongly discouraged=20 > under the initial charter of the PMOL WG, except to advise a=20 > protocol development WG when they are evaluating a new work=20 > proposal for related performance metrics. >=20 > The PMOL WG will also be guided by a document describing how=20 > memos defining performance metrics are intended to advance=20 > along the IETF Standards track (draft-bradner-metricstest). >=20 > PMOL WG will take advantage of expertise and seek to avoid=20 > overlap with other standards development organizations, such=20 > as ETSI STQ, ITU-T SG 12, ATIS IIF, ATIS PRQC, and others. >=20 > Milestones > June 08 SIP Performance Metrics Draft to IESG Review Sept 08=20 > PMOL Framework and Guidelines Draft to IESG Review >=20 >=20 >=20 > _______________________________________________ > PMOL mailing list > PMOL@ietf.org > https://www1.ietf.org/mailman/listinfo/pmol >=20 _______________________________________________ PMOL mailing list PMOL@ietf.org https://www1.ietf.org/mailman/listinfo/pmol Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1IhP9m-0005TL-ED; Mon, 15 Oct 2007 08:34:50 -0400 Received: from pmol by megatron.ietf.org with local (Exim 4.43) id 1IhP9k-0005Ri-5R for pmol-confirm+ok@megatron.ietf.org; Mon, 15 Oct 2007 08:34:48 -0400 Received: from [10.91.34.44] (helo=ietf-mx.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1IhP9X-0005NF-7P for pmol@ietf.org; Mon, 15 Oct 2007 08:34:35 -0400 Received: from mail146.messagelabs.com ([216.82.245.131]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1IhP9M-000786-1v for pmol@ietf.org; Mon, 15 Oct 2007 08:34:29 -0400 X-VirusChecked: Checked X-Env-Sender: acmorton@att.com X-Msg-Ref: server-9.tower-146.messagelabs.com!1192451642!6990499!1 X-StarScan-Version: 5.5.12.14.2; banners=-,-,- X-Originating-IP: [144.160.128.149] Received: (qmail 10406 invoked from network); 15 Oct 2007 12:34:03 -0000 Received: from sbcsmtp9.sbc.com (HELO flph024.enaf.ffdc.sbc.com) (144.160.128.149) by server-9.tower-146.messagelabs.com with AES256-SHA encrypted SMTP; 15 Oct 2007 12:34:03 -0000 Received: from enaf.ffdc.sbc.com (localhost.localdomain [127.0.0.1]) by flph024.enaf.ffdc.sbc.com (8.14.0/8.14.0) with ESMTP id l9FCY1qB030427 for ; Mon, 15 Oct 2007 05:34:02 -0700 Received: from flph023.ffdc.sbc.com (flph023.ffdc.sbc.com [150.234.117.36]) by flph024.enaf.ffdc.sbc.com (8.14.0/8.14.0) with ESMTP id l9FCXtRF030364 for ; Mon, 15 Oct 2007 05:34:01 -0700 Received: from ffdc.sbc.com (localhost.localdomain [127.0.0.1]) by flph023.ffdc.sbc.com (8.14.0/8.14.0) with ESMTP id l9FCXsDW025876 for ; Mon, 15 Oct 2007 05:33:55 -0700 Received: from maillennium.att.com (dns.maillennium.att.com [135.25.114.99]) by flph023.ffdc.sbc.com (8.14.0/8.14.0) with ESMTP id l9FCXkVA025804 for ; Mon, 15 Oct 2007 05:33:52 -0700 Message-Id: <200710151233.l9FCXkVA025804@flph023.ffdc.sbc.com> Received: from acmt.att.com (ehuda01.mt.att.com[135.16.251.73](misconfigured sender)) by maillennium.att.com (mailgw1) with SMTP id <20071015123346gw10010gdce>; Mon, 15 Oct 2007 12:33:46 +0000 X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Mon, 15 Oct 2007 08:32:26 -0400 To: pmol@ietf.org From: Al Morton Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Spam-Score: -2.5 (--) X-Scan-Signature: f66b12316365a3fe519e75911daf28a8 Subject: [PMOL] Revised Charter X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: pmol-bounces@ietf.org Folks, here's the latest version of the charter text, addressing most, if not all, comments to date. Al Proposed Charter (0.3) Performance Metrics at Other Layers WG (PMOL) The successful implementation and operation of IP based applications often depends on some underlying performance measurement infrastructure that helps service operators or network managers to recognize when performance is unsatisfactory and identify problems affecting service quality. Standardized performance metrics add the desirable features of consistent implementation, interpretation, nd comparison. The IETF has two Working Groups dedicated to the development of performance metrics however each has strict limitations in their charters: - The Benchmarking Methodology WG has addressed a range of networking technologies and protocols in their long history (such as IEEE 802.3, ATM, Frame Relay, and Routing Protocols), but the charter strictly limits their performance characterizations to the laboratory environment. - The IP Performance Metrics WG has the mandate to develop metrics applicable to the performance of Internet data delivery, but it is specifically prohibited from developing metrics that characterize traffic (such as a VoIP stream). The IETF also has current and completed activities related to the reporting of application performance metrics (e.g. RAQMON and RTCP XR) and is also actively involved in the development of reliable transport protocols which would affect the relationship between IP performance and application performance. Thus there is a gap in the currently chartered coverage of IETF WGs: development of performance metrics for IP-based applications that operate over UDP, TCP, SCTP, DCCP, Forward Error Correction (FEC) and other robust transport protocols, and that can be used to characterize traffic on live networks. The working group will focus on the completion of two RFCs: 1. A PMOL framework and guidelines memo that documents the motivation for work to define performance metrics for applications transported over IETF-specified protocols, and how that work fills a need and a gap in IETF-chartered work (motivation and gap identification having been part of the original BOF proposal). The framework will describe the necessary elements of performance metric drafts (such as the formal definition, purpose, and units of measure) and the various types of metrics that may be prepared in this work (such as metrics derived from other metrics, possibly on lower layers). The framework will also address the need to specify the intended audience and the motivation for the performance metrics. There will also be guidelines for a performance metric development process that includes entry criteria for new proposals (how a proposal might be evaluated for possible endorsement by a protocol development working group), and how a successful proposal will be developed by PMOL WG in cooperation with a protocol development WG. 2. A proof-of-concept RFC defining performance metrics for SIP, based on draft-malas-performance-metrics. This memo would serve as an example of the framework and the PMOL development process in the IETF. Discussion of new work proposals is strongly discouraged under the initial charter of the PMOL WG, except to advise a protocol development WG when they are evaluating a new work proposal for related performance metrics. The PMOL WG will also be guided by a document describing how memos defining performance metrics are intended to advance along the IETF Standards track (draft-bradner-metricstest). PMOL WG will take advantage of expertise and seek to avoid overlap with other standards development organizations, such as ETSI STQ, ITU-T SG 12, ATIS IIF, ATIS PRQC, and others. Milestones June 08 SIP Performance Metrics Draft to IESG Review Sept 08 PMOL Framework and Guidelines Draft to IESG Review _______________________________________________ PMOL mailing list PMOL@ietf.org https://www1.ietf.org/mailman/listinfo/pmol Return-path: Received: from [127.0.0.1] (helo=stiedprmman1.va.neustar.com) by megatron.ietf.org with esmtp (Exim 4.43) id 1Ih1wt-0006Gq-9E; Sun, 14 Oct 2007 07:47:59 -0400 Received: from pmol by megatron.ietf.org with local (Exim 4.43) id 1Ih1wr-0006GV-Uj for pmol-confirm+ok@megatron.ietf.org; Sun, 14 Oct 2007 07:47:57 -0400 Received: from [10.90.34.44] (helo=chiedprmail1.ietf.org) by megatron.ietf.org with esmtp (Exim 4.43) id 1Ih1wm-00066y-By; Sun, 14 Oct 2007 07:47:52 -0400 Received: from de307622-de-outbound.net.avaya.com ([198.152.71.100]) by chiedprmail1.ietf.org with esmtp (Exim 4.43) id 1Ih1wl-0007q5-VK; Sun, 14 Oct 2007 07:47:52 -0400 X-IronPort-AV: E=Sophos;i="4.21,273,1188792000"; d="scan'208";a="65676287" Received: from unknown (HELO 307622ANEX5.global.avaya.com) ([135.64.140.14]) by de307622-de-outbound.net.avaya.com with ESMTP; 14 Oct 2007 07:47:48 -0400 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Sun, 14 Oct 2007 13:47:07 +0200 Message-ID: X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: REVISED Approval to Post Version -00 Internet-Drafts for the 70th IETF Meeting in Vancouver, BC, Canada Thread-Index: AcgM+j5zh8Ax/7b7SyK5IpOLNKh4ewBXZa8Q From: "Romascanu, Dan (Dan)" To: , , X-Spam-Score: 0.0 (/) X-Scan-Signature: e1e48a527f609d1be2bc8d8a70eb76cb Cc: Subject: [PMOL] FW: REVISED Approval to Post Version -00 Internet-Drafts for the 70th IETF Meeting in Vancouver, BC, Canada X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: pmol-bounces@ietf.org =20 =20 -----Original Message----- From: Internet-Drafts Administrator [mailto:internet-drafts@ietf.org]=20 Sent: Friday, October 12, 2007 8:01 PM To: Working Group Chairs Cc: iesg@ietf.org Subject: REVISED Approval to Post Version -00 Internet-Drafts for the 70th IETF Meeting in Vancouver, BC, Canada=20 Dear IETF Working Group Chairs: In order to expedite the processing of the many version -00 I-Ds that the Secretariat receives before an IETF meeting, we ask that you please notify the Secretariat prior to the initial submission cutoff date of all version -00 I-Ds that you expect to approve for posting as WG documents.=20 Please indicate if a new version -00 I-D replaces any existing I-Ds, and if so, which ones. Please send the filenames of your approved -00 I-Ds to internet-drafts@ietf.org, or use the "Initial Version WG Draft Approval Tool" https://datatracker.ietf.org/cgi-bin/wg/wg_init_rev_approval.cgi to pre-approve drafts. We would appreciate receiving your list of approved drafts five (5) business days prior to the cutoff date for -00 submissions, or by Monday, November 5th at 9:00 AM ET (14:00 UTC/GMT) for the 70th IETF Meeting. If you send your list by e-mail, then please include the word "Approved I-Ds" in the "Subject" field. This procedure will help to ensure that version -00 I-Ds are posted in a timely manner, allowing more time for review by the public. Thank you for your cooperation in this matter. The Internet-Drafts Administrator FYI: The Internet-Draft cutoff dates as well as other significant dates for the 70th IETF Meeting can be found at http://www3.ietf.org/meetings/70-cutoff_dates.html. _______________________________________________ PMOL mailing list PMOL@ietf.org https://www1.ietf.org/mailman/listinfo/pmol