Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id D04053A6ACE for ; Mon, 26 Oct 2009 09:07:08 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.66 X-Spam-Level: X-Spam-Status: No, score=-1.66 tagged_above=-999 required=5 tests=[AWL=-0.669, BAYES_00=-2.599, SARE_LWSHORTT=1.24, URI_HEX=0.368] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZOpXIZ3JY8xF for ; Mon, 26 Oct 2009 09:07:07 -0700 (PDT) Received: from av-tac-bru.cisco.com (weird-brew.cisco.com [144.254.15.118]) by core3.amsl.com (Postfix) with ESMTP id 0764A28C287 for ; Mon, 26 Oct 2009 09:07:06 -0700 (PDT) X-TACSUNS: Virus Scanned Received: from strange-brew.cisco.com (localhost.cisco.com [127.0.0.1]) by av-tac-bru.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id n9QEHZ8O021572; Mon, 26 Oct 2009 15:17:35 +0100 (CET) Received: from [10.55.43.53] (ams-bclaise-8714.cisco.com [10.55.43.53]) by strange-brew.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id n9QEHYmM009688; Mon, 26 Oct 2009 15:17:35 +0100 (CET) Message-ID: <4AE5AF7E.8040802@cisco.com> Date: Mon, 26 Oct 2009 15:17:34 +0100 From: Benoit Claise User-Agent: Thunderbird 2.0.0.23 (Windows/20090812) MIME-Version: 1.0 To: Al Morton , Loki Jorgenson References: <200812120631.mBC6VXam018361@alph001.aldc.att.com> In-Reply-To: <200812120631.mBC6VXam018361@alph001.aldc.att.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: pmol@ietf.org Subject: Re: [PMOL] Fwd: RE: WGLC on pmol-metrics-framework-01 X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Oct 2009 16:07:08 -0000 Loki, Thanks for your feedback. I have added the statement that this framework is valid for both active and passive measurements. All the editorial comments have been taken into account. Regards, Benoit. > Forwarded at Loki's request... > >> X-VirusChecked: Checked >> X-Env-Sender: ljorgenson@apparentnetworks.com >> X-Msg-Ref: server-13.tower-121.messagelabs.com!1228755277!24935304!1 >> X-StarScan-Version: 5.5.12.14.2; banners=-,-,- >> X-Originating-IP: [209.139.228.52] >> X-SpamReason: No, hits=0.5 required=7.0 tests=BODY_RANDOM_LONG >> X-AuditID: ac108108-00000e4c00000774-d5-493d514d1d00 >> Subject: RE: [PMOL] WGLC on pmol-metrics-framework-01 >> Date: Mon, 8 Dec 2008 08:54:35 -0800 >> X-MS-Has-Attach: >> X-MS-TNEF-Correlator: >> thread-topic: [PMOL] WGLC on pmol-metrics-framework-01 >> thread-index: AclUuJeWbGY6AyloRw+sWjN4K8ltvABneGJgAL/HSyA= >> From: "Loki Jorgenson" >> To: "Al Morton" >> X-Brightmail-Tracker: AAAAAA== >> >> Al - any reason why my post to PMOL did not make it onto the list? >> >> Loki Jorgenson >> Apparent Networks >> t 604 433 2333 ext 105 >> m 604 250-4642 >> >> -----Original Message----- >> From: Loki Jorgenson >> Sent: Thursday, December 04, 2008 16:49 >> To: pmol@ietf.org >> Cc: Alan Clark >> Subject: RE: [PMOL] WGLC on pmol-metrics-framework-01 >> >> Pardon my late comments as I have been unavailable for PMOL >> participation as of late. >> >> One request for clarification as well as some tidying of prose. >> >> 1) Is there any value in defining the relationship of this framework to >> active or passive measurement methodologies? >> >> Originally I had thought not. It appeared that this document was >> intended to be high-level and agnostic to active or passive >> methodologies, and therefore said nothing differentiate between the two. >> >> >> However, there is a reference in Section 6 (Security Considerations) >> that states that "security considerations that apply to any active >> measurements of live networks are relevant here" with a pointer to >> IPPM-developed RFC4656, OWAMP. IPPM thus far limits itself to active >> methodologies. This very mildly suggests that this framework is >> specific to active techniques. >> >> In Purpose and Scope, the reference is "for... metrics... that can be >> used to characterize traffic on live networks and services" - which >> sounds distinctly passive, insofar as "traffic" suggests applications >> and not active probing. >> >> And, various cross-references to other IPPM drafts and RFCs have been >> directly incorporated (e.g. compagg in at least Section 3.3). However, >> examples to the Telchemy burst-loss algorithm and RFC 4546 may be >> applied to either passive or active. >> >> Further, if passive is included, then security considerations specific >> to passive should apply in Section 6 as well (e.g. anonymization). I >> don't have a handy sample reference for that. >> >> Not to make more of it than necessary, but maybe it is worth addressing >> explicitly. Perhaps a sentence or two within the Purpose and Scope >> section declaring its applicability with regard to common measurement >> methodologies. >> >> 2) Awkward sentence: >> >> Section 3.4.2 (v) >> >> "Short intervals or frequent sampling provides a richer source of >> information that can be helpful in assessing application performance >> however can lead to excessive measurement data." >> >> It is comprehensible but somewhat awkward. Suggest a couple of changes: >> >> "A SHORT SAMPLING INTERVAL or frequent sampling provides a richer source >> of information that MAY be helpful in assessing application performance >> BUT MAY ALSO lead to excessive measurement data." >> >> (Capitals indicate changes, not use of capitals in the document - e.g. >> MAY here is not the usual IETF directive convention) >> >> 3) Ambiguous sentence: >> >> Section 3.4.2 (v) >> >> "Long measurement or sampling intervals reduce that amount of reported >> and collected data however may be insufficient to truly understand >> application performance or service quality if this varies with time." >> >> Semantically challenged although more or less readable. Suggest: >> >> "Long measurement or sampling intervals reduce THE amount of reported >> and collected data SUCH THAT IT may be insufficient to ___ understand >> application performance or service quality INSOFAR AS THE MEASURED >> QUANTITY MAY vary SIGNIFICANTLY with time." >> >> 4) Awkward sentence: >> >> Section 3.4.3 (iv) >> >> "For example, if the metric is a short term running average packet delay >> variation (e.g. PPDV as defined in RFC3550) however this value is >> reported at intervals of 6-10 seconds this results in a sampling model >> which may have limited accuracy if packet delay variation is >> non-stationary." >> >> A very long sentence with missing or awkward conjunction. Suggest: >> >> "For example, if the metric is a short term running average packet delay >> variation (e.g. PPDV as defined in RFC3550) AND (YET) this value is >> reported at intervals of 6-10 seconds, THE RESULTING MEASUREMENT may >> have limited accuracy WHEN packet delay variation is non-stationary." >> >> 5) Typographic error: >> >> Section 3.4.5 (Measurement Timing:) >> >> "... due to time-of-day load network load changes." >> >> Suggest: >> >> "... due to time-of-day changes in network load." >> >> >> 6) Inconsistent notation: >> >> Section 3.5.1 >> >> "... a 10 percent variation ..." (two instances) >> >> Could using number word and "percent" for all instances or numeric value >> and '%'. Suggest: >> >> "... a 10% variation ..." (two instances) >> >> >> End of Comments >> >> ------------------------------------------------------------------------ >> ------------- >> >> Message: 1 >> Date: Tue, 02 Dec 2008 10:11:00 -0500 >> From: Al Morton >> Subject: [PMOL] WGLC on pmol-metrics-framework-01 >> To: pmol@ietf.org >> Cc: bmwg@ietf.org, ippm@ietf.org >> >> PMOL WG, >> cc IPPM and BMWG WGs, >> >> This message begins a WG Last Call on the following draft: >> >> Title : Framework for Performance Metric Development >> Author(s) : A. Clark >> Filename : draft-ietf-pmol-metrics-framework-01.txt >> Pages : 14 >> Date : 2008-11-02 >> >> > > _______________________________________________ > PMOL mailing list > PMOL@ietf.org > https://www.ietf.org/mailman/listinfo/pmol Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id D19B228C29E for ; Mon, 26 Oct 2009 09:07:07 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.451 X-Spam-Level: X-Spam-Status: No, score=-2.451 tagged_above=-999 required=5 tests=[AWL=0.147, BAYES_00=-2.599, HTML_MESSAGE=0.001] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BVZKFkwUbb1q for ; Mon, 26 Oct 2009 09:07:06 -0700 (PDT) Received: from av-tac-bru.cisco.com (weird-brew.cisco.com [144.254.15.118]) by core3.amsl.com (Postfix) with ESMTP id 2ACB43A6ACE for ; Mon, 26 Oct 2009 09:07:06 -0700 (PDT) X-TACSUNS: Virus Scanned Received: from strange-brew.cisco.com (localhost.cisco.com [127.0.0.1]) by av-tac-bru.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id n9QEFBTQ021039; Mon, 26 Oct 2009 15:15:11 +0100 (CET) Received: from [10.55.43.53] (ams-bclaise-8714.cisco.com [10.55.43.53]) by strange-brew.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id n9QEFA9o004831; Mon, 26 Oct 2009 15:15:11 +0100 (CET) Message-ID: <4AE5AEEE.4010903@cisco.com> Date: Mon, 26 Oct 2009 15:15:10 +0100 From: Benoit Claise User-Agent: Thunderbird 2.0.0.23 (Windows/20090812) MIME-Version: 1.0 To: "Christou, Christos [USA]" References: <46E7B21EF4A45749B2432ABDCDE6FEA103E8EAF5@MCLNEXVS12.resource.ds.bah.com> In-Reply-To: <46E7B21EF4A45749B2432ABDCDE6FEA103E8EAF5@MCLNEXVS12.resource.ds.bah.com> Content-Type: multipart/alternative; boundary="------------000503030304070300050503" Cc: pmol@ietf.org Subject: Re: [PMOL] application performance metrics X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Oct 2009 16:07:07 -0000 This is a multi-part message in MIME format. --------------000503030304070300050503 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Hi Chris, In the new framework draft version I just posted, I inserted definitions for QoE, QoS, and I have a place holder for application performance metric, for which I'm after a generic definition. See if this works for you. Regards, Benoit. > Hello, > > Having reviewed the work of this working group and listening to the > presentation at the Ops Area open meeting in Minneapolis, I would like > to understand from WG participants whether there is interest in > expanding the work of this WG to define performance metrics for > different application types? > > In discussing this with Al and others, I wanted to see if participants > think this is needed, if it is too difficult, out of scope for the > IETF, if the appropriate metrics have already been defined, etc. For > example, ITU-T SG 12 has definied G.1080 and G.1081 to define > different "QoE" metrics--is this sufficient? > > It seems like there has been much work over the years on how to > measure MOS. However, what is the parameter that is used for other > applications? If this has previously been defined, please do include > a pointer... > > Thanks--just wanted to gauge the list's thoughts on this, > > Chris > ------------------------------------------------------------------------ > > _______________________________________________ > PMOL mailing list > PMOL@ietf.org > https://www.ietf.org/mailman/listinfo/pmol > --------------000503030304070300050503 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Hi Chris,

In the new framework draft version I just posted, I inserted definitions for QoE, QoS, and I have a place holder for application performance metric, for which I'm after a generic definition.
See if this works for you.

Regards, Benoit.
Hello,
 
Having reviewed the work of this working group and listening to the presentation at the Ops Area open meeting in Minneapolis, I would like to understand from WG participants whether there is interest in expanding the work of this WG to define performance metrics for different application types? 
 
In discussing this with Al and others, I wanted to see if participants think this is needed, if it is too difficult, out of scope for the IETF, if the appropriate metrics have already been defined, etc.  For example, ITU-T SG 12 has definied G.1080 and G.1081 to define different "QoE" metrics--is this sufficient? 
 
It seems like there has been much work over the years on how to measure MOS.  However, what is the parameter that is used for other applications?  If this has previously been defined, please do include a pointer...
 
Thanks--just wanted to gauge the list's thoughts on this,
 
Chris

_______________________________________________
PMOL mailing list
PMOL@ietf.org
https://www.ietf.org/mailman/listinfo/pmol

--------------000503030304070300050503-- Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 22B1E28C2AC for ; Mon, 26 Oct 2009 07:21:06 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.414 X-Spam-Level: X-Spam-Status: No, score=-2.414 tagged_above=-999 required=5 tests=[AWL=0.185, BAYES_00=-2.599] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id T1G8xU0l2WCL for ; Mon, 26 Oct 2009 07:21:05 -0700 (PDT) Received: from av-tac-bru.cisco.com (weird-brew.cisco.com [144.254.15.118]) by core3.amsl.com (Postfix) with ESMTP id EFE9728C242 for ; Mon, 26 Oct 2009 07:21:04 -0700 (PDT) X-TACSUNS: Virus Scanned Received: from strange-brew.cisco.com (localhost.cisco.com [127.0.0.1]) by av-tac-bru.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id n9QEKjjY022143; Mon, 26 Oct 2009 15:20:45 +0100 (CET) Received: from [10.55.43.53] (ams-bclaise-8714.cisco.com [10.55.43.53]) by strange-brew.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id n9QEKimx015021; Mon, 26 Oct 2009 15:20:45 +0100 (CET) Message-ID: <4AE5B03C.2060704@cisco.com> Date: Mon, 26 Oct 2009 15:20:44 +0100 From: Benoit Claise User-Agent: Thunderbird 2.0.0.23 (Windows/20090812) MIME-Version: 1.0 To: "Fardid, Reza" References: <3BC5D44AB71A754FAE3CA68A7597EABE0F83C2D6@ZANEVS03.cc-ntd1.covad.com> In-Reply-To: <3BC5D44AB71A754FAE3CA68A7597EABE0F83C2D6@ZANEVS03.cc-ntd1.covad.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: pmol@ietf.org Subject: Re: [PMOL] [ippm] WGLC on pmol-metrics-framework-01 X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Oct 2009 14:21:06 -0000 Hi Fardid, Thanks for your feedback. > Hi, > > I have a number of comments on the PMOL framework draft: > > 1.1 Since RAQMON is mentioned, adding an Informative reference to its > framework is appropriate. > Done. > 3.3.2 Since Index is a derived metric, it is best to specify how its > components contribute to its measurement beyond a function definition. > An index composed of N orthogonal metrics all of which are measurable > has more "usefulness" than an index whose measurement is based on a > small subset of these N metrics. > Could you please provide some text for this. > 3.4.2 (vi) Does measurement point define the measurement domain? If not, > then measurement domain (intra-, inter-) is another factor to consider. > Security restrictions may prevent inter-domain measurement of certain > metrics. > Done. > 3.4.3 Are both active and passive measurement types within scope? If so, > then measurement type is another factor to consider. > I realize I forgot this one is the version I just posted. I'll write this down in my personal to do list. > 3.5 Some metrics and their measurements may be affected by middleboxes. > I can think of the following: > > 3.5.3 Middlebox presence > > Presence of a middlebox, e.g., proxy, NAT, redirect server, session > border controller (SBC), and application layer gateway (ALG) may add > variability to or restrict the scope of measurements of a metric. For > example, an SBC that does not process RTP loopback packets may block or > locally terminate this traffic rather then pass it through to its > target. > Inserted. Regards, Benoit. > Thanks, > Reza Fardid > > > > > > -----Original Message----- > From: ippm-bounces@ietf.org [mailto:ippm-bounces@ietf.org] On Behalf Of > Al Morton > Sent: Tuesday, December 02, 2008 7:11 AM > To: pmol@ietf.org > Cc: bmwg@ietf.org; ippm@ietf.org > Subject: [ippm] WGLC on pmol-metrics-framework-01 > > PMOL WG, > cc IPPM and BMWG WGs, > > This message begins a WG Last Call on the following draft: > > Title : Framework for Performance Metric Development > Author(s) : A. Clark > Filename : draft-ietf-pmol-metrics-framework-01.txt > Pages : 14 > Date : 2008-11-02 > > > This memo describes a framework and guidelines for the development of > performance metrics that are beyond the scope of existing working > group charters in the IETF. In this version, the memo refers to a > Performance Metrics Entity, or PM Entity, which may in future be a > working group or directorate or a combination of these two. > > A URL for this Internet-Draft is: > http://www.ietf.org/internet-drafts/draft-ietf-pmol-metrics-framework-01 > .txt > > There has been discussion of this draft for over a year, > mostly on the PMOL list and at meetings. The Author believes he > has addressed all comments to date. > > Please weigh-in on whether or not this Draft should be forwarded > to the Area Directors for publication as a Standards Track RFC. > Send your comments to the PMOL list or the co-chairs. > > The Last Call will be open till January 2, 2009. > > thanks for your review and comments, > Al Morton > co-chair, PMOL WG > > _______________________________________________ > ippm mailing list > ippm@ietf.org > https://www.ietf.org/mailman/listinfo/ippm > _______________________________________________ > PMOL mailing list > PMOL@ietf.org > https://www.ietf.org/mailman/listinfo/pmol > Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 146AC28C2B6 for ; Mon, 26 Oct 2009 07:18:29 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.388 X-Spam-Level: X-Spam-Status: No, score=-2.388 tagged_above=-999 required=5 tests=[AWL=0.211, BAYES_00=-2.599] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id S9T09Z9XH7jR for ; Mon, 26 Oct 2009 07:18:28 -0700 (PDT) Received: from av-tac-bru.cisco.com (weird-brew.cisco.com [144.254.15.118]) by core3.amsl.com (Postfix) with ESMTP id 5623A28C283 for ; Mon, 26 Oct 2009 07:18:25 -0700 (PDT) X-TACSUNS: Virus Scanned Received: from strange-brew.cisco.com (localhost.cisco.com [127.0.0.1]) by av-tac-bru.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id n9QEB4BD020208; Mon, 26 Oct 2009 15:11:05 +0100 (CET) Received: from [10.55.43.53] (ams-bclaise-8714.cisco.com [10.55.43.53]) by strange-brew.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id n9QEB4To027272; Mon, 26 Oct 2009 15:11:04 +0100 (CET) Message-ID: <4AE5ADF7.1010304@cisco.com> Date: Mon, 26 Oct 2009 15:11:03 +0100 From: Benoit Claise User-Agent: Thunderbird 2.0.0.23 (Windows/20090812) MIME-Version: 1.0 To: Al Morton References: <200802220137.m1M1bQYr009212@klph001.kcdc.att.com> In-Reply-To: <200802220137.m1M1bQYr009212@klph001.kcdc.att.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: pmol@ietf.org Subject: Re: [PMOL] Suggestions for the framework draft X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Oct 2009 14:18:29 -0000 Al, Thanks for your suggestion. All inserted. Regards, Benoit. > PMOL, > > While reviewing the SIP metrics draft, I found a few places > where the framework draft needed further development. > > I suggest some text in several sections below. I'm not > exactly sure where most of them fit in the outline, but > section 4.2 is strawman text for an empty section. > > Al > (as a participant) > > 3.x. Organization of Results > > The IPPM Framework [RFC2330] organizes the results of metrics into > three related notions: > > 1. singleton, an elementary instance, or "atomic" value > > 2. sample, a set of singletons with some common properties and some > varying properties. > > 3. statistic, a value derived from a sample through deterministic > calculation, such as the mean. > > Many metrics can use this organization for the results, with or > without the term names used by IPPM working group. Section 11 of > [RFC2330] should consulted for further details. > > 3.y. Parameters, the variables of a metric > > Metrics are completely defined when all options and input variables > have been identified and considered. These variables are sometimes > left unspecified in a metric definition, and their general name > indicates that the user must set them and report them with the > results. Such variables are called "parameters" in the IPPM metric > template. The scope of the metric, the time at which it was > conducted, the settings for timers and the thresholds for counters > are all examples of parameters. > > All memos defining performance metrics SHOULD identify ALL key > parameters for each metric. > > 3.z. Packet Measurement Details > > Another key aspect of the IPPM Framework [RFC2330] is the discussion > of "wire time". Internet hosts require physical resources to observe > packet traffic, and they can contribute to the performance observed. > Further, the observations SHOULD be related to some instant of time > (such as a specific bit of a packet), because a packet contains many > bits and exists on physical media long enough to introduce some > ambiguity. Section 10.2 of [RFC2330] should be consulted. > > 3.a. Identifying and Categorizing the Audience > > Many of the aspects of metric definition and reporting, even the > selection or determination of the essential metrics, depend on who > will use the results, and for what purpose. The question, "How will > the results be used?" usually yields important factors to consider > when developing performance metrics. > > All memos defining performance metrics SHOULD identify the primary > audience and its associated requirements. The audience can influence > both the definition of metrics and the methods of measurement. > > The key areas of variation between different metric users include: > > o Suitability of passive measurements of live traffic, or active > measurements using dedicated traffic > > o Measurement in laboratory environment, or on a network of deployed > devices > > o Accuracy of the results > > o Access to measurement points and configuration information > > o Measurement topology (point-to-point, point-to-multipoint) > > o Scale of the measurement system > > o Measurements conducted on-demand, or continuously > > o Required reporting formats > > > > > 4.2. Proposal Approval > > The process depends on the form that the PM Entity ultimately takes, > Directorate or working group. > > In all cases, the proposal will need to achieve consensus, in the > corresponding protocol development working group (or alternatively, > an "Area" working group with broad charter), that there is interest > and a need for the work. > > IF the PM Entity is a Directorate, > > THEN Approval SHOULD include the following steps > > o consultation with the PM Directorate, using this framework memo > > o consultation with Area Director(s) > > o and possibly IESG approval of a new or revised charter for the > working group > > IF the PM Entity is a Working Group, and the protocol development > working group decides to take up the work under its charter, > > THEN the approval is the same as the PM Directorate steps above, with > the possible additional assignment of a PM Advisor for the work item. > > IF the PM Entity is a Working Group, and the protocol development > working group decides it does not have sufficient expertise to > take-up the work, or the proposal falls outside the current charter, > > THEN > > Approval SHOULD include the following steps > > o identification of protocol expertise to support metric development > > o consensus in the PM working group that there is interest and a > need for the work, and that a memo conforming to this framework > can be successfully developed > > o consultation with Area Director(s) > > o IESG approval of a revised charter for the PM working group > > > _______________________________________________ > PMOL mailing list > PMOL@ietf.org > http://www.ietf.org/mailman/listinfo/pmol > Return-Path: X-Original-To: pmol@ietf.org Delivered-To: pmol@core3.amsl.com Received: by core3.amsl.com (Postfix, from userid 0) id 11B1028C285; Mon, 26 Oct 2009 07:15:01 -0700 (PDT) From: Internet-Drafts@ietf.org To: i-d-announce@ietf.org Content-Type: Multipart/Mixed; Boundary="NextPart" Mime-Version: 1.0 Message-Id: <20091026141502.11B1028C285@core3.amsl.com> Date: Mon, 26 Oct 2009 07:15:02 -0700 (PDT) Cc: pmol@ietf.org Subject: [PMOL] I-D Action:draft-ietf-pmol-metrics-framework-03.txt X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Oct 2009 14:15:02 -0000 --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the Performance Metrics for Other Layers Working Group of the IETF. Title : Framework for Performance Metric Development Author(s) : A. Clark, B. Claise Filename : draft-ietf-pmol-metrics-framework-03.txt Pages : 22 Date : 2009-10-26 This document describes a framework and a process for developing performance metrics for IP-based applications that operate over reliable or datagram transport protocols, and that can be used to characterize traffic on live networks and services. The framework refers to a Performance Metrics Entity, or PM Entity, which may in future be a working group or directorate or a combination of these two. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-pmol-metrics-framework-03.txt Internet-Drafts are also available by anonymous FTP at: ftp://ftp.ietf.org/internet-drafts/ Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --NextPart Content-Type: Message/External-body; name="draft-ietf-pmol-metrics-framework-03.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <2009-10-26070541.I-D@ietf.org> --NextPart-- Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id A09703A67E2 for ; Mon, 26 Oct 2009 07:10:21 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.352 X-Spam-Level: X-Spam-Status: No, score=-2.352 tagged_above=-999 required=5 tests=[AWL=0.246, BAYES_00=-2.599, HTML_MESSAGE=0.001] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ySvsLYxOh5jG for ; Mon, 26 Oct 2009 07:10:20 -0700 (PDT) Received: from av-tac-bru.cisco.com (weird-brew.cisco.com [144.254.15.118]) by core3.amsl.com (Postfix) with ESMTP id 851B73A6783 for ; Mon, 26 Oct 2009 07:10:20 -0700 (PDT) X-TACSUNS: Virus Scanned Received: from strange-brew.cisco.com (localhost.cisco.com [127.0.0.1]) by av-tac-bru.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id n9QEAWUT020068 for ; Mon, 26 Oct 2009 15:10:32 +0100 (CET) Received: from [10.55.43.53] (ams-bclaise-8714.cisco.com [10.55.43.53]) by strange-brew.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id n9QEAWsS026009 for ; Mon, 26 Oct 2009 15:10:32 +0100 (CET) Message-ID: <4AE5ADD7.7020304@cisco.com> Date: Mon, 26 Oct 2009 15:10:31 +0100 From: Benoit Claise User-Agent: Thunderbird 2.0.0.23 (Windows/20090812) MIME-Version: 1.0 To: pmol@ietf.org Content-Type: multipart/alternative; boundary="------------040505020504070300000207" Subject: [PMOL] [Fwd: New Version Notification for draft-ietf-pmol-metrics-framework-03] X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Oct 2009 14:10:21 -0000 This is a multi-part message in MIME format. --------------040505020504070300000207 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Dear all, As agreed with Al Morton, I took the editor role in order to finish this document. I have integrated many suggested improvements compared to the previous version (which expired). As you can see there is a TO DO list, and some EDITOR'S NOTES throughout the document. All comments, feedbacks, improvements, text are welcome. Regards, Benoit. -------- Original Message -------- Subject: New Version Notification for draft-ietf-pmol-metrics-framework-03 Date: Mon, 26 Oct 2009 07:05:41 -0700 (PDT) From: IETF I-D Submission Tool To: bclaise@cisco.com CC: alan.d.clark@telchemy.com A new version of I-D, draft-ietf-pmol-metrics-framework-03.txt has been successfuly submitted by Benoit Claise and posted to the IETF repository. Filename: draft-ietf-pmol-metrics-framework Revision: 03 Title: Framework for Performance Metric Development Creation_date: 2009-10-26 WG ID: pmol Number_of_pages: 22 Abstract: This document describes a framework and a process for developing performance metrics for IP-based applications that operate over reliable or datagram transport protocols, and that can be used to characterize traffic on live networks and services. The framework refers to a Performance Metrics Entity, or PM Entity, which may in future be a working group or directorate or a combination of these two. The IETF Secretariat. --------------040505020504070300000207 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit Dear all,

As agreed with Al Morton, I took the editor role in order to finish this document.
I have integrated many suggested improvements compared to the previous version (which expired).
As you can see there is a TO DO list, and some EDITOR'S NOTES throughout the document.
All comments, feedbacks, improvements, text are welcome.

Regards, Benoit.

-------- Original Message --------
Subject: New Version Notification for draft-ietf-pmol-metrics-framework-03
Date: Mon, 26 Oct 2009 07:05:41 -0700 (PDT)
From: IETF I-D Submission Tool <idsubmission@ietf.org>
To: bclaise@cisco.com
CC: alan.d.clark@telchemy.com 


A new version of I-D, draft-ietf-pmol-metrics-framework-03.txt has been successfuly submitted by Benoit Claise and posted to the IETF repository.

Filename:	 draft-ietf-pmol-metrics-framework
Revision:	 03
Title:		 Framework for Performance Metric Development
Creation_date:	 2009-10-26
WG ID:		 pmol
Number_of_pages: 22

Abstract:
This document describes a framework and a process for developing
performance metrics for IP-based applications that operate over
reliable or datagram transport protocols, and that can be used to
characterize traffic on live networks and services.  The framework
refers to a Performance Metrics Entity, or PM Entity, which may in
future be a working group or directorate or a combination of these
two.
                                                                                  


The IETF Secretariat.
--------------040505020504070300000207-- Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 720553A69A5 for ; Tue, 6 Oct 2009 08:37:17 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.517 X-Spam-Level: X-Spam-Status: No, score=-2.517 tagged_above=-999 required=5 tests=[AWL=0.083, BAYES_00=-2.599, SPF_PASS=-0.001] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jQ3imJRrAyqe for ; Tue, 6 Oct 2009 08:37:16 -0700 (PDT) Received: from nostrum.com (nostrum-pt.tunnel.tserv2.fmt.ipv6.he.net [IPv6:2001:470:1f03:267::2]) by core3.amsl.com (Postfix) with ESMTP id 376413A68FC for ; Tue, 6 Oct 2009 08:37:15 -0700 (PDT) Received: from [192.168.2.2] (pool-173-71-53-15.dllstx.fios.verizon.net [173.71.53.15]) (authenticated bits=0) by nostrum.com (8.14.3/8.14.3) with ESMTP id n96FcoLh053459 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Tue, 6 Oct 2009 10:38:50 -0500 (CDT) (envelope-from rjsparks@nostrum.com) Message-Id: <285328B5-AAFC-4D55-9810-D9D9310271C8@nostrum.com> From: Robert Sparks To: Al Morton In-Reply-To: <200909301149.n8UBnMco001647@klpd017.kcdc.att.com> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v936) Date: Tue, 6 Oct 2009 10:38:50 -0500 References: <72A0E3C5-322B-4FEE-B565-9659581BBFC4@nostrum.com> <200909301149.n8UBnMco001647@klpd017.kcdc.att.com> X-Mailer: Apple Mail (2.936) Received-SPF: pass (nostrum.com: 173.71.53.15 is authenticated by a trusted mechanism) Cc: pmol@ietf.org Subject: Re: [PMOL] Several questions/suggestions from my review of draft-ietf-pmol-sip-perf-metrics-04 X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Oct 2009 15:37:17 -0000 Thanks Al - I'm going to enter a discuss pointing to my original message in the PMOL archives. The tracker will send email to you and the authors. I wanted to let you know I had seen this and that the message the tracker sends you is not a prod. RjS On Sep 30, 2009, at 6:49 AM, Al Morton wrote: > Robert, > > thanks for your questions & suggestions. Daryl and I will get > together to address them, and will be in touch soon. > > Al (as co-author) > > +++++++++++++++++ > > PMOL WG, > If you have strong opinions about any of the areas > that Robert identified in his message, feel free to > post to the list on the topic(s) > (using a new and descriptive subject line, please). > > thanks, > Al (as pmol co-chair) > > At 02:31 PM 9/29/2009, Robert Sparks wrote: >> Hi All - >> >> I have several concerns about draft-ietf-pmol-sip-perf-metrics that I >> would like to discuss. >> I've asked for a dedicated RAI-review of this document, so there may >> be additional comments >> later, but I wanted to get these to you now so we can start working >> through them. >> >> These comments are more-or-less in document order, with a couple of >> nits moved to the end. >> I've numbered them to help split responses into threads later. When >> replying to just one of the >> items below, please change the subject line to indicate what you're >> replying to. >> >> Thanks, >> >> RjS > Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 4D7E13A6894; Fri, 2 Oct 2009 19:32:33 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.523 X-Spam-Level: X-Spam-Status: No, score=-6.523 tagged_above=-999 required=5 tests=[AWL=0.076, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BSlD34KVtZxx; Fri, 2 Oct 2009 19:32:31 -0700 (PDT) Received: from zrtps0kp.nortel.com (zrtps0kp.nortel.com [47.140.192.56]) by core3.amsl.com (Postfix) with ESMTP id 8D5233A6852; Fri, 2 Oct 2009 19:32:31 -0700 (PDT) Received: from zrtphxs1.corp.nortel.com (zrtphxs1.corp.nortel.com [47.140.202.46]) by zrtps0kp.nortel.com (Switch-2.2.6/Switch-2.2.0) with ESMTP id n932XLO01087; Sat, 3 Oct 2009 02:33:21 GMT Received: from [47.141.31.158] ([47.141.31.158]) by zrtphxs1.corp.nortel.com with Microsoft SMTPSVC(6.0.3790.3959); Fri, 2 Oct 2009 22:33:21 -0400 From: "Dale Worley" To: rai@ietf.org Content-Type: text/plain Organization: Nortel Networks Date: Fri, 02 Oct 2009 22:33:20 -0400 Message-Id: <1254537200.4303.12.camel@khone.us.nortel.com> Mime-Version: 1.0 X-Mailer: Evolution 2.12.3 (2.12.3-5.fc8) Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 03 Oct 2009 02:33:21.0478 (UTC) FILETIME=[DFC29E60:01CA43D1] X-Mailman-Approved-At: Sat, 03 Oct 2009 02:15:10 -0700 Cc: alan@telchemy.com, pmol@ietf.org, Mary Barnes , rjsparks@nostrum.com Subject: [PMOL] RAI-ART review of draft-ietf-pmol-sip-perf-metrics-04 by Dale R. Worley X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Oct 2009 02:32:33 -0000 RAI-ART review of draft-ietf-pmol-sip-perf-metrics-04 Dale R. Worley I am the assigned RAI-ART reviewer for draft-ietf-pmol-sip-perf-metrics-04.txt. For background on RAI-ART, please see the FAQ at . Please resolve these comments along with any other Last Call comments you may receive. This draft is on the right track but has open issues, described in the review. My comments are divided into groups of varying significance: - Situations where the draft appears to have omitted measurements that would be of value, and which appear to be of the same category as measurements that are included in the draft. However, this omission may have been deliberate due to the intended scope of the draft, in which case the intended limitation should be described (and perhaps justified) in the "Scope" section. - Deficiencies in the exposition of how various timing measurements are to be taken. These may also mask technical issues about how the measurements are intended to be taken. - Editorial issues, mostly involving aligning terminology with the common usage in SIP discussions. The items in these groups are numbered with discontinuous sets of numbers. Situations where the draft appears to have omitted measurements that would be of value, and which appear to be of the same category as measurements that are included in the draft. However, this omission may have been deliberate due to the intended scope of the draft, in which case the intended limitation should be described (and perhaps justified) in the "Scope" section. 100. Is the intention to restrict attention to signaling (SIP) alone? In our experience, performance problems first come to users' attention in media (RTP), and any environment with tolerable media performance has more than adequate signaling performance. 101. The measurements appear to be designed to closely parallel performance metrics of TDM telephone systems. This may be intentional, but this draft omits a number of measurements that do not closely parallel TDM performance metrics, but are nonetheless important for the performance of SIP systems, even when limiting attention to "traditional telephone" usage. (See later items for specifics.) 102. Many of the metrics appear to form natural triples: - Average delay when the operation (of a particular class) was successful. - Average delay when the operation was unsuccessful. - Percentage of operations of the class which were successful. The metrics scheme would be clarified and made more systematic if this grouping was defined as an overall property of the metric scheme and then applied as such to the various classes of operations: registration request, session request, session establishment, session disconnect, etc. E.g., although "Failed session completion delay" (average BYE-with-failure response delay) is defined, the *percentage* of failed session completions is not defined, despite that that metric should be very low in any well-functioning network, and thus can provide a valuable performance indicator. 103. Some metrics distinguish between "failure in the network" and "failure at the destination user agent"; e.g., "Session Defects Ratio". Beware that distinguishing between these two cases is very difficult and cannot easily be done by listing response codes for the two cases. (I've been developing an I-D for a couple of years on how to deal with this problem -- I have yet to devise a good solution.) It may also be desirable to apply this distinction to other operations other than initial INVITEs -- this would effectively add further metrics to the "natural triple" described in item 102, as it divides the class of failures into two sub-classes. 104. In regard to "telephone calls" (INVITE-initiated sessions), there are three metrics described (each of which gives rise to a triple of metrics): - Session request (INVITE to 180) - Session setup (INVITE to final response) - Session disconnect (BYE to final response) In addition, there is a metric for REGISTER requests. But there are a number of additional operations whose performance is important to the performance of SIP signaling, and which may differ greatly from the performance of the above operations (and for which the users' expectations may be very different from the above operations): - re-INVITEs (These are particularly important as re-INVITE is used to implement on-hold and off-hold operations, and users expect those operations to complete much quicker than initial INVITEs.) - UPDATE (if it is used in practice) - initial SUBSCRIBE (for dialog events) - re-SUBSCRIBE - NOTIFY Several metrics might be defined regarding REFER requests, which are used to implement blind and consultative transfers. In addition, MESSAGE requests can be sent either out-of-dialog or within-dialog, and one expects the performance of the two cases to be quite different, so there should be separate metrics for the two cases. 105. Similar to the problem with detecting "failure in the network", determining when session request is finished (that is, ringing starts) is difficult to specify. A 180 response is clear indication that a user has been notified, but other 18x responses may be considered as successful setup in some situations. E.g., a 182 Queued message may be considered "end of session request" if one is concerned about the performance of the network, although it does not indicate the start of useful communication from the user's point of view. 106. In regard to issues 105 and 103, it may be necessary to allow that metrics may be "parametrized" by attaching a specification of which response codes are treated as having which meaning. In any case, allowance needs to be made that experience with the metrics may show that the various specified sets of response codes need to be modified to maximize the usefulness of the metrics. Deficiencies in the exposition of how various timing measurements are to be taken. However, these may mask technical issues about how the measurements are intended to be taken. 200. Section 3 of the draft gives a nice standard for how the time from sending a request to receiving a response is to be measured ("T1 to T4"), including within it a standard for how to measure the time of sending and receiving ("first bit sent" vs. "last bit received"). However, in many places (e.g., section 4.1) these standards are not referenced, but rather they are restated. This leads the reader to have to check each metric definition to see if it is defined in the same way as described in section 3, and to wonder if all of the metrics are defined in the same way. Better would be to have section 3 note explicitly that all metrics use this definition of time delay. 201. Metrics involving INVITE should explicitly note that ACK is not considered as part of the time delay; all time delays are measured from sending the INVITE to receiving its response. Or rather, there should be an overall declaration of this standard in section 3. 202. The discussion of "Successful session duration SDT" shows some examples but does not clearly indicate how all four cases are to be handled. (The cases are the combinations of: measurement at the originating end vs. measurement at the terminating end, sending BYE vs. receiving BYE.) There is also some confusion regarding T1 and T4, which are specified in multiple different ways in this section. 203. Consideration should be made in the various definitions of calls that are timed-out by the session-timer mechanism or other session-keepalive mechanisms. (Unless it is known that the intended networks will not use session-keepalives, or that session-keepalive failures will not cause UAs to see differing signaling flows.) 204. As Robert notes, the "Hops per Request" measurement is unlikely to be useful for out-of-dialog requests, because it does not capture redirections handled by proxies and failed forks, and so is poorly correlated with the amount of work needed to deliver a request to its destination. However, this metric may be useful for in-dialog requests, as there is usually no redirection and (AFAIK) B2BUAs maintain the Max-Forwards value. Editorial issues, mostly involving aligning terminology with the common usage in SIP discussions. 300. Many of the metrics appear to be intended to report the average of a quantity that is measured for many executions of an operation. But most of these metrics do not state that averaging is to be done, they describe in detail how a delay quantity is to be measured, but do not mention that averaging is to be done. (E.g., the word "average" appears only twice in the draft.) 301. The term "session" seems to always be used to describe what is called a "dialog" in the SIP world, seeing has how the media (RTP) session is never discussed. Is this a telecom usage, or should it be replaced with "dialog"? 302. As Robert notes, T1 and T4 are used in ways that conflict with the use of those symbols in RFC 3261. They also lead one to wonder what happened to T2 and T3. But perhaps these symbols are a reference to a terminology used in TDM performance measurement and should be retained due to that standardization. 303. In measurements regarding a single request and its response(s), the roles UAC and UAS are well-defined. But in session-duration measurements, the roles of UAC and UAS are defined only within individual request-response transactions. In that case, a different terminology should be used. (E.g., "originator" vs. "terminator"? -- Surely the telecom world has words for this.)