Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id F34193A69ED for ; Fri, 30 Jul 2010 08:04:27 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.459 X-Spam-Level: X-Spam-Status: No, score=-2.459 tagged_above=-999 required=5 tests=[AWL=0.140, BAYES_00=-2.599] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BXO5CWqbYMdB for ; Fri, 30 Jul 2010 08:04:27 -0700 (PDT) Received: from av-tac-bru.cisco.com (weird-brew.cisco.com [144.254.15.118]) by core3.amsl.com (Postfix) with ESMTP id A52253A69BF for ; Fri, 30 Jul 2010 08:04:26 -0700 (PDT) X-TACSUNS: Virus Scanned Received: from strange-brew.cisco.com (localhost.cisco.com [127.0.0.1]) by av-tac-bru.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id o6UF14QQ008784; Fri, 30 Jul 2010 17:01:04 +0200 (CEST) Received: from [10.55.92.55] (dhcp-10-55-92-55.cisco.com [10.55.92.55]) by strange-brew.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id o6UF13Z1008781; Fri, 30 Jul 2010 17:01:04 +0200 (CEST) Message-ID: <4C52E92F.7080006@cisco.com> Date: Fri, 30 Jul 2010 17:01:03 +0200 From: Benoit Claise User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.1.11) Gecko/20100711 Thunderbird/3.0.6 MIME-Version: 1.0 To: Al Morton References: <201007301030.o6UAUs7T005666@alpd052.aldc.att.com> In-Reply-To: <201007301030.o6UAUs7T005666@alpd052.aldc.att.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: pmol@ietf.org Subject: Re: [PMOL] Call for New Metrics Development Directorate Volunteers X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Jul 2010 15:04:28 -0000 Al, Yep. Interested. Regards, Benoit. > PMOL, > > If you are interested in reviewing Internet-drafts > from the perspective of developing solid performance > metrics and using our draft framework as guidance, > please contact me off-list. > > Latest draft of PMOL Framework/Guidelines: > http://tools.ietf.org/html/draft-ietf-pmol-metrics-framework > > regards, > Al > > _______________________________________________ > PMOL mailing list > PMOL@ietf.org > https://www.ietf.org/mailman/listinfo/pmol Return-Path: X-Original-To: pmol@ietf.org Delivered-To: pmol@core3.amsl.com Received: by core3.amsl.com (Postfix, from userid 0) id E975B3A691B; Fri, 30 Jul 2010 06:15:01 -0700 (PDT) From: Internet-Drafts@ietf.org To: i-d-announce@ietf.org Content-Type: Multipart/Mixed; Boundary="NextPart" Mime-Version: 1.0 Message-Id: <20100730131501.E975B3A691B@core3.amsl.com> Date: Fri, 30 Jul 2010 06:15:01 -0700 (PDT) Cc: pmol@ietf.org Subject: [PMOL] I-D Action:draft-ietf-pmol-sip-perf-metrics-06.txt X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Jul 2010 13:15:02 -0000 --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the Performance Metrics for Other Layers Working Group of the IETF. Title : Basic Telephony SIP End-to-End Performance Metrics Author(s) : D. Malas, A. Morton Filename : draft-ietf-pmol-sip-perf-metrics-06.txt Pages : 29 Date : 2010-07-30 This document defines a set of metrics and their usage to evaluate the performance of end-to-end Session Initiation Protocol (SIP) for telephony services in both production and testing environments. The purpose of this document is to combine a standard set of common metrics, allowing interoperable performance measurements, easing the comparison of industry implementations. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-pmol-sip-perf-metrics-06.txt Internet-Drafts are also available by anonymous FTP at: ftp://ftp.ietf.org/internet-drafts/ Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --NextPart Content-Type: Message/External-body; name="draft-ietf-pmol-sip-perf-metrics-06.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <2010-07-30061310.I-D@ietf.org> --NextPart-- Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 9C2FA3A67FA; Fri, 30 Jul 2010 05:27:27 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: 0.731 X-Spam-Level: X-Spam-Status: No, score=0.731 tagged_above=-999 required=5 tests=[AWL=-0.665, BAYES_20=-0.74, HELO_EQ_MODEMCABLE=0.768, HOST_EQ_MODEMCABLE=1.368] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Vo1476pmji4n; Fri, 30 Jul 2010 05:27:26 -0700 (PDT) Received: from ondar.cablelabs.com (ondar.cablelabs.com [192.160.73.61]) by core3.amsl.com (Postfix) with ESMTP id 35DD33A693F; Fri, 30 Jul 2010 05:27:26 -0700 (PDT) Received: from kyzyl.cablelabs.com (kyzyl [10.253.0.7]) by ondar.cablelabs.com (8.14.4/8.14.4) with ESMTP id o6UCRoKM004592; Fri, 30 Jul 2010 06:27:50 -0600 Received: from srvxchg.cablelabs.com (10.5.0.15) by kyzyl.cablelabs.com (F-Secure/fsigk_smtp/303/kyzyl.cablelabs.com); Fri, 30 Jul 2010 06:27:50 -0700 (MST) X-Virus-Status: clean(F-Secure/fsigk_smtp/303/kyzyl.cablelabs.com) Received: from srvxchg.cablelabs.com ([10.5.0.15]) by srvxchg ([10.5.0.15]) with mapi; Fri, 30 Jul 2010 06:27:50 -0600 From: Daryl Malas To: "pmol@ietf.org" , "rai@ietf.org" Date: Fri, 30 Jul 2010 06:27:47 -0600 Thread-Topic: [RAI] [PMOL] RE: SIP Perf Telephony Metrics: SER and SEER Updates Thread-Index: Acsu/2FvroY5LByvTPqBebFJWs5sOgAALobAADiWsSA= Message-ID: <114DAD31379DFA438C0A2E39B3B8AF5D01422BEF0F@srvxchg> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Approved: ondar Subject: [PMOL] [RAI] RE: SIP Perf Telephony Metrics: SER and SEER Updates X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Jul 2010 12:27:27 -0000 All, Please ignore this as the text surrounding these two metrics has change, an= d these questions no longer pertain to the draft. If you have further ques= tions or comments, please let me know. Regards, Daryl=20 -----Original Message----- From: Daryl Malas=20 Sent: Thursday, July 29, 2010 11:28 AM To: 'rai@ietf.org' Subject: FW: [PMOL] SIP Perf Telephony Metrics: SER and SEER Updates Forwarding this to the RAI area list for input from "SIP Experts" participa= ting in this area. Regards, Daryl=20 -----Original Message----- From: pmol-bounces@ietf.org [mailto:pmol-bounces@ietf.org] On Behalf Of Dar= yl Malas Sent: Thursday, July 29, 2010 11:21 AM To: pmol@ietf.org Cc: Robert Sparks Subject: [PMOL] SIP Perf Telephony Metrics: SER and SEER Updates All, During the IESG review process, there were some comments about the response= codes used in the SER and SEER metrics. I have updated the text in these = metrics and included the updates below. Please review these and provide an= y feedback. Specifically, are the response codes and their purpose clear, = and are the right set of response codes for the purposes of these metrics. = If it is unclear the purpose of these metrics, please let me know. Regards, Daryl ---------------------- 4.6. Session Establishment Ratio (SER) This metric is used to detect the ability of a terminating UA or downstream proxy to successfully establish sessions per new session INVITE requests. SER is defined as the number of new session INVITE requests resulting in a 200 OK response, to the total number of attempted INVITE requests less the total number of attempted INVITE requests less INVITE requests resulting in a 3XX, 401, 402, and 407 response. The 3XX, 401, 402 and 407 response codes were chosen, because they indicate an acceptable recoverable UA effect without the interaction of an individual user of the UA. This metric is similar to Answer Seizure Ratio (ASR) defined in [E.411]. It is measured at the originating UA only. The output value of this metric is numerical and SHOULD be adjusted to indicate a percentage of successfully established sessions. The SER is calculated using the following formula metric. In order to simplify the formula, the following variable is used to summarize multiple SIP responses: a =3D 3XX, 401, 402, and 407 # of INVITE Requests w/ associated 200 OK SER =3D --------------------------------------------------------- x 100 (Total # of INVITE Requests)-(# of INVITE Requests w/ 'a' Response) The following message exchange provides an example of identifiable events necessary for inputs in determining session establishment as described above: UA1 UA2 | | |INVITE | +----------->|------------------>| | | 180| | |<------------------| Session Established | | | | | | | 200| +----------->|<------------------| | | The following is an example call flow including a SIP 302 Redirect response. UA1 UA2 UA3 | | | +----------->|INVITE | | | |------------------>| | INVITE w/ 'a' Response | | | | | 302| | +----------->|<------------------| | | | | |INVITE | +----------->|-------------------------------------->| | | | | | 180| Session Established |<--------------------------------------| | | | | | 200| +----------->|<--------------------------------------| | | 4.7. Session Establishment Effectiveness Ratio (SEER) This metric is complimentary to SER, but is intended to exclude the potential effects of the target UA from the metric. SER is defined as the number of INVITE requests resulting in a 200 OK response and INVITE requests resulting in a 480, 486, or 600; less the total number of attempted INVITE requests less INVITE requests resulting in a 3XX, 401, 402, and 407 response. The response codes 480, 486 and 600 were chosen, because they clearly indicate the effect of an individual user of the UA. It is possible an individual user could cause a negative effect on the UA. For example, they may have misconfigured the UA causing a response code not directly related to a SSP, but this cannot be easily determined from an intermediary B2BUA somewhere inbetween the originating and terminating UA. With this in consideration, response codes such as 415 and 420 were not included in the numerator of the metric. If future response codes provide similar effects, they should be considered in addition with this metric. This metric is similar to Network Effectiveness Ratio (NER) defined in [E.411]. It is measured at the originating UA only. The output value of this metric is numerical and SHOULD be adjusted to indicate a percentage of successfully established sessions less common UAS failures. In order to simplify the formula, the following variable is used to summarize multiple SIP responses: a =3D 3XX, 401, 402, and 407 The SEER is calculated using the following formula: # of INVITE Requests w/ associated 200 OK, 480, 486, or 600 SEER =3D -------------------------------------------------------- x 100 (Total # of INVITE Requests)-(# of INVITE Requests w/ 'a' Response) Reference the example flow is Section 4.6. _______________________________________________ PMOL mailing list PMOL@ietf.org https://www.ietf.org/mailman/listinfo/pmol Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id C3A263A6840 for ; Fri, 30 Jul 2010 03:31:16 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -105.479 X-Spam-Level: X-Spam-Status: No, score=-105.479 tagged_above=-999 required=5 tests=[AWL=0.317, BAYES_00=-2.599, MSGID_FROM_MTA_HEADER=0.803, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ebMRDpnFkTz7 for ; Fri, 30 Jul 2010 03:31:14 -0700 (PDT) Received: from mail146.messagelabs.com (mail146.messagelabs.com [216.82.241.147]) by core3.amsl.com (Postfix) with ESMTP id 518E33A6885 for ; Fri, 30 Jul 2010 03:30:43 -0700 (PDT) X-VirusChecked: Checked X-Env-Sender: acmorton@att.com X-Msg-Ref: server-13.tower-146.messagelabs.com!1280485862!11731541!1 X-StarScan-Version: 6.2.4; banners=-,-,- X-Originating-IP: [144.160.20.145] Received: (qmail 19458 invoked from network); 30 Jul 2010 10:31:03 -0000 Received: from sbcsmtp6.sbc.com (HELO mlpd192.enaf.sfdc.sbc.com) (144.160.20.145) by server-13.tower-146.messagelabs.com with DHE-RSA-AES256-SHA encrypted SMTP; 30 Jul 2010 10:31:03 -0000 Received: from enaf.sfdc.sbc.com (localhost.localdomain [127.0.0.1]) by mlpd192.enaf.sfdc.sbc.com (8.14.4/8.14.4) with ESMTP id o6UAVIU3009084 for ; Fri, 30 Jul 2010 06:31:18 -0400 Received: from alpd052.aldc.att.com (alpd052.aldc.att.com [130.8.42.31]) by mlpd192.enaf.sfdc.sbc.com (8.14.4/8.14.4) with ESMTP id o6UAVDMc009024 for ; Fri, 30 Jul 2010 06:31:13 -0400 Received: from aldc.att.com (localhost.localdomain [127.0.0.1]) by alpd052.aldc.att.com (8.14.3/8.14.3) with ESMTP id o6UAUvTb005757 for ; Fri, 30 Jul 2010 06:30:57 -0400 Received: from dns.maillennium.att.com (dns.maillennium.att.com [135.25.114.99]) by alpd052.aldc.att.com (8.14.3/8.14.3) with ESMTP id o6UAUs7T005666 for ; Fri, 30 Jul 2010 06:30:54 -0400 Message-Id: <201007301030.o6UAUs7T005666@alpd052.aldc.att.com> Received: from acmt.att.com (vpn-135-70-82-100.vpn.swst.att.com[135.70.82.100](misconfigured sender)) by maillennium.att.com (mailgw1) with SMTP id <20100730103052gw100s4p70e>; Fri, 30 Jul 2010 10:30:53 +0000 X-Originating-IP: [135.70.82.100] X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Fri, 30 Jul 2010 06:30:00 -0400 To: pmol@ietf.org From: Al Morton Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Subject: [PMOL] Call for New Metrics Development Directorate Volunteers X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Jul 2010 10:31:17 -0000 PMOL, If you are interested in reviewing Internet-drafts from the perspective of developing solid performance metrics and using our draft framework as guidance, please contact me off-list. Latest draft of PMOL Framework/Guidelines: http://tools.ietf.org/html/draft-ietf-pmol-metrics-framework regards, Al Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id BDC4928C2A6 for ; Fri, 30 Jul 2010 01:41:05 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.599 X-Spam-Level: X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hHH2xDpkalF0 for ; Fri, 30 Jul 2010 01:41:04 -0700 (PDT) Received: from mail-ww0-f44.google.com (mail-ww0-f44.google.com [74.125.82.44]) by core3.amsl.com (Postfix) with ESMTP id 9091C28C29B for ; Fri, 30 Jul 2010 01:40:34 -0700 (PDT) Received: by wwj40 with SMTP id 40so1020705wwj.13 for ; Fri, 30 Jul 2010 01:39:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=RIodq6ucxZ1itqHXMAzYgk0T4BSJCE2eHcLbKXKGTNs=; b=Uq+wlKr20r2mureqP6tnsDfxfNGpdIUYqyTm31ziwF8J0BADY97kdqHXRdR2IeZgiQ ucAkpRWm1Eiq2g9/aW7vJQyE5qQH5RNzE01U7HP9oTL1g6Ps4ChnrmQw6lzaLDc5oN63 Ah4mR6p+Xp2i96AvnBYTxONM0dzihiwRanlOk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=JhaLwhpP1NqsTOlqbML75FllhlfnkN7Get2IG24D5tD8/NlQUrGJF+VWOtyVKbtf+3 UzzNdrewb88VL085CgcQrjjrpiJpphrYOlTDcKLu771Z5nbncjwb4v89+T27wYYaylSw VwJZrsXHrOc3QZruun09Qna2Ri9NRWtqyQC3E= MIME-Version: 1.0 Received: by 10.216.145.198 with SMTP id p48mr1358882wej.18.1280479185421; Fri, 30 Jul 2010 01:39:45 -0700 (PDT) Received: by 10.216.133.207 with HTTP; Fri, 30 Jul 2010 01:39:45 -0700 (PDT) In-Reply-To: <201007300814.o6U8EgnZ004975@alpd052.aldc.att.com> References: <201006281417.o5SEHthw022056@alpd052.aldc.att.com> <201007291545.o6TFjpgR006323@alpd052.aldc.att.com> <201007300814.o6U8EgnZ004975@alpd052.aldc.att.com> Date: Fri, 30 Jul 2010 14:09:45 +0530 Message-ID: From: Vinayak Hegde To: Al Morton Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: pmol@ietf.org Subject: Re: [PMOL] Informal pmol lunch session at IETF-78 X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Jul 2010 08:41:05 -0000 The HTTP latency draft v00 has been submitted. You can access it at https://datatracker.ietf.org/doc/draft-vinayak-http-latency/ Some sections have to be expanded and marked as TBD Regards Vinayak On Fri, Jul 30, 2010 at 1:45 PM, Al Morton wrote: > PMOL, > > Here's the list of items we will cover today: > > Latest draft of Framework/Guidelines > http://tools.ietf.org/html/draft-ietf-pmol-metrics-framework > > Questions/Issues on SIP Draft > http://tools.ietf.org/html/draft-ietf-pmol-sip-perf-metrics-05 > http://www.ietf.org/mail-archive/web/pmol/current/msg00230.html > > HTTP Latency Metrics =A0draft-vinayak-http-latency-00 > (this may be posted in time for the meeting). > > We'll cover these topics in the most convenient order. > > We'll start at 11:30, preferring to finish early if we can > (and eat lunch after the meeting). > > Al > > At 11:46 AM 7/29/2010, Al Morton wrote: >> >> Thanks to Dan, the meeting place is >> >> IESG office room - Rhine 2.8 >> >> See you there, >> Al >> pmol co-chair >> >> At 10:18 AM 6/28/2010, Al Morton wrote: >>> >>> Folks, >>> >>> the pmol wg will meet informally for a (bring-your-own) lunch on >>> Friday, July 30, at 11:30 local time. >>> >>> the meeting place will be announced later. >>> >>> the Agenda includes the status of both our current drafts, >>> and any other business. >>> >>> regards, >>> Al >>> pmol co-chair >>> >>> _______________________________________________ >>> PMOL mailing list >>> PMOL@ietf.org >>> https://www.ietf.org/mailman/listinfo/pmol >> >> _______________________________________________ >> PMOL mailing list >> PMOL@ietf.org >> https://www.ietf.org/mailman/listinfo/pmol > > _______________________________________________ > PMOL mailing list > PMOL@ietf.org > https://www.ietf.org/mailman/listinfo/pmol > Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id D73D73A6AB8 for ; Fri, 30 Jul 2010 01:14:29 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -105.448 X-Spam-Level: X-Spam-Status: No, score=-105.448 tagged_above=-999 required=5 tests=[AWL=0.349, BAYES_00=-2.599, MSGID_FROM_MTA_HEADER=0.803, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vmCffSUnBMRG for ; Fri, 30 Jul 2010 01:14:26 -0700 (PDT) Received: from mail146.messagelabs.com (mail146.messagelabs.com [216.82.241.147]) by core3.amsl.com (Postfix) with ESMTP id D5E653A69BD for ; Fri, 30 Jul 2010 01:14:25 -0700 (PDT) X-VirusChecked: Checked X-Env-Sender: acmorton@att.com X-Msg-Ref: server-7.tower-146.messagelabs.com!1280477689!17532013!1 X-StarScan-Version: 6.2.4; banners=-,-,- X-Originating-IP: [144.160.20.145] Received: (qmail 11177 invoked from network); 30 Jul 2010 08:14:49 -0000 Received: from sbcsmtp6.sbc.com (HELO mlpd192.enaf.sfdc.sbc.com) (144.160.20.145) by server-7.tower-146.messagelabs.com with DHE-RSA-AES256-SHA encrypted SMTP; 30 Jul 2010 08:14:49 -0000 Received: from enaf.sfdc.sbc.com (localhost.localdomain [127.0.0.1]) by mlpd192.enaf.sfdc.sbc.com (8.14.4/8.14.4) with ESMTP id o6U8F4ML019272 for ; Fri, 30 Jul 2010 04:15:04 -0400 Received: from alpd052.aldc.att.com (alpd052.aldc.att.com [130.8.42.31]) by mlpd192.enaf.sfdc.sbc.com (8.14.4/8.14.4) with ESMTP id o6U8F1sG019252 for ; Fri, 30 Jul 2010 04:15:01 -0400 Received: from aldc.att.com (localhost.localdomain [127.0.0.1]) by alpd052.aldc.att.com (8.14.3/8.14.3) with ESMTP id o6U8Ejvh005067 for ; Fri, 30 Jul 2010 04:14:45 -0400 Received: from dns.maillennium.att.com (dns.maillennium.att.com [135.25.114.99]) by alpd052.aldc.att.com (8.14.3/8.14.3) with ESMTP id o6U8EgnZ004975 for ; Fri, 30 Jul 2010 04:14:42 -0400 Message-Id: <201007300814.o6U8EgnZ004975@alpd052.aldc.att.com> Received: from acmt.att.com (vpn-135-70-168-4.vpn.mwst.att.com[135.70.168.4](misconfigured sender)) by maillennium.att.com (mailgw1) with SMTP id <20100730081441gw100s4p6se>; Fri, 30 Jul 2010 08:14:41 +0000 X-Originating-IP: [135.70.168.4] X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Fri, 30 Jul 2010 04:15:00 -0400 To: pmol@ietf.org From: Al Morton In-Reply-To: <201007291545.o6TFjpgR006323@alpd052.aldc.att.com> References: <201006281417.o5SEHthw022056@alpd052.aldc.att.com> <201007291545.o6TFjpgR006323@alpd052.aldc.att.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Subject: Re: [PMOL] Informal pmol lunch session at IETF-78 X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Jul 2010 08:14:30 -0000 PMOL, Here's the list of items we will cover today: Latest draft of Framework/Guidelines http://tools.ietf.org/html/draft-ietf-pmol-metrics-framework Questions/Issues on SIP Draft http://tools.ietf.org/html/draft-ietf-pmol-sip-perf-metrics-05 http://www.ietf.org/mail-archive/web/pmol/current/msg00230.html HTTP Latency Metrics draft-vinayak-http-latency-00 (this may be posted in time for the meeting). We'll cover these topics in the most convenient order. We'll start at 11:30, preferring to finish early if we can (and eat lunch after the meeting). Al At 11:46 AM 7/29/2010, Al Morton wrote: >Thanks to Dan, the meeting place is > >IESG office room - Rhine 2.8 > >See you there, >Al >pmol co-chair > >At 10:18 AM 6/28/2010, Al Morton wrote: >>Folks, >> >>the pmol wg will meet informally for a (bring-your-own) lunch on >>Friday, July 30, at 11:30 local time. >> >>the meeting place will be announced later. >> >>the Agenda includes the status of both our current drafts, >>and any other business. >> >>regards, >>Al >>pmol co-chair >> >>_______________________________________________ >>PMOL mailing list >>PMOL@ietf.org >>https://www.ietf.org/mailman/listinfo/pmol > >_______________________________________________ >PMOL mailing list >PMOL@ietf.org >https://www.ietf.org/mailman/listinfo/pmol Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id E516428C10B for ; Thu, 29 Jul 2010 08:45:36 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -105.409 X-Spam-Level: X-Spam-Status: No, score=-105.409 tagged_above=-999 required=5 tests=[AWL=0.387, BAYES_00=-2.599, MSGID_FROM_MTA_HEADER=0.803, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id dfVV-gHvE4SH for ; Thu, 29 Jul 2010 08:45:36 -0700 (PDT) Received: from mail146.messagelabs.com (mail146.messagelabs.com [216.82.241.147]) by core3.amsl.com (Postfix) with ESMTP id B5C3B28C0E1 for ; Thu, 29 Jul 2010 08:45:35 -0700 (PDT) X-VirusChecked: Checked X-Env-Sender: acmorton@att.com X-Msg-Ref: server-7.tower-146.messagelabs.com!1280418358!17490305!1 X-StarScan-Version: 6.2.4; banners=-,-,- X-Originating-IP: [144.160.20.145] Received: (qmail 31989 invoked from network); 29 Jul 2010 15:45:58 -0000 Received: from sbcsmtp6.sbc.com (HELO mlpd192.enaf.sfdc.sbc.com) (144.160.20.145) by server-7.tower-146.messagelabs.com with DHE-RSA-AES256-SHA encrypted SMTP; 29 Jul 2010 15:45:58 -0000 Received: from enaf.sfdc.sbc.com (localhost.localdomain [127.0.0.1]) by mlpd192.enaf.sfdc.sbc.com (8.14.4/8.14.4) with ESMTP id o6TFkDDj001333 for ; Thu, 29 Jul 2010 11:46:13 -0400 Received: from alpd052.aldc.att.com (alpd052.aldc.att.com [130.8.42.31]) by mlpd192.enaf.sfdc.sbc.com (8.14.4/8.14.4) with ESMTP id o6TFkAmZ001236 for ; Thu, 29 Jul 2010 11:46:10 -0400 Received: from aldc.att.com (localhost.localdomain [127.0.0.1]) by alpd052.aldc.att.com (8.14.3/8.14.3) with ESMTP id o6TFjsFC006492 for ; Thu, 29 Jul 2010 11:45:54 -0400 Received: from mailgw1.maillennium.att.com (mailgw1.maillennium.att.com [135.25.114.99]) by alpd052.aldc.att.com (8.14.3/8.14.3) with ESMTP id o6TFjpgR006323 for ; Thu, 29 Jul 2010 11:45:51 -0400 Message-Id: <201007291545.o6TFjpgR006323@alpd052.aldc.att.com> Received: from acmt.att.com (vpn-135-70-245-180.vpn.east.att.com[135.70.245.180](misconfigured sender)) by maillennium.att.com (mailgw1) with SMTP id <20100729154550gw100s4p53e>; Thu, 29 Jul 2010 15:45:50 +0000 X-Originating-IP: [135.70.245.180] X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Thu, 29 Jul 2010 11:46:09 -0400 To: pmol@ietf.org From: Al Morton In-Reply-To: <201006281417.o5SEHthw022056@alpd052.aldc.att.com> References: <201006281417.o5SEHthw022056@alpd052.aldc.att.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Subject: Re: [PMOL] Informal pmol lunch session at IETF-78 X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Jul 2010 15:45:37 -0000 Thanks to Dan, the meeting place is IESG office room - Rhine 2.8 See you there, Al pmol co-chair At 10:18 AM 6/28/2010, Al Morton wrote: >Folks, > >the pmol wg will meet informally for a (bring-your-own) lunch on >Friday, July 30, at 11:30 local time. > >the meeting place will be announced later. > >the Agenda includes the status of both our current drafts, >and any other business. > >regards, >Al >pmol co-chair > >_______________________________________________ >PMOL mailing list >PMOL@ietf.org >https://www.ietf.org/mailman/listinfo/pmol Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 3317D3A69E2; Thu, 29 Jul 2010 02:20:56 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: 1.032 X-Spam-Level: * X-Spam-Status: No, score=1.032 tagged_above=-999 required=5 tests=[AWL=-1.105, BAYES_50=0.001, HELO_EQ_MODEMCABLE=0.768, HOST_EQ_MODEMCABLE=1.368] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id W9YENx2kZYNW; Thu, 29 Jul 2010 02:20:54 -0700 (PDT) Received: from ondar.cablelabs.com (ondar.cablelabs.com [192.160.73.61]) by core3.amsl.com (Postfix) with ESMTP id 60E943A69EE; Thu, 29 Jul 2010 02:20:54 -0700 (PDT) Received: from kyzyl.cablelabs.com (kyzyl [10.253.0.7]) by ondar.cablelabs.com (8.14.4/8.14.4) with ESMTP id o6T9LH6o005135; Thu, 29 Jul 2010 03:21:17 -0600 Received: from srvxchg.cablelabs.com (10.5.0.15) by kyzyl.cablelabs.com (F-Secure/fsigk_smtp/303/kyzyl.cablelabs.com); Thu, 29 Jul 2010 03:21:17 -0700 (MST) X-Virus-Status: clean(F-Secure/fsigk_smtp/303/kyzyl.cablelabs.com) Received: from srvxchg.cablelabs.com ([10.5.0.15]) by srvxchg ([10.5.0.15]) with mapi; Thu, 29 Jul 2010 03:21:17 -0600 From: Daryl Malas To: "pmol@ietf.org" Date: Thu, 29 Jul 2010 03:21:09 -0600 Thread-Topic: [PMOL] SIP Perf Telephony Metrics: SER and SEER Updates Thread-Index: Acsu/2FvroY5LByvTPqBebFJWs5sOg== Message-ID: <114DAD31379DFA438C0A2E39B3B8AF5D01422BEE03@srvxchg> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Approved: ondar Cc: "rai@ietf.org" , Robert Sparks Subject: [PMOL] SIP Perf Telephony Metrics: SER and SEER Updates X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Jul 2010 09:20:56 -0000 All, During the IESG review process, there were some comments about the response= codes used in the SER and SEER metrics. I have updated the text in these = metrics and included the updates below. Please review these and provide an= y feedback. Specifically, are the response codes and their purpose clear, = and are the right set of response codes for the purposes of these metrics. = If it is unclear the purpose of these metrics, please let me know. Regards, Daryl ---------------------- 4.6. Session Establishment Ratio (SER) This metric is used to detect the ability of a terminating UA or downstream proxy to successfully establish sessions per new session INVITE requests. SER is defined as the number of new session INVITE requests resulting in a 200 OK response, to the total number of attempted INVITE requests less the total number of attempted INVITE requests less INVITE requests resulting in a 3XX, 401, 402, and 407 response. The 3XX, 401, 402 and 407 response codes were chosen, because they indicate an acceptable recoverable UA effect without the interaction of an individual user of the UA. This metric is similar to Answer Seizure Ratio (ASR) defined in [E.411]. It is measured at the originating UA only. The output value of this metric is numerical and SHOULD be adjusted to indicate a percentage of successfully established sessions. The SER is calculated using the following formula metric. In order to simplify the formula, the following variable is used to summarize multiple SIP responses: a =3D 3XX, 401, 402, and 407 # of INVITE Requests w/ associated 200 OK SER =3D --------------------------------------------------------- x 100 (Total # of INVITE Requests)-(# of INVITE Requests w/ 'a' Response) The following message exchange provides an example of identifiable events necessary for inputs in determining session establishment as described above: UA1 UA2 | | |INVITE | +----------->|------------------>| | | 180| | |<------------------| Session Established | | | | | | | 200| +----------->|<------------------| | | The following is an example call flow including a SIP 302 Redirect response. UA1 UA2 UA3 | | | +----------->|INVITE | | | |------------------>| | INVITE w/ 'a' Response | | | | | 302| | +----------->|<------------------| | | | | |INVITE | +----------->|-------------------------------------->| | | | | | 180| Session Established |<--------------------------------------| | | | | | 200| +----------->|<--------------------------------------| | | 4.7. Session Establishment Effectiveness Ratio (SEER) This metric is complimentary to SER, but is intended to exclude the potential effects of the target UA from the metric. SER is defined as the number of INVITE requests resulting in a 200 OK response and INVITE requests resulting in a 480, 486, or 600; less the total number of attempted INVITE requests less INVITE requests resulting in a 3XX, 401, 402, and 407 response. The response codes 480, 486 and 600 were chosen, because they clearly indicate the effect of an individual user of the UA. It is possible an individual user could cause a negative effect on the UA. For example, they may have misconfigured the UA causing a response code not directly related to a SSP, but this cannot be easily determined from an intermediary B2BUA somewhere inbetween the originating and terminating UA. With this in consideration, response codes such as 415 and 420 were not included in the numerator of the metric. If future response codes provide similar effects, they should be considered in addition with this metric. This metric is similar to Network Effectiveness Ratio (NER) defined in [E.411]. It is measured at the originating UA only. The output value of this metric is numerical and SHOULD be adjusted to indicate a percentage of successfully established sessions less common UAS failures. In order to simplify the formula, the following variable is used to summarize multiple SIP responses: a =3D 3XX, 401, 402, and 407 The SEER is calculated using the following formula: # of INVITE Requests w/ associated 200 OK, 480, 486, or 600 SEER =3D -------------------------------------------------------- x 100 (Total # of INVITE Requests)-(# of INVITE Requests w/ 'a' Response) Reference the example flow is Section 4.6. Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 29B3D3A6BE7 for ; Mon, 26 Jul 2010 08:01:04 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.116 X-Spam-Level: X-Spam-Status: No, score=-2.116 tagged_above=-999 required=5 tests=[AWL=-0.118, BAYES_00=-2.599, HTML_MESSAGE=0.001, J_CHICKENPOX_16=0.6] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9fDuqraQ4DeF for ; Mon, 26 Jul 2010 08:00:57 -0700 (PDT) Received: from av-tac-bru.cisco.com (weird-brew.cisco.com [144.254.15.118]) by core3.amsl.com (Postfix) with ESMTP id 074493A6BFF for ; Mon, 26 Jul 2010 08:00:56 -0700 (PDT) X-TACSUNS: Virus Scanned Received: from strange-brew.cisco.com (localhost.cisco.com [127.0.0.1]) by av-tac-bru.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id o6QEFc0x029899; Mon, 26 Jul 2010 16:15:38 +0200 (CEST) Received: from [10.61.97.2] (dhcp-10-61-97-2.cisco.com [10.61.97.2]) by strange-brew.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id o6QEFbvX029893; Mon, 26 Jul 2010 16:15:37 +0200 (CEST) Message-ID: <4C4D9889.40008@cisco.com> Date: Mon, 26 Jul 2010 16:15:37 +0200 From: Benoit Claise User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.1.11) Gecko/20100711 Thunderbird/3.0.6 MIME-Version: 1.0 To: "Schwarz, Albrecht (Albrecht)" References: <201006281417.o5SEHthw022056@alpd052.aldc.att.com> <4C496AA2.1060903@cisco.com> <5F7BCCF5541B7444830A2288ABBEBC961D46913AB3@FRMRSSXCHMBSD2.dc-m.alcatel-lucent.com> In-Reply-To: <5F7BCCF5541B7444830A2288ABBEBC961D46913AB3@FRMRSSXCHMBSD2.dc-m.alcatel-lucent.com> Content-Type: multipart/alternative; boundary="------------000608060607020708050300" Cc: "pmol@ietf.org" Subject: Re: [PMOL] Terminology; RE: Informal pmol lunch session at IETF-78 X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Jul 2010 15:01:04 -0000 This is a multi-part message in MIME format. --------------000608060607020708050300 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit Albrecht, I would prefer not to redo the ITU-T discussion on the distinction between application and protocol, as this is that relevant here. Potentially around "applications transported over IETF-specified protocols". Sure, we could improve this. Note: https://datatracker.ietf.org/doc/draft-ietf-pmol-metrics-framework/ version 4 has been posted. Regards, Benoit. > > Also, there is the famous question, what is an application and what > is a protocol? > Benoit, > you know the answer is dependent on the used definition for that terms. > Thus, the draft is missing a terminology section and correspondent > references. > There are some suited definitions for protocol, but the term > application is more difficult ("there are much more definitions arround"). > You know our discussion in ITU-T SG13 Q.17 on "application > identification" within Y.dpireq: > I'm suggesting different notions of application (in T09-SG13-C-0654R1): > > Terminology: > > · "*Application*" -> "The notion of ,application' within Y.dpireq may > comprise: > > 1. the "*application protocol type*" (as carried in the application > protocol layer), > example: audio application data encoded in "MPEG-1 Audio Layer 3 > (MP3)"; > 2. the *application as a served user instance* of the application > protocol type[1] , e.g."*Audio-over-Packet > application*" > example: VoIP, VoLTE, VoIMS, VoNGN, VoP2P; > 3. the "*provider specific application*" for Audio-over-Packet, > example: 3GPP provider VoIP, Skype VoIP > 4. ... > > ------------------------------------------------------------------------ > > [1] e.g. the application definition like "/a > collection of user tasks which require processing, storage and > communications functions to carry them out/" > > Looks like that pmol is meaning "application protocol" when talking > about "application", right? > E.g., "applications transported over over IETF-specified protocols" > should be more precise "application protocol data transported over ..." > pmol becomes too vague without terminology definitions IMHO. > Regards, > Albrecht > > ------------------------------------------------------------------------ > *From:* pmol-bounces@ietf.org [mailto:pmol-bounces@ietf.org] *On > Behalf Of *Benoit Claise > *Sent:* Freitag, 23. Juli 2010 12:11 > *To:* Romascanu, Dan (Dan) > *Cc:* pmol@ietf.org > *Subject:* Re: [PMOL] Informal pmol lunch session at IETF-78 > > Dear all, > > Slightly against the IETF policy, but in the interest of making > progress, here is an updated version of the PMOL framework. > In this new version, I addressed all open issues, except two: > - Regarding the "Application Performance Metric", there is no > Application Performance Metric definition in E800. So I made my > own. Do you know of any other ITU standard that could have such a > definition? Also, there is the famous question, what is an > application and what is a protocol? I guess we don't want to go > there... ITU solved it in Y.1562 : Framework for higher layer > protocol performance parameters and their measurement, by > expressing the notion of higher-layer protocols, i.e. above TCP/UDP > - To map RFC5706, should we call this document "Guidelines for > Considering New Performance Metric Development"? > > I will post this draft on Monday as, "/The I-D submission tool > will be reopened at midnight, 2010-07-26/" > > Regards, Benoit. >> I would like to make sure that discussing the follow-up of pmol is part >> of the 'other business'. We need to wrap up the long time due documents >> in works and to make a recommendation about similar work will be made in >> the IETF in the future. >> >> Dan >> >> >> >>> -----Original Message----- >>> From:pmol-bounces@ietf.org [mailto:pmol-bounces@ietf.org] On >>> Behalf Of Al Morton >>> Sent: Monday, June 28, 2010 5:18 PM >>> To:pmol@ietf.org >>> Subject: [PMOL] Informal pmol lunch session at IETF-78 >>> >>> Folks, >>> >>> the pmol wg will meet informally for a (bring-your-own) lunch >>> on Friday, July 30, at 11:30 local time. >>> >>> the meeting place will be announced later. >>> >>> the Agenda includes the status of both our current drafts, >>> and any other business. >>> >>> regards, >>> Al >>> pmol co-chair >>> >>> _______________________________________________ >>> PMOL mailing list >>> PMOL@ietf.org >>> https://www.ietf.org/mailman/listinfo/pmol >>> >>> >> _______________________________________________ >> PMOL mailing list >> PMOL@ietf.org >> https://www.ietf.org/mailman/listinfo/pmol >> > --------------000608060607020708050300 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Albrecht,

I would prefer not to redo the ITU-T discussion on the distinction between application and protocol, as this is that relevant here.
Potentially around "applications transported over IETF-specified protocols". Sure, we could improve this.

Note: https://datatracker.ietf.org/doc/draft-ietf-pmol-metrics-framework/ version 4 has been posted.

Regards, Benoit.
> Also, there is the famous question, what is an application and what is a protocol?
 
Benoit,
you know the answer is dependent on the used definition for that terms.
Thus, the draft is missing a terminology section and correspondent references.
There are some suited definitions for protocol, but the term application is more difficult ("there are much more definitions arround").
 
You know our discussion in ITU-T SG13 Q.17 on "application identification" within Y.dpireq:
I'm suggesting different notions of application (in T09-SG13-C-0654R1):

Terminology:

·        “Application” -> „The notion of ‚application’ within Y.dpireq may comprise:

  1. the “application protocol type” (as carried in the application protocol layer),
    example: audio application data encoded in “MPEG-1 Audio Layer 3 (MP3)”;
  2. the application as a served user instance of the application protocol type[1], e.g.“Audio-over-Packet application”
    example: VoIP, VoLTE, VoIMS, VoNGN, VoP2P;
  3. the “provider specific application” for Audio-over-Packet,
    example: 3GPP provider VoIP, Skype VoIP
  4. …

[1]   e.g. the application definition like “a collection of user tasks which require processing, storage and communications functions to carry them out“

 
 
Looks like that pmol is meaning "application protocol" when talking about "application", right?
 
E.g., "applications transported over over IETF-specified protocols" should be more precise "application protocol data transported over ..."
 
pmol becomes too vague without terminology definitions IMHO.

 
Regards,
Albrecht
 

From: pmol-bounces@ietf.org [mailto:pmol-bounces@ietf.org] On Behalf Of Benoit Claise
Sent: Freitag, 23. Juli 2010 12:11
To: Romascanu, Dan (Dan)
Cc: pmol@ietf.org
Subject: Re: [PMOL] Informal pmol lunch session at IETF-78

Dear all,

Slightly against the IETF policy, but in the interest of making progress, here is an updated version of the PMOL framework.
In this new version, I addressed all open issues, except two:
- Regarding the "Application Performance Metric", there is no Application Performance Metric definition in E800. So I made my own. Do you know of any other ITU standard that could have such a definition? Also, there is the famous question, what is an application and what is a protocol? I guess we don't want to go there... ITU solved it in Y.1562 : Framework for higher layer protocol performance parameters and their measurement, by expressing the notion of higher-layer protocols, i.e. above TCP/UDP
- To map RFC5706, should we call this document "Guidelines for Considering New Performance Metric Development"?

I will post this draft on Monday as, "The I-D submission tool will be reopened at midnight, 2010-07-26"

Regards, Benoit.
I would like to make sure that discussing the follow-up of pmol is part
of the 'other business'. We need to wrap up the long time due documents
in works and to make a recommendation about similar work will be made in
the IETF in the future. 

Dan


  
-----Original Message-----
From: pmol-bounces@ietf.org [mailto:pmol-bounces@ietf.org] On 
Behalf Of Al Morton
Sent: Monday, June 28, 2010 5:18 PM
To: pmol@ietf.org
Subject: [PMOL] Informal pmol lunch session at IETF-78

Folks,

the pmol wg will meet informally for a (bring-your-own) lunch 
on Friday, July 30, at 11:30 local time.

the meeting place will be announced later.

the Agenda includes the status of both our current drafts, 
and any other business.

regards,
Al
pmol co-chair

_______________________________________________
PMOL mailing list
PMOL@ietf.org
https://www.ietf.org/mailman/listinfo/pmol

    
_______________________________________________
PMOL mailing list
PMOL@ietf.org
https://www.ietf.org/mailman/listinfo/pmol


--------------000608060607020708050300-- Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 2D1F13A687C for ; Mon, 26 Jul 2010 07:34:02 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.437 X-Spam-Level: X-Spam-Status: No, score=-2.437 tagged_above=-999 required=5 tests=[AWL=-0.789, BAYES_00=-2.599, HELO_EQ_FR=0.35, HTML_MESSAGE=0.001, J_CHICKENPOX_16=0.6] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CWphcNBI+AKq for ; Mon, 26 Jul 2010 07:33:45 -0700 (PDT) Received: from smail2.alcatel.fr (smail2.alcatel.fr [64.208.49.57]) by core3.amsl.com (Postfix) with ESMTP id 165633A6A83 for ; Mon, 26 Jul 2010 07:33:42 -0700 (PDT) Received: from FRMRSSXCHHUB02.dc-m.alcatel-lucent.com (FRMRSSXCHHUB02.dc-m.alcatel-lucent.com [135.120.45.62]) by smail2.alcatel.fr (8.14.3/8.14.3/ICT) with ESMTP id o6QEXgMT028263 (version=TLSv1/SSLv3 cipher=RC4-MD5 bits=128 verify=NOT); Mon, 26 Jul 2010 16:33:57 +0200 Received: from FRMRSSXCHMBSD2.dc-m.alcatel-lucent.com ([135.120.45.52]) by FRMRSSXCHHUB02.dc-m.alcatel-lucent.com ([135.120.45.62]) with mapi; Mon, 26 Jul 2010 16:33:45 +0200 From: "Schwarz, Albrecht (Albrecht)" To: Benoit Claise Date: Mon, 26 Jul 2010 16:34:00 +0200 Thread-Topic: Terminology; RE: [PMOL] Informal pmol lunch session at IETF-78 Thread-Index: AcsszQxSujTan0oUT2Wc2f3IRAgSJwAAYAPQ Message-ID: <5F7BCCF5541B7444830A2288ABBEBC961D46913C5D@FRMRSSXCHMBSD2.dc-m.alcatel-lucent.com> References: <201006281417.o5SEHthw022056@alpd052.aldc.att.com> <4C496AA2.1060903@cisco.com> <5F7BCCF5541B7444830A2288ABBEBC961D46913AB3@FRMRSSXCHMBSD2.dc-m.alcatel-lucent.com> <4C4D9889.40008@cisco.com> In-Reply-To: <4C4D9889.40008@cisco.com> Accept-Language: de-DE, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: de-DE, en-US Content-Type: multipart/alternative; boundary="_000_5F7BCCF5541B7444830A2288ABBEBC961D46913C5DFRMRSSXCHMBSD_" MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.64 on 155.132.188.80 Cc: "pmol@ietf.org" Subject: Re: [PMOL] Terminology; RE: Informal pmol lunch session at IETF-78 X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Jul 2010 14:34:13 -0000 --_000_5F7BCCF5541B7444830A2288ABBEBC961D46913C5DFRMRSSXCHMBSD_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Benoit, that's ok for me. Finally the crucial point is a precise, unambiguous defin= ition of the performance metric(s) itself. Regards, Albrecht Perhaps a side note: Example 1: RTCP XR QoE metrics might be an example for the "application as = served user instance": http://tools.ietf.org/html/draft-ietf-avt-rtcp-xr-qoe-00 Example 2: Then there are performance metrics arround the H.264 video decoder, thus pe= rformance metrics related directly to an application protocol type. ________________________________ From: Benoit Claise [mailto:bclaise@cisco.com] Sent: Montag, 26. Juli 2010 16:16 To: Schwarz, Albrecht (Albrecht) Cc: Romascanu, Dan (Dan); pmol@ietf.org Subject: Re: Terminology; RE: [PMOL] Informal pmol lunch session at IETF-78 Albrecht, I would prefer not to redo the ITU-T discussion on the distinction between = application and protocol, as this is that relevant here. Potentially around "applications transported over IETF-specified protocols"= . Sure, we could improve this. Note: https://datatracker.ietf.org/doc/draft-ietf-pmol-metrics-framework/ v= ersion 4 has been posted. Regards, Benoit. > Also, there is the famous question, what is an application and what is a = protocol? Benoit, you know the answer is dependent on the used definition for that terms. Thus, the draft is missing a terminology section and correspondent referenc= es. There are some suited definitions for protocol, but the term application is= more difficult ("there are much more definitions arround"). You know our discussion in ITU-T SG13 Q.17 on "application identification" = within Y.dpireq: I'm suggesting different notions of application (in T09-SG13-C-0654R1): Terminology: * "Application" -> "The notion of 'application' within Y.dpireq may = comprise: 1. the "application protocol type" (as carried in the application protoco= l layer), example: audio application data encoded in "MPEG-1 Audio Layer 3 (MP3)"; 2. the application as a served user instance of the application protocol = type[1], e.g."Audio-over-Packet application" example: VoIP, VoLTE, VoIMS, VoNGN, VoP2P; 3. the "provider specific application" for Audio-over-Packet, example: 3GPP provider VoIP, Skype VoIP 4. ... ________________________________ [1] e.g. the application definition like "a colle= ction of user tasks which require processing, storage and communications fu= nctions to carry them out" Looks like that pmol is meaning "application protocol" when talking about "= application", right? E.g., "applications transported over over IETF-specified protocols" should = be more precise "application protocol data transported over ..." pmol becomes too vague without terminology definitions IMHO. Regards, Albrecht ________________________________ From: pmol-bounces@ietf.org [mailto:pmol-boun= ces@ietf.org] On Behalf Of Benoit Claise Sent: Freitag, 23. Juli 2010 12:11 To: Romascanu, Dan (Dan) Cc: pmol@ietf.org Subject: Re: [PMOL] Informal pmol lunch session at IETF-78 Dear all, Slightly against the IETF policy, but in the interest of making progress, h= ere is an updated version of the PMOL framework. In this new version, I addressed all open issues, except two: - Regarding the "Application Performance Metric", there is no Application P= erformance Metric definition in E800. So I made my own. Do you know of any = other ITU standard that could have such a definition? Also, there is the fa= mous question, what is an application and what is a protocol? I guess we do= n't want to go there... ITU solved it in Y.1562 : Framework for higher laye= r protocol performance parameters and their measurement, by expressing the = notion of higher-layer protocols, i.e. above TCP/UDP - To map RFC5706, should we call this document "Guidelines for Considering = New Performance Metric Development"? I will post this draft on Monday as, "The I-D submission tool will be reope= ned at midnight, 2010-07-26" Regards, Benoit. I would like to make sure that discussing the follow-up of pmol is part of the 'other business'. We need to wrap up the long time due documents in works and to make a recommendation about similar work will be made in the IETF in the future. Dan -----Original Message----- From: pmol-bounces@ietf.org [mailto:pmol-boun= ces@ietf.org] On Behalf Of Al Morton Sent: Monday, June 28, 2010 5:18 PM To: pmol@ietf.org Subject: [PMOL] Informal pmol lunch session at IETF-78 Folks, the pmol wg will meet informally for a (bring-your-own) lunch on Friday, July 30, at 11:30 local time. the meeting place will be announced later. the Agenda includes the status of both our current drafts, and any other business. regards, Al pmol co-chair _______________________________________________ PMOL mailing list PMOL@ietf.org https://www.ietf.org/mailman/listinfo/pmol _______________________________________________ PMOL mailing list PMOL@ietf.org https://www.ietf.org/mailman/listinfo/pmol --_000_5F7BCCF5541B7444830A2288ABBEBC961D46913C5DFRMRSSXCHMBSD_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
Benoit,
that's ok for me. Finally the crucial point is a p= recise,=20 unambiguous definition of the performance metric(s) itself.
 
Regards,
Albrecht
 
Perhaps a side note:
Example 1: RTCP XR QoE metrics might be an example= for the=20 "application as served user instance":
http://to= ols.ietf.org/html/draft-ietf-avt-rtcp-xr-qoe-00
 
Example 2:
Then there are performance metrics arround the H.2= 64 video=20 decoder, thus performance metrics related directly to an application protoc= ol=20 type.
 

From: Benoit Claise [mailto:bclaise@c= isco.com]=20
Sent: Montag, 26. Juli 2010 16:16
To: Schwarz, Albre= cht=20 (Albrecht)
Cc: Romascanu, Dan (Dan);=20 pmol@ietf.org
Subject: Re: Terminology; RE: [PMOL] Informal pmo= l=20 lunch session at IETF-78

Albrecht,

I would prefer not to redo the ITU-T discuss= ion=20 on the distinction between application and protocol, as this is that rele= vant=20 here.
Potentially around "applications transported=20 over=20 IETF-specified protocols". Sure, we could improve this.

Not= e: https://datatracker.ietf.org/doc/draft-ietf-pmol-metrics-framework/= =20 version 4 has been posted.

Regards,=20 Benoit.
> Also, there is the famous question, what is an application and= what=20 is a protocol?
 
Benoit,
you know the answer is d= ependent on=20 the used definition for that terms.
Thus, the draft is missi= ng a=20 terminology section and correspondent references.
There are some suited de= finitions=20 for protocol, but the term application is more difficult ("there are mu= ch=20 more definitions arround").
&nb= sp;
You know our discussion = in ITU-T=20 SG13 Q.17 on "application identification" within=20 Y.dpireq:
I'm suggesting different= notions of=20 application (in T09-SG13-C-0654R1):

<= FONT=20 size=3D3>Terminology:

= ·       = ;=20 = “Application” -> „The notion of ‚application’ within Y.dp= ireq may=20 comprise:

  1. the=20 “application protocol type” (as carried in the app= lication protocol=20 layer),
    example: audio application data encoded in “MPEG-1 A= udio Layer=20 3 (MP3)”;    =20
  2. the application = as a served=20 user instance of the application protocol type[1],=20 e.g.“Audio-over-Packet application”
    example: V= oIP, VoLTE,=20 VoIMS, VoNGN, VoP2P;    =20
  3. the = 220;provider=20 specific application” for Audio-over-Packet,
    example: 3G= PP provider=20 VoIP, Skype VoIP    =20
  4. …

<= SPAN=20 class=3DMsoFootnoteReference>[1]=20   e.g. the application definition like “a co= llection of=20 user tasks which require processing, storage and communications functio= ns to=20 carry them=20 out“

&nb= sp;
&nb= sp;
Looks like that pmol is = meaning=20 "application protocol" when talking about "application",=20 right?
&nb= sp;
E.g., "applications transported=20 over over IETF-specified=20 protocols" should be more precise "application protocol data transported over= =20 ..."
&nb= sp;
pmol becomes too vague without terminology=20 definitions IMHO.

 
Regards,
Albrecht
 

From: pmol-bounces@ietf.org [mailto:pmol-bounces@ietf.org]=20 On Behalf Of Benoit Claise
Sent: Freitag, 23. Juli 2= 010=20 12:11
To: Romascanu, Dan (Dan)
Cc: pmol@ietf.org
Subject: Re= :=20 [PMOL] Informal pmol lunch session at IETF-78

Dea= r=20 all,

Slightly against the IETF policy, but in the interest of= =20 making progress, here is an updated version of the PMOL framework.In=20 this new version, I addressed all open issues, except two:
- Regar= ding=20 the "Application Performance Metric", there is no Application Perform= ance=20 Metric definition in E800. So I made my own. Do you know of any other= ITU=20 standard that could have such a definition? Also, there is the famous= =20 question, what is an application and what is a protocol? I guess we d= on't=20 want to go there... ITU solved it in Y.1562 : Framework for higher la= yer=20 protocol performance parameters and their measurement, by expressing = the=20 notion of higher-layer protocols, i.e. above TCP/UDP
- To map RFC5= 706,=20 should we call this document "Guidelines for Considering New Performa= nce=20 Metric Development"?

I will post this draft on Monday as, "= The=20 I-D submission tool will be reopened at midnight,=20 2010-07-26"

Regards, Benoit.
I would like to make sure that discussin=
g the follow-up of pmol is part
of the 'other business'. We need to wrap up the long time due documents
in works and to make a recommendation about similar work will be made in
the IETF in the future.=20

Dan


  
-----Original Message-----
From: pmol-bounces@ietf.org [mailto:pmol-bounces@ietf.org] On=20
Behalf Of Al Morton
Sent: Monday, June 28, 2010 5:18 PM
To: pmol@ietf.org
Subject: [PMOL] Informal pmol lunch session at IETF-78

Folks,

the pmol wg will meet informally for a (bring-your-own) lunch=20
on Friday, July 30, at 11:30 local time.

the meeting place will be announced later.

the Agenda includes the status of both our current drafts,=20
and any other business.

regards,
Al
pmol co-chair

_______________________________________________
PMOL mailing list
PMOL@ietf.org
https://www.ietf.org/mailman/listinfo/pm=
ol

    
_____________________________________=
__________
PMOL mailing list
PMOL@ietf.org
https://www.ietf.org/mailman/listinfo/pm=
ol


= --_000_5F7BCCF5541B7444830A2288ABBEBC961D46913C5DFRMRSSXCHMBSD_-- Return-Path: X-Original-To: pmol@ietf.org Delivered-To: pmol@core3.amsl.com Received: by core3.amsl.com (Postfix, from userid 0) id 15F903A6BF6; Mon, 26 Jul 2010 06:30:02 -0700 (PDT) From: Internet-Drafts@ietf.org To: i-d-announce@ietf.org Content-Type: Multipart/Mixed; Boundary="NextPart" Mime-Version: 1.0 Message-Id: <20100726133004.15F903A6BF6@core3.amsl.com> Date: Mon, 26 Jul 2010 06:30:02 -0700 (PDT) Cc: pmol@ietf.org Subject: [PMOL] I-D Action:draft-ietf-pmol-metrics-framework-04.txt X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Jul 2010 13:30:04 -0000 --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the Performance Metrics for Other Layers Working Group of the IETF. Title : Framework for Performance Metric Development Author(s) : A. Clark, B. Claise Filename : draft-ietf-pmol-metrics-framework-04.txt Pages : 22 Date : 2010-07-26 This document describes a framework and a process for developing performance metrics of protocols and applications transported over over IETF-specified protocols, and that can be used to characterize traffic on live networks and services. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-pmol-metrics-framework-04.txt Internet-Drafts are also available by anonymous FTP at: ftp://ftp.ietf.org/internet-drafts/ Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --NextPart Content-Type: Message/External-body; name="draft-ietf-pmol-metrics-framework-04.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <2010-07-26062731.I-D@ietf.org> --NextPart-- Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id C07C83A6407 for ; Fri, 23 Jul 2010 08:05:19 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -4.264 X-Spam-Level: X-Spam-Status: No, score=-4.264 tagged_above=-999 required=5 tests=[AWL=1.384, BAYES_00=-2.599, HELO_EQ_FR=0.35, HTML_MESSAGE=0.001, J_CHICKENPOX_16=0.6, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EDY1g0ZDDXCY for ; Fri, 23 Jul 2010 08:05:12 -0700 (PDT) Received: from smail5.alcatel.fr (smail5.alcatel.fr [64.208.49.27]) by core3.amsl.com (Postfix) with ESMTP id 75A123A6894 for ; Fri, 23 Jul 2010 08:05:11 -0700 (PDT) Received: from FRMRSSXCHHUB03.dc-m.alcatel-lucent.com (FRMRSSXCHHUB03.dc-m.alcatel-lucent.com [135.120.45.63]) by smail5.alcatel.fr (8.14.3/8.14.3/ICT) with ESMTP id o6NF5K6C000414 (version=TLSv1/SSLv3 cipher=RC4-MD5 bits=128 verify=NOT); Fri, 23 Jul 2010 17:05:20 +0200 Received: from FRMRSSXCHMBSD2.dc-m.alcatel-lucent.com ([135.120.45.52]) by FRMRSSXCHHUB03.dc-m.alcatel-lucent.com ([135.120.45.63]) with mapi; Fri, 23 Jul 2010 17:05:20 +0200 From: "Schwarz, Albrecht (Albrecht)" To: Benoit Claise , "Romascanu, Dan (Dan)" Date: Fri, 23 Jul 2010 17:05:40 +0200 Thread-Topic: Terminology; RE: [PMOL] Informal pmol lunch session at IETF-78 Thread-Index: AcsqUf86jQJbg22nQbiJ/7NLTDLwSAAJTBpA Message-ID: <5F7BCCF5541B7444830A2288ABBEBC961D46913AB3@FRMRSSXCHMBSD2.dc-m.alcatel-lucent.com> References: <201006281417.o5SEHthw022056@alpd052.aldc.att.com> <4C496AA2.1060903@cisco.com> In-Reply-To: <4C496AA2.1060903@cisco.com> Accept-Language: de-DE, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: de-DE, en-US Content-Type: multipart/alternative; boundary="_000_5F7BCCF5541B7444830A2288ABBEBC961D46913AB3FRMRSSXCHMBSD_" MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.64 on 155.132.188.13 Cc: "pmol@ietf.org" Subject: [PMOL] Terminology; RE: Informal pmol lunch session at IETF-78 X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Jul 2010 15:05:19 -0000 --_000_5F7BCCF5541B7444830A2288ABBEBC961D46913AB3FRMRSSXCHMBSD_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable > Also, there is the famous question, what is an application and what is a = protocol? Benoit, you know the answer is dependent on the used definition for that terms. Thus, the draft is missing a terminology section and correspondent referenc= es. There are some suited definitions for protocol, but the term application is= more difficult ("there are much more definitions arround"). You know our discussion in ITU-T SG13 Q.17 on "application identification" = within Y.dpireq: I'm suggesting different notions of application (in T09-SG13-C-0654R1): Terminology: * "Application" -> "The notion of 'application' within Y.dpireq may = comprise: 1. the "application protocol type" (as carried in the application protoco= l layer), example: audio application data encoded in "MPEG-1 Audio Layer 3 (MP3)"; 2. the application as a served user instance of the application protocol = type[1], e.g."Audio-over-Packet application" example: VoIP, VoLTE, VoIMS, VoNGN, VoP2P; 3. the "provider specific application" for Audio-over-Packet, example: 3GPP provider VoIP, Skype VoIP 4. ... ________________________________ [1] e.g. the application definition like "a colle= ction of user tasks which require processing, storage and communications fu= nctions to carry them out" Looks like that pmol is meaning "application protocol" when talking about "= application", right? E.g., "applications transported over over IETF-specified protocols" should = be more precise "application protocol data transported over ..." pmol becomes too vague without terminology definitions IMHO. Regards, Albrecht ________________________________ From: pmol-bounces@ietf.org [mailto:pmol-bounces@ietf.org] On Behalf Of Ben= oit Claise Sent: Freitag, 23. Juli 2010 12:11 To: Romascanu, Dan (Dan) Cc: pmol@ietf.org Subject: Re: [PMOL] Informal pmol lunch session at IETF-78 Dear all, Slightly against the IETF policy, but in the interest of making progress, h= ere is an updated version of the PMOL framework. In this new version, I addressed all open issues, except two: - Regarding the "Application Performance Metric", there is no Application P= erformance Metric definition in E800. So I made my own. Do you know of any = other ITU standard that could have such a definition? Also, there is the fa= mous question, what is an application and what is a protocol? I guess we do= n't want to go there... ITU solved it in Y.1562 : Framework for higher laye= r protocol performance parameters and their measurement, by expressing the = notion of higher-layer protocols, i.e. above TCP/UDP - To map RFC5706, should we call this document "Guidelines for Considering = New Performance Metric Development"? I will post this draft on Monday as, "The I-D submission tool will be reope= ned at midnight, 2010-07-26" Regards, Benoit. I would like to make sure that discussing the follow-up of pmol is part of the 'other business'. We need to wrap up the long time due documents in works and to make a recommendation about similar work will be made in the IETF in the future. Dan -----Original Message----- From: pmol-bounces@ietf.org [mailto:pmol-boun= ces@ietf.org] On Behalf Of Al Morton Sent: Monday, June 28, 2010 5:18 PM To: pmol@ietf.org Subject: [PMOL] Informal pmol lunch session at IETF-78 Folks, the pmol wg will meet informally for a (bring-your-own) lunch on Friday, July 30, at 11:30 local time. the meeting place will be announced later. the Agenda includes the status of both our current drafts, and any other business. regards, Al pmol co-chair _______________________________________________ PMOL mailing list PMOL@ietf.org https://www.ietf.org/mailman/listinfo/pmol _______________________________________________ PMOL mailing list PMOL@ietf.org https://www.ietf.org/mailman/listinfo/pmol --_000_5F7BCCF5541B7444830A2288ABBEBC961D46913AB3FRMRSSXCHMBSD_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
> Also, there is the famous question, what is an application and wha= t is a=20 protocol?
 
Benoit,
you know the answer is dependent on the used defin= ition for=20 that terms.
Thus, the draft is missing a terminology section a= nd=20 correspondent references.
There are some suited definitions for protocol, bu= t the=20 term application is more difficult ("there are much more definitions=20 arround").
 
You know our discussion in ITU-T SG13 Q.17 on "app= lication=20 identification" within Y.dpireq:
I'm suggesting different notions of application (i= n=20 T09-SG13-C-0654R1):

Terminology:

·      &= nbsp;=20 “Application” -> = „The= notion of=20 ‚application’ within Y.dpireq may comprise:

  1. the “application protocol type̶= 1; (as=20 carried in the application protocol layer),
    example: audio application= data=20 encoded in “MPEG-1 Audio Layer 3=20 (MP3)”;
  2. the ap= plication=20 as a served user instance of the application protocol type[1], e.g.“Audio-over-Packet application&= #8221;=20
    example: VoIP, VoLTE, VoIMS, VoNGN,=20 VoP2P;
  3. the “provider specific application&= #8221; for=20 Audio-over-Packet,
    example: 3GPP provider VoIP, Skype=20 VoIP
  4. …

[1]   e.g. the application definition li= ke “a collection of user tasks which requ= ire=20 processing, storage and communications functions to carry them=20 out“

 
 
Looks like that pmol is meaning "application proto= col" when=20 talking about "application", right?
 
E.g., "applications transported=20 over over IETF-sp= ecified=20 protocols" should be more precise "application protocol data transported over=20 ..."
 
pmol becomes too vague without terminology defin= itions=20 IMHO.
 
Regards,
Albrecht
 

From: pmol-bounces@ietf.org=20 [mailto:pmol-bounces@ietf.org] On Behalf Of Benoit=20 Claise
Sent: Freitag, 23. Juli 2010 12:11
To: Romasca= nu,=20 Dan (Dan)
Cc: pmol@ietf.org
Subject: Re: [PMOL] Infor= mal=20 pmol lunch session at IETF-78

Dear all,

Slightly against the IETF policy, but in the= =20 interest of making progress, here is an updated version of the PMOL=20 framework.
In this new version, I addressed all open issues, except=20 two:
- Regarding the "Application Performance Metric", there is no=20 Application Performance Metric definition in E800. So I made my own. Do y= ou=20 know of any other ITU standard that could have such a definition? Also, t= here=20 is the famous question, what is an application and what is a protocol? I = guess=20 we don't want to go there... ITU solved it in Y.1562 : Framework for high= er=20 layer protocol performance parameters and their measurement, by expressin= g the=20 notion of higher-layer protocols, i.e. above TCP/UDP
- To map RFC5706,= =20 should we call this document "Guidelines for Considering New Performance= =20 Metric Development"?

I will post this draft on Monday as, "The = I-D=20 submission tool will be reopened at midnight, 2010-07-26"

Rega= rds,=20 Benoit.
I would like to make sure that discussing th=
e follow-up of pmol is part
of the 'other business'. We need to wrap up the long time due documents
in works and to make a recommendation about similar work will be made in
the IETF in the future.=20

Dan


  
-----Original Message-----
From: pmol-bounces@ietf.org [mailto:pmol-bounces@ietf.org] On=20
Behalf Of Al Morton
Sent: Monday, June 28, 2010 5:18 PM
To: pmol@=
ietf.org
Subject: [PMOL] Informal pmol lunch session at IETF-78

Folks,

the pmol wg will meet informally for a (bring-your-own) lunch=20
on Friday, July 30, at 11:30 local time.

the meeting place will be announced later.

the Agenda includes the status of both our current drafts,=20
and any other business.

regards,
Al
pmol co-chair

_______________________________________________
PMOL mailing list
PMOL@ietf=
.org
https://www.ietf.org/mailman/listinfo/pmol

    
_____________________________________=
__________
PMOL mailing list
PMOL@ietf=
.org
https://www.ietf.org/mailman/listinfo/pmol

--_000_5F7BCCF5541B7444830A2288ABBEBC961D46913AB3FRMRSSXCHMBSD_-- Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 885323A6890 for ; Fri, 23 Jul 2010 03:29:21 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -0.545 X-Spam-Level: X-Spam-Status: No, score=-0.545 tagged_above=-999 required=5 tests=[AWL=-1.787, BAYES_50=0.001, HTML_MESSAGE=0.001, SARE_LWSHORTT=1.24] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rqCRsFjsBO-S for ; Fri, 23 Jul 2010 03:29:18 -0700 (PDT) Received: from av-tac-bru.cisco.com (weird-brew.cisco.com [144.254.15.118]) by core3.amsl.com (Postfix) with ESMTP id 0969B3A6A59 for ; Fri, 23 Jul 2010 03:29:16 -0700 (PDT) X-TACSUNS: Virus Scanned Received: from strange-brew.cisco.com (localhost.cisco.com [127.0.0.1]) by av-tac-bru.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id o6NAAi5x015500; Fri, 23 Jul 2010 12:10:44 +0200 (CEST) Received: from [10.55.43.51] (ams-bclaise-8712.cisco.com [10.55.43.51]) by strange-brew.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id o6NAAgWT015443; Fri, 23 Jul 2010 12:10:43 +0200 (CEST) Message-ID: <4C496AA2.1060903@cisco.com> Date: Fri, 23 Jul 2010 12:10:42 +0200 From: Benoit Claise User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.1.10) Gecko/20100512 Thunderbird/3.0.5 MIME-Version: 1.0 To: "Romascanu, Dan (Dan)" References: <201006281417.o5SEHthw022056@alpd052.aldc.att.com> In-Reply-To: Content-Type: multipart/mixed; boundary="------------090807010301010703070702" Cc: pmol@ietf.org Subject: Re: [PMOL] Informal pmol lunch session at IETF-78 X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Jul 2010 10:29:21 -0000 This is a multi-part message in MIME format. --------------090807010301010703070702 Content-Type: multipart/alternative; boundary="------------000108060606030809040804" --------------000108060606030809040804 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Dear all, Slightly against the IETF policy, but in the interest of making progress, here is an updated version of the PMOL framework. In this new version, I addressed all open issues, except two: - Regarding the "Application Performance Metric", there is no Application Performance Metric definition in E800. So I made my own. Do you know of any other ITU standard that could have such a definition? Also, there is the famous question, what is an application and what is a protocol? I guess we don't want to go there... ITU solved it in Y.1562 : Framework for higher layer protocol performance parameters and their measurement, by expressing the notion of higher-layer protocols, i.e. above TCP/UDP - To map RFC5706, should we call this document "Guidelines for Considering New Performance Metric Development"? I will post this draft on Monday as, "/The I-D submission tool will be reopened at midnight, 2010-07-26/" Regards, Benoit. > I would like to make sure that discussing the follow-up of pmol is part > of the 'other business'. We need to wrap up the long time due documents > in works and to make a recommendation about similar work will be made in > the IETF in the future. > > Dan > > > >> -----Original Message----- >> From: pmol-bounces@ietf.org [mailto:pmol-bounces@ietf.org] On >> Behalf Of Al Morton >> Sent: Monday, June 28, 2010 5:18 PM >> To: pmol@ietf.org >> Subject: [PMOL] Informal pmol lunch session at IETF-78 >> >> Folks, >> >> the pmol wg will meet informally for a (bring-your-own) lunch >> on Friday, July 30, at 11:30 local time. >> >> the meeting place will be announced later. >> >> the Agenda includes the status of both our current drafts, >> and any other business. >> >> regards, >> Al >> pmol co-chair >> >> _______________________________________________ >> PMOL mailing list >> PMOL@ietf.org >> https://www.ietf.org/mailman/listinfo/pmol >> >> > _______________________________________________ > PMOL mailing list > PMOL@ietf.org > https://www.ietf.org/mailman/listinfo/pmol > --------------000108060606030809040804 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Dear all,

Slightly against the IETF policy, but in the interest of making progress, here is an updated version of the PMOL framework.
In this new version, I addressed all open issues, except two:
- Regarding the "Application Performance Metric", there is no Application Performance Metric definition in E800. So I made my own. Do you know of any other ITU standard that could have such a definition? Also, there is the famous question, what is an application and what is a protocol? I guess we don't want to go there... ITU solved it in Y.1562 : Framework for higher layer protocol performance parameters and their measurement, by expressing the notion of higher-layer protocols, i.e. above TCP/UDP
- To map RFC5706, should we call this document "Guidelines for Considering New Performance Metric Development"?

I will post this draft on Monday as, "The I-D submission tool will be reopened at midnight, 2010-07-26"

Regards, Benoit.
I would like to make sure that discussing the follow-up of pmol is part
of the 'other business'. We need to wrap up the long time due documents
in works and to make a recommendation about similar work will be made in
the IETF in the future. 

Dan


  
-----Original Message-----
From: pmol-bounces@ietf.org [mailto:pmol-bounces@ietf.org] On 
Behalf Of Al Morton
Sent: Monday, June 28, 2010 5:18 PM
To: pmol@ietf.org
Subject: [PMOL] Informal pmol lunch session at IETF-78

Folks,

the pmol wg will meet informally for a (bring-your-own) lunch 
on Friday, July 30, at 11:30 local time.

the meeting place will be announced later.

the Agenda includes the status of both our current drafts, 
and any other business.

regards,
Al
pmol co-chair

_______________________________________________
PMOL mailing list
PMOL@ietf.org
https://www.ietf.org/mailman/listinfo/pmol

    
_______________________________________________
PMOL mailing list
PMOL@ietf.org
https://www.ietf.org/mailman/listinfo/pmol

--------------000108060606030809040804-- --------------090807010301010703070702 Content-Type: text/plain; name="draft-ietf-pmol-metrics-framework-04.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="draft-ietf-pmol-metrics-framework-04.txt" Network Working Group A. Clark Internet-Draft Telchemy Incorporated Intended status: BCP B. Claise Expires: January 24, 2011 Cisco Systems, Inc. July 23, 2010 Framework for Performance Metric Development draft-ietf-pmol-metrics-framework-04 Abstract This document describes a framework and a process for developing performance metrics of protocols and applications transported over over IETF-specified protocols, and that can be used to characterize traffic on live networks and services. Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. Status of this Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on January 24, 2011. Copyright Notice Copyright (c) 2010 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of Clark & Claise Expires January 24, 2011 [Page 1] Internet-Draft Perf. Metric Framework July 2010 publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. This document may contain material from IETF Documents or IETF Contributions published or made publicly available before November 10, 2008. The person(s) controlling the copyright in some of this material may not have granted the IETF Trust the right to allow modifications of such material outside the IETF Standards Process. Without obtaining an adequate license from the person(s) controlling the copyright in such materials, this document may not be modified outside the IETF Standards Process, and derivative works of it may not be created outside the IETF Standards Process, except to format it for publication as an RFC or to translate it into languages other than English. Clark & Claise Expires January 24, 2011 [Page 2] Internet-Draft Perf. Metric Framework July 2010 Table of Contents 1. TO DO . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.1. Background and Motivation . . . . . . . . . . . . . . . . 4 2.2. Organization of this document . . . . . . . . . . . . . . 5 3. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 6 3.1. Performance Metrics Entity (PM Entity) . . . . . . . . . . 6 3.2. Quality of Service . . . . . . . . . . . . . . . . . . . . 6 3.3. Quality of Experience . . . . . . . . . . . . . . . . . . 6 3.4. Application Performance Metric . . . . . . . . . . . . . . 7 4. Purpose and Scope . . . . . . . . . . . . . . . . . . . . . . 7 5. Relationship between QoS, QoE and Application Performance Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 6. Metrics Development . . . . . . . . . . . . . . . . . . . . . 8 6.1. Identifying and Categorizing the Audience . . . . . . . . 8 6.2. Definitions of a Metric . . . . . . . . . . . . . . . . . 9 6.3. Computed Metrics . . . . . . . . . . . . . . . . . . . . . 10 6.3.1. Composed Metrics . . . . . . . . . . . . . . . . . . . 10 6.3.2. Index . . . . . . . . . . . . . . . . . . . . . . . . 10 6.4. Metric Specification . . . . . . . . . . . . . . . . . . . 11 6.4.1. Outline . . . . . . . . . . . . . . . . . . . . . . . 11 6.4.2. Normative parts of metric definition . . . . . . . . . 11 6.4.3. Informative parts of metric definition . . . . . . . . 12 6.4.4. Performance Metric Definition Template . . . . . . . . 13 6.4.5. Example: Burst Packet Loss Frequency . . . . . . . . . 14 6.5. Dependencies . . . . . . . . . . . . . . . . . . . . . . . 15 6.5.1. Timing accuracy . . . . . . . . . . . . . . . . . . . 15 6.5.2. Dependencies of metric definitions on related events or metrics . . . . . . . . . . . . . . . . . . 15 6.5.3. Relationship between application performance and lower layer metrics . . . . . . . . . . . . . . . . . 15 6.5.4. Middlebox presence . . . . . . . . . . . . . . . . . . 16 6.6. Organization of Results . . . . . . . . . . . . . . . . . 16 6.7. Parameters, the variables of a metric . . . . . . . . . . 16 7. Performance Metric Development Process . . . . . . . . . . . . 17 7.1. New Proposals for Metrics . . . . . . . . . . . . . . . . 17 7.2. Reviewing Metrics . . . . . . . . . . . . . . . . . . . . 17 7.3. Proposal Approval . . . . . . . . . . . . . . . . . . . . 18 7.4. PM Entity Interaction with other WGs . . . . . . . . . . . 19 7.5. Standards Track Performance Metrics . . . . . . . . . . . 19 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 19 9. Security Considerations . . . . . . . . . . . . . . . . . . . 19 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 20 11. References . . . . . . . . . . . . . . . . . . . . . . . . . . 20 11.1. Normative References . . . . . . . . . . . . . . . . . . . 20 11.2. Informative References . . . . . . . . . . . . . . . . . . 20 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 22 Clark & Claise Expires January 24, 2011 [Page 3] Internet-Draft Perf. Metric Framework July 2010 1. TO DO o Regarding the "Application Performance Metric", there is no Application Performance Metric definition in E800. So I made my own. Do you know of any other ITU standard that could have such a definition? Also, there is the fameous question, what is an application and what is a protocol? I guess we don't want to go there... ITU solved it in Y.1562 : Framework for higher layer protocol performance parameters and their measurement, by expressing the notion of higher-layer protocols, i.e. above TCP/ UDP o To map RFC5706, should we call this document "Guidelines for Considering New Performance Metric Development" 2. Introduction Many networking technologies, applications, or services, are distributed in nature, and their performance may be impacted by IP impairments, server capacity, congestion and other factors. It is important to measure the performance of applications and services to ensure that quality objectives are being met and to support problem diagnosis. Standardized metrics help to ensure that performance measurement is implemented consistently and to facilitate interpretation and comparison. There are at least three phases in the development of performance standards. They are: 1. Definition of a performance metric and its units of measure 2. Specification of a method of measurement 3. Specification of the reporting format During the development of metrics, it is often useful to define performance objectives and expected value ranges. However, this is not defined as part of the metric specification. 2.1. Background and Motivation Although the IETF has two active Working Groups (WGs) dedicated to the development of performance metrics, they each have strict limitations in their charters: - The Benchmarking Methodology WG has addressed a range of networking technologies and protocols in their long history (such as IEEE 802.3, Clark & Claise Expires January 24, 2011 [Page 4] Internet-Draft Perf. Metric Framework July 2010 ATM, Frame Relay, and Routing Protocols), but the charter strictly limits their performance characterizations to the laboratory environment. - The IP Performance Metrics (IPPM) WG has developed a set of standard metrics that can be applied to the quality, performance, and reliability of Internet data delivery services. The IPPM metrics development is applicable to live IP networks, but it is specifically prohibited from developing metrics that characterize traffic at upper layers, such as a VoIP stream. A BOF held at IETF-69 introduced the IETF community to the possibility of a generalized activity to define standardized performance metrics. The existence of a growing list of Internet- Drafts on performance metrics (with community interest in development, but in un-chartered areas) illustrates the need for additional performance work. The majority of people present at the BOF supported the proposition that IETF should be working in these areas, and no one objected to any of the proposals. The IETF does have current and completed activities related to the reporting of application performance metrics: for example the Real- time Application Quality-of-Service Monitoring (RAQMON) Framework RFC 4710 [RFC4710], which extends the remote network monitoring (RMON) family of specifications to allow real-time quality-of-service (QoS) monitoring of various applications that run on devices such as IP phones, pagers, Instant Messaging clients, mobile phones, and various other handheld computing devices. The IETF is also actively involved in the development of reliable transport protocols which would affect the relationship between IP performance and application performance. Thus there is a gap in the currently chartered coverage of IETF WGs: development of performance metrics for protocols above and below the IP-layer that can be used to characterize performance on live networks. This document refers to a Performance Metrics Entity, or PM Entity, which may in future be a WG or directorate or a combination of these two, whose goal is to coordinate the performance metric development at the IETF. 2.2. Organization of this document This document is divided in two major sections beyond the "Purpose and Scope" section. The first is a definition and description of a performance metric and its key aspects. The second defines a process Clark & Claise Expires January 24, 2011 [Page 5] Internet-Draft Perf. Metric Framework July 2010 to develop these metrics that is applicable to the IETF environment. 3. Terminology 3.1. Performance Metrics Entity (PM Entity) The Performance Metrics Entity, or PM Entity, which may in future be a WG or directorate or a combination of these two, must coordinate the performance metric development at the IETF. Similar to The "Guidelines for Considering Operations and Management of New Protocols and Protocol Extensions" RFC 5706 [RFC5706], which is the reference document for the IETF Operations Directorate, this document should be consulted as part of the new performance metric review. The PM Entity should be composed of experts in the performance community, potentially selected from the IPPM, BMWG, and PMOL WGs. 3.2. Quality of Service The Quality of Service (QoS) is defined similarly to the ITU "QoS experienced/perceived by customer/user (QoE)" E800 [E800], i.e.: "Totality of characteristics of a telecommunications service that bear on its ability to satisfy stated and implied needs of the user of the service." 3.3. Quality of Experience The Quality of Experience (QoE) is defined similarly to the ITU "QoS experienced/perceived by customer/user (QoE)" E800 [E800], i.e.: "a statement expressing the level of quality that customers/users believe they have experienced." NOTE 1 - The level of QoS experienced and/or perceived by the customer/user may be expressed by an opinion rating. NOTE 2 - QoE has two main man components: quantitative and qualitative. The quantitative component can be influenced by the complete end-to-end system effects (network infrastructure). NOTE 3 - The qualitative component can be influenced by user expectations, ambient conditions, psychological factors, application context, etc. NOTE 4 - QoE may also be considered as QoS delivered, received, and interpreted by a user with the pertinent qualitative factors influencing his/her perception of the service. Clark & Claise Expires January 24, 2011 [Page 6] Internet-Draft Perf. Metric Framework July 2010 3.4. Application Performance Metric A measure of performance, specific to an application. For example, the FTP response time for a complete file download, the DNS response time to resolve the IP address, a database loging time, etc.. 4. Purpose and Scope The purpose of this document is to define a framework and a process for developing performance metrics for protocols above and below the IP-layer (such as IP-based applications that operate over reliable or datagram transport protocols), that can be used to characterize traffic on live networks and services. As such, this document will not define any performance metrics. The scope of this document includes the support of metric definition for any protocol developed by the IETF. However this document is not intended to supercede existing working methods within WGs that have existing chartered work in this area. This process is not intended to govern performance metric development in existing IETF WG that are focused on metrics development, such as IPPM and BMWG. However, the framework and guidelines may be useful in these activities, and MAY be applied where appropriate. A typical example is the development of performance metrics to be exported with the IPFIX protocol RFC 5101 [RFC5101], with specific IPFIX information elements RFC 5102 [RFC5102], which would benefit from the framework in this document. The framework in this document applies to performance metrics derived from both active and passive measurements. 5. Relationship between QoS, QoE and Application Performance Metrics Network QoS deals with the network and network protocol performance, while QoE deals with the assessment of a user's experience in a context of a task or a service. As a result, the topic of Application Performance Metrics includes the opportunities to quantify performance at layers between IP and the user. For example, network QoS metrics (packet loss, delay, and delay variation [RFC5481]) can be used to estimate Application Performance Metrics (de-jitter buffer size and RTP-layer packet loss), then combined with other known aspects of a VoIP application (such as codec type) to estimate a Mean Opinion Score (MOS) [P.800]. However, the QoE for a particular VoIP user depends on the specific context, such as a casual conversation, a business conference call, or an emergency Clark & Claise Expires January 24, 2011 [Page 7] Internet-Draft Perf. Metric Framework July 2010 call. Finally, QoS and Application Performance Metrics are quantitative, while QoE is qualitative. Also network QoS and Application Performance Metrics can be directly or indirectly evident to the user, while the QoE is directly evident. 6. Metrics Development This section provides key definitions and qualifications of performance metrics. 6.1. Identifying and Categorizing the Audience Many of the aspects of metric definition and reporting, even the selection or determination of the essential metrics, depend on who will use the results, and for what purpose: in order to properly maintain service quality? or to identify and quantify problems? The question, "How will the results be used?" usually yields important factors to consider when developing performance metrics. All documents defining performance metrics SHOULD identify the primary audience and its associated requirements. The audience can influence both the definition of metrics and the methods of measurement. The key areas of variation between different metric users include: o Suitability of passive measurements of live traffic, or active measurements using dedicated traffic o Measurement in laboratory environment, or on a network of deployed devices o Suitability of passive measurements of live traffic, or active measurements using dedicated traffic o Measurement in laboratory environment, or on a network of deployed devices o Accuracy of the results o Access to measurement points and configuration information o Measurement topology (point-to-point, point-to-multipoint) o Scale of the measurement system Clark & Claise Expires January 24, 2011 [Page 8] Internet-Draft Perf. Metric Framework July 2010 o Measurements conducted on-demand, or continuously o Required reporting formats 6.2. Definitions of a Metric A metric is a measure of an observable behavior of an networking technology, an application, or a service. Most of the time, the metric can be directly measured. However, sometimes, the metric definition is computed: it assumes some implicit or explicit underlying statistical process. In such case, the metric is an estimate of a parameter of this process, assuming that the statistical process closely models the behavior of the system. A metric should serve some defined purpose. This may include the measurement of capacity, quantifying how bad some problem is, measurement of service level, problem diagnosis or location and other such uses. A metric may also be an input to some other process, for example the computation of a composite metric or a model or simulation of a system. Tests of the "usefulness" of a metric include: (i) the degree to which its absence would cause significant loss of information on the behavior or performance of the application or system being measured (ii) the correlation between the performance metric, the QoS G1000 [G1000] and QoE delivered to the user (person or other application) (iii) the degree to which the metric is able to support the identification and location of problems affecting service quality. (iv) the requirement to develop policies (Service Level Agreement, and potentially Service Level Contract) based on the metric. For example, consider a distributed application operating over a network connection that is subject to packet loss. A Packet Loss Rate (PLR) metric is defined as the mean packet loss rate over some time period. If the application performs poorly over network connections with high packet loss rate and always performs well when the packet loss rate is zero then the PLR metric is useful to some degree. Some applications are sensitive to short periods of high loss (bursty loss) and are relatively insensitive to isolated packet loss events; for this type of application there would be very weak correlation between PLR and application performance. A "better" metric would consider both the packet loss rate and the distribution of loss events. If application performance is degraded when the PLR Clark & Claise Expires January 24, 2011 [Page 9] Internet-Draft Perf. Metric Framework July 2010 exceeds some rate then a useful metric may be a measure of the duration and frequency of periods during which the PLR exceeds that rate. 6.3. Computed Metrics 6.3.1. Composed Metrics Some metrics may not be measured directly, but may be composed from base metrics that have been measured. A composed metric is derived from other metrics by applying a deterministic process or function (e.g., a composition function). The process may use metrics that are identical to the metric being composed, or metrics that are dissimilar, or some combination of both types. Usually the base metrics have a limited scope in time or space, and they can be combined to estimate the performance of some larger entities. Some examples of composed metrics and composed metric definitions are: Spatial composition is defined as the composition of metrics of the same type with differing spatial domains [RFC5835] [I-D.ietf-ippm-spatial-composition]. For spatially composed metrics to be meaningful, the spatial domains should be non-overlapping and contiguous, and the composition operation should be mathematically appropriate for the type of metric. Temporal composition is defined as the composition of sets of metrics of the same type with differing time spans [RFC5835]. For temporally composed metrics to be meaningful, the time spans should be non- overlapping and contiguous, and the composition operation should be mathematically appropriate for the type of metric. Temporal aggregation is a summarization of metrics into a smaller number of metrics that relate to the total time span covered by the original metrics. An example would be to compute the minimum, maximum and average values of a series of time sampled values of a metric. In the context of flow records in IP Flow Informatin eXport (IPFIX), the IPFIX Mediation: Framework [I-D.ietf-ipfix-mediators-framework] also discusses some aspects of the temporal and spatial composition. 6.3.2. Index An Index is a metric for which the output value range has been selected for convenience or clarity, and the behavior of which is selected to support ease of understanding (e.g. G.107 R Factor). Clark & Claise Expires January 24, 2011 [Page 10] Internet-Draft Perf. Metric Framework July 2010 The deterministic function for an index is often developed after the index range and behavior have been determined. 6.4. Metric Specification 6.4.1. Outline A metric definition MUST have a normative part that defines what the metric is and how it is measured or computed and SHOULD have an informative part that describes the metric and its application. 6.4.2. Normative parts of metric definition The normative part of a metric definition MUST define at least the following: (i) Metric Name Metric names MUST be unique within the set of metrics being defined and MAY be descriptive. (ii) Metric Description The description MUST explain what the metric is, what is being measured and how this relates to the performance of the system being measured. (iii) Method of Measurement or Caculation This method of measurement or calculation MUST define what is being measured or computed and the specific algorithm to be used. Does the measurement involve active or only passive measurements? Terms such as "average" should be qualified (e.g. running average or average over some interval). Exception cases SHOULD also be defined with the appropriate handling method. For example, there are a number of commonly used metrics related to packet loss; these often don't define the criteria by which a packet is determined to be lost (vs very delayed) or how duplicate packets are handled. For example, if the average packet loss rate during a time interval is reported, and a packet's arrival is delayed from one interval to the next then was it "lost" during the interval during which it should have arrived or should it be counted as received? Some parameters linked to the method MAY also be reported, in order to fully interpret the performance metric. For example, the time interval, the load, the minimum packet loss, etc... (iv) Units of measurement Clark & Claise Expires January 24, 2011 [Page 11] Internet-Draft Perf. Metric Framework July 2010 The units of measurement MUST be clearly stated. (v) Measurement Point(s) If the measurement is specific to a measurement point this SHOULD be defined. The measurement domain MAY also be defined. Specifically, if measurement points are spread across domains, the measurement domain (intra-, inter-) is another factor to consider. In some cases, the measurement requires multiple measurement points: all measurement points SHOULD be defined, including the measurement domain(s). (vi) Measurement timing The acceptable range of timing intervals or sampling intervals for a measurement and the timing accuracy required for such intervals MUST be specified. Short sampling intervals or frequent samples provide a rich source of information that can help to assess application performance but may lead to excessive measurement data. Long measurement or sampling intervals reduce the amount of reported and collected data such that it may be insufficient to understand application performance or service quality insofar as the measured quantity may vary significantly with time. In case of multiple measurement points, the potential requirement for synchronized clocks must be clearly specified. In the specific example of the IP delay variation application metric, the different aspects of synchronized clocks are discussed in [RFC5481]. 6.4.3. Informative parts of metric definition The informative part of a metric specification is intended to support the implementation and use of the metric. This part SHOULD provide the following data: (i) Implementation The implementation description MAY be in the form of text, algorithm or example software. The objective of this part of the metric definition is to assist implementers to achieve a consistent result. (ii) Verification The metric definition SHOULD provide guidance on verification testing. This may be in the form of test vectors, a formal verification test method or informal advice. Clark & Claise Expires January 24, 2011 [Page 12] Internet-Draft Perf. Metric Framework July 2010 (iii) Use and Applications The use and applications description is intended to assist the "user" to understand how, when and where the metric can be applied, and what significance the value range for the metric may have. This MAY include a definition of the "typical" and "abnormal" range of the metric, if this was not apparent from the nature of the metric. For example: (a) it is fairly intuitive that a lower packet loss rate would equate to better performance. However the user may not know the significance of some given packet loss rate, (b) the speech level of a telephone signal is commonly expressed in dBm0. If the user is presented with: Speech level = -7 dBm0 this is not intuitively understandable, unless the user is a telephony expert. If the metric definition explains that the typical range is -18 to -28 dBm0, a value higher than -18 means the signal may be too high (loud) and less than -28 means that the signal may be too low (quiet), it is much easier to interpret the metric. (iv) Reporting Model The reporting model definition is intended to make any relationship between the metric and the reporting model clear. There are often implied relationships between the method of reporting metrics and the metric itself, however these are often not made apparent to the implementor. For example, if the metric is a short term running average packet delay variation (e.g. PPDV as defined in RFC3550) and this value is reported at intervals of 6-10 seconds, the resulting measurement may have limited accuracy when packet delay variation is non-stationary. 6.4.4. Performance Metric Definition Template Normative o Metric Name o Metric Description o Method Clark & Claise Expires January 24, 2011 [Page 13] Internet-Draft Perf. Metric Framework July 2010 o Units of measurement o Measurement Timing Informative o Implementation Guidelines o Verification o Use and Applications o Reporting Model 6.4.5. Example: Burst Packet Loss Frequency The burst packet loss frequency can be observed at different layers. The following example is specific to RTP RFC 3550 [RFC3550]. Metric Name: BurstPacketLossFrequency Metric Description: A burst of packet loss is defined as a longest period starting and ending with lost packets during which no more than Gmin consecutive packets are received. The BurstPacketLossFrequency is defined as the number of bursts of packet loss occurring during a specified time interval (e.g. per minute, per hour, per day). If Gmin is set to 0 then a burst of packet loss would comprise only consecutive lost packets, whereas a Gmin of 16 would define bursts as periods of both lost and received packets (sparse bursts) having a loss rate of greater than 5.9%. Method: Bursts may be detected using the Markov Model algorithm defined in RFC3611. The BurstPacketLossFrequency is calculated by counting the number of burst events within the defined measurement interval. A burst that spans the boundary between two time intervals shall be counted within the later of the two intervals. Units of Measurement: Bursts per time interval (e.g. per second, per hour, per day) Measurement Timing: This metric can be used over a wide range of time intervals. Using time intervals of longer than one hour may prevent the detection of variations in the value of this metric due to time- of-day changes in network load. Timing intervals should not vary in duration by more than +/- 2%. Implementation Guidelines: See RFC3611. Clark & Claise Expires January 24, 2011 [Page 14] Internet-Draft Perf. Metric Framework July 2010 Verification Testing: See Appendix for C code to generate test vectors. Use and Applications: This metric is useful to detect IP network transients that affect the performance of applications such as Voice over IP or IP Video. The value of Gmin may be selected to ensure that bursts correspond to a packet loss rate that would degrade the performance of the application of interest (e.g. 16 for VoIP). Reporting Model: This metric needs to be associated with a defined time interval, which could be defined by fixed intervals or by a sliding window. 6.5. Dependencies 6.5.1. Timing accuracy The accuracy of the timing of a measurement may affect the accuracy of the metric. This may not materially affect a sampled value metric however would affect an interval based metric. Some metrics, for example the number of events per time interval, would be directly affected; for example a 10% variation in time interval would lead directly to a 10% variation in the measured value. Other metrics, such as the average packet loss rate during some time interval, would be affected to a lesser extent. If it is necessary to correlate sampled values or intervals then it is essential that the accuracy of sampling time and interval start/ stop times is sufficient for the application (for example +/- 2%). 6.5.2. Dependencies of metric definitions on related events or metrics Metric definitions may explicitly or implicitly rely on factors that may not be obvious. For example, the recognition of a packet as being "lost" relies on having some method to know the packet was actually lost (e.g. RTP sequence number), and some time threshold after which a non-received packet is declared as lost. It is important that any such dependencies are recognized and incorporated into the metric definition. 6.5.3. Relationship between application performance and lower layer metrics Lower layer metrics may be used to compute or infer the performance of higher layer applications, potentially using an application performance model. The accuracy of this will depend on many factors including: Clark & Claise Expires January 24, 2011 [Page 15] Internet-Draft Perf. Metric Framework July 2010 (i) The completeness of the set of metrics - i.e. are there metrics for all the input values to the application performance model? (ii) Correlation between input variables (being measured) and application performance (iii) Variability in the measured metrics and how this variability affects application performance 6.5.4. Middlebox presence Presence of a middlebox RFC 3303 [RFC3303], e.g., proxy, NAT, redirect server, session border controller (SBC), and application layer gateway (ALG) may add variability to or restrict the scope of measurements of a metric. For example, an SBC that does not process RTP loopback packets may block or locally terminate this traffic rather then pass it through to its target. 6.6. Organization of Results The IPPM Framework [RFC2330] organizes the results of metrics into three related notions: o singleton, an elementary instance, or "atomic" value. o sample, a set of singletons with some common properties and some varying properties. o statistic, a value derived from a sample through deterministic calculation, such as the mean. Many metrics can use this organization for the results, with or without the term names used by IPPM WG. Section 11 of RFC 2330 [RFC2330] should consulted for further details. 6.7. Parameters, the variables of a metric Metrics are completely defined when all options and input variables have been identified and considered. These variables are sometimes left unspecified in a metric definition, and their general name indicates that the user must set them and report them with the results. Such variables are called "parameters" in the IPPM metric template. The scope of the metric, the time at which it was conducted, the settings for timers and the thresholds for counters are all examples of parameters. All documents defining performance metrics SHOULD identify ALL key parameters for each metric. Clark & Claise Expires January 24, 2011 [Page 16] Internet-Draft Perf. Metric Framework July 2010 7. Performance Metric Development Process 7.1. New Proposals for Metrics This process is intended to add additional considerations to the processes for adopting new work as described in RFC 2026 [RFC2026] and RFC 2418 [RFC2418]. The following entry criteria will be considered for each proposal. Proposals SHOULD be prepared as Internet Drafts, describing the metrics and conforming to the qualifications above as much as possible. Proposals SHOULD be vetted by the corresponding protocol development WG prior to discussion by the PM Entity. This aspect of the process includes an assessment of the need for the metrics proposed and assessment of the support for their development in IETF. Proposals SHOULD include an assessment of interaction and/or overlap with work in other Standards Development Organizations. Proposals SHOULD specify the intended audience and users of the metrics. The development process encourages participation by members of the intended audience. Proposals SHOULD survey the existing standards work in the area and identify additional expertise that might be consulted, or possible overlap with other standards development orgs. Proposals SHOULD identify any security and IANA requirements. Security issues could potentially involve revealing of user identifying data or the potential misuse of active test tools. IANA considerations may involve the need for a metrics registry. 7.2. Reviewing Metrics Each metric SHOULD be assessed according to the following list of qualifications: o Unambiguously defined? o Units of Measure Specified? o Measurement Interval Specified? o Measurement Errors Identified? Clark & Claise Expires January 24, 2011 [Page 17] Internet-Draft Perf. Metric Framework July 2010 o Repeatable? o Implementable? o Assumptions concerning underlying process? o Use cases? o Correlation with application performance/ user experience? 7.3. Proposal Approval New work item proposals SHALL be approved using the existing IETF process. The process depends on the form that the PM Entity ultimately takes, Directorate or WG. In all cases, the proposal will need to achieve consensus, in the corresponding protocol development WG (or alternatively, an "Area" WG with broad charter), that there is interest and a need for the work. IF the PM Entity is a Directorate, THEN approval SHOULD include the following steps o consultation with the PM Directorate, using this framework document o consultation with Area Director(s) o and possibly IESG approval of a new or revised charter for the WG IF the PM Entity is a WG, and the protocol development WG decides to take up the work under its charter, THEN the approval is the same as the PM Directorate steps above, with the possible additional assignment of a PM Advisor for the work item. IF the PM Entity is a WG, and the protocol development WG decides it does not have sufficient expertise to take-up the work, or the proposal falls outside the current charter, THEN approval SHOULD include the following steps o identification of protocol expertise to support metric development Clark & Claise Expires January 24, 2011 [Page 18] Internet-Draft Perf. Metric Framework July 2010 o consensus in the PM Entity that there is interest and a need for the work, and that a document conforming to this framework can be successfully developed o consultation with Area Director(s) o IESG approval of a revised charter for the PM WG 7.4. PM Entity Interaction with other WGs The PM Entity SHALL work in partnership with the related protocol development WG when considering an Internet Draft that specifies performance metrics for a protocol. A sufficient number of individuals with expertise must be willing to consult on the draft. If the related WG has concluded, comments on the proposal should still be sought from key RFC authors and former chairs, or from the WG mailing list if it was not closed. Existing mailing lists SHOULD be used however a dedicated mailing list MAY be initiated if necessary to facilitate work on a draft. In some cases, it will be appropriate to have the IETF session discussion during the related protocol WG session, to maximize visibility of the effort to that WG and expand the review. 7.5. Standards Track Performance Metrics The PM Entity will manage the progression of PM RFCs along the Standards Track. See [I-D.bradner-metricstest]. This may include the preparation of test plans to examine different implementations of the metrics to ensure that the metric definitions are clear and unambiguous (depending on the final form of the draft above). 8. IANA Considerations This document makes no request of IANA. Note to RFC EDITOR: this section may be removed on publication as an RFC. 9. Security Considerations In general, the existence of framework for performance metric development does not constitute a security issue for the Internet. Metric definitions may introduce security issues and this framework recommends that those defining metrics should identify any such risk Clark & Claise Expires January 24, 2011 [Page 19] Internet-Draft Perf. Metric Framework July 2010 factors. The security considerations that apply to any active measurement of live networks are relevant here. See [RFC4656]. The security considerations that apply to any passive measurement of specific packets in live networks are relevant here as well. See the security considerations in [RFC5475]. 10. Acknowledgements The authors would like to thank Al Morton, Dan Romascanu, Daryl Malas and Loki Jorgenson for their comments and contributions. 11. References 11.1. Normative References [RFC2026] Bradner, S., "The Internet Standards Process -- Revision 3", BCP 9, RFC 2026, October 1996. [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. [RFC2418] Bradner, S., "IETF Working Group Guidelines and Procedures", BCP 25, RFC 2418, September 1998. [RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M. Zekauskas, "A One-way Active Measurement Protocol (OWAMP)", RFC 4656, September 2006. 11.2. Informative References [E800] "ITU-T Recommendation E.800. SERIES E: OVERALL NETWORK OPERATION, TELEPHONE SERVICE, SERVICE OPERATION AND HUMAN FACTORS". [G1000] "ITU-T Recommendation G.1000. Communications Quality of Service: A framework and definitions". [I-D.bradner-metricstest] Bradner, S. and V. Paxson, "Advancement of metrics specifications on the IETF Standards Track", draft-bradner-metricstest-03 (work in progress), August 2007. Clark & Claise Expires January 24, 2011 [Page 20] Internet-Draft Perf. Metric Framework July 2010 [I-D.ietf-ipfix-mediators-framework] Kobayashi, A., Claise, B., Muenz, G., and K. Ishibashi, "IPFIX Mediation: Framework", draft-ietf-ipfix-mediators-framework-07 (work in progress), July 2010. [I-D.ietf-ippm-spatial-composition] Morton, A. and E. Stephan, "Spatial Composition of Metrics", draft-ietf-ippm-spatial-composition-15 (work in progress), July 2010. [P.800] "ITU-T Recommendation P.800. : Methods for subjective determination of transmission quality". [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, "Framework for IP Performance Metrics", RFC 2330, May 1998. [RFC3303] Srisuresh, P., Kuthan, J., Rosenberg, J., Molitor, A., and A. Rayhan, "Middlebox communication architecture and framework", RFC 3303, August 2002. [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. Jacobson, "RTP: A Transport Protocol for Real-Time Applications", STD 64, RFC 3550, July 2003. [RFC4710] Siddiqui, A., Romascanu, D., and E. Golovinsky, "Real-time Application Quality-of-Service Monitoring (RAQMON) Framework", RFC 4710, October 2006. [RFC5101] Claise, B., "Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information", RFC 5101, January 2008. [RFC5102] Quittek, J., Bryant, S., Claise, B., Aitken, P., and J. Meyer, "Information Model for IP Flow Information Export", RFC 5102, January 2008. [RFC5475] Zseby, T., Molina, M., Duffield, N., Niccolini, S., and F. Raspall, "Sampling and Filtering Techniques for IP Packet Selection", RFC 5475, March 2009. [RFC5481] Morton, A. and B. Claise, "Packet Delay Variation Applicability Statement", RFC 5481, March 2009. [RFC5706] Harrington, D., "Guidelines for Considering Operations and Management of New Protocols and Protocol Extensions", RFC 5706, November 2009. Clark & Claise Expires January 24, 2011 [Page 21] Internet-Draft Perf. Metric Framework July 2010 [RFC5835] Morton, A. and S. Van den Berghe, "Framework for Metric Composition", RFC 5835, April 2010. Authors' Addresses Alan Clark Telchemy Incorporated 2905 Premiere Parkway, Suite 280 Duluth, Georgia 30097 USA Phone: Fax: Email: alan.d.clark@telchemy.com URI: Benoit Claise Cisco Systems, Inc. De Kleetlaan 6a b1 Diegem 1831 Belgium Phone: +32 2 704 5622 Fax: Email: bclaise@cisco.com URI: Clark & Claise Expires January 24, 2011 [Page 22] --------------090807010301010703070702--