Subject: [PMOL] Remote Participation URLs (Dublin)
From: Al Morton <[email-masked]>
To: [email-masked]
Date: Mon, 28 Jul 2008 09:21:29 -0400
List-Id: Performance Metrics at Other Layers <pmol.ietf.org>

For those who wish to participate remotely, we have:

Start time:  Today, July 29, 15:20-1720 in Dublin, Ireland (GMT+1:00)

Much of the info below is available on the tools team agenda:
http://tools.ietf.org/agenda/72/

Jabber Instructions
http://www.ietf.org/meetings/text_conf.html
Jabber Room
[email-masked]

Slides, not all posted yet (scroll down to pmol in OPS area)
https://datatracker.ietf.org/meeting/72/materials.html

Unicast Audio
http://videolab.uoregon.edu/events/ietf/
pmol is on Channel 4 in Newcastle

Al
pmol co-chair

_______________________________________________
PMOL mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/pmol

Content-class: urn:content-classes:message
Date: Wed, 23 Jul 2008 17:24:34 +0200
From: "Romascanu, Dan (Dan)" <[email-masked]>
To: <[email-masked]>
Subject: [PMOL] comments on draft-ietf-pmol-metrics-framework-00.txt
Sender: [email-masked]

Please find below my contributor comments on
draft-ietf-pmol-metrics-framework-00.txt

1. Section 1.1: s/Although the IETF has two Working Groups/Although the
IETF currently has two active Working Groups/

2. Section 3.3: s/Temporal Aggregation is a summarization of metrics
into/Temporal Aggregation is a summarization of sets of metrics into/

3. Section 3.4.1 - it is not clear what is the difference between 'what
the metric is' and 'describes the metrics'. Maybe the intent here is
s/describes the metric/describes the rationale of the metric/? 

4. (iii) Measurement Method - should not we relate here also to the
measurement points? For example in the sip-performance-metrics draft
some of the metrics are measured only at UACs, other at UAC or UAS, etc.

5. The 2119 capitalization in section 3.4.2 is inconsistent. Maybe the
MUST in the first phrase is enough and the rest needs not be
capitalized. 

6. There are alignment problems with the paragraphs in 3.4.3

7. 3.4.3 (ii) - is conformance testing really a SHOULD in our metrics
documents? Or maybe just a MAY? I am hesitating on this, we define very
little conformance testing in the IETF

8. 3.4.4 - if Conformance Testing is to stay, it should be part of the
template. 

9. It is not clear to me what is the scope of 3.5 and in any case I do
not believe it should be located here in the document. Maybe its place
is rather in section 4, as a checklist for reviewing PM documents, or
maybe in an Annex. 

10. The text in 3.6 and the whole subsection should be merged with the
text about reporting models in 3.4.4

11. Section 4.2 - What proposal approval are we talking about here? The
approval of initial work, or the final approval? In any case I think
that we just need to refer to the IETF processes to charter new work, or
to approve documents. We do not intent to invent a new process here, but
to either support other WGs with the expertise of a cross-area
directorate, or create the framework of a new WG. I am not sure this
section is needed at all. 

12. Section 4.3 - here we should explain in a few words the two options
1. a cross-area directorate which is called to give advice on the
development of new PMOL metrics in other WGs and reviews the documents
as part of the WGLC and IESG review process 2. a dedicated WG within the
OPS Area which creates the framework for other areas protocol experts to
develop here their PM definition documents. In both cases there is a
need to interact - in case #1 as a WG-directorate review team
relationship, in case #2 as a protocol WG- PMOL WG relation. The
decision should be left IMO to the IESG 

13. Section 4.3 - I think that the requirement for a dedicated mailing
list for each effort is exaggerated - the protocol WG mail list and/or
the PMOL (directorate or WG) list should be enough. 

Thanks and Regards,

Dan

_______________________________________________
PMOL mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/pmol

Content-class: urn:content-classes:message
Date: Wed, 23 Jul 2008 14:26:00 +0200
From: "Romascanu, Dan (Dan)" <[email-masked]>
To: "Al Morton" <[email-masked]>, <[email-masked]>
Cc: [email-masked]
Subject: Re: [PMOL] [Sipping] WG Last Call: draft-ietf-pmol-sip-perf-metrics-01
Sender: [email-masked]

Please find below my contributor comments. 

 1. idnits complains on a number of references problems:

  -- Looks like a reference, but probably isn't: 'RFC 1157' on line
     1149
'[RFC 1157] and via future extensions to the SIP Management...'

Please add [RFC1157] as reference

  == Unused Reference: '12' is defined on line 1282, but no explicit
     reference was found in the text
     '[12]  Faber, T., Touch, J. and W. Yue, "The TIME-WAIT state in TCP
a...'

  == Unused Reference: '13' is defined on line 1286, but no explicit
     reference was found in the text
     '[13]  Crocker, D. and Overell, P.(Editors), "Augmented BNF for
Synta...'

Please take out [12] and [13] if not needed. 

  -- Possible downref: Non-RFC (?) normative reference: ref. '7'

  -- Possible downref: Non-RFC (?) normative reference: ref. '8'

  -- Possible downref: Non-RFC (?) normative reference: ref. '9'

These three are probably OK

  ** Downref: Normative reference to an Informational RFC: RFC 2330
(ref.
     '11')

Does this need to be a normative reference? 

  -- Obsolete informational reference (is this intentional?): RFC 2234
(ref.
     '13') (Obsoleted by RFC 4234)

Is this intentional, or can 2234 be replaced by RFC 4234. Of course if
[13] will be taken out (see above) no need to answer this. 

2. Introduction - 'responsiveness of the network' - is it really only
the network, or rather 'responsiveness of the network and application
servers'? 

3. same, why is DO NOT capitalized? 

4. section 4.4 - why does SDT need to be adjusted to indicate 'minutes
and seconds' and not just seconds? 

5. section 4.8 - the definition of SD seems broken. If the output value
of this metric is numerical and should be adjusted to indicate a
percentage of defective sessions, the formula seems to be SD = (# of
Defective Sessions) / (Total # of Session Requests). Am I missing
something? 

6. For consistency reasons should not metrics define in 4.8, 4.9, 4.10
be all named as Rate? 

7. RFC1157 is historic. Use RFC3411 to refer to SNMP

8. I suggest to add RAQMON (RFC4710) with possible future extensions to
the set of methods for transferring metrics information enumerated in
6.4

9. Sections 6.5 an 6.6 seem to deal with issues that are beyond the
scope of the SIP performance metrics. I think their place is not in the
normative part of this document, but maybe in the PMOL framework
document. I suggest to take out these sections completely or move them
to an Appendix to make clear that they are non-normative. 

Dan

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Al Morton
> Sent: Thursday, June 26, 2008 5:48 AM
> To: [email-masked]
> Cc: [email-masked]
> Subject: [Sipping] WG Last Call: draft-ietf-pmol-sip-perf-metrics-01
> 
> PMOL WG,
> cc SIPPING WG,
> 
> This message begins a WG Last Call on the following:
> 
>         Title		: SIP End-to-End Performance Metrics
_______________________________________________
PMOL mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/pmol

Date: Mon, 21 Jul 2008 22:07:42 -0400
To: [email-masked]
From: Al Morton <[email-masked]>
Subject: Re: [PMOL] WG Last Call: draft-ietf-pmol-sip-perf-metrics-01
Sender: [email-masked]

Hi Daryl,

At 10:48 PM 6/25/2008, Al Morton wrote:
>This message begins a WG Last Call on the following:
>A URL for this Internet-Draft is:
>http://www.ietf.org/internet-drafts/draft-ietf-pmol-sip-perf-metrics-01.txt

my comments as a pmol participant follow,
Al

(I have several editorial suggestions which I will pass
to Daryl at the upcoming meeting.)

The major comment is the same one I made for the ccamp
GMPLS LSP metrics draft...
>-----------------
>There is a taxonomy of possible outcomes for set-up protocols.
>Each set-up attempt will have one of the following outcomes:
>
>1. Success
>2. Failure
>3. Incorrect Set-up (wrong destination)
>
>The same basic outcomes apply to release attempts.
>-----------------
...but here it can mostly be addressed
by changing the names of a few metrics.

In the current text, the set-up time metrics cover the
Successful outcome and the Failure outcome, but attempt
to do it with a *single* metric, AFAICT.
For Registration, its RRD (Registration Request Delay, sec 4.1),
and for Sessions, its SRD (Session Request Delay, in 4.2).
I don't think that Successful Delay and Failure Delay values
should be combined in the same distribution, and the draft seems
to allow that with only one _RD for either Registration or Session.

Later, the draft introduces the notion of an
Ineffective Session Attempts (ISA) when defining another important
metric, Session Establishment Rate (sec 4.6).
IMO, the ISA should be introduced as an alternative to the Successful SRD.
There should also be an Ineffective Registration Attempt (IRA).

Here's how the outline might change to do all this:

    4. SIP Performance Metrics

       4.1. Registration Request Metrics
          4.1.1. Successful REGISTER Delay (SRD)
                         ends with 200 OK
          4.1.2. Ineffective Registration Attempt (IRA)
                         count attempts with with non-200
          4.1.3. Failed REGISTER Delay (FRD)
                         ends with non-200, (some are just Timer F?)

       4.2. Session Request Metrics
          4.2.1. Successful Session-Setup Delay (SSD)
                         ends with 200 OK
          4.1.2. Ineffective Session-Setup Attempt (ISA)
                         count attempts with with non-200
          4.2.3. Failed Session-Setup Delay (FSD)
                         ends with non-200, (some are just Timer F?)
          4.2.X. Instant Messaging
(the possibilities here are similar to the above,
I think IM may deserve its own section.)

       4.3. Session Disconnect Delay
          4.3.1. Successful Session Disconnect Delay (SDD)
                         ends with 200 OK
          4.3.2. Ineffective (Session) Disconnect Attempt (IDA)
                         count attempts with with non-200
          4.3.2. Failed-Disconnect Delay (SFD)
                         ends with non-200, (some are just Timer F?)

       4.4. Session Time (ST)
(was Session Duration Time, SDT, the terms
Duration and Time are Redundant,
and the memo uses SD below)

          4.4.1. Session Time with successful completion ST
                        from 200 to BYE
          4.4.2. Session Time with failed completion SFT
                         ends with non-BYE, (just Timer F?)

       4.5. Hops per Request (HpR)
       4.6. Session Establishment Rate (SER)
(IMO, this should be Ratio,  and 1,$s /Rate/Ratio/g )

          4.6.1. Instant Messaging

       4.7. Session Establishment Efficiency Rate (SEER)
       4.8. Session Defects (SD)
       4.9. Ineffective Session Attempts (ISA)
(text moves to 4.2.2)
       4.10. Session Disconnect Failures (SDF)
(text moves to 4.3.2)

Other comments:
---------------

Section 2, Terminology, has the following definition:
>    End-to-End - This is described as two or more elements utilized for
>    initiating a request, receiving the request, and responding to the
>    request.  It encompasses elements as necessary to be involved in a
>    session dialog between the originating user agent client (UAC),
>    destination user agent server (UAS), and any interim proxies (may
>    also include back-to-back user agent's (B2BUA's)). This may be
>    relative to a single operator's set of elements or extend to
>    encompass all elements (if beyond a single operator's network)
>    associated with a session.

There are lots of definitions for "end-to-end" in standards,
so we should try to be consistent with them, and clear about
this particular context.  I suggest:

End-to-End - This phrase defines the scope of communications as
the logical extremes, covering two or more elements utilized for
initiating a request, receiving the request, and responding to the
request.  It encompasses elements necessarily involved in a
session dialog between the originating user agent client (UAC),
destination user agent server (UAS), and any interim proxies (may
also include back-to-back user agents (B2BUA's)). End-to-End may
denote a single operator's set of elements or extend to
encompass all elements (if beyond a single operator's network)
associated with a session dialog.

-----------------------

Section 4.

For all the  relevant metrics, it strikes me that the length
of the Request (INVITE) may affect the performance, especially if the
Request message is fragmented. It seems like the Request message length
and the MTU should be parameters for all relevant metrics.

-----------------------

Section 4.7 Session Establishment Efficiency Rate (SEER)
...
           # of INVITE Requests w/ associated 200OK, 480, 486, or 600
    SEER = --------------------------------------------------------------
      (Total # of INVITE Requests)-(# of INVITE Requests w/ 'a' Response)

This ratio has all the Requests that get a legitimate response
divided by the total requests, reduced by the "abnormal" responses.

I think I could understand this metric better if the denominator
was not reduced by the "abnormal" responses. That way it would be the
ratio of the legitimate responses to the total requests, and it would
be a ratio of 1 when signaling goes as planned, despite the effects
of the terminating UAS.  The metric would have the following ratio:

           # of INVITE Requests w/ associated 200OK, 480, 486, or 600
    SEER = --------------------------------------------------------------
                          (Total # of INVITE Requests)

----------------------

I might have some more comments, but don't want to hold
this message up any longer.

_______________________________________________
PMOL mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/pmol

Date: Mon, 14 Jul 2008 14:49:46 -0400
To: [email-masked]
From: Al Morton <[email-masked]>
Subject: [PMOL] Agenda preparation for pmol session at IETF-72
Sender: [email-masked]

PMOL WG,

Our Dublin meeting is currently scheduled as follows:

Tuesday, July 29, 2008, 1520-1720 Afternoon Session II
Room: Newcastle

We will have at least three agenda items:
1. WG Status
http://tools.ietf.org/wg/pmol/
2. SIP Metrics Draft (resolve WGLC comments)
http://tools.ietf.org/html/draft-ietf-pmol-sip-perf-metrics-01
3. Framework and Guidelines Draft (please read and comment!)
http://tools.ietf.org/html/draft-ietf-pmol-metrics-framework-00

If time allows, the co-chairs are willing to consider
an additional agenda item on performance metric development
covered by the PMOL framework and guidelines memo.

Please contact the co-chairs directly with any proposals
or comments relevant to the agenda.

thanks,
Alan and Al
pmol co-chairs

_______________________________________________
PMOL mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/pmol

Content-class: urn:content-classes:message
Date: Mon, 14 Jul 2008 10:21:07 -0600
From: "Daryl Malas" <[email-masked]>
To: <[email-masked]>
Cc: [email-masked]
Subject: Re: [PMOL] [Sipping] WG Last Call: draft-ietf-pmol-sip-perf-metrics-01
Sender: [email-masked]

All,

The major changes in this version include removing all of the "average"
calculations, essentially allowing implementers to use the root
algorithms as they see fit.  The other major change is an addition,
provided by Al Morton, of "Time Interval Measurement and Reporting"
details.  

A few other nits and clean-up issues were also revised in this version.

Thank you,

Daryl

----------------
Daryl Malas
CableLabs
(o) +1 303 661 3302
(f) +1 303 661 9199
mailto:[email-masked]

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Al Morton
> Sent: Monday, July 14, 2008 9:54 AM
> To: [email-masked]
> Cc: [email-masked]
> Subject: Re: [Sipping] WG Last Call: 
> draft-ietf-pmol-sip-perf-metrics-01
> 
> Reminder - WGLC ends July 25, 2008.
> 
> PMOL WG,
> cc SIPPING WG,
> 
> This message begins a WG Last Call on the following:
> 
>         Title		: SIP End-to-End Performance Metrics
> 	Author(s)	: D. Malas
> 	Filename	: draft-ietf-pmol-sip-perf-metrics-01.txt
> 	Pages		: 29
> 	Date		: 2008-6-25
> 	
>     This document defines a set of metrics and their usage to evaluate
>     the performance of end-to-end Session Initiation Protocol 
> (SIP) based
>     services in both production and testing environments.  
> The purpose of
>     this document is to combine a set of common metrics, allowing
>     interoperable performance measurements, easing the comparison of
>     industry implementations.
> 
> A URL for this Internet-Draft is:
> http://www.ietf.org/internet-drafts/draft-ietf-pmol-sip-perf-m
etrics-01.txt
> 
> There has been discussion of this draft for 2 years, first in 
> the SIPPING WG, and then in the PMOL WG.  The Author believes 
> he has addressed all comments to date.
> 
> Please weigh-in on whether or not this Draft should be 
> forwarded to the Area Directors for publication as a 
> Standards Track RFC.
> Send your comments to the PMOL list or the co-chairs.
> 
> The Last Call will be open till July 25, 2008.
> 
> thanks for your review and comments,
> Alan and Al
> co-chairs, PMOL WG
> 
> _______________________________________________
> Sipping mailing list  https://www.ietf.org/mailman/listinfo/sipping
> This list is for NEW development of the application of SIP 
> Use [email-masked] for questions on current 
> sip Use [email-masked] for new developments of core SIP
> 
_______________________________________________
PMOL mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/pmol

Date: Mon, 14 Jul 2008 11:54:16 -0400
To: [email-masked]
From: Al Morton <[email-masked]>
Cc: [email-masked]
Subject: Re: [PMOL] WG Last Call: draft-ietf-pmol-sip-perf-metrics-01
Sender: [email-masked]

Reminder - WGLC ends July 25, 2008.

PMOL WG,
cc SIPPING WG,

This message begins a WG Last Call on the following:

        Title		: SIP End-to-End Performance Metrics
	Author(s)	: D. Malas
	Filename	: draft-ietf-pmol-sip-perf-metrics-01.txt
	Pages		: 29
	Date		: 2008-6-25
	
    This document defines a set of metrics and their usage to evaluate
    the performance of end-to-end Session Initiation Protocol (SIP) based
    services in both production and testing environments.  The purpose of
    this document is to combine a set of common metrics, allowing
    interoperable performance measurements, easing the comparison of
    industry implementations.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-pmol-sip-perf-metrics-01.txt

There has been discussion of this draft for 2 years, first in
the SIPPING WG, and then in the PMOL WG.  The Author believes he
has addressed all comments to date.

Please weigh-in on whether or not this Draft should be forwarded
to the Area Directors for publication as a Standards Track RFC.
Send your comments to the PMOL list or the co-chairs.

The Last Call will be open till July 25, 2008.

thanks for your review and comments,
Alan and Al
co-chairs, PMOL WG

_______________________________________________
PMOL mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/pmol

From: [email-masked]
To: [email-masked]
Date: Fri,  4 Jul 2008 12:45:01 -0700 (PDT)
Cc: [email-masked]
Subject: [PMOL] I-D Action:draft-ietf-pmol-metrics-framework-00.txt
Sender: [email-masked]

--NextPart

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the Performance Metrics for Other Layers Working Group of the IETF.

	Title           : Framework for Performance Metric Development
	Author(s)       : A. Clark
	Filename        : draft-ietf-pmol-metrics-framework-00.txt
	Pages           : 15
	Date            : 2008-07-04

This memo describes a framework and guidelines for the development of
performance metrics that are beyond the scope of existing working
group charters in the IETF.  In this version, the memo refers to a
Performance Metrics Entity, or PM Entity, which may in future be a
working group or directorate or a combination of these two.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-pmol-metrics-framework-00.txt

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

--NextPart
Content-Type: Message/External-body;
	name="draft-ietf-pmol-metrics-framework-00.txt";
	site="ftp.ietf.org";
	access-type="anon-ftp";
	directory="internet-drafts"

Content-Type: text/plain
Content-ID: <[email-masked]>

--NextPart
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
PMOL mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/pmol

--NextPart--

From: "Alan Clark" <[email-masked]>
To: <[email-masked]>
Date: Fri, 4 Jul 2008 15:46:50 -0400
Subject: [PMOL] FW: New Version Notification for draft-ietf-pmol-metrics-framework-00
Sender: [email-masked]

Apologies for the delay in getting the updated framework draft out, but here
it is!!

Alan

-----Original Message-----
From: IETF I-D Submission Tool [mailto:[email-masked]] 
Sent: Friday, July 04, 2008 3:44 PM
To: [email-masked]
Subject: New Version Notification for draft-ietf-pmol-metrics-framework-00 

A new version of I-D, draft-ietf-pmol-metrics-framework-00.txt has been
successfuly submitted by Alan Clark and posted to the IETF repository.

Filename:	 draft-ietf-pmol-metrics-framework
Revision:	 00
Title:		 Framework for Performance Metric Development
Creation_date:	 2008-07-04
WG ID:		 pmol
Number_of_pages: 15

Abstract:
This memo describes a framework and guidelines for the development of
performance metrics that are beyond the scope of existing working
group charters in the IETF.  In this version, the memo refers to a
Performance Metrics Entity, or PM Entity, which may in future be a
working group or directorate or a combination of these two.

The IETF Secretariat.

_______________________________________________
PMOL mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/pmol