Subject: [PMOL] FW: I-D ACTION:draft-ietf-pmol-sip-perf-metrics-01.txt
From: "Romascanu, Dan (Dan)" <[email-masked]>
To: <[email-masked]>
Date: Thu, 26 Jun 2008 12:45:18 +0200
List-Id: Performance Metrics at Other Layers <pmol.ietf.org>

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Thursday, June 26, 2008 3:30 AM
To: [email-masked]
Cc: [email-masked]
Subject: I-D ACTION:draft-ietf-pmol-sip-perf-metrics-01.txt 

A New Internet-Draft is available from the on-line Internet-Drafts
directories.
This draft is a work item of the Performance Metrics for Other Layers
Working Group of the IETF.

	Title		: SIP End-to-End Performance Metrics
	Author(s)	: D. Malas
	Filename	: draft-ietf-pmol-sip-perf-metrics-01.txt
	Pages		: 29
	Date		: 2008-6-25
	
This document defines a set of metrics and their usage to evaluate 
   the performance of end-to-end Session Initiation Protocol (SIP) based

   services in both production and testing environments.  The purpose of

   this document is to combine a set of common metrics, allowing 
   interoperable performance measurements, easing the comparison of 
   industry implementations.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-pmol-sip-perf-metrics-01.
txt

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.
_______________________________________________
PMOL mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/pmol

Date: Wed, 25 Jun 2008 22:48:03 -0400
To: [email-masked]
From: Al Morton <[email-masked]>
Cc: [email-masked]
Subject: [PMOL] WG Last Call: draft-ietf-pmol-sip-perf-metrics-01
Sender: [email-masked]

PMOL WG,
cc SIPPING WG,

This message begins a WG Last Call on the following:

        Title		: SIP End-to-End Performance Metrics
	Author(s)	: D. Malas
	Filename	: draft-ietf-pmol-sip-perf-metrics-01.txt
	Pages		: 29
	Date		: 2008-6-25
	
    This document defines a set of metrics and their usage to evaluate
    the performance of end-to-end Session Initiation Protocol (SIP) based
    services in both production and testing environments.  The purpose of
    this document is to combine a set of common metrics, allowing
    interoperable performance measurements, easing the comparison of
    industry implementations.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-pmol-sip-perf-metrics-01.txt

There has been discussion of this draft for 2 years, first in
the SIPPING WG, and then in the PMOL WG.  The Author believes he
has addressed all comments to date.

Please weigh-in on whether or not this Draft should be forwarded
to the Area Directors for publication as a Standards Track RFC.
Send your comments to the PMOL list or the co-chairs.

The Last Call will be open till July 25, 2008.

thanks for your review and comments,
Alan and Al
co-chairs, PMOL WG

_______________________________________________
PMOL mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/pmol

Date: Wed, 25 Jun 2008 11:36:51 +0800 
From: [email-masked]
To: [email-masked], [email-masked], [email-masked], [email-masked], [email-masked], [email-masked],
Cc: [email-masked], [email-masked], [email-masked], [email-masked], [email-masked], [email-masked], [email-masked], [email-masked], [email-masked]
Subject: [PMOL] LSP DPPM draft updated.
Sender: [email-masked]

--===============1307016464==
Content-Type: multipart/alternative;
	 boundary="=-sinamail_alt_c659913727155f305ed33c48ac01836f"

--=-sinamail_alt_c659913727155f305ed33c48ac01836f
Content-Type: text/plain;
	charset=GBK
Content-Transfer-Encoding: base64
Content-Disposition: inline

--=-sinamail_alt_c659913727155f305ed33c48ac01836f
Content-Type: text/html; 
	charset=GBK
Content-Transfer-Encoding: base64
Content-Disposition: inline

--=-sinamail_alt_c659913727155f305ed33c48ac01836f--

--===============1307016464==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
PMOL mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/pmol

--===============1307016464==--

Content-class: urn:content-classes:message
Date: Wed, 11 Jun 2008 15:05:13 +0200
From: "Romascanu, Dan (Dan)" <[email-masked]>
To: <[email-masked]>
Subject: [PMOL] IPPM Rechartering BOF
Sender: [email-masked]

The following BOF is proposed for the Dublin IETF meeting. 

IPPM Rechartering BOF 
Status: will be approved 
Responsible AD: Lars Eggert 
Description: IPPM is nearing the end of its chartered work, and is
discussing to recharter. Because several of the proposed new work items
go significantly beyond its current scope (passive measurements, SLA
validation), I'm encouraging them to discuss these in a BOF rather than
just within the WG. 
Discussions: [email-masked] 

If the PMOL folks have any comments or questions please let me know in
the next day or so. The IAB and IESG telechat that will discuss the BOF
proposals is scheduled for tomorrow. 

Thanks and Regards,

Dan
_______________________________________________
PMOL mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/pmol

From: "Weiqiang Sun" <[email-masked]>
To: "Henk Uijterwaal" <[email-masked]>, "IETF IPPM WG" <[email-masked]>, <[email-masked]>
Date: Mon, 9 Jun 2008 16:53:13 +0800
Importance: Normal
Cc: [email-masked]
Subject: Re: [PMOL] FW: [bmwg] Soliciting comments on GMPLS performance metrics draft inCCAMP
Sender: [email-masked]

Hi all,

It seems that my SJTU mail account has some problems with the IETF lists 
(maybe blacklisted?).
I am resending this email from  my Gmail account. Sorry if you receive 
multiple copies of this.

Best regards,
Weiqiang Sun
Shanghai Jiao Tong University

--------------------------------------------------
From: "Weiqiang Sun" <[email-masked]>
Sent: Thursday, June 05, 2008 9:15 PM
To: "Henk Uijterwaal" <[email-masked]>; "IETF IPPM WG" <[email-masked]>; 
<[email-masked]>
Cc: <[email-masked]>
Subject: Re: [PMOL] FW: [bmwg] Soliciting comments on GMPLS performance 
metrics draft inCCAMP

> Hi Henk,
>
> Many thanks for going through the draft and for the comments.
> Please see online for my responses.
>
> Best regards,
> Weiqiang
>
> --------------------------------------------------
> From: "Henk Uijterwaal" <[email-masked]>
> Sent: Thursday, June 05, 2008 7:04 PM
> To: "IETF IPPM WG" <[email-masked]>; <[email-masked]>
> Cc: <[email-masked]>; "Weiqiang Sun" <[email-masked]>
> Subject: Re: [PMOL] FW: [bmwg] Soliciting comments on GMPLS performance 
> metrics draft inCCAMP
>
>> Hi all,
>>
>>> Hi IPPMers and BMWGers,
>>
>>> Recently the LSP dynamic provisioning performance draft is adopted as a 
>>> working item of CCAMP.
>>
>> Some comments, reading this as a non-expert on GMPLS.
>>
>> 3.4. Units.
>>
>>  I don't think this is quite correct: you start to measure at T1 and
>>  wait for the setup to complete.  However, it doesn't, so at some point
>>  you become bored and stop the measurement.  You report undefined.
>>  However, one then has to report in the metrics parameters when the
>>  measurement will be stopped.  This value should be added to the
>>  metrics parameters.  (Note: this will apply to more metrics).
> Yes I agree that this parameter is very important and should be carefully 
> chosen, provided that different switching technologies may differ 
> drastically in the delay when performing switching operations. For this 
> reason we have a little bit discussion regarding this threshhold:
>   o  A given methodology will have to include a way to determine
>      whether a latency value is infinite or whether it is merely very
>      large.  Simple upper bounds could be used.  But GMPLS networks may
>      accommodate many kinds of devices.  For example, some photonic
>      cross-connects (PXCs) have to move the micro mirrors.  This
>      physical motion may take several milliseconds.  But the common
>      electronic switches finish the nodal process within several
>      microseconds.  So the unidirectional LSP setup delay varies
>      drastically from a network to another.  In practice, the upper
>      bound should be chosen carefully.
> However, I think it is a good idea to put this parameter in the "metric 
> parameter" section. This also makes the results more reasonable, since 
> different threshholds may lead to different results.
>
>>
>> 4.3. Metrics Parameters.
>>  In IPPM speak, Lamba refers to a rate but it also specifies a 
>> distribution
>>  for the interval between probe packets.  For example, poisson or 
>> periodic.
>>  The draft should mention which distribution to use.
> We did define poisson arrival process for the lambda. However, it may be 
> interesting to add some more arrival statistical models (with periodic as 
> an example). We will consider this in the next revision.
>
>>
>> 12.1. Statistics.
>>  I think you better report the 5th or 10th percentile of the 
>> distributions,
> Yes we do have the percentile of metric in 12.3, although we do not 
> specify which exact value should be used.
>
>>  rather than the minimum (and the same for the maximum).  This excludes
>>  data points where something went wrong in the measurement process.
> I think it is reasonable to keep the minimum value as this in normal cases 
> can reflect the least amount of time needed to set up/release an LSP 
> (something that is pretty much inherent for the control plane). And I 
> agree that we don't need the maximum. Luckily enough we do not have one in 
> the current version. :)
>
> Anyway, I find a little mistake in 12.3:
>   Given a metric and a percent X between 0% and 100%, the Xth
>   percentile of all the dT values in the sample.  In addition, the
>   unidirectional LSP setup delay percentile is undefined if the sample
>   is empty.
> It will be changed into:
>   Given a metric and a percent X between 0% and 100%, the Xth
>   percentile of all the dT values in the sample.  In addition, the
>   percentile is undefined if the sample is empty.
>
>
>>
>> Henk
>>
>>
>> -- 
>> ------------------------------------------------------------------------------
>> Henk Uijterwaal                           Email: 
>> henk.uijterwaal(at)ripe.net
>> RIPE Network Coordination Centre http://www.amsterdamned.org/~henk
>> P.O.Box 10096          Singel 258         Phone: +31.20.5354414
>> 1001 EB Amsterdam      1016 AB Amsterdam  Fax: +31.20.5354445
>> The Netherlands        The Netherlands    Mobile: +31.6.55861746
>> ------------------------------------------------------------------------------
>>
>> Is one of the choices leaving the office open?
>>                                       Alan Greenspan on the next 
>> elections
>> 
_______________________________________________
PMOL mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/pmol

Date: Thu, 05 Jun 2008 15:21:14 -0400
To: [email-masked], [email-masked]
From: Al Morton <[email-masked]>
Cc: 'Rajiv Papneja' <[email-masked]>, [email-masked], 'xqwei' <[email-masked]>, [email-masked], 'zhangguoying' <[email-masked]>, [email-masked], 'Bin Gu' <[email-masked]>, [email-masked], 'gjhhit' <[email-masked]>, 'blithe' <[email-masked]>
Subject: Re: [PMOL] [ippm] Soliciting comments on GMPLS performance metrics draft in CCAMP
Sender: [email-masked]

--===============0594539723==
Content-Type: text/html; charset="us-ascii"

<html>
<body>
At 04:44 AM 5/29/2008, Weiqiang Sun wrote:<br><br>
<blockquote type=cite class=cite cite="">The authors would now like to
solicit comments among the performance metric experts, as this document
contains notations that are commonly seen in IPPM and BMWG. The authors
believe that your expertise can help us further improve this document,
with regard to the definition of the metrics and the structure as well.
Please send your comments directly to the authors, or, to the CCAMP
working group mail list
(<a href="mailto:[email-masked]">[email-masked]</a>). We do
appreciate your input.</blockquote><br>
As I've said in the past, this draft is an excellent start,<br>
and because it adopts the metric template used in bmwg and ippm,<br>
it's an easy read. Thanks for doing that part.<br><br>
more below,<br>
Al<br><br>
General comments:<br>
-----------------<br>
There is a taxonomy of possible outcomes for set-up protocols.<br>
Each set-up attempt will have one of the following outcomes:<br><br>
1. Success<br>
2. Failure<br>
3. Incorrect Set-up (wrong destination)<br><br>
The same basic outcomes apply to release attempts.<br><br>
In the current text, the set-up time metrics cover the <br>
Successful outcome (they could be called Successful *.* Setup
Time).<br><br>
When a setup attempt fails, the setup time is Undefined. Stop there.<br>
Please remove the &quot;informally infinite&quot; phrase between <br>
parentheses, because it places the failure outcome on the<br>
same dimension as success. As I've pointed out above, <br>
the 3 outcomes are mutually exclusive.&nbsp; I know this text came <br>
from IPPM RFCs, but this is a case where you should not follow<br>
that precedent, IMHO.&nbsp; For more nit-picking background, read<br>
<a href="http://tools.ietf.org/id/draft-morton-ippm-reporting-metrics-05.txt" eudora="autourl">
http://tools.ietf.org/id/draft-morton-ippm-reporting-metrics-05.txt<br>
<br>
</a>Back to the Failure Outcome.&nbsp; I can imagine that there will
be<br>
technologies for which a metric on Setup Failure Ratio may be <br>
more important than the Successful setup time - I may not care<br>
if the setup takes 1 second or 10 seconds, but the 50% failure<br>
ratio is annoying...<br><br>
So, I propose to add metrics to cover the Failure outcome,<br>
defined as a count of failures=Undefined setup times divided by the<br>
total attempts. This would be a single outcome for a singleton<br>
metric and a ratio for metrics with multiple LSP setups.<br>
(I saw the metric in 12.4, it doesn't quite have the right wording<br>
or emphasis - this metric should be a part of all setup and release<br>
scenarios).<br><br>
Adding metrics to cover the Incorrect setup/release outcome<br>
would complete the taxonomy.&nbsp; I can accept that the Incorrect<br>
outcome may be quite rare in practice, but having the metric <br>
defined and ready is a reasonable use of resources. You can<br>
note that these outcomes may be rare in some technologies as <br>
part of the discussion.<br><br>
Specific Comments:<br>
------------------<br>
(You can generalize these to all applicable sections)<br><br>
3.3&nbsp; Metric Parameters<br><br>
&nbsp; o&nbsp; ID0, the ingress LSR ID<br><br>
&nbsp; o&nbsp; ID1, the egress LSR ID<br><br>
You should use these ID#s in the definition of the metric.<br><br>
&nbsp; o&nbsp; T, a time<br><br>
I've found that the two word description of a parameter is <br>
not always sufficient for me.&nbsp; I suggest:<br><br>
&nbsp; o&nbsp; T, a time when the setup is attempted<br><br>
I suggest to add a parameter, for clarity:<br><br>
&nbsp; o&nbsp; WAITTIME, the maximum waiting time for successful
setup<br><br>
and mention this parameter in the sections where discussed, <br>
like 3.5 and 3.6.<br><br>
----------<br><br>
13. Discussion<br><br>
The control-plane-only nature of these metrics should be clear(er)<br>
in the Introduction/Scope.<br><br>
Question:<br>
How important is it to have data-plane verification of the <br>
apparent control-plane success/failure/incorrect performance?<br><br>
<br><br>
<br><br>
</body>
</html>

--===============0594539723==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
PMOL mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/pmol

--===============0594539723==--

From: "Weiqiang Sun" <[email-masked]>
To: "Henk Uijterwaal" <[email-masked]>, "IETF IPPM WG" <[email-masked]>, <[email-masked]>
Date: Thu, 5 Jun 2008 22:15:04 +0900
Importance: Normal
Cc: [email-masked]
Subject: Re: [PMOL] FW: [bmwg] Soliciting comments on GMPLS performance metrics draft inCCAMP
Sender: [email-masked]

Hi Henk,

Many thanks for going through the draft and for the comments.
Please see online for my responses.

Best regards,
Weiqiang

--------------------------------------------------
From: "Henk Uijterwaal" <[email-masked]>
Sent: Thursday, June 05, 2008 7:04 PM
To: "IETF IPPM WG" <[email-masked]>; <[email-masked]>
Cc: <[email-masked]>; "Weiqiang Sun" <[email-masked]>
Subject: Re: [PMOL] FW: [bmwg] Soliciting comments on GMPLS performance 
metrics draft inCCAMP

> Hi all,
>
>> Hi IPPMers and BMWGers,
>
>> Recently the LSP dynamic provisioning performance draft is adopted as a 
>> working item of CCAMP.
>
> Some comments, reading this as a non-expert on GMPLS.
>
> 3.4. Units.
>
>  I don't think this is quite correct: you start to measure at T1 and
>  wait for the setup to complete.  However, it doesn't, so at some point
>  you become bored and stop the measurement.  You report undefined.
>  However, one then has to report in the metrics parameters when the
>  measurement will be stopped.  This value should be added to the
>  metrics parameters.  (Note: this will apply to more metrics).
Yes I agree that this parameter is very important and should be carefully 
chosen, provided that different switching technologies may differ 
drastically in the delay when performing switching operations. For this 
reason we have a little bit discussion regarding this threshhold:
   o  A given methodology will have to include a way to determine
      whether a latency value is infinite or whether it is merely very
      large.  Simple upper bounds could be used.  But GMPLS networks may
      accommodate many kinds of devices.  For example, some photonic
      cross-connects (PXCs) have to move the micro mirrors.  This
      physical motion may take several milliseconds.  But the common
      electronic switches finish the nodal process within several
      microseconds.  So the unidirectional LSP setup delay varies
      drastically from a network to another.  In practice, the upper
      bound should be chosen carefully.
However, I think it is a good idea to put this parameter in the "metric 
parameter" section. This also makes the results more reasonable, since 
different threshholds may lead to different results.

>
> 4.3. Metrics Parameters.
>  In IPPM speak, Lamba refers to a rate but it also specifies a 
> distribution
>  for the interval between probe packets.  For example, poisson or 
> periodic.
>  The draft should mention which distribution to use.
We did define poisson arrival process for the lambda. However, it may be 
interesting to add some more arrival statistical models (with periodic as an 
example). We will consider this in the next revision.

>
> 12.1. Statistics.
>  I think you better report the 5th or 10th percentile of the 
> distributions,
Yes we do have the percentile of metric in 12.3, although we do not specify 
which exact value should be used.

>  rather than the minimum (and the same for the maximum).  This excludes
>  data points where something went wrong in the measurement process.
I think it is reasonable to keep the minimum value as this in normal cases 
can reflect the least amount of time needed to set up/release an LSP 
(something that is pretty much inherent for the control plane). And I agree 
that we don't need the maximum. Luckily enough we do not have one in the 
current version. :)

Anyway, I find a little mistake in 12.3:
   Given a metric and a percent X between 0% and 100%, the Xth
   percentile of all the dT values in the sample.  In addition, the
   unidirectional LSP setup delay percentile is undefined if the sample
   is empty.
It will be changed into:
   Given a metric and a percent X between 0% and 100%, the Xth
   percentile of all the dT values in the sample.  In addition, the
   percentile is undefined if the sample is empty.

>
> Henk
>
>
> -- 
> ------------------------------------------------------------------------------
> Henk Uijterwaal                           Email: 
> henk.uijterwaal(at)ripe.net
> RIPE Network Coordination Centre 
> http://www.amsterdamned.org/~henk
> P.O.Box 10096          Singel 258         Phone: +31.20.5354414
> 1001 EB Amsterdam      1016 AB Amsterdam  Fax: +31.20.5354445
> The Netherlands        The Netherlands    Mobile: +31.6.55861746
> ------------------------------------------------------------------------------
>
> Is one of the choices leaving the office open?
>                                       Alan Greenspan on the next elections
> 

_______________________________________________
PMOL mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/pmol

Date: Thu, 05 Jun 2008 12:04:09 +0200
From: Henk Uijterwaal <[email-masked]>
To: IETF IPPM WG <[email-masked]>, [email-masked]
Cc: Weiqiang Sun <[email-masked]>, [email-masked]
Subject: Re: [PMOL] FW: [bmwg] Soliciting comments on GMPLS performance metrics draft inCCAMP
Sender: [email-masked]

Hi all,

> Hi IPPMers and BMWGers,

> Recently the LSP dynamic provisioning performance draft is adopted as a 
> working item of CCAMP. 

Some comments, reading this as a non-expert on GMPLS.

3.4. Units.

  I don't think this is quite correct: you start to measure at T1 and
  wait for the setup to complete.  However, it doesn't, so at some point
  you become bored and stop the measurement.  You report undefined.
  However, one then has to report in the metrics parameters when the
  measurement will be stopped.  This value should be added to the
  metrics parameters.  (Note: this will apply to more metrics).

4.3. Metrics Parameters.
  In IPPM speak, Lamba refers to a rate but it also specifies a distribution
  for the interval between probe packets.  For example, poisson or periodic.
  The draft should mention which distribution to use.

12.1. Statistics.
  I think you better report the 5th or 10th percentile of the distributions,
  rather than the minimum (and the same for the maximum).  This excludes
  data points where something went wrong in the measurement process.

Henk

-- 
------------------------------------------------------------------------------
Henk Uijterwaal                           Email: henk.uijterwaal(at)ripe.net
RIPE Network Coordination Centre          http://www.amsterdamned.org/~henk
P.O.Box 10096          Singel 258         Phone: +31.20.5354414
1001 EB Amsterdam      1016 AB Amsterdam  Fax: +31.20.5354445
The Netherlands        The Netherlands    Mobile: +31.6.55861746
------------------------------------------------------------------------------

Is one of the choices leaving the office open?
                                       Alan Greenspan on the next elections
_______________________________________________
PMOL mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/pmol