Subject: Re: [PMOL] Informal pmol lunch session at IETF-78
From: "Romascanu, Dan (Dan)" <[email-masked]>
To: "Al Morton" <[email-masked]>, <[email-masked]>
Date: Mon, 28 Jun 2010 16:53:03 +0200
List-Id: Performance Metrics at Other Layers <pmol.ietf.org>

I would like to make sure that discussing the follow-up of pmol is part
of the 'other business'. We need to wrap up the long time due documents
in works and to make a recommendation about similar work will be made in
the IETF in the future. 

Dan

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Al Morton
> Sent: Monday, June 28, 2010 5:18 PM
> To: [email-masked]
> Subject: [PMOL] Informal pmol lunch session at IETF-78
> 
> Folks,
> 
> the pmol wg will meet informally for a (bring-your-own) lunch 
> on Friday, July 30, at 11:30 local time.
> 
> the meeting place will be announced later.
> 
> the Agenda includes the status of both our current drafts, 
> and any other business.
> 
> regards,
> Al
> pmol co-chair
> 
> _______________________________________________
> PMOL mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/pmol
> 

Date: Mon, 28 Jun 2010 10:18:13 -0400
To: [email-masked]
From: Al Morton <[email-masked]>
Subject: [PMOL] Informal pmol lunch session at IETF-78

Folks,

the pmol wg will meet informally for a (bring-your-own) lunch on
Friday, July 30, at 11:30 local time.

the meeting place will be announced later.

the Agenda includes the status of both our current drafts,
and any other business.

regards,
Al
pmol co-chair

From: Daryl Malas <[email-masked]>
To: Marian Delkinov <[email-masked]>, Robert Sparks <[email-masked]>
Date: Fri, 4 Jun 2010 09:06:40 -0600
Accept-Language: en-US
acceptlanguage: en-US
Cc: "[email-masked]" <[email-masked]>
Subject: Re: [PMOL] DISCUSS: draft-ietf-pmol-sip-perf-metrics

Mario,

Thanks for the feedback.  It appears I may have gone to far to the "basic
telephony" side, and I will change the metric names back to their -04 names.

Regards,

Daryl

On 6/4/10 6:09 AM, "Marian Delkinov" <[email-masked]> wrote:

>  
> Regarding the new issues with -05:
> 
> I have similar concerns about relabeling the following metrics:
> 
> Session Duration Time (SDT) --> Call Hold Time (CHT)
> 
> Although this metric corresponds to Call Holding Time in the telephony, this
> may lead to confusion that it's exactly the same thing. Still, here we refer
> to sessions that may not always relate to calls. So, I would prefer keeping
> the term "Session".
> 
> Session Establishment Ratio (SER) --> Answer Seizure Ratio (ASR)
> 
> The term "Seizure" is used in the telephony signalling. In SIP we have
> "Invite". If in the telephony we use "Answer to Seizure Ratio" to identify the
> % of answered vs all initiated calls, I strongly feel we should avoid using
> the same terms "Answer" and "Seizure" in SIP. If there is any real need to
> change the metric label, it would make more sense to use "Invite Success Rate"
> or similar. 
> 
> Session Establishment Effectiveness Ratio (SEER) --> Network Effectiveness
> Ratio (NER)
> The same thoughts: If we use NER, perhaps many will start comparing the
> measurement results with the TDM telephony, while here we have larger scope
> and different results.
> 
> In earlier Comments Robert wrote:
> 19 The ratio metrics don't define (or convey) the interval that totals
>    are taken over. Are these supposed to be "# requests received since
>    this instance was manufactured' or "since last reboot" or "since last
>    reset of statistics" or something else? What is the implementation
>    supposed to report when the denominator of a ratio is 0?
> 
> I propose a text to be inserted for all ratio metrics (may be valid for time
> metrics as well) specifying recommended measurements granularity of a 15 min
> interval. That interval can be used for benchmarking. All other granularities
> can also be supported, e.g. 1 min (for real time measurements), 3 min, 5 min,
> 30 min, 60 min etc. Thus various reports can later be generated based on these
> granularities. The same granularity intervals are widely used for the similar
> metrics in the TDM Telephony. How the counters are organized can be left to
> the manufacturers, but I also thought about the option to consider a standard
> MIB (like MIB II) for the counters that SIP end-to-end performance metrics are
> derived from.
> 
> At the end of chapter 4.7:
> "Reference the example flow is Section 4.7."
> Looks like the intention was to refer to Section 4.6?
> 
> Regards!
> Mario.
> 
> 
> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Robert
> Sparks
> Sent: Thursday, 03 June, 2010 16:57
> To: Daryl Malas
> Cc: [email-masked]
> Subject: Re: [PMOL] DISCUSS: draft-ietf-pmol-sip-perf-metrics
> 
> PDD is the one I feel strongly about.
> 
> If any of the others risk the same concerns, it would make sense to switch
> them back too.
> 
> RjS
> 
> On Jun 3, 2010, at 9:49 AM, Daryl Malas wrote:
> 
>> Hi Robert,
>> 
>> Is it your opinion I switch all of the metrics back to their original
>> names or just PDD?
>> 
>> Regards,
>> 
>> Daryl
>> 
>>> 
>>> ======= NEW ISSUES with -05 =======
>>> 
>>> 27 I think relabeling SRD as PDD is a mistake. At the very least it is
>>>   a change of such magnitude that it should be re-reviewed by the working
>>>   group. SRD as defined by -04 is similar, but not an exact reflection of
>>>   PDD. Saying it is "like" Post-Dial-Delay as defined in the PSTN is risky
>>>   enough. Using the same name makes it even more likely that someone will
>>>   come to the conclusion that it is measuring exactly the same thing. It
>>>   does not. For example, it fails to capture any delay from when
>>>   the user "finishes dialing" to when the INVITE is generated due to
>>>   DNS processing. It would be possible to engineer highly constrained
>>>   networks (that didn't use DNS or allow redirection or forking for example)
>>>   where the metric might behave very much like PDD in the PSTN, but that
>>>   will not be true in the general case.
>>> 
>>> 
>> 
> 
> _______________________________________________
> PMOL mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/pmol
> _______________________________________________
> PMOL mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/pmol

From: Marian Delkinov <[email-masked]>
To: Robert Sparks <[email-masked]>, Daryl Malas <[email-masked]>
Date: Fri, 4 Jun 2010 14:09:01 +0200
Accept-Language: en-US
acceptlanguage: en-US
Cc: "[email-masked]" <[email-masked]>
Subject: Re: [PMOL] DISCUSS: draft-ietf-pmol-sip-perf-metrics

Regarding the new issues with -05:

I have similar concerns about relabeling the following metrics:

Session Duration Time (SDT) --> Call Hold Time (CHT)

Although this metric corresponds to Call Holding Time in the telephony, this may lead to confusion that it's exactly the same thing. Still, here we refer to sessions that may not always relate to calls. So, I would prefer keeping the term "Session". 

Session Establishment Ratio (SER) --> Answer Seizure Ratio (ASR)

The term "Seizure" is used in the telephony signalling. In SIP we have "Invite". If in the telephony we use "Answer to Seizure Ratio" to identify the % of answered vs all initiated calls, I strongly feel we should avoid using the same terms "Answer" and "Seizure" in SIP. If there is any real need to change the metric label, it would make more sense to use "Invite Success Rate" or similar. 

Session Establishment Effectiveness Ratio (SEER) --> Network Effectiveness Ratio (NER)
The same thoughts: If we use NER, perhaps many will start comparing the measurement results with the TDM telephony, while here we have larger scope and different results.

In earlier Comments Robert wrote:
19 The ratio metrics don't define (or convey) the interval that totals
   are taken over. Are these supposed to be "# requests received since
   this instance was manufactured' or "since last reboot" or "since last
   reset of statistics" or something else? What is the implementation
   supposed to report when the denominator of a ratio is 0?

I propose a text to be inserted for all ratio metrics (may be valid for time metrics as well) specifying recommended measurements granularity of a 15 min interval. That interval can be used for benchmarking. All other granularities can also be supported, e.g. 1 min (for real time measurements), 3 min, 5 min, 30 min, 60 min etc. Thus various reports can later be generated based on these granularities. The same granularity intervals are widely used for the similar metrics in the TDM Telephony. How the counters are organized can be left to the manufacturers, but I also thought about the option to consider a standard MIB (like MIB II) for the counters that SIP end-to-end performance metrics are derived from.

At the end of chapter 4.7:
"Reference the example flow is Section 4.7."
Looks like the intention was to refer to Section 4.6?

Regards!
Mario.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Robert Sparks
Sent: Thursday, 03 June, 2010 16:57
To: Daryl Malas
Cc: [email-masked]
Subject: Re: [PMOL] DISCUSS: draft-ietf-pmol-sip-perf-metrics

PDD is the one I feel strongly about. 

If any of the others risk the same concerns, it would make sense to switch them back too.

RjS

On Jun 3, 2010, at 9:49 AM, Daryl Malas wrote:

> Hi Robert,
> 
> Is it your opinion I switch all of the metrics back to their original 
> names or just PDD?
> 
> Regards,
> 
> Daryl
> 
>> 
>> ======= NEW ISSUES with -05 =======
>> 
>> 27 I think relabeling SRD as PDD is a mistake. At the very least it is
>>   a change of such magnitude that it should be re-reviewed by the working
>>   group. SRD as defined by -04 is similar, but not an exact reflection of
>>   PDD. Saying it is "like" Post-Dial-Delay as defined in the PSTN is risky
>>   enough. Using the same name makes it even more likely that someone will
>>   come to the conclusion that it is measuring exactly the same thing. It
>>   does not. For example, it fails to capture any delay from when
>>   the user "finishes dialing" to when the INVITE is generated due to
>>   DNS processing. It would be possible to engineer highly constrained
>>   networks (that didn't use DNS or allow redirection or forking for example)
>>   where the metric might behave very much like PDD in the PSTN, but that
>>   will not be true in the general case.
>> 
>> 
> 

_______________________________________________
PMOL mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/pmol

Date: Thu, 03 Jun 2010 15:59:34 -0400
To: [email-masked]
From: Al Morton <[email-masked]>
Subject: [PMOL] Fwd: IETF 78 - Meeting and Sponsorship Information

>From: IETF Secretariat <[email-masked]>
>To: Working Group Chairs <[email-masked]>
>Subject: IETF 78 - Meeting and Sponsorship Information
>Date: Thu,  3 Jun 2010 12:51:40 -0700 (PDT)
>X-BeenThere: [email-masked]
>X-Mailman-Version: 2.1.9
>List-Id: Working Group Chairs <wgchairs.ietf.org>
>List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/wgchairs>,
>         <mailto:[email-masked]?subject=unsubscribe>
>List-Archive: <http://www.ietf.org/mail-archive/web/wgchairs>
>List-Post: <mailto:[email-masked]>
>List-Help: <mailto:[email-masked]?subject=help>
>List-Subscribe: <https://www.ietf.org/mailman/listinfo/wgchairs>,
>         <mailto:[email-masked]?subject=subscribe>
>Sender: [email-masked]
>
>Working Group Chairs,
>
>Can you please forward this message to your individual working group
>emails lists.  We want to ensure that as many people as possible are aware
>of the sponsorship opportunities available at IETF meetings.
>
>Thank you.
>=====================
>78th IETF Meeting
>Maastricht, Netherlands
>July 25-30, 2010
>
>1. Sponsorship Opportunities
>2. Registration Types
>3. Visas and Letters of Invitation
>4. Accommodations & Breakfast Information
>5. IETF 79 (Beijing) Visa Information
>
>1) Sponsorship Opportunities
>There are still sponsorship opportunities and benefits for high profile
>company/organizational exposure at the upcoming IETF meeting in
>Maastricht, Netherlands from July 25-30, 2010.  All sponsorship fees go
>directly to fund the operations of the IETF.  See the options at:
>http://www.ietf.org/meeting/78/index.html "Sponsorship Opportunities"
>under "General".  Each of the sponsorship options provide extended and
>repeat exposure at this weeklong meeting.
>
>Please contact Drew Dvorshak, [email-masked], with any questions
>and/or to reserve your organization's spot.  Thanks in advance for
>supporting the critical work of the IETF!
>
>2) Register online at:  http://www.ietf.org/meetings/78/
>
>3) Letters of Invitation and Visas
>Letters of Invitation
>After you complete the meeting registration process, you will be given the
>option to request a letter of invitation. You can request the Letter of
>Invitation as soon as you finish registration, or at a later time by
>following the link provided in the confirmation email.
>
>Visas
>Please check the Netherlands Ministry of Foreign Affairs site
>(http://www.minbuza.nl/en/Services/Consular_Services/Visa) for list of
>countries with visa exemptions.
>
>We recommend you give yourself at least one month to complete the visa
>process.
>
>4) Accommodations & Breakfast Information
>http://www.ietf.org/meeting/78/hotel.html
>
>5) If you want to start planning for your trip to IETF 79 in Beijing, we
>have visa information online here:
>http://www.ietf.org/meeting/79/visa.html
>
>Only 52 days until IETF 78!

From: Robert Sparks <[email-masked]>
Date: Thu, 3 Jun 2010 09:56:45 -0500
To: Daryl Malas <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Subject: Re: [PMOL] DISCUSS: draft-ietf-pmol-sip-perf-metrics

PDD is the one I feel strongly about. 

If any of the others risk the same concerns, it would make sense to switch them back too.

RjS

On Jun 3, 2010, at 9:49 AM, Daryl Malas wrote:

> Hi Robert,
> 
> Is it your opinion I switch all of the metrics back to their original names
> or just PDD?
> 
> Regards,
> 
> Daryl
> 
>> 
>> ======= NEW ISSUES with -05 =======
>> 
>> 27 I think relabeling SRD as PDD is a mistake. At the very least it is
>>   a change of such magnitude that it should be re-reviewed by the working
>>   group. SRD as defined by -04 is similar, but not an exact reflection of
>>   PDD. Saying it is "like" Post-Dial-Delay as defined in the PSTN is risky
>>   enough. Using the same name makes it even more likely that someone will
>>   come to the conclusion that it is measuring exactly the same thing. It
>>   does not. For example, it fails to capture any delay from when
>>   the user "finishes dialing" to when the INVITE is generated due to
>>   DNS processing. It would be possible to engineer highly constrained
>>   networks (that didn't use DNS or allow redirection or forking for example)
>>   where the metric might behave very much like PDD in the PSTN, but that
>>   will not be true in the general case.
>> 
>> 
> 

From: Daryl Malas <[email-masked]>
To: Robert Sparks <[email-masked]>
Date: Thu, 3 Jun 2010 08:49:35 -0600
Accept-Language: en-US
acceptlanguage: en-US
Cc: "[email-masked]" <[email-masked]>
Subject: [PMOL]  Re: DISCUSS: draft-ietf-pmol-sip-perf-metrics

Hi Robert,

Is it your opinion I switch all of the metrics back to their original names
or just PDD?

Regards,

Daryl

> 
> ======= NEW ISSUES with -05 =======
> 
> 27 I think relabeling SRD as PDD is a mistake. At the very least it is
>    a change of such magnitude that it should be re-reviewed by the working
>    group. SRD as defined by -04 is similar, but not an exact reflection of
>    PDD. Saying it is "like" Post-Dial-Delay as defined in the PSTN is risky
>    enough. Using the same name makes it even more likely that someone will
>    come to the conclusion that it is measuring exactly the same thing. It
>    does not. For example, it fails to capture any delay from when
>    the user "finishes dialing" to when the INVITE is generated due to
>    DNS processing. It would be possible to engineer highly constrained
>    networks (that didn't use DNS or allow redirection or forking for example)
>    where the metric might behave very much like PDD in the PSTN, but that
>    will not be true in the general case.
> 
>