Subject: Re: [ippm] [Fwd: IPPM expert review request]

From: "Weiqiang Sun" <[email-masked]>
Date: Wed, 19 Aug 2009 15:28:17 +0800

More responses below:

RS4.
Should be more specific as to why this metric is no simple function of 
single uni-directional LSP Setup Delay metric, i.e. why can it not be 
deduced from single LSP Setup Delay?
[sunwq]
This point is raised regarding the sentence "The time needed to setup a 
large number of LSPs during a short time period can not be deduced by single 
LSP setup delay."
One reason is that when a large number of LSPs in being setup during a short 
period of time, the control plane may be crowded or even overwhelmed by the 
large number of signaling and routing messages. This may significantly slow 
down the processing of a particular signaling message, which will results in 
longer LSP provisioning delays.
Are you suggesting that we add similar explanations to the document? We'd 
love to do that if you feel it is necessary.

RS5.
The current definition only addresses multiple uni-directional LSP setups 
between two nodes ID0 and ID1. What about the the case when ID0 is to setup 
multiple uni-directional LSP setups to different egress nodes ID1, ID2, 
ID3,.,IDN?
[sunwq]
Yes, it is a useful case. Likewise you would expect a case to establish a 
varied number of (instead of one) LSPs to a set of destinations. Our 
suggestion is that this case is approached by integrating the defined 
methodologies of single/multiple LSPs provisioning delay, in the form of a 
particular testing case, rather than a formal testing methodology. This is 
in fact a trade-off between complexity and accuracy.

Best regards,
Weiqiang

--
Weiqiang Sun
Shanghai Jiao Tong University
http://front.sjtu.edu.cn/~sunwq/

--------------------------------------------------
From: "Weiqiang Sun" <[email-masked]>
Sent: Wednesday, August 19, 2009 2:48 PM
To: "Rschrage" <[email-masked]>; "'Lou Berger'" 
<[email-masked]>; "'Henk Uijterwaal'" <[email-masked]>
Cc: <[email-masked]>; "'BRUNGARD, DEBORAH A,	ATTLABS'" 
<[email-masked]>; "'IETF IPPM WG'" <[email-masked]>; <[email-masked]>
Subject: Re: [ippm] [Fwd: IPPM expert review request]

> Hi Reinhard,
>
> Many thanks for doing the careful review. We appreciate your many wording 
> suggestions. We will go through these one by one and adopt those we feel 
> appropriate. In this email I will not list all the comments since these 
> will not lead to major technical changes.
>
> Below I try to respond to the points you raised in the revision.
>
> RS1.
> This would imply that the metric may produce a range of several values at 
> one time with the minimum value to be taken from that range.
> I believe what the authors are trying to say is, that the LSP Setup Delay 
> Metric provides a lower bound on all possible LSP Delay Metrics of 
> actually instantiated LSPs, or in other words: an actual LSP Delay Metric 
> cannot get smaller than an LSP Setup Delay Metric.
> [sunwq]
> This point is raised regarding the motivation of the sigleton definition 
> of Single Uni-directional LSP Setup Delay (ie. 4.1, para 2). We are trying 
> to say that the minimum value of this metric reflects the (likely) single 
> lsp setup delay when the control plane is lightly loaded. The formal 
> definition of the minimum value is in "14.1 The minimum of metric"
> In fact we have inherited this from the IPPM documents (see RFC 2681, page 
> 2 section 1.1 bullet 3).
>
> RS2.
> How? Should there be a methodology be defined for this?
> [sunwq]
> This point is raised regarding the sentense in our methodologies 
> sections - "Make sure that the network has enough resource to set up the 
> requested LSP."
> We have assumed that test personnel should have adequate expertise in 
> allocating enough resources for the testing. The allocation of resources 
> for the testing purpose can be very test specific and we believe it is 
> quite outside the scope of this document.
>
> RS3.
> Too vague? As both timestamps are to be taken on the same ingress node, 
> they should be taken at the same application/network level.
> [sunwq]
> This point is raised regarding the sentense "If the corresponding RESV 
> message arrives within a reasonable period of time, take the timestamp 
> (T2) as soon as possible upon receipt of the message."
> Yes, ideally the timestamp should be taken immediately after the reception 
> of the RESV message. The wording "as soon as possible" is used to allow 
> for some flexibility when RESV message needs to be propagated to certain 
> modules where timestamp-taking is most appropriate.
> And again, this text is inherited from the IPPM documents (see RFC 2681 
> page 7, bullet 3).
>
> Thanks again and looking forward to your further comments and revisions.
>
> Weiqiang
>
> --
> Weiqiang Sun
> Shanghai Jiao Tong University
> http://front.sjtu.edu.cn/~sunwq/
>
> --------------------------------------------------
> From: "Rschrage" <[email-masked]>
> Sent: Tuesday, August 18, 2009 8:23 PM
> To: "'Lou Berger'" <[email-masked]>; "'Henk Uijterwaal'" <[email-masked]>
> Cc: <[email-masked]>; "'BRUNGARD, DEBORAH A, ATTLABS'" 
> <[email-masked]>; <[email-masked]>; "'IETF IPPM WG'" <[email-masked]>; 
> <[email-masked]>
> Subject: RE: [ippm] [Fwd: IPPM expert review request]
>
>> Hello,
>>
>> sorry for late response-
>>
>> Nevertheless here is a first edition of my revision of doc
>> draft-ietf-ccamp-lsp-dppm-06.
>>
>> The doc has been saved in Word 2003 format to easily track changes.
>> I am using Kaspersky Anti Virus 2010 software so I trust there should be 
>> no
>> unpleasant surprises when downloading the attached word doc.
>>
>>
>> I thought it would be advisable to first address the suggested comments 
>> and
>> questions upto and including the uni-directional LSP setup delay metric 
>> and
>> after that to continue with the bi-directional and sample definitions as
>> covered in the doc.
>>
>> Please let me have your thoughts on this.
>>
>> Many thanks.
>> Reinhard Schrage
>> Tel: +49 (0) 5137 909540
>> Mobile: +49 (0) 172 26.36.046
>> [email-masked]
>>
>> -----Original Message-----
>> From: [email-masked] [mailto:[email-masked]] On Behalf Of 
>> Lou
>> Berger
>> Sent: Dienstag, 28. Juli 2009 14:59
>> To: Henk Uijterwaal
>> Cc: [email-masked]; BRUNGARD, DEBORAH A, ATTLABS;
>> [email-masked]; IETF IPPM WG
>> Subject: Re: [ippm] [Fwd: IPPM expert review request]
>>
>> Hank/Reinhard,
>>
>> Thank you very much for undertaking this review.  Please cc
>> [email-masked] on any comments you may have.
>>
>> Lou
>>
>> On 7/28/2009 6:47 AM, Henk Uijterwaal wrote:
>>> Lou, others,
>>>
>>>> We received the request for a review of a document currently under
>>>> discussion in the CCAMP WG, please see below.  Is there anybody who
>>>> has time to do this review in the near future?
>>>
>>> Reinhard Schrage ([email-masked]) has voluntered to do 
>>> this.
>>>
>>> Reinhard: please post anything you find to both the list and the 
>>> authors.
>>> And thank you for doing this.
>>>
>>> Henk
>>>
>>>>
>>>> Matt & Henk
>>>>
>>>> -------- Original Message --------
>>>> Subject: IPPM expert review request
>>>> Date: Fri, 24 Jul 2009 15:57:41 -0400
>>>> From: Lou Berger <[email-masked]>
>>>> To: [email-masked]
>>>> CC: Brungard, Deborah A, ALABS <[email-masked]>, [email-masked],
>>>> zhangguoying <[email-masked]>
>>>>
>>>> Hi,
>>>>     We, the CCAMP WG chairs, would like to request that the IPPM WG
>>>> review
>>>> a draft that is progressing through the CCAMP WG.  This work applies
>>>> IPPM approaches to GMPLS.  The document we'd like reviewed is
>>>> available at:
>>>>
>>>> http://tools.ietf.org/html/draft-ietf-ccamp-lsp-dppm-06
>>>>
>>>> Is this acceptable?  Can you undertake this review?  Alternatively, we
>>>> can just last call the document in your WG (it has already passed CCAMP
>>>> WG LC).
>>>>
>>>> Thank you,
>>>> Lou (and Deborah)
>>>>
>>>>
>>>
>>>
>> _______________________________________________
>> ippm mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/ippm
>>

---

From: Lou Berger <[email-masked]>
Date: Wed, 19 Aug 2009 12:20:24 -0400

Tom,
	This document covers evaluation metrics that you'd expect to see
produced by a test tool rather than from the systems (DUTs) themselves.
 In short, no GMPLS network node is expected to implement this spec.

WRT the quoted text.  I viewed this as referring to a benchmarking tool
that is deployed in an operational network.  I can see how this can be
misinterpreted and I have no objection to asking that the text be
removed from the document.

Will dropping this text address your concerns?

Note that this document has been a WG document for well over a year and
was first discussed in the WG ~2.5 years ago.

Lou

On 8/19/2009 11:18 AM, Thomas D. Nadeau wrote:
>     Lou,
> 
>     I'd like to ask a question about this draft. How does it relate to (or
> not) to any of the GMPLS MIBs? As you know, the MIBs contain performance
> counters. If these metrics present in this draft do not coincide with the
> MIBs, this will cause more work for implementations. In particular, the
> following statement in this document worries me:
> 
>    On the other hand, it can also be used in operational environments for
>    carriers to monitor the control plane operation in real-time.  For
>    example, a new object can be added to GMPLS TE STD MIB [RFC4802] so
>    that the current and past control plane performance can be monitored
>    through network management systems.  The extension of TE-MIB to
>    support the defined metrics is outside the scope of this document.
> 
> 
>     Does the WG really want to progress a document that requires these
> changes?
> 
>     --Tom
> 
> 
> 
> On 8/19/09 9:57 AM, "Rschrage" <[email-masked]> wrote:
> 
>> Hi all,
>>
>> pls find attached a further updated edition of previous revised doc
>> I believe this is as far as we can go for the moment before a mutual
>> satisfactory understanding has been reached and a new draft has been issued
>> by the authors.
>>
>> Again, pls feel free to comment.
>>
>> Many thanks.
>> Brgds.
>> Reinhard Schrage
>> Tel: +49 (0) 5137 909540
>> Sent: Mittwoch, 19. August 2009 09:28
>> To: Rschrage; 'Lou Berger'; 'Henk Uijterwaal'
>> Cc: [email-masked]; [email-masked]; 'BRUNGARD, DEBORAH A,
>> ATTLABS'; 'IETF IPPM WG'
>> CCAMP mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/ccamp
>

---

From: "Rschrage" <[email-masked]>
Date: Wed, 19 Aug 2009 15:57:38 +0200

Hi all,

Many thanks.
Brgds.
Reinhard Schrage
Tel: +49 (0) 5137 909540

-----Original Message-----

More responses below:

RS4.

LSP setup delay."

RS5.

Best regards,
Weiqiang

---

From: "Thomas D. Nadeau" <[email-masked]>
Date: Wed, 19 Aug 2009 11:18:29 -0400

Lou,

    --Tom

-- 
Principal Architect - 21CN Networks

---

From: "Thomas D. Nadeau" <[email-masked]>
Date: Wed, 19 Aug 2009 15:15:06 -0400

> Will dropping this text address your concerns?

    Yes and no. For the current document I think yes, but from a wider WG
perspective, does the WG believe that MIBs can be used for the purposes of
measuring metrics (say as the target for a tool like you refer to above)?
Or does the WG feel new MIBs (or other APIs) should be developed to support
such a tool?  The worry I have, as an operator, is the introduction of such
metrics without a standardized means of getting at them in devices. This
will result in a hodgepodge of CLI-like mechanisms for anyone wishing to
produce a measurement tool, thus increasing the costs to operators wishing
to use said tools. 
 
> was first discussed in the WG ~2.5 years ago.

    That is a valid point, but as you know many documents unfortunately
don't get the attention they deserve until a LC is issued. :(

    --Tom

---

From: "Weiqiang Sun" <[email-masked]>
Date: Fri, 21 Aug 2009 14:58:38 +0800

Hi Reinhard,

Thanks for sending this part. See below for my responses.

[RS7]
Is there any help or suggestion on how to actually compute a usable 
Lambda_m?
[sunwq]
No. Depending on the performance of the implementation, the lambda_m can be 
very different, even under the same topology. A proper value of Lambda_m can 
only be tuned in the testing.

[RS8]
How to compare metrics that have been generated by two different Poisson 
processes, i.e. e.g. two different arrival rates?
[sunwq]
No way. Do we need to do that anyway?

[RS9]
These sample tuples are dependent on the arrival rate Lambda_m. How to 
deduct this input parameter from a resulting sample?
For instance, if two result sample sets are given, how can we be sure that 
they can be compared?
[sunwq]
My understanding of the sample is that it is a sequence of singleton values 
obtained under a set of parameters. The parameters, such as Lambda_m, are 
not visible in the sample. However, when reporting the sample, the 
parameters must also be reported.

[RS10]
A superposition of two different Poisson processes is being used here.
How do you map the resulting RESV messages to the correct process, i.e. how 
to differentiate e.g. a late RESV message resulting from third PATH message 
in fourth invocation from an early RESV message solicited by first PATH 
message in fourth invocation?
[sunwq]
RESV messages can be differentiated by either MSG ID or non-overlapping LSP 
ID, or combined. The RSVP-TE protocol does that automatically. So I believe 
this is an RSVP-TE implementation issue, rather than a testing methodology 
issue.

[RS11]
????
[sunwq]
Well, the full text is like: "The percentile of Metric is defined as: given 
a metric and a percent X between 0% and 100%, the Xth percentile of all the 
dT values in the sample." Would it help a bit if sentence is replaced by the 
full text above?
Please be noted that this text is inherited from the IPPM docs (See RFC 
2681, page 16, 4.1).

[RS12]
As stipulated in RFC2330 the EDF is defined as a function F(x) which for any 
x gives the fractional proportion of the total measurements that were <= x. 
(see RFC2330 11.3 p26)
Hence F(90)=0.25, F(100)=0.5,F(110)=0.75, F(500)=1 and therefore the 50th 
percentile is defined as 100 as opposed to 110.
[sunwq]
Yes you are right with regard to RFC 2330. But we have used exactly the same 
example as in RFC 2681 (page 16, section 4.1). The good thing is that the 
difference of the two calculations is very small when the number of 
measurement is large. Change needed?

And I realized that the points you have raised covered the main part of the 
text. We would expect your further comments regarding our responses. 
Otherwise we will assume that you are happy with it and will proceed to 
change the text based on the revision you sent in your previous email :)

Thank you again for your time in reviewing the draft so carefully.

Best regards,
Weiqiang

--------------------------------------------------
From: "Rschrage" <[email-masked]>
Sent: Wednesday, August 19, 2009 9:57 PM
To: "'Weiqiang Sun'" <[email-masked]>; "'Lou Berger'" <[email-masked]>; 
"'Henk Uijterwaal'" <[email-masked]>
Cc: <[email-masked]>; <[email-masked]>; "'IETF IPPM WG'" 
<[email-masked]>
Subject: Re: [CCAMP] [ippm] [Fwd: IPPM expert review request]

---

From: "Weiqiang Sun" <[email-masked]>
Date: Thu, 27 Aug 2009 14:25:34 +0800

Hi Reinhard,

Thanks for responding. Please see inline.

> [RS7a]
> I am a bit at loss here.
> It was my understanding that you use a metric to measure certain network
> quantities, e.g. throughput, set up delays, etc.
> In a second step you'd use these measured metrics to determine whether the
> systems under test meet your previously expected or defined quality
> parameters. If not, you'd make changes to the components or network 
> design,
> measure again, and compare the previous set of data to the one after the
> change and see whether you have improved and are moving into the right
> direction.
>
> You seem to be coming from the other end, i.e. tweak the input parameters 
> to
> suit your system under test.
[sunwq]
OK, I get your point here.
What you said is true. I had thought by saying ``a usable lambda_m", you are 
saying that a lambda_m that is *best achievable* by the DUT/SUT.
If you are expecting certain performance from the DUT/SUT, then very likely 
you will have in mind an "expected" lambda_m, won't you?
The methodology here does not suggest how to select lambda_m. This in fact 
can provide flexibility. Either the test will use an expected lambda_m, or 
tune the lambda_m that reflects the best system performance, depending on 
the purpose of the test. Does this make sense?

> [RS8a]
> Pls also see my comments above.
> If you cannot compare two sets of measurements how will you then arrive at
> an objective study of the network components under test?
[sunwq]
Maybe I misunderstood you again here. The Poisson arrival rate is an 
approximation of real traffic load. Of course it is worthwhile to find out 
how DUT/SUT performs when traffic load changes. But what else do you expect 
by comparing two data sets from different traffic load? Did I miss something 
here?

> [RS9a]
> OK. Noted and understood.
[sunwq]
Great, thanks.

> [RS10a]
> OK. Noted and understood.
[sunwq]
Great, thanks.

> [RS11a]
> The way it was written in the doc it was just a fragment of a sentence.
> Unfortunately RFC 2681, page 16, 4.1. also does not give a proper
> definition, as it too is just a fragmented sentence.
> RFC 2330, p.26, 11.3 gives a better explanation, as:
> "We will use the term 'percentile' to refer to the smallest value of x for
> which the empirical distribution function is greater or equal a given
> percentage"
[sunwq]
Well, to tell the truth I was troubled by this too. Later I realized that 
one reason that might lead to such a definition is that the authors of 2681 
had assumed that ``percentile" is a ``well known" term, and does not 
inheriting its definition from RFC 2330. And quite surprisingly wikipedia 
says "there is no standard definition of percentile." But wikipedia also 
says ``however all definitions yield similar results when the number of 
observations is large."

I will replace the original example with the following to avoid such a 
dilemma (of conflicting calculations in two RFCs).
============original text===============
   Example: suppose we take a sample and the results are: Stream1 = <
   <T1, 100 msec>, <T2, 110 msec>, <T3, undefined>, <T4, 90 msec>, <T5,
   500 msec> >

   Then the 50th percentile would be 110 msec, since 90 msec and 100
   msec are smaller, and 110 and 500 msec are larger (undefined values
   are not counted in).
============end of original text========

============new text===============
   Example: suppose we take a sample and the results are: Stream1 = <
   <T1, 100 msec>, <T2, 110 msec>, <T3, undefined>, <T4, 90 msec>, <T5,
   500 msec>, <T6, 350msec> >

   Then the 60th percentile would be 110 msec, since 90 msec and 100
   msec are smaller, and 110, 350 and 500 msec are larger (undefined values
   are not counted in).

   A more detailed definition of percentile may be found in RFC 2330.
============end of new text========

> [RS12a]
> As outlined above RFC 2330 as opposed to RFC2681 gives a proper definition
> of percentile. Therefore we should stick to RFC2330 and change 
> accordingly.
>
> It should be noted though, that also RFC2330 defines percentile 
> differently
> than it is done in the statistical literature, but we can live with this, 
> as
> it makes the definition easier to apply to measurement samples.
Good point, thanks for explaining. I hope the change above is satisfactory 
for you.

Thanks and best regards,
Weiqiang

>
> [------------------End of new comments---------------------]
>
>

---

From: "Weiqiang Sun" <[email-masked]>
Date: Thu, 27 Aug 2009 14:26:09 +0800

Hi Reinhard,

[sunwq]

[sunwq]

[sunwq]
Great, thanks.

[sunwq]
Great, thanks.

[sunwq]

Thanks and best regards,
Weiqiang

---

From: "Rschrage" <[email-masked]>
Date: Wed, 26 Aug 2009 17:04:40 +0200

Hi Weiqiang,

sorry for late reply-

My responses/comments follow below your points.

We seem to be coming from different view points

Brgds.
Reinhard Schrage
Tel: +49 (0) 5137 909540

-----Original Message-----
From: Weiqiang Sun [mailto:[email-masked]] 
Sent: Freitag, 21. August 2009 08:59
To: Rschrage; 'Lou Berger'; 'Henk Uijterwaal'
Cc: [email-masked]; [email-masked]; 'IETF IPPM WG'
Subject: Re: [CCAMP] [ippm] [Fwd: IPPM expert review request]

Hi Reinhard,

[RS7]

[RS7a]
I am a bit at loss here.

parameters. If not, you'd make changes to the components or network design,

You seem to be coming from the other end, i.e. tweak the input parameters to
suit your system under test.

[RS8]

[RS8a]

[RS9]

[RS9a]

[RS10]

[RS10a]

[RS11]
????
[sunwq]

full text above?

[RS11a]

[RS12]

[RS12a]
As outlined above RFC 2330 as opposed to RFC2681 gives a proper definition
of percentile. Therefore we should stick to RFC2330 and change accordingly.

It should be noted though, that also RFC2330 defines percentile differently
than it is done in the statistical literature, but we can live with this, as
it makes the definition easier to apply to measurement samples.

Best regards,
Weiqiang

---

From: "Rschrage" <[email-masked]>
Date: Mon, 31 Aug 2009 19:16:13 +0200

Hi Weiqiang,

thanks for commenting.
My responses inline as usual;

Brgds.
Reinhard Schrage
Tel: +49 (0) 5137 909540

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Weiqiang Sun
Sent: Donnerstag, 27. August 2009 08:26
To: Rschrage; 'Lou Berger'; 'Henk Uijterwaal'
Cc: [email-masked]; [email-masked]; 'IETF IPPM WG'
Subject: Re: [ippm] [CCAMP] [Fwd: IPPM expert review request]

Hi Reinhard,

[sunwq]

[RS]
OK - under the assumption of an experienced user.

[sunwq]

here?

[RS]
I was thinking along the lines of say starting with a reasonably large
lambda_m, get the metric sample for that and then subsequently e.g.
shortening lambda_m, comparing the resulting metric tuples. I was looking
for a guideline on how to possibly compare these resulting tuples.
As this is a stochastic process even shorter lambda_m values can still
generate longer interarrival times (surely with lower probability but still
possible). Makes sense?
[sunwq]
Great, thanks.

[sunwq]
Great, thanks.

[sunwq]

[RS]
OK then.

Thanks and best regards,
Weiqiang

---

From: "Weiqiang Sun" <[email-masked]>
Date: Tue, 1 Sep 2009 09:58:32 +0800

Hi Reinhard,

I appreciate your response. I am glad that we resolved all other concerns 
:).
Please see one last comment below.

> generate longer interarrival times (surely with lower probability but 

Larger lambda_m represents higher arrival rate hence will result in shorter 
average inter-arrival delay.
I guess what you want to say is that shorter lambda_m may generate shorter 
inter-arrival times. Yes it is true. But once we decided to get a ``sample", 
instead of a singleton value, then very likely we are interested in the 
system performance on average (and of course, in some other statistics of 
the sample), possibly averaged over a large number of singleton values. In 
this case each singleton value does not have much individual significance. 
This also can be explained by the fact the the processing of signaling 
messages on each NE are themselves random processes. When you generate the 
requests with constant inter-arrival time, you may still get singleton 
values that vary drastically. So it does not make sense to compare any two 
singleton values. However, once averaged over a large number of singleton 
values, the result is meaningful.

And I believe the same thing happens in IP performance measurement. For 
example, in RFC 2681, Poisson process is suggested but no guideline of how 
to compare two samples is provided. It think it makes sense that the 
documents provides methodologies on how to perform a measurement, and leaves 
the task of interpreting the results to the testers, or the owners of 
DUT/SUTs.

Hope this helps!

Best,
Weiqiang