Subject: RE: [ippm] Composition of Metrics

From: "Robert Holley \(rholley\)" <[email-masked]>
Date: Thu, 9 Jun 2005 14:21:58 -0400

Roman,

It seems like three (3) is a special case of two (2).

But I agree that this work should be done.

Regards
Bob 

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of [email-masked]
> Sent: Thursday, June 09, 2005 1:54 PM
> To: Maurizio Molina
> Cc: [email-masked]; Lei Liang; [email-masked]; STEPHAN 
> Emile RD-CORE-LAN; GN-JRA1-list
> Subject: Re: [ippm] Composition of Metrics
> 
> 
> 
> 
> 
> the composition of metrics is on everybody's mind ( a trivial 
> statement)
> and I guess the solution is elusive.
> I have seen two broad approaches  or maybe  three
> (1) Cannot be done and violates all known physical and 
> mathematical laws.
> (2) must be done and any reasonable method we agree on will do
> (3) some approximation to the end to end performance from sub-segments
> metrics  for practical purposes can be achieved using
> addition ( latency), probability based( packet loss) 
> aggregates that, while
> inaccurate, from the SP  perspective will do.
> 
> I am somehow leaning towards the option (3). There is in my 
> view an absence
> of hard guidelines how to do the recombination
> but I guess there is plenty of pragmatic approaches 
> implemented or used.
> And considering the nature of the problem
> pragmatic appoach is the best we can do. To this point, maybe 
> under  IPPM
> WG we should develop the document
> with requirements and a summary of pragmatic solutions from 
> different SPs.
> In my view such a document would certainly help all of us 
> involved to focus
> on some best practices or methods
> roman k
> 
> 
> 
> 
>                                                               
>                                                                  
>                       "Maurizio Molina"                       
>                                                                  
>                       <maurizio.molina@da        To:       
> "STEPHAN Emile RD-CORE-LAN" <[email-masked]>,      
>                       nte.org.uk>                 
> [email-masked]                                                 
>                
>                       Sent by:                   cc:       
> "Lei Liang" <[email-masked]>, "GN-JRA1-list"                  
>                       [email-masked]         
> <[email-masked]>                                       
>                
>                       rg                         Subject:  
> Re: [ippm] Composition of Metrics                                   
>                                                               
>                                                                  
>                                                               
>                                                                  
>                       06/09/2005 12:21 PM                     
>                                                                  
>                                                               
>                                                                  
>                                                               
>                                                                  
> 
> 
> 
> 
> Hi Emile,
> 
> I compared your two drafts, and summarizing I would say (correct me if
> I'm wrong) that:
> 
> 1) Type-P-METRIC-XX-trajectory in the old draft corresponds to the
> Type-P-METRIC-stream in the new one.
> 
> 2) The new draft contains the one-to-group metrics
> 
> 3) The asynchronous metrics were ruled out.
> 
> Regarding the last point, I understand that if you have a set of hosts
> <H0, H1, ...Hn> along a path and a set of delays <dT1, dT2, 
> ...dTn> obtained
> by *different* probes (thus asynchronous...) is too naïve to say that
> the delay's sum is a good representation of the singleton 
> one-way-delay
> on the path. The path delay and its partitioning in the different
> portions could of course be obtained if we had a trajectory metric,
> which is a good thing but is not given by any currently 
> widely deployed
> measurement protocol.
> 
> So, it looks like the IPPM WG preferred just to prepare the container
> for something that it is still to come and not to say 
> anything about how
> to compose existing metric. What I think would be useful is giving
> guidelines about the composition of metric *statistics* (e.g. the mean
> -> trivial - the variance or the percentiles -> not trivial 
> because the
> independence on the several path portions comes into play). I 
> know that
> testing such an independence hypothesis is difficult or impossible, is
> it a reason for not saying anything about how to compose metric
> statistics (which is something that operators would like to do)?.
> Furthermore, in the fourth bullet point of RFC 2330 there was exactly
> the statement that the specification of the composition would 
> come along
> with some reasoning about its correctness (see below).
> Apologies if I'm raising a point that has already been discussed and
> buried in te WG. In this case, I'd appreciate pointers to 
> mail archives,
> etc.
> Regards,
> Maurizio
> 
> STEPHAN Emile RD-CORE-LAN wrote:
> 
> >Dear Maurizio
> >
> >2 years ago I tried to introduce this point in the IPPM WG, see
> http://www.watersprings.org/pub/id/draft-stephan-ippm-spatial-
> metrics-01.txt
> . This ID defines metrics for spatial composition and spatial
> decomposition. This point was presented and discussed in the IPPM WG.
> >My understanding of the IPPM WG conclusion is that it is 
> quite tricky to
> define composition metrics respectful of the RFC2330 because 
> it is possible
> to prove that such composition is not deterministic.
> >
> >Consequently Lei and I are focusing on the definition of 
> spatial metrics
> to decompose end-to-end measures and one-to-group metrics in
> http://www.ietf.org/internet-drafts/draft-stephan-ippm-multime
> trics-00.txt.
> 
> >
> >
> >Regards
> >Emile
> >
> >
> >
> >>-----Message d'origine-----
> >>De : [email-masked] [mailto:[email-masked]] 
> De la part de
> >>Maurizio Molina
> >>Envoyé : mercredi 1 juin 2005 20:47
> >>À : [email-masked]
> >>Objet : [ippm] Composition of Metrics
> >>
> >>Dear all,
> >>In RFC 2330 (Framework for IP Performance Metrics), the Spatial and
> >>temporal composition of metrics is defined (see extract below).
> >>However, in none of the RFCs defining the metrics the 
> problem appears to
> >>be tackled. Could somebody clarify me if there is a reason why this
> >>piece of work has not been undertaken?
> >>Thanks,
> >>Maurizio
> >>
> >>
> >>9.1. Spatial Composition of Metrics
> >>.....
> >>   When the definition of a metric includes a conjecture 
> that the metric
> >>   across the path is related to the metric across the 
> subpaths of the
> >>   path, that conjecture constitutes a claim that the 
> metric exhibits
> >>   spatial composition.  The definition should then include:
> >> +    the specific conjecture applied to the metric,
> >> +    a justification of the practical utility of the composition in
> >>      terms of making accurate measurements of the metric 
> on the path,
> >> +    a justification of the usefulness of the composition 
> in terms of
> >>      making analysis of the path using A-frame concepts 
> more effective,
> >>      and
> >> +    an analysis of how the conjecture could be incorrect.
> >>
> >>9.2. Temporal Composition of Formal Models and Empirical Metrics
> >>
> >>......
> >>
> >>   When the definition of a metric includes a conjecture 
> that the metric
> >>   across the path at a given time T is related to the 
> metric across the
> >>   path for a set of other times, that conjecture 
> constitutes a claim
> >>   that the metric exhibits temporal composition.  The 
> definition should
> >>   then include:
> >>
> >> +    the specific conjecture applied to the metric,
> >> +    a justification of the practical utility of the composition in
> >>      terms of making accurate measurements of the metric 
> on the path,
> >>      and
> >> +    a justification of the usefulness of the composition 
> in terms of
> >>      making analysis of the path using A-frame concepts 
> more effective.
> >>
> >>
> >>
> >>
> >>_______________________________________________
> >>ippm mailing list
> >>[email-masked]
> >>https://www1.ietf.org/mailman/listinfo/ippm
> >>
> >
> >
> >
> 
> 
> 
> _______________________________________________
> ippm mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/ippm
> 
> 
> 

_______________________________________________
ippm mailing list
[email-masked] 
https://www1.ietf.org/mailman/listinfo/ippm

---

From: Al Morton <[email-masked]>
Date: Fri, 10 Jun 2005 08:42:22 -0400

>But I agree that this work should be done.

As Roman alluded, there is some work in progress.
I mentioned ITU-T work in my talk at IETF-62, it's
much along the lines of (2) and (3).  One of the areas
investigated there: what summary of the sample metrics collected
on each path segment will produce a decent estimate of e2e path?

I would encourage discussion of the topic here,
because there will likely be benefit derived from IPPM's
collective expertise. From the start, we should recognize the
tension between accuracy and practicality (as always),
and folks who need to combine metrics need to be clear
about the degree of inaccuracy they can live with.

In the spirit of contribution, I promise to summarize
what I've added/seen so far, in ID or slide-ware form.

(astonished that I've added another item to my ToDo list),
Al

---

From: [email-masked]
Date: Sat, 11 Jun 2005 12:21:44 -0400

I guess my concern is that if you let expediency drive the definitions, 
you can easily wind up with nonsense.

A good example of this is the common "high water mark" metric that you 
find often in MIBs. The information content that you get from this is 
not worth the cycles spent on it. How often does it occur? Out of how 
many samples? etc. etc.

Metric composition is difficult because of a number of factors, not the 
least of which is that often metrics on successive links or in 
successive networks on an end to end path are not independent. If all 
you are interested in are means and what you want to know is the mean 
of a sum of random variables (e.g. end to end delay), then no problem; 
add the means and that is perfectly valid.  If you want to approximate 
other compositions of metrics, though, you have to understand the best 
and worst that you can do with your approximation and the limitations 
(like when it is a reasonable approximation and when it is simply not 
valid) and doing that is something like work. If the WG is willing to 
do that work, then I say "wonderful" and let's get busy. If not, then I 
think that you are not only wasting your time, but at worst producing 
something seriously misleading.

My 2 cents...

Regards,
Phil Chimento

---

From: Al Morton <[email-masked]>
Date: Sat, 11 Jun 2005 17:45:27 -0400

At 12:21 PM 06/11/2005, [email-masked] wrote:
>If all you are interested in are means and what you want to know is the 
>mean of a sum of random variables (e.g. end to end delay), then no 
>problem; add the means and that is perfectly valid.  If you want to 
>approximate other compositions of metrics, though, you have to understand 
>the best and worst that you can do with your approximation and the 
>limitations (like when it is a reasonable approximation and when it is 
>simply not valid) and doing that is something like work. If the WG is 
>willing to do that work, then I say "wonderful" and let's get busy. If 
>not, then I think that you are not only wasting your time, but at worst 
>producing something seriously misleading.
Well said, Phil.

Almost everything we come-up with will be an approximation,
and this group can bring considerable scrutiny to bear on
the problem, assuming enough active contributors are interested.

Each metric will have its own challenges, and any composition
method should have a scope/application space.

We've heard interest from several people so far...

Anyone else?

Al

---

From: Henk Uijterwaal <[email-masked]>
Date: Mon, 13 Jun 2005 11:43:59 +0200

Speaking as chair...

>We've heard interest from several people so far...

Yes, there seems to be interest, so I think we should see if there are
sufficient volunteers to work on this.  We should also define the scope
of this work.   So, if you are interested and can spend time on this,
drop me a note.  I'll also put it on the agenda for Paris,

Henk

(who has been extremely busy and is neglecting IPPM a bit).

------------------------------------------------------------------------------
Henk Uijterwaal                           Email: henk.uijterwaal(at)ripe.net
RIPE Network Coordination Centre          http://www.amsterdamned.org/~henk
P.O.Box 10096          Singel 258         Phone: +31.20.5354414
1001 EB Amsterdam      1016 AB Amsterdam  Fax: +31.20.5354445
The Netherlands        The Netherlands    Mobile: +31.6.55861746
------------------------------------------------------------------------------

Look here junior, don't you be so happy.
And for Heaven's sake, don't you be so sad.                 (Tom Verlaine)

---

From: Lei Liang <[email-masked]>
Date: Thu, 16 Jun 2005 13:13:21 +0100

Hi, Henk,
  I would like to contribute in this topic as well.

Best regards,
Lei Liang