Subject: [PMOL] Re: draft-morton-perf-metrics-framework-00

From: Al Morton <[email-masked]>
Date: Tue, 23 Oct 2007 22:28:20 -0400

Hi Brian,

The draft is intended for discussion on the <[email-masked]> list.

You've raised a worthwhile consideration for the draft, so
I have cc'd the pmol list for further discussion.

regards,
Al

At 06:24 PM 10/23/2007, Brian E Carpenter wrote:
>Al,
>
>I'm not sure where the draft is being discussed, so please feel
>free to add in any appropriate cc's, including your co-author,
>whose address is missing in the draft.
>
>This comment was triggered by seeing the draft charter for
>pmol, but I think it really belongs in the document and not in
>the charter.
>
>It's fairly clear for the traditional ippm/bmwg metrics what
>is meant by "all other things being equal." Once you start
>discussing metrics for the application level, as in pmol,
>that's much less clear. I believe that any definition of
>an upper layer metric should include a discussion of the extent
>to which lower layer performance must be held stable for the
>upper layer metric to be of value, and in fact what lower layer
>metrics MUST be measured simultaneously and what bounds on
>those metrics must be met. I'd envisage statements of the type
>
>   When metric Foobar is measured, metrics Barfoo and Foofoo must
>   be measured simultaneously between the same endpoints.
>   Results for Foobar must be accompanied by the results for Barfoo
>   and Foofoo. If the observed variance of Barfoo or Foofoo exceeds
>   <level>, the results for Foobar must be labelled as 'unreliable'.
>
>As a one word summary, this can be called 'dependencies', but
>I think the framework needs to be explicit about it.
>
>      Brian
>--
>Regards
>    Brian Carpenter
>    University of Auckland
>

_______________________________________________
PMOL mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/pmol

---

From: "Aamer Akhter (aakhter)" <[email-masked]>
Date: Wed, 24 Oct 2007 08:22:10 -0400

Al, et al,

> >   <level>, the results for Foobar must be labelled as 'unreliable'.

This makes sense. 

I wonder however, about the cases where the application (or L7 entity etc) that is aware of the underlying disruptive transport and is working around it. 

Example: the work of digital fountain. I think in these cases, the metrics for the underlying would be even more important/relevant, but perhaps the final result shouldn't be labeled unreliable.

Thoughts?

-- 
Aamer Akhter / [email-masked]
Ent & Commercial Systems, cisco Systems

> -----Original Message-----
> From: Al Morton [mailto:[email-masked]]
> Sent: Tuesday, October 23, 2007 10:28 PM
> To: Brian E Carpenter
> Cc: Nevil Brownlee; [email-masked]
> Subject: [PMOL] Re: draft-morton-perf-metrics-framework-00
> 
> Hi Brian,
>

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 25 Oct 2007 09:35:30 +1300

> Example: the work of digital fountain. I think in these cases, the metrics for the underlying would be even more important/relevant, but perhaps the final result shouldn't be labeled unreliable.

My thought is that the variance in the "upper" metric will be very hard to
understand if there is a large variance in the "lower" metrics. There seem
likely to be non-linear effects at work.  If the upper metric is being
used only as an observation (e.g. to check SLA conformance) that is one
thing, but if it's being used to make any kind of prediction or to analyze
the behaviour of the upper layer protocol, I think describing it as
unreliable is justified.

     Brian

---

From: "Loki Jorgenson" <[email-masked]>
Date: Thu, 25 Oct 2007 12:03:35 -0700

Brian/Aamer

"Unreliable" more likely reflects ignorance on our part, more than any
fundamental incoherence between upper and lower layer behaviors.  I
would posit that there exists a coupling between any two layers/metrics
that can be discovered and described.  The work of PMOL could include
the definition of such couplings.

Optimally, there would be an abstracted interface between layers that is
parameterized in terms of existing well-defined quantities - for
example, IPDV or One-way Loss Pattern.  Thus expressions of higher level
metrics (take MOS derived from E-model as an exemplar) can be mapped
explicitly (if not simply) back to underlying packet metrics.

Then the relationship between metrics such as associated variances are
explicitly defined by statistical means and not simply a subject of
off-hand and potentially arbitrary thresholds for "reliability".  Mind,
those coupling description may be quite complex in how they related and
compound underlying metrics.

This might also provide an explicit framework for application or
middleware behaviors to be expressed as translation functions within the
coupling context.  Accounting for adaptive behaviors for "network aware"
applications will otherwise be very challenging.

Of course, I may be just being wishful that such a thing is possible.
It certainly seems non-trivial.

Loki Jorgenson
Apparent Networks
t   604 433 2333 ext 105
m   604 250-4642

-----Original Message-----
From: Brian E Carpenter [mailto:[email-masked]] 
Sent: Wednesday, October 24, 2007 1:36 PM
To: Aamer Akhter (aakhter)
Cc: Nevil Brownlee; [email-masked]
Subject: Re: [PMOL] Re: draft-morton-perf-metrics-framework-00

On 2007-10-25 01:22, Aamer Akhter (aakhter) wrote:
> I wonder however, about the cases where the application (or L7 entity
etc) that is aware of the underlying disruptive transport and is working
around it. 
> 
> Example: the work of digital fountain. I think in these cases, the
metrics for the underlying would be even more important/relevant, but
perhaps the final result shouldn't be labeled unreliable.
> Thoughts?

My thought is that the variance in the "upper" metric will be very hard
to understand if there is a large variance in the "lower" metrics. There
seem likely to be non-linear effects at work.  If the upper metric is
being used only as an observation (e.g. to check SLA conformance) that
is one thing, but if it's being used to make any kind of prediction or
to analyze the behaviour of the upper layer protocol, I think describing
it as unreliable is justified.

     Brian