Subject: [ippm] Metric requirements posed in Model-based-metrics

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Mon, 7 Oct 2013 09:11:09 -0400

Hi Matt,

While we (Joachim and I) were developing revisions to the RFC2330 update draft,
we considered the metric requirements you wrote in section 3 of the MBM draft:
http://tools.ietf.org/html/draft-ietf-ippm-model-based-metrics-00#section-3

We've had several exchanges about the three new requirements for metrics:

- Metrics must be actionable (test, repair, verify)
- Metrics must be vantage point-invariant
- Metrics must be repeatable by multiple parties

and felt this would be a topic to open-up with you and on the list.

All three are great ideals to have, but given the flexibility with which
IPPM metrics are usually specified it seems that these requirements are
more applicable to specific instances of metrics that are "ready to measure",
with a single methodology and a very clear purpose.  
IOW, 
s/Metrics/Measurements/

We also note some interactions between these requirements. For example,
a measurement which is "actionable" may only possess this property 
at a sub-set of vantage points. Also, some of the most meaningful 
vantage points will not necessarily be accessible by all parties.

Some specific comments on the three points (excerpts from our discussion) 
below. We'll be glad to provide more details as necessary.

regards,
Joachim and Al

Actionable: 
Although this correctly mentions ISPs, it is perfectly valid and
possibly more important for *users* who test to get information they can
use for identifying the cause and direct repairs or other mitigations.

Vantage point invariant: 
An example scenario which very likely does not fulfill this
requirement is a series of time-slotted networks. Just think about
Time-Slotted Randomness Cancellation in a mobile-to-mobile test or
round-trip measurements from a mobile with uplink and downlink being
time-slotted paths. Abstracting from one of these paths makes
simplifying assumptions which are not acceptable in the general case (as
demonstrated by Joachim's IEEE TIM paper, ref available, 
more detail in http://tools.ietf.org/html/draft-ietf-ippm-2330-update-00#section-3.4 ).
The requirement to consider the remainder of the network path as being
ideal is very, very hard in this context.
The requirements for an "ideal sub-path" requires
study that can likely only be completed when one understands
the specific measurement being made. So again we have a concept which
appears to apply to particular measurements, not metric definitions
applicable to a wide range of measurement scenarios.

Metrics must be repeatable by multiple parties:
As noted above, it's difficult to rely on the "ideal sub-path" 
concept to attain this feature. Alternatives, including remote
control with user-initiated measurements may be sufficient,
and the reality is that everyone needs a (remote) testing 
partner for active measurements.