Subject: [ippm] Metric registry considerations

From: Joachim Fabini <[email-masked]>
Date: Wed, 31 Jul 2013 13:38:14 +0200

Dear group,

to detail on my point started at the microphone, please find below a 
rather formal approach to registries. Metric registries imho have a 
clearly hierarchical concept to which software engineering offers 
well-proven solutions - why do not reuse these? Starting from top to 
bottom in the list below, abstraction level moves from a theoretic 
concept to concrete implementation (I'll stick to the C++ naming in the 
following).

1. Metric _Templates_ (the Type_P stuff): abstract metrics, not 
depending on any specific parameter types. Eg: Type_P one-way-delay. 
These templates are highly generic meta-concepts, offering operations 
which act on an abstract type P (I really like that name! ;). These are 
the RFCs defining metrics.

2. Metric _Types_ (Instantiations of Templates): Metric Template (1) 
plus a set of parameter _types_ (names, definitions, which are mandatory 
to safeguard repeatability for the respective metric), i.e., protocol 
type (SIP, ICMP, whatever), packet size type, or packet size range type, 
source address type, destination address type, port type.

3. Metric _Instances_ (Variables in C++, Instances of Metric Types). 
These must define values for any of the parameters types specified.

The only metric which is "executable", i.e., safeguards the 
repeatibility condition and allows for unique identification, is #3. 
However, such a metric registry is illusory (must define all possible 
combinations of all parameters) (protocol type ICMPv4, payload size 
25-37 bytes, flags ..., etc.).

However, an intermediate solution between #2 and #3 could work and be 
usable in practice as reference registry. The key is to define 
acceptable default parameters (which have been already established in 
practice, see RTD ICMPv4 echo with 32/64 byte payload).

What is needed
A) an abstract registry (#2 above), which is NOT used/referenced by 
measurements, maintained by IPPM and stores all metrics defined by IPPM 
together with their parameter names (types) and descriptions which are 
known to (potentially) influence on results. I find it extremly 
important to make the decision on Type-P explicit. Otherwise we end up 
comparing apples and oranges (see the payload-dependent delay in my 
presentation as an example).

Ideally, we can agree in this registry on default values for some or all 
of these parameters and state mandatory pre-requisites.
Example:
ID-M1 Metric round-trip-delay. Parameters: (NetProtocol->default=IPv6, 
TransProtocol->default=ICMP, packetSizeRange->default=64-64, 
packetSizeRangeAlgorithm->default=none, Flags->default=none, 
interPacketDelayRange->default=1-1s, interPacketDelayAlgorithm=none, 
SourceAddress, DestinationAddress, SPort=none, DPort=none, 
AdditionalIPHeaders=none...)

B) However, this default set won't work stand-alone. Therefore for 
public use we must define a second registry. Here we can refine all 
metric types from A) and specialize them with other parameter values.

Example:
ID 1: Metric round-trip-delay-default (ref. ID-M1) ModParam: 
SourceAddress=mandatoryUser, DestinationAddress=mandatoryUser
ID 2: Metric round-trip-delay-default-IPv4 (ref. ID-M1) ModParam: 
Protocol:IPv4, ... (Description)
ID 3: Metric round-trip-delay-default-UPD (ref. ModParam: Protocol:UDP, 
SourceAddress=mandatory, DestinationAddress=mandatory, SPort=mandatory, 
Dport=mandatory, ... (Description)
ID 4: Metric round-trip-delay-default-SIP (ref. ModParam: Protocol:UDP, ..

For better reference the metric name should probably be unique and the 
protocol name part of the metric name (don't stick to that).

Concerning Benoit's remark on user requests "give me one 
round-trip-delay, one one-way delay": I think that one key to success is 
to define ALL parameters but at the same time provide as many default 
values as possible.
However, the user must be requested to EXPLICITLY mention all parameter 
value changes relative to the default set (if he needs special features).

The question is, whether:
a) we can list all relevant parameters for any metric type,
b) agree on default values for these parameters,
c) define a process on how "custom-tailored" metrics should be defined 
and referenced.

best regards
Joachim
PS: Just my unbiased view  on how I would tackle this topic with focus 
on usability while maintaing the main metric property (repeatability) 
and a common discussion ground.

---

From: <[email-masked]>
Date: Wed, 31 Jul 2013 14:57:25 +0200

Joachim,

thanks. Your proposal of defaults (in general) came to my mind after following this mornings discussion. They seem to be a reasonable way forward. But we should be careful to avoid an interpretation that defaults are the only way to derive useful measurements.

I think to recall that IPPM wanted to allow for utmost flexibility in measurement parametrisation where that wouldn't break the concepts of IPPM. I found no text in RFC23330. IPPM wanted to avoid a bias by any of the metric specifications.

The lack of "MUST" requirements in the metric specifications supports the point of giving as much as possible flexibility to those measuring.

But IPPM also demands that measurements must be repeatable. And here default definitions fit well, if they ensure repeatability under otherwise identical conditions.

Regards,

Ruediger

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Joachim Fabini
Sent: Wednesday, July 31, 2013 1:38 PM
To: [email-masked]
Subject: [ippm] Metric registry considerations

Dear group,

What is needed

Example:

best regards
Joachim
PS: Just my unbiased view  on how I would tackle this topic with focus
on usability while maintaing the main metric property (repeatability)
and a common discussion ground.
_______________________________________________
ippm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ippm