Subject: Re: [ippm] metric registry parameter names

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 1 Oct 2015 09:59:26 +0200

On Wed, Sep 30, 2015 at 05:06:11PM +0000, STARK, BARBARA H wrote:
> > I think it would be desirable if parameters of metrics have well defined
> > names. In section 4.3.5 of draft-morton-ippm-initial-registry-00.txt I see:
> > 
> >    o  Upper limit on Poisson distribution (values above this limit will
> >       be clipped and set to the limit value). (if fixed, Upper limit =
> >       30 seconds.)
> > 
> > So what is the name of this option if I want to configure it?
> 
> I agree. The problem here is that the referenced specification is a bit vague wrt to certain "parameters", and the registry is being used as a means of tightening up the specification. I think that's fine, but what I would recommend is that the Reference Definition field include the name and definition of all parameters that are not explicitly named and defined in the referenced spec. I would rather the Fixed and Run-time Parameters simply provide a parsable list of parameters (and their values for Fixed Parameters), with names taken either directly from the referenced spec or from a list of additional parameters defined in the Reference Definition. 
> 
> In the Act_IP_UDP_Round-trip_Delay_Raw_95th-percentile_Poisson example, the Reference Definition would need to include text along the lines of...
> 
> The following parameters are defined and/or provided default values here:
> "upper_limit" is the upper time limit on Poisson distribution (values above this limit will be clipped and set to the limit value). Default is "upper_limit"="30 seconds".
> "seq_num" is an 8-byte field inside the payload of the test packet. The Src host shall set the initial value of seq_num in the payload of the first test packet and increment it by 1 with each subsequent test packet.
> "timestamp" is an 8 byte field inside the payload of the test packet. The Src host shall set the timestamp value in each test packet payload.
> 
> Both "seq_num" and "timestamp" need to be listed as Run-time Parameters, and not Fixed.
> 
> BTW, the lambda definition in the registry sample may be wrong. RFC 2681 defines lambda as "a rate in reciprocal seconds" and 1/lambda as a time interval between packets; but the sample registry entry defines 1/lambda as a rate (packets/second). I think perhaps the registry parameter needs to be lambda and not 1/lambda.
> 
> And since there seems to be a need to tighten RFC 2681 wrt default value of lambda, I recommend putting such default recommendations also in Reference Definition, with the other parameter definitions and defaults.
> Default "lambda"="1 packet/second" 
>

The quoting is perhaps helpful but still subtle. I would rather have
something like this (inspired by good old man pages):

The following parameters are defined and/or provided default values
here:

 upper-limit: The upper-limit defines the time limit on Poisson
              distribution (values above this limit will be clipped
              and set to the limit value). Default of the upper-limit
              is 30 seconds.

 seq-num:     The seq-num is an 8-byte field inside the payload of the
              test packet. The source host shall set the initial value
              of seq_num in the payload of the first test packet and
              increment it by 1 with each subsequent test packet.
   
 timestamp:   The timestamp is an 8 byte field inside the payload
              of the test packet. The source host shall set the
	      timestamp value in each test packet payload.

[...]

For automation, it would be nice to have a machine readable format
that might also indicate other aspects such as data type and units or
precision where they make sense. But a rendering such as shown above
makes it easy for developers to identify the set of options and to
lookup specific definitions.

PS: I personally prefer to not use underscores but dashes instead (the
    command line tools on Unix-like systems I am familiar with tend to
    use dashes instead of underscores).

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Thu, 1 Oct 2015 15:29:37 +0000

> The quoting is perhaps helpful but still subtle. I would rather have something
> like this (inspired by good old man pages):
> 

In the Reference Definition where additional parameters are defined, I think this makes sense. BTW, I've been thinking about the default for parameters defined inside a registry entry, and think this probably does need to also be provided in the Run-time Parameters field. I could envision different registry entries that reference the same RFC using different defaults for registry-defined parameters.

> [...]
> 
> For automation, it would be nice to have a machine readable format that
> might also indicate other aspects such as data type and units or precision
> where they make sense. But a rendering such as shown above makes it easy
> for developers to identify the set of options and to lookup specific
> definitions.

I would like for Identifier, URI, Run-time Parameters, Fixed Parameters, and some field under Output to be highly machine-readable/parsable. I was actually wondering if we could just use JSON format for the parameter and output fields that need to be parsable. My expectation is that a developer would be able to code a single executable for all registry entries with URI starting with "urn:ietf:params:performance:metric:Act_IP_UDP_Periodic-var_UDP-Payload-142_One-way_Delay" (see section 7 of draft-morton-ippm-initial-registry for this example). The Controller (or whatever configures the Controller) could get the fixed and default parameter values from the registry entry in order to set these in the MA (for a measurement task). In looking at this particular example, that would also suggest that output-type (which can be "raw" or "summary") and calculation (which can be "Percentile95", "Mean", "Min", "Max", or "Std_Dev") need to be provided either as Fixed Parameters or as Output Parameters defined in an Output category column.

>     use dashes instead of underscores).

I'm not particular. As long as there are rules/conventions. I would prefer if we just pick some existing well-defined rules/conventions and point to them for the fields that need to be parsable. The other fields do also need conventions (like man pages) to make them easily human readable. I'll gladly support any commonly-used convention.
Barbara

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Thu, 1 Oct 2015 23:00:19 -0400

Hi Barbara and Juergen,

I'm sorry to join in late (and at late hours).  
Here's what I take away as additional considerations 
or requirements for the registry format (which must be 
reflected in the initial entries, of course):

Parameters (Fixed and Run-time) MUST have well defined names.
For Human readers, the hanging indent style will work, and the
names and definitions that do not appear in the Reference Method
Specification should appear in that column, with other additional
details.

Now, some other items (possibly) needing discussion/contribution:

The example, Act_IP_UDP_Round-trip_Delay_Raw_95th-percentile_Poisson,
includes requirements on the UDP Payload, that it contain a sequence
number, a time stamp, and padding. This aspect of the Fixed parameters
is a hold-over from our earliest examples and drafts.  
I suggest:
 - noting that the Reference Method requires sequence numbers to 
   establish packet correspondence.
 - noting that the chosen measurement protocol will dictate the format of 
   sequence numbers and time-stamps, if they are conveyed in the packet.
 - providing the payload size in octets as a Fixed parameter.
The sequence numbers or timestamps are not Run-time parameters.
They must be determined by the measurement process itself after the
Fixed and Runtime parameters are passed to that process.

Also, in the Act_IP_UDP_Round-trip_Delay_Raw_95th-percentile_Poisson example
(and others that were copied from this one, I somehow reversed the meaning of 
lambda when attempting to provide a simple data format (NTP short format, 32 bit), 
I couldn't format lambda easily as a rate, so went for 1/lambda as a time
(and the corresponding edits were screwed-up == thanks for catching that Barbara).

Default values for Run-time parameters: IPPM has never attempted to agree 
on defaults, for 1/lambda, for example.  From the earliest comparisons (1998?),
RIPE and Advanced didn't use the same values for lambda.  Also, some Run-time
parameters (Source IP address) cannot have meaningful defaults.  So, I propose
that parameters are either Fixed (with values) or Run-time (no value).

Automation/Machine parsing: We discussed this in Dallas, and I listened to the 
recording again last night. In the exchange between Marcelo and Barbara, I think 
there was agreement (for someone) to propose a JSON format for Fixed Parameters.
Today, Barbara expressed an interest in Index, URI, Fixed, Run-time, and some field 
under the Output category to be highly machine parse-able.  
I'd like to be sure that some machine is actually going to read all this 
material. We started the registry assuming that human implementers would 
need to read the entries. If there's a use case where machines access
the registry themselves, whenever they need to, I'm interested to hear about it.
"I, for one, welcome our new computer overlords." ;-/

Last for this thread: Naming conventions, with word separation.
Please make some referenceable suggestions: I don't think we need/want
to create anything new.

Last^2: It was clear to me that Barbara did some extensive background
reading into IPPM's literature/RFCs. Welcome to IPPM, Barbara, you're one
of us now.

regards,
Al

> -----Original Message-----
> From: ippm [mailto:[email-masked]] On Behalf Of STARK, BARBARA H
> Sent: Thursday, October 01, 2015 11:30 AM
> To: Juergen Schoenwaelder
> Cc: [email-masked]
> Subject: Re: [ippm] metric registry parameter names
> 
> *** Security Advisory: This Message Originated Outside of AT&T ***.
> Reference http://cso.att.com/EmailSecurity/IDSP.html for more
> information.
> 
> In the Reference Definition where additional parameters are defined, I
> think this makes sense. BTW, I've been thinking about the default for
> parameters defined inside a registry entry, and think this probably does
> need to also be provided in the Run-time Parameters field. I could
> envision different registry entries that reference the same RFC using
> different defaults for registry-defined parameters.
> 
> > [...]
> >
> I would like for Identifier, URI, Run-time Parameters, Fixed Parameters,
> and some field under Output to be highly machine-readable/parsable. I
> was actually wondering if we could just use JSON format for the
> parameter and output fields that need to be parsable. My expectation is
> that a developer would be able to code a single executable for all
> registry entries with URI starting with
> "urn:ietf:params:performance:metric:Act_IP_UDP_Periodic-var_UDP-Payload-
> 142_One-way_Delay" (see section 7 of draft-morton-ippm-initial-registry
> for this example). The Controller (or whatever configures the
> Controller) could get the fixed and default parameter values from the
> registry entry in order to set these in the MA (for a measurement task).
> In looking at this particular example, that would also suggest that
> output-type (which can be "raw" or "summary") and calculation (which can
> be "Percentile95", "Mean", "Min", "Max", or "Std_Dev") need to be
> provided either as Fixed Parameters or as Output
>   Paramet
>  ers defined in an Output category column.
> 
> I'm not particular. As long as there are rules/conventions. I would
> prefer if we just pick some existing well-defined rules/conventions and
> point to them for the fields that need to be parsable. The other fields
> do also need conventions (like man pages) to make them easily human
> readable. I'll gladly support any commonly-used convention.
> Barbara
> 
> _______________________________________________
> ippm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ippm

---

From: marcelo bagnulo braun <[email-masked]>
Date: Mon, 12 Oct 2015 10:04:04 +0200

So, trying to wrap this one.
I suggest to add:

In the fixed parameter section, the following text:

In any case, Parameters MUST have well defined names.
For Human readers, the hanging indent style will work, and the
names and definitions that do not appear in the Reference Method
Specification should appear in this column.

In the Packet Generation Stream section, it now reads:

<t>Stream Parameters: The names, values and formats of input factors
for each type of stream. For example, the average packet rate
and distribution truncation value for streams with
Poisson-distributed inter-packet sending times.</t>

In the Sampling distribution section:

<t>Parameters: The names, values and formats of parameters, if any.</t>

In the run time parameters section:

<t>Parameters MUST have well defined names.
For Human readers, the hanging indent style will work, and the
names and definitions that do not appear in the Reference Method
Specification should appear in this column</t>

would this address this issue?

Regards, marcelo

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Thu, 15 Oct 2015 19:37:48 -0400

Hi Marcelo,

This is *almost* a wrap -- my issue is with the 
"Packet Generation Stream" section.

you wrote:
> <t>Stream Parameters: The names, values and formats of input factors for
> each type of stream. For example, the average packet rate and
> distribution truncation value for streams with Poisson-distributed
> inter-packet sending times.</t>

But see one example in: 
https://tools.ietf.org/html/draft-morton-ippm-initial-registry-00#section-4.3.2

This section needs to include the reference to the stream generation
method, and that's usually where the stream parameters will be defined.
For Poisson, it's RFC 2330. For Periodic, it's RFC 3432.

AND, the parameters ONLY appear in the Fixed Parameters section, 
     OR in the Run-time Parameters section.

It would be redundant to have the parameters in Two sections, right?

So I suggest:

Packet Stream Generation <note word re-ordering, reads better to me>

   This section gives the Reference Method for generating the packet traffic that is the
   basis for measurement.  For IPPM metrics, this is called the Stream,
   and can easily be described by citing one of the relevant RFCs and
   any specific details, such as the selected method among several 
   alternatives.

For what it's worth, I plan to revise section 4 of the initial contents draft:
https://tools.ietf.org/html/draft-morton-ippm-initial-registry-00#section-4

and when we have agreement on the comments that have been raised in the last few 
weeks, I'll go back and bring the rest of the sections into sync.

regards,
Al

> From: ippm [mailto:[email-masked]] On Behalf Of marcelo bagnulo
> braun
> Sent: Monday, October 12, 2015 4:04 AM
> To: MORTON, ALFRED C (AL); STARK, BARBARA H; Juergen Schoenwaelder
> Cc: [email-masked]
> For Human readers, the hanging indent style will work, and the names and
> definitions that do not appear in the Reference Method Specification
> should appear in this column.
> 
> <t>Stream Parameters: The names, values and formats of input factors for
> each type of stream. For example, the average packet rate and
> distribution truncation value for streams with Poisson-distributed
> inter-packet sending times.</t>
> 
> For Human readers, the hanging indent style will work, and the names and
> definitions that do not appear in the Reference Method Specification
> should appear in this column</t>
> 
> > Here's what I take away as additional considerations or requirements
> > for the registry format (which must be reflected in the initial
> > entries, of course):
> >
> > For Human readers, the hanging indent style will work, and the names
> > and definitions that do not appear in the Reference Method
> > Also, in the Act_IP_UDP_Round-trip_Delay_Raw_95th-percentile_Poisson
> > example (and others that were copied from this one, I somehow reversed
> > the meaning of lambda when attempting to provide a simple data format
> > (NTP short format, 32 bit), I couldn't format lambda easily as a rate,
> > so went for 1/lambda as a time (and the corresponding edits were
> screwed-up == thanks for catching that Barbara).
> >
> > Default values for Run-time parameters: IPPM has never attempted to
> > agree on defaults, for 1/lambda, for example.  From the earliest
> > comparisons (1998?), RIPE and Advanced didn't use the same values for
> > lambda.  Also, some Run-time parameters (Source IP address) cannot
> > have meaningful defaults.  So, I propose that parameters are either
> Fixed (with values) or Run-time (no value).
> >
> > Automation/Machine parsing: We discussed this in Dallas, and I
> > listened to the recording again last night. In the exchange between
> > Marcelo and Barbara, I think there was agreement (for someone) to
> propose a JSON format for Fixed Parameters.
> > Today, Barbara expressed an interest in Index, URI, Fixed, Run-time,
> > and some field under the Output category to be highly machine parse-
> able.
> > I'd like to be sure that some machine is actually going to read all
> > this material. We started the registry assuming that human
> > implementers would need to read the entries. If there's a use case
> > where machines access the registry themselves, whenever they need to,
> I'm interested to hear about it.
> > reading into IPPM's literature/RFCs. Welcome to IPPM, Barbara, you're
> > one of us now.
> >
> > regards,
> > Al
> >
> >
> >
> >
> >> In the Reference Definition where additional parameters are defined,
> >> I think this makes sense. BTW, I've been thinking about the default
> >> for parameters defined inside a registry entry, and think this
> >> probably does need to also be provided in the Run-time Parameters
> >> field. I could envision different registry entries that reference the
> >> same RFC using different defaults for registry-defined parameters.
> >>
> >>> [...]
> >>>
> >>> that might also indicate other aspects such as data type and units
> >>> or precision where they make sense. But a rendering such as shown
> >>> above makes it easy for developers to identify the set of options
> >>> and to lookup specific definitions.
> >> I would like for Identifier, URI, Run-time Parameters, Fixed
> >> Parameters, and some field under Output to be highly
> >> machine-readable/parsable. I was actually wondering if we could just
> >> use JSON format for the parameter and output fields that need to be
> >> parsable. My expectation is that a developer would be able to code a
> >> single executable for all registry entries with URI starting with
> >> "urn:ietf:params:performance:metric:Act_IP_UDP_Periodic-var_UDP-Paylo
> >> ad- 142_One-way_Delay" (see section 7 of
> >> draft-morton-ippm-initial-registry
> >> output-type (which can be "raw" or "summary") and calculation (which
> >> can be "Percentile95", "Mean", "Min", "Max", or "Std_Dev") need to be
> >> prefer if we just pick some existing well-defined rules/conventions
> >> and point to them for the fields that need to be parsable. The other
> >> fields do also need conventions (like man pages) to make them easily
> >> human readable. I'll gladly support any commonly-used convention.
> >> Barbara
> >>

---

From: marcelo bagnulo braun <[email-masked]>
Date: Fri, 16 Oct 2015 08:56:50 +0200

below...

>       OR in the Run-time Parameters section.

i can understand that
but the streams do have associated parameters that need to be provisioned
These can be provisioned in the fixed or run time parameters i agree.

>     any specific details, such as the selected method among several

I would add something along these lines. The packet generation stream 
may require parameters such as the the average packet rate and 
distribution truncation value for streams with Poisson-distributed 
inter-packet sending times. In case such parameters are needed, they 
should be included either in the Fixed parmaeter column or in the run 
time parameter column, depending on whehter they will be fixed or will 
be an input for the metric.

would that make sense?

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Fri, 16 Oct 2015 07:32:01 -0400

more in-line... [ACM}

> Sent: Friday, October 16, 2015 2:57 AM
...
> but the streams do have associated parameters that need to be
> provisioned These can be provisioned in the fixed or run time parameters
> i agree.
[ACM] 
Good, thanks!

> be an input for the metric.
[ACM] 
Yes, I like it, thanks!

Al

> > and when we have agreement on the comments that have been raised in
> > the last few weeks, I'll go back and bring the rest of the sections
> into sync.
> >
> > regards,
> > Al
> >
> >> From: ippm [mailto:[email-masked]] On Behalf Of marcelo
> >> bagnulo braun
> >> for each type of stream. For example, the average packet rate and
> >>> The example,
> >>> Act_IP_UDP_Round-trip_Delay_Raw_95th-percentile_Poisson,
> >>> number, a time stamp, and padding. This aspect of the Fixed
> >>> parameters is a hold-over from our earliest examples and drafts.
> >>> I suggest:
> >>> example (and others that were copied from this one, I somehow
> >>> reversed the meaning of lambda when attempting to provide a simple
> >>> data format (NTP short format, 32 bit), I couldn't format lambda
> >>> easily as a rate, so went for 1/lambda as a time (and the
> >>> corresponding edits were
> >>> comparisons (1998?), RIPE and Advanced didn't use the same values
> >>> for lambda.  Also, some Run-time parameters (Source IP address)
> >>> cannot have meaningful defaults.  So, I propose that parameters are
> >>> either
> >>> Please make some referenceable suggestions: I don't think we
> >>> need/want to create anything new.
> >>>
> >>> Last^2: It was clear to me that Barbara did some extensive
> >>> background reading into IPPM's literature/RFCs. Welcome to IPPM,
> >>> Barbara, you're one of us now.
> >>>
> >>> regards,
> >>> Al
> >>>
> >>>
> >>>
> >>>
> >>>>> The quoting is perhaps helpful but still subtle. I would rather
> >>>>> have something like this (inspired by good old man pages):
> >>>>>
> >>>>>    seq-num:     The seq-num is an 8-byte field inside the payload
> of
> >> the
> >>>> In the Reference Definition where additional parameters are
> >>>> defined, I think this makes sense. BTW, I've been thinking about
> >>>> the default for parameters defined inside a registry entry, and
> >>>> think this probably does need to also be provided in the Run-time
> >>>> Parameters field. I could envision different registry entries that
> >>>> reference the same RFC using different defaults for registry-
> defined parameters.
> >>>>
> >>>>> [...]
> >>>>>
> >>>> machine-readable/parsable. I was actually wondering if we could
> >>>> just use JSON format for the parameter and output fields that need
> >>>> to be parsable. My expectation is that a developer would be able to
> >>>> code a single executable for all registry entries with URI starting
> >>>> with
> >>>> Controller) could get the fixed and default parameter values from
> >>>> the registry entry in order to set these in the MA (for a
> >>>> measurement
> >> task).
> >>>> output-type (which can be "raw" or "summary") and calculation
> >>>> (which can be "Percentile95", "Mean", "Min", "Max", or "Std_Dev")
> >>>> need to be provided either as Fixed Parameters or as Output
> >>>>     Paramet
> >>>>>       command line tools on Unix-like systems I am familiar with
> >>>>> tend
> >> to
> >>>> and point to them for the fields that need to be parsable. The
> >>>> other fields do also need conventions (like man pages) to make them
> >>>> easily human readable. I'll gladly support any commonly-used
> convention.
> >>>> Barbara
> >>>>

---

From: marcelo bagnulo braun <[email-masked]>
Date: Sat, 17 Oct 2015 08:49:16 +0200

ok, i made this change and i also made a similar change for the sampling 
distribution, that may also require parameters