Subject: [ippm] Alvaro Retana's Discuss on draft-ietf-ippm-metric-registry-22: (with DISCUSS and COMMENT)

From: Alvaro Retana via Datatracker <[email-masked]>
Date: Tue, 03 Dec 2019 14:37:29 -0800

Alvaro Retana has entered the following ballot position for
draft-ietf-ippm-metric-registry-22: Discuss

When responding, please keep the subject line intact and reply to all
email addresses included in the To and CC lines. (Feel free to cut this
introductory paragraph, however.)

Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
for more information about IESG DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-ippm-metric-registry/

----------------------------------------------------------------------
DISCUSS:
----------------------------------------------------------------------

I have two separate issues that I would like to DISCUSS.

(1) Approval of the initial performance metrics entries (in
draft-ietf-ippm-initial-registry).

This document describes the format of the registry, and the initial entries are
defined in draft-ietf-ippm-initial-registry.  However, the registration policy
of Specification Required would not be met if the entries in
draft-ietf-ippm-initial-registry are approved without expert review.

As I mentioned in my ballot for draft-ietf-ippm-initial-registry, I believe
that because both documents are being processed at the same time, and the new
entries have been reviewed by the WG, IESG Approval [rfc8126] can be used.

I can think of at least three ways to address this DISCUSS point (there may be
others):

a. Designated Experts for this document can be assigned and the formal review
can be done.

b. The text in this document can explicitly say that the entries in
draft-ietf-ippm-initial-registry are to be approved using IESG Approval.

c. The Responsible AD can add a Management Item to the Telechat for the IESG to
explicitly approve (beyond approval for the publication of
draft-ietf-ippm-initial-registry) the new entries.

I am ok with either choice, but would prefer Option c because it would be
faster and cause less churn.

(2)

§8.1 (Adding new Performance Metrics to the Performance Metrics Registry)
defines the following process for entries that are "not yet widely deployed":

   If the proposed registry entry is defined in an RFC but is not yet
   widely deployed, there SHOULD be a statement in the RFC that says the
   proposed registry entry is not ready for registration, and use SHOULD
   employ a private/experimental ID.  It is the responsibility of the
   document authors to submit the request to IANA when the proposed
   registry entry is ready for official registration.

Considering the Specification Required policy and the fact that the RFC has
already gone through all the reviews required for publication (including expert
review, as mentioned in the same section), how will it work for the "authors to
submit the request to IANA when the proposed registry entry is ready for
official registration"?  What specification will be presented to IANA to
satisfy the registration requirement?   It seems to me that the statement
mentioned above would prevent the official registration in the first place, and
that same statement (still present in the RFC) should prevent a second review
of the same document from resulting in an official registration.

This process needs more discussion and clarity for it to work.

[Not part of this DISCUSS point, but related.]

a. Section 8 talks about instructions about handling of the registry.  Perhaps
it should be part of the IANA Considerations section.

b. §7.1.1 contains additional instructions for IANA, including the reservation
of a private/experimental range of Identifiers.  This test should also be part
of the IANA Considerations section.

----------------------------------------------------------------------
COMMENT:
----------------------------------------------------------------------

(1)  I don't understand why this document is a BCP and not on the Standards
Track.  The Shepherd writeup says that the status "is best current
practice...as explained in the draft", but the document only justifies it this
way:

   Based on [RFC8126] Section 4.3, this document is processed as Best
   Current Practice (BCP) [RFC2026].

rfc8126/§4.3 talks about the Hierarchical Allocation registration policy.  I'm
lost.

It does seem strange that this document is classified as a BCP when it is
defining a registry...when many other documents serving the same function are
simply on the Standards Track.  It would be ideal if there was an actual
justification.

Related to being a BCP.  Should this document be part of BCP 170?

(2) Nits:

s/any other organization that wishes to create a Performance Metrics Registry
MAY use the same formatting specifications/any other organization that wishes
to create a Performance Metrics Registry may use the same formatting
specifications      There is no Normative value, just stating a fact.

s/Expert Review [RFC8126]policy/Expert Review [RFC8126] policy

s/Submission to IANA MAY be during IESG review/Submission to IANA may be during
IESG review     Just stating a fact...no Normative value.

---

From: Barry Leiba <[email-masked]>
Date: Wed, 4 Dec 2019 09:57:46 -0500

Hi, Álvaro.  On the first DISCUSS point, I don't see an issue.  If it
had all been done in one document it would be unremarkable, and we
would not think that a formal review by the DE would be needed.  In
this case, it's clear that this is a single product of the working
group that has, for editorial convenience, been split into two
documents.  I think there's neither need nor value in worrying about
it further for these initial registrations.

But...
For future registrations let me suggest that there's a fourth
alternative to your list.  Is it the working group's intent that if an
IETF-consensus document should want to make a future registration --
say, another document out of the IPPM WG -- it would have to go
through expert review even though it has working group consensus, and,
in theory, the DE could override WG consensus?

If so, then we're fine as we are.  But if not, then:

d. The registration policy could be set as "Specification Required or
IETF Review".  We use that in a number of registries, and it means
that an IETF-consensus document can register items without formal
expert review, though, of course, we would expect the DE(s) to
participate in forming the IETF consensus on it.  But specifications
that do not have IETF consensus -- from other RFC streams or non-RFC
sources -- would have to get a formal DE review.

The working group might consider whether this is or isn't what they
want before we cast the policy in stone.

Barry

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Wed, 4 Dec 2019 15:31:30 +0000

Hi Alvaro,

Thanks for your review and comments. 

Please see replies marked [acm] below,
Al

> -----Original Message-----
> From: Alvaro Retana via Datatracker [mailto:[email-masked]]
> Sent: Tuesday, December 3, 2019 5:37 PM
> To: The IESG <[email-masked]>
> Cc: [email-masked]; [email-masked];
> [email-masked]; [email-masked]
> Subject: Alvaro Retana's Discuss on draft-ietf-ippm-metric-registry-22:
> (with DISCUSS and COMMENT)
> 
> Please refer to https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__www.ietf.org_iesg_statement_discuss-2Dcriteria.html&d=DwIDaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=EtTYlWuZoXD0BCznVTc0bk6Y3qChy
> oUO9dp0prUdtp0&s=v3ONAc_XSQinjyxQv6WtNdhAF5xxs0vstrGaYRWe13M&e=
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__datatracker.ietf.org_doc_draft-2Dietf-2Dippm-2Dmetric-
> 2Dregistry_&d=DwIDaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=EtTYlWuZoXD0BCznVTc0bk6Y3qChy
> oUO9dp0prUdtp0&s=1f72Np2tfL7sqsWqx4tt_RTCu4bLV7znBDNprUiTcl4&e=
> 
> 
> 
[acm] 
C. is my preference too, although it is worth noting that Mirja has
begun to form the Experts Group, with two of the initial-registry
authors as the Experts.

> This process needs more discussion and clarity for it to work.
[acm] 
Yes, I see your point. This a paragraph we added at the IANA desk 
during IETF-106, to resolve a question raised, and it seems we didn't
read it critically. Let me suggest several revisions as a paragraph 
below:

If the proposed registry entry is defined in an RFC but is not yet 
widely deployed, there SHOULD be a statement in the RFC that says the 
proposed registry entry is not ready for registration because it lacks 
deployment (in the IANA Considerations Section), and SHOULD employ a 
private/experimental ID when used in practice. The IANA and Expert 
reviews of the proposed registry entry take place as normally required, 
and registration SHALL be held. When the authors are ready to show 
evidence of widespread deployment, they SHOULD submit the evidence 
to IANA to be evaluated in consultation with the Performance Metric 
Experts for registration at that time.

> a. Section 8 talks about instructions about handling of the registry.
> Perhaps it should be part of the IANA Considerations section.
[acm] 
Section 8 contains instructions for all the actors, Experts, etc. 
but perhaps we can make that clear with a change to the title:
"Processes for Managing the Performance Metric Registry Group"

> of a private/experimental range of Identifiers.  This test should also be
> part of the IANA Considerations section.
[acm] 
OK, all the details in 7.1.1 are now part of Section 10.3, for Identifiers.

> practice...as explained in the draft", but the document only justifies it
> this way:
> 
> rfc8126/§4.3 talks about the Hierarchical Allocation registration policy.
> I'm lost.
> 
> defining a registry...when many other documents serving the same function
> are simply on the Standards Track.  It would be ideal if there was an actual
> justification.
[acm] 
Aha, I finally sorted this out.  8126 obsoleted 5226, and changed the 
outline completely. (someone said 5226 is obsolete,
the number was changed and the requirement section was not checked, 
mea culpa).

RFC 5226 said:
4.  Creating a Registry
...
4.3.  Updating IANA Guidelines for Existing Registries

   Updating the registration process for an already existing (i.e.,
   previously created) namespace (whether created explicitly or
   implicitly) follows a process similar to that used when creating a
   new namespace.  That is, a document is produced that makes reference
   to the existing namespace and then provides detailed guidelines for
   handling assignments in each individual namespace.  Such documents
   are normally processed as Best Current Practices (BCPs)
   [IETF-PROCESS].

and this memo was written while 5226 was In-force. It now seems a bit 
tenuous to have used this section as the justification for BCP, but
many other opinions were involved over time.

I don't see a similar requirement in 8126.

I don't think anyone would have a problem if this memo is Standards Track now.

I am deleting the sentence you cited above (at end of section 3)
and changing the status in the working version.

> specifications      There is no Normative value, just stating a fact.
[acm] 
OK

> s/Expert Review [RFC8126]policy/Expert Review [RFC8126] policy
[acm] 
This is left-over from the pre-IETF-106 discussion with IANA.
The agreed policy is now Specification Required (which includes
Expert Review).

> IESG review     Just stating a fact...no Normative value.
[acm] OK

All these changes implemented in the working version.

---

From: Alvaro Retana <[email-masked]>
Date: Wed, 4 Dec 2019 08:43:34 -0800

On December 4, 2019 at 10:31:42 AM, MORTON, ALFRED C (AL) wrote:

Al:

Hi!

For the approval of the initial registry point, I think we'll need
Mirja to comment.

...
> > already gone through all the reviews required for publication (including
> > expert review, as mentioned in the same section), how will it work for the
> > "authors to submit the request to IANA when the proposed registry entry is
> > ready for official registration"? What specification will be presented to
> > IANA to satisfy the registration requirement? It seems to me that the
> > statement mentioned above would prevent the official registration in the
> > first place, and that same statement (still present in the RFC) should
> > prevent a second review of the same document from resulting in an official
> > registration.
> >

This new text is not explicit on what "evidence of widespread
deployment", nor whether that would be considered a specification in
line with the registration policy.  It also suffers from the point
that Alissa mentioned in her DISCUSS of leaving the published RFC with
the "old" text saying that the metric is not ready.

Thanks for addressing my other comments.

Alvaro.

---

From: Mirja Kuehlewind <[email-masked]>
Date: Wed, 4 Dec 2019 17:51:39 +0100

Hi Alvaro,

I just added this afternoon a management item to approve the experts for this registry and both experts are authors of draft-ietf-ippm-metric-registry, so I assume we can be sure that they did a review :-) I would think we don’t need any additional process/separate management item here. But we can also do that if that’s better for you…?

@Al: We changed the registration policy recently but didn’t we also talk about making it “Specification required or IETF Review”? Can you remind me of the outcome of that discussion? I guess the goal was to also have the experts to look at the details of the registry even when we publish an RFC, however, it should probably not be the case that the experts can block the registration of an assignment in an RFC, no? Or is this exactly covering the case discussed below where an RFC is published but the metric is not widely deployed yet?

Mirja

> On 4. Dec 2019, at 17:43, Alvaro Retana <[email-masked]> wrote:
> 
> ....

---

From: Alvaro Retana <[email-masked]>
Date: Wed, 4 Dec 2019 09:50:27 -0800

On December 4, 2019 at 11:51:42 AM, Mirja Kuehlewind wrote:

Mirja:

Hi!

> I just added this afternoon a management item to approve the experts for this
> registry and both experts are authors of draft-ietf-ippm-metric-registry, so
> I assume we can be sure that they did a review :-) I would think we don’t
> need any additional process/separate management item here. But we can also do
> that if that’s better for you…?

Approving the experts is fine with me -- then it is up to IANA how the
initial entries are approved.

Thanks!

Alvaro.

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Wed, 4 Dec 2019 18:22:04 +0000

Hi Mirja and Alvaro,

please see below, thanks!
Al

> From: Mirja Kuehlewind [mailto:[email-masked]]
> Sent: Wednesday, December 4, 2019 11:52 AM
> To: Alvaro Retana <[email-masked]>
> Cc: MORTON, ALFRED C (AL) <[email-masked]>; The IESG
> <[email-masked]>; [email-masked]; draft-ietf-ippm-metric-
> [email-masked]; [email-masked]; [email-masked]
> Subject: Re: Alvaro Retana's Discuss on draft-ietf-ippm-metric-registry-
> 22: (with DISCUSS and COMMENT)
> 
> Hi Alvaro,
> 
> I just added this afternoon a management item to approve the experts for
> this registry and both experts are authors of draft-ietf-ippm-metric-
> registry, so I assume we can be sure that they did a review :-) I would
> think we don’t need any additional process/separate management item here.
> But we can also do that if that’s better for you…?
> 
> @Al: We changed the registration policy recently but didn’t we also talk
> about making it “Specification required or IETF Review”? Can you remind me
> of the outcome of that discussion? 
[acm] 
We decided on "Specification Required"  which includes Expert Review,
and covers IETF RFCs and Specifications from other SDOs.

> I guess the goal was to also have the
> experts to look at the details of the registry even when we publish an
> RFC, however, it should probably not be the case that the experts can
> block the registration of an assignment in an RFC, no? Or is this exactly
> covering the case discussed below where an RFC is published but the metric
> is not widely deployed yet?
[acm] 
The Experts have to evaluate the completeness of the Registry entry proposed.
Also, all the Criteria in section 5 must be met.

Experts don't necessarily block an RFC-to-be for their registry entry, 
but the incomplete registry entry would have to be re-worked and re-submitted.
Also, deployment is part of the criteria.

I'm trying to conjure some new text that says this for the draft...

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Wed, 4 Dec 2019 18:37:52 +0000

Hi Alvaro and Alissa,

I'll give this paragraph one more try, taking your comments
into account...  please see below, and feel free to edit!

Al

> From: Alvaro Retana [mailto:[email-masked]]
> Sent: Wednesday, December 4, 2019 11:44 AM
> To: MORTON, ALFRED C (AL) <[email-masked]>; The IESG <[email-masked]>
> Cc: [email-masked]; [email-masked]; [email-masked]; draft-ietf-
> [email-masked]
> Subject: RE: Alvaro Retana's Discuss on draft-ietf-ippm-metric-registry-
[acm] 

If an RFC-to-be includes a Performance Metric and a proposed Performance 
Metrics Registry entry, but the IANA and Performance Metric Expert review 
determines that one or more of the Section 5 criteria have not been met, 
then the IESG approval process MUST proceed with the proposed Performance 
Metrics Registry entry removed from the text. When the RFC-to-be authors 
are ready to show evidence of meeting the criteria in section 5, they 
SHOULD re-submit the proposed Performance Metrics Registry entry to IANA 
to be evaluated in consultation with the Performance Metric Experts for 
registration at that time.

> Thanks for addressing my other comments.
[acm] 
you're welcome!

---

From: Alvaro Retana <[email-masked]>
Date: Wed, 4 Dec 2019 14:34:58 -0800

On December 4, 2019 at 1:38:07 PM, MORTON, ALFRED C (AL) wrote:

> registration at that time.

This text basically says that if the criteria in §5 is not met, then
the specific entry must not be in the RFC.  At some point in the
future (when the §5 criteria is met), publication of the entry can
proceed -- presumably in a different RFC.

As Alissa mentioned in her DISCUSS, the text needs to be generalized
to cover specifications from other SDOs.  I'm not sure how preventing
publication would work there.

The criteria in §5 talks about "industry interest, or has seen
deployment", which makes me think about how early allocation would
work.  If interest/deployment is part of the criteria to be in the
registry, then early allocation "to facilitate desired or required
implementation and deployment experience" [rfc7120] would seem to not
meet the requirement.  Wouldn't it?

Do you (authors/WG) see value in documenting private/experimental
metrics?  I imagine that one way to prove interest/deployment is to
experiment (vs early allocation).  These would not pass the §5 hurdle,
but may need to be documented somehow.  In the IETF the WG could limit
publication to a draft (for example), but in other SDOs an actual
specification could end up being published...

Sorry to add more questions; I just want to make sure all the bases are covered.

Thanks!

Alvaro.

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Thu, 5 Dec 2019 12:25:02 +0000

HI Alvaro,
I'll try now to answer your questions.
Al

> Sent: Wednesday, December 4, 2019 5:35 PM
[acm] 
Yes.
> publication would work there.
[acm] 
It doesn't apply to other SDOs.
There are process points that only apply to IETF and RFCs-to-be,
such as the one we are discussing.
IANA can receive a request from other SDOs directly, and
we cover those cases separately. IWO, we do not generalize
every instance of RFC to "spec", because IANA review 
coincides with IIESG review.

> meet the requirement.  Wouldn't it?
[acm] 
We only have mentioned use of private or experimental IDs,
which would facilitate the desired metric and 
measurement deployment and experience,
and in this case the Registry entries are not reviewed
until afterward.

> specification could end up being published...
[acm] 
The WG has been informed of 2 Private Performance Metric Registries 
being constructed and used. One is in Brazil, another is in the BBF. 

> Sorry to add more questions; I just want to make sure all the bases are
[acm] 
Thank you.
In this case we are designing a people-process, and we can trust that 
when we designate smart and fair people to administrate it, we will 
only then cover all bases.

---

From: Alvaro Retana <[email-masked]>
Date: Thu, 5 Dec 2019 09:59:46 -0800

On December 5, 2019 at 7:30:22 AM, MORTON, ALFRED C (AL) wrote:

Al:

Hi!

...
...
> coincides with IIESG review.

Your answer made go look at §8.1 again and the paragraph we're
discussing in context.  Just one suggestion: s/then the IESG approval
process MUST proceed with the proposed Performance Metrics Registry
entry removed from the text./then the proposed Performance Metrics
Registry entry MUST be removed from the text.

I trust that this text will make it into your next update, so I'm
clearing my DISCUSS.

Thanks!

Alvaro.

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Fri, 6 Dec 2019 18:53:50 +0000

Hi Alvaro and Alissa, and other ADs who supported Alvaro's DISCUSS #2,

Alvaro and I have come to agreement on revised text,
please see below.

Al

> Sent: Thursday, December 5, 2019 1:00 PM
[acm] 
Thanks, I've made that change in the working text, and it will appear
in the next version.  Thanks for clearing your DISCUSS.

Al

---

From: Alissa Cooper <[email-masked]>
Date: Fri, 13 Dec 2019 05:19:12 -0500

Hi Al,

> determines that one or more of the Section 5 criteria have not been met,

IANA has no role in evaluating the metrics against the Section 5 criteria, so I think they should be dropped from this sentence.

> registration at that time.

For the last sentence I would suggest “Once evidence exists that the Performance Metric meets the criteria in section 5, the proposed Performance Metrics Registry entry SHOULD be submitted … .” The people who write the RFC need not be the same people who later submit the metric to the registry.

Thanks,
Alissa

> On Dec 6, 2019, at 1:53 PM, MORTON, ALFRED C (AL) <[email-masked]> wrote:
>

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Fri, 13 Dec 2019 15:48:10 +0000

Thanks for the two suggestions, Alissa.

Done and Done in the working version of the text.

Al

> From: Alissa Cooper [mailto:[email-masked]]
> Sent: Friday, December 13, 2019 5:19 AM
> To: MORTON, ALFRED C (AL) <[email-masked]>
> Cc: Alvaro Retana <[email-masked]>; IESG <[email-masked]>;
> [email-masked]; [email-masked]; [email-masked]; draft-ietf-ippm-
> [email-masked]
> IANA has no role in evaluating the metrics against the Section 5 criteria,
> so I think they should be dropped from this sentence.
> 
> For the last sentence I would suggest “Once evidence exists that the
> Performance Metric meets the criteria in section 5, the proposed
> Performance Metrics Registry entry SHOULD be submitted … .” The people who
> write the RFC need not be the same people who later submit the metric to
> the registry.
> 
> Thanks,
> Alissa
> 
>