Subject: Re: [RTG-DIR] Rtgdir telechat review of draft-ietf-anima-autonomic-control-plane-13

From: Toerless Eckert <[email-masked]>
Date: Mon, 7 May 2018 21:30:19 +0200

Joel,

Sorry for the late reply. Bufferbloat.

Thank you  very much for the review.

Uploaded first round fixes for your review to github, will push out -14
when i am through with the other reviews.

Version:

https://raw.githubusercontent.com/anima-wg/autonomic-control-plane/master/draft-ietf-anima-autonomic-control-plane/draft-ietf-anima-autonomic-control-plane-14.txt

Diff to -13:

http://tools.ietf.org/tools/rfcdiff/rfcdiff.pyht?url1=https://www.ietf.org/id/draft-ietf-anima-autonomic-control-plane-13.txt&url2=https://raw.githubusercontent.com/anima-wg/autonomic-control-plane/master/draft-ietf-anima-autonomic-control-plane/draft-ietf-anima-autonomic-control-plane-14.txt

Comments inline below. (n) additional explanations at end of mail.

Cheers
   Toerless

On Thu, Mar 01, 2018 at 06:18:45PM -0800, Joel Halpern wrote:
> Reviewer: Joel Halpern
> Review result: Ready
> 
> I have been selected as the Routing Directorate reviewer for this draft. The
> Routing Directorate seeks to review all routing or routing-related drafts as
> they pass through IETF last call and IESG review, and sometimes on special
> request. The purpose of the review is to provide assistance to the Routing ADs.
> For more information about the Routing Directorate, please see
> ???http://trac.tools.ietf.org/area/rtg/trac/wiki/RtgDir
> 
> Although these comments are primarily for the use of the Routing ADs, it would
> be helpful if you could consider them along with any other IETF Last Call
> comments that you receive, and strive to resolve them through discussion or by
> updating the draft.
> 
> Document: draft-ietf-anima-autonomic-control-plane-13
> Reviewer: Joel Halpern
> Review Date: 1-March-2018
> IETF LC End Date: N/A
> Intended Status: Proposed Standard
> 
> Summary: I have some minor concerns about this document that I think should be
> resolved before publication.
> 
> Major: N/A (Note that three comment is marked *** as they may be more
> significant.)
> 
> Minor:
>     The description of the ACP In the introduction should probably note that
>     the addition of Management capability is beyond the definition provided in
>     section 5 of RFC 7575, and is included to complete the needed set of
>     functionality.

Added this to the end of introduction before discussion of stable connectivity,
seems to fit best there:

<t>The ACP provides secure IPv6 connectivity, therefore it can not only be used as the secure connectivity for self-management as required for the ACP in <xref target="RFC7575"/>, but it can also be used as the secure connectivity for traditional (centralized) management. The ACP can be implemented and operated without any other components of autonomic networks, except for the GRASP protocol which it leverages.</t> 

>     The definition of data-plane seems to assume that all forwarding (and
>     control?) in all nodes is structured in terms of VRFs.  That seems to
>     overly constrain implementation architecture.  Using VRF to to describe the
>     ACP functionality is a little odd, but likely defnesible.  Declaring that
>     all other functionality is in VRFs seems a step too far.

Oh... rathole ;-)
I spent a lot of time pondering this problem.  Given how i think the
classical way VRFs are implemented in routers is actually NOT a good
way to implement the ACP, i tried long to figure out better terminology.

But: VRF is actually the most logical way to specify the interoperability,
routing and forwarding aspects of the ACP without really implying a specific
implementation approach.

For example:
If we would have used a term like "virtual router", we would have
run the risk of excluding the option to implement ACP as a "classical
router OS VRF". On the other hand, by using the term VRF (including
our definitions what we mean with it), i can perfectly implement
the ACP as a virtual router, as a function in a hypervisor or
as a classical router OS VRF. VRF only says separate routing and
forwarding (see section 6.9).

Note too that we have explicitly specified in the terminology section
that we're NOT implying any of the MPLS/VPN-PE specific functions of
a VRF. Like othrer RFCs as well that use the term VRF without having
anything to do with MPLS/VPN.

In response to you bringing up the naming issue, i went through the
doc and removed "VRF" where it was used for the data-plane, so as not
to imply the use of VRF other than for the ACP itself.

There is still some amount of VRF in the "VRF selection" section, thats
really too hard to describe without VRF - would just make it too convoluted.

>     The definition of Intent in the terminology section seems to be different
>     from the cited definition in the anima reference model.  "Intent" there
>     seems to refer to a policy language, not an interface.

Thanks. Good catch. Fixed.

>     It is unclear how the flexible policy defined in section 5 bullet 2 (about
>     which nodes are ACP peer candidates) is consistent with autonomic
>     operation.  It seems that the flexibility is important, so there should be
>     some explanation here about how this is consonant with the stated goals.  I
>     understand that the bootstrap comes from BRSKI, but I do not think that is
>     where the policy comes from?

Would rather not like to add more suggestive text, and thats at best what
i could add. The default policy is the best "autonomic" behavior we know how
to make work: aka: try to connect ACP to all neighbors you can discover. And
we have only defined with DULL GRASP how to find subnet adjacent neighbors.

The main reason to mention policy is so that there is some leeway to do
more or even (sigh) less than all direct neighbors.

See also (1) at end.

>     The BNF is section 6.1.1 seems to have some oddities.  It may be that none
>     of these can happen for reasons that are described elsewhere, but as
>     written:
>   If the local-info is empty, the localpart will be "rfcxxxx." and thus the
>   domain info will be "rfcxxxx.@domain".  Is the period really required in the
>   absence of the local-info.

modified local-part = key   "." local-info
to       local-part = key [ "." local-info ]

>   If the rsub is empty, but the extensions are not
>   empty, it looks like there will be two plus signs (following the optional
>   acp-address) before the first extension.  Is that intended as the way to
>   indicate the absence of the rsub?

Yes. Otherwise it would be ambiguous to parse. Haven't
written something about this because i think readers eithrer
don't care or the can figure it out easily.

>   It also seems odd that the rsub (which is a
>   domain extension) appears in the local-part of the domain information, not in
>   the domain name.  Even though for hashing purposes it is concatenated, with a
>   period, with the domain name.
>  If these oddities are intentional, then there ought to be explanations so the
>  reader is not confused.

The difference between domain and routing-subdomain are explained directly after
the BNF:

 <t>"domain" is used to indicate the ACP Domain across which all ACP nodes trust each other and are willing to build ACP channel to each other.  See <xref target="certcheck"/>.

 <t><t>"routing-subdomain" is the autonomic subdomain that is used to calculate the hash for the ULA prefix of the ACP address of the node. ...

Added:

"rsub" needs to be in the "local-part"; it could not syntactically be separated from "domain-name" if "domain" is just a domain name. It also makes it easier for domain name to be a valid e-mail target.

>     I presume there is not an assumption that all ULA addressed communicating
>     with a node which supports the ACP will be over the ACP.  As I understand
>     it, ULAs may well have other uses outside of ANIMA / the ACP.  Which leads
>     to a question about how the text in 6.1.3 "If the CDP URL uses an IPv6 ULA,
>     the ACP node will try to reach it via the ACP." is expected to be supported?

The reason is that the ACP certificate maintenance runs across the ACP so that
we have a simple security model for the ACP certificate lifecycle. We do not
want to have to figure out the impact of attacks against connectivity to the CDP
when it runs across the data plane.  After all, these are ACP certificates are
also renewed via EST across the ACP.

I added the following explanation sentence:

Reaching the CDP across the ACP implies that the CDPs 
need to be reachable via the ACP, for example by running CDPs on 
registrars or on devices connected to the ACP via ACP connect (see <xref target="ACPconnect">).</t>

See also (2) at end.

>     The discovery text in section6.3 states that discovery should be run in
>     every physical interface.  Is this intended to be restricted by policy, as
>     described in section 5, bullet 2?

As said above for 5 bullet 2, it is permissible to be modified, the ACP
draft does not suggest or recommends any modifications other than those in 6.3,
aka: physical interfaces.

> ***    The explanation in section 6.5 on channel selection (and security
> protocol selection) is worded as if the described behavior will lead to
> reliable interoperation.  The text on why there are limitations (reasonable)
> doe not explain why mandating dTLS support would not be a reasonable step.   (I
> hope I missing something, as otherwise this is a major problem rather than
> being minor.)  It is also hard to align this with the requirements in section
> 6.7.3.

I think there is no issue. Let me explain:

We beat ourselves up for quite a long time about MTI channel options 
(Mandatory To Implement) and figured there is no one-size-fits-all.

The first challenge for dTLS was that there is no RFC specifying how to run
IPv6 over dTLS. There are widely supported / deployed variations
like OpenConnect etc. running IP/IPv6 over dTLS, but they all
havesome home-brewed complex session maintenance mechanisms because
they where built for hub&spoke user VPN solutions. We do not only not
need that additional complexity, there is also no RFC for them (only
expired individual drafts). 

In the absence of a simple standardized specification and deployment
experience in non-constrained equipment with simple IPv6 over dTLS, we 
did not want to make it MTI for devices that had a better alternative.

I then tried to figure out what would need to be written to specify
how to run IPv6 over dTLS, and that became 6.7.2.

Many type of constrained devices want to run everything over
dTLS because they already got that code for CoAP etc. pp, and
they do often not have IPsec code in their software. Therefore
dTLS is the better MTI for those devices.

I have modified the dTLS MUST sentence to make it clearer:

A constrained ACP node that can not support IPsec MUST support dTLS.
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^

>     6.8.2.1 has text about "hop-by-hop reliable retransmission as provided for
>     by TCP".  I am unable to determine what this is saying.

Changed sentence to:

<t>Such shorter periodic retransmission of datagrams would result in
more traffic and processing overhead in the ACP than the hop-by-hop
reliable retransmission mechanism by TCP and duplicate elimination by GRASP.</t>

>     If I have used section 6.10.3 properly, when the Zone-ID is 0, all nodes
>     within the ULA are within a flat address space.  that does turn the address
>     into an identifier.  It is, as far as I can tell from the description,
>     still a locator in that packets will be routed to the node using the
>     address including the node identifier.  It just means that routing has to
>     work in /127 addresses.  (There is also a reference to "identifier" in
>     6.10.3.1.  From the usage there, I think that the intention is that this is
>     the generic "name" for the node.  Given that it is routable, it is simply
>     an addresses, not an identifier or a locator.)

My driver license number is (intended to be) an identifier. It stays an indentifier
even if someone starts building a network with a routing table for driver license
numbers. Likewise, the intent of zone=0 numbers is to be identifiers. And
yes, they do double a locators when you have a flat routing table which is
what we do in this doc.

We just didn't describe options where zone=0 would not double as a locator,
that is meant to be left for future work if we see sufficient use cases for it.
We just tried to make the definitions extensible and define the intent of the
format to be that of identifiers.

>     The description in section 6.10 / 6.10.1 / 6.10.2 is about the schema
>     identification is very confusing, and arguably a bad design.

I guess you mean 6.10.2 / 6.10.3 and 6.10.4 ?

>     The two schemas use the same schema identifier.
                                          ^^^^^^^^^^

Type ?

      The difference in interpretation
>     is actually provided by the last bit in the upper 64.  If these are the
>     same schema, then it should not need a bit to differentiate them.  They
>     should be explained as a single schema.  Which would presumably result in
>     the Z bit being part of the zone / subnet field.   If they are two
>     different schemas, then move the differentiation to part of the schema
>     identifier (making it 3 bits).

We did not want to make Z part of the base schemas 6.10.2 "Type" code because
that would also make the Vlong scheme one bit shorter and that would reduce
virtualization space in Vlong by a factor of 2.

So, yes, encoding wise, 6.10.4 manual sub addressing space was carved out
of the 6.10.3 zone address space via the Z bit, but thats just an encoding aspect.

Functionally, 6.10.3/6.10.4 are quite different, and therefore it is IMHO
textually correct to have them in different subsections.

Carving out bits like Z isn't ideal, but in between the ULA prefix,
the 64 bit local part we must have for external interfaces and
best utilization of bits, we tried to rather optimize address space
utilization and not beauty of encoding.

Having  at the end allows (as written) use of /63 instead of  2 * /64
routes. I am not sure how important this would be in the future.

If you feel strongly about more beauty in encoding,
i can move Z to the left and remove the notion of the possible /63 route
optimization possible through the placement of Z.

>     This sort-of different but sort-of the same
>     makes implementation "interesting".  Even if this is extra bit is only
>     needed for schema 00b, it should be in front of the Zone / subnet, not
>     after it. This leads to the obvious observation that either we need two
>     bits, of which we will have 3 settings, and we will use the fourth setting
>     as an extension mechanism if we ever need more formats, or we need at least
>     three bits all the time because we seriously think there will be more
>     formats.

I very much hope there is no "interesting" implementation aspect;
nothing in the current scheme that makes implemenation harder. If you
have an example that you fear, i would like to hear it.

To me, we are just defining address space allocation schemes.

>     In section 6.10.5, after stating that it is not even known what the needed
>     usage is for more V values

Which sentence are you referring to when you say "not even known what the
needed usage is" ? I can't find it, but i would like to rectify it
if it actually is still there.

The two addressing space Zone and Vlong do certainly address two very different
network and node design considerations. In the Zone addressing format, we
wanted to have a scheme for potentialy very large networks. Enterprises
plus their attached IoT networks with hundreds of thousands of nodes.
Worldwide enterprise WN/campus/branches. Or massive DCs. The one V bit
in this space is just to solve the single most likely problematic implementation
consideration of ANI: Allowing ANI to have a complete port number space of
itself, therefore not conflicting with traditional management port number
usage (if needed). Or similar.

The Vlong addressing space goes the other way, allowing to build out
management plane or ASA more easily as containers or other lightweight
component models where each such component has its own address. We
also had brainstormed some network layer functions (proxies) that had to
eat up address space, so would use V bits.

Both approaches are IMHO very valid and important options that we wanted to be able to
support and let developers and deployments choose which one fits
them best.

>     the document goes on to introduce the
>     complexity of classful nodes numbers, with the leading bit indicating how
>     long the node-number is.  Yes, the rest of the text in that bullet tries to
>     explain why you need both sizes.  It seems like needless complexity that
>     begs for mistakes in implementation.

I was taking the example use cases we brainstormed into account, and
some of them  would require a lot more addresses in the node than others,
therefore these two choices.

Ultimately, we need to break through the chicken and egg problem
of how to build more self-managing networks. These will require
a different number of addresses inside nodes. Anything more intelligent
than address space allocations would become a complex dependency
making implementation/adoption even slower. I don't think that
offering 2, 256, 64k addresses per node is too much flexibility.
It IMHO necessary and sufficient to explore and we'll see what sticks.

>     In section 6.11.1.11 in describing the prefix lengths, I thought that the
>     point of zone addressing was to allow the use of /64 prefixes.  But it
>     seems here that we will not do so ?

As mentioned above: This document does not describe the routing setup for /64
(or /63 with Z-bit) routing aggregation. There are multiple options,
but we could not conclude on a single solution that we felt to be
appropriate for this document. Instead it was only very important
to carve out addressing space to support this.

See also (3) at end.

>     Section 6.12.5 says "ACP nodes MUST NOT derive their ACP virtual interface
>     IPv6 link local address from their IPv6 link local address used on the
>     underlying interface".  But it does not then seem to provide any advice on
>     what is expected to be used.  Is the loopback ACP supposed to be used
>     somehow?  Or?

Changed paragraph to following:

        <t>The ACP virtual interface IPv6 link local address can be derived from
        any appropriate local mechanism such as <xref target="EUI48"/> or <xref target="EUI64"/>. 
        It MUST NOT depend on something that is attackable from the data-plane such 
        as the IPv6 link-local address of the underlying physcial interface, which 
        can be attacked by SLAAC, or parameters of the secure channel encapsulation header 
        that may not be protected by the secure channel mechanism.</t>

>     The "configuration" description in 8.2.1 provides a nice description of
>     what information is needed, and what choices are allowed.  It does NOT
>     describe a method for configuring the information.  Please tune the
>     introductory material.

Hope i am guessing correctly what you're alluding to. Fixed text:

<t>On the ACP node, remote ACP neighbors are configured explicitly.
   The parameters of such a "connection" are described in the following ABNF.
   ...

<t>Explicit configuration of a remote-peer according to this ABNF
                                            ^^^^^^^^^^^^^^^^^^^^^
provides all the information to build a secure channel

> ***    Section 10.3.2 paragraph 2 says that devices should change the meaning
> of "admin down" to mean "down for everything except ACP / Anima".    I can
> understand why, in an autonomic network, such a state is desirable.  However,
> that really should be a different state from "admin down".  Operators already
> understand "admin down" as meaning that this interface will not be used for any
> traffic.  Changing that is fairly drastic.  "admin limited" or "admin ACP-Only"
> would seem much better than changing the meaning of "admin down".  The
> justification in the text seems to be a desire to prevent an operator from
> doing what he intends.  that seems backwards.  (Note that the distinction
> between administrative state vs operational state, aka failed, is already well
> understood.)

Well, this is in an informative section, so hopefully something we can agree to disagree.

I strongly believe that the most easy way to operationalize the ACP and
achieve its goals is what we have written. 

The historic equivalence of "down" = "admin down" = "physical down" is one of
the the most fundamental problem of remote management in networks.

We really need to start thinking of separate layers of (remote) management.
Network services on one hand and physical infra on the other.

The first thing to do this is to separate operations such as "down"
into "admin down" that affects networks services and "phy down" that
affects the physcial infra. 

If you have existing commands like "shutdown" that today do both,
the safe mapping is to let them do the safe thing in the future, e.g:
only "admin down", and introduce new commands for the phy operations,
e.g.: "phy shutdown" or the like.
And then protect those dangerous commands even further against unintentional or
intentional misuse.

Think of ACP/ANI also as a seamless inband version of a DCN. As an operator
of the actual data network, you also didn't have any access to bring down the DCN
and cut yourself off from the network you manage. YOu could only screw
up that network you're meant to manage.

And if your network consists of VMs in a DC, a "shutdown" on their
interfaces will also not bring down physical interfaces necessary
to reach the VM.

In optical networks you often have an inband physcial ethernet management
channel. Making/breaking that one is completely different from making/breaking
your data-plane, aka: data fiber colors. 

> ***    The notion in 10.3.6 that the device should refuse to disable
> functionality when an authorized administrator directs such seems flatly wrong.

The authorization to break your own remote management connectivity
would be a property of the certificate. clearly whoever enrolled
the device with a certificate denying that capaility to the operator
did NOT authorize the administrator to break connectivity.

> Editorial / Nits:
>     Section 6.3, in talking about DULL GRASP refers to the grasp internet
>     draft.  It refers to section 3.5.2.2.  there is no 3.5.2.2 in the grasp
>     I-D.  As best I can guess the reference should be 2.5.2.

Thanks

>     In the various formats in section 6.10, the lowest bit of the upper 64 bits
>     is mandated to be 0.  Presumably, there is some reason for doing this.  It
>     would be nice to explain it.

Sorry, not clear what you're referring to. Can you give me an
explicit reference ?

>     In section 6.10.5, in the text about how many V values are allowed, the
>     text reads: "2^16 - 65536" which would be 0.  I believe that the intention
>     is "2^16 (i.e. 65536)"

Thanks. Meant to say "2^16 = 65536", but now its your i.e.

>    Section 9.2.2 paragraph 2 begins "Group members must overall the secured". 
>    While I can guess what is intended, I would prefer that were in grammatical
>    English so as to avoid guessing.

Hmm.. not sure i catched the grammatical error i did, but i figured that 
the text was missing some better explanation what it meant. Added the following:

<t>Group members must be protected against attackers so that there is no easy way to compromise them,
or use them as a proxy for attacking other devices across the ACP. For example, management plane functions (transport ports) should only be reachable from the ACP but not the data-plane.
Especially for those management plane functions that have no good protection by themselves because they
do not have secure end-to-end transport and to whom ACP does not only provides automatic reliable 
 connectivity but also protection against attacks. 

>     Section 10.5 provides the requirements that were met by choosing RPL for
>     the routing protocol.  Likely RPL is a reasonable choice.  Since this
>     section does not provide comparison or list any alternatives, I would
>     suggest changing "explanation" to "motivation".

Done.

--------------------------------------------------------------------------

(1) Examples of good additional discovery would be methods to do it
via L3 or where additional parameters ae needed, we discussed some options
to help bring up interfaces like PPP requiring credentials or the like.
On the "challenged product" side, we have seen products unwilling to
bring up interfaces during bootstrap because those interfaces require
additional license keys for activation *sigh*. 

(2)side note: We started from classical PKI model with CDP/OCSP but didn't bother
much about it, so this is a bit of a step-child: Most of the ANI authors 
are fans of short lived certificates to eliminate CRL/CDP because through ACP 
and BRSKI, we can make lifetimes shorter than typical CDP validity intervals:
ACP adds reliability, EST automation, BRSKI can provides re-enrollment even
when certificate has expired so ACP can not be vbuilt. Therefore
we didn't bother much in making CDPs as easy to use as we could, e.g.: no GRASP
objective to auto-announce/discover CDP like we did for EST renewal. but
i  didn't want to blackhole the text with these considerations even further. ]

(3) The most simple option we have seen asked for in actual deployments was for
manual config of such zone edges, not IMHO really appropriate
for this document. More like something for informational docs. Even
in that case you could explore the traditional method where you
actually hand out zone != 0 address via the domain cert, set up
RPL roots for each zone and connect the roots. Or you do simply rely
on zone 0 addresses, rely only on the default route in each zone
and use something like lisp between zone roots to map from zone=0
as identifier to remote root locator. Aka: everything
worth describing in other documents.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 7 May 2018 19:05:10 -0400

Comments in line.  Areas of clear agreement elided.

> Sorry for the late reply. Bufferbloat.
Okay.  Thank you for addressing a number of the points effectively. 
Let's see if we can solve the remaining items.

> I spent a lot of time pondering this problem.  ...
> 

Good enough.  Thank you.

> more or even (sigh) less than all direct neighbors.

After looking at this, I think I can live with what you already have.  I 
think there are some issues, but let it be.

> don't care or the can figure it out easily.

I think it would help to state it explicitly.

(section 6.1.1)
> "rsub" needs to be in the "local-part"; it could not syntactically be separated from "domain-name" if "domain" is just a domain name. It also makes it easier for domain name to be a valid e-mail ta…

Okay.  It is odd, but works.

> registrars or on devices connected to the ACP via ACP connect (see <xref target="ACPconnect">).</t>

While a useful addition, that is not what I was asking about.  The text 
as written seems to say that simply because the RDP uses an IPv6 ULA, 
the node will know to use ACP for the communication.  I understand why 
we awant the communication to use ACP.  I don't understand how that is 
achieved.  The way the text is written, it seems to imply that it is a 
property of IPv6 ULAs.  Which I don't think is your intention.
So I would like to see clarification.

> aka: physical interfaces.

I think you simply need to say "Unless modified by policy as noted 
earlier, ..."  As currently written, the text prohibits the policy 
over-reide you want.

> I think there is no issue. Let me explain:

I think I understand your reasoning, and the various constraints that 
you face.  However, when all is said and done, you seem to have two 
mechanisms, and no requirement that all devices must support one of 
them.  Which seems to mean that two independent devices may not have an 
implementation of the channel / security selection that leads to 
interoperability.  That seems a major issue.  Saying "we couldn't find a 
good answer" is not usually an acceptable IETF answer for interoperability.
  (text retained for future discussion)
> A constrained ACP node that can not support IPsec MUST support dTLS.
The change helps, but as noted above does not seem sufficient.

...
> format to be that of identifiers.

Since the design specifically uses the same field as both the identifier 
and the locator, I do not think it is fair or appropriate to describe it 
as being purely an identifier.    The fact that you could possibly use 
it as an identifier later doesn't make it so.

> I guess you mean 6.10.2 / 6.10.3 and 6.10.4 ?
Yes.

>>      The two schemas use the same schema identifier.
Yes.

Part of my reaction to this whole section is that it looks much like the 
mistake we made originally in defining classful IP addresses.  You are 
trying to guess the correct use cases, and defining specific address 
structures with specific encodings for each one.  With only a few 
encodings available.  This seems to be a recipe for being wrong later in 
ways that hurt.

> optimization possible through the placement of Z.

Since the two different uses of the Z bit with type 00b are quite 
distinct, trying to enable aggregation seems counter-productive.

While I hate to push on a point you describe as aesthetics, having the Z 
bit, occuring after the Zone / subnet-ID field, but defining whether the 
field is a zone or a subnet, seems awkward for insufficient reason.

I would recommend moving the Z bit up so it is after the type, and at 
the front of each of the two sections reiterate that this subscheme is 
indicated by the Z bit being { 0 | 1 }.

> To me, we are just defining address space allocation schemes.

I suppose likely implementations will use mask definitions that will 
make it work.  I pity the code reviewer.

> if it actually is still there.

The text reads "further values use via definition in future work."  In 
the zone-id scheme, it is one bit.  For some reason, it is 8 or 16 bits 
here.

> therefore these two choices.

See my comments above.  Trying to encode specific use cases into the 
address format seems a problematic answer to an admittedly difficult 
question.

What would break if you didn't define any of this?

> It IMHO necessary and sufficient to explore and we'll see what sticks.

Variation is needed.  That was true of allocations in IP.  encoding the 
variations proved a bad choice.

> to carve out addressing space to support this.
The net effect seems to be to prohibt the very things that you went to a 
lot of trouble to enable in your addressing design.

...
> Well, this is in an informative section, so hopefully something we can agree to disagree.

If this were truly informative, it would not belong in this document at 
all.  Changing the behavior of widely understood configuration behaviors 
is NOT a small thing.  Changing the permissions models for widely 
understood configuration operations is also NOT a small thing.

My suggestion is that you remove all of this material.  If you feel it 
is important, write an I-D for standards track on evolving the 
configuration model for robust ANIMA.  While I think the change is a 
mistake, you clearly have the right to cause the discussion.

Hiding the topic in an "informational" note in the ACP document seems 
VERY wrong.

Also note that in your discussion below, you talk about the equivalence 
of admin down and physical down.  Most management models I know of treat 
these as quite distinct.  SO that is at best a red herring.

(text retained for context of further discussions.)

> did NOT authorize the administrator to break connectivity.

The security model change seems as problematic as the above 
configuration change.

>> Editorial / Nits:
.
> Sorry, not clear what you're referring to. Can you give me an

I should ahve removed this note.  It was a side-effect of the 
presentation and ordering, not a problem on its own.  Sorry.

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 8 May 2018 14:59:53 +1200

CCs trimmed:

On 08/05/2018 11:05, Joel M. Halpern wrote:
...
>>>      the ACP node will try to reach it via the ACP." is expected to be supported?

I guess I read that sentence as simply meaning:
"If the CDP URL uses an IPv6 ULA within the ACP's ULA prefix,
the ACP node will try to reach it via the ACP." which is a trivial
statement. If that's not what it means, I am confused too.

   Brian

---

From: Toerless Eckert <[email-masked]>
Date: Sat, 12 May 2018 04:09:44 +0200

Diff:

http://tools.ietf.org/tools/rfcdiff/rfcdiff.pyht?url1=https://raw.githubusercontent.com/anima-wg/autonomic-control-plane/2ae8f47399ae0d0811cb45209186d01f9e0d3077/draft-ietf-anima-autonomic-control-plane/draft-ietf-anima-autonomic-control-plane-14.txt&url2=https://raw.githubusercontent.com/anima-wg/autonomic-control-plane/54c9c16f3444b1db13c7cb05744e543dfcdfb63b/draft-ietf-anima-autonomic-control-plane/draft-ietf-anima-autonomic-control-plane-14.txt

Latest version:
https://raw.githubusercontent.com/anima-wg/autonomic-control-plane/54c9c16f3444b1db13c7cb05744e543dfcdfb63b/draft-ietf-anima-autonomic-control-plane/draft-ietf-anima-autonomic-control-plane-14.txt

> Comments in line.  Areas of clear agreement elided.

Comments in line.  Removed sections closed by you.

> I think it would help to state it explicitly.

<t>In this specification, the "acp-address" field is REQUIRED, but future variations (see <xref target="adopting-acp"/> may use local information to derive the ACP address. In this case, "acp-address" could be empty. Such a variation would be indicated by an appropriate "extension". If "acp-address" is empty, and "rsub" is empty too, the "local-part" will have the format "rfcSELF + + extension(s)". The two plus characters are necessary so the node can unambiguously parse that both "acp-address" and "rsub" are empty.</t>

> Okay.  It is odd, but works.

Any easier choice coming to mind ? I wouldn't know how.

> While a useful addition, that is not what I was asking about.  The text as
> written seems to say that simply because the RDP uses an IPv6 ULA, the node
> will know to use ACP for the communication.  I understand why we awant the
> communication to use ACP.  I don't understand how that is achieved.  The way
> the text is written, it seems to imply that it is a property of IPv6 ULAs.
> Which I don't think is your intention.

Originally it was meant to be indicated by ULA address, yes, but i think
its better/simpler just to decide based on DNS being used.

> So I would like to see clarification.

Ok, refined the text as follows:

If the CDP URL uses an IPv6 address
(ULA address when using the addressing rules specified in this document),
the ACP node will connect to the CDP via the ACP. If the CDP uses a
domain name, the ACP node will connect to the CDP via the data-plane.</t>

<t>It is common to use domains name for CDP(s), but there is no requirement
for the ACP to support DNS. Any DNS lookup in the data-plane is
not only a possible security issue, but it would also not indicate whether 
the resolved address is meant to be reachable across the ACP. Therefore,
the use of an IPv6 address versus the use of a DNS name doubles as an 
indicator whether or not to reach the CDP via the ACP.</t>

<t>A CDP can be reachable across the ACP either by running it on a
node with ACP or by connecting its node via ACP connect (see <xref target="ACPconnect"/>). 
The CDP SHOULD use an ACP domain certificate for its HTTPs connections.
The connecting ACP node SHOULD verify that the CDP certificate used 
during the HTTPs connection has the same ACP address as indicated in the
CDP URL of the nodes ACP domain certificate</t>

> I think you simply need to say "Unless modified by policy as noted earlier,
> ..."  As currently written, the text prohibits the policy over-reide you
> want.

changed to:

Unless modified by policy as noted earlier (<xref target="overview"/> bullet point 2.),
native interfaces (e.g.: physical interfaces on physical nodes) SHOULD be initialized ...

> I think I understand your reasoning, and the various constraints that you
> face.  However, when all is said and done, you seem to have two mechanisms,
> and no requirement that all devices must support one of them.  Which seems
> to mean that two independent devices may not have an implementation of the
> channel / security selection that leads to interoperability.  That seems a
> major issue.  Saying "we couldn't find a good answer" is not usually an
> acceptable IETF answer for interoperability.

Think of channel protocols like interface types. We also do not have
a requirement that all nodes MUST have WiFI or 100 Mbps ethernet or 40 Gbps
fiber. And yet we manage to build an Internet across all those device
interconnecting them all. Same thing with channel security protocols.

The requirement right now describes two profiles (6.7.3):

"baseline ACP node" - MUST support IPsec
"constrained ACP node" - node that can not support IPsec, MUST support dTLS.

In the same way you use a WiFi access point connecting your ethernet with
your WiFi equipment, you would use a baseline ACP node that also supports dTLS
to interconnect constrained and unconstrained nodes.

This also well matches how constrained node networks are built:
Some gateway node that is not constrained, but also supports
whatever (mostly constrained) network/radio the constrained nodes use.

If you think this should be explained more in more detail in the doc,
let me know. I could simply add above example as text.

> The change helps, but as noted above does not seem sufficient.

See above, the "can not support IPsec" was only meant as a definition of
what a constrained node is, but not to change the basic property of ACP networks
that different links / nodes can have different protocol options and that
there is no "one-fits-all" MTI necessary.

> Since the design specifically uses the same field as both the identifier and
> the locator, I do not think it is fair or appropriate to describe it as
> being purely an identifier.    The fact that you could possibly use it as an
> identifier later doesn't make it so.

Do we (IETF) have any normative or otherwise uncontested definition for the criteria
required to call an address an identifier ? 

Here are the things that make the ACP address an identifier to me:

- It is assigned once to the device and not meant to change over the lifetime
  of the device.
- It does not change when i move the node to a different location
  and re-connect the device.
- It is the unique part of the ACP domain certificate (other than the private key)
  and used to identify the device (privat key is used to authenticate it,
  but the private key may change over time, for example when renewing the certificate).

Would be happy to elaborate this in the doc more explicitly, e.g.: like written
above.

> structures with specific encodings for each one.  With only a few encodings
> available.  This seems to be a recipe for being wrong later in ways that
> hurt.

The only equivalent to classfull addressing i see is for the fixed prefix-length
handed out to each node, 1 bit (2 addresses) with zone addresses, 8/16 bits
(256/64k address) for Vlong, but its NOT assignment to network segments 
as in IPv4 A/B/C but to nodes.

We didn't introduce choices lightly. There are compromises to be made when you
want some lightweight address allocation scheme, not expecting to have complex
address registry or management system in the backend but simple registrars.

see below for more

> Since the two different uses of the Z bit with type 00b are quite distinct,
> trying to enable aggregation seems counter-productive.
> 
> I would recommend moving the Z bit up so it is after the type, and at the
> front of each of the two sections reiterate that this subscheme is indicated
> by the Z bit being { 0 | 1 }.

Done. (see diff).

> I suppose likely implementations will use mask definitions that will make it
> work.  I pity the code reviewer.

I have read the code and tested in the lab IPv6 embedded RP adressing
and didn't feel pityfull about the job. But i weigh the overall system
complexity. Embedded-RP simplified a lot of other parts of the
multicast system at the cost of introducing an addressing scheme.
I think similarily, the ACP addressing scheme simplifies the overall system
complexity, especially registrars.

> The text reads "further values use via definition in future work."  In the
> zone-id scheme, it is one bit.  For some reason, it is 8 or 16 bits here.

Thanks, changed sentence to:

V: Virtualization bit: Values 0 and 1 are assigned in the same way as in the Zone-ID sub-scheme.

The existing text, now directly after the list, explains examples
when a device would likely need 256 or 64k addresses:

existing text:
..."1" bit Node-Numbers are intended for ACP nodes that are ACP edge nodes ...

> See my comments above.  Trying to encode specific use cases into the address
> format seems a problematic answer to an admittedly difficult question.
> 

One could save the 2.5 bits (Type, Z) indicating the structure of the
address inside the address by encoding the designated format of the
address instead in an extension field of the ACP address information in
the certificate. We would certainly still want to have the same type
of addresses, e.g.: 2,256,64k local addresses, option for zones and
address to assign to ACP connect interfaces (manual addressing scheme) 
IMHO. So, at best we could save the 2.5 bits if we wanted to keep the
functionality, which i think is all required.

I would be quite afraid of doing such a drastic change this late in
the process because i think it will take a long time to foresee the
resulting complexity introduced for registars and any other place
that now can deduct the semantic from the address and would then have
to access to what the certificate indicates in a different way (e.g.:
access to the certificate). Note that the length of the ACP information
field in the cert is also limited, so this type of extension might end
up looking even uglier to be short *sigh*.

I was always thinking that this type of more flexible option would
be a good next-step, but not a safe first step. It would make a lot
more sense after hearing implementation/deployment experiences,
especially to correctly support important options we may have
overlooked in the current schemes. Just redoing what we have differently
to save 2.5 bits isn't worth it IMHO.

> variations proved a bad choice.

Let me repeat that we are not doing any network/subnet prefix encoding which is
what IPv4 A/B/C was. And there are IMHO good examples how encoding in IPv6
did save the day from complexity (at least all the stuff done for IPv6
multicast, ASM/SSM, zones, Embedded-RP) was highly useful.

> The net effect seems to be to prohibt the very things that you went to a lot
> of trouble to enable in your addressing design.

Maybe i do not understand your comment correctly, but i do not
think it is correct. We just have not defined recommended mechanisms
how to set up routing to use /64 zone routes. That is something IMHO
better done in a followup document because all the cases where we
did see a need for znes where really very specific to particular
deployment styles and would not be applicable to the mayority of ACPs.
Nevertheless, we felt those cases where important enough to reserve
bits for them through the zone field.

> If this were truly informative, it would not belong in this document at all.
> Changing the behavior of widely understood configuration behaviors is NOT a
> small thing.  Changing the permissions models for widely understood
> configuration operations is also NOT a small thing.
> 
> My suggestion is that you remove all of this material.  If you feel it is
> important, write an I-D for standards track on evolving the configuration
> model for robust ANIMA.  While I think the change is a mistake, you clearly
> have the right to cause the discussion.
> 
> Hiding the topic in an "informational" note in the ACP document seems VERY
> wrong.
> 
> Also note that in your discussion below, you talk about the equivalence of
> admin down and physical down.  Most management models I know of treat these
> as quite distinct.  SO that is at best a red herring.

Hmm. I really only CLI derived from Cisco IOS, and there the
interface level "shutdown" command doubles IMHO both as admin down and
as phy down, so i would contend that this is a red herring. It actually
does show up in diagnostic CLI as "admin down", but is also bringing down
phy state. Therefore it is also used to diagnose phy, .e.g: shutting down
one side of a supposed direct p2p link to verify the phy change on the 
other side. That practical experience is what the text and direction is
based on.

Whats an example where admin/phy down are separate in your experience ?

I don't know how you define "truly informative". I simply meant
informational as compared to standards track wrt. to not impacting
interoperability, but also as a precursor work to give opportunity
for gaining more experience before attempting to go standard.

If/when i find time i definitely would like to investigate how this
could be formalized through a YANG model for ACP so that it can become standard,
but i definitely agree that especially this part of the YANG model would
be highly difficult to get to before any experience is gained with
this.

I therefore would very much hesitate removing this text from the document, it was
specifically asked for several times in the working group to understand
how the promised reliability of the ACP against operator/contoller
operation could be achieved. Having this in a first round (this document)
as "informational" is IMHO a good and important thing. 

> The security model change seems as problematic as the above configuration
> change.

We have recurringly heard from customers that mistakes in managing remote
equipment and loosing connectivity is a mayor pain point. Creating
an operational option that limits/disables operators ability to
impact phy level connectivity that remote management depends upon and therefore
reducing this risk and cost factor seems to me like a very logical pah
forward. Whether or not this should be indicated in the certificate
or differently is definitely wide open.

I agree that none of these operational interface level changes are
a piece of cake given how we have in the IETF not tried to define
different access levels to configuration so far, at least not
between some phy/infra access level and network-services access level.

As said above in my first reply that you didn't comment on, i think a
good part of what i am looking for may happen through other means
already, such as when a router is just a VM running on an OS. At least
in early versions of this that i'd seen, this implicitly disabled
the ability to modify phy state from the router VM, and of course
complaints where raised that phy mods where necessary, but at least this
evolution could give raise to pause and rethink HOW to permit doing this.

So i would at least like to pledge with you that it is a good thing to
have this type of discussion in an informational way in this RFC instead
of doing nothing or trying to go standard YANG directly as a first step.
It did get into the spec because of the desire of the working group.

> I should ahve removed this note.  It was a side-effect of the presentation
> and ordering, not a problem on its own.  Sorry.

Cheers
    Toerless

---

From: "Joel M. Halpern" <[email-masked]>
Date: Sun, 13 May 2018 19:38:35 -0400

Trimmed.
Top posting for clarity.  Mostly, agreement.
On the MTI Security, I will leave it to the relevant ADs to decide if 
they find what is in there sufficient.

On the identifier vs locator, I think the way you are using the terms is 
confusing.  But it is not worth continued argumentation.Some text 
elaborating along the lines you provide (retained below) would help.

I have tried reading the section on addressing inside the ACP more 
carefully.  Thank you for moving the Z bit.
I have retained the earlier discussion on this topic below, in case we 
still need it.
Reading the section, I think that the problem is more basic.

I agree that the document needs to define that we are using a ULA.  And 
because of the certificate behaviors, there is significant advantage in 
defining what ULA prefix is used.

Given that, I do not see what use there is in all the other format 
definition.  I suspect I am missing something very basic.  Why not leave 
this to the deploying operator?  It is not like it can be autonomic, 
since something has to tell the devices what format to use, and what 
numbers within some of the formats to use.  I could imagine something 
that said "in the absence of configuration, construct device addresses 
according to one specific algorithm, so as to simplify bootstrap.
But it would be a simple scheme.

Yours,
Joel

...
> Would be happy to elaborate this in the doc more explicitly, e.g.: like written

(Context: Addressing inside the ACP.)

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 14 May 2018 17:23:51 +1200

Joel,

Consider this:

>    The 64 bit Node-ID is derived and composed as follows:
> 
>    o  Registrar-ID (48 bit): A number unique inside the domain that
>       identifies the registrar which assigned the Node-ID to the node.
>       A MAC address of the registrar can be used for this purpose.
> 
>    o  Node-Number: A number which is unique for a given registrar, to
>       identify the node.  This can be a sequentially assigned number.

plus

>    If a node learns through a future autonomic method or through
>    configuration that it is part of a zone, it MUST also respond to its
>    ACP address with that Zone-ID.

Basically it's a scheme for all autonomic nodes to get a unique
address in a potentially flat address space, with the help of
a registrar node, and without the help of a human. My understanding
is that the nodes know the format because it's built into the
code that each node runs to join the ACP. It's certainly not the
only possible scheme. Maybe that should be stated?

Regards
   Brian

On 14/05/2018 11:38, Joel M. Halpern wrote:
> ....

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 14 May 2018 11:04:25 +0200

a) Will try to do better explanation of identifier nature of addresses in
   next commit.

b) what do we loose if get rid of the choices. I kinda thought i had answered
   this already, but i really don't know where your concern starts...

   1. We use ULA because we want to be able to bring up ACP networks without
      requiring a registry allocation of a prefix, and ULA is the correct
      private address space. And deriving ULA Global ID from hash of domain
      name is nice.

      Sounds as if you agree with this.
    
   2. We want to be able to allocate addresses to ACP/ANI devices
      without any additional configuration from multiple independently
      running registrars. Thats why the global unique ID of the registrar
      is part of the address in both zone and Vlong format.

      Thats very fundamental to the ease of deployment of ACP/ANI nodes
      (i say ANI, because its really ACP+BRSKI required here to get a simple standard
       solution to create the cert, otherwise youre down to a vendor specific
       cert provisioning solution extended with ACP info support)

      i hope this is uncontentuous
   
   3. a) We wanted to be able to support in the future "zone" = area/region-level
      route aggregation, thats why the zone scheme reserves those 13 bits.

      b) We also know use cases where nodes need more than 2 addresses,
      thats why we have have the 8/16 bit Vlong option. 

      c) We can not do everything autonomically (== addresses assigned from registrars),
      so we also reserve address space for traditional config (manual address space).

      Are these three sub-spaces zone,manual,vlong contenuous ?
   
   As mentioned in prior email if i wanted to avoid those ca. 3 bits of
   format indication, i would need to indicate the semantic of addresses
   somehow differently to the nodes, e.g: via some extension to the ACP format,
   but i would want to be able to signal the same information.

   So for example: because we have a standardized addressing scheme,
   a node that allows configuration of "ACP connect" can make sure
   that the address configured on an ACP cnnect interface can never be a ULA
   address that would conflict with an automatic ACP address assignment done
   by a registrar. So i protect my autonomic address space from misconfiguration.
   Which is IMHO extremely important. So this is the key justification for
   the manual address space allocation. 

   Distinguishing between zone and Vlong address spaces is necessary because
   we just can't fit zone, registrar-ID and node-number and 16 bit of per-node bits
   into a single addressing scheme.

   E voila, three (IMHO: simpler) simple options. Likely networks will use all three.
 
Cheers
    Toerless

On Sun, May 13, 2018 at 07:38:35PM -0400, Joel M. Halpern wrote:
> I have retained the earlier discussion on this topic below, in case we still
> need it.
> this to the deploying operator?  It is not like it can be autonomic, since
> something has to tell the devices what format to use, and what numbers
> within some of the formats to use.  I could imagine something that said "in
> the absence of configuration, construct device addresses according to one
> specific algorithm, so as to simplify bootstrap.
> _______________________________________________
> Anima mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/anima

-- 
---
[email-masked]

---

From: "jmh.direct" <[email-masked]>
Date: Mon, 14 May 2018 08:49:56 -0400

I think the I understand what you describe.  As far as I can tell, the ACP DOES NOT AND SHOYLD NOT MANDATE The address assignment mechanism.  But it is the address assignment mechanism that drives the address allocation.
As such, it seems like the ACP itself does not care about this.  That is,why the routing scheme seemed confusing after the address allocation scheme.
I would expect to find the address structure for autonomic addressed in the same document as the mechanisms for autonomic address assignment.
Yours,Joel

Sent via the Samsung Galaxy S® 6, an AT&T 4G LTE smartphone
-------- Original message --------From: Toerless Eckert <[email-masked]> Date: 5/14/18  05:04  (GMT-05:00) To: "Joel M. Halpern" <[email-masked]> Cc: [email-masked], [email-masked], [email-masked], [email-masked] Subject: Re: [Anima] Rtgdir telechat review of
  draft-ietf-anima-autonomic-control-plane-13 
a) Will try to do better explanation of identifier nature of addresses in
   next commit.

    

   

   

 
Cheers
    Toerless

-- 
---
[email-masked]

---

From: Toerless Eckert <[email-masked]>
Date: Tue, 15 May 2018 04:58:07 +0200

On Mon, May 14, 2018 at 08:49:56AM -0400, jmh.direct wrote:
> I think the I understand what you describe.
> As far as I can tell, the ACP DOES NOT AND SHOYLD NOT MANDATE
> the address assignment mechanism.  But it is the address assignment
> mechanism that drives the address allocation.

But that exactly is the conclusion of the multi-year ACP work in ANIMA WG.
The ACP of an unconfigured, but only certificate enrolled device needs to
be able to bring up routing and addressing for the ACP autonomically,
aka: without any other dependencies.

This ACP draft as it stands only depends on BRSKI and some unmodified CA
to be implemented and deployed. There is no NOC with any address management
functions required. At minimum, addressing is solely done from th registrar,
and you can have many registrars in a network and there is no chance
for inconsistent address configuration.

Please propose any other solution that you are thinking of in enough detail
to discuss it, and i think we can show it has more, and often future/undefined
problems to be resolved. 

> I would expect to find the address structure for autonomic addressed
> in the same document as the mechanisms for autonomic address assignment.

The ANIMA autonomic address asignment draft is relying on the ACP to be
operational end-to-end to signal amongst its components via ACP GRASP the
address/prefix assignments. It can by definition not be used to bring up 
routing and addressing of the ACP itself. It likely also required more
centralized functionality. For what it tries to do, it is elegant, flexible and
all that because it does not have to bother about bootstrap addressing and
routing for itself as the ACP has to do.

https://en.wikipedia.org/wiki/Bootstrapping#/media/File:Zentralbibliothek_Solothurn_-_M%C3%BCnchhausen_zieht_sich_am_Zopf_aus_dem_Sumpf_-_a0400.tif

BRSKI+ACP are doing what poor Muenchhausen is trying to do in
what i think is a quite lightweight, very secure, scalable and
for the job at hand (ugly) quite elegant way.

> That is, why the routing scheme seemed confusing after the address allocation scheme.

If you can be more explicit about what you think is confusing, i am
happy to make any confusing text better.

Note that routing protocol/profile and addressing scheme are all subject
to future enhancements easily. But in the design of anima, we would like
to have operator defined address  policy to be part of intent, and
that would not only requir a lot more future work, it would also make
addressing assigned this way less reliable and would not allow to use
addresses managed dyanmically be used by components that are used to implement
the Intent infrastructure or any future ANI components (ASA etc.) that
intent or address management depends on.

Cheers
    Toerless

-- 
---
[email-masked]

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 14 May 2018 23:15:58 -0400

(Sorry about the excess caps.  Did not intentd to shout.)

 From some off-line disucssions, I understand that the actual address 
allocation is tied to the registration process.  The registration 
process is not described in this document.

As far as I can tell, the ACP does not depend upon the address 
allocation strategy.  The address allocation strategy does require the 
ACP.  And it requires and is closely coupled to the registration process.

If that is true, the address allocation seems to be better described 
with the registration process.  Which I presume is in the BRSKI document.

All I would put in here is the basic ULA allocation (hash based 
generation) mechanism.

Yours,
Joel

---

From: Toerless Eckert <[email-masked]>
Date: Tue, 15 May 2018 04:58:07 +0200

Cheers
    Toerless

-- 
---
[email-masked]

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 14 May 2018 23:15:58 -0400

Yours,
Joel