Subject: [Netconf] draft-ietf-netconf-zerotouch-21 feedback

From: Tarko Tikan <[email-masked]>
Date: Sat, 10 Mar 2018 21:02:08 +0200

hey,

I was pointed towards this draft during a discussion around operational 
problems in current ZTP world. This is my feedback from mid-sized SP who 
would like to use this draft as soon it is implemented.

1) Not many SPs buy their higher-end devices (used for enterprise 
services, L3VPNs etc.) in big quantities. Due to smaller quantities, 
they are not available customized from the manufacturer. Hence the need 
for proper ZTP.

1.1) Many times the devices are bought via distributors without direct 
interaction between manufacturer and SP.

I might be mistaken but currently it looks like secure bootstrap is not 
possible without properly precustomized devices. If hw-model, os-name 
etc. are not sent in get-bootstrapping-data in untrusted mode it becomes 
impossible to generate correct configuration for the device.

2) Currently used ZTP interface information should be sent in 
get-bootstrapping-data. Minimally ifName and vlan ID (for the scenarios 
where ZTP DHCP client sends out DHCP discovery on all 4k vlan IDs).

This is required for devices that might have multiple ZTP capable 
interfaces (router with DSL and SFP interface for example) and it's not 
known which interface might be used during installation. Without this 
information it again becomes impossible to generate correct 
configuration for the device.

3) zerotouch/enabled? MUST be restored to "true" when device is reset to 
factory config using a push-button or CLI command. There have been 
negative examples where even factory default customization is wiped 
clean during the push-button factory reset.

Thanks to the authors for the work, it's good to see work being done in 
this, still operationally problematic, area.

-- 
tarko

---

From: Kent Watsen <[email-masked]>
Date: Mon, 12 Mar 2018 18:35:58 +0000

Hi Tarko,

> would like to use this draft as soon it is implemented.

Thanks for the feedback.

> they are not available customized from the manufacturer. Hence the
> need for proper ZTP.

Sounds familiar.

> interaction between manufacturer and SP.

Understood.  The draft doesn't standardize an ownership-tracking solution.
It is left to the manufacturers to devise a strategy that works well with
how their devices are sold and used.

> impossible to generate correct configuration for the device.

I'm curious what you mean by the first sentence.  Looking to the 2nd
sentence for clues, be aware that those three input parameters are
optional and only needed in select cases.  When get-bootstrapping-data is 
sent to an untrusted bootstrap server, the primary input is the device's
authentication credentials (e.g., TLS-level client certificate) with
which the bootstrap server can use to return signed data.  Please take 
a look at Appendix A (Promoting a Connection from Untrusted to Trusted).

> configuration for the device.

I see.  One or more additional input parameters would be needed to 
address this.  Do you think a single parameter "interface" (the name
of the ZTP interface used) would suffice, or perhaps a generic "opaque"
parameter for a vendor-specific text format would be more flexible?
Currently, if needed, vendors could define a vendor-specific parameters
but, of course, that could affect interoperability - as not all bootstrap
servers will handle the additional input the same way.

> 3) zerotouch/enabled? MUST be restored to "true" when device is 
> reset to factory config using a push-button or CLI command. There 
> have been negative examples where even factory default customization
> is wiped clean during the push-button factory reset.

True, and it is the intention of the draft to enable devices to be
"reset" in the field, thereby returning them to their initial states,
as described in Section 5.   Perhaps the draft isn't saying this 
clearly enough?  Please let us know which section you were looking
at when thinking this.

> this, still operationally problematic, area.

Cheers,
Kent

---

From: Tarko Tikan <[email-masked]>
Date: Sun, 18 Mar 2018 21:07:21 +0200

hey,

> a look at Appendix A (Promoting a Connection from Untrusted to Trusted).

Well I'm worried about the implications of the trust model. Getting the 
owner CA trust anchor onto the devices from factory would be as 
complicated as todays ZTP. Getting ownership vouchers from the 
manufacturer is better but from practical perspective it would be best 
if there would be insecure mode where device blindly trusts whatever is 
served by bootstrapping server.

> servers will handle the additional input the same way.

The first concern is ZTP in general so it'll do even if it is vendor 
specific. But such things tend to be forgotten by vendors and if the 
interface parameter is optional then we might end up in the same 
situation as today - ZTP exists on paper but unusable in real life 
unless you can directly map device serial number to intended role.

It would be tempting to make it a single parameter filled with ifName of 
relevant interface. But that would be problematic in cases where device 
sends DHCP discoveries with all possible VLAN IDs while the subinterface 
for that VLAN might not exist. So when interface parameter is filled 
with ge-0/0/0 for example but vlan 123 is in use, the parameter should 
say ge-0/0/0.123.

> clearly enough?  Please let us know which section you were looking

I was reading 5.1 when I thought about that. I mentioned this because I 
have seen situations where devices will end up in some limbo state after 
factory reset even when they were ZTP capable from the factory. IMHO if 
someone claims they support draft-ietf-netconf-zerotouch then their 
devices MUST return to zerotouch/enabled?=true in every factory reset 
situation imaginable.

-- 
tarko

---

From: Kent Watsen <[email-masked]>
Date: Mon, 19 Mar 2018 11:54:14 +0000

Hi Tarko,

> served by bootstrapping server.

This draft does not enable putting an "owner CA trust anchor" on a device
from manufacturing.  Section 5.1 describes devices possessing a list of 
trust anchor certs for *well-known* bootstrap servers and issuers of 
ownership vouchers.  By well-known, we mean something along the lines 
of a MASA (manufacturer assured signing authority), not a deployment-
specific trust anchor for a specific owner.

Insecure zero touch bootstrapping mechanisms have been available for some
time.  What's new with this work is that it is secured.  There cannot be 
an insecure mode, as that would then enable a degradation-attack, whereby
the attacker blocks the device from reaching secured resources, causing 
the device to ultimately land on an insecure resource the attacker controls.

Looking at the above, it seems that your concern hasn't been addressed
yet or, perhaps stated more accurately, I haven't understood it yet.  Can
you try stating it another way?

>> Currently, if needed, vendors could define a vendor-specific parameter

SZTP is usable in real life today, but there is the expectation that, for
devices with multiple interfaces, that a specific port would always be 
used, at least within that owner's organization.  This works pretty well,
but there are cases where optional hardware (i.e., an LTE modem) might
be attached, which can introduce variability in the initial configuration
that should be pushed to the device.  This draft addresses this (kind of)
via the "hw-model" input parameter and name-mangling (e.g., srx250-lte).
This is a little bit hacky, but trying to come up with something better
was more complicated than I wanted to get into at this time.  Vendor-
specific input parameters can be used now, if needed.

Regarding VLANs, I haven't seen this as a problem yet, and I'm not sure
why.  Perhaps it just hasn't come up with the range of devices that I've
been working with though all, I think without exception, support VLANs.
Maybe it has to do with how the devices are deployed (e.g. as an access
gateway) and so VLAN isn't an issue, or maybe deployments have use an
untagged VLAN interface.

I'm unclear how big of an issue this is.  What I see most is that the
NMS has a model of the network and thus knows in advance what initial
configuration individual devices need.  Consider the case where the 
initial config is placed onto a removable storage device, such that
the NMS has to anticipate what the device needs.  

You started this saying that the device may have multiple ZTP capable
interfaces and it's not known which interface might be used during 
installation.  It's important to note that SZTP-itself doesn't care
which interface is used to get the bootstrapping data (it can even
be read off a removable storage device).  What matters is what 
interface(s) the initial config needs to configure, which may not
necessarily be the same as the interface used for the SZTP call.
My claim is that this is fairly predictable within a deployment, 
such that canned responses work great.  Can you provide a more 
concrete example where this doesn't work?

> devices MUST return to zerotouch/enabled?=true in every factory reset 

Agreed, and I think Section 5.1 is saying this, albeit perhaps not clearly
enough.  Could you provide the OLD/NEW text that you'd like to see there?

Kent // contributor

---

From: Tarko Tikan <[email-masked]>
Date: Wed, 28 Mar 2018 14:39:24 +0300

hey,

> specific trust anchor for a specific owner.

OK, fair enough. I'm not sure how this will work out in L3VPN/private 
management scenarios where internet access might not be available.

> the device to ultimately land on an insecure resource the attacker controls.

I fully agree with the security part. But I find this draft interesting 
because it might finally be something that is actually usable in 
production in multivendor environment. While insecure ZTP 
implementations exist today, they are not really usable unless you have 
the strict serial-number to the devicetype to the customer mapping.

> concrete example where this doesn't work?

Agree about the ZTP interface part and in most cases, the interface 
thats being used for ZTP bootstrapping, is the interface thats needs to 
be used towards the SP network.

Concrete example where you cannot predict the interface is device with 
1G copper and 1G SFP interface. Either can be used based on customer 
location:

- customer colocated at the AN site = use copper

- customer connected to AN via fiber = use SFP

-- 
tarko