Subject: [netmod] what if <running> starts out invalid?

From: Andy Bierman <[email-masked]>
Date: Fri, 17 Jul 2009 09:24:49 -0700

Hi,

Not sure if this is also a 4741bis issue...

Let's say the agent loads a module at boot-time or run-time,
and it immediately invalidates the <running> config?
(e.g., missing mandatory top-level node).

Or the NV-startup contents are not valid for the set
of loaded modules at boot-time? (e.g., missing or unknown data)

It means that the <validate> operation on the <running> config
can fail, because agent implementations are likely to keep
going, and let an operator fix things (may be partially disabled
until this done).  Crash-and-reboot-loop is a bad option,
but I guess it is a valid implementation choice.

The agent will not allow any <edit-config> to the <running>
to succeed unless the validate tests pass (SOP for <running>).

It also means that the <candidate> starts out invalid and the
agent will not let any <commit> succeed until the problems
with the validation have been fixed.

Not sure what text (if any) needs to be written in the YANG draft.

Andy

---

From: Andy Bierman <[email-masked]>
Date: Mon, 20 Jul 2009 02:20:10 -0700

Balazs Lengyel wrote:
> Hello,
> We sad earlier that running must be always valid. Is that a YANG or a
> NETCONF requirement on the server?

So are we saying that if the <startup>
has even 1 missing leaf in it, the NETCONF
agent must shutdown and refuse to run?

Since there is no access control on <startup>,
this means if a hacker can somehow cause the
<startup> to be invalid (by 1 leaf), and
then cause the box to reboot, then the NETCONF
standard mandates that the device is FUBAR
until the operator replaces the <startup> somehow?

What about loading modules are run-time?
Add 1 mandatory leaf, and the box rolls over and dies?
Without any chance for the operator to create the new leaf?

> Balazs

Andy

> Andy Bierman wrote:
>> _______________________________________________
>> netmod mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/netmod
>>
>

---

From: Balazs Lengyel <[email-masked]>
Date: Mon, 20 Jul 2009 10:40:10 +0200

Hello,
During NETMOD the discussions it was stated nomerous times, that the running configuration MUST 
be valid at all times; e.g. this is the reason we can't refer to state data in must 
expressions. While I am not totally happy with this, this was the latest news about running. 
Are we saying now that in some cases running might not be valid?

Also I very strongly think, that if running is valid at all times this MUST be stated in YANG 
and/or in the architecture document. Actually it is a strong requirement on the NETCONF server 
by YANG, that it should keep running valid.

Balazs

-- 
Balazs Lengyel                       Ericsson Hungary Ltd.
System Manager
ECN: 831 7320                        Fax: +36 1 4377792
Tel: +36-1-437-7320                  email: [email-masked]

---

From: Balazs Lengyel <[email-masked]>
Date: Mon, 20 Jul 2009 10:41:29 +0200

Hello,
We sad earlier that running must be always valid. Is that a YANG or a NETCONF requirement on 
the server?
Balazs

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 20 Jul 2009 23:38:29 +0200 (CEST)

Hi,

> agent must shutdown and refuse to run?

Short answer: Yes.

I can't see how an agent can start with an invalid config in the
general case.  In your simple example, somewhere in the code someone
expects this mandatory leaf to exist, and behave accordingly.  If the
code can start without this leaf, it wasn't mandatory - the code
obviously took some default action.  This should be reflected in the
data model.

> Since there is no access control on <startup>,

What?  Where is this specified?

> until the operator replaces the <startup> somehow?

If there is no access control on startup, a hacker can do much more
damage than this...

> Without any chance for the operator to create the new leaf?

If we say that running MAY be invalid, i.e. that the box is supposed
to run with any invalid config, what is the purpose of the validation
anyway?

/martin

---

From: Andy Bierman <[email-masked]>
Date: Tue, 21 Jul 2009 04:59:17 -0700

> obviously took some default action.  This should be reflected in the

How come CLI-based routers don't won't this way?
Could it be because this is way too fragile?

I can't see how a YANG module with a mandatory
top-level node in it can ever be deployed in
a real system.

By definition, it means the manager MUST set the node.
The agent is NOT ALLOWED to set it.  So once the module
is loaded into the system, the agent is dead.

I don't see how <startup> config and new modules
are kept in perfect alignment, within a single
protocol operation.

Andy

---

From: Andy Bierman <[email-masked]>
Date: Tue, 21 Jul 2009 05:34:28 -0700

Martin Bjorklund wrote:
> Agreed.  So do we agree on the above that startup must be valid?
> 

In my code, startup is just a file that is input into <running>,
using an extreme 'continue-on-error' algorithm.  Any bad config
is tossed, and and good config is used.

The operator may be able to fix a broken config,
but certainly not if the box shuts down and
refuses to accept any NETCONF sessions.

> As you said - if there are no top-level mandatory nodes (or within
> NP-containers...) then it will work.  Maybe add something about this
> in the guidelines document?
> 

I don't like the idea that YANG has perfectly valid
constructs that are like poison pills -- just
loading a valid YANG module into an agent should
not force the box to shutdown. We need NETCONF agents
to work with any valid YANG module that could ever be written.

IMO, either we make this work in the NETCONF protocol,
or change YANG so top-level mandatory nodes are not allowed.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 21 Jul 2009 14:10:26 +0200 (CEST)

/martin

---

From: Balazs Lengyel <[email-masked]>
Date: Tue, 21 Jul 2009 16:10:20 +0200

> is loaded into the system, the agent is dead.
In Ericsson such top level nodes are typically created by the network element itself. It is 
always already there immediately, once you loaded/installed the module, and can not be deleted.

Balazs

---

From: Balazs Lengyel <[email-masked]>
Date: Tue, 21 Jul 2009 16:19:09 +0200

Hello,
This all leads me to a nice idea:
- delete-config is mandatory to support on startup (if you have startup).
- you can get a reboot immediately after delete-config of startup
- this means that
a) Every box that supports startup must be able to run with an empty configuration
or
b) Every box that supports startup must itself create a minimum config after 
delete-config/startup, that will allow it to run
or
c) I am wrong somewhere

Balazs

---

From: Andy Bierman <[email-masked]>
Date: Tue, 21 Jul 2009 08:02:39 -0700

Balazs Lengyel wrote:
> In Ericsson such top level nodes are typically created by the network
> element itself. It is always already there immediately, once you
> loaded/installed the module, and can not be deleted.
> 

by definition, that is not a mandatory node, and not
part of the problem.

Andy

---

From: Andy Bierman <[email-masked]>
Date: Tue, 21 Jul 2009 08:17:56 -0700

> - delete-config is mandatory to support on startup (if you have startup).

yes, but there may be access control in your way.
I choose to tag the <delete-config> operation as 'secure',
meaning by default only the 'super user' can invoke it.
(oops - I forgot we have no official users, groups, or
anything else ACM-wise, but IMO the concept of a super-user account
that bypasses all access control is fairly important.)

> a) Every box that supports startup must be able to run with an empty

including all the access control config wiped out -- yes.
that is why it is dangerous to allow anybody to do this.

> or
> b) Every box that supports startup must itself create a minimum config
> after delete-config/startup, that will allow it to run
> or

same as (a).
The agent will always create any nodes it must have,
if they are missing.  Partial or empty startup makes
no difference here.

Andy

---

From: Balazs Lengyel <[email-masked]>
Date: Tue, 21 Jul 2009 17:19:32 +0200

> by definition, that is not a mandatory node, and not
If a list has min-elements 1; it is a mandatory element as defined in YANG chapter 3.1.
It can be a top level node in a YAM.

AFAIK it is not forbidden for the element itself to create the above element.
Balazs

---

From: Andy Bierman <[email-masked]>
Date: Tue, 21 Jul 2009 08:35:04 -0700

> It can be a top level node in a YAM.

yes.  I think there are 3 problems, or features,
depending on how you look at it.
By 'top-level', we of course mean a top-level exposed
data node (could be within a choice or nested NP containers).

   - top-level mandatory=true
   - top-level min-elements > 0
   - anywhere a must-stmt fails

> AFAIK it is not forbidden for the element itself to create the above

correct -- I think the <streams> subtree in the notification RFC
mandates that the agent create 1 entry for the NETCONF stream.
This could be min-elements 1; (IMO).

Only the mandatory-stmt is picky about how the entry got there.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 21 Jul 2009 22:25:40 +0200 (CEST)

> is tossed, and and good config is used.

So what happens if there is a must expression that says that a > b,
and it is not.  Do you delete a or b?  What if both are mandatory?
What if a list has a max-elements constraint and there are more - do
you randomly delete one entry?  What if a must expression in general
returns false, how do you know which node to delete?

I don't think this idea works.  Also, it implies that someone somehow
was able to write invalid data into startup.  If the server ensures
that startup is valid, then this is not a problem.

> refuses to accept any NETCONF sessions.

The point is that if startup (and running) are always valid, this
situation does not occur.

I am not sure what you argue for - is it:

  1) startup is allowed to be invalid
  2) running is allowed to be invalid

> to work with any valid YANG module that could ever be written.

Of course.  Continuing your example, if the new module has a mandatory
top-level node, when it is loaded into the server (implementation
dependant btw), the server must make sure that a value for the
mandatory node is provided.

/martin

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 21 Jul 2009 22:28:05 +0200 (CEST)

> by definition, that is not a mandatory node, and not

I think Balazs means that they use models with mandatory top-level
nodes, and in order to handle it, a "factory default" value is
provided.  The node is still mandatory in the data model, even though
the box provides an initial value.

/martin

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 21 Jul 2009 22:32:09 +0200 (CEST)

> > - delete-config is mandatory to support on startup (if you have startup).

[...]

> > a) Every box that supports startup must be able to run with an empty

[...]

> if they are missing.  Partial or empty startup makes

Agreed.  I think delete-config on startup can be viewed as a reset to
factory defaults, which might be empty.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Tue, 21 Jul 2009 14:08:03 -0700

> Andy Bierman <[email-masked]> wrote:
....
> and it is not.  Do you delete a or b?  What if both are mandatory?

neither.

the node with the must-stmt would get flagged with the error,
so it would be deleted.
that error will ripple up until a 'detachable' point is reached,
or maybe all the way to the top-level node.  All the tainted nodes would
get deleted.

> returns false, how do you know which node to delete?

I think all the entries would get deleted, because I would tag
the parent with the error.  So bye-bye all child nodes,
not just the max-elements-exceeded node.

> that startup is valid, then this is not a problem.

What about a SW image upgrade, or just a reload with new modules
that have some mandatory (or just desired) config?

  1) lock target=startup
  2) delete-config target=startup
  3) copy-config source=@my-new-config target=startup
  4) (offline change CLI parameters for agent to reflect
     new <startup> settings and modules expected)
  5) restart

The agent MUST NOT validate the contents of <my-new-config>
against the current settings/modules for this to work.

I suppose a special operation for <upgrade> would be
cleaner, but the WG is busy with other things now.

Andy

---

From: Phil Shafer <[email-masked]>
Date: Tue, 21 Jul 2009 17:08:31 -0400

>situation does not occur.

My take on this discussion is that invalid startup configs are an
ugly fact of life, and while it would be nice to say "don't do
that", they will inevitably occur.  Do we need to say what devices
should do, or do we leave implementors to do their best?

Scenario one: sw install changes the rules for config but doesn't
update the startup.

Scenario two: startup is on removal media and the user inserts media
with an invalid startup config.

This isn't an easy decision on the device side, since discarding
invalid config may lead to evil acts (no firewall rules) that will
not be noticed until evil things happen, and stopping dead means
your device becomes an inefficient space heater.

FWIW, I went with option two, since (a) a dead device gets noticed
and fixed, (b) open firewalls are bad news, and (c) space heaters
are pretty cool.  If we heat all of space, we won't need spacesuits.

Thanks,
 Phil

---

From: Martin Bjorklund <[email-masked]>
Date: Tue, 21 Jul 2009 23:39:36 +0200 (CEST)

Phil Shafer <[email-masked]> wrote:
> that", they will inevitably occur.

[back pedaling a bit]

Yes, you are right.  I have a problem with Andy's original email, and
also the title of this thread -- I don't think running can ever be
invalid.  I will make sure that my startup is valid at the time it is
written, but as you say, I can not guarantee that the startup file is
valid with the sw when the device boots.  (I can guarantee that
running is valid though).

> Do we need to say what devices

I would prefer the latter.  Then Andy can do his delete and you and I
can make our space heaters.

/martin

---

From: Andy Bierman <[email-masked]>
Date: Tue, 21 Jul 2009 14:39:52 -0700

> are pretty cool.  If we heat all of space, we won't need spacesuits.

really?
is that what the CLI does too?
Just 1 bad CLI command and the router refuses to
do anything at all?

What about plug-in modules that aren't there,
but the config for them is there?  What if the YANG file for
that module isn't even available, not just the instrumentation?

What about 1 module that was updated, but the config was not,
and the YANG update rules are not followed to the letter,
so the startup has invalid config for 1 module?

Tossing the valid config for all the rest of the modules
seems harsh.  The agent can always hard-wire a check-list
of must-have-modules if it wants, and act like a brick
when the need arises.

I think I am agreeing with you that it is an implementation
specific matter how the agent will handle certain errors
in the <startup> config.

Andy

---

From: Phil Shafer <[email-masked]>
Date: Tue, 21 Jul 2009 17:31:33 -0400

>can make our space heaters.

Sounds good.  I agree that running should never be invalid.  If a
box doesn't enforce a rule on running, that it's not a rule.  (Or
it's a deviation from the rule).

Thanks,
 Phil

---

From: Phil Shafer <[email-masked]>
Date: Tue, 21 Jul 2009 17:35:21 -0400

>really?

Honest.

>is that what the CLI does too?

Nope.

>Just 1 bad CLI command and the router refuses to

Of course not.  A user at the CLI gets an error and the
command is ignored.  But at boot time (startup), there's
no user to give an error to.

>specific matter how the agent will handle certain errors

Cool.  So what do we need to say in the spec?  Just mention
that care should be taken that startup be valid, but the
real world might interfer and the behavior of the device
in the face of an invalid startup config is left to the device
implementor?

Thanks,
 Phil

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 22 Jul 2009 08:12:45 +0200

> the box provides an initial value.

So these "factory defaults" are not (necessarily) part of the data
model, and these mandatory nodes are not set by the manager. It looks
like we are going to repeat the same discussion we had about two years
ago ...

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 22 Jul 2009 10:04:59 +0200 (CEST)

> like we are going to repeat the same discussion we had about two years

No, I hope not.  "factory defaults" should maybe be "factory initial
data" - it is simply the initial set of data, if any, in the data
store.

/martin

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 22 Jul 2009 10:19:51 +0200

> data" - it is simply the initial set of data, if any, in the data

Well, they are set "by default", without any manager being involved. My
personal conclusion from that old discussion was:

1. The only sensible definition of "mandatory" that doesn't depend on a
particular device behaviour is this: A configuration parameter is
mandatory if it is required for a (sub)system or protocol to operate
properly.

2. The notion of default values is broader than what can be possibly
specified in a data model.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Balazs Lengyel <[email-masked]>
Date: Wed, 22 Jul 2009 12:33:47 +0200

Hello Martin,
You should out this into 4741bis! Maybe add it to TRAC as a start?
Balazs

---

From: Andy Bierman <[email-masked]>
Date: Fri, 24 Jul 2009 17:20:25 -0700

> running is valid though).

I think you are right.

We need to document all this correctly in yang/usage/4741bis.
If we are saying that certain errors are OK to
continue (invalid-value) but others are fatal errors
(mandatory/min-elements), then it needs to be written down.

It is OK for the agent to prune 'bad startup config'
as it is loaded into <running>.  The operator should
be able to choose "stop-on-error" or "continue-on-error"
for processing the startup config. (e.g., a CLI parm).

Even if "continue-on-error" is done by the agent,
the <running> config MUST be completely YANG-validated OK
or the agent will not start.

The problem is that we have no way to send out notifications
to the operator that this problem is happening.  But via polling,
the device will disappear from radar, and get attention, as Phil noted.

The "rollback-on-error" option also applies to <startup>.
It could mean "try the current <startup>, and go back to
the last <startup> if it does not validate".  (But I digress... ;-)

In summary:

  - <candidate>, <startup> and <url> can fail the <validate> operation
  - if <running> fails the initial startup <validate> operation, the
    agent must not accept NETCONF sessions (cannot force a shutdown).

So what happens if the device capabilities or sub-systems
change during run-time?  This goes back to the old question
of pre-provisioning.  What if 1 implementation keeps the 'dead' config,
and another deletes it, causing an invalid <running> config?

I assume its OK to have subtrees in the <running> config
that may not have hardware or for whatever reason are
not installed at the moment.

But the manager cannot tell (other than adding 'enabled' flags to
all the data structures), the difference between live config
and pending config.  It's possible that the agent implementation
doesn't even know for sure.  That support would need to be built-in.

I think the standard should eventually deal with this somehow.

Andy