Subject: Re: [i2rs] Two thoughts on an ephemeral data store

From: "Russ White" <[email-masked]>
Date: Thu, 2 Oct 2014 07:49:51 -0400

> If objects that are always intended to be ephemeral are never going to
> conflict (but may lie on top of) local config, there is never a conflict.
> There is simply referential integrity issues if the underlying local
config is
> deleted and ephemeral config remains.
> 
> If the objects are always disjoint, there is never a conflict.
> 
> But if you permit an ephemeral object to override a local config one
within
> the schema, we have this issue.  Our options would appear to be:

A lot of this discussion centers around the concept of a "configuration
change" and, more specifically, "static routes." This is actually what I was
trying to avoid early on by separating the concept of configuration state
from the concept of forwarding state... But, since we are here, one possible
point -- If I2RS is treated as, essentially, an alternate "protocol process"
running off box, rather than "yet another configuration mechanism," then
this problem falls out. There would be no such thing as a "static route
installed by I2RS," there would only be routes installed in the local
routing table that would be persistent only if the I2RS agent makes certain
to reinstall it after any "system event" that causes the route to fall out
of the table. 

Policy within a protocol (though I generally don't like the idea of
modifying individual routes at the protocol level) would be treated the same
way -- not as a route map or RPL statement "configured" on the device, but
as an individual change to an individual route. If an update is received for
this route, the local agent must remodify the route, rather than assuming
any modifications made would be persistent across updates to that specific
reachability information.

If this is all true, then there _should be_ no problems with something being
configured in two places -- if a single piece of reachability information is
installed into the local routing table twice, normal processing rules apply,
regardless of the source (generally speaking, the route with the lower admin
distance wins, no matter the source). I2RS would never "configure" a "static
route," but rather attempt to install a piece of reachability information
into the routing table, which then either succeeds or fails, depending on
the processing rules involved. The static route configuration isn't touched
by I2RS, so there's no issue of "which configuration wins."

I hope all that made sense.

:-)

Russ

---

From: Jeffrey Haas <[email-masked]>
Date: Thu, 2 Oct 2014 10:03:33 -0400

Russ,

On Thu, Oct 02, 2014 at 07:49:51AM -0400, Russ White wrote:
> change" and, more specifically, "static routes." 

It's important to be really cautious and not conflate this as being just as
simple as "static routes".  For cases that look just like a routing
protocol, sure.  But some of these cases are closer to extending existing 
configuration state, not injecting new forwarding state.

---

From: "Russ White" <[email-masked]>
Date: Fri, 3 Oct 2014 07:44:14 -0400

> simple as "static routes".  For cases that look just like a routing
protocol, sure.
> But some of these cases are closer to extending existing configuration
state,
> not injecting new forwarding state.

So the case we're aiming for is to be able to change, say, the community
string on a BGP route that's sitting in the BGP table, rather than in the
RIB... Two things:

- I consider this use case unwise. If you want to change the community
string (or MED, or Local Pref, or AS Path) on a BGP route, then change the
config so the route is impacted through normal processing on the router,
rather than touching the route directly through an outside interface.
Changing the config so the route is modified moves the problem out of the
(strict bounds of the) I2RS space, and into configuration proper, where the
operator can see the change in the config, etc. Playing with routes as they
sit in the local protocol table without changing the configuration, and
hence without telling the protocol what we're doing, is just asking for
routing loops and other problems, and doing so from the outside so the
operator must look in multiple places to figure out why the problem occurred
is just asking for trouble and long MTTRs, IMHO.

- If we insist on playing with routes in the local protocol tables, then we
need to restrict ourselves to touching _routes_, those individual ephemeral
things that change over time, don't persist over a reboot, and aren't
configured on the box in any static way. Here, again, just like when
interacting with the RIB, we should touch the _route_, not the policy, so
the same rules apply as I pointed out in my original email -- if the route
is updated, the agent needs to be notified, which then needs to notify the
application, which needs to re-apply the policy if needed. Some
implementations might allow the agent to be programed to react to route
changes, but that's completely out of scope here -- it's a local agent
problem.

To put it another way -- we're trying to implement a policy from an external
box by way of directly touching routing information. We are _not_ trying to
influence how the router processes routes. We're _not_ configuring or
modifying policy, we're modifying routing information in various tables.
Different things entirely.

Where we are running into problems is that we didn't start with a clearly
defined set of objectives or definitions. "Configuration" means "slow,"
while "control plane" means "fast" -- a poor definition that's essentially
opened every possible configuration semantic up to I2RS control. "Policy" is
the community string on a BGP route -- not, it isn't. Policy is what happens
to that route because it carries a specific community string or the reason
the community string is changed, not the community string itself. We're
mixing all these different things up; expecting to have a clear cut solution
to a muddy problem. Ain't happenin'.

If we back up and think through what it is we're trying to _do_, we can take
this whole problem of "when do we write to the configuration" off the table.
We're touching _routes_, not policy, no matter which table or database they
happen to live in. Routes are _always_ ephemeral, and the policy indicators
(those things policies use to do things to a route, or expressions of
policy) carried in the route are always ephemeral -- so there shouldn't even
be a discussion about writing something to a configuration. There shouldn't
be anything "writable" in anything I2RS modifies on the router, plain and
simple. If there is, then we've gone out of scope, IMHO.

:-)

Russ

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 3 Oct 2014 10:24:51 -0400

> string on a BGP route that's sitting in the BGP table, rather than in the

And from there you pick an example and play reductio ad absurdum. :-)

That's fine, doing that to specific use cases is helpful.  However, I think
part of the issue is that at the end of the day, this is an *interface*
to the routing system.  Being able to read state has always been pretty
clear.  We're spending all of our time debating the *write* semantics.
They're tricky when they're crossing boundaries.

The use cases where the configuration state we're injecting (configuration
in the netconf sense, basically we've done a write operation) and it's
completely disjoint from existing local configuration has always been pretty
clear.  Whether such things like in a unified datastore with local config or
in something that requires a completely new protocol to set has never been
an issue.

Our issues all come when we want to interact with that local config -
whether in an augmentation sense or a referential sense.

For the moment, let's work through your reductio.

> rather than touching the route directly through an outside interface.

We already have BGP attributes getting "config" via other mechanisms.  AIGP
metric and the bandwidth extended communities are examples of these.

We've previously talked about influencing of routing via programmatic
changes to the cost community.

> hence without telling the protocol what we're doing,

I don't recall any cases where we're saying "go forth and meddle with things
in a way that the underlying system isn't aware of".  Matter of fact, for
the IGP cases, we've explicitly put interference with the underlying LSAs
as out of bounds because the protocols wouldn't deal with it.

If you think we're specifically chartered to meddle with state in ways that
don't make sense, I think you've gotten a bit lost.  Not to mention I can't
see someone supporting a model that has such impact.

> interacting with the RIB, we should touch the _route_, not the policy,

Policy is dynamically changed in lots of ways.  Again, it has to make sense
in the system.

As a random example, JUNOS has had ephemeral policy configuration for years.

> modifying policy, we're modifying routing information in various tables.

Policy is a table depending on one's perspective.

> to a muddy problem. Ain't happenin'.

I want a way to be able to write "state" into boxes that influences
behaviors based on configuration models.  Those models define what one is
trying to do.  People want to do lots of things.

And it's not so much "I".  I already work on a platform that has a defined
protocol and currently one language that lets me do this sort of thing.
Customers want more broadly supported versions of such a thing and better
models to get at and influence the device.

> We're touching _routes_, not policy,

Perhaps you are.  Go forth with the knowledge that others want to do
different things.

- Jeff

---

From: "Russ White" <[email-masked]>
Date: Fri, 3 Oct 2014 14:57:02 -0400

> Policy is dynamically changed in lots of ways.  Again, it has to make
sense in
> the system.

What we need to do is separate policy from policy instruments... A community
string, metric, etc., is _not_ a policy, it's a "policy instrument." In
other words, an I2RS policy can change a community, but the reason you set
the community is a policy, not the community. So what we are saying is two
different things:

1. Find routes with a destination of 10.1.1.0/24; add community 1 to them.
Do this this one time to route currently in the table.

2. For any route with a destination of 10.1.1.0/24, add community 1 to them
starting now, and lasing until I tell you to do otherwise.

The difference is subtle, but important. The first can take the semantic --
"set community 1 on routes matching 10.1.1.0/24." The second must take the
semantic, "install a policy that examines inbound routes and sets community
1 on them if they match 10.1.1.0/24." 

If we're trying to do the second, we're trying to do the wrong thing. We
shouldn't be programming policy into the box -- that's what netconf is for.
We should be trying adjust individual routes based on a policy that's opaque
to the box -- the routing process on box only sees the result of the policy,
not the policy itself. 

> models to get at and influence the device.

So in JUNOS, you have the ability to instruct BGP to install a route map or
RPL statement to change inbound routes, or to set a filter on inbound routes
in addition to existing CLI configured filters, that the user can only see
by examining the scripts running on the box as well as the actual CLI
configured filters? I don't actually think you do... You probably have a set
of instructions that configure something so new filters and policies show up
in the CLI, and another set of options that will let you change actual
routes, but I don't think you have a set of options that allow you to put in
what is essentially an RPL statement without actually having that RPL
statement show up in the configuration, or modify the inbound filter of BGP
without modifying the configuration the user sees through other interfaces.

Again, the difference is subtle, but important. 

If I2RS' goal is to replace RPL configurations in the CLU, then we've
already stepped outside the bounds of the routing system, and into the
bounds of netconf and the configuration/network management systems. 

> > If we back up and think through what it is we're trying to _do_, we
> > can take this whole problem of "when do we write to the configuration"
off
> the table.

Sorry, Jeff, but I don't think the WG actually knows what it wants to do --
we don't have a carefully articulated well defined set of goals or
definitions. As I read the use cases, I don't see _any_ use case that says,
"I want to set an RPL based filter that changes every route received by the
process, or acts as a nonconfigured filter on the box." What I see is, "I
need to inject this route here, remove that route there, or modify that
route over there."

Completely different things.

We need to know what "policy" means, and we need to know what a "route"
means. We need to be able to state clearly and easily where I2RS ends, and
where NETCONF begins. It's not that configuring policy is a "bad thing,"
it's that we need to scope the work to a reasonable set of actual work
items, especially when there are other WG's working in the same space.
Otherwise, we're just trying to boil the ocean -- a sure recipe for failure.

To bring this back to the beginning -- we can play with how to build, store,
etc., the YANG models all day long -- but unless we put some definitions
around what I2RS is _not_ doing, we're never going to be able to make
progress. 

:-)

Russ