Subject: [i2rs] Architecture Discussion 2: Persistence (ephemeral vs. permanent)

From: Alia Atlas <[email-masked]>
Date: Thu, 12 Sep 2013 10:47:52 -0400

In draft-ietf-i2rs-architecture-00 Sec 5.2, it says:

"The I2RS Agent will not attempt to retain or reapply state across

   routing element reboot.  Determination of whether state still applies
   depends heavily on the causes of reboots, and reapplication is at
   least as likely to cause problems as it is to provide for correct
   operation.  [[Editor's note: This topics needs more discussion in the
   working group.]]"

I would like to start that discussion and rapidly conclude it with the needed

changes (removal of the Editor's note or otherwise).

My personal impression is that simplifying to just ephemeral state was

viewed as a good idea.  What do others think?  Does anyone strongly

disagree?

Thanks,

Alia (WG-chair hat off)

---

From: Joe Marcus Clarke <[email-masked]>
Date: Thu, 12 Sep 2013 11:02:18 -0400

> viewed as a good idea.  What do others think?  Does anyone strongly

I agree that ephemeral state is a good idea.  But I think we need to be
a bit clearer on where the ephemerality lies.  That is, if the Agent
goes away independent of the routing system, does the state go away?
Does the RS retain the state until reboot of the network element?

IMHO, the ephemeral state should be maintained with the Agent, and if it
goes away (if it can go away independent from the RS), then the state
should go away with it.

Joe

> 
> 
> Thanks,
> 
> Alia (WG-chair hat off)
> 
> 
> 
> _______________________________________________
> i2rs mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/i2rs
> 

-- 
Joe Marcus Clarke, CCIE #5384,         |          |
SCJP, SCSA, SCNA, SCSECA, VCP        |||||      |||||
Distinguished Services Engineer ..:|||||||||::|||||||||:..
Phone: +1 (919) 392-2867         c i s c o  S y s t e m s
Email: [email-masked]

----------------------------------------------------------------------------

---

From: Alia Atlas <[email-masked]>
Date: Thu, 12 Sep 2013 11:11:00 -0400

Hi Joe,

> goes away (if it can go away independent from the RS), then the state

[Alia] This is a really good point to clarify.  If the I2RS Agent goes
away, cleaning up
ephemeral state that it had programmed seems like it would require an
additional gate-keeper
to keep track of that state and do the cleaning up.   That seems painful to
me in terms of
complexity and storage.  I am assuming an unplanned Agent departure rather
than an orderly
disabling of the Agent.     If the Agent simply fails, then IMHO, the
ephemeral state should
remain - but it is no longer owned by a particular client and could be
overwritten...  If an Agent is
disabled, then it can notify each client that that client's state has been
preempted and do an
orderly (?) removal of the state.  The gotcha with the orderly removal of
the state would be understanding
the dependencies of the state so that it can be cleanly removed.

[Alia] So, I think we're picturing almost opposite things for this behavior
- possibly based upon the
particular case.   What do you think?

Alia

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 12 Sep 2013 11:11:29 -0400

My perspective on ephemeral was that it was ephemeral from the 
perspective of the device with the I2RS agent.  If that device reboots, 
the state is gone.  No persistent storage.

I do not want the I2RS agent to have to maintain TCP connections with 
all I2RS clients that have state in the device.  Such a connection is 
overhead both in terms of state and packets, for very little gain.  In 
general, if the client created the state, it should remain until the 
client explicitly clears it.  Presumably, as part of operational 
behaviors, the clients take care to keep track of what they have done in 
a robust fashion.
There are ways to create cleanup mechanisms, but I would not recommend 
that as a common practice.

Yours,
Joel

---

From: Joe Marcus Clarke <[email-masked]>
Date: Thu, 12 Sep 2013 11:18:16 -0400

> that as a common practice.

I was thinking the same thing from the Client perspective.  That is, the
state is maintained by the Agent.  My comment was what happens if the
Agent goes away, but the RS remains (i.e., a cold start of the Agent
independent from the underlying network element RS)?  The reason

I think the RS state needs to go away with the Agent is that if the
Agent reboots, and it comes back, how could Clients operate on their
previous state?  The Agent would have no knowledge of that state since
the Agent maintains to persistent storage.  Therefore, you could get
state in the RS that is impossible to remove without a reboot of the
underlying NE.  That is bad.

Joe

---

From: Joe Marcus Clarke <[email-masked]>
Date: Thu, 12 Sep 2013 11:22:16 -0400

On 9/12/13 11:11 AM, Alia Atlas wrote:
> Hi Joe,
> 
> On Thu, Sep 12, 2013 at 11:02 AM, Joe Marcus Clarke <[email-masked]
> <mailto:[email-masked]>> wrote:
> 
> to keep track of that state and do the cleaning up.   That seems painful
> to me in terms of
> complexity and storage.  I am assuming an unplanned Agent departure
> rather than an orderly
> disabled, then it can notify each client that that client's state has
> been preempted and do an
> orderly (?) removal of the state.  The gotcha with the orderly removal
> of the state would be understanding
> [Alia] So, I think we're picturing almost opposite things for this
> behavior - possibly based upon the 

See my response to Joel.  I think we have to deal with this complexity
because if we don't and the Agent goes away, how could Clients then come
in to remove their state?  That is, the RS retains the Agent-programmed
state, but when the Agent reboots, it has no state.  At the worst, we
have RS state that a Client cannot manipulate.  At best, we would open
RS state to Clients that did not make the original change.

I think we have to say, to an implementor, if the Agent goes away, the
RS must handle cleaning up all state installed by that Agent.

Joe

>     > [email-masked] <mailto:[email-masked]>
>     Phone: +1 (919) 392-2867 <tel:%2B1%20%28919%29%20392-2867>         c
>     i s c o  S y s t e m s
>     Email: [email-masked] <mailto:[email-masked]>
>

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 12 Sep 2013 11:22:03 -0400

Okay.  Then I misunderstood your note and I agree with you.  If the I2RS 
Agent in a device goes down, all the state it created in the device 
needs to go away.

Thank you,
Joel

---

From: Alia Atlas <[email-masked]>
Date: Thu, 12 Sep 2013 11:31:04 -0400

> RS state to Clients that did not make the original change.

[Alia] I don't think that we've included locking for I2RS - so I can't see
how we'd have
RS that a client cannot manipulate.  Whether state is removed(changed back
to the
default)  or left, we will definitely have the ability for a Client to add
or change the RS
state.

> RS must handle cleaning up all state installed by that Agent.

[Alia] I think that it could be either way - that the state remains but can
be overwritten
or that the RS has to clean up the state.  I'm concerned that in the RS
cleans up the state
case, there is no way for the Client to learn that its state has been
removed (assuming a
failure on the part of the Agent rather than an orderly removal).  I'd
prefer to see the state
remain for that reason for the Agent-failure case.  What do you think?

Alia

---

From: Joe Marcus Clarke <[email-masked]>
Date: Thu, 12 Sep 2013 11:44:14 -0400

On 9/12/13 11:31 AM, Alia Atlas wrote:
> [Alia] I don't think that we've included locking for I2RS - so I can't
> see how we'd have
> RS that a client cannot manipulate.  Whether state is removed(changed
> back to the
> default)  or left, we will definitely have the ability for a Client to
> add or change the RS
> state.

Fair enough.

> [Alia] I think that it could be either way - that the state remains but
> can be overwritten

I think the risk of an Agent reboot allowing Client B to change Client
A's state warrants the RS cleanup.

Plus, if the Client is subscribed to a pub-sub bus then it could
presumably learn about an Agent cold start via a published message.  At
that point, the client could re-establish its state to the Agent.

If the Client is not part of the bus, it could periodically connect to
the Agent and learn that it was rebooted via the protocol...yeah, this
sounds a bit like SNMP, but it seems reasonable to me.

Joe

---

From: Alia Atlas <[email-masked]>
Date: Thu, 12 Sep 2013 12:29:03 -0400

Hi Joe,

More in-line (good to have the conversation)

> A's state warrants the RS cleanup.

[Alia] The risk that client B comes and changes the state after an Agent
reboot justifies the RS cleanup of that state before??  Remember that
Client A and B aren't supposed to be overwriting each other's data.  We've
defined that as an error case and that policy between the clients should
prevent it.

> that point, the client could re-establish its state to the Agent.

[Alia] Hmm, I know there was a draft discussed about pub-sub methods in
general - but I don't think that we've really nailed down how pub-sub would
work.  Absent anything, I'd assume that the I2RS agent is whom the client
registers with to get its notifications.  Now, if we want to have a
different system being a bus (with the scale implications there), that
might be different - but I don't think we've really talked about that.  So,
my assumption is that the Agent provides the notifications to Clients that
have registered - and when the Agent reboots, there are no Clients
registered to get the notifications.  If you have a different model (or
different thoughts on what to do for the pub-sub), want to start that
discussion (perhaps in a different thread)?

> sounds a bit like SNMP, but it seems reasonable to me.

[Alia] It doesn't sound reasonable to me... at least not to avoid what
should be an error condition.  Can we discuss more?  Do you have additional
justification for the significant added complexity.

Regards,
Alia

---

From: Joe Marcus Clarke <[email-masked]>
Date: Thu, 12 Sep 2013 13:33:06 -0400

On 9/12/13 12:29 PM, Alia Atlas wrote:
> Client A and B aren't supposed to be overwriting each other's data.
>  We've defined that as an error case and that policy between the clients
> should prevent it. 

It's an error because the Agent says it's an error because it knows the
current state.

> general - but I don't think that we've really nailed down how pub-sub
> would work.  Absent anything, I'd assume that the I2RS agent is whom the
> client registers with to get its notifications.  Now, if we want to have
> a different system being a bus (with the scale implications there), that
> might be different - but I don't think we've really talked about that.
>  So, my assumption is that the Agent provides the notifications to
> Clients that have registered - and when the Agent reboots, there are no
> Clients registered to get the notifications.  If you have a different
> model (or different thoughts on what to do for the pub-sub), want to
> start that discussion (perhaps in a different thread)?

I had envisioned that as a potential model, too.  I thought TCP
connections would be one way to ensure that the Client knew the pub-sub
mechanism was still in place, and if that got torn down, the Client
would know to renegotiate the session with the Agent.

Since TCP is expensive, another model might be a UDP thing, where the
Client wouldn't know if the Agent went away.  But in this case, the
Client would likely have a periodic check to the Agent to make sure the
Agent is still healthy and still the Client would know if the Agent
rebooted.

A third model I thought of was a broker model in which the Agent
rebooting would be a broadcast event to all subscribers.

In either case, I saw that the Client would have some means of knowing
the Agent went away.

> should be an error condition.  Can we discuss more?  Do you have
> additional justification for the significant added complexity.

One other thing I haven't raised is the operator impact.  How else could
an operator (i.e., owner/admin of a device) flush the I2RS changes?
There might be a malicious change or just some bad state, and they want
to clear that up.  A restart of the Agent could do it without reloading
the device.

Joe

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 12 Sep 2013 21:06:42 +0200 (CEST)

> remain for that reason for the Agent-failure case.  What do you think?

It seems to me that the important thing is that the RS state and Agent
state needs to be consistent.  Since the state is not persistent, at
least a reboot of the entire device will remove all RS and Agent state
- so this is an event that a client must be able to learn about.

Depending on the implementation, there may be other events that lead
to the same result (state removed).  If the Agent is implemented as
one OS process and keeps its state there, a restart of this process
must also remove the RS state (otherwise we have an inconsistency).
But I think it would be wrong to assume that all implementations do
this.  Maybe there are no processes at all, or maybe the Agent state
is kept intact even if the Agent process restarts.

As someone noted, maybe you want an explicit operation in I2RS to
trigger this event - without rebooting the device, and without
assuming that there is an Agent process that can be restarted.

So I think it would be better to define how a client can detect that
the state has been removed, and let implementations trigger this event
when they have to.

/martin

---

From: Joe Marcus Clarke <[email-masked]>
Date: Thu, 12 Sep 2013 15:24:40 -0400

> the state has been removed, and let implementations trigger this event

I agree with this.  Would you also agree, then, that the Agent should be
the keeper of the state, and when it dies, so does the state it added to
the RS?

Joe

---

From: Alia Atlas <[email-masked]>
Date: Thu, 12 Sep 2013 15:27:59 -0400

Hi Martin,

in-line

On Thu, Sep 12, 2013 at 3:06 PM, Martin Bjorklund <[email-masked]> wrote:

> > On Thu, Sep 12, 2013 at 11:22 AM, Joe Marcus Clarke <[email-masked]
> >wrote:
> can

[Alia] Implementations will vary, but my assumption has been that,
conceptually,
the I2RS Agent stores the client identity and so on associated with data
that is
written - while the RS continues to store just the subset of what was
written.

> the state has been removed, and let implementations trigger this event

[Alia] I agree that building in the ability to notify a client that all
I2RS state
or all of that client's state has been removed is useful.  However, doing so
outside of the I2RS agent seems problematic...

Alia

---

From: Alia Atlas <[email-masked]>
Date: Thu, 12 Sep 2013 15:49:30 -0400

Hi Joe,

In-line responses again.

> It's an error because the Agent says it's an error because it knows the

[Alia] But we (the WG) said that it was an error because we expect
coordination
between the applications to avoid multiple writers of the same data.  It's
not an
error because the Agent says so - the Agent would just report back what
happened
(successful write or fail due to higher priority).

> Agent is still healthy and still the Client would know if the Agent

[Alia] As Joel had said, I think we're trying to avoid the need for
persistent
connections just to confirm liveness.  It's a heavy-handed way to do it.
 That said,
we could use a mechanism to determine if the RS is up and if the associated
Agent is up.  Of course, there can be multiple Agents per RS - each
supporting
different services or such. For a router reboot, a Client that is learning
topology
would hear very quickly that a router has failed or appeared.

[Alia] So - developing a liveness detection mechanism is useful.  But what
would
be useful is the ability for the RS to notify interested clients on a
failure.  Otherwise,
what frequency would suffice for checking - when traffic can be being
actively dropped
or misrouted if the Agent has failed.    Also - are we designing for the
error-case of
a crashed Agent rather than the normal case of a gracefully shut down Agent?

> rebooting would be a broadcast event to all subscribers.

[Alia] I've been assuming, I guess, that Clients would hear about router
failures via
the topology DM based from the IGP.

> In either case, I saw that the Client would have some means of knowing

[Alia] I can see that if the Client knew that the Agent went away
(ungracefully), that
having the associated RS clean up state would be fine.  I haven't yet heard
a fully baked
design where the client would know rapidly.  We (you?) could come up with
one, but
that needs to be written down and agreed to.  Without that, I hear that the
Client's state
might suddenly disappear without the Client being able to tell - and am
much more comfortable
leaving it there to be removed either via CLI or to be read and then
written again by the Client
when it reconnects.

[Alia] To be clear, I didn't have an opinion about this aspect before you
raised the question -
and I think that exploring the reasonable solutions and requirements for
them is a good idea.

> to clear that up.  A restart of the Agent could do it without reloading

[Alia]  Why wouldn't using the CLI to flush the I2RS changes work?  Also,
that
is a graceful restart of the Agent - where the Agent can tell each Client
that all
its written data has been preempted because the Agent is rebooting.  I'm
also
concerned that the behavior make sense for if/when the Agent fails.  I
could be
persuaded not to worry about that (Agents shouldn't crash) - but I'd prefer
not to
mandate complexity for that case.

Alia

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 12 Sep 2013 22:06:19 +0200 (CEST)

> the keeper of the state, and when it dies, so does the state it added to

I agree that it should be a valid implementation strategy to let the
Agent keep the state, and thus if it dies, such an implementation must
remove the state state it added to the RS.  But I don't think all
implementations have to do it this way.  All implementations must do:
"if i2rs state gone then also rs state gone".

/martin

---

From: Joe Marcus Clarke <[email-masked]>
Date: Fri, 13 Sep 2013 00:58:29 -0400

On 9/12/13 3:49 PM, Alia Atlas wrote:
> between the applications to avoid multiple writers of the same data.
>  It's not an

My point was that the Agent knows the priority.  This isn't passed to
the RS.

> different services or such. For a router reboot, a Client that is
> learning topology
> [Alia] So - developing a liveness detection mechanism is useful.  But
> what would

I was raising the point of both.  In the graceful case, the Agent clears
all state and notifies all Clients.  But the crash is a possibility, and
we should stipulate what should happen to state.

I don't think we want the RS doing anything to the Clients directly.  I
think perhaps an Agent->Client heartbeat is a useful thing if the Client
asks to be notified about changes.

> the topology DM based from the IGP. 

Router failure of Agent failure?

> having the associated RS clean up state would be fine.  I haven't yet
> heard a fully baked
> design where the client would know rapidly.  We (you?) could come up
> with one, but
> that needs to be written down and agreed to.  Without that, I hear that
> the Client's state

I've been kicking around the idea of a pub-sub "heartbeat."  This would
give the Client awareness of the Agent's "thereness."  If the heartbeat
carried a boot count, the Client might not need for multiple dropped
heartbeats before reapplying state.  I don't know if this is an ideal
way to confirm liveness of the agent, though I'm happy to propose it
more formally.  I think the exact approach would depend on what the
pub-sub mechanism is determined to be.

> [Alia]  Why wouldn't using the CLI to flush the I2RS changes work?

It would.  I didn't realize you were okay with a graceful restart of the
Agent purging all RS state for which it is responsible.  If you are,
then that is handled.

>  Also, that
> is a graceful restart of the Agent - where the Agent can tell each
> Client that all 
> persuaded not to worry about that (Agents shouldn't crash) - but I'd
> prefer not to

I think we should.  Because if there is a crash from the Agent, how then
can an admin purge state?  There would no longer be a clean way to do
this.  I'm trying to think what an operator might like here.

Joe

---

From: Russ White <[email-masked]>
Date: Fri, 13 Sep 2013 07:47:55 -0700

> this.  I'm trying to think what an operator might like here.

IMHO:

- no heartbeat or any other persistent connection
- if the controller crashes, on startup up it should connect to all the clients it knows about (some controller state needs to be persistent across reboot) and send at least one null or other command
- each packet/command the controller sends contains a "boot count," or "session I'd," or something like that; this must increment each time the controller reboots or resets its state
- if a client receives a command/packet with the "wrong" session I'd (or whatever it's called) it clears any state associated with that controller

This doesn't require the agent to keep persistent state, but allows current state to remain without persistent communication. It's possible to design a graceful restart mechanism (like bgp), but lets leave it out of the discussion until we get a better handle on what the protocols and models actually look like. We can speculate, but the devil is in the details in graceful restart.

:-)

Russ

<><
[email-masked]
?@?.com (engineer without a portfolio)

---

From: Alia Atlas <[email-masked]>
Date: Fri, 13 Sep 2013 10:57:08 -0400

Hi Joe,

Summarizing up top (but leaving context)

For a graceful Agent restart (i.e. not a crash), I think the behavior can
be implementation-dependent.  The implementation can either notify all
Clients that have data written that the state is being preempted/torn down
or it can just notify all Clients (that it knows of) that the Agent is
going down.  These are important notifications to capture in the I2RS Agent
Info-Model.  I don't think that either needs to be explicitly required as
the only mode.  I am certainly willing to be convinced otherwise.

For the case of an Agent crashing, I'm quite reluctant to create a
heartbeat or persistent connection.   This is a lot of complexity to add
for what should be a quite rare case.  If the Agent crashes, leave the
state and the Client will get an error when it tries to connect or
otherwise request something.  This does argue that on connection, the
Client should get an indication of something like an Agent boot count
/session id (and boot time).   Russ's idea of storing known clients for the
agent to connect to on reboot makes sense to me.

I really dislike the idea of adding significant complexity to handle an
edge failure case.

Alia

---

From: Joe Marcus Clarke <[email-masked]>
Date: Fri, 13 Sep 2013 13:57:11 -0400

> - if a client receives a command/packet with the "wrong" session I'd (or whatever it's called) it clears any state associated with that controller

This approach is fine with me, too.  I just didn't know if a heartbeat
would be less desirable than having the Agent maintain a list of
interested Clients.

And for terminology sake, I think you mean Agent not "controller?"  The
controller would be more like the Client.

Joe

---

From: "Russ White" <[email-masked]>
Date: Fri, 13 Sep 2013 16:51:00 -0400

> This approach is fine with me, too.  I just didn't know if a heartbeat
would be
> less desirable than having the Agent maintain a list of interested
Clients.

Okay, I'm lost in the terminology again. :-) I think of the controller as
being the off box device that's doing the injection, while the agent is the
on box piece doing the actual manipulation of the devices routing
information. So I'm thinking that the "controller" (or the "client?") would
be one some sort of box that would be likely to have a spinning drive and
the accompanying database constructs to keep the information required to
persist over a reboot --while the "agent," residing on the router/forwarding
device itself, would be more likely to be lightweight, etc. Further, the
point seems, to me, to allow the router/forwarding device to reboot into a
"clean state" represented by local configuration. 

To wit --if I've committed a command that causes the router/forwarding
device to crash, I don't want that command to persist when it reboots, nor
do I want it to be in some unknown state that might, or might not, include
stuff put into policy from the "controller" over the time it's been
operating. A clean slate to "locally stored config" seems like the safest
bet all around (?).

:-)

Russ

---

From: Joe Marcus Clarke <[email-masked]>
Date: Sat, 14 Sep 2013 12:47:02 -0400

> "clean state" represented by local configuration. 

Yes, I'm with you.  Controller == I2RS Client.  The part of my email you
quote seems in line with what you've written here.

In your previous email you seemed to use client and controller to mean
different things.  At one point you said,

"if the controller crashes, on startup up it should connect to all the
clients it knows about (some controller state needs to be persistent
across reboot) and send at least one null or other command"

To me, I read this as:

"if the _Agent_ crashes, on startup up it should connect to all the
_Clients_ it knows about (some _Agent_ state needs to be persistent
across reboot) and send at least one null or other command"

Did I read that right?

> operating. A clean slate to "locally stored config" seems like the safest

Agreed (I think everyone is in agreement on that), but in this case I
look at the Agent as possibly being a separate isolated process from the
router/forwarding device.  So the Agent can crash independent of the
forwarding device.  I'm saying that if that happens, the state in the RS
that was put there by the Agent should be removed.

Joe

---

From: Joe Marcus Clarke <[email-masked]>
Date: Sat, 14 Sep 2013 12:50:19 -0400

On 9/13/13 10:57 AM, Alia Atlas wrote:
> For a graceful Agent restart (i.e. not a crash), I think the behavior
> can be implementation-dependent.  The implementation can either notify
> all Clients that have data written that the state is being
> preempted/torn down or it can just notify all Clients (that it knows of)
> that the Agent is going down.  These are important notifications to
> capture in the I2RS Agent Info-Model.  I don't think that either needs
> to be explicitly required as the only mode.  I am certainly willing to
> be convinced otherwise.

I think I'm on the same page with you here.  If the Agent is being
gracefully restarted, it flushes state, accepts no further connections,
and notifies all known Clients that state is being torn down.  That
works for me.

> /session id (and boot time).   Russ's idea of storing known clients for
> the agent to connect to on reboot makes sense to me.

As I said in that other thread, I'm okay with the persistent list of
Clients.  I didn't think that would be more desirable that the
heartbeat.  However, I think something needs to be done to notify
Clients in case of a crash.  Else they might think everything is healthy
when it's not.  It might be minutes, hours, days, never(?) before they
reconnect to learn otherwise.

Joe

> On Fri, Sep 13, 2013 at 12:58 AM, Joe Marcus Clarke <[email-masked]

---

From: Scott Whyte <[email-masked]>
Date: Sat, 14 Sep 2013 20:41:02 +0100

On Thursday, September 12, 2013, Alia Atlas wrote:

> On Thu, Sep 12, 2013 at 11:22 AM, Joe Marcus Clarke <[email-masked]<javascript:_e({}, 'cvml', [email-masked]');>
> > wrote:
>
>> > On Thu, Sep 12, 2013 at 11:02 AM, Joe Marcus Clarke <[email-masked]<javascript:_e({}, 'cvml', [email-masked]');>
>> > <mailto:[email-masked] <javascript:_e({}, 'cvml',
>> [email-masked]');>>> wrote:
>> >

I don't understand why ephemeral state is held by the agent.  The client
has to track what it sent to which agent.  Surely a client could easily be
informed an agent has lost its brain, and ask for a dump of the RS state it
is eligible to receive from that agent?  Failing in place is a very useful
thing, and what's of interest here is the RS state not the agent state.

>> >     > [email-masked] <javascript:_e({}, 'cvml', [email-masked]');><mailto:
>> [email-masked] <javascript:_e({}, 'cvml', [email-masked]');>>
>> >     Email: [email-masked] <javascript:_e({}, 'cvml',
>> [email-masked]');> <mailto:[email-masked] <javascript:_e({},
>> 'cvml', [email-masked]');>>
>> >
>> >
>> Email: [email-masked] <javascript:_e({}, 'cvml',
>> [email-masked]');>
>>
>>

-- 
There was never any more inception than there is now,
Nor any more youth or age than there is now,
And will never be any more perfection than there is now,
Nor any more heaven or hell than there is now.
  -Walt Whitman

---

From: Scott Whyte <[email-masked]>
Date: Sat, 14 Sep 2013 20:41:03 +0100

On Thursday, September 12, 2013, Alia Atlas wrote:

> On Thu, Sep 12, 2013 at 11:44 AM, Joe Marcus Clarke <[email-masked]<javascript:_e({}, 'cvml', [email-masked]');>
> > wrote:
>

A pub-sub method makes a lot of sense here.  It offloads reliability out of
the I2RS protocols and into the systems supporting client-agent
communications.

I'm not sure I agree that a client is a direct subscriber, at least in the
sense that brokers are very useful for scalability.  Using them I can
design a system where each agent talks to N brokers, where N is reasonable
for me (whether 2 or 20), and the broker-agent communication is independent
of the broker-client communication.  It might be very nice to have the
broker-agent communication be UDP and the broker-client communication be
TCP for example, where I purposefully put the broker topologically close to
the agents and the clients can be hierarchically distributed throughout the
network.

I also think we need to consider that SDN enables a distributed control
plane, but IMO its necessary for I2RS to provide specific transport /
format requirements for all the various latency cases.  Giving SDN
implementors the ability to solve their specific needs via a standardized
way to negotiate capabilities seems to have much broader utility.

In the same vein, I don't think we can just limit the information an I2RS
agent provides to traditional "routing sytem" control plane state.  For
example, data plane state greatly impacts TE decisions.  Someone may want
to get 3000+ interface stats off a box every 30ms in order to do very
real-time TE, and someone else may want that data to do billing.  Let's
enable both at the same time.

> justification for the significant added complexity.

Given push and pull capabilities, having the agent push a notification that
initiates some kind of state synchronization on the client would be ideal.
 Push capabilities have a lot of nice features beyond a heart-beat, but
that probably does deserve another thread.

-Scott

---

From: Scott Whyte <[email-masked]>
Date: Sat, 14 Sep 2013 20:47:41 +0100

On Sat, Sep 14, 2013 at 8:41 PM, Scott Whyte <[email-masked]> wrote:
>> reboot justifies the RS cleanup of that state before??  Remember that Client
>> A and B aren't supposed to be overwriting each other's data.  We've defined
>> that as an error case and that policy between the clients should prevent it.
>>
>>>
>> different system being a bus (with the scale implications there), that might
>> be different - but I don't think we've really talked about that.  So, my
>> assumption is that the Agent provides the notifications to Clients that have
>> registered - and when the Agent reboots, there are no Clients registered to
>> get the notifications.  If you have a different model (or different thoughts
>> on what to do for the pub-sub), want to start that discussion (perhaps in a
>> different thread)?
>
>
> sense that brokers are very useful for scalability.  Using them I can design
> a system where each agent talks to N brokers, where N is reasonable for me
> (whether 2 or 20), and the broker-agent communication is independent of the
> broker-client communication.  It might be very nice to have the broker-agent
> communication be UDP and the broker-client communication be TCP for example,
> where I purposefully put the broker topologically close to the agents and
> the clients can be hierarchically distributed throughout the network.
>
> plane, but IMO its necessary for I2RS to provide specific transport / format

s/necessary/unnecessary/

> requirements for all the various latency cases.  Giving SDN implementors the
> ability to solve their specific needs via a standardized way to negotiate
> capabilities seems to have much broader utility.
>
> example, data plane state greatly impacts TE decisions.  Someone may want to
> get 3000+ interface stats off a box every 30ms in order to do very real-time
> TE, and someone else may want that data to do billing.  Let's enable both at
> the same time.
>
>
>>
>>
>>>
> Push capabilities have a lot of nice features beyond a heart-beat, but that
> probably does deserve another thread.
>
> -Scott
>
>
>>
>>
>> Regards,
>> Alia
>>
>>>
>>>
>>> Joe
>>>
>>> --

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Sun, 15 Sep 2013 16:11:45 +0200

> the agents and the clients can be hierarchically distributed throughout the

I am not sure what exactly a 'broker' is. I did not see this term
defined in the architecture document nor in the problem statement
document. Whatever this 'broker' is precisely, is it needed for I2RS
1.0 or something that may come along later?

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Scott Whyte <[email-masked]>
Date: Sun, 15 Sep 2013 20:54:06 +0100

> 1.0 or something that may come along later?

A pub-sub broker is an intermediary commonly used in distributed
software applications.  I was responding to Alia's assertion
client-agent communication is always direct; I don't think that will
be true even for I2RS 1.0 deployments and assuming there will be
intermediaries certainly affects the data model (in the RFC3444
sense).

-Scott

> Fax:   +49 421 200 3103

---

From: "Joel M. Halpern" <[email-masked]>
Date: Sun, 15 Sep 2013 16:34:23 -0400

The model that is described in the I2RS archtiecture currently does not 
talk about, assume, or use a pub-sub broker.

The use of broker in the WG discussion has been about an I2RS client 
that is making requests on behalf of other entities.  The protocol those 
entities use with the "broker" might be I2RS< or something else.  I2RS 
itself does not care.  The reason we care about these brokers is that 
they motive having two identities associated with a request, the client 
that sent the request and the originator of the request.

Yours,
Joel

---

From: Joe Marcus Clarke <[email-masked]>
Date: Tue, 17 Sep 2013 15:28:38 -0400

> 1.0 or something that may come along later?

It was not described in the architecture, but it is not an entirely
foreign concept to I2RS discussions.  Nancy Cam-Winget brought it up
during the IETF-87 session, and it is described in
draft-camwinget-i2rs-pubsub-sec-00.  I realize this is not a WG document
yet, and that the broker is not an element of the architecture yet, but
a broker does represent a way of doing scalable pub-sub.

Joe

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 17 Sep 2013 21:51:32 +0200

> a broker does represent a way of doing scalable pub-sub.

If brokers do impact data or information modeling decisions, I think
they should be discussed in the architecture explicitely. If the
architecture document then concludes that all that is needed to
support brokers is to be able to deal with multiple identities, fine.
Right now, it is kind of difficult for people not too deeply involved
to relate WG mailing list discussions to the content of WG documents.

/js

---

From: "Toru Okatsu (tokatsu)" <[email-masked]>
Date: Wed, 18 Sep 2013 04:29:04 +0000

Hi Joe and Alia,

I also think that if the agent has gone, information created by the agent in the RS should go away.
It is like when the BGP process has gone, BGP routes should go away

In terms of the notification of the agent availability, it depends on how fast it should be detected.

Notification message from the agent to known client in Russ's idea would be sent within less than 1 minute.
This is because a crashed process is restarted within 1 minutes on most of platforms.

The heartbeat message like BGP keep alive would be useful to detect the failure in more than 1 minutes.
If any message is exchanged between the agent and the client, no heartbeat is sent.
If no message is exchanged for more than 5 minutes, a heartbeat message is sent.
This is for cases where the agent restart time is long or never.

I don't think I2RS architecture have to cover all cases. 
I don't think I2RS client should detect the failure of the agent in sub-second.
By using HA technology like a hot-standby of the agent process, the possibility of the agent crash would be mitigated.
But I think it is out of scope of I2RS architecture. 

Thanks,

Toru
> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Joe Clarke (jclarke)
> Sent: Sunday, September 15, 2013 1:50 AM
> To: Alia Atlas
> Cc: [email-masked]
> Subject: Re: [i2rs] Architecture Discussion 2: Persistence (ephemeral vs.
> permanent)
> 
> > preempted/torn down or it can just notify all Clients (that it knows
> > of) that the Agent is going down.  These are important notifications
> > to capture in the I2RS Agent Info-Model.  I don't think that either
> > needs to be explicitly required as the only mode.  I am certainly
> > willing to be convinced otherwise.
> 
> I think I'm on the same page with you here.  If the Agent is being gracefully
> restarted, it flushes state, accepts no further connections, and notifies
> all known Clients that state is being torn down.  That works for me.
> 
> >
> As I said in that other thread, I'm okay with the persistent list of Clients.
> I didn't think that would be more desirable that the heartbeat.  However,
> I think something needs to be done to notify Clients in case of a crash.
> Else they might think everything is healthy when it's not.  It might be
> minutes, hours, days, never(?) before they reconnect to learn otherwise.
> 
> Joe
> 
> >
> > I really dislike the idea of adding significant complexity to handle
> > an edge failure case.
> >
> > Alia
> >
> >
> >     >     In either case, I saw that the Client would have some means
> of
> >     knowing
> >     >     the Agent went away.
> >     >
> >     >

---

From: Joe Marcus Clarke <[email-masked]>
Date: Wed, 18 Sep 2013 17:56:45 -0400

> This is because a crashed process is restarted within 1 minutes on most of platforms.

Russ' idea could potentially recover as fast as the Agent can recover.
The moment the Agent comes back, it notifies all previously subscribed
Clients.

> I don't think I2RS architecture have to cover all cases. 

Agreed.  However, Agent clean restart and Agent crash are two common
scenarios I think are worth handling.

Joe