Subject: [OSPF] Cascaded Graceful Restarts with NSF for OSPF protocol Question

From: "Abhay D.S" <[email-masked]>
Date: Fri, 19 Oct 2007 11:56:23 +0800

hi all,

Suppose I am providing real-time transactions over IP in a
non NSR environment(I dont invest much on getting a backup RP i.e).

But I have NSF, owing to the ease of which OSPF can be used, I
use OSPF to manage connections to WAN.

And I have bought new I/0 modules which require a restart for
all my routers for proper installation.

Luckily, I have inservice upgrade on the boxes.

Assuming that I am a new operator without any certifications under
my belt, I dont have a beautiful NMS with colored buttons.

But I must implement the new I/O cards and its features without disturbing
the users who are using the network for routing transactions such as data
backup over network or voice forwarding and other distributed applications
running over TCP/IP using my network.

I want to graceful restart all my routers one by one, called as cascaded
re-starts, how can I make sure the transition is done seamlessly.

I want the "this thing" to be automatic, since each router is graceful
restarted
and then then the next one is re-started and then the next one,
automatically.

I can write a script to do that, but I am not sure the forwarding will be
safe,
because it is difficult to control the network transient conditions.

Any mechanism available ?.

Thanks and Regards,
Abhay

_______________________________________________
OSPF mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ospf

---

From: Acee Lindem <[email-masked]>
Date: Fri, 19 Oct 2007 09:13:14 -0400

Hi Abhay,
Most of this borders on the implementation of the networking  
equipment's in-service upgrade. For now, let's just assume that you  
have redundant data place connectivity for the new line cards and  
solely need do the graceful restarts as indicated below. Again, this  
borders more on implementation. However, now the OSPFv2 MIB as  
described in RFC 4750 does support a variable which could be polled  
to determine when restart on a given router has completed.

  ospfRestartStatus OBJECT-TYPE
        SYNTAX       INTEGER { notRestarting (1),
                               plannedRestart (2),
                               unplannedRestart (3)
                             }
        MAX-ACCESS   read-only
        STATUS       current
        DESCRIPTION
           "Current status of OSPF graceful restart."
        ::= { ospfGeneralGroup 21 }

Hope this helps,
Acee

---

From: Erblichs <[email-masked]>
Date: Fri, 19 Oct 2007 12:43:54 -0700

Abhay D.S., et al,

	Their are multiple issues that need to be addressed
	before you can even attempt to sucessfully perform
	a cascade restart:

	1) NSF is a patented process that is specific to
	   one well-known company.

	2) Their is no GUARANTEE that any Restart will
	   be sucessful, that the LSDB will be stable and/or
	   that helper(s) exist.

	3) It would be my initial expectation that
	   resumption of recieving "aliveness" from a
	   restarting router COULD be a passive method.

	Mitchell Erblich
	-----------------

---

From: "Abhay D.S" <[email-masked]>
Date: Sat, 20 Oct 2007 11:24:10 +0800

Hello Acee,

a) Restart complete variable poll is one method after restart 

I am trying to find a method before restart to be deterministic.

A simple analogy in TCP.

For example..in TCP, just after connection establishment, I want
start sending data, at the same time the remote link had some problem.

a) Scenario 1

0sec .....5 Sec...(ok).....10 Sec (ok).....15 Sec (ok).....[DONE]

Continue restart y/n ? : y

Attempting graceful restart.....

[OK]

b) Scenario 2

0sec .....5 Sec...(ok).....10 Sec (Wait a While)....[Retrying due to network conditions]

0sec .....5 Sec...(Wait a While).....10 Sec (Ok)...15 Sec (ok).....[DONE]

Continue restart y/n ? : y

Attempting graceful restart......

[OK]

c) Scenario 3

0sec .....5 Sec...(ok).....10 Sec (Wait a While)....[Retrying due to network conditions]

0sec .....5 Sec...(Wait a While).....10 Sec (Ok)...15 Sec (Wait a While).....[Retrying due to network conditions]

0sec .....5 Sec...(ok).....10 Sec (Wait a While)....[Retrying due to network conditions]

Continuing restart may affect forwarding...
Continue anyway ? y/n .....n

[OK]

The time line above considers a network stable.

After the restart is done I can follow method a) 

Thanks for your reply,
Regards,
Abhay

  ----- Original Message ----- 
  From: Acee Lindem 
  To: Abhay D.S 
  Cc: [email-masked] 
  Sent: Friday, October 19, 2007 9:13 PM
  Subject: Re: [OSPF] Cascaded Graceful Restarts with NSF for OSPF protocol Question

  Hi Abhay,
  Most of this borders on the implementation of the networking equipment's in-service upgrade. For now, let's just assume that you have redundant data place connectivity for the new line cards and solely need do the graceful restarts as indicated below. Again, this borders more on implementation. However, now the OSPFv2 MIB as described in RFC 4750 does support a variable which could be polled to determine when restart on a given router has completed. 

    
  Hope this helps,
  Acee

    hi all,

    Thanks and Regards,
    Abhay

---

From: "Abhay D.S" <[email-masked]>
Date: Sat, 20 Oct 2007 11:27:11 +0800

Mitchell,

I wanted an automated and deterministic number 2.

Any ideas or any attempts in the direction of 2 ?

With Regards,
Abhay

----- Original Message -----
From: "Erblichs" <[email-masked]>
To: "Abhay D.S" <[email-masked]>
Cc: <[email-masked]>
Sent: Saturday, October 20, 2007 3:43 AM
Subject: Re: [OSPF] Cascaded Graceful Restarts with NSF for OSPF
protocolQuestion

---

From: Erblichs <[email-masked]>
Date: Sat, 20 Oct 2007 01:02:15 -0700

Abhay D.S.,

	Sorry about the top post, but..

	TCP has a slow-start mechanism with ACK
	 clocking to verify completion of xmit'ed data,
	until a threshold is met or,

	Until 2 or 3 DUP ACKs are seen and then
	rexmited or a coarse grain timeout occurs,

	However, TCP does not guarantee that the
	recvr will actually recieve data xmited
	by the send, just that if the data arrives
	then, it will send acks..

	And again, their is no guarantee that the
	acks will be recieved, so the sender can
	send more data.

	Mitchell Erblich
	-----------------

> 0sec .....5 Sec...(Wait a While).....10 Sec (Ok)...15 Sec (Wait a
> While).....[Retrying due to network conditions]
> 
>      equipment's in-service upgrade. For now, let's just assume
>      that you have redundant data place connectivity for the new
>      line cards and solely need do the graceful restarts as
>      indicated below. Again, this borders more on implementation.
>      However, now the OSPFv2 MIB as described in RFC 4750 does
>      support a variable which could be polled to determine when
>      restart on a given router has completed.
> 
>     ---------------------------------------------------------------

---

From: Erblichs <[email-masked]>
Date: Sat, 20 Oct 2007 01:21:19 -0700

Abhay D.S.,

	Actually it is just a bit more 
	complicated.

	With OSPF, we should start with a case
	of two routers.

	How does the restarting router know
	that the restart is done and how does
	the other router know that the restart
	is done?

	The restarting router can determine
	whether NSF and restart is done. Whether
	restart required a a standard resync of 
	the LSDB is almost irrelevant.

	So, the question is why must Gracefull
	be successful? It basicly just "reduces"
	overhead while NOT distrupting forwarding.

	Thus, It is done when it re-starts sending HELLOs
	and re-extablishes full adjs from the perspective
	of the 2nd router.  The hellos are the aliveness 
	criteria from my earlier post.

`	So, you MIGHT want to track the nbr FSM
	if you have access to the OSPF source code.

	Mitchell Erblich
	----------------

---

From: "Abhay D.S" <[email-masked]>
Date: Sat, 20 Oct 2007 18:12:30 +0800

Hi Mitchell,
Agreed,
I rather not have a TCP style in OSPF graceful restart.

But I am talking about TCP Timeout, but an application
has already done much work to send data to end point, before
time out occurs.

I would rather have a  request reply mechanism , which is similar
to pinging end points, before actually beginning the signaling.

Which I have shown in the time line.

Stressing the point of only what I want before the irreversible restart.

Regards,
Abhay

----- Original Message -----
From: "Erblichs" <[email-masked]>
To: "Abhay D.S" <[email-masked]>
Cc: "Acee Lindem" <[email-masked]>; <[email-masked]>
Sent: Saturday, October 20, 2007 4:02 PM
Subject: Re: [OSPF] Cascaded Graceful Restarts with NSF for OSPF
protocolQuestion

---

From: Curtis Villamizar <[email-masked]>
Date: Mon, 22 Oct 2007 23:53:49 -0400

> 	until a threshold is met or,

Not a good description of TCP's window mechanism but off topic anyway.

> 	rexmited or a coarse grain timeout occurs,

3 Dups.  Its called fast retransmit.

The timer is based on an estimate of RTT unless there is no basis to
determine RTT.

> 	then, it will send acks..

Are you claiming TCP doesn't guarentee arrival of data?  It does.

> 	acks will be recieved, so the sender can

The sender retransmits in the complete absense of ACKs after 3 seconds
if the initial retransmit is lost.  Then doubles the timer up to 75
seconds (odd default) and keeps that up for 15 minutes (by default,
all this is changeable).

> 	Mitchell Erblich

So what's your point?  Acee's response regarding using the MIB is
still valid.  With SNMP you *in theory* have to retransmit but if your
NMS is not braindead and overrunning equipment with SNMP gets then
you'll get the response because it is on IP prec 6 the whole way
(unless your OPs staff is also not too swift).  If you do lose an SNMP
get, a retry or even a series of retries is not the end of the world.

Curtis

btw - There is also FTTCP (fault tolerant) from Avici (patented but I
beleive the patent is invalid) which keeps TCP going even if the
primary server goes down (hacked TCP does application level 2 phase
commit on primary and standby server before sending ACK).  Amber and
others used a secondary that snoops on the TCP data for another form
of fault tolerant TCP adequate for BGP and manangement sessions.  But
this is OSPF WG so we're off topic.

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 23 Oct 2007 00:00:02 -0400

> 	overhead while NOT distrupting forwarding.

More important it does not slosh all of the BGP routes (hundreds of
thousands of them) away from the restarting router and then back.
In an IGP only transient routing nasties due to inconsistent notion of
whether that router is up or not is something to be avoided (you know,
transient loops, transient blackholes, and all that).

> 	if you have access to the OSPF source code.

Looks to me to be more like a windmill than a dragon.

Curtis

---

From: "Abhay D.S" <[email-masked]>
Date: Wed, 24 Oct 2007 10:10:43 +0800

Curtis,Erblich,

Could we focus on the key question of
cascaded restarts, how to make sure the topology
is stable before and after each restart ?. Automatically.

How to make a deterministic planning, so we can know
where the holes are and what is the impact.

Has anyone thought on these lines ?

Thanks,
Abhay

----- Original Message -----
From: "Curtis Villamizar" <[email-masked]>
To: "Erblichs" <[email-masked]>
Cc: "Abhay D.S" <[email-masked]>; <[email-masked]>
Sent: Tuesday, October 23, 2007 12:00 PM
Subject: Re: [OSPF] Cascaded Graceful Restarts with NSF for
OSPFprotocolQuestion

---

From: "Nitin Kakkar" <[email-masked]>
Date: Tue, 23 Oct 2007 21:04:40 -0700

Abhay,
  I am wondering if you can use LLS or similar mechanisms instead of GR?
  I can also think of something like an ospf task/process (with a
different router id) in the same memory address space, acting as
virtual/temporary router. This process borrow configurations from
restarting process, pretend itself to be a new router on link, form
adjacency with peer but does not do route calculations (unless you want
forwarding refreshed).

This does not require nbr to support GR, LLS or any other draft, but
require your system to support temporary interfaces.

Hope I am making some sense.
Regards
Nitin

-----Original Message-----
From: Abhay D.S [mailto:[email-masked]] 
Sent: Tuesday, October 23, 2007 7:11 PM
To: [email-masked]; Erblichs
Cc: [email-masked]
Subject: Re: [OSPF] Cascaded Graceful Restarts with NSF
forOSPFprotocolQuestion

Curtis,Erblich,

Thanks,
Abhay

----- Original Message -----
From: "Curtis Villamizar" <[email-masked]>
To: "Erblichs" <[email-masked]>
Cc: "Abhay D.S" <[email-masked]>; <[email-masked]>
Sent: Tuesday, October 23, 2007 12:00 PM
Subject: Re: [OSPF] Cascaded Graceful Restarts with NSF for
OSPFprotocolQuestion

---

From: "Abhay D.S" <[email-masked]>
Date: Wed, 24 Oct 2007 14:24:11 +0800

Hi Nitin,

I am thinking on the lines of preversing forwarding
in a stable toplogy as much as possible
which might become unstable due to cascaded restarts.

In protocol lore LLS or GR signalling, which I feel should
have some pre conditions for them to minimize fwding loss.

I want have an automatic mechanism to make sure the preconditions
exist after every device graceful restart using any mechanism, since
they are all connected and have ripple effect.

If this mechanism is automated, I am in more control of the
whole forwading paradigm.

With Regards,
Abhay

----- Original Message -----
From: "Nitin Kakkar" <[email-masked]>
To: "Abhay D.S" <[email-masked]>; <[email-masked]>; "Erblichs"
<[email-masked]>
Cc: <[email-masked]>
Sent: Wednesday, October 24, 2007 12:04 PM
Subject: RE: [OSPF] Cascaded Graceful Restarts with NSF
forOSPFprotocolQuestion

Abhay,

-----Original Message-----
From: Abhay D.S [mailto:[email-masked]]
Sent: Tuesday, October 23, 2007 7:11 PM
To: [email-masked]; Erblichs
Cc: [email-masked]
Subject: Re: [OSPF] Cascaded Graceful Restarts with NSF
forOSPFprotocolQuestion

Curtis,Erblich,

Thanks,
Abhay

----- Original Message -----
From: "Curtis Villamizar" <[email-masked]>
To: "Erblichs" <[email-masked]>
Cc: "Abhay D.S" <[email-masked]>; <[email-masked]>
Sent: Tuesday, October 23, 2007 12:00 PM
Subject: Re: [OSPF] Cascaded Graceful Restarts with NSF for
OSPFprotocolQuestion

---

From: "Abhay D.S" <[email-masked]>
Date: Wed, 24 Oct 2007 17:10:33 +0800

Hi Nitin,
I want reliability before the restart.

So I can predict the result of it and
then I can delay it if I want to by making
corrections.

Just like...looks good..before I begin
the signaling.

The mechanism described by Mitchell,Acee
and yours do not provide reliability before
restart.

Folks, tell me something I dont know.

With Regards,
Abhay

----- Original Message -----
From: "Nitin Kakkar" <[email-masked]>
To: "Abhay D.S" <[email-masked]>; <[email-masked]>; "Erblichs"
<[email-masked]>
Cc: <[email-masked]>

Abhay,

-----Original Message-----
From: Abhay D.S [mailto:[email-masked]]
Sent: Tuesday, October 23, 2007 7:11 PM
To: [email-masked]; Erblichs
Cc: [email-masked]
Subject: Re: [OSPF] Cascaded Graceful Restarts with NSF
forOSPFprotocolQuestion

Curtis,Erblich,

Thanks,
Abhay

----- Original Message -----
From: "Curtis Villamizar" <[email-masked]>
To: "Erblichs" <[email-masked]>
Cc: "Abhay D.S" <[email-masked]>; <[email-masked]>
Sent: Tuesday, October 23, 2007 12:00 PM
Subject: Re: [OSPF] Cascaded Graceful Restarts with NSF for
OSPFprotocolQuestion