Subject: RE: OAM functionality (Was: RE: can egress know the ingressof a packet?)

From: [email-masked]
Date: Thu, 8 Jun 2000 22:37:46 +0100

See comments below, Regards Neil

	[Harrison,N,Neil,INC1 X]  <Snip> 

> > As a network operator I have the following requirements:
> > - I need to detect failure of an LSP. (I don't want to wait for the
> > 	customer to report a failure)
> 
> This is a good requirement statement.
	[Harrison,N,Neil,INC1 X]  Good.  Then perhaps we can agree that all
defects need defining and tools providing to allow an operator to detect
them in advance of a customer reporting a fault.  So for LSPs there are at
least the following:
	-	simple connectivity failures (sourced from server or MPLS
layer......and we need to distinguish......note that ISP may not own
physical layer, cf earlier prot-sw issue on mail) 
	-	swapped LSPs
	-	mismerging of >=2 LSps
	-	self-mismerging
	As far as I can tell TTL is our only standardised OAM tool in the
MPLS layer.  This does not detect/diagnose any of above, but provides damage
limitation in case of self-mismerging.  A simple keepalive OAM pkt sent
periodically from the source of each LSP (and constaining the absolute LSP
source address) to its sink will however detect/diagnose all of these.
	I am open to better/other solutions.

> > 	There are more failure states of LSPs than standard destination
> > 	based routing because relative labels are used for switching. For
> > 	example we can have LSP traffic delivered to the wrong destination
> > 	by a simple bug on a single LSR (it is far less likely to happen
> > 	on an IP network using per hop routing because if
> > 	a router sends it the wrong way the next router will probably
> > 	send it back (get a routing loop) or send it the right way). How
> > 	do I (as an operator) detect this failure?
> 
> If it were a TE tunnel an IBGP adjacency going down be a real big
> hint.  This isn't supposed to happen (the tunnel going down without
> the control plane knowing about it).
	[Harrison,N,Neil,INC1 X]  If your customer's traffic was leaking
into another LSP how would this help detect and then diagnose it?  We are
considering more than simple breaks here.

> > 	Sure I could use a remote ping function on the customer premises 
> > 	equipment and when it fails raise an alarm. But I have to create
> > 	a full mesh of addresses to ping to cover all the customers
> > 	address space and sites (in the case of IP VPNs). This seems a bit
> > 	clumsy to me. And for IP VPNs what happens if the customers are
> using
> > 	the same 10.0.0.0 address space, chances are if my ping gets 
> > 	misdelivered they'll still be a box in somebody else's network that
> > 	will respond to the address I'm pinging (although we'll need 2
> faults
> > 	for that icmp echo-reply to get back).
> 
> OAM packets would not use the same hardware resources that the VPN
> traffic would unless they travel transparently in which case they
> would behave a lot like ping packets.
	[Harrison,N,Neil,INC1 X]  I thought Peter had already established
ping is not adequate.

> > 	When MPLS is used for TE what do I ping from where? If the TE trunk
> > 	fails won't I still get a response from ping because normal per-hop
> > 	based routing will take over? So we need a TE tunnel liveness
> > 	indicator.
> 
> Apparently the type of failure you are looking for is a mismatch
> between the control plane and the actual forwarding (an implementation
> bug or fault).
	[Harrison,N,Neil,INC1 X]  Not necessarily.  Why could there not be a
fault in the fowarding processing (in the user-plane) of which the
control-plane is in blissful ignorance? 
>   If the control plane knows of a failure, your NMS
> should know about it.
> 
> If the tunnel goes down and the control plane recovers by reverting to
> per-hop forwarding, then it ought to tell the NMS that it has done so.
	[Harrison,N,Neil,INC1 X]  If I had a pound (or even dollar) for each
time someone told me that the NMS should solve my problem I would a very
rich person.  I am not rich......and I am still waiting for the all-seeing
TMN solution 12 years after hearing it was 'coming soon'.  NMS solutions
should be a last resort IMHO.  Defects need to be handled in a distributed
manner at the network level......esp when one considers international
scenarios.

> > - I need tools to isolate where the fault is. Which LSR is causing the
> problem?
> > 	It would be useful if when I get a failure in the network the alarms
> > 	raised are a useful indicator of where the fault is. Then I can
> > 	start using tools like ping and traceroute to narrow down where
> > 	the fault is.
> 
> The type of fault you seem to be describing is where a label
> spontaneously evaporates or some similar occurance where the control
> plane is not aware that forwarding isn't working.
> 
> In some subset of such cases, counters that should normally be zero
> would be incremented.  For example, the fault may increment counters
> of labels for which there was no valid table entry or for which some
> internal table was invalid.  If no counters are incremented traffic
> levels should either fall to zero or differ radically at ingress and
> egress.
	[Harrison,N,Neil,INC1 X]  I would be grateful if you could please
explain how this works for all the misbranching cases I noted earlier.  If
all vendors can convince me they have identical solutions, and that my
operations people can diagnose the problem, and that customers will be
protected from leaking (out or in) traffic then OK.  Note - I really should
not rely on any customer generated traffic for diagnosis.....since this may
have large quiescent periods for one reason (and you did agree above that
customer's should not be used as the defect diagnosis tool). 

> For TE tunnels, a zero or near zero traffic rate would give an
> indication that such a fault had occurred,
	[Harrison,N,Neil,INC1 X]  This seems to indicate only a simple
connectivity break is being considered....it does not cater for the other
connectivity defects.
>  assuming that in an
> Internet backbone zero traffic never happens.  This is not a safe
> assumption for VPNs or enterprise since zero traffic can be possible.
	[Harrison,N,Neil,INC1 X]  Agreed...re above point that customers
traffic can be quiescent and the detection tool should not therefore rely on
customer traffic.

> For those cases were zero traffic is possible, add some traffic for
> the purpose of measurement (run a ping) and since you are pinging, you
> might as well verify that the pings go through.
	[Harrison,N,Neil,INC1 X]  Pings have problems as Peter pointed out
in prior mail.  Again this only considers simple breaks (though it might
detect some of the other misbranching cases....though at the wrong
point/layer).  Moreover, it does not address consequent actions.......which
are important, esp when LSPs are nested in a client/server layer
relationship.

> If a hardware fault has caused a table entry to rot, then using an OAM
> packet that takes a different path (handled via router alert label)
> would not detect the fault.
> 
> A big help in preventing this inability to detect faults is to put
> parity or ECC on internal data paths and where possible internal data
> sets.  This exposes the most common cause of internal data rot
> (unreliable internal data paths due to marginal or intermittent
> components that could otherwise result in the wrong value being set).
> 
> > Peter.
> 
> I don't see the point to specifying OAM packets if they do exactly
> what a ping would do and I don't yet see why the OAM packet would do
> anything different from ordinary traffic (therefore making pings a
> good candidate since they are ordinary traffic).
	[Harrison,N,Neil,INC1 X]  They don't.  We needs tools specific for
the MPLS layer itself.  And we need consequent actions defined, eg
forwarding of forward/backward defect indications to LSP trail terminations
points to take the appropriate consequent actions.
>   The issue then
> becomes specifying how pings and counters on the LSRs should be
> handled in order to notify the NMS.  Some might argue that this is out
> of scope for the MPLS WG.
	[Harrison,N,Neil,INC1 X]  I won't buy any OAM solutions that are
predicated on NMSs and their international co-operation.  Defects that
affect customer traffic have to be addressed/handled at the network level.

> One of the big problems with PVCs configured from the NMS (in ATM) had
> been the NMS missing one of the hops due to an undetected failure when
> trying to configure the switches.  In the absense of a control plane
> (use of PVCs) this can only be detected by some type of traffic not
> going through.  The same would be true of MPLS if it were used in this
> way.  Introducing OAM to MPLS in order to run without a control plane
> (configuring LSPs from the NMS) may be misguided logic.
	[Harrison,N,Neil,INC1 X]  Who is avocating this?

> Thats all from me on this.  If you still see a need for OAM packets
> then keep in mind that it is very late in the evolution of the MPLS
> encapsulation and changing the encapsulation is not an option that you
> will be able to propose without strong opposition.
	[Harrison,N,Neil,INC1 X]  Already noted.  And totally agree that
whatever is proposed should keep a close eye on backwards compatibilty.  One
final point on last sentence above.  It depends on who is doing the
'opposing'.  It really should be the operators call if they want certain OAM
functions to be addressed in a standardised and non-proprietary
manner....there is no point in asking operators for requirements and then
manufacturers ignoring/opposing them.  If a significant number of other
operators oppose such requirements though, then fair enough.

> Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 09 Jun 2000 18:53:57 -0400

> them in advance of a customer reporting a fault.  So for LSPs there are at

OK.  We at least agree that we need to start with a statement of the
probelm we are trying to solve.

> physical layer, cf earlier prot-sw issue on mail) 

Diagnosing physical layer problems would depend on each physical layer
unless we wanted simply to treat the physical layer as opaque and get
a measure of how well it is doing.  For that application something
alone the lines of LQM is well suited.  OAM is not since you can't put
enough probe packets into the network to start with.

> 	-	self-mismerging

Problems of this type fall into two categories:

  1.  The control software has bugs and has either signaled wrong
      labels or assigned incorrect table entries.  SNMP query of an
      MPLS MIB which reveals ingress FEC mappings, the insegments, and
      outsegment would allow tunnels to be verified and allow this
      type of problem to be identified.  We already have such a thing.

  2.  The control software has it right, but the lower level software
      or hardware doesn't agree with the control software.  In this
      case the traffic verifying the packet flow must take a path
      throught the hardware that is as close to identical to the
      actual traffic as possible, or make a very accurate measurement
      of the actual traffic.

In the latter case, you can send probe packets thorugh the tunnel but
they really should have an identical top label (or label stack under
test).  LQM does somwhat better in that if there are a few million PPS
of traffic and only a small amount of LQM packets, there is a huge
sampling.

Even with OAM or a yet to be specified MPLS LQM there is still a
problem of verifying the mapping of FEC into LSPs at the ingress.  If
there are 70,000 IP routes at an ingress LSR and only 700 tunnels, OAM
or LQM only covers 1% of the entries needed to make traffic flow in
the right direction.  If anything is done in this area, I'd favor
specifying an LQM for MPLS as I suggested in a prior email.

The only way that the 70,000 other entries can be verified would be to
send probe packets (IP OAM, traceroute, whatever) to each destination.
If 100 PPS of probe traffic is injected, then each route is checked
only once every 10 minutes but this is better than with MPLS OAM where
it is never checked.

Curtis

---

From: Peter Willis <[email-masked]>
Date: Mon, 12 Jun 2000 17:17:34 +0100

A simple illustration of how iBGP in MPLS IP VPNs can become non-congruent
with the LSP; I use a route reflector.

If I have a LSP between PEs but I'm using BGP route reflectors (for scaling
because I have hundreds of PEs) then a LSR in the middle can fail but the
BGP session will stay up. If the LSR fails totally then I'll get my
traditional SNMP traps from neighboring LSRs (hopefully) BUT if only
the label switching bit failed and the CPU in the LSR stayed up I could find
that the LDP and L2 keepalives are still working, may be even IP process
switching, but the label "fast path" is down  How do I now
detect failure of the LSP?

Lets try counting packets in and out of the LSP, will that work?
No it won't! It may be that the customer only uses that LSP once a month,
 so with no traffic its difficult to tell the LSP is down. So you're going 
to get a fault report from the customer when they try to use it on the last
midnight of the month.

But I think we've agreed we need a continuity verfication function? 

It may be too futuristic to suggest that when network providers start
offering MPLS as a network service the network provider will never see
the IP addresses of the customers so anything using ping or traceroute
to a customer address is inappropriate.

Peter.

---

From: Curtis Villamizar <[email-masked]>
Date: Mon, 12 Jun 2000 23:37:34 -0400

In message <[email-masked]>, Peter Willis
> to get a fault report from the customer when they try to use it on the last

If exactly zero packets are sent, then a small number of probe packets
can be added to make sure the traffic is non-zero.

If so, then perhaps a "data-sink" label is needed just to allow probe
packets to be silently discarded after the last label being measured
is popped.

Curtis