Subject: RE: OAM functionality (Was: RE: can egress know the ingress of a packet?)

From: [email-masked]
Date: Mon, 5 Jun 2000 22:42:38 +0100

Hi Florian,

I have been involved with much of the recent ATM OAM work.......but came in
late in the day when, unfortunately, it was too late to change the stds
(though believe me we tried).  Its a very complex story and I don't want to
waste people's time by going through its history here (speak to me privately
if you want the details).  But in essence, there are too many optional OAM
combinations and not enough thought was given early enough to the basic
architectural OAM functional relationships (especially harmonised defect
state handling at near-end and far-end of trails)......which is why after
many years (maybe 8 or so) we do not even have any agreement of how to
measure availability on ATM trails (and which also means, as a by-product,
that the up-state-only QoS measurements can become worthless).

Based on this experience, and working closely with my IP network design
colleagues, we have pared down our functional requirements (and their
manifestation as OAM pkts) to what we believe are the essential minimal in
order to (i) allow operations people to maintain and fault-find and (ii)
protect customer traffic from wrong delivery under various connectivity
failure scenarios.

I really ought to get something drafted so that other people can more
constructively criticise it......there may indeed be better ways to provide
the functionality we would like to see.   I will try and do something over
the next few weeks or so and circulate to those who have expressed an
interest so far. 

Regards, Neil

> -----Original Message-----
> From:	Florian-Daniel Otel [SMTP:[email-masked]]
> Sent:	Monday, June 05, 2000 3:41 PM
> To:	Shahram Davari
> Cc:	[email-masked]'; [email-masked]; [email-masked];
> [email-masked]; [email-masked]; [email-masked]
> Subject:	OAM functionality (Was: RE: can egress know the ingress of a
> packet?)
> 
> 
> Sharam,
> 
> > Hi Neil,
> 
> > I also think we need some sort of OAM function at MPLS layer. And as you
> > said, to be able to process OAM cells, they must be recognized by LSRs.
> But
> > it might be hard to change the definition of TTL field at this point in
> > time. So I have a different proposal. Why don't we assign/reserve some
> Label
> > values to be used only by OAM packets. This is very analogous to ATM, in
> > which, some VPI/VCIs are reserved for OAM cells (e.g. VPI=0 and VCI
> =0-15).
> > For example we could assign Label = 0 for the Keepalive messages that
> you
> > described.
> 
> > I am interested in writing such an I-D, and in fact my company is expert
> in
> > ATM OAM, so we could leverage that knowledge for the MPLS OAM.
> 
> As Neil already pointed out, there are several people intrested in O&M 
> functionality in MPLS, and for several reasons (you can also check the
> discussion between Neil and Vishal on TEwg mailing list on 17 Apr &
> ff. I also had some off-list questions about it). 
> 
> But the impression i got (and please, someone correct me if I'm wrong) 
> is that, as the Theimer OAM requirements draft clearly states:
> 
> "[..]we do not intend to simply adopt the OAM functions and procedures
> previously defined  for other technologies (in particular ATM), as
> this would probably lead to very complex and expensive
> implementations[..]"
> 
> 
> 
> Putting it bluntly, the way i read this is that "we simply don't want
> to turn MPLS into ATM-2 by adding all possible types of OAM functions".
> 
> 
> So, my take is (and, by all means, please correct me if i'm wrong)
> that in addition to the (rather vague IMHO) Theimer draft there you
> have all other proposals for recovery, protection, restoration and
> loop prevention but no other general O&M spec than the above
> draft. Why is that and how to solve (if ever need be) the OAM
> needs...well, that's it's beyond my knowledge (and the reason I'm 
> asking about it here).
> 
> 
> My $0.02 worth.
> 
> With kind regards,
> 
> Florian.

---

From: Ben Mack-Crane <[email-masked]>
Date: Tue, 06 Jun 2000 08:08:36 -0500

Hello Neil,

I think a draft covering MPLS OAM that can set requirements and define
the essential minimal OAM functions and mechanisms would be extremely 
useful.  As Florian mentions, there are several drafts addressing 
various specific items and one general one so far.  This work needs 
to be progressed and some common mechanism(s) for carrying out the 
variety of OAM functions should be defined.

I have been thinking about this from the viewpoint of the recovery
framework and protection switching drafts to which I have contributed.
In looking at MPLS based recovery, it becomes clear that a simple
and reliable path continuity test would be very useful.  Also a fast
communication path to convey a fault indication is useful if rapid 
protection switching is required.

A simple way to identify OAM packets, while allowing them to
follow the same data path as user packets (when required), is
essential to these functions.  If this can be done without disruption
to the embedded MPLS base, as Kireeti has indicated, this would be 
ideal.  I think there are some promising possibliities being discussed
in the related thread on "OAM labels".

I would be happy to help out with your proposed draft, if needed (though
I can't claim to have any special talents for editing ASCII state diags.).

Regards,
Ben

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 06 Jun 2000 11:12:17 -0400

> protection switching is required.

Pardon my ignorance but to what extent does ping and traceroute
implemented by the LSR meet these particular OAM needs?

It might help to start by identifying which features the OAM needs to
accomoddate that are currently missing (timestamp, record route,
other?).

For example, ping with record route doesn't work unless the MPLS
router alert and an ability to modify the underlying IP packet is
implemented (which hasn't even been suggested, and I am not implying
that it is a good idea).  Then we can consider whether a MPLS specific
ping with record route (or OAM packet) is needed to prevent DOS
attack, or whether one ping for all underlying layers is still OK (and
if one ping is better, then whether LSRs should provide configurable
ability to either rate limit or block setting of MPLS router alert
based on presence of RR IP options).

We might want to ask ISPs how valuable RR IP option on ping has been
and how valuable the equivalent ATM OAM cells have been.  We may find
that the former is used occasionally at most and the latter even less
or not at all for IP over ATM networks.

> in the related thread on "OAM labels".

draft-ietf-mpls-icmp-01.txt ?

Lets not reinvent the wheel if we don't have to.

Curtis

---

From: Peter Willis <[email-masked]>
Date: Tue, 06 Jun 2000 18:04:13 +0100

> or not at all for IP over ATM networks.

I have been network manager of BT's Internet services for 4 years. The RR
option on ping was used very rarely. ATM OAM is essential to the operation
of the network (we have an extensive ATM network). The ATM OAM is used
by the routers to detect failures in ATM VCs and raise SNMP traps. The
up/down status of the router interface is also dependent on ATM OAM so
we can run resilient ATM services to customers without running a routing
protocol to the customers' CPE/NTE.

As a network operator I have the following requirements:
- I need to detect failure of an LSP. (I don't want to wait for the
	customer to report a failure)
	There are more failure states of LSPs than standard destination
	based routing because relative labels are used for switching. For
	example we can have LSP traffic delivered to the wrong destination
	by a simple bug on a single LSR (it is far less likely to happen
	on an IP network using per hop routing because if
	a router sends it the wrong way the next router will probably
	send it back (get a routing loop) or send it the right way). How
	do I (as an operator) detect this failure?
	Sure I could use a remote ping function on the customer premises 
	equipment and when it fails raise an alarm. But I have to create
	a full mesh of addresses to ping to cover all the customers
	address space and sites (in the case of IP VPNs). This seems a bit
	clumsy to me. And for IP VPNs what happens if the customers are using
	the same 10.0.0.0 address space, chances are if my ping gets 
	misdelivered they'll still be a box in somebody else's network that
	will respond to the address I'm pinging (although we'll need 2 faults
	for that icmp echo-reply to get back).
	When MPLS is used for TE what do I ping from where? If the TE trunk
	fails won't I still get a response from ping because normal per-hop
	based routing will take over? So we need a TE tunnel liveness
	indicator. 
- I need tools to isolate where the fault is. Which LSR is causing the problem?
	It would be useful if when I get a failure in the network the alarms
	raised are a useful indicator of where the fault is. Then I can
	start using tools like ping and traceroute to narrow down where
	the fault is.

Peter.

---

From: Ben Mack-Crane <[email-masked]>
Date: Wed, 07 Jun 2000 08:55:55 -0500

Hello Curtis,

Perhaps I'm sporting more ignorance in this matter.  I had not
considered ping as a path continuity test.  The idea is to have
a test at the MPLS layer (OK ping is a bit above that...maybe 
not a problem) that would follow the data path (even through
LSRs) so it can detect disconnected and misconnected LSPs.

If the LSP egress expected ping's regularly from the ingress,
lack of pings would detect a broken connection.  If the ping 
contains data that identifies the ingress and path (path trace 
info), some (all?) misconnections can be detected.  (Is adding
specific data to a ping OK, or is this frowned upon?)

The remaining question is how to detect the location of the
dis-or-misconnection.  If intermediate LSRs can monitor this path
trace ping, that would help.  So here we're back to the question
of how to identify OAM packets in transit...  I don't know of
an existing mechanism for this.  (Router alert doesn't seem to 
fit the bill since, I assume, router alert packets follow
a different data path through an LSR from user packets and thus 
will not reflect all possible connection faults.)

In any case, I agree ping is worth considering.

I'm not trying to reinvent the wheel -- just trying to figure
out how to make the cart roll straight with all the varied wheels
we have to work with...

Regards,
Ben

---

From: Curtis Villamizar <[email-masked]>
Date: Wed, 07 Jun 2000 14:57:16 -0400

> specific data to a ping OK, or is this frowned upon?)

IP ping can use the IP record route IP option.  On a BSD system this
is done using "ping -R".  There is no corresponding MPLS RR option.

Without -R, ping is a continuity test only, though it does rely on
having a return path so it tests 2 way continuity.

traceroute uses a limited TTL and detects the TTL expired ICMP
response.  traceroute also requires a return path.

draft-ietf-mpls-icmp-01.txt extends the ICMP processing to allow
traceroute to list LSRs.  I thought I remembered a recommendation to
send the ICMP packet through the tunnel for an unreachable destination
in order to support VPNs, but I didn't see that in the draft.  In a
broken VPN LSP, the packets will never be able to complete the round
trip.  If a VPN LSP must travel inside an upper VPN, then the upper
LSP can be tested to find a fault (since a return address exists for
the ingress) but the lower VPN LSP cannot if the IP source address is
not reachable in the global routing table.  The lower LSP only has one
LSR looking at the label (the egress looks at the lower label after
the top label is popped, no other LSR looks at it) so this is not a
serious limitation.

> In any case, I agree ping is worth considering.

There is no real need for the intermediate to monitor.  What is needed
is a means to test connectivity and a means to determine the location
of a fault.  Ping and traceroute offer this.

> out how to make the cart roll straight with all the varied wheels

A strength and down side of ping and traceroute have always been their
bidirectional nature.  It is a strength in that any host can run ping
and traceroute and minimal coordination is required (IP routers or LSR
need only correctly generate ICMP).  Some of the downsides are:

  1.  Round trip times are reported rather than one way delay.
  2.  There is no way to record route for MPLS on ping (but traceroute
      still works).
  3.  It is not always obvious whether a problem resides in the
      forward or the reverse path when a ping or traceroute fails.

The approach of sending MPLS OAM packets also has its down sides.  For
example, one potential LSP failure is in the mapping of an FEC to the
first LSP insegment at the ingress.  If this fails and the OAM packets
are injected into the LSP through some other means, this failure will
not be detected.  Passing an IP packet (ping echo request or
traceroute udp packet) from an external interface with exercise this
and detect the failure.  The same applies to failures involving the
outsegment at the egress.  To make matters worse, failures can occur
that involve only one particular ingress interface.

Curtis

---

From: Ron Bonica <[email-masked]>
Date: Wed, 07 Jun 2000 21:52:52 -0500

> in order to support VPNs, but I didn't see that in the draft.

Hi Curtis,

You saw that recommendation in Section 2.3.2 of
draft-ietf-mpls-label-encaps-07. Section 5 of draft-ietf-mpls-icmp-01
explicitly defers to draft-ietf-mpls-label-encaps-07 in this matter.

                                   Ron

---

From: Peter Willis <[email-masked]>
Date: Thu, 08 Jun 2000 11:22:08 +0100

Problems with Ping and Traceroute Alone as OAM Tools For MPLS
-------------------------------------------------------------

1 - Failure of LSP does not imply ping will fail.
	An LSP X is established between PE A and PE D. To test continuity
	a ping process runs on A pinging D. If LSP X fails the ping may
	still be successful because D may still be reachable via per-hop
	routing. Example fault is that the LSR process on LSR C fails but its
	destination based per-hop routing process continues.

2 - Complexity of using ping as a proactive continuity check.
	It is essential to a network operator that failures in any LSP are
	raised as alarms to its NMS ASAP. To this end we have alarms from
	SDH, ATM, FR, PPP, HDLC and L2TP that causes routers to send traps 
	to NMS.  We need a similar function from MPLS. Even if pings fail
	when LSPs fail the complexity of running ping processes on all routers
	to test all LSPs is not acceptable and not current practice for other
	network technologies.
	Ping is ideally suited as a fault finding tool but lacks the 
	integration with the network protocols to be an effective pro-active
	monitoring tool within a complex network.

	- Especially in an IP VPN Environment.
	In an IP VPN environment only CEs within a VPN can ping other CEs
	within the same VPN. That is a ping can not run from the PE routers
	to test LSP connectivity between CEs. It is possible to run pings
	from the CEs and by making every CE also a member of the NMS VPN we
	can get the CEs to send network performance information back to the
	NMS. However how many CEs does 1 CE have to ping to measure LSP
	connectivity? The CEs have to be owned by the network operator in
	order for the network operator to provider this service. It is not
	acceptable to network operators to force customers to monitor their
	VPN service in order to raise alarms of LSP failures where customers
	own their own CPE.

3 - Non-uniqueness of IP Addresses in an IP VPN Service.
	In an IP VPN service different customers will be reusing the same
	address space. This leads to the possibility that if a LSP is
	misdirected that a ping test would still suceed. Usually this
	will require 2 faults, that is the outgoing and return LSPs to
	be misdirected however this is feasible as a single bug on a single
	LSR could lead to this. There is also the possibility that if
	CE a is using registered addresses to source the ping and pinging
	a private address, in the event of a misdirection of an LSP, the
	wrong device would respond and send the response back to the
	registered address via the Internet (I've actually seen this happen!).

4 - Failure of Traceroute where customers have their own MPLS network in an
	IP VPN environment.
	In the simple IP VPN service where we only offer standard IP 
	connectivity to customers traceroute may suffice (Although tracerouting
	to a customer's private address from within the provider core is
	not possible!) However if in the future there is a market for selling
	MPLS connectivity to customers in an IP VPN service traceroute
	will have many shortcommings. (I can imagine large corporate customers
	running their own MPLS networks and wanting to buy some service from
	network operators to connect their large MPLS networks together).
	In this situation traceroute is an insufficient tool for the
	customer to use and for the network operator to use. (I don't ever
	see how the network operator could traceroute into the customer's
	network but if the customer has tools to traceroute across their
	nets and the providers nets in an end-to-end manner it would be useful.)

5 - Seperation of IP-FEC failures from MPLS LSR label switching failures 
	requires seperate tools.
	It is true that MPLS OAM would not detect failures in with IP FEC and
	other possible IP routing failures. However this is reasonable,
	similarly ATM OAM does not spot IP failures. The question is can
	we depend on IP customers doing their own continuity tests and
	reporting failures to the network operators? If MPLS has OAM then
	at least the network operator can detect a significant number of
	failures in their customers' service. 
	In the event of failures in the IP FEC would it be helpful to have
	information that rules out failures in the LSRs label switching?
	I've run an ISP with a FR backbone and it was always useful to be
	able to tell if the fault was in the IP routing or the FR switches.
	MPLS OAM would give us the same ability to tell the difference
	between "simple" LSR label switching failure and more "complex" IP
	routing failures.

Peter.

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 08 Jun 2000 09:23:38 -0400

In message <[email-masked]>, Peter Willis
> 	customer to report a failure)

This is a good requirement statement.

> 	do I (as an operator) detect this failure?

If it were a TE tunnel an IBGP adjacency going down be a real big
hint.  This isn't supposed to happen (the tunnel going down without
the control plane knowing about it).

> 	for that icmp echo-reply to get back).

OAM packets would not use the same hardware resources that the VPN
traffic would unless they travel transparently in which case they
would behave a lot like ping packets.

> 	based routing will take over? So we need a TE tunnel liveness

Apparently the type of failure you are looking for is a mismatch
between the control plane and the actual forwarding (an implementation
bug or fault).  If the control plane knows of a failure, your NMS
should know about it.

If the tunnel goes down and the control plane recovers by reverting to
per-hop forwarding, then it ought to tell the NMS that it has done so.

> 	start using tools like ping and traceroute to narrow down where

The type of fault you seem to be describing is where a label
spontaneously evaporates or some similar occurance where the control
plane is not aware that forwarding isn't working.

In some subset of such cases, counters that should normally be zero
would be incremented.  For example, the fault may increment counters
of labels for which there was no valid table entry or for which some
internal table was invalid.  If no counters are incremented traffic
levels should either fall to zero or differ radically at ingress and
egress.

For TE tunnels, a zero or near zero traffic rate would give an
indication that such a fault had occurred, assuming that in an
Internet backbone zero traffic never happens.  This is not a safe
assumption for VPNs or enterprise since zero traffic can be possible.

For those cases were zero traffic is possible, add some traffic for
the purpose of measurement (run a ping) and since you are pinging, you
might as well verify that the pings go through.

If a hardware fault has caused a table entry to rot, then using an OAM
packet that takes a different path (handled via router alert label)
would not detect the fault.

A big help in preventing this inability to detect faults is to put
parity or ECC on internal data paths and where possible internal data
sets.  This exposes the most common cause of internal data rot
(unreliable internal data paths due to marginal or intermittent
components that could otherwise result in the wrong value being set).

> Peter.

I don't see the point to specifying OAM packets if they do exactly
what a ping would do and I don't yet see why the OAM packet would do
anything different from ordinary traffic (therefore making pings a
good candidate since they are ordinary traffic).  The issue then
becomes specifying how pings and counters on the LSRs should be
handled in order to notify the NMS.  Some might argue that this is out
of scope for the MPLS WG.

One of the big problems with PVCs configured from the NMS (in ATM) had
been the NMS missing one of the hops due to an undetected failure when
trying to configure the switches.  In the absense of a control plane
(use of PVCs) this can only be detected by some type of traffic not
going through.  The same would be true of MPLS if it were used in this
way.  Introducing OAM to MPLS in order to run without a control plane
(configuring LSPs from the NMS) may be misguided logic.

Thats all from me on this.  If you still see a need for OAM packets
then keep in mind that it is very late in the evolution of the MPLS
encapsulation and changing the encapsulation is not an option that you
will be able to propose without strong opposition.

Curtis

---

From: Peter Willis <[email-masked]>
Date: Thu, 08 Jun 2000 16:59:23 +0100

> The issue then

I can see that a system employing pingers on CPE, polling of counters, 
monitors of iBGP and other control plane aspects would deliver some of the OAM 
requirements. Such a system requires the deployment of proprietary components
to interface to the propreitary aspects of the LSRs. When I introduce
more than one vendor into the core network I have to rework these MPLS "OAM"
systems to interface to that new vendor.
On the other hand if MPLS OAM was incorporated as an optional standard then
I could deploy any vendors LSRs in my network that implemented the MPLS OAM
standard without having to rework my systems. Indeed if an MPLS OAM standard
was available how many network operators would not ask for conformance to it?
There will also be future issues when customers require MPLS services or 
customers
require IP VPNs implemented across more than one network operator. In such a
situation to get end-to-end MPLS OAM would require integration at the NMS 
level,
an MPLS OAM standard would be an easier way of facilitating this interworking.

By the way a common cause of internal "data rot" is software with bugs in it so
a system that can handle software and hardware failures is highly desirable.

Peter.

---

From: Curtis Villamizar <[email-masked]>
Date: Thu, 08 Jun 2000 17:13:13 -0400

In message <[email-masked]>, Peter Willis
 writes:
> requirements.

Welcome to the world of Internet operations.  Simple things that work
is what got us this far.  I'm still a bit skeptical about the OAM
approach.

> By the way a common cause of internal "data rot" is software with bugs
> in it so a system that can handle software and hardware failures is highly
> desirable.
> 
> Peter.

This is true.  The point is that the control plane and the forwarding
hardware have different information.  This is supposed to be a very
rare exception and it is supposed to get fixed.  If the software
works, then microcode or hardware may fail.  This is supposed to be
even more rare.  However, unlike software, hardware doesn't last
forever and is expected to eventually fail.  It is better if it
doesn't fail in subtle ways, hence my comment on parity or ECC in
internal hardware data paths and even data registers where possible.

Back to the requirement.  We have, in order of expected frequency, the
following types of failures:

  1.  Failures that are detected by the control plane.  For example,
      link failures or complete hardware component failures.

  2.  Software failures that are not detectable at the control plane.
      These will exist iff there remain software bugs which allow
      inconsistencies between control plane software and low level
      software or hardware.

  3.  Hardware failures that are not detectable at the control plane.
      These will exist iff there remain data paths or highly
      persistent data storage (like forwarding tables) that is not
      protected by ECC or parity.

OAM is not needed for the first set because the control plane will
recover and/or report the fault.

The third case is better covered by making sure all registers can be
read back and periodically checking hardware registers to
make sure that they match the software copies.

The second and third cases can be detected by sending probe traffic
through the network (OAM) but 100% coverage cannot be assured.  

The problem may occur at:

  1.  ingress,
  2.  midpoint, or
  3.  egress.

For example, if the ingress node loses the hardware entry that is
supposed to map a given IP prefix into an LSP, the same ingress may
successfully inject OAM packets because those do not enter as IP with
that prefix.  This is probably by far the most common error and it is
not detectable by OAM packets.  This is also not detectable by ping
unless the ping uses an IP destination corresponding to the faulty
entry.  This is detectable by a traceroute, because traceroute is IP
and can have the identical destination (and source, but that gets a
bit more tricky) and can be given a limited TTL that just verifies the
tunnel.

Note that an RFC2547 style VPNs there are multiple forwarding tables
in the provider edge routers.  These forwarding tables can only be
verified by injecting traffic through the same interface that would
exercise the same table entries.  Sounds to me a lot like a ping would
do the job nicely while an OAM packet that already had an MPLS
encapsulation would not.

If a midpoint loses a hardware entry for a given label, and the OAM
packet has a special label that kicks the OAM packet to software to be
counted, checked, or manipulated in some way, it still hasn't caught
the problem where the hardware table for that label is wrong since it
consults software.

The OAM label is diverted at the egress before it hits the hardware
that does things like POP the label and adjust the packet length,
adjust the underlying TTL (if IP), ECN, etc.  So the OAM packet
doesn't detect this fault either.

One thing that might help, but is not ATM OAM like is some form of LQM
for MPLS.  For example, the ingress could place a packet into the
stream with the label stack it wants to measure and a special LQM
label (in the reserved range) below it and place LSP counters in the
underlying packet.  A special outsegment would be needed at the
ingress to put the label stack and current counters in place.  At the
egress, the last forwarding label would be POPed and the reserved
label would be revealed.  The hardware would have to place its LSP
counters in the packet and forward to software for exception
processing.  The software can then subtract the counters of two
successive MPLS LQM (as is done in RFC1989 PPP LQM) to determine the
traffic rate and loss along the LSP.  Only the ingress and egress
would need to do this.  If the egress didn't know how, it would count
and silently discard the packet and you'd be no worse off.

MPLS LQM would definitely require hardware change so expect to hear
some screams.  On the positive side, only the ingress and egress need
to do it and it can be deployed incrementally.

Curtis