Subject: RE: OAM functionality (Was: RE: can egress know the ingressof a packet?)

From: [email-masked]
Date: Fri, 9 Jun 2000 00:07:02 +0100

See remarks below.  Regards, Neil

	[Harrison,N,Neil,INC1 X]  <Snip>
>  Back to the requirement.  We have, in order of expected frequency, the
> following types of failures:
> 
>   1.  Failures that are detected by the control plane.  For example,
>       link failures or complete hardware component failures.
> 
>   2.  Software failures that are not detectable at the control plane.
>       These will exist iff there remain software bugs which allow
>       inconsistencies between control plane software and low level
>       software or hardware.
> 
>   3.  Hardware failures that are not detectable at the control plane.
>       These will exist iff there remain data paths or highly
>       persistent data storage (like forwarding tables) that is not
>       protected by ECC or parity.
> 
> OAM is not needed for the first set because the control plane will
> recover and/or report the fault.
	[Harrison,N,Neil,INC1 X]  And why is this?  It's because the
control-plane had OAM functions designed into it at the outset.  But the
control-plane is not one single entity.....at its coarsest it comprises (i)
addressing (ii) a routing protocol (iii) a signalling protocol (at least for
CO-type routings).  What we really mean here is that the routing protocol
will (eventually) detect the failures........providing they are simple
breaks and not more complex misbranching cases.   We don't have to worry
about misbranching in IP........each pkt contains an absolute source (and
indeed sink) address....subject to global uniqueness issues.  This is not
true in MPLS.  Labels are ostensibly only unique per interface and are
therefore relative addresses.  BTW, to also rely on the 'control-plane'
requires it to have total congruence with the user plane.........can we
really be so prescriptive on this? 

	Last point on this paragraph wrt observation of this being highest
expected failure cause frequency.  I agree L1 server failures should be
expected to yield the highest frequency, and these will manifest themselves
as simple connectivity breaks in the server's client layers.  (This is also
a good reason not to react too quickly with prot-sw in clients....esp if one
does not own the L1).  But I suspect one will find that the L1 has OAM which
detects its own failures and reacts accordingly.....so our focus should not
be on detection/diagnosis of defects sourced in the L1, but those sourced in
MPLS.  

> The third case is better covered by making sure all registers can be
> read back and periodically checking hardware registers to
> make sure that they match the software copies.
	[Harrison,N,Neil,INC1 X]  Operators cannot be prescriptive about
manufacturer implementations .  We would hope they pay attention to internal
nodal defects as suggested.

> The second and third cases can be detected by sending probe traffic
> through the network (OAM) but 100% coverage cannot be assured.
	[Harrison,N,Neil,INC1 X]  Probes seem like a good way to get some
form of statistical sampling of one's network behaviour (at the IP layer),
but to scale to 100% to cover all cases is a non-starter.
Collection/processing costs alone would rule this out.  In any case it is
overkill.  CF The PM cell in ATM.  Probes are good for doing sampling in the
case of QoS metrics (eg errored pkts, abs/var on delay, etc) that one would
not normally be concerned about in a workhorse OAM sense.......here basic
connectivity assurances are the most important.  So whilst probes have their
use (sampling or ad hoc detailed trouble-shooting) this is not a scaleable
solution in my view.  

> The problem may occur at:
> 
>   1.  ingress,
>   2.  midpoint, or
>   3.  egress.
> 
> For example, if the ingress node loses the hardware entry that is
> supposed to map a given IP prefix into an LSP, the same ingress may
> successfully inject OAM packets because those do not enter as IP with
> that prefix.  This is probably by far the most common error and it is
> not detectable by OAM packets.
	[Harrison,N,Neil,INC1 X]  This is describing a defect which is in
the IP layer......or perhaps more accurately in the IP->MPLS layer
adaptation function.  So it is not valid to say MPLS OAM will not detect
this....it indeed should not.
>   This is also not detectable by ping
> unless the ping uses an IP destination corresponding to the faulty
> entry.  This is detectable by a traceroute, because traceroute is IP
> and can have the identical destination (and source, but that gets a
> bit more tricky) and can be given a limited TTL that just verifies the
> tunnel.
	[Harrison,N,Neil,INC1 X]  Then we have a tool at the right layer for
this particular defect.

> Note that an RFC2547 style VPNs there are multiple forwarding tables
> in the provider edge routers.  These forwarding tables can only be
> verified by injecting traffic through the same interface that would
> exercise the same table entries.  Sounds to me a lot like a ping would
> do the job nicely while an OAM packet that already had an MPLS
> encapsulation would not.
	[Harrison,N,Neil,INC1 X]  Agreed.  We are talking of IP->MPLS
adaptation again I think.  So the IP layer needs to detect this since it is
not a defect of the MPLS layer. [Note - If we had some decent functional
architecture diagrams I think this would help us a great deal......Andy Reid
and Mike Sexton, if you are out there reading this please come to our
rescue......I'll forward this mail to you anyway]
>  
> If a midpoint loses a hardware entry for a given label, and the OAM
> packet has a special label that kicks the OAM packet to software to be
> counted, checked, or manipulated in some way, it still hasn't caught
> the problem where the hardware table for that label is wrong since it
> consults software.
	[Harrison,N,Neil,INC1 X]  I would not expect mid points of an LSP to
alter a CV (Connectivity Verification) OAM pkt in any way.  They should be
treated as normal user-plane traffic.  CV pkts are generated at the LSP
source point and only examined again at the LSP sink point.  If there are
lower layer LSPs, then the higher layer LSP simply gets encapsulated and
passes transparently.  Or at the least that's the model I have in mind for
all this.

> The OAM label is diverted at the egress before it hits the hardware
> that does things like POP the label and adjust the packet length,
> adjust the underlying TTL (if IP), ECN, etc.  So the OAM packet
> doesn't detect this fault either.
	[Harrison,N,Neil,INC1 X]  Need proper functional architecture to
decide whether the issues being described are a function of the client layer
or the server layer, ie do they come after or before the LSP trail
termination point?  But they all sound client layer to me and so look like
they should be a function of the client layer to check 'correctness'.

> One thing that might help, but is not ATM OAM like is some form of LQM
> for MPLS.  For example, the ingress could place a packet into the
> stream with the label stack it wants to measure and a special LQM
> label (in the reserved range) below it and place LSP counters in the
> underlying packet.  A special outsegment would be needed at the
> ingress to put the label stack and current counters in place.  At the
> egress, the last forwarding label would be POPed and the reserved
> label would be revealed.  The hardware would have to place its LSP
> counters in the packet and forward to software for exception
> processing.  The software can then subtract the counters of two
> successive MPLS LQM (as is done in RFC1989 PPP LQM) to determine the
> traffic rate and loss along the LSP.  Only the ingress and egress
> would need to do this.  If the egress didn't know how, it would count
> and silently discard the packet and you'd be no worse off.
> 
> MPLS LQM would definitely require hardware change so expect to hear
> some screams.  On the positive side, only the ingress and egress need
> to do it and it can be deployed incrementally.
	[Harrison,N,Neil,INC1 X]  I'd like to know more about this.....I'll
check out the refs you gave (thanks).  In the stuff I have been looking at I
tried to keep the OAM as minimal and consistent as I could (having been
through the ATM 'experience').  My basic Connectvity Verification pkt is
nothing more that a sort of keepalive that is sent LSP_trail_source to
LSP_trail_sink on a periodic basis.  This detects all the connectivity
defects that might occur.  I also designed a variation on this which simply
counted the client layer pkts/octets sent between sucessive CV pkts....which
I called a CV+P pkt.  Its not something I would recommend for day-day use
(processing burden) but is simply there as a 1-step-up ad hoc diagnostic
tool for looking at pkt loss (and something that could be picked-off, by a
search for the absolute LSP source address, at intermediate points along an
LSP for subnetwork partition monitoring).  More than this (ie full QoS
metric suite) would require a sampling regime, like the probes discussed
earlier.

> Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 09 Jun 2000 19:31:05 -0400

In message <[email-masked]>, nei
[email-masked] writes:
> control-plane had OAM functions designed into it at the outset.  

No.  Because the control plane can be queried with SNMP and
periodically verfied.

> But the

All of which have (or should have) MIBs.

> nodal defects as suggested.

I doubt that you are suggesting that we ignore this type of problem.
If we did we could properly instrument the control plane (with MIBs)
and we're done.

> 	[Harrison,N,Neil,INC1 X]  Then we have a tool at the right layer for

So you agree that traceroute is the right tool for this.  OK we agree.

> rescue......I'll forward this mail to you anyway]

So OAM doesn't adress this either.  We agree here too.

> passes transparently.  Or at the least that's the model I have in mind for

That would mean that a special top label that flags a packet as OAM is
out of the question.  Good.  We agree again.

> they should be a function of the client layer to check 'correctness'.

So OAM does not address this either.  OK.  Further agreement.

> metric suite) would require a sampling regime, like the probes discussed

It might be possible to have two types of packets.  

One is an OAM connectivity verification that is injected at the
ingress and goes under the last label that you are testing
connectivity for.  When POP occurs, the OAM packet is simply counted
and possibly something in the packet is recorded (a sequence number
for example).  The ingress can inject these at some prescribed
(settable) rate.  SNMP can use two successive polls, subtract the
sequence numbers and subtract the counters and determine how many
probe packets (OAM packets) have been lost.  To have some chance at
detecting 0.1% loss you need to send at least 1,000 OAM packets.
This could be implemented entirely in software.

The other type would be LQM packets.  This would require hardware
assist to implement.  If there were 10^5 PPS on an LSP very small
levels of loss could be detected and extremely accurate loss
measurements could be made even if one LQM packet per second were
sent.

Of course both would be optional but there would be very little
incentive not to offer the first.

Curtis