Subject: Problem with BFD with ECMP

From: Manav Bhatia <[email-masked]>
Date: Fri, 9 May 2014 13:32:20 +0530

Hi,

There is a problem with how BFD/S-BFD works.

Sec 3.2 draft-aldrin-bfd-seamless-use-case-01 explains how we can use
S-BFD to quickly validate the data plane before pushing the data out.

There is a potential issue here.

Assume I send an S-BFD ping and I get a pong back.I believe that the
path is up and i start sending traffic towards the Reflector, brimming
with confidence, since i believe the data plane is up.

Assume that there is ECMP happening in the network somewhere that
splits the traffic to paths p1 and p2.

Its possible that the S-BFD packet gets hashed to the path p1, while
the data traffic that i send gets hashed to path p2.

If path p2 is down (or is congested and drops packets), then i drop
all traffic -- or some, in case of congestion. This however is
unexpected since i always get 100% response for my S-BFD ping packets.

I understand that if path p2 is down (because of an issue in one of
the nodes/links in the path), the ECMP path will not contain p2, since
the routing protocols will naturally remove this from the ECMP list.
So, its only while the network is reconverging (which should only last
for a few secs) that we will see this.

Is my understanding correct?

If yes, then is the scenario where S-BFD and data traffic following
different paths something that we want to look at fixing (either here
or some other WG)? This problem has already been solved in the MPLS
context with the Entropy label. I am talking specifically about
non-MPLS networks.

Cheers, Manav

---

From: Mach Chen <[email-masked]>
Date: Fri, 9 May 2014 08:45:19 +0000

Hi Manav,

The issue you raised is valid, but this issue should not be specific to S-BFD only, it's a common issue. 

And IMHO, in the pure IP scenario, since the hash is normally based on the four/five-tuple, if BFD and the data have the same IP header, the hash result should be the same. Unless you use s-bfd to detect a segment of the path, means the destination of the S-BFD will be different from the user data. Then you described situation will happen.

Best regards,
Mach

> -----Original Message-----
> From: Rtg-bfd [mailto:[email-masked]] On Behalf Of Manav Bhatia
> Sent: Friday, May 09, 2014 4:02 PM
> To: [email-masked]
> Subject: Problem with BFD with ECMP
> 
> Hi,
> 
> Sec 3.2 draft-aldrin-bfd-seamless-use-case-01 explains how we can use S-BFD to
> quickly validate the data plane before pushing the data out.
> 
> Assume I send an S-BFD ping and I get a pong back.I believe that the path is up
> and i start sending traffic towards the Reflector, brimming with confidence, since i
> believe the data plane is up.
> 
> Assume that there is ECMP happening in the network somewhere that splits the
> traffic to paths p1 and p2.
> 
> Its possible that the S-BFD packet gets hashed to the path p1, while the data
> traffic that i send gets hashed to path p2.
> 
> If path p2 is down (or is congested and drops packets), then i drop all traffic -- or
> some, in case of congestion. This however is unexpected since i always get 100%
> response for my S-BFD ping packets.
> 
> I understand that if path p2 is down (because of an issue in one of the
> nodes/links in the path), the ECMP path will not contain p2, since the routing
> protocols will naturally remove this from the ECMP list.
> So, its only while the network is reconverging (which should only last for a few
> secs) that we will see this.
> 
> If yes, then is the scenario where S-BFD and data traffic following different paths
> something that we want to look at fixing (either here or some other WG)? This
> problem has already been solved in the MPLS context with the Entropy label. I
> am talking specifically about non-MPLS networks.
> 
> Cheers, Manav

---

From: Manav Bhatia <[email-masked]>
Date: Fri, 9 May 2014 14:28:18 +0530

Hi Mac,

> The issue you raised is valid, but this issue should not be specific to S-BFD only, it's a common issue.

Yes. Thats why i said BFD/S-BFD. I just took S-BFD as an example,
since thats what i was looking at.

>
> And IMHO, in the pure IP scenario, since the hash is normally
> based on the four/five-tuple, if BFD and the data have the same IP
> header, the hash result should be the same. Unless you use s-bfd to

No, thats the point. The hash with 5 tuple will be different since
S-BFD headers will have a different protocol number, source port
number and destination port number than the data traffic thats we're
sending. Since these fields are different the hash may or may not be
the same as that of the BFD/S-BFD packet.

My point is this:

How do i ensure that in an IP only network, my data traffic always
follows the same path as the OAM packet that i had sent to verify the
data plane?

Will it work if i encapsulate both the OAM and the data traffic inside
a GRE tunnel so that both use the same hashing fields? I may want to
do this for certain flows where i want a very deterministic behavior.

Cheers, Manav

>  detect a segment of the path, means the destination of the S-BFD will
>  be different from the user data. Then you described situation will happen.
>
> Best regards,
> Mach
>

---

From: Mach Chen <[email-masked]>
Date: Fri, 9 May 2014 10:00:42 +0000

Hi Manav,

> From: Manav Bhatia [mailto:[email-masked]]
> Sent: Friday, May 09, 2014 4:58 PM
> To: Mach Chen
> Cc: [email-masked]
> Subject: Re: Problem with BFD with ECMP
> 
> Hi Mac,
> 
> Yes. Thats why i said BFD/S-BFD. I just took S-BFD as an example, since thats what
> i was looking at.
> 
> >
> > And IMHO, in the pure IP scenario, since the hash is normally based on
> > the four/five-tuple, if BFD and the data have the same IP header, the
> > hash result should be the same. Unless you use s-bfd to
> 
> No, thats the point. The hash with 5 tuple will be different since S-BFD headers
> will have a different protocol number, source port number and destination port
> number than the data traffic thats we're sending. Since these fields are different
> the hash may or may not be the same as that of the BFD/S-BFD packet.

Yes, indeed.

> 
> My point is this:
> 
> How do i ensure that in an IP only network, my data traffic always follows the
> same path as the OAM packet that i had sent to verify the data plane?

It's always desired, but it's hard, and most of time, no perfect solution. 

> 
> Will it work if i encapsulate both the OAM and the data traffic inside a GRE tunnel
> so that both use the same hashing fields? I may want to do this for certain flows
> where i want a very deterministic behavior.

Yes, use a tunnel to encapsulate both the OAM and the data traffic is a candidate. But for a deployed network, such changes may not accepted, you cannot change the existing deployment model. 

Best regards,
Mach

> >> path is up and i start sending traffic towards the Reflector,
> >> brimming with confidence, since i believe the data plane is up.
> >>
> >> the nodes/links in the path), the ECMP path will not contain p2,
> >> since the routing protocols will naturally remove this from the ECMP list.
> >> context with the Entropy label. I am talking specifically about non-MPLS
> networks.
> >>
> >> Cheers, Manav
> >

---

From: Manav Bhatia <[email-masked]>
Date: Fri, 9 May 2014 15:42:07 +0530

Hi Mach,

> It's always desired, but it's hard, and most of time, no perfect solution.

Sure the question is: Do we want to fix this?

> Yes, use a tunnel to encapsulate both the OAM and the data traffic is a candidate.
> But for a deployed network, such changes may not accepted, you cannot change the
> existing deployment model.

We can look at the solution space later.

The first thing is -- Do we want to solve this issue? I would wager
that its something that we would want to solve. Just want to hear what
others think about this.

Cheers, Manav

---

From: Loa Andersson <[email-masked]>
Date: Fri, 09 May 2014 12:52:39 +0200

Mach and Manav,

I guess that the answer to this type of question is somewhere around
"Is it worth doing?". If we see an easy way to fix it and we are pretty
sure that it will be deployed, then let us go ahead and do it. If the
fix is shady and no one really want it, then just leave it.

No in those terms where are we?

/Loa

-- 

Loa Andersson                        email: [email-masked]
Senior MPLS Expert                          [email-masked]
Huawei Technologies (consultant)     phone: +46 739 81 21 64

---

From: "Nobo Akiya (nobo)" <[email-masked]>
Date: Fri, 9 May 2014 12:36:44 +0000

Hi Manav, Mach, Loa,

On some data planes (ex: MPLS with Entropy Label), it's less difficult to use OAM packets to [likely] exercise specific ECMP paths as long as there are ways to discover the entropies.

On some others (ex: IP network), it's more difficult and I fully agree with what Loa said.

-Nobo

> From: Rtg-bfd [mailto:[email-masked]] On Behalf Of Loa
> Andersson
> Sent: Friday, May 09, 2014 6:53 AM
> To: Manav Bhatia; Mach Chen
> Cc: [email-masked]
> I guess that the answer to this type of question is somewhere around "Is it
> worth doing?". If we see an easy way to fix it and we are pretty sure that it
> will be deployed, then let us go ahead and do it. If the fix is shady and no
> one really want it, then just leave it.
> 
> >>> Will it work if i encapsulate both the OAM and the data traffic
> >>> inside a GRE tunnel so that both use the same hashing fields? I may
> >>> want to do this for certain flows where i want a very deterministic
> behavior.
> >>
> >> But for a deployed network, such changes may not accepted, you cannot
> >> change the existing deployment model.
> >
> >>>>   detect a segment of the path, means the destination of the S-BFD
> >>>> will be different from the user data. Then you described situation will
> happen.
> >>>>
> >>>> Best regards,
> >>>> Mach
> >>>>
> >>>>> Sec 3.2 draft-aldrin-bfd-seamless-use-case-01 explains how we can
> >>>>> use S-BFD to quickly validate the data plane before pushing the data
> out.
> >>>>>
> >>>>> Assume I send an S-BFD ping and I get a pong back.I believe that
> >>>>> the path is up and i start sending traffic towards the Reflector,
> >>>>> Its possible that the S-BFD packet gets hashed to the path p1,
> >>>>> while the data traffic that i send gets hashed to path p2.
> >>>>>
> >>>>> If path p2 is down (or is congested and drops packets), then i
> >>>>> drop all traffic -- or some, in case of congestion. This however
> >>>>> is unexpected since i always get 100% response for my S-BFD ping
> packets.
> >>>>>
> >>>>> I understand that if path p2 is down (because of an issue in one
> >>>>> of the nodes/links in the path), the ECMP path will not contain
> >>>>> p2, since the routing protocols will naturally remove this from the
> ECMP list.
> >>>>> If yes, then is the scenario where S-BFD and data traffic
> >>>>> following different paths something that we want to look at fixing
> >>>>> (either here or some other WG)? This problem has already been
> >>>>> solved in the MPLS context with the Entropy label. I am talking
> >>>>> specifically about non-MPLS
> >>> networks.
> >>>>>
> >>>>> Cheers, Manav
> >>>>
> >
> 
> --
> 
>

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 9 May 2014 09:26:51 -0400

> No in those terms where are we?

We are in the same state of Doom that we have been in for years. :-)

ECMP is a well known but not well specified bit of Internet architecture.
In some cases, it's a vendor differentiated feature and may differ depending
on platform, line card(s) and software revision.  

Addressing the this in a general fashion would require some mechanism that
permitted exposing an algebra that documents traffic hashing on a hop by hop
basis, where hops may also include internals of a given device.

While I suspect such an effort is, I believe,  mathematically tractable, I
don't believe such a proposal would be well received.

In my opinion, I think we must proceed with an understanding that this
situation is status quo.  This means mechanisms that intend to probe
reachability have to contain a reasonable level of internal mistrust. "I can
see the far end, this path *might* work."

-- Jeff

---

From: Gregory Mirsky <[email-masked]>
Date: Fri, 9 May 2014 14:31:04 +0000

Hi Manav, et. al,
I believe that we cannot say that ECMP scenario you've described, i.e. ECMP convergence, been solved in MPLS by application of ELI/EL. As in case of IP network, convergence of MPLS network constrained by time for fault detection and re-hashing available paths in a FIB/LFIB. During that period MPLS packets will be dropped on p2. After that return path p2 that carried flow with EL L1 will become p2' and packets with L1 will get to the destination. Would you agree?

	Regards,
		Greg

-----Original Message-----
From: Rtg-bfd [mailto:[email-masked]] On Behalf Of Manav Bhatia
Sent: Friday, May 09, 2014 1:02 AM
To: [email-masked]
Subject: Problem with BFD with ECMP

Hi,

Sec 3.2 draft-aldrin-bfd-seamless-use-case-01 explains how we can use S-BFD to quickly validate the data plane before pushing the data out.

Assume I send an S-BFD ping and I get a pong back.I believe that the path is up and i start sending traffic towards the Reflector, brimming with confidence, since i believe the data plane is up.

Assume that there is ECMP happening in the network somewhere that splits the traffic to paths p1 and p2.

Its possible that the S-BFD packet gets hashed to the path p1, while the data traffic that i send gets hashed to path p2.

If path p2 is down (or is congested and drops packets), then i drop all traffic -- or some, in case of congestion. This however is unexpected since i always get 100% response for my S-BFD ping packets.

I understand that if path p2 is down (because of an issue in one of the nodes/links in the path), the ECMP path will not contain p2, since the routing protocols will naturally remove this from the ECMP list.
So, its only while the network is reconverging (which should only last for a few secs) that we will see this.

If yes, then is the scenario where S-BFD and data traffic following different paths something that we want to look at fixing (either here or some other WG)? This problem has already been solved in the MPLS context with the Entropy label. I am talking specifically about non-MPLS networks.

Cheers, Manav

---

From: Gregory Mirsky <[email-masked]>
Date: Fri, 9 May 2014 14:46:57 +0000

Hi Nobo, et. al,
I'm not sure that ELI/EL was intended to pin a flow to the particular ECMP path. My understanding of EL is that it makes easier to distribute flows, e.g. by assigning EL labels from label range N labels wide, [M, M+N] when there are N available paths in an ECMP. I believe that there is no expectation to predict with any level of certainty which particular path a flow will follow but only expectation that flow with EL M+2 will take different path with flow tagged by EL M+3. Certainty is required though when ECMP is viewed as a link, e.g. micro-BFD.

	Regards,
		Greg

-----Original Message-----
From: Rtg-bfd [mailto:[email-masked]] On Behalf Of Nobo Akiya (nobo)
Sent: Friday, May 09, 2014 5:37 AM
To: Loa Andersson; Manav Bhatia; Mach Chen
Cc: [email-masked]
Subject: RE: Problem with BFD with ECMP

Hi Manav, Mach, Loa,

-Nobo

> "Is it worth doing?". If we see an easy way to fix it and we are 
> pretty sure that it will be deployed, then let us go ahead and do it. 
> If the fix is shady and no one really want it, then just leave it.
> 
> >>> inside a GRE tunnel so that both use the same hashing fields? I 
> >>> may want to do this for certain flows where i want a very 
> >>> deterministic
> behavior.
> >>
> >> Yes, use a tunnel to encapsulate both the OAM and the data traffic 
> >> is a
> candidate.
> >> But for a deployed network, such changes may not accepted, you 
> >> cannot change the existing deployment model.
> >
> > that its something that we would want to solve. Just want to hear 
> > what others think about this.
> >
> > Cheers, Manav
> >
> >>
> >> Best regards,
> >> Mach
> >>
> >>>
> >>> Cheers, Manav
> >>>
> >>>>   detect a segment of the path, means the destination of the 
> >>>> S-BFD will be different from the user data. Then you described 
> >>>> situation will
> happen.
> >>>>
> >>>> Best regards,
> >>>> Mach
> >>>>
> >>>>> Sec 3.2 draft-aldrin-bfd-seamless-use-case-01 explains how we 
> >>>>> can use S-BFD to quickly validate the data plane before pushing 
> >>>>> the data
> out.
> >>>>>
> >>>>> the path is up and i start sending traffic towards the 
> >>>>> Reflector, brimming with confidence, since i believe the data plane is up.
> >>>>>
> >>>>> following different paths something that we want to look at 
> >>>>> fixing (either here or some other WG)? This problem has already 
> >>>>> been solved in the MPLS context with the Entropy label. I am 
> >>>>> talking specifically about non-MPLS
> >>> networks.
> >>>>>
> >>>>> Cheers, Manav
> >>>>
> >
> 
> --
> 
>

---

From: Sam Aldrin <[email-masked]>
Date: Fri, 9 May 2014 07:54:51 -0700

Hi folks,

As Greg said, EL/ELI cannot predict which packet takes which path. It is only used to distribute flows of data packets, with an assumption that hashing will occur to utilize load balanced paths.
MPLS OAM cannot predict this either. That is why, as a vendor, one implements various tools like trace, path trace etc.
I see it no different to non-MPLS networks either.

As far as sec 3.2 of the use case document is concerned, S-BFD is not replacing other OAM tools. It still performs cc checks.
Solving ECMP issue is a never ending and don’t see one either, unless load balancing is standardized and all intermediate nodes confirmed to that, IMHO.
Till that time we have to play hash tag’s. :D

cheers
-sam

---

From: Manav Bhatia <[email-masked]>
Date: Fri, 9 May 2014 20:35:17 +0530

Ok, my point about MPLS being more predictable was this:

You slap on an EL to an OAM packet and it takes *some* path. Now, you
if you slap on the same EL to the data traffic then it would too take
the *same* path. So there is some level of determinism, which i stated
was missing in pure IP.

Cheers, Manav

---

From: Gregory Mirsky <[email-masked]>
Date: Fri, 9 May 2014 15:09:12 +0000

Hi Manav,
yes, there's problem with satisfying "in-band" requirement for IP OAM. I recall that Nobo and I had discussed it though briefly and agreed that if "in-band" OAM requirement is indeed strong then hashing should not use port numbers, e.g. be based on IP SA/DA.

	Regards,
		Greg

-----Original Message-----
From: Manav Bhatia [mailto:[email-masked]] 
Sent: Friday, May 09, 2014 8:05 AM
To: Sam Aldrin
Cc: Gregory Mirsky; Nobo Akiya (nobo); [email-masked]; Mach Chen; [email-masked]
Subject: Re: Problem with BFD with ECMP

You slap on an EL to an OAM packet and it takes *some* path. Now, you if you slap on the same EL to the data traffic then it would too take the *same* path. So there is some level of determinism, which i stated was missing in pure IP.

Cheers, Manav

On Fri, May 9, 2014 at 8:24 PM, Sam Aldrin <[email-masked]> wrote:
>>>>>> How do i ensure that in an IP only network, my data traffic 
>>>>>> always follows the same path as the OAM packet that i had sent to 
>>>>>> verify the
>>> data plane?
>>>>>
>>>>>>>> Assume I send an S-BFD ping and I get a pong back.I believe 
>>>>>>>> that the path is up and i start sending traffic towards the 
>>>>>>>> drop all traffic -- or some, in case of congestion. This 
>>>>>>>> however is unexpected since i always get 100% response for my 
>>>>>>>> S-BFD ping
>>> packets.
>>>>>>>>
>>>>>>>> I understand that if path p2 is down (because of an issue in 
>>>>>>>> one of the nodes/links in the path), the ECMP path will not 
>>>>>>>> contain p2, since the routing protocols will naturally remove 
>>>>>>>> this from the
>>> ECMP list.

---

From: Sam Aldrin <[email-masked]>
Date: Fri, 9 May 2014 08:10:26 -0700

Yes and no. 
What if the man in the middle only performs load balance with 3 label stack depth + ip HDR + udp port number (EL is beyond 3 lbl stack depth). The behavior of OAM packet differs from data packet.

There is some flexibility in MPLS / MPLS OAM (destn IP addr &&// EL) but not deterministic though.
We don’t in IP or BFD

-sam

---

From: John E Drake <[email-masked]>
Date: Fri, 9 May 2014 16:53:49 +0000

Greg,

RFC 4379 with Nobo's updates will tell you a range of labels that can be used to cause an MPLS packet to take a particular ECMP path.  Putting any label from that range on a packet will force it to take that path. 

Yours Irrespectively,

John

> From: Rtg-bfd [mailto:[email-masked]] On Behalf Of Gregory Mirsky
> Sent: Friday, May 09, 2014 7:47 AM
> To: Nobo Akiya (nobo); Loa Andersson; Manav Bhatia; Mach Chen
> Cc: [email-masked]
> I'm not sure that ELI/EL was intended to pin a flow to the particular ECMP
> path. My understanding of EL is that it makes easier to distribute flows, e.g. by
> assigning EL labels from label range N labels wide, [M, M+N] when there are N
> available paths in an ECMP. I believe that there is no expectation to predict
> with any level of certainty which particular path a flow will follow but only
> expectation that flow with EL M+2 will take different path with flow tagged by
> EL M+3. Certainty is required though when ECMP is viewed as a link, e.g.
> micro-BFD.
> 
> 	Regards,
> 		Greg
> 
> On some data planes (ex: MPLS with Entropy Label), it's less difficult to use
> OAM packets to [likely] exercise specific ECMP paths as long as there are ways
> to discover the entropies.
>

---

From: "Nobo Akiya (nobo)" <[email-masked]>
Date: Fri, 9 May 2014 17:28:36 +0000

Hi John, Greg,

[John wrote]
> RFC 4379 with Nobo's updates will tell you a range of labels that can be used
> to cause an MPLS packet to take a particular ECMP path.  Putting any label
> from that range on a packet will force it to take that path.

Correct. One would still want to run LSP tree/path trace (RFC4379 + multipath info for EL) every now and then to refresh the (paths + ELs), but that's the intent. And we if could use S-BFD for in-band monitoring of every paths, that's a fairly good coverage (not perfect, but fairly good) for MPLS.

[Greg wrote]
> > I'm not sure that ELI/EL was intended to pin a flow to the particular
> > ECMP path. My understanding of EL is that it makes easier to
> > distribute flows, e.g. by assigning EL labels from label range N
> > labels wide, [M, M+N] when there are N available paths in an ECMP. I
> > believe that there is no expectation to predict with any level of
> > certainty which particular path a flow will follow but only
> > expectation that flow with EL M+2 will take different path with flow
> tagged by EL M+3. Certainty is required though when ECMP is viewed as a
> link, e.g.
> > micro-BFD.

If possible, I do prefer end-to-end tests as that is easier to provide confidence that things are working end-to-end. MPLS, that's do-able. IP ... very tricky as other mentioned. So what you are proposing looks very interesting from that angle.

-Nobo

> From: John E Drake [mailto:[email-masked]]
> Sent: Friday, May 09, 2014 12:54 PM
> To: Gregory Mirsky; Nobo Akiya (nobo); Loa Andersson; Manav Bhatia; Mach
> Chen
> Cc: [email-masked]
> RFC 4379 with Nobo's updates will tell you a range of labels that can be used
> to cause an MPLS packet to take a particular ECMP path.  Putting any label
> from that range on a packet will force it to take that path.
> 
> Yours Irrespectively,
> 
> John
> 
> > I'm not sure that ELI/EL was intended to pin a flow to the particular
> > ECMP path. My understanding of EL is that it makes easier to
> > distribute flows, e.g. by assigning EL labels from label range N
> > labels wide, [M, M+N] when there are N available paths in an ECMP. I
> > believe that there is no expectation to predict with any level of
> > certainty which particular path a flow will follow but only
> > expectation that flow with EL M+2 will take different path with flow
> tagged by EL M+3. Certainty is required though when ECMP is viewed as a
> link, e.g.
> > micro-BFD.
> >
> > 	Regards,
> > 		Greg
> >
> > From: Rtg-bfd [mailto:[email-masked]] On Behalf Of Nobo
> > Akiya
> > (nobo)
> > On some data planes (ex: MPLS with Entropy Label), it's less difficult
> > to use OAM packets to [likely] exercise specific ECMP paths as long as
> > there are ways to discover the entropies.
> >
> > > >>> always follows the same path as the OAM packet that i had sent
> > > >>> to verify the
> > > data plane?
> > > >>
> > > >> Yes, use a tunnel to encapsulate both the OAM and the data
> > > >> traffic is a
> > > candidate.
> > > > The first thing is -- Do we want to solve this issue? I would
> > > > wager that its something that we would want to solve. Just want to
> > > > hear what others think about this.
> > > >
> > > > Cheers, Manav
> > > >
> > > >>
> > > >> Best regards,
> > > >> Mach
> > > >>
> > > >>>
> > > >>> Cheers, Manav
> > > >>>
> > > >>>>> can use S-BFD to quickly validate the data plane before
> > > >>>>> pushing the data
> > > out.
> > > >>>>>
> > > >>>>> Reflector, brimming with confidence, since i believe the data
> > > >>>>> plane is
> > up.
> > > >>>>>
> > > >>>>> fixing (either here or some other WG)? This problem has
> > > >>>>> already been solved in the MPLS context with the Entropy
> > > >>>>> label. I am talking specifically about non-MPLS
> > > >>> networks.
> > > >>>>>
> > > >>>>> Cheers, Manav
> > > >>>>
> > > >
> > >
> > > --
> > >
> > >

---

From: "George Swallow (swallow)" <[email-masked]>
Date: Fri, 9 May 2014 17:56:49 +0000

Manav -

Wish it were true.  But take a look at this from 6790:

----------------
4.3.  Transit LSR

   Transit LSRs MAY operate with no change in forwarding behavior.  The
   following are suggestions for optimizations that improve load
   balancing, reduce the amount of packet data processed, and/or enhance
   backward compatibility.

   If a transit LSR recognizes the ELI, it MAY choose to load balance
   solely on the following label (the EL); otherwise, it SHOULD use as
   much of the whole label stack as feasible as keys for the load-
   balancing function.  In any case, reserved labels MUST NOT be used as
   keys for the load-balancing function.

   Some transit LSRs look beyond the label stack for better load-
   balancing information.  This is a simple, backward-compatible
   approach in networks where some ingress LSRs impose ELs and others
   don't.  However, this is of limited incremental value if an EL is
   indeed present and requires more packet processing from the LSR.  A
   transit LSR MAY choose to parse the label stack for the presence of
   the ELI and look beyond the label stack only if it does not find it,
   thus retaining the old behavior when needed, yet avoiding unnecessary
   work if not needed.

------------------

Even if all transits routers implement EL and use it in their hash,
implementers are free to take stuff from elsewhere in the label stack
and/or IP header.  So you are not guaranteed to get the same path for OAM
and data.  You can only be guaranteed with the above assumptions if

A) The label stack for the OAM and data are identical
B) The IP headers are identical
C) The UDP ports are not used in the hash by any of the transit routers
D) None of the transit routers use stuff (like URLs) further into the
packet in the hash

George

On 5/9/14 11:05 AM, "Manav Bhatia" <[email-masked]> wrote:

>> As Greg said, EL/ELI cannot predict which packet takes which path. It
>>is only used to distribute flows of data packets, with an assumption
>>that hashing will occur to utilize load balanced paths.
>> MPLS OAM cannot predict this either. That is why, as a vendor, one
>>implements various tools like trace, path trace etc.
>> As far as sec 3.2 of the use case document is concerned, S-BFD is not
>>replacing other OAM tools. It still performs cc checks.
>> Solving ECMP issue is a never ending and don¹t see one either, unless
>>load balancing is standardized and all intermediate nodes confirmed to
>>that, IMHO.
>> Till that time we have to play hash tag¹s. :D
>>
>> cheers
>> -sam
>>>ECMP path. My understanding of EL is that it makes easier to distribute
>>>flows, e.g. by assigning EL labels from label range N labels wide, [M,
>>>M+N] when there are N available paths in an ECMP. I believe that there
>>>is no expectation to predict with any level of certainty which
>>>particular path a flow will follow but only expectation that flow with
>>>EL M+2 will take different path with flow tagged by EL M+3. Certainty
>>>is required though when ECMP is viewed as a link, e.g. micro-BFD.
>>>
>>>       Regards,
>>>               Greg
>>>
>>>Akiya (nobo)
>>>>>>>>>plane is up.

---

From: "George Swallow (swallow)" <[email-masked]>
Date: Fri, 9 May 2014 18:02:03 +0000

John -

Small point.  Most hash functions that I've looked at take bits from
various places to include in the hash.  More low order bits are used that
high order.  

As a result what you get back from LSP Ping is much more like a "set of
labels" than a "label range".
Most likely you would see only small sets of successive labels, if you got
any at all.

George

On 5/9/14 12:53 PM, "John E Drake" <[email-masked]> wrote:

>Greg,
>
>RFC 4379 with Nobo's updates will tell you a range of labels that can be
>used to cause an MPLS packet to take a particular ECMP path.  Putting any
>label from that range on a packet will force it to take that path.
>
>Yours Irrespectively,
>
>John
>
>>ECMP
>>tagged by
>>to use
>> On some others (ex: IP network), it's more difficult and I fully agree
>>with what
>> Loa said.
>> 
>> -Nobo
>>

---

From: "Nobo Akiya (nobo)" <[email-masked]>
Date: Fri, 9 May 2014 18:13:21 +0000

I also would like to point out

draft-ietf-mpls-forwarding-09
Section 2.4.5.1

   2.  If an ELI label is found, then if the LSR supports entropy label,
       the EL label field in the next label entry (the EL) SHOULD be
       used and label entries below that label SHOULD NOT be used and
       the MPLS payload SHOULD NOT be used.  See below this list for
       reasons.

Hopefully vendors implementing EL will seriously consider above "SHOULD".

-Nobo

> From: Rtg-bfd [mailto:[email-masked]] On Behalf Of George
> Swallow (swallow)
> Sent: Friday, May 09, 2014 1:57 PM
> To: Manav Bhatia; Sam Aldrin
> Cc: [email-masked]
> D) None of the transit routers use stuff (like URLs) further into the packet in
> the hash
> 
> George
> 
> >>> I'm not sure that ELI/EL was intended to pin a flow to the
> >>>particular ECMP path. My understanding of EL is that it makes easier
> >>>to distribute flows, e.g. by assigning EL labels from label range N
> >>>labels wide, [M,
> >>>M+N] when there are N available paths in an ECMP. I believe that
> >>>M+there
> >>>particular path a flow will follow but only expectation that flow
> >>>with EL M+2 will take different path with flow tagged by EL M+3.
> >>>Certainty is required though when ECMP is viewed as a link, e.g. micro-
> BFD.
> >>>
> >>>       Regards,
> >>>               Greg
> >>>
> >>> On some data planes (ex: MPLS with Entropy Label), it's less
> >>>difficult to use OAM packets to [likely] exercise specific ECMP paths
> >>>as long as there are ways to discover the entropies.
> >>>
> >>>> I guess that the answer to this type of question is somewhere
> >>>> around "Is it worth doing?". If we see an easy way to fix it and we
> >>>> are pretty sure that it will be deployed, then let us go ahead and do it.
> >>>>>>>>> Assume that there is ECMP happening in the network
> somewhere
> >>>> that

---

From: "George Swallow (swallow)" <[email-masked]>
Date: Fri, 9 May 2014 18:16:37 +0000

> Hopefully vendors implementing EL will seriously consider above "SHOULD".

Agreed!

George

On 5/9/14 2:13 PM, "Nobo Akiya (nobo)" <[email-masked]> wrote:

>>packet in

---

From: John E Drake <[email-masked]>
Date: Fri, 9 May 2014 19:02:26 +0000

Snipped, comment inline

Yours Irrespectively,

John

> From: Nobo Akiya (nobo) [mailto:[email-masked]]
> Sent: Friday, May 09, 2014 10:29 AM
> To: John E Drake; Gregory Mirsky; Loa Andersson; Manav Bhatia; Mach Chen
> Cc: [email-masked]
> > RFC 4379 with Nobo's updates will tell you a range of labels that can
> > be used to cause an MPLS packet to take a particular ECMP path.
> > Putting any label from that range on a packet will force it to take that path.
> 
> Correct. One would still want to run LSP tree/path trace (RFC4379 + multipath
> info for EL) every now and then to refresh the (paths + ELs), but that's the
> intent. And we if could use S-BFD for in-band monitoring of every paths, that's
> a fairly good coverage (not perfect, but fairly good) for MPLS.
> 

[JD]  It was always my intention to do exactly this but I never got around to updating RFC 4379

---

From: Manav Bhatia <[email-masked]>
Date: Sat, 10 May 2014 05:38:15 +0530

Hi Greg,

This is very difficult to mandate since in most cases we would like to
spread the traffic flows across multiple ECMP paths. Its only in
certain very specific cases where we would want the data to strictly
follow the IP OAM (S-BFD/BFD) packet. This i was wondering could be
possibly be achieved by encapsulating the S-BFD and the data packet
inside an IP-in-IP or a GRE header. This way the 5 tuple hash
computation would yield the same result for the OAM and the data
packets.

Cheers, Manav

---

From: Gregory Mirsky <[email-masked]>
Date: Sat, 10 May 2014 02:04:34 +0000

Hi Manav,
I agree that for continuity check, which what (S-)BFD is, "in-band" doesn't seem to be really required. Perhaps that is because IP is connectionless and is not viewed as transport layer unlike, for example, MPLS-TP. But I believe that there are other IP OAM mechanisms that would have much stronger need to keep in-band with IP particular service or be able to pin its path. These, I think, are IP Performance Measurement OAM. While passive PM natively is in-band as measurements performed on data traffic itself, active PM, for example TWAMP, would have the same issue with ECMP segments as we're discussing. But question of active PM OAM is more in scope of IPPM and LMAP WGs. As for BFD, I think that unless there's requirement to use it for connectivity verification in addition to continuity checking passing over diverse ECMP member segments seems OK to me.

	Regards,
		Greg

-----Original Message-----
From: Manav Bhatia [mailto:[email-masked]] 
Sent: Friday, May 09, 2014 5:08 PM
To: Gregory Mirsky
Cc: Sam Aldrin; Nobo Akiya (nobo); [email-masked]; Mach Chen; [email-masked]
Subject: Re: Problem with BFD with ECMP

Hi Greg,

This is very difficult to mandate since in most cases we would like to spread the traffic flows across multiple ECMP paths. Its only in certain very specific cases where we would want the data to strictly follow the IP OAM (S-BFD/BFD) packet. This i was wondering could be possibly be achieved by encapsulating the S-BFD and the data packet inside an IP-in-IP or a GRE header. This way the 5 tuple hash computation would yield the same result for the OAM and the data packets.

Cheers, Manav