Subject: BFD on LAG interfaces

From: "Bhatia, Manav (Manav)" <[email-masked]>
Date: Tue, 13 Dec 2011 23:51:27 +0530

Hi,

We had a very lively debate on the BFD mailing list a few months back on standardizing BFD over the LAG interfaces.

We have written a draft that proposes a new approach that fixes the issues that are seen with how folks do BFD over the LAGs today.

Draft available here:
http://tools.ietf.org/id/draft-mmm-bfd-on-lags-00.txt

Would be great to comments from the WG.

Cheers, 
Marc, Mach and Manav

--

Abstract

   This document proposes a mechanism to run BFD on Link Aggregation
   Group (LAG) interfaces.  It does so by running an independent BFD
   session on every LAG member link.

   A dedicated well-known multicast IP address for both IPv4 and IPv6 is
   introduced as the destination IP address of the BFD packets when
   running BFD on the member links of the LAG.

   There is currently no standard that describes how BFD should run on
   LAG interfaces.  As a result multiple non-interoperable BFD
   implementations for LAG interfaces exist.  This draft provides a
   short overview as a context for the new proposed mechanism.

---

From: Jeff Tantsura <[email-masked]>
Date: Tue, 13 Dec 2011 15:02:25 -0500

Hi Manav,

IMHO BFD over LAG session type (single vs multi) should be transparent to the upper layer protocols.
When L3 protocol registers with RIB for BFD status it is not interested in type of BFD session it is interested in its state.
So to keep logic simple from L3 protocol point of view it would be:

LAG constituent down && min-links has not been met -> L3 protocol doesn't get notified
LAG constituent down && min-links has been met -> L3 protocol gets BFD DOWN

As for IPv6 I'd suggest to use LL (FF02) as this is for single hop BFD

Please keep in mind - we were the first vendor to implement this (5 years ago) and have had many discussions with our customers on implementation, so there's quite some experience.
 
P.S. Personally I don't really like "pipes" everywhere, much too ALU :)

Regards,
Jeff

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Bhatia, Manav (Manav)
Sent: Tuesday, December 13, 2011 10:21 AM
To: [email-masked]
Subject: BFD on LAG interfaces

Hi,

Draft available here:

Cheers, 
Marc, Mach and Manav

--

Abstract

---

From: Tom Sanders <[email-masked]>
Date: Wed, 14 Dec 2011 01:48:15 +0530

> LAG constituent down && min-links has been met -> L3 protocol gets BFD DOWN

How do you detect that a LAG constituent link is down? LACP? Its this
(dependency on LAG which converges in order of seconds) that this
draft wishes to avoid. I think its about how fast one can detect each
individual LAG member link going down (by running BFD over each link).

Toms

---

From: Jeff Tantsura <[email-masked]>
Date: Tue, 13 Dec 2011 17:41:44 -0500

Tom,

That's exactly what we have been doing for years, BFD session per LAG constituent.

Regards,
Jeff

-----Original Message-----
From: Tom Sanders [mailto:[email-masked]] 
Sent: Tuesday, December 13, 2011 12:18 PM
To: Jeff Tantsura
Cc: [email-masked]
Subject: Re: BFD on LAG interfaces

> LAG constituent down && min-links has not been met -> L3 protocol 
> doesn't get notified LAG constituent down && min-links has been met -> 
> L3 protocol gets BFD DOWN

How do you detect that a LAG constituent link is down? LACP? Its this (dependency on LAG which converges in order of seconds) that this draft wishes to avoid. I think its about how fast one can detect each individual LAG member link going down (by running BFD over each link).

Toms

---

From: "Bhatia, Manav (Manav)" <[email-masked]>
Date: Wed, 14 Dec 2011 05:48:05 +0530

Hi Jeff,
 
> IMHO BFD over LAG session type (single vs multi) should be 
> transparent to the upper layer protocols.
> When L3 protocol registers with RIB for BFD status it is not 
> interested in type of BFD session it is interested in its state.
> LAG constituent down && min-links has not been met -> L3 
> protocol doesn't get notified LAG constituent down && 
> min-links has been met -> L3 protocol gets BFD DOWN

This is precisely what the draft suggests.

You run BFD for a LAG and several L3 applications register for it. Each member link of the LAG runs a BFD session. Sec 3.3 discusses about the concluded state of the BFD session on the LAG. What is relevant for BFD on LAG is that the concluded state is the overall state of the LAG.

If min-links are up then concluded state is UP. 

L3 applications are notified once the concluded state goes down.

> As for IPv6 I'd suggest to use LL (FF02) as this is for single hop BFD

Yes, we should do that.

We should have also mentioned that TTL for all IPv4 multicast packets MUST be 1.

Cheers, Manav

---

From: Yael Frank Dayan <[email-masked]>
Date: Wed, 14 Dec 2011 17:46:09 +0200

Hi,
Wouldn't the correct protocol be a version of BFD that runs over Ethernet?
How is IP relevant to a LAG member?

Thanks,
Yael
-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Bhatia, Manav (Manav)
Sent: Tuesday, December 13, 2011 8:21 PM
To: [email-masked]
Subject: BFD on LAG interfaces

Hi,

Draft available here:

Cheers, 
Marc, Mach and Manav

--

Abstract

---

From: Tom Sanders <[email-masked]>
Date: Wed, 14 Dec 2011 21:44:06 +0530

Yael,

There is no BFD that runs directly over ethernet. I think one of the
ideas behind the draft is to reuse as much of the existing BFD code as
possible. Just treating a constituent link of a LAG as a regular
physical port and running BFD on top of it looks really simple to me.
This can also easily be extended to work on ASICs that are designed to
do BFD in HW.

Why complicate this with another encapsulation and such?

Toms

-- 
Toms.

---

From: Nitin Bahadur <[email-masked]>
Date: Wed, 14 Dec 2011 08:24:23 -0800

I read through the draft and I personally do not like the idea of using a reserved mcast address.....since it masks
one of the IP addresses used for more strict checking on whether the packet is indeed intended for the recipient.

I would prefer using a different UDP port number and keeping everything the same. That way IP addr checks can be enforced.
And based on the different UDP port, the receiving entity will know that it needs to take constituent interface into account as well.

We have been meaning to write a draft for a long time on this....just been bogged down with other stuff.

Thanks
Nitin

On 12/13/11 4:18 PM, "Bhatia, Manav (Manav)" <[email-masked]> wrote:

Hi Jeff,

Yes, we should do that.

Cheers, Manav

---

From: "Reshad Rahman (rrahman)" <[email-masked]>
Date: Wed, 14 Dec 2011 10:28:12 -0600

I think the concern being raised is layering violation. Theoretically you could have IPv4 not working on a constituent link but other type of traffic would work fine.

Regards,
Reshad.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Tom Sanders
Sent: Wednesday, December 14, 2011 11:14 AM
To: Yael Frank Dayan
Cc: [email-masked]
Subject: Re: BFD on LAG interfaces

Yael,

Toms

-- 
Toms.

---

From: Gregory Mirsky <[email-masked]>
Date: Wed, 14 Dec 2011 11:40:17 -0500

Dear Manav,
Thank you for forwarding information, otherwise I'd have missed the announcement.
Please find my comments to the document:
- described issues with BFD over LAG seem to be vendor-specific, implementation dependent, i.e. not caused by RFC 5880 and RFC 5881. As result, tag line of Section 2.1 Existing implementations makes generalization to far as it seems it is not based on any poll or research but, I assume, only on experience of authors;
- any suggestion on relationship between LMM and BFD - implementation specific and doesn't seem to be appropriate in BFD standard;
- are you planning to update RFC 5880 and/or RFC 5881 by introducing new BFD state variable in Section 3.3?
- I think that BFD over LAG member links can use 127/8 for IPv4 address family and 0:0:0:0:0:FFFF:127.0.0.0/104 for IPv6.

Thank you for your kind consideration and I'm looking forward to your feedback.

	Regards,
		Greg

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Bhatia, Manav (Manav)
Sent: Tuesday, December 13, 2011 10:21 AM
To: [email-masked]
Subject: BFD on LAG interfaces

Hi,

Draft available here:

Cheers,
Marc, Mach and Manav

--

Abstract

---

From: Gregory Mirsky <[email-masked]>
Date: Wed, 14 Dec 2011 11:43:02 -0500

Hi Tom,
I'll note that we have BFD over MPLS with IP encapsulation. IMO, nothing prevents us from defining BFD over Ethernet with IP encapsulation with much of RFC 5885 being of good use.

	Regards,
		Greg 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Tom Sanders
Sent: Wednesday, December 14, 2011 8:14 AM
To: Yael Frank Dayan
Cc: [email-masked]
Subject: Re: BFD on LAG interfaces

Yael,

There is no BFD that runs directly over ethernet. I think one of the ideas behind the draft is to reuse as much of the existing BFD code as possible. Just treating a constituent link of a LAG as a regular physical port and running BFD on top of it looks really simple to me.
This can also easily be extended to work on ASICs that are designed to do BFD in HW.

Toms

--
Toms.

---

From: "Bhatia, Manav (Manav)" <[email-masked]>
Date: Wed, 14 Dec 2011 22:43:39 +0530

Hi Nitin,

> I read through the draft and I personally do not like the 
> idea of using a reserved mcast address.....since it masks one 
> of the IP addresses used for more strict checking on whether 
> the packet is indeed intended for the recipient.

We had internally discussed the idea of using a Unicast IP and we felt that using multicast was better since that avoided the need for L3 to come before a member link of a LAG could be declared operational. 

Using the IP address of the interface leads into a cyclic loop as you cant bring up the member links since L3 is not up, and you cant bring up L3 since the member links are still down.

On a point-to-point link isn't the discriminator value good enough to check if the packet is indeed for the recipient?

Cheers, Manav

> I would prefer using a different UDP port number and keeping 
> everything the same. That way IP addr checks can be enforced.

> And based on the different UDP port, the receiving entity 
> will know that it needs to take constituent interface into 
> account as well.
> 
> We have been meaning to write a draft for a long time on 
> this....just been bogged down with other stuff.
> 
> Thanks
> Nitin
> 
> 
> You run BFD for a LAG and several L3 applications register 
> for it. Each member link of the LAG runs a BFD session. Sec 
> 3.3 discusses about the concluded state of the BFD session on 
> the LAG. What is relevant for BFD on LAG is that the 
> concluded state is the overall state of the LAG.
>

---

From: "Bhatia, Manav (Manav)" <[email-masked]>
Date: Wed, 14 Dec 2011 22:50:18 +0530

Hi Greg,

> - described issues with BFD over LAG seem to be 
> vendor-specific, implementation dependent, i.e. not caused by 
> RFC 5880 and RFC 5881. As result, tag line of Section 2.1 
> Existing implementations makes generalization to far as it 
> seems it is not based on any poll or research but, I assume, 
> only on experience of authors;

And what makes you believe that this is an implementation specific issue?

Is there a standard on how BFD should work on LAGs? Do you round robin packets, do you send them over one port, what do you do?

> - any suggestion on relationship between LMM and BFD - 
> implementation specific and doesn't seem to be appropriate in 
> BFD standard;

LMM is a generic module name. You can call it whatever you want as long as you understand what it means. 

> - are you planning to update RFC 5880 and/or RFC 5881 by 
> introducing new BFD state variable in Section 3.3?

I am not sure if having a concluded state for LAG requires updating 5880/5881. An implementation could just overload its internal state to match the state of the LAG. They could work with the existing data structures. Does this mean that it does not require updating 5880/5881? I don't know ..

> - I think that BFD over LAG member links can use 127/8 for 
> IPv4 address family and 0:0:0:0:0:FFFF:127.0.0.0/104 for IPv6.

Just trying to understand - how is this better than using a mcast address? You don't need to reserve an address in this case. Any other benefit?

> Thank you for your kind consideration and I'm looking forward 

And you just got it! :)

Cheers, Manav

---

From: "Bhatia, Manav (Manav)" <[email-masked]>
Date: Wed, 14 Dec 2011 22:57:59 +0530

BTW, we also have the other mode defined in Sec 4 which uses BFD over a Big Pipe mechanism on top of the running BFD over each member link. In that mode, BFD runs over each member link to manage the state of the LAG. On top of that, we run another BFD session between the two IP end points just like its done today. So, now the L3 applications really depend upon the second BFD session and it takes care of the strict checking point that you had brought up and the layering violation issue that somebody else had brought up.

In the draft this is an optional mode and implementations SHOULD support it.

Cheers, Manav

> From: Nitin Bahadur [mailto:[email-masked]] 
> Sent: Wednesday, December 14, 2011 9:54 PM
> To: Bhatia, Manav (Manav); Jeff Tantsura; [email-masked]

---

From: Gregory Mirsky <[email-masked]>
Date: Wed, 14 Dec 2011 12:54:16 -0500

Hi Manav,
Thank you for your most expedient response. Please find my notes in-lined tagged by GIM>>

	Regards,
		Greg 

-----Original Message-----
From: Bhatia, Manav (Manav) [mailto:[email-masked]] 
Sent: Wednesday, December 14, 2011 9:20 AM
To: Gregory Mirsky; [email-masked]
Subject: RE: BFD on LAG interfaces

Hi Greg,

> - described issues with BFD over LAG seem to be vendor-specific, 
> implementation dependent, i.e. not caused by RFC 5880 and RFC 5881. As 
> result, tag line of Section 2.1 Existing implementations makes 
> generalization to far as it seems it is not based on any poll or 
> research but, I assume, only on experience of authors;

GIM>> BFD (RFC 5880 + RFC 5881) over LAG, IMO, is not different from BFD over any other 1 hop link and it monitors connectivity between two IP addresses regardless of through or on which LC BFD packets been originated/processed. Such details are implementation specific and if because of different method of originating/processing BFD packets such BFD over LAG is not interoperating, then someone got it wrong.

> - any suggestion on relationship between LMM and BFD - implementation 
> specific and doesn't seem to be appropriate in BFD standard;

GIM>> I compare with RFC 5885 which does not define MPLS and BFD inerworking because it is implementation specific. IMO, similar applies to BFD over LAG (per constituent link or not).

GIM>> Changing, i.e. overloading, context of a variable defined in RFC, IMO, is updating existing standard. If you want ineroperability, of course.

GIM>> No unneccessary allocations, i.e. re-use what you have - this is good enough for me.

And you just got it! :)

Cheers, Manav

---

From: Marc Binderberger <[email-masked]>
Date: Wed, 14 Dec 2011 19:24:33 +0100

Hello Tom and Yael,

so it's proven: the memory of a mailing list is smaller than 5 month. Actually not bad ;-)  We had Juli/August already an intensive discussion about this (see subject "Solicit comments on BFD for Interface draft")

First I don't think there is "the correct" protocol. True, RFC5880 would allow to run BFD directly on Ethernet. But:

- maybe we should have added this to the draft, but it's never clear upfront if adding something causes more confusion or more clarity: this draft is about BFD on Bundles, i.e. Ethernet and POS (!) bundles. This is the technical reason why the IP layer seems the right one.

- the customers I know had an IP focus. Liked to have BFD packets they already know when sniffing the wire, using ACLs etc.. Soft reason, true.

- the developers (again the ones I know) preferred to re-use as much as possible, thus IP was ideal. Another soft reason.

Regards, Marc

--
Marc Binderberger           <[email-masked]>

---

From: Marc Binderberger <[email-masked]>
Date: Wed, 14 Dec 2011 19:43:23 +0100

Hello Nitin,

> one of the IP addresses used for more strict checking on whether the packet is indeed intended for the recipient.

well, the bundle links are back-to-back between the 2 devices. So it's not a LAN with multiple (> 2) devices. The discriminator (when non-zero), the destination multicast IP, potentially the Multicast MAC would not be enough for a consistency check?
Can you give an example for what scenario you have in mind?

> I would prefer using a different UDP port number and keeping everything the same. That way IP addr checks can be enforced.

For the differentiation of per-member sessions that would do, of course. What the mcast IP does though is it comes on Ethernet automatically with an mcast MAC. The problem this solves is if you plan to have a tight coupling between the LMM and BFD, mainly that BFD must go "up" before a member link is added to the LAG, then ARP is likely not available. Programming the mcast MAC in the Ethernet hw does the trick to receive the BFD packets.

> And based on the different UDP port, the receiving entity will know that it needs to take constituent interface into account as well.

that's true.

> We have been meaning to write a draft for a long time on this....just been bogged down with other stuff.

And now we have all the different implementations in the field :-)

Thanks & Regards,
Marc

---

From: "Bhatia, Manav (Manav)" <[email-masked]>
Date: Thu, 15 Dec 2011 01:04:15 +0530

Hi Greg,
 
> Is there a standard on how BFD should work on LAGs? Do you 
> round robin packets, do you send them over one port, what do you do?
> 
> GIM>> BFD (RFC 5880 + RFC 5881) over LAG, IMO, is not 
> different from BFD over any other 1 hop link and it monitors 

There is a difference. In non lag, single hop links, there is no ambiguity over the physical port which needs to be used when transmitting the packets. Replace, this link with a "n" port lag, and you suddenly don't know which link to use when TXing the packets. 

> connectivity between two IP addresses regardless of through 
> or on which LC BFD packets been originated/processed. Such 
> details are implementation specific and if because of 
> different method of originating/processing BFD packets such 
> BFD over LAG is not interoperating, then someone got it wrong.

There are multiple way to run BFD over a LAG and neither is "incorrect".

1. Spray BFD pkts in a round robin fashion over all links of LAG
2. Use one of the active ports of the LAG
3. Send BFD packets over all member links using the same Discriminator values and hope that the other end will be able to make some sense out of it
4. Do what this draft suggests 
5. some more ..

If different vendors pick up different approaches to do BFD over LAG then how do you get them to interoperate?

In addition to this, we also want to avoid depending upon LACP to detect failed member links, since LACP has convergence in O(secs). I will not rehash the  long and protracted discussion that we had last time about why operators want to rely on BFD instead of .1ag and .3ah for fast link failure detection.

> implementation 
> LMM is a generic module name. You can call it whatever you 
> want as long as you understand what it means. 
> 
> GIM>> I compare with RFC 5885 which does not define MPLS and 
> BFD inerworking because it is implementation specific. IMO, 
> similar applies to BFD over LAG (per constituent link or not).

Youre saying that you don't like the term LMM. Sure, it can be replaced with something that's more generic. This is a nit that can be addressed. At this point of time I would like the WG members to focus on whether the mechanism described in this draft solves their problem (of running BFD over LAGs) or not. We can deal with the stylistic issues later. 

> I am not sure if having a concluded state for LAG requires 
> updating 5880/5881. An implementation could just overload its 
> internal state to match the state of the LAG. They could work 
> with the existing data structures. Does this mean that it 
> does not require updating 5880/5881? I don't know ..
> 
> GIM>> Changing, i.e. overloading, context of a variable 
> defined in RFC, IMO, is updating existing standard. If you 
> want ineroperability, of course.

I am still not convinced that introducing a variable in the internal data structures can affect interoperability. Again, as I had stated earlier - I would for now want us to focus on whether we have a plausible solution to a problem or not.

> Just trying to understand - how is this better than using a 
> mcast address? You don't need to reserve an address in this 
> case. Any other benefit?
> 

So, youre suggesting that we use an IPv4 address like 127.0.0.1 instead of a IANA assigned multicast address. Sure, what about the MAC address? What MAC should I use? What do I ARP for?

Cheers, Manav

---

From: Gregory Mirsky <[email-masked]>
Date: Wed, 14 Dec 2011 15:19:55 -0500

Hi Manav,
Another round of thanks for your interest and in-lining my notes with tag GIM>>. Hope we're converging.

	Regards,
		Greg

-----Original Message-----
From: Bhatia, Manav (Manav) [mailto:[email-masked]] 
Sent: Wednesday, December 14, 2011 11:34 AM
To: Gregory Mirsky; [email-masked]
Subject: RE: BFD on LAG interfaces

Hi Greg,
 
> Is there a standard on how BFD should work on LAGs? Do you round robin 
> packets, do you send them over one port, what do you do?
> 

GIM>> IMO, it is irrelevant over which port of a LAG BFD packets been transported/received. BFD packets in RFC 5881 are addressed to an IP address, not a physical port. Hence where an IP packet been originated and/or processed is irrelelvant.

GIM>> BFD over LAG must operate according to RFC 5881 and thus any particular "way" of originating and/or processing BFD control packet is, repeating myself, irrelevant to BFD state. Modes you've listed are implementation specific attempts to optimize or address some other objectives of particular vendor, HW shortcomings and, IMNSHO, not something to standardize.

GIM>> That is the substance of the proposal, IMO:
- how to run BFD over LAG's constituent links;
- how to run BFD over LAG's constituent links and BFD over LAG concurrently.
And no need to argue which one is better, I'm not technology biased.

GIM>> Manav, it is not about the term but effective standartization of internals of BFD and LMM interworking. What and how BFD signals to its clients, IMO, is not in the scope of the BFD WG.

GIM>> I'm afraid that you hardly can control topics being discussed once you submit the text for review.

GIM>> Request to allocate Ethernet Multicast Address to be used for p2p Ethernet links is reasonable and in line with idea suggested in draft-fbb-mpls-tp-ethernet-addressing-00.

Cheers, Manav

---

From: Dave Katz <[email-masked]>
Date: Wed, 14 Dec 2011 14:19:13 -0800

Reading between the lines, it appears that the fundamental goal is to replace LACP, at least as a liveness mechanism, as the detection time is too slow.  This should be called out explicitly.  This means that you need a BFD session per leg, and an aggregation mechanism for the resulting state.

On the first point, the problem is that the whole point of having LAGs is to fold multiple links into one;  as such, from the IP layer, the component links are invisible; the service access for the individual legs is at the Ethernet layer.  Thus, from an architectural standpoint, the only thing that makes sense would be to run BFD directly over Ethernet (which we went out of our way to make possible in 5880 but couldn't get any traction from the Ethernet folks to adopt it).  As soon as you try to operate at the IP layer, you run smack into implementation-specific problems, as there is fundamentally no IP binding to individual legs of a LAG.  So to run this over IP, you require implementations to expose the individual legs as having IP bindings, even if they are kludged (which is what is being suggested here).  This is, well, *ugly*.

Assuming that you manage to run BFD over individual legs somehow, achieving the second point (state aggregation) does not require modification to 5880.  What it *does* require, however, is for the BFD-over-bundle spec to call out how that aggregation occurs (an additional state machine).  What comes out of the top of that state machine is what gets coupled to the apps.  This is discussed (albeit briefly) in RFC 5883.  There's no new ground here other than to be specific about how the state machine works.  It may be necessary to run an additional BFD session, independent of the per-leg sessions, in order to succinctly communicate such concepts as AdminDown for the whole bundle, etc.  Such a session can run at a sedate rate since the per-leg sessions will take care of the more intense liveness testing.

As far as this particular draft goes, I think trying to make this work over IP creates a whole pile of issues that cannot be solved in a standard way, because it essentially requires that BFD tap into private APIs in order to work.

The real issue here is that the Ethernet folks have clearly dropped the ball.  The right answer (IMHO) is to fix LACP so that it can operate at reasonable speeds.  That *is* part of its job, no?  With a LACP that Did The Right Thing, BFD just operates over the bundle and doesn't duplicate a bunch of functionality.  Anybody go to IEEE meetings?

If this is too tall an order, the Not Quite As Right thing would be to define an Ethernet encapsulation for BFD, as it could then operate on well-defined APIs and not be an ugly layer violation.  This will still make the Ethernet folks squeal, but it doesn't require them to do anything other than to allocate an EtherType code point and possibly a well-known multicast address (though even this can be punted by just using broadcast if the link is truly point-to-point).

The disagreement about physical TX port binding is simply a manifestation of the fact that trying to do this at the IP layer is fundamentally undefined from a standards point of view, as it requires sending IP packets over links that have no IP binding, and as such is fundamentally hackery.

--Dave

---

From: Marc Binderberger <[email-masked]>
Date: Wed, 14 Dec 2011 23:31:10 +0100

Hello Gregory and Manav,

this is a fast "match". I hope I can follow - let me try ;-)

For section 2.x I think the definition of BFD over Big Pipe ("BBP") - or "BFD over LAG" as you name it - was not section 2.1 but 2.2:

----snip----snap----snip----snap----
   o  BFD must work no matter what member link, packets are sent and
      received on.
   o  the Rx/Tx link can change any time and/or regularly with every
      change pattern without causing BFD to fail

   This allows to use the LAG like any other interface and RFC 5880 and
   RFC 5881 can be used without modification.
----snip----snap----snip----snap----

and this should be in-line with your comments:

>> different from BFD over any other 1 hop link

> GIM>> IMO, it is irrelevant over which port of a LAG BFD packets been transported/received. BFD packets in RFC 5881 are addressed to an IP address, not a physical port. Hence where an IP packet bee…

I don't see a difference between Gregory's statements and the draft section 2.2 - what am I missing? :-)

Maybe the recommendation for round-robin is causing confusion? For me I was reading this more as "worked well for others, give this a try". You can ignore recommendations.

> And no need to argue which one is better, I'm not technology biased.

I agree. Actually the proposal also does a "definition on the fly" of what the BFD over LAG ("BBP") is, in section 2.2.  
My personal view is that when you run over member links and BBP concurrently then there is no better - they improve each other.

> GIM>> I'm afraid that you hardly can control topics being discussed once you submit the text for review.

"hardly" is true but is not identical with "no" control ;-)
Seriously: when we named it concluded state then there was no intention to affect RFC5880. Actually the per-member-link sessions are RFC5880 without any additional state. The concluded state comes with the LAG:

   What is relevant for BFD on LAG is that the
   concluded state is the overall state of the LAG.

As "BFD over LAG's constituent links" doesn't have a BFD session for the overall LAG we had to name this "overall state" somehow. We use "concluded state".

So this new state variable may exist in an implementation but then it belongs to this draft, not to RFC5880. It has relevance for the report back to the layer-3 clients, as described in section 4. But the packets on the wire follow RFC5880. 

> GIM>> Request to allocate Ethernet Multicast Address to be used for p2p Ethernet links is reasonable and in line with idea suggested in draft-fbb-mpls-tp-ethernet-addressing-00.

So why not the all-host multicast address? No extra allocation and comes with a multicast MAC.

Reason why I feel uncomfortable with 127/8: used by BFD for LSP (RFC5884) already. The Multicast IP was meant for a simple differentiation

About the LMM I have to think a bit - or maybe I wait for Manav's reply :-)

Thanks for your comments!

Regards, Marc

---

From: "Bhatia, Manav (Manav)" <[email-masked]>
Date: Thu, 15 Dec 2011 06:28:55 +0530

Hi Dave,
 
> Reading between the lines, it appears that the fundamental 
> goal is to replace LACP, at least as a liveness mechanism, as 
> the detection time is too slow.  This should be called out 
> explicitly.  This means that you need a BFD session per leg, 
> and an aggregation mechanism for the resulting state.

You've put it quite succinctly. 

Really, the goal is simple - How do we run BFD over a LAG interfaces while ensuring that link failures don't take more than a few milli secs to be detected. LACP was out since its too slow. Eth OAM was out since operators in IP/MPLS space wanted to stay away from it. That left us with just BFD to modify, which is what we did.

> 
> On the first point, the problem is that the whole point of 
> having LAGs is to fold multiple links into one;  as such, 
> from the IP layer, the component links are invisible; the 

Yes.

> service access for the individual legs is at the Ethernet 
> layer.  Thus, from an architectural standpoint, the only 
> thing that makes sense would be to run BFD directly over 
> Ethernet (which we went out of our way to make possible in 
> 5880 but couldn't get any traction from the Ethernet folks to 

Which is precisely why we wanted to do this at the BFD layer.

> adopt it).  As soon as you try to operate at the IP layer, 
> you run smack into implementation-specific problems, as there 
> is fundamentally no IP binding to individual legs of a LAG.  
> So to run this over IP, you require implementations to expose 
> the individual legs as having IP bindings, even if they are 
> kludged (which is what is being suggested here).  This is, 

Yes its slight convoluted as you let the IP layer decide the fate of the lower layer.

> 
> Assuming that you manage to run BFD over individual legs 
> somehow, achieving the second point (state aggregation) does 
> not require modification to 5880.  What it *does* require, 
> however, is for the BFD-over-bundle spec to call out how that 
> aggregation occurs (an additional state machine).  What comes 
> out of the top of that state machine is what gets coupled to 
> the apps.  This is discussed (albeit briefly) in RFC 5883.  
> There's no new ground here other than to be specific about 
> how the state machine works.  It may be necessary to run an 
> additional BFD session, independent of the per-leg sessions, 
> in order to succinctly communicate such concepts as AdminDown 
> for the whole bundle, etc.  Such a session can run at a 
> sedate rate since the per-leg sessions will take care of the 
> more intense liveness testing.

And this is the second mode that we describe in Sec 4. The additional BFD session that youre alluding to is the BFD over the Big Pipe in our document and it does exactly what you have described. We have BFD sessions per member links of the LAG that control the state of that link. We then have an additional BFD session, independent of the per member LAG sessions, which controls the state of the whole LAG.

> As far as this particular draft goes, I think trying to make 
> this work over IP creates a whole pile of issues that cannot 
> be solved in a standard way, because it essentially requires 
> that BFD tap into private APIs in order to work.

I am not sure I understand what private APIs are you referring to. I think the confusion stems from the two modes that we have defined in Sec 4. 

In the first mode we just run BFD over the member links. There is no additional BFD running over the whole bundle. In this mode, BFD is simply replacing LACP. It keeps track of all the links that are alive and informs the LAG manager (or some entity, beyond the scope of this document) about it. The IP interface over the LAG is torn down once the number of remaining links reaches the minimum threshold number. BFD is thus used to decide the fate of an IP interface. 

We had kept this mode as the default mode.

I am now having second thoughts on whether this is the right thing to do or not.

Then there is the second mode described in Section 4.

Establish BFD sessions over each member link. The aggregate result of this decides the fate of only the LAG (and not that of the IP interface defined above it). Establish another BFD session over the LAG (which is what most implementations do today) that runs sedate timers and ensures IP reachability with the other end. This mode gives us the best of both worlds.

Would it make more sense to keep this mode as default and completely do away with the first mode that we have defined?

> The real issue here is that the Ethernet folks have clearly 
> dropped the ball.  The right answer (IMHO) is to fix LACP so 
> that it can operate at reasonable speeds.  That *is* part of 
> its job, no?  With a LACP that Did The Right Thing, BFD just 
> operates over the bundle and doesn't duplicate a bunch of 
> functionality.  Anybody go to IEEE meetings?

That would be ideal. 

Cheers, Manav

---

From: Dave Katz <[email-masked]>
Date: Wed, 14 Dec 2011 19:01:23 -0800

> I am not sure I understand what private APIs are you referring to. I think the confusion stems from the two modes that we have defined in Sec 4. 

My point is that running IP over an individual leg is an unnatural act, as it is architecturally incompatible with how LAGs are supposed to work (IMHO).  As a result, implementors have to hack something into their implementations to sort-of-pretend-but-not-really that IP is enabled on the individual legs.

There is no good way to spec such a thing because there isn't supposed to be any IP binding to the individual leg.  What you're doing, in effect, is modifying the definition of a LAG in order to shoehorn BFD into the picture, which is rather out-of-scope for the BFD group (and for the IETF in general).  I don't see how it would be possible to standardize such a thing.

> I am now having second thoughts on whether this is the right thing to do or not.

It's a reasonable thing if all you're doing is replacing LACP with BFD as the per-leg liveness test;  it makes BFD invisible to applications, as the service interface is the top of the LAG.  This is subtly different than having a BFD-over-the-top session in addition to the individual legs, as doing so provides visible BFD semantics rather than LAG semantics, and the service interface reflects those semantics (so you can potentially do things like AdminDown and the like).
> Would it make more sense to keep this mode as default and completely do away with the first mode that we have defined?

I'm agnostic about the choices;  I can see some benefit in both.  But the real problem is how to do the per-link stuff in a reasonable way.

--Dave

---

From: "Bhatia, Manav (Manav)" <[email-masked]>
Date: Thu, 15 Dec 2011 10:10:50 +0530

Hi Dave,

I think philosophically we're having a similar discussion that we had many many moons ago over BGP being extended to carry non routing information. Few asserted that it made trifle little sense for BGP to carry all the additional information it was being envisaged to carry. However, the conclusion drawn after a long protracted discussion was that if BGP is extensible and can do something, then why reinvent the wheel and why not use the existing mechanism instead?

In a similar vein, we have an issue here (faster detection of failed links) that needs to be addressed. BFD today on LAG doesn't really work (explained in the draft). This needs to be fixed. We can either approach IEEE and wait endlessly for them to come up with something, or we can do something on our own.

Its not an engineering driven problem - it's a problem that people out there are seeing (there were operators who did speak up last time).

What we're saying is that while BFD was not originally meant to monitor the forwarding plane of the constituent links of a LAG, lets do it now, because there is a need. Does it require us to pretend a few things? Yes - maybe it does. And, mostly it doesn't. 

More inline ..

> There is no good way to spec such a thing because there isn't 
> supposed to be any IP binding to the individual leg.  What 
> you're doing, in effect, is modifying the definition of a LAG 
> in order to shoehorn BFD into the picture, which is rather  

I don't think we're doing anything to the LAG (http://en.wikipedia.org/wiki/Link_aggregation). We're not changing what the LAG is. What we're perhaps changing is the part where LACP fits in. We're saying that instead of using LACPDUs to detect failed links we'll use BFD. To avoid the deadlock between L3 and the lower layers, we will use multicast. This is one component of the solution and involves no layer violation.

On top of this you may run the regular RFC 5880 compliant BFD with *zero* changes.

The two together should solve all the problems. However, its NOT mandatory to run the on-the-top-BFD session.

> > In the first mode we just run BFD over the member links. 
> There is no additional BFD running over the whole bundle. In 
> this mode, BFD is simply replacing LACP. It keeps track of 
> all the links that are alive and informs the LAG manager (or 
> some entity, beyond the scope of this document) about it. The 
> IP interface over the LAG is torn down once the number of 
> remaining links reaches the minimum threshold number. BFD is 
> thus used to decide the fate of an IP interface. 
> > 
> It's a reasonable thing if all you're doing is replacing LACP 
> with BFD as the per-leg liveness test;  it makes BFD 
> invisible to applications, as the service interface is the 
> top of the LAG.  

This is the default mode that we're suggesting. Just run micro BFD sessions on each member link. Applications are oblivious to the presence of BFD. They only rely on the state of the LAG.

> This is subtly different than having a 
> BFD-over-the-top session in addition to the individual legs, 
> as doing so provides visible BFD semantics rather than LAG 

So, I think this is where we have a disconnect.

To me, the micro BFD sessions just have the visibility at the LAG level. They control the fate of the LAG.

You don't need to run the BFD-over-the-top-session, but if you want to do that they that session is completely independent of the micro BFD sessions that you are running underneath.

> > Would it make more sense to keep this mode as default and 
> completely do away with the first mode that we have defined?
> 
> I'm agnostic about the choices;  I can see some benefit in 
> both.  

Same here. Which is why we decided to include both. 

> But the real problem is how to do the per-link stuff 
> in a reasonable way.

I think what we're proposing is quite reasonable.

You cant use Unicast IPv4 (for ARP resolution etc)
You don't want to define a new IEEE encapsulation where you get a new ether type from IEEE.
You want the ASICs that do BFD in HW to support this scheme with zero changes.
You want to reuse as much code as possible for implementing the new mechanism.
You want the solution to be generic enough to work in IP/MPLS and non IP/MPLS environments.

If you look at what we're proposing then it meets all the above requirements.

1. Use a well known IANA allocated multicast IP address. 
2. Derive the dest MAC from the dest IP. 
3. Use your port's MAC as the source MAC.
4. Start sending BFD packets as described in rfc 5880. One can pretty much use the same state machine that's described there.
5. Use the Your Discriminator value to demultiplex the BFD packets to the right micro BFD session (BFD session on member link).
6. If either the remote system signals Down state or the Detection Time expires, the micro BFD session advances to Down state and the LAG manager is informed which promptly removes the member link from the LAG.

The rest would be taken care of by the LAG manager as its done in all the existing implementations today.

Cheers, Manav

---

From: Mach Chen <[email-masked]>
Date: Thu, 15 Dec 2011 04:59:50 +0000

Hi Dave,

Please see inline...

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Dave Katz
> Sent: Thursday, December 15, 2011 11:01 AM
> To: Bhatia, Manav (Manav)
> Cc: [email-masked]
> > I am not sure I understand what private APIs are you referring to. I think the
> confusion stems from the two modes that we have defined in Sec 4.
> 
> My point is that running IP over an individual leg is an unnatural act, as it is
> architecturally incompatible with how LAGs are supposed to work (IMHO).  As

Cited from RFC 5880:

Section 1.: Introduction

"The goal of Bidirectional Forwarding Detection (BFD) is to provide
   low-overhead, short-duration detection of failures in the path
   between adjacent forwarding engines, including the interfaces, data
   link(s), and, to the extent possible, the forwarding engines
   themselves.

   An additional goal is to provide a single mechanism that can be used
   for liveness detection over any media, at any protocol layer, with a
   wide range of Detection Times and overhead, to avoid a proliferation
   of different methods."

Section 2. Design
".BFD can provide failure detection on any kind of path between
   systems, including direct physical links, virtual circuits, tunnels,
   MPLS Label Switched Paths (LSPs), multihop routed paths, and
   unidirectional links (so long as there is some return path, of
   course)."

According to the above description (e.g., any media, any protocol layer, direct physical links...), IMHO, running BFD over an individual leg should be one of the goals of BFD and it is allowed and natural.

> a result, implementors have to hack something into their implementations to
> sort-of-pretend-but-not-really that IP is enabled on the individual legs.

I agree if the LAG is an L2 interface, but for an IP enabled LAG, the individual legs could be IP enabled as well. Suppose that if the solution is applied to Bundle interface.

> 
> There is no good way to spec such a thing because there isn't supposed to be
> any IP binding to the individual leg.  What you're doing, in effect, is modifying
> the definition of a LAG in order to shoehorn BFD into the picture, which is
> rather out-of-scope for the BFD group (and for the IETF in general).  I don't see
> how it would be possible to standardize such a thing.

I think the question is that do we really want to solve the issue, whether BFD is a proper tool? According the discussions on the list, the requirements from some SPs, several individual similar implementations in the field, IMHO, the answer is yes.

Actually, the draft proposes a new application of BFD, the client is the LAG/Bundle Interface.  

Best regards,
Mach

> >
> > In the first mode we just run BFD over the member links. There is no
> additional BFD running over the whole bundle. In this mode, BFD is simply
> replacing LACP. It keeps track of all the links that are alive and informs the LAG
> manager (or some entity, beyond the scope of this document) about it. The IP
> interface over the LAG is torn down once the number of remaining links reaches
> the minimum threshold number. BFD is thus used to decide the fate of an IP
> interface.
> >
> It's a reasonable thing if all you're doing is replacing LACP with BFD as the
> per-leg liveness test;  it makes BFD invisible to applications, as the service
> interface is the top of the LAG.  This is subtly different than having a
> BFD-over-the-top session in addition to the individual legs, as doing so provides
> visible BFD semantics rather than LAG semantics, and the service interface
> reflects those semantics (so you can potentially do things like AdminDown and
> the like).
> >
> > Establish BFD sessions over each member link. The aggregate result of this
> decides the fate of only the LAG (and not that of the IP interface defined above
> it). Establish another BFD session over the LAG (which is what most
> implementations do today) that runs sedate timers and ensures IP reachability
> with the other end. This mode gives us the best of both worlds.
> >
> > Would it make more sense to keep this mode as default and completely do
> away with the first mode that we have defined?
> 
> I'm agnostic about the choices;  I can see some benefit in both.  But the real
> problem is how to do the per-link stuff in a reasonable way.
> 
> --Dave

---

From: Greg Mirsky <[email-masked]>
Date: Wed, 14 Dec 2011 21:52:31 -0800

Dear Dave and All,
I think that Mach brought the right point and that problem we can work is
not just BFD over LAG's constituent link but, as Dave pointed earlier, BFD
over Ethernet, with or without IP addressing. And I'll draw parallel with
RFC 5885 and RFC 6428 that use BFD over MPLS, i.e. ACH VCCV encapsulation
without IP header.

Regards,
Greg

On Wed, Dec 14, 2011 at 7:01 PM, Dave Katz <[email-masked]> wrote:

> > I am not sure I understand what private APIs are you referring to. I
> think the confusion stems from the two modes that we have defined in Sec 4.
>
> My point is that running IP over an individual leg is an unnatural act, as
> it is architecturally incompatible with how LAGs are supposed to work
> (IMHO).  As a result, implementors have to hack something into their
> implementations to sort-of-pretend-but-not-really that IP is enabled on the
> individual legs.
>
> There is no good way to spec such a thing because there isn't supposed to
> be any IP binding to the individual leg.  What you're doing, in effect, is
> modifying the definition of a LAG in order to shoehorn BFD into the
> picture, which is rather out-of-scope for the BFD group (and for the IETF
> in general).  I don't see how it would be possible to standardize such a
> thing.
>
> >
> replacing LACP. It keeps track of all the links that are alive and informs
> the LAG manager (or some entity, beyond the scope of this document) about
> it. The IP interface over the LAG is torn down once the number of remaining
> links reaches the minimum threshold number. BFD is thus used to decide the
> fate of an IP interface.
> >
> It's a reasonable thing if all you're doing is replacing LACP with BFD as
> the per-leg liveness test;  it makes BFD invisible to applications, as the
> service interface is the top of the LAG.  This is subtly different than
> having a BFD-over-the-top session in addition to the individual legs, as
> doing so provides visible BFD semantics rather than LAG semantics, and the
> service interface reflects those semantics (so you can potentially do
> things like AdminDown and the like).
> >
> > Establish BFD sessions over each member link. The aggregate result of
> this decides the fate of only the LAG (and not that of the IP interface
> defined above it). Establish another BFD session over the LAG (which is
> what most implementations do today) that runs sedate timers and ensures IP
> reachability with the other end. This mode gives us the best of both worlds.
> >
> I'm agnostic about the choices;  I can see some benefit in both.  But the
> real problem is how to do the per-link stuff in a reasonable way.
>
> --Dave
>
>

---

From: Marc Binderberger <[email-masked]>
Date: Fri, 16 Dec 2011 18:20:06 +0100

Hello Dave,

trying to understand the "private API" statement.

>> [...] 

If you receive an IP packet on a member link then is it this problem you face (?):

(i)   an IP/UDP socket can be opened for the whole LAG, as it has an IP address. Likely the socket would exist per-system and extensions would allow to either bind it to the LAG or to receive the interface information together with the packet

(ii)  opening an IP/UDP socket for a member link interface can not be expected to work as the member link may not be "IP activated".

(iii) opening an "Ethernet socket" for a particular Ethertype and interface should work (assuming your OS offers this kind of socket)

Unless in (i) the implementation provides you with the information of the member link the packet was received on  or  allows you in (ii) to bind the IP/UDP socket to the member link , unless one of this is given one could not use IP/UDP encapsulation for the micro sessions.

Does this describe the problem?

Now for BFD in RFC5880/5881 we also need the interface information to handle your_discr zero packets. Why was it okay to assume you know this information from  the API but we cannot assume that the member-interface information is provided in (i) above?

Or in other words: what API model can we assume when defining a standard?

Thanks & Regards,
Marc

---

From: Nitin Bahadur <[email-masked]>
Date: Thu, 22 Dec 2011 11:34:53 -0800

Hi Manav,

> Using the IP address of the interface leads into a cyclic loop as you cant bring up the member links since L3 is not up, and
> you cant bring up L3 since the member links are still down.

I think we are talking of 2 different things here. In vanilla BFD mode today, the IP link comes up first and then BFD starts on it. BFD in no way
brings down a Sonet link for instance. So why do you want BFD to bring down an Ethernet link? If you want that to happen, you have to use L2 encap
or use ethernet oam.

If all member links are down, then the AE will be down and L3 will be down as well. Once a single AE member comes up, the AE bundle should
come up. Once the AE is UP, your ARP will work as well and L3 will come UP. And then your BFD on that AE member will come as well.
Now that BFD is UP, you can feed that into routing protocols or other protocols that can feed off that. When the 2nd AE member comes up, the 2nd BFD
session will also come up. And you can feed that again into whatever protocol you want.

Note that "you can bring the member link UP even when L3 is not up". The member link just comes up itself....maybe I am missing something very basic here.

> On a point-to-point link isn't the discriminator value good enough to check if the packet is indeed for the recipient?

         On a p2p link yes....but you are assuming that the P2P link doesn't have any L2 switches in the middle. If there are L2 switches in the middle, you
          could have mis-routed packets coming in and in that case the discriminator value is not sufficient.

Thanks
Nitin

---

From: Nitin Bahadur <[email-masked]>
Date: Thu, 22 Dec 2011 11:58:51 -0800

And just to clarify a little bit more....

I don't want BFD to replace LACP. I want it to supplement LACP. Just like today if we run BFD for OSPF adjacencies, we don't get rid of OSPF hellos...
But if the BFD session goes down, we tell OSPF about it and OSPF reacts to that. Similarly (pardon my lack of knowledge about LACP), LACP can react to
BFD member sessions down. The link can be brought up by LACP at a slower interval. The goal should be fast failure detection and not necessarily fast
Link-up following a failure correction.

Thanks
Nitin

On 12/22/11 11:34 AM, "Nitin Bahadur" <[email-masked]> wrote:

Hi Manav,

Thanks
Nitin

---

From: Marc Binderberger <[email-masked]>
Date: Thu, 22 Dec 2011 22:57:09 +0100

Hello Nitin,

I do not disagree but the goal you define may be too restricted.

> The goal should be fast failure detection

We should all be able to agree on this. This is the minimum achievement.

But already this goal may have two options: with and without LACP. While I would recommend LACP to customers it does not mean it's always possible.

Without LACP you likely want to prevent a LAG link "going up" after you just received a BFD Down state. But how?

Even with LACP you could - maybe not a very likely scenario - have LACP connectivity but BFD is interrupted. Again stopping an immediate link "going up" again may be a good thing.

For now I prefer an L3 encaps that is not preventing BFD to control the "going up". We may drop it when we learn the solution covering, lets say, 90% of the setups (LACP supported by BFD down?) is small & elegant while the 100% solution is a complex beast. But we don't know this yet.

And of course we don't know if - for a standardization - it is possible to define a packet Rx/Tx API at layer-3. Another reason why I think after a good discussion about pros and cons of Unicast, Multicast, new UDP port we should stop that discussion and wait for the outcome of other discussions, e.g. interaction between BFD and 802.1X state machine.
(still waiting for Dave Katz to shed some light on the "private API" comment he made :-)

For the "switch in the middle", maybe this wasn't clear but draft-mmm is not supporting this for the per-member BFD. Nor do I think any other draft I'm aware of. It's back-to-back. More precise: the per-member sessions and the overall "BLM" is between directly connected neighbours, as mentioned in RFC5882, section 7.3. This is why we allow a combination of it with "BBP" (single session over LAG), as the single session can go through an Ethernet switch; this is scenario 7.4 in RFC5882.

I assume you mention the switch to explain why the per-interface IP address is a good thing, as it allows a check if the BFD packet arrived at the right interface? Now for LAG we have N ports and all the BFD packets arriving will have the same destination IP address if you use the interface IP;  I think this IP check must be dropped for micro sessions.

Instead the "interface information" must be used. Or some out-of-band mechanism informs you about the remove discriminator, similar to RFC5884 (I'm not promoting this, I don't like it too much due to the complexity but for the sake of a more complete discussion I mention it)

"Interface information" is another interesting point for the Rx/Tx API in the standard. Can we demand this information for a layer-3 API?
(not asking if we code-wise can do it. We probably all _do_ already).

Too long, my email. In short: I don't think we can get to a conclusion about Unicast/Multicast until we have discussed the other topics and then see what is needed.

Regards, Marc

---

From: Alexander Vainshtein <[email-masked]>
Date: Fri, 23 Dec 2011 08:39:19 +0200

Marc, Nitin and all,

Quoting from Marc's email:

"I don't think we can get to a conclusion about Unicast/Multicast until we have discussed the other topics and then see what is needed."

I fully agree with this statement.

Regards,
     Sasha

________________________________________
From: [email-masked] [[email-masked]] On Behalf Of Marc Binderberger [[email-masked]]
Sent: Thursday, December 22, 2011 11:57 PM
To: Nitin Bahadur
Cc: [email-masked]
Subject: Re: BFD on LAG interfaces

Hello Nitin,

Regards, Marc

This e-mail message is intended for the recipient only and contains information which is CONFIDENTIAL and which may be proprietary to ECI Telecom. If you have received this transmission in error, please inform us by e-mail, phone or fax, and then delete the original and all copies thereof.

---

From: "Bhatia, Manav (Manav)" <[email-masked]>
Date: Tue, 27 Dec 2011 21:13:33 +0530

Yes, this sounds reasonable.

What we're looking at right now is this:

1. We use LACP to bring up the links. 
2. We establish micro BFD sessions per link (using ingress port + source MAC as a unique identifier)
3. If BFD session goes down, we inform the 802.1ax so that it can bring down the state so that the link is NOT used for forwarding.
4. Wait for LACP to bring up the member port of the LAG. Once its UP, BFD sessions again establish and the process repeats.
 
So, we use BFD only to bring down the link.
 
Once the link is up, we use IP encapsulated BFD (no L2 encap required) with either unicast or multicast addressing scheme. 

This solution however requires LACP to work in tandem with the micro BFD sessions for the LAG. This will work for all scenarios except the one where a provider does NOT want to run LACP on the LAG (and I am not sure why somebody would NOT want LACP btw).

Cheers, Manav

> Sent: Friday, December 23, 2011 1:29 AM
> To: Nitin Bahadur; Bhatia, Manav (Manav); Jeff Tantsura; 
> [email-masked]
> I don't want BFD to replace LACP. I want it to supplement 
> LACP. Just like today if we run BFD for OSPF adjacencies, we 
> don't get rid of OSPF hellos...
> But if the BFD session goes down, we tell OSPF about it and 
> OSPF reacts to that. Similarly (pardon my lack of knowledge 
> about LACP), LACP can react to BFD member sessions down. The 
> link can be brought up by LACP at a slower interval. The goal 
> should be fast failure detection and not necessarily fast

---

From: Alexander Vainshtein <[email-masked]>
Date: Tue, 27 Dec 2011 20:59:40 +0200

Manav,
The scheme you've presented looks OK to me.

Regarding your question - why somebody would NOT want LACP - I can only add that micro-BFD sessions, in my opinion, are used to replace lacking failure indications from the physical layer. In these scenarios working without LACP would be quite problematic in any case... 

(Not sure if this is the right answer, but it is the best one I can give:-)

Regards,
     Sasha

________________________________________
From: [email-masked] [[email-masked]] On Behalf Of Bhatia, Manav (Manav) [[email-masked]]
Sent: Tuesday, December 27, 2011 5:43 PM
To: Nitin Bahadur; [email-masked]
Subject: RE: BFD on LAG interfaces

Cheers, Manav

---

From: Tom Sanders <[email-masked]>
Date: Thu, 29 Dec 2011 21:08:16 +0530

Sasha,

I agree. Instead of us speculating, can the operators who are on this
list speak on whether they are looking at BFD managing their LAGs
without LACP? If Yes, then why do they want this? It would be a shame
to over-engineer a solution when a simple one is available.

Toms

-- 
Toms.

---

From: Marc Binderberger <[email-masked]>
Date: Thu, 29 Dec 2011 18:23:38 +0100

Hello Tom, Alexander et al.,

in general I agree with your statements. BFD is _not_ LACP. BFD should be planned as a booster for the detection. And keep it simple. On the other hand we need answers for the various cases.

The problem is to bring up the link with protocol P1 but bring it down with protocol P2. The underlying assumption is that P1 is interrupted as well when P2 goes down. True for a clean link cut, questionable if something goes wrong with the layer-3 alone. Then P1 (LACP or L1) may be up and thus the LAG port is active while P2 (BFD) stays down.

Maybe we can live with it (?).

As far as I know about existing implementations they seem to avoid the above situation by either:

(i) use BFD to control - alone or in combination with LACP - the port going active and going down

(ii) use BFD to update routing protocols alone but do not affect the LAG.

Maybe someone has implemented the "P1 brings up, P2 brings down" and can talk about the experience?

Regards, Marc

---

From: Gregory Mirsky <[email-masked]>
Date: Thu, 29 Dec 2011 14:10:45 -0500

Dear Sasha, tom, et al.,
I concur with Sasha and Tom that in presence of LACP so-called micro-BFD, a.k.a. BFD over LAG member link(s), can be viewed as substitute to Layer 1 failure indication. IMHO, use of micro-BFD without LACP or with propritary extensions of LACP (mLACP) might be for further study. But it looks as to replace LACP we'll need to change character of BFD from simple connectivity monitoring to make it TE-aware.

	Regards,
		Greg  

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Tom Sanders
Sent: Thursday, December 29, 2011 7:38 AM
To: Alexander Vainshtein
Cc: [email-masked]
Subject: Re: BFD on LAG interfaces

Sasha,

I agree. Instead of us speculating, can the operators who are on this list speak on whether they are looking at BFD managing their LAGs without LACP? If Yes, then why do they want this? It would be a shame to over-engineer a solution when a simple one is available.

Toms

On 28 December 2011 00:29, Alexander Vainshtein <[email-masked]> wrote:
> From: [email-masked] [[email-masked]] On Behalf Of 
> Bhatia, Manav (Manav) [[email-masked]]
> 2. We establish micro BFD sessions per link (using ingress port + 
> source MAC as a unique identifier) 3. If BFD session goes down, we inform the 802.1ax so that it can bring down the state so that the link is NOT used for forwarding.
>> I don't want BFD to replace LACP. I want it to supplement LACP. Just 
>> like today if we run BFD for OSPF adjacencies, we don't get rid of 
>> OSPF hellos...
>> But if the BFD session goes down, we tell OSPF about it and OSPF 
>> reacts to that. Similarly (pardon my lack of knowledge about LACP), 
>> LACP can react to BFD member sessions down. The link can be brought 
>> up by LACP at a slower interval. The goal should be fast failure 
>> detection and not necessarily fast Link-up following a failure 
>> correction.
>>
>> Thanks
>> Nitin
>>
>

--
Toms.

---

From: "Nobo Akiya (nobo)" <[email-masked]>
Date: Thu, 29 Dec 2011 16:33:04 -0800

Hello Marc, et al,

> The problem is to bring up the link with protocol P1 but bring it down
> with protocol P2. The underlying assumption is that P1 is interrupted
> as well when P2 goes down. True for a clean link cut, questionable if
> something goes wrong with the layer-3 alone. Then P1 (LACP or L1) may
> be up and thus the LAG port is active while P2 (BFD) stays down.
> 

"Can we live with it" is indeed the primary question. If consensus is
yes, then secondary question becomes what action should LMM take (if any
at all)? LMM will know P1 is up but P2 has not come up for "long" time:
hours or even days. Should LMM continue to use such link, or should it
time out (and stop using the link) eventually? If we want to keep it
simple, then the answer here would be "no timeout mechanism" here as
well.

> Maybe someone has implemented the "P1 brings up, P2 brings down" and
can
> talk about the experience?

I'm curious on this. Side effect here is traffic blackhole. To elaborate
the side effect, here are couple of scenarios which can happen.
Assumption here is there is one link which P1 is up and P2 has not come
up due to layer-3 specific failure. Let's label this link L1.

(1) Routing protocols "handshake" happens to take place over non-L1 link
and traffic shifts over 802.1ax, but traffic going over L1 will be
blackholed.
(2) Static route, once 802.1ax state is up, will install route over it,
but traffic going over L1 will be blackholed.

Certainly would like to hear input from vendors if someone has
implemented this model, but would also like to hear from operators if
this potential issue is something "we can live this".

Regards,
Nobo

> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Marc Binderberger
> Sent: Friday, December 30, 2011 2:24 AM
> To: Alexander Vainshtein; Tom Sanders
> Cc: [email-masked]
> in general I agree with your statements. BFD is _not_ LACP. BFD should
> be planned as a booster for the detection. And keep it simple. On the
> other hand we need answers for the various cases.
> 
> The problem is to bring up the link with protocol P1 but bring it down
> with protocol P2. The underlying assumption is that P1 is interrupted
> as well when P2 goes down. True for a clean link cut, questionable if
> something goes wrong with the layer-3 alone. Then P1 (LACP or L1) may
> be up and thus the LAG port is active while P2 (BFD) stays down.
> 
> Maybe someone has implemented the "P1 brings up, P2 brings down" and
can
> talk about the experience?
> 
> 
> Regards, Marc
> 
> 
> 
> 
> >> Regarding your question - why somebody would NOT want LACP - I can
> only add that micro-BFD sessions, in my opinion, are used to replace
> lacking failure indications from the physical layer. In these
scenarios
> working without LACP would be quite problematic in any case...
> >>
> >> From: [email-masked] [[email-masked]] On Behalf
> Of Bhatia, Manav (Manav) [[email-masked]]
source
> MAC as a unique identifier)
> >> 3. If BFD session goes down, we inform the 802.1ax so that it can
bring
> down the state so that the link is NOT used for forwarding.
> >> 4. Wait for LACP to bring up the member port of the LAG. Once its
UP,
> BFD sessions again establish and the process repeats.
> >>
> >> Once the link is up, we use IP encapsulated BFD (no L2 encap
required)
> with either unicast or multicast addressing scheme.
> >>
> >> This solution however requires LACP to work in tandem with the
micro
> BFD sessions for the LAG. This will work for all scenarios except the
> one where a provider does NOT want to run LACP on the LAG (and I am
not
> sure why somebody would NOT want LACP btw).
> >>
> >> Cheers, Manav
> >>
> >> This e-mail message is intended for the recipient only and contains
> information which is CONFIDENTIAL and which may be proprietary to ECI
> Telecom. If you have received this transmission in error, please
inform
> us by e-mail, phone or fax, and then delete the original and all
copies
> thereof.