Subject: Multicast vs Unicast (was BFD on LAG interfaces)

From: Tom Sanders <[email-masked]>
Date: Thu, 22 Dec 2011 06:00:35 +0530

The two options before the WG are:

1. Use IP encapsulation
2. Use direct L2 encapsulation.

With (1) we have an option of (1a) Using Multicast and (1b) Using Unicast.

Can we arrive on a consensus between choosing 1a and 1b?

Given that the new draft has this nice Appendix that succinctly
discusses this, can the proponents of (1b) justify why they think (1a)
is not good enough.

Lets try closing the different threads so that we can converge towards
a solution.

Toms

On 20 December 2011 06:24, Glen Kent <[email-masked]> wrote:
> Some more reasons for going in for multicast given at the end of the
> new version.
>
> http://tools.ietf.org/html/draft-mmm-bfd-on-lags-01#appendix-A.1
>
> I dont think there is a need to specify that the TTL=1 since the
> multicast addresses have a link local scope. Am i missing something?
>
> Thanks,
> Glen
>
> On Tue, Dec 20, 2011 at 5:08 AM, Marc Binderberger <[email-masked]> wrote:
>> Hello Nobo,
>>
>>> I see new revision out there, will have to go check it out.
>>
>> Thanks for the interest :-)
>>
>>
>>> The other question that I have on this draft is, do we really need/want
>>> to reserve IP multicast address for this purpose. Most cases, if not
>>> all, where operator wants to use BFD to protect LAG at member level
>>> should have some sort of IP address assigned at LAG.
>>
>> I have seen there had been some "unnumbered" discussion but actually that is not the reason. Actually "unnumbered" was put aside when we discussed the original draft-chen-bfd-interface and the list feedback was to focus on LAG with IP at that time.
>>
>>
>> What we want to achieve:
>>
>> - you can have a member-link session (aka "micro session")
>> - you can have additionally a BBP session (the single session over LAG) for the main LAG interface
>>
>> On the receiving side, what do you do to differentiate these sessions (think about the your_discr zero case)?
>>
>> (i) different address
>> (ii) different UDP port
>>
>>
>> Both should work. We started the draft with (i) in mind and I don't see a hard reason to change this.
>>
>> For (i) you then come up with the idea of a "dedicated address". 127/8 was proposed by Gregory and again, that would work although ... how do you differentiate this from a BFD-LSP (RFFC5884) packet arriving on the LAG interface?  So a Multicast address seems another option.
>>
>>
>>> Resulting behavior will be less deviating from RFC5881: same
>>> demux procedures can be used when discriminator is zero(0),
>>
>> so can it for the link-local multihop address I think - no? Now we do mention this in section 3.2, as a reminder that for your_Discr zero the main difference is the ingress interface:
>>
>>   The demultiplexing of a received packet is solely based on the Your
>>   Discriminator field, if this field is nonzero.  For the initial Down
>>   packet of a micro session this value may be zero.  In this case
>>   demultiplexing MUST be based on some combination of other fields
>>   which MUST include the interface information of the member link.
>>
>>
>>> no need to
>>> come up with special case of TTL=1 which was discussed.
>>
>>
>> oh-oh I missed this part! Have to go through the emails. TTL = 1 ?!?  Our draft is not talking about this (section 3 mentions RFC5880, so you would have a TTL of 255)
>>
>>
>>> I'm questioning
>>> this aspect since I didn't see any strong arguments for burning up IP
>>> multicast address.
>>
>> actually I wonder if all-router or all-hosts wouldn't do the job. We could probably even use 255.255.255.255 as the setup is a direct link, back to back. Open to this. But if it turns out we need one, well then we need one. Or alternatively - see (ii) above - a new IANA UDP port. Not sure what is better/worse.
>>
>>
>> Regards, Marc
>>
>>
>>
>>
>>>
>>> Thanx,
>>> Nobo
>>>
>>>> -----Original Message-----
>>>> From: Marc Binderberger [mailto:[email-masked]]
>>>> Sent: Monday, December 19, 2011 5:33 PM
>>>> To: Nobo Akiya (nobo)
>>>> Cc: Alexander Vainshtein; Dave Katz; [email-masked]
>>>> Subject: Re: BFD on LAG interfaces
>>>>
>>>> Hello Nobo,
>>>>
>>>> thanks for the comments. Let me check this with the new version we are
>>>> just preparing (almost done).
>>>>
>>>> And yes, a cleaner description when the creation/deletion is part of
>>> it.
>>>> But I also think this is where Alexander's comment came from: without
>>>> some modifications in 802.1AX it will not be possible to lock the
>>> "going
>>>> UP" with BFD. Otherwise you always run into questions and issues -
>>> starting
>>>> LACP first? Or BFD first? Or both in parallel?
>>>>
>>>>
>>>>> Generally speaking, when up/down use different methods, I've seen a
>>>>> system fall into continuous flap cycle. Down by BFD, up by
>>> protocol/LMM
>>>>> is one possible approach, and your draft wants to allow this for
>>>>> interoperability purpose. I can understand the need, but side effect
>>>> can
>>>>> be nasty, and it (possible continuous flap) can be described?
>>>>
>>>>
>>>> *sigh* yes ... maybe we should write a draft on flap dampening? :-)
>>>>
>>>>
>>>> Let me go through our latest (internal) version if that addresses your
>>>> points.
>>>>
>>>>
>>>> Thanks!
>>>>
>>>> REgards, Marc
>>>>
>>>>
>>>>
>>>>
>>>> On 2011-12-19, at 2:59 , Nobo Akiya (nobo) wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> Section 3.2.
>>>>>
>>>>>> A member link of the LAG is not used anymore for data forwarding
>>>>> when
>>>>>> the particular BFD session running over that link goes down.  The
>>>>>> member link MUST be removed from the LAG.  The BFD session for the
>>>>>> link remains, i.e. it is not deleted.
>>>>>
>>>>> It is not clear when BFD session should get deleted (nor created).
>>> With
>>>>> above statement, BFD session could remain even when operator decides
>>>> to
>>>>> shut down a member, depending on how it's interpreted/implemented.
>>> When
>>>>> LACP is used, LMM will startup LACP only when interface is up. L2
>>> check
>>>>> starts only when L1 check passes. Check is done from bottom layer
>>>>> upwards. Going by same idea, BFD session should only be created when
>>>> L2
>>>>> check passes (LACP), when L2 check is being used. If L2 check is not
>>>>> being used, then L1 check is required for BFD session to get
>>> created.
>>>>> Absence in L1/L2 check pass should cause corresponding BFD sessions
>>>> to
>>>>> get deleted.
>>>>>
>>>>>
>>>>>>> 3. The situation when LAG goes down due to insufficient number of
>>>>> active
>>>>>> links must be reversible, i.e., the LAG must be able to go UP once
>>>>> this
>>>>>> condition disappears.
>>>>>>
>>>>>> yes, I would prefer this ;-)
>>>>>>
>>>>>>> Do these notes capture the problem space correctly? Or did I miss
>>>>>> something important?
>>>>>>>
>>>>>>> One point that is not clear to me is the timeframe for bringing
>>>>>> constituent links of a LAG up. I would expect that the requirements
>>>>> for
>>>>>> that could be much more relaxed (e.g., in order to avoid link
>>>>> flapping).
>>>>>> Is that correct? If it is, one possible implication is that it is
>>>>> possible
>>>>>> to use different mechanisms for bringing links DOWN and UP. To the
>>>>> best
>>>>>> of my understanding, this is how BFD is supposed to be used with
>>>>> routing
>>>>>> adjacencies today, i.e. BFD can bring the adjacency DOWN quite
>>> fast,
>>>>> but
>>>>>> it has to be brought UP by the appropriate routing protocol.
>>>>>>
>>>>>> the draft allows this, section 3.2 "To avoid this deadlock ...". So
>>>>> using
>>>>>> BFD to bring down fast alone is covered. Now RFC5882, e.g. section
>>>>> 4.1,
>>>>>> mentions that waiting for BFD's Up notification may be useful. Thus
>>>> we
>>>>>> allowed this in the draft as well.
>>>>>>
>>>>>>
>>>>>> [I skip the LACP, CFM part. We had this. Nothing wrong technically
>>>> but
>>>>>> somehow customers push BFD teams to deliver]
>>>>>
>>>>> Generally speaking, when up/down use different methods, I've seen a
>>>>> system fall into continuous flap cycle. Down by BFD, up by
>>> protocol/LMM
>>>>> is one possible approach, and your draft wants to allow this for
>>>>> interoperability purpose. I can understand the need, but side effect
>>>> can
>>>>> be nasty, and it (possible continuous flap) can be described?
>>>>>
>>>>>
>>>>> Section 4, first bullet.
>>>>>
>>>>> LAG will be taken down when BFD fails on sufficient members (LAG
>>>>> interface state will be down). LAG will come up potentially when BFD
>>>> on
>>>>> sufficient members come up (LAG interface state will be up). This
>>>>> behavior implies that all applications will benefit (by monitoring
>>> LAG
>>>>> interface state) even if those applications do not configure BFD on
>>>> LAG.
>>>>> Sure, one can configure applications with BFD over LAG, but that's
>>> not
>>>>> an absolute requirement for this model. Statements regarding this
>>>>> behavior will be beneficial to readers.
>>>>>
>>>>> Thanx,
>>>>> Nobo
>>>>>
>>>>
>>>> --
>>>> Marc Binderberger           <[email-masked]>
>>>
>>
>> --
>> Marc Binderberger           <[email-masked]>
>>

-- 
Toms.

---

From: Jeff Tantsura <[email-masked]>
Date: Wed, 21 Dec 2011 19:58:13 -0500

Hi,

/*customer/product manager mode*/
If we set aside religious discussion about layer violation/etc IMHO 1a is just fine, it allows reusing existing BFD implementations with minimum changes, hence decreases time to market == happy customers.
I think we should still be keeping our focus on IP LAG, as we are trying to extend RFC5881 to work over LAG where BFD is used is to track IPv4 and IPv6 connectivity between directly connected systems 
IMO for L2 LAG Eth OAM would make much more sense (eg. .1ag aware LAG).

Regards,
Jeff

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Tom Sanders
Sent: Wednesday, December 21, 2011 4:31 PM
To: [email-masked]
Subject: Multicast vs Unicast (was BFD on LAG interfaces)

1. Use IP encapsulation

Given that the new draft has this nice Appendix that succinctly discusses this, can the proponents of (1b) justify why they think (1a) is not good enough.

Lets try closing the different threads so that we can converge towards a solution.

Toms

On 20 December 2011 06:24, Glen Kent <[email-masked]> wrote:
>>> The other question that I have on this draft is, do we really 
>>> need/want to reserve IP multicast address for this purpose. Most 
>>> cases, if not all, where operator wants to use BFD to protect LAG at 
>>> member level should have some sort of IP address assigned at LAG.
>>
>>> Resulting behavior will be less deviating from RFC5881: same demux 
>>> procedures can be used when discriminator is zero(0),
>>
>> oh-oh I missed this part! Have to go through the emails. TTL = 1 ?!?  
>> Our draft is not talking about this (section 3 mentions RFC5880, so 
>> you would have a TTL of 255)
>>
>>
>>> I'm questioning
>>> this aspect since I didn't see any strong arguments for burning up 
>>> IP multicast address.
>>
>>>> thanks for the comments. Let me check this with the new version we 
>>>> are just preparing (almost done).
>>>>
>>>> But I also think this is where Alexander's comment came from: 
>>>> without some modifications in 802.1AX it will not be possible to 
>>>> lock the
>>> "going
>>>>> Generally speaking, when up/down use different methods, I've seen 
>>>>> a system fall into continuous flap cycle. Down by BFD, up by
>>> protocol/LMM
>>>> Let me go through our latest (internal) version if that addresses 
>>>> your points.
>>>>
>>>>
>>>> Thanks!
>>>>
>>>> REgards, Marc
>>>>
>>>>
>>>>
>>>>
>>>>>> member link MUST be removed from the LAG.  The BFD session for 
>>>>>> the link remains, i.e. it is not deleted.
>>>>>
>>>>> check passes (LACP), when L2 check is being used. If L2 check is 
>>>>> not being used, then L1 check is required for BFD session to get
>>> created.
>>>>> Generally speaking, when up/down use different methods, I've seen 
>>>>> a system fall into continuous flap cycle. Down by BFD, up by
>>> protocol/LMM

--
Toms.

---

From: Tom Sanders <[email-masked]>
Date: Thu, 22 Dec 2011 06:46:49 +0530

Oh and btw, i would too prefer option (1a), i.e. going the multicast way.

Toms

-- 
Toms.

---

From: "Nobo Akiya (nobo)" <[email-masked]>
Date: Thu, 22 Dec 2011 06:46:53 -0800

Hello Tom, Jeff, et al,

I feel obligated to reply to this thread, so here we go.

> to work over LAG where BFD is used is to track IPv4
> and IPv6 connectivity between directly connected systems

Core idea of running BFD per member is for link verification, not for IPv4 and IPv6 connectivity verification. However, depending on how it's implemented, you could verify pieces of IP forwarding, and this will be more possible if unicast and actually 802.1ax assigned IP address was to be used.

> > I think we should still be keeping our focus on IP LAG, as we are trying
> to extend RFC5881 to work over LAG

We will have minor differences with RFC5881 when we start using multicast address.

For example:

   On a multiaccess network, BFD Control packets MUST be transmitted
   with source and destination addresses that are part of the subnet
   (addressed from and to interfaces on the subnet).

Sure, BFD per member will work only on point-to-point links, but ether is still considered multi-access media. Subtle and minor, yes. But less likely to deviate if we use unicast IP address.

Some arguments given against unicast IP approach in the draft lacks some information. For example, if well-known MAC is used with unicast IP, ARP point is really not valid. Another will be that, destination address required. Simplicity is good, but nobody has argued against this config as too much/complicated. If we consider those, then unicast IP doesn't look so bad.

And, there's also the aspect of, if we were to use unicast IP, what changes are needed to run BFD per member and same AF BBP on main 802.1ax interface.

Don't get me wrong, I'm not against multicast IP usage on this technology (1a), but things have just moved so fast with this draft ... I'm just creating some noise to see if we have exhausted necessary discussions.

In any case, myself and few others, including all MMM authors, have been exchanging some emails outside of the list. One of us should be able to get back with conclusion soon.

Regards,
Nobo

> Sent: Thursday, December 22, 2011 10:17 AM
> To: Jeff Tantsura
> Cc: [email-masked]
> Subject: Re: Multicast vs Unicast (was BFD on LAG interfaces)
> 
> > If we set aside religious discussion about layer violation/etc IMHO
> 1a is just fine, it allows reusing existing BFD implementations with
> minimum changes, hence decreases time to market == happy customers.
> > I think we should still be keeping our focus on IP LAG, as we are trying
> to extend RFC5881 to work over LAG where BFD is used is to track IPv4
> and IPv6 connectivity between directly connected systems
> > Given that the new draft has this nice Appendix that succinctly discusses
> this, can the proponents of (1b) justify why they think (1a) is not good
> enough.
> >
> >>> I have seen there had been some "unnumbered" discussion but actually
> that is not the reason. Actually "unnumbered" was put aside when we
> discussed the original draft-chen-bfd-interface and the list feedback
> was to focus on LAG with IP at that time.
> >>>
> >>>
> >>> What we want to achieve:
> >>>
> >>> For (i) you then come up with the idea of a "dedicated address". 127/8
> was proposed by Gregory and again, that would work although ... how do
> you differentiate this from a BFD-LSP (RFFC5884) packet arriving on the
> LAG interface?  So a Multicast address seems another option.
> >>>
> >>>
> >>> so can it for the link-local multihop address I think - no? Now we
> do mention this in section 3.2, as a reminder that for your_Discr zero
> the main difference is the ingress interface:
> >>>
> >>> actually I wonder if all-router or all-hosts wouldn't do the job.
> We could probably even use 255.255.255.255 as the setup is a direct link,
> back to back. Open to this. But if it turns out we need one, well then
> we need one. Or alternatively - see (ii) above - a new IANA UDP port.
> Not sure what is better/worse.
> >>>
> >>>
> >>> Regards, Marc
> >>>
> >>>
> >>>
> >>>
> >>>>
> >>>> Thanx,
> >>>> Nobo
> >>>>

---

From: "Bhatia, Manav (Manav)" <[email-masked]>
Date: Thu, 22 Dec 2011 21:25:37 +0530

>    (addressed from and to interfaces on the subnet).

I am not sure if quoting 5881 really helps here, since 5881 also states:

	BFD Control packets MUST be transmitted in UDP packets with
	destination port 3784, within an IPv4 or IPv6 packet. 

For Unicast we're thinking of using a different UDP port, so a violation is done there as well.

Also note that we may, at some point of time, start supporting BFD for multipoint networks. In that case, we may end up receiving BFD packets with a multicast destination IP.

[snipped]
 
> Some arguments given against unicast IP approach in the draft 
> lacks some information. For example, if well-known MAC is 
> used with unicast IP, ARP point is really not valid. Another 

In my private email I had pointed out that this is an unnatural design.

Using a well known L2 multicast MAC address for unicast L3 is quite kludgy. Router Implementations look at the dest MAC and if it's the system's or the port's MAC or an L3 multicast MAC then those packets are sent to the L3 engine. Packets with any other MAC (IS-IS is an exception) are considered for L2 switching. Now, by introducing well known MACs that L3 can use we start complicating the HW by introducing a list of special MACs (besides IS-IS) that now need to be processed by L3. Doing it for micro sessions is setting up a bad precedent. One day we'll find that we have a list of 50+ special MACs that L3 needs to process.

>From system scaling, this is not good. Its architecturally a hack. 

Currently we have instances where we use a multicast IP address for Unicast purposes (Sections 8.1 and 8.2 of RFC 2328)

[snipped]

> 
> In any case, myself and few others, including all MMM 
> authors, have been exchanging some emails outside of the 
> list. One of us should be able to get back with conclusion soon.

Yup, and this should happen very soon!

Cheers, Manav