Subject: Re: BFD over VXLAN: Trapping BFD Control packet at VTEP

From: Greg Mirsky <[email-masked]>
Date: Thu, 1 Aug 2019 08:40:14 -0400

Hi Dinesh,
as I understand, Sridhar is on vacation. In the meantime, this is his
response to the question on using VTEP's MAC address as the destination MAC
in the inner Ethernet frame:

T. Sridhar
Sun, Jun 30, 7:13 PM
to Reshad, Martin, [email-masked], Matthew, Sam, Jeffrey
Reshad,

Sorry - could not respond earlier.

Joel is right - the draft does impose a requirement on the inner MAC being
the same as the VTEP MAC and eats into the tenant MAC space which is not a
desirable approach. You *could* use VNI0 as the one only VNI to specify
this but that's not ideal either since we did not impose any restriction on
the VNI space.

Btw,  I also think I should modify the comment I had made below about using
the inner IP to be the same as the VTEP IP since there is no guarantee that
the inner MAC frame is an IP packet.

Thanks,
Sridhar

On 6/26/19, 6:59 PM, "Reshad Rahman (rrahman)" <[email-masked]> wrote:

    Thank you Sridhar. One concern which came up is wrt the use of the
destination VTEP MAC address as dest MAC in the inner IP header (see
attached email). Could you please comment on that aspect?

    Regards,
    Reshad.

I understand that RFC 7348 maybe is not clear on that issue. I'd like to
understand how the existing implementations behave, process VXLAN header
and the inner Ethernet frame to minimize changes BFD over VXLAN may impose
on the implementation.

Regards,
Greg

On Thu, Aug 1, 2019 at 12:48 AM Dinesh Dutt <[email-masked]> wrote:

> I don't understand his objection. My recommendation is to understand that
> before we propose new text. I fear otherwise that we'll have a new draft in
> a few months to address the issue of using non-mgmt VNI.
>
> Dinesh
> On Jul 31, 2019, 12:07 PM -0700, Greg Mirsky <[email-masked]>,
> wrote:
>
> Hi Dinesh,
> if I recall correctly, T.Sridhar has noted that VTEP's MAC must not be
> used as the destination MAC address in the inner Ethernet frame.
>
>
>
> Also, I should have been more precise in the proposed text, please see the
> updated version to stress that the management VNI MUST NOT be one of the
> tenant's VNIs:
> NEW TEXT:
>
> An operator MUST select a VNI number to be used as Management VNI.
> Management VNI number MUST NOT be one of the tenant's VNIs to prevent
> sending VXLAN packets received on Management VNI to a tenant. VNI number 1
> is RECOMMENDED as the default for Management VNI.
>
>
> On Wed, Jul 31, 2019 at 2:25 PM Dinesh Dutt <[email-masked]> wrote:
>
>> Hi Greg,
>>
>> On Wed, Jul 31, 2019 at 9:20 AM Greg Mirsky <[email-masked]>
>> wrote:
>>
>>> Hi Dinesh,
>>> thank you for your consideration of the proposal and questions. What
>>> would you see as the scope of testing the connectivity for the specific
>>> VNI? If it is tenant-to-tenant, then VTEPs will treat these packets as
>>> regular user frames. More likely, these could be Layer 2 OAM, e.g. CCM
>>> frames. The reason to use 127/8 for IPv4, and 0:0:0:0:0:FFFF:7F00:0/104 for
>>> IPv6 is to safeguard from leaking Ethernet frames with BFD Control packet
>>> to a tenant.
>>> You've suggested using a MAC address to trap the control packet at VTEP.
>>> What that address could be? We had proposed using the dedicated MAC and
>>> VTEP's MAC and both raised concerns among VXLAN experts. The idea of using
>>> Management VNI may be more acceptable based on its similarity to the
>>> practice of using Management VLAN.
>>>
>>
>> If you use the inner IP address as the VTEP IP address, then use the MAC
>> address that the VTEP would respond with when replying to an ARP for that
>> VTEP IP address. If a VXLAN expert disagrees with this, could you kindly
>> tell me who it is so that I can understand their disagreement? So this
>> handles the case where the VNI is not a user-tenant VNI. If the VNI used in
>> the BFD packet is a user-tenant VNI, then the receiving VTEP MUST have an
>> IP address in that VNI (mapped to a VRF) else you cannot use that VNI in
>> the BFD packet. Why won't this combination address all the cases you've
>> listed? What am I missing? Define VNI 1 as a possible use, not VNI 0. I
>> objected to VNI 0 because there are too many switching siicon out there and
>> some of them will not be able to handle this scenario.
>>
>> Dinesh
>>
>>>
>>> Regards,
>>> Greg
>>>
>>> On Wed, Jul 31, 2019 at 12:03 PM Dinesh Dutt <[email-masked]> wrote:
>>>
>>>> Hi Greg,
>>>>
>>>> As long as the inner MAC address is such that the packet is trapped to
>>>> the CPU, it should be fine for use as an inner MAC is it not? Stating that
>>>> is better than trying to force a management VNI. What if someone wants to
>>>> test connectivity on a specific VNI? I would not pick a loopback IP address
>>>> for this since that address range is host/node local only. Is there a
>>>> reason you're not using the VTEP IP as the inner IP address ?
>>>>
>>>> Dinesh
>>>>
>>>> On Wed, Jul 31, 2019 at 5:48 AM Greg Mirsky <[email-masked]>
>>>> wrote:
>>>>
>>>>> Dear All,
>>>>> thank you for your comments, suggestions on this issue, probably the
>>>>> most challenging for this specification. In the course of our discussions,
>>>>> we've agreed to abandon the request to allocate the dedicated MAC address
>>>>> to be used as the destination MAC address in the inner Ethernet frame.
>>>>> Also, earlier using VNI 0 was changed from mandatory to one of the options
>>>>> an implementation may offer to an operator. The most recent discussion was
>>>>> whether VTEP's MAC address might be used as the destination MAC address in
>>>>> the inner Ethernet frame. As I recall it, the comments from VXLAN experts
>>>>> equally split with one for it and one against. Hence I would like to
>>>>> propose a new text to resolve the issue. The idea is to let an operator
>>>>> select Management VNI and use that VNI in VXLAN encapsulation of BFD
>>>>> Control packets:
>>>>> NEW TEXT:
>>>>>
>>>>> An operator MUST select a VNI number to be used as Management VNI.
>>>>> VXLAN packet for Management VNI MUST NOT be sent to a tenant. VNI number 1
>>>>> is RECOMMENDED as the default for Management VNI.
>>>>>
>>>>> With that new text, what can be the value of the destination MAC in
>>>>> the inner Ethernet? I tend to believe that it can be anything and ignored
>>>>> by the reciever VTEP. Also, if the trapping is based on VNI number, the
>>>>> destination IP address of the inner IP packet can from the range 127/8 for
>>>>> IPv4, and for IPv6 from the range 0:0:0:0:0:FFFF:7F00:0/104. And lastly,
>>>>> the TTL to be set to 1 (no change here).
>>>>>
>>>>> Much appreciate your comments, questions, and suggestions.
>>>>>
>>>>> Best regards,
>>>>> Greg
>>>>>
>>>>

---

From: Dinesh Dutt <[email-masked]>
Date: Wed, 31 Jul 2019 21:48:41 -0700

Dinesh
On Jul 31, 2019, 12:07 PM -0700, Greg Mirsky <[email-masked]>, wrote:

Hi Dinesh,
if I recall correctly, T.Sridhar has noted that VTEP's MAC must not be used
as the destination MAC address in the inner Ethernet frame.

On Wed, Jul 31, 2019 at 2:25 PM Dinesh Dutt <[email-masked]> wrote:

> Hi Greg,
>
> On Wed, Jul 31, 2019 at 9:20 AM Greg Mirsky <[email-masked]> wrote:
>
>> Hi Dinesh,
>>>> With that new text, what can be the value of the destination MAC in the
>>>> inner Ethernet? I tend to believe that it can be anything and ignored by
>>>> the reciever VTEP. Also, if the trapping is based on VNI number, the

---

From: Dinesh Dutt <[email-masked]>
Date: Thu, 1 Aug 2019 09:33:55 -0700

I understand his point now. It arose in my head because of a difference in
perspective. On a hardware switch, pretty much every implementation I've
seen reuses one of the device's MAC address across all the local VNIs. Even
in VMW's case, Even on a server, NIC's MAC address is by default reused
across all VNIs. It seems some software switches have taken liberties with
this.

Given this information, if we want to ensure any proposal we come up with
will work with existing packet switching silicon and software switch
implementation,  I also would prefer not to define something that'll be
overridden or worked around soon enough by a new draft. With these
assumptions, I see the problem definition as:

   - checking VTEP-to-VTEP connectivity in the overlay. Checking
   VTEP-to-VTEP connectivity in the underlay is a multi-hop BFD scenario.
   - I suspect whoever wants to check connectivity in the overlay will also
   want to check the connectivity in a specific VNI(s), Otherwise why are they
   checking this in the overlay? There maybe existing mechanisms such as IEEE
   OAM frames to check overlay in a specific VNI, but given my past
   experience, I suspect people will want a single unified mechanism using
   VXLAN (at least I see this in the data center use case).

I believe you;ve implicitly excluded the case of bullet 2 currently. I'm
not in favor of that for the reasons I stated above.

I see the primary solution principles as:

   - BFD packet addressed to the VTEP MUST not leak beyond the VTEP..
   - BFD packets exchanged between endpoints behind VTEPs MUST work as
   well.

So, on the critical question of how do we address the VTEP, I see we have
three possibilities:

   - Pick a MAC/IP pair that belongs to the VTEP in the VNI that you wish
   to send the packet in. These three parameters MUST be configurable: {VNI,
   MAC, IP}. The default VNI for this can be VNI 1.
   - Pick a management VNI and use that VNI to trap control packets to the
   CPU. I don't like this option because we're burning a VNI for just this
   purpose.
   - Pick a well-defined MAC/IP for a given VNI. This is the solution that
   tries to use the 127/8 address for example. But this is a Martian packet,
   is it not? I don't like it for this reason.

Does this make sense?

Dinesh

On Thu, Aug 1, 2019 at 5:40 AM Greg Mirsky <[email-masked]> wrote:

> Btw,  I also think I should modify the comment I had made below about
> using the inner IP to be the same as the VTEP IP since there is no
> guarantee that the inner MAC frame is an IP packet.
>
> Thanks,
> Sridhar
>
>
>
>> Also, I should have been more precise in the proposed text, please see
>> the updated version to stress that the management VNI MUST NOT be one of
>> the tenant's VNIs:
>> NEW TEXT:
>>
>>>> You've suggested using a MAC address to trap the control packet at
>>>> VTEP. What that address could be? We had proposed using the dedicated MAC
>>>> and VTEP's MAC and both raised concerns among VXLAN experts. The idea of
>>>> using Management VNI may be more acceptable based on its similarity to the

---

From: Santosh P K <[email-masked]>
Date: Fri, 2 Aug 2019 10:27:13 -0700

I have cross checked point raised about MAC address usage. It is possible
that tenant could be using physical MAC address and when a packet comes
with valid VNI with a MAC address that is being used by tenant then packet
will be sent to that tenant. This rules out the fact that we could use
physical MAC address as inner MAC to ensure packets get terminated at VTEP
itself.

Thanks
Santosh P K

On Wed, Jul 31, 2019 at 11:00 AM Santosh P K <[email-masked]>
wrote:

> Joel,
>    Thanks for your inputs. I checked implementation within Vmware. Perhaps
> I should have been more clear about MAC address space while checking
> internally. I will cross check again for the same and get back on this
> list.
>
> Thanks
> Santosh P K
>
> On Wed, Jul 31, 2019 at 10:54 AM Joel M. Halpern <[email-masked]>
> wrote:
>
>> Sorry to ask a stupid question.  Whose implementation?
>>
>> The reason I ask is that as far as I can tell, since the tenant does not
>> have any control access to the VTEP, there is no reason for the VTEP to
>> have a MAC address in the tenant space.  Yes, the device has a physical
>> MAC address.  But the tenant could well be using that MAC address.  Yes,
>> they would be violating the Ethernet spec.  But the whole point of
>> segregation is not to care about such issues.
>>
>> On the other hand, if you tell me that the VMWare implementation has an
>> Ethernet address that is part of the tenant space, well, they made up
>> this particular game.
>>
>> Yours,
>> Joel
>>
>> On 7/31/2019 1:44 PM, Santosh P K wrote:
>> > I have checked with implementation in data path. When we receive a
>> > packet with valid VNI then lookup for MAC will happen and it is VTEP
>> own
>> > MAC then it will be trapped to control plane for processing. I think we
>> > can have following options
>> > 1. Optional managment VNI
>> > 2. Mandatory inner MAC set to VTEP mac
>> > 3. Inner IP TTL set to 1 to avoid forwarding of packet via inner IP
>> > address.
>> >
>> >
>> > Thoughts?
>> >
>> > Thansk
>> > Santosh P K
>> >
>> > On Wed, Jul 31, 2019 at 9:20 AM Greg Mirsky <[email-masked]
>> > <mailto:[email-masked]>> wrote:
>> >
>> >     Hi Dinesh,
>> >     would you see as the scope of testing the connectivity for the
>> >     specific VNI? If it is tenant-to-tenant, then VTEPs will treat these
>> >     packets as regular user frames. More likely, these could be Layer 2
>> >     OAM, e.g. CCM frames. The reason to use 127/8 for IPv4, and
>> >     0:0:0:0:0:FFFF:7F00:0/104 for IPv6 is to safeguard from leaking
>> >     Ethernet frames with BFD Control packet to a tenant.
>> >     VTEP. What that address could be? We had proposed using the
>> >     dedicated MAC and VTEP's MAC and both raised concerns among VXLAN
>> >     experts. The idea of using Management VNI may be more acceptable
>> >     based on its similarity to the practice of using Management VLAN.
>> >
>> >     Regards,
>> >     Greg
>> >
>> >     On Wed, Jul 31, 2019 at 12:03 PM Dinesh Dutt <[email-masked]
>> >     <mailto:[email-masked]>> wrote:
>> >
>> >         Hi Greg,
>> >
>> >         As long as the inner MAC address is such that the packet is
>> >         trapped to the CPU, it should be fine for use as an inner MAC is
>> >         it not? Stating that is better than trying to force a management
>> >         VNI. What if someone wants to test connectivity on a specific
>> >         VNI? I would not pick a loopback IP address for this since that
>> >         address range is host/node local only. Is there a reason you're
>> >         not using the VTEP IP as the inner IP address ?
>> >
>> >         Dinesh
>> >
>> >         On Wed, Jul 31, 2019 at 5:48 AM Greg Mirsky
>> >         <[email-masked] <mailto:[email-masked]>> wrote:
>> >
>> >             Dear All,
>> >             thank you for your comments, suggestions on this issue,
>> >             probably the most challenging for this specification. In the
>> >             course of our discussions, we've agreed to abandon the
>> >             request to allocate the dedicated MAC address to be used as
>> >             the destination MAC address in the inner Ethernet frame.
>> >             Also, earlier using VNI 0 was changed from mandatory to one
>> >             of the options an implementation may offer to an operator.
>> >             The most recent discussion was whether VTEP's MAC address
>> >             might be used as the destination MAC address in the inner
>> >             Ethernet frame. As I recall it, the comments from VXLAN
>> >             experts equally split with one for it and one against. Hence
>> >             I would like to propose a new text to resolve the issue. The
>> >             idea is to let an operator select Management VNI and use
>> >             that VNI in VXLAN encapsulation of BFD Control packets:
>> >             NEW TEXT:
>> >
>> >                 An operator MUST select a VNI number to be used as
>> >                 Management VNI. VXLAN packet for Management VNI MUST NOT
>> >                 be sent to a tenant. VNI number 1 is RECOMMENDED as the
>> >                 default for Management VNI.
>> >
>> >             With that new text, what can be the value of the destination
>> >             MAC in the inner Ethernet? I tend to believe that it can be
>> >             anything and ignored by the reciever VTEP. Also, if the
>> >             trapping is based on VNI number, the destination IP address
>> >             of the inner IP packet can from the range 127/8 for IPv4,
>> >             and for IPv6 from the range 0:0:0:0:0:FFFF:7F00:0/104. And
>> >             lastly, the TTL to be set to 1 (no change here).
>> >

---

From: Dinesh Dutt <[email-masked]>
Date: Fri, 2 Aug 2019 10:44:34 -0700

Thanks for verifying this. On Linux and hardware routers that I'm aware of
(Cisco circa 2012 and Cumulus), the physical MAC address is reused across
the VNIs on the VTEP. Did you check on a non-VMW device? This is more for
my own curiosity.

To address the general case, can we not define a well-known (or reserve
one) unicast MAC address for use with VTEP? If the MAC address is
configurable in BFD command, this can be moot.

Dinesh

On Fri, Aug 2, 2019 at 10:27 AM Santosh P K <[email-masked]>
wrote:

>>    Thanks for your inputs. I checked implementation within Vmware.
>> Perhaps I should have been more clear about MAC address space while
>> checking internally. I will cross check again for the same and get back on
>> this list.
>>
>> Thanks
>> Santosh P K
>>

---

From: "Joel M. Halpern" <[email-masked]>
Date: Fri, 2 Aug 2019 14:03:31 -0400

If I am reading your various emails correctly Dinesh (and I may have 
missed something) you are trying to use the MAC address because you want 
to be able to send these BFD packets over arbitrary VNI to monitor the 
VNI.  That is not a requirement identified in the document.  It is not 
even a problem I understand, since all the VNI between an ingress and 
egress VTEP share fate.

Yours,
Joel

On 8/2/2019 1:44 PM, Dinesh Dutt wrote:
> Thanks for verifying this. On Linux and hardware routers that I'm aware 
> of (Cisco circa 2012 and Cumulus), the physical MAC address is reused 
> across the VNIs on the VTEP. Did you check on a non-VMW device? This is 
> more for my own curiosity.
> 
> On Fri, Aug 2, 2019 at 10:27 AM Santosh P K 
> <[email-masked] <mailto:[email-masked]>> wrote:
> 
>     I have cross checked point raised about MAC address usage. It is
>     possible that tenant could be using physical MAC address and when a
>     packet comes with valid VNI with a MAC address that is being used by
>     tenant then packet will be sent to that tenant. This rules out the
>     fact that we could use physical MAC address as inner MAC to ensure
>     packets get terminated at VTEP itself.
> 
>     Thanks
>     Santosh P K
> 
>     On Wed, Jul 31, 2019 at 11:00 AM Santosh P K
>     <[email-masked] <mailto:[email-masked]>>
>     wrote:
> 
>         Joel,
>             Thanks for your inputs. I checked implementation within
>         Vmware. Perhaps I should have been more clear about MAC address
>         space while checking internally. I will cross check again for
>         the same and get back on this list.
> 
>         Thanks
>         Santosh P K
> 
>         On Wed, Jul 31, 2019 at 10:54 AM Joel M. Halpern
>         <[email-masked] <mailto:[email-masked]>> wrote:
> 
>              > packet with valid VNI then lookup for MAC will happen and
>             it is VTEP own
>              > On Wed, Jul 31, 2019 at 9:20 AM Greg Mirsky
>             <[email-masked] <mailto:[email-masked]>
>              > <mailto:[email-masked]
>             <mailto:[email-masked]>>> wrote:
>              >
>              >     Hi Dinesh,
>              >     On Wed, Jul 31, 2019 at 12:03 PM Dinesh Dutt
>             <[email-masked] <mailto:[email-masked]>
>              >     <mailto:[email-masked] <mailto:[email-masked]>>>
>             wrote:
>              >
>              >         Hi Greg,
>              >
>              >         <[email-masked]
>             <mailto:[email-masked]> <mailto:[email-masked]
>             <mailto:[email-masked]>>> wrote:
>              >
>              >             Dear All,

---

From: "Joel M. Halpern" <[email-masked]>
Date: Fri, 2 Aug 2019 18:17:05 -0400

Your response seems to miss two points:

First, the problem you describe is not what the document says it is 
solving.  To the degree it discusses it at all, the document says "   In 
most cases, a single BFD session is sufficient for the given VTEP to 
monitor the reachability of a remote VTEP, regardless of the number of 
VNIs in common. "

Second, you assume the existence of an IP address for a VTEP within a 
VNI.  As with the MAC address, the VTEP does not have an IP address 
within the VNI.  Some implementations may have created such a thing, but 
the general construct, as defined to date, does not support such.

In short, you are requiring a behavior that violates the architectural 
structure of overlay / underlay separation, and common usage.  And you 
are doing so to support a use case that the working group has not 
indicated in the document as important.

Yours,
Joel

On 8/2/2019 5:01 PM, Dinesh Dutt wrote:
> Joel,
> 
> You understood correctly.
> 
> The VNIs may not share fate due to misconfiguration. And I strongly 
> suspect someone will want to use BFD for that because its about checking 
> path continuity as stated by the draft. As long as there's a valid IP 
> (because it's BFD) owned by the VTEP in that VNI, you can use BFD in 
> that VNI. Thats all that you need to dictate.  That IP address has a MAC 
> address and you can use that on the inner frame. That is all normal 
> VXLAN processing. The outer IP is always that of the VTEP.
> 
> Dinesh
> 
> On Fri, Aug 2, 2019 at 11:03 AM Joel M. Halpern <[email-masked] 
> <mailto:[email-masked]>> wrote:
> 
>      > <[email-masked]
>     <mailto:[email-masked]>
>     <mailto:[email-masked]
>     <mailto:[email-masked]>>> wrote:
>      >
>      >     <[email-masked]
>     <mailto:[email-masked]>
>     <mailto:[email-masked]
>     <mailto:[email-masked]>>>
>      >     wrote:
>      >
>      >         Joel,
>      >         <[email-masked] <mailto:[email-masked]>
>     <mailto:[email-masked] <mailto:[email-masked]>>> wrote:
>      >
>      >             have any control access to the VTEP, there is no
>     reason for
>      >             the VTEP to
>      >             have a MAC address in the tenant space.  Yes, the
>     device has
>      >             a physical
>      >              > I have checked with implementation in data path.
>     When we
>      >             receive a
>      >              > packet with valid VNI then lookup for MAC will
>     happen and
>      >             it is VTEP own
>      >              > MAC then it will be trapped to control plane for
>      >             processing. I think we
>      >              > can have following options
>      >              > 1. Optional managment VNI
>      >              > 2. Mandatory inner MAC set to VTEP mac
>      >              > 3. Inner IP TTL set to 1 to avoid forwarding of packet
>      >             via inner IP
>      >              > address.
>      >              >
>      >              >
>      >              > Thoughts?
>      >              >
>      >              > Thansk
>      >              > Santosh P K
>      >              >
>      >              > On Wed, Jul 31, 2019 at 9:20 AM Greg Mirsky
>     <mailto:[email-masked] <mailto:[email-masked]>>
>      >              > <mailto:[email-masked]
>     <mailto:[email-masked]>>>> wrote:
>      >              >
>      >              >     Hi Dinesh,
>      >              >     thank you for your consideration of the
>     proposal and
>      >             questions. What
>      >              >     would you see as the scope of testing the
>      >             connectivity for the
>      >              >     specific VNI? If it is tenant-to-tenant, then
>     VTEPs
>      >             will treat these
>      >              >     packets as regular user frames. More likely, these
>      >             could be Layer 2
>      >              >     OAM, e.g. CCM frames. The reason to use 127/8 for
>      >             IPv4, and
>      >              >     0:0:0:0:0:FFFF:7F00:0/104 for IPv6 is to safeguard
>      >             from leaking
>      >              >     Ethernet frames with BFD Control packet to a
>     tenant.
>      >              >     You've suggested using a MAC address to trap the
>      >             control packet at
>      >              >     VTEP. What that address could be? We had proposed
>      >             using the
>      >              >     dedicated MAC and VTEP's MAC and both raised
>     concerns
>      >             among VXLAN
>      >              >     experts. The idea of using Management VNI may
>     be more
>      >             acceptable
>      >              >     based on its similarity to the practice of using
>      >             Management VLAN.
>      >              >
>      >              >     Regards,
>      >              >     Greg
>      >              >
>      >              >     On Wed, Jul 31, 2019 at 12:03 PM Dinesh Dutt
>     <mailto:[email-masked] <mailto:[email-masked]>>
>      >              >     <mailto:[email-masked]
>     <mailto:[email-masked]> <mailto:[email-masked]
>     <mailto:[email-masked]>>>>
>      >             wrote:
>      >              >
>      >              >         Hi Greg,
>      >              >
>      >              >         As long as the inner MAC address is such
>     that the
>      >             packet is
>      >              >         trapped to the CPU, it should be fine for
>     use as
>      >             an inner MAC is
>      >              >         it not? Stating that is better than trying to
>      >             force a management
>      >              >         VNI. What if someone wants to test
>     connectivity
>      >             on a specific
>      >              >         VNI? I would not pick a loopback IP
>     address for
>      >             this since that
>      >              >         address range is host/node local only. Is
>     there a
>      >             reason you're
>      >              >         not using the VTEP IP as the inner IP
>     address ?
>      >              >
>      >              >         Dinesh
>      >              >
>      >              >         On Wed, Jul 31, 2019 at 5:48 AM Greg Mirsky
>      >              >         <[email-masked]
>     <mailto:[email-masked]>> <mailto:[email-masked]
>     <mailto:[email-masked]>>>> wrote:
>      >              >
>      >              >             Dear All,
>      >              >             thank you for your comments,
>     suggestions on
>      >             this issue,
>      >              >             probably the most challenging for this
>      >             specification. In the
>      >              >             course of our discussions, we've agreed to
>      >             abandon the
>      >              >             request to allocate the dedicated MAC
>     address
>      >             to be used as
>      >              >             the destination MAC address in the inner
>      >             Ethernet frame.
>      >              >             Also, earlier using VNI 0 was changed from
>      >             mandatory to one
>      >              >             of the options an implementation may
>     offer to
>      >             an operator.
>      >              >             The most recent discussion was whether
>     VTEP's
>      >             MAC address
>      >              >             might be used as the destination MAC
>     address
>      >             in the inner
>      >              >             Ethernet frame. As I recall it, the
>     comments
>      >             from VXLAN
>      >              >             experts equally split with one for it
>     and one
>      >             against. Hence
>      >              >             I would like to propose a new text to
>     resolve
>      >             the issue. The
>      >              >             idea is to let an operator select
>     Management
>      >             VNI and use
>      >              >             that VNI in VXLAN encapsulation of BFD
>      >             Control packets:
>      >              >             NEW TEXT:
>      >              >
>      >              >                 An operator MUST select a VNI
>     number to
>      >             be used as
>      >              >                 Management VNI. VXLAN packet for
>      >             Management VNI MUST NOT
>      >              >                 be sent to a tenant. VNI number 1 is
>      >             RECOMMENDED as the
>      >              >                 default for Management VNI.
>      >              >
>      >              >             With that new text, what can be the
>     value of
>      >             the destination
>      >              >             MAC in the inner Ethernet? I tend to
>     believe
>      >             that it can be
>      >              >             anything and ignored by the reciever VTEP.
>      >             Also, if the
>      >              >             trapping is based on VNI number, the
>      >             destination IP address
>      >              >             of the inner IP packet can from the range
>      >             127/8 for IPv4,
>      >              >             and for IPv6 from the range
>      >             0:0:0:0:0:FFFF:7F00:0/104. And
>      >              >             lastly, the TTL to be set to 1 (no
>     change here).
>      >              >
>      >              >             Much appreciate your comments,
>     questions, and
>      >             suggestions.
>      >              >
>      >              >             Best regards,
>      >              >             Greg
>      >              >
>      >
>

---

From: Dinesh Dutt <[email-masked]>
Date: Fri, 2 Aug 2019 14:01:21 -0700

Joel,

The VNIs may not share fate due to misconfiguration. And I strongly suspect
someone will want to use BFD for that because its about checking path
continuity as stated by the draft. As long as there's a valid IP (because
it's BFD) owned by the VTEP in that VNI, you can use BFD in that VNI. Thats
all that you need to dictate.  That IP address has a MAC address and you
can use that on the inner frame. That is all normal VXLAN processing. The
outer IP is always that of the VTEP.

Dinesh

---

From: Dinesh Dutt <[email-masked]>
Date: Fri, 2 Aug 2019 15:41:23 -0700

The assumption of an IP address within any VNI is suspect that way. What's
special about a single VNI, the management VNI? The VTEP IP address does
not belong in reality in any VNI.

Dinesh

On Fri, Aug 2, 2019 at 3:17 PM Joel M. Halpern <[email-masked]> wrote:

> >      >             The reason I ask is that as far as I can tell, since
> the
> >      >             tenant does not
> >      >             MAC address.  But the tenant could well be using that
> MAC
> >      >             address.  Yes,
> >      >             they would be violating the Ethernet spec.  But the
> whole
> >      >             point of
> >      >             Ethernet address that is part of the tenant space,
> well,
> >      >             they made up
> >      >             this particular game.
> >      >
> >      >             Yours,
> >      >             Joel
> >      >
> >      >              > I have checked with implementation in data path.
> >     When we
> >      >             receive a
> >      >              > 3. Inner IP TTL set to 1 to avoid forwarding of
> packet
> >      >             via inner IP
> >      >              > address.
> >      >              >
> >      >              >
> >      >              > Thoughts?
> >      >              >
> >      >              > Thansk
> >      >              > Santosh P K
> >      >              >
> >      >              >     thank you for your consideration of the
> >     proposal and
> >      >             questions. What
> >      >              >     specific VNI? If it is tenant-to-tenant, then
> >     VTEPs
> >      >             will treat these
> >      >              >     packets as regular user frames. More likely,
> these
> >      >             could be Layer 2
> >      >              >     OAM, e.g. CCM frames. The reason to use 127/8
> for
> >      >             IPv4, and
> >      >              >     0:0:0:0:0:FFFF:7F00:0/104 for IPv6 is to
> safeguard
> >      >             from leaking
> >      >              >     You've suggested using a MAC address to trap
> the
> >      >             control packet at
> >      >              >     VTEP. What that address could be? We had
> proposed
> >      >             using the
> >      >              >     dedicated MAC and VTEP's MAC and both raised
> >     concerns
> >      >             among VXLAN
> >      >              >     experts. The idea of using Management VNI may
> >     be more
> >      >             acceptable
> >      >              >     based on its similarity to the practice of
> using
> >      >             Management VLAN.
> >      >              >
> >      >              >     Regards,
> >      >              >     Greg
> >      >              >
> >      >              >     <mailto:[email-masked]
> >      >              >         As long as the inner MAC address is such
> >     that the
> >      >             packet is
> >      >              >         trapped to the CPU, it should be fine for
> >     use as
> >      >             an inner MAC is
> >      >              >         it not? Stating that is better than trying
> to
> >      >             force a management
> >      >              >         VNI. What if someone wants to test
> >     connectivity
> >      >             on a specific
> >      >              >         VNI? I would not pick a loopback IP
> >     address for
> >      >             this since that
> >      >              >         address range is host/node local only. Is
> >     there a
> >      >             reason you're
> >      >              >             thank you for your comments,
> >     suggestions on
> >      >             this issue,
> >      >              >             course of our discussions, we've
> agreed to
> >      >             abandon the
> >      >              >             request to allocate the dedicated MAC
> >     address
> >      >             to be used as
> >      >              >             the destination MAC address in the
> inner
> >      >             Ethernet frame.
> >      >              >             Also, earlier using VNI 0 was changed
> from
> >      >             mandatory to one
> >      >              >             of the options an implementation may
> >     offer to
> >      >             an operator.
> >      >              >             The most recent discussion was whether
> >     VTEP's
> >      >             MAC address
> >      >              >             might be used as the destination MAC
> >     address
> >      >             in the inner
> >      >              >             Ethernet frame. As I recall it, the
> >     comments
> >      >             from VXLAN
> >      >              >             experts equally split with one for it
> >     and one
> >      >             against. Hence
> >      >              >             I would like to propose a new text to
> >     resolve
> >      >             the issue. The
> >      >              >             idea is to let an operator select
> >     Management
> >      >             VNI and use
> >      >              >                 An operator MUST select a VNI
> >     number to
> >      >             be used as
> >      >              >                 be sent to a tenant. VNI number 1
> is
> >      >             RECOMMENDED as the
> >      >              >             With that new text, what can be the
> >     value of
> >      >             the destination
> >      >              >             MAC in the inner Ethernet? I tend to
> >     believe
> >      >             that it can be
> >      >              >             anything and ignored by the reciever
> VTEP.
> >      >             Also, if the
> >      >              >             of the inner IP packet can from the
> range
> >      >             127/8 for IPv4,
> >      >              >             Much appreciate your comments,
> >     questions, and
> >      >             suggestions.
> >      >              >
> >      >              >             Best regards,
> >      >              >             Greg
> >      >              >
> >      >
> >
>

---

From: "Joel M. Halpern" <[email-masked]>
Date: Fri, 2 Aug 2019 20:08:13 -0400

What is special about the management VNI is precisely that it is NOT a 
tenant VNI.  The VxLAN administration does know how it allocates VNI to 
tenants, and which VNI it has allocated.  In contrast, it does not know 
which IP addresses or MAC adddresses teh tenant is using or may plan to use.

Yours,
Joel

On 8/2/2019 6:41 PM, Dinesh Dutt wrote:
> The assumption of an IP address within any VNI is suspect that way. 
> What's special about a single VNI, the management VNI? The VTEP IP 
> address does not belong in reality in any VNI.
> 
> Dinesh
> 
> On Fri, Aug 2, 2019 at 3:17 PM Joel M. Halpern <[email-masked] 
>      > On Fri, Aug 2, 2019 at 11:03 AM Joel M. Halpern
>      >     missed something) you are trying to use the MAC address
>     because you
>      >     want
>      >      > Thanks for verifying this. On Linux and hardware routers
>     that I'm
>      >     aware
>      >      > of (Cisco circa 2012 and Cumulus), the physical MAC address is
>      >     reused
>      >      > across the VNIs on the VTEP. Did you check on a non-VMW
>     device?
>      >     This is
>      >      > more for my own curiosity.
>      >      >
>      >      > To address the general case, can we not define a
>     well-known (or
>      >     reserve
>      >      > one) unicast MAC address for use with VTEP? If the MAC
>     address is
>      >      > configurable in BFD command, this can be moot.
>      >      >
>      >      > Dinesh
>      >      >
>      >      > On Fri, Aug 2, 2019 at 10:27 AM Santosh P K
>      >      > <[email-masked]
>     <mailto:[email-masked]>>
>     <mailto:[email-masked]>>>> wrote:
>      >      >
>      >      >     I have cross checked point raised about MAC address
>     usage. It is
>      >      >     possible that tenant could be using physical MAC
>     address and
>      >     when a
>      >      >     packet comes with valid VNI with a MAC address that is
>     being
>      >     used by
>      >      >     tenant then packet will be sent to that tenant. This rules
>      >     out the
>      >      >     fact that we could use physical MAC address as inner
>     MAC to
>      >     ensure
>      >      >     packets get terminated at VTEP itself.
>      >      >
>      >      >     Thanks
>      >      >     Santosh P K
>      >      >
>      >      >     On Wed, Jul 31, 2019 at 11:00 AM Santosh P K
>      >      >     <[email-masked]
>     <mailto:[email-masked]>>
>     <mailto:[email-masked]>>>>
>      >      >     wrote:
>      >      >
>      >      >         Joel,
>      >      >             Thanks for your inputs. I checked
>     implementation within
>      >      >         Vmware. Perhaps I should have been more clear
>     about MAC
>      >     address
>      >      >         space while checking internally. I will cross
>     check again for
>      >      >         the same and get back on this list.
>      >      >
>      >      >         Thanks
>      >      >         Santosh P K
>      >      >
>      >      >         On Wed, Jul 31, 2019 at 10:54 AM Joel M. Halpern
>      >      >         <[email-masked] <mailto:[email-masked]>
>     <mailto:[email-masked] <mailto:[email-masked]>>
>      >     <mailto:[email-masked] <mailto:[email-masked]>
>     <mailto:[email-masked] <mailto:[email-masked]>>>> wrote:
>      >      >
>      >      >             Sorry to ask a stupid question.  Whose
>     implementation?
>      >      >
>      >      >             The reason I ask is that as far as I can tell,
>     since the
>      >      >             tenant does not
>      >      >             have any control access to the VTEP, there is no
>      >     reason for
>      >      >             the VTEP to
>      >      >             have a MAC address in the tenant space.  Yes, the
>      >     device has
>      >      >             a physical
>      >      >             MAC address.  But the tenant could well be
>     using that MAC
>      >      >             address.  Yes,
>      >      >             they would be violating the Ethernet spec. 
>     But the whole
>      >      >             point of
>      >      >             segregation is not to care about such issues.
>      >      >
>      >      >             On the other hand, if you tell me that the VMWare
>      >      >             implementation has an
>      >      >             Ethernet address that is part of the tenant
>     space, well,
>      >      >             they made up
>      >      >             this particular game.
>      >      >
>      >      >             Yours,
>      >      >             Joel
>      >      >
>      >      >             On 7/31/2019 1:44 PM, Santosh P K wrote:
>      >      >              > I have checked with implementation in data
>     path.
>      >     When we
>      >      >             receive a
>      >      >              > packet with valid VNI then lookup for MAC will
>      >     happen and
>      >      >             it is VTEP own
>      >      >              > MAC then it will be trapped to control
>     plane for
>      >      >             processing. I think we
>      >      >              > can have following options
>      >      >              > 1. Optional managment VNI
>      >      >              > 2. Mandatory inner MAC set to VTEP mac
>      >      >              > 3. Inner IP TTL set to 1 to avoid
>     forwarding of packet
>      >      >             via inner IP
>      >      >              > address.
>      >      >              >
>      >      >              >
>      >      >              > Thoughts?
>      >      >              >
>      >      >              > Thansk
>      >      >              > Santosh P K
>      >      >              >
>      >      >              > On Wed, Jul 31, 2019 at 9:20 AM Greg Mirsky
>      >      >             <[email-masked]
>     <mailto:[email-masked]>>
>      >     <mailto:[email-masked] <mailto:[email-masked]>
>     <mailto:[email-masked] <mailto:[email-masked]>>>
>      >      >              > <mailto:[email-masked]
>     <mailto:[email-masked]>
>     <mailto:[email-masked]>>>>> wrote:
>      >      >              >
>      >      >              >     Hi Dinesh,
>      >      >              >     thank you for your consideration of the
>      >     proposal and
>      >      >             questions. What
>      >      >              >     would you see as the scope of testing the
>      >      >             connectivity for the
>      >      >              >     specific VNI? If it is
>     tenant-to-tenant, then
>      >     VTEPs
>      >      >             will treat these
>      >      >              >     packets as regular user frames. More
>     likely, these
>      >      >             could be Layer 2
>      >      >              >     OAM, e.g. CCM frames. The reason to use
>     127/8 for
>      >      >             IPv4, and
>      >      >              >     0:0:0:0:0:FFFF:7F00:0/104 for IPv6 is
>     to safeguard
>      >      >             from leaking
>      >      >              >     Ethernet frames with BFD Control packet
>     to a
>      >     tenant.
>      >      >              >     You've suggested using a MAC address to
>     trap the
>      >      >             control packet at
>      >      >              >     VTEP. What that address could be? We
>     had proposed
>      >      >             using the
>      >      >              >     dedicated MAC and VTEP's MAC and both
>     raised
>      >     concerns
>      >      >             among VXLAN
>      >      >              >     experts. The idea of using Management
>     VNI may
>      >     be more
>      >      >             acceptable
>      >      >              >     based on its similarity to the practice
>     of using
>      >      >             Management VLAN.
>      >      >              >
>      >      >              >     Regards,
>      >      >              >     Greg
>      >      >              >
>      >      >              >     On Wed, Jul 31, 2019 at 12:03 PM Dinesh
>     Dutt
>      >      >             <[email-masked] <mailto:[email-masked]>
>      >     <mailto:[email-masked] <mailto:[email-masked]>
>      >      >              >     <mailto:[email-masked]
>     <mailto:[email-masked]>
>     <mailto:[email-masked] <mailto:[email-masked]>
>      >     <mailto:[email-masked] <mailto:[email-masked]>>>>>
>      >      >             wrote:
>      >      >              >
>      >      >              >         Hi Greg,
>      >      >              >
>      >      >              >         As long as the inner MAC address is
>     such
>      >     that the
>      >      >             packet is
>      >      >              >         trapped to the CPU, it should be
>     fine for
>      >     use as
>      >      >             an inner MAC is
>      >      >              >         it not? Stating that is better than
>     trying to
>      >      >             force a management
>      >      >              >         VNI. What if someone wants to test
>      >     connectivity
>      >      >             on a specific
>      >      >              >         VNI? I would not pick a loopback IP
>      >     address for
>      >      >             this since that
>      >      >              >         address range is host/node local
>     only. Is
>      >     there a
>      >      >             reason you're
>      >      >              >         not using the VTEP IP as the inner IP
>      >     address ?
>      >      >              >
>      >      >              >         Dinesh
>      >      >              >
>      >      >              >         On Wed, Jul 31, 2019 at 5:48 AM
>     Greg Mirsky
>      >      >              >         <[email-masked]
>     <mailto:[email-masked]>
>     <mailto:[email-masked]>>> <mailto:[email-masked]
>     <mailto:[email-masked]>
>     <mailto:[email-masked]>>>>> wrote:
>      >      >              >
>      >      >              >             Dear All,
>      >      >              >             thank you for your comments,
>      >     suggestions on
>      >      >             this issue,
>      >      >              >             probably the most challenging
>     for this
>      >      >             specification. In the
>      >      >              >             course of our discussions,
>     we've agreed to
>      >      >             abandon the
>      >      >              >             request to allocate the
>     dedicated MAC
>      >     address
>      >      >             to be used as
>      >      >              >             the destination MAC address in
>     the inner
>      >      >             Ethernet frame.
>      >      >              >             Also, earlier using VNI 0 was
>     changed from
>      >      >             mandatory to one
>      >      >              >             of the options an
>     implementation may
>      >     offer to
>      >      >             an operator.
>      >      >              >             The most recent discussion was
>     whether
>      >     VTEP's
>      >      >             MAC address
>      >      >              >             might be used as the
>     destination MAC
>      >     address
>      >      >             in the inner
>      >      >              >             Ethernet frame. As I recall it, the
>      >     comments
>      >      >             from VXLAN
>      >      >              >             experts equally split with one
>     for it
>      >     and one
>      >      >             against. Hence
>      >      >              >             I would like to propose a new
>     text to
>      >     resolve
>      >      >             the issue. The
>      >      >              >             idea is to let an operator select
>      >     Management
>      >      >             VNI and use
>      >      >              >             that VNI in VXLAN encapsulation
>     of BFD
>      >      >             Control packets:
>      >      >              >             NEW TEXT:
>      >      >              >
>      >      >              >                 An operator MUST select a VNI
>      >     number to
>      >      >             be used as
>      >      >              >                 Management VNI. VXLAN
>     packet for
>      >      >             Management VNI MUST NOT
>      >      >              >                 be sent to a tenant. VNI
>     number 1 is
>      >      >             RECOMMENDED as the
>      >      >              >                 default for Management VNI.
>      >      >              >
>      >      >              >             With that new text, what can be the
>      >     value of
>      >      >             the destination
>      >      >              >             MAC in the inner Ethernet? I
>     tend to
>      >     believe
>      >      >             that it can be
>      >      >              >             anything and ignored by the
>     reciever VTEP.
>      >      >             Also, if the
>      >      >              >             trapping is based on VNI
>     number, the
>      >      >             destination IP address
>      >      >              >             of the inner IP packet can from
>     the range
>      >      >             127/8 for IPv4,
>      >      >              >             and for IPv6 from the range
>      >      >             0:0:0:0:0:FFFF:7F00:0/104. And
>      >      >              >             lastly, the TTL to be set to 1 (no
>      >     change here).
>      >      >              >
>      >      >              >             Much appreciate your comments,
>      >     questions, and
>      >      >             suggestions.
>      >      >              >
>      >      >              >             Best regards,
>      >      >              >             Greg
>      >      >              >
>      >      >
>      >
>

---

From: "Joel M. Halpern" <[email-masked]>
Date: Fri, 2 Aug 2019 20:38:45 -0400

I am going by what the draft says its purpose is.  If you (Dinesh) want 
the draft to fulfill a different purpose, then either ask the chairs to 
take this draft back to the WG, or write a separate draft.
As currently written, the behavior Greg proposed meets the needs, and 
does so in a way that is consistent with VxLAN.

Yours,
Joel

On 8/2/2019 8:30 PM, Dinesh Dutt wrote:
> What is the stated purpose of this BFD session? The VTEP reachability is 
> determined by the underlay, I don't need VXLAN-encaped packet for that. 
> Do we agree?
> 
> If I want to test the VXLAN encap/decap functionality alone, picking any 
> single VNI maybe fine. But is this all any network operator wants? Why? 
> In what situations has this been a problem? I suspect operators also 
> want to verify path continuity over a specific VNI. If you say this is 
> not defined by the document, I disagree because the current version 
> talks about controlling the number of BFD sessions between the VTEPs 
> (see section 3). More importantly, this is a real problem that operators 
> like to verify.
> 
> Dinesh
> 
> On Fri, Aug 2, 2019 at 5:08 PM Joel M. Halpern <[email-masked] 
>      > On Fri, Aug 2, 2019 at 3:17 PM Joel M. Halpern
>      >     solving.  To the degree it discusses it at all, the document
>     says "
>      >       In
>      >     within the VNI.  Some implementations may have created such a
>     thing,
>      >     but
>      >      > You understood correctly.
>      >      >
>      >      > The VNIs may not share fate due to misconfiguration. And I
>     strongly
>      >      > suspect someone will want to use BFD for that because its
>     about
>      >     checking
>      >      > path continuity as stated by the draft. As long as there's a
>      >     valid IP
>      >      > (because it's BFD) owned by the VTEP in that VNI, you can
>     use BFD in
>      >      > that VNI. Thats all that you need to dictate.  That IP address
>      >     has a MAC
>      >      > address and you can use that on the inner frame. That is
>     all normal
>      >      > VXLAN processing. The outer IP is always that of the VTEP.
>      >      >
>      >      > Dinesh
>      >      >
>      >      > On Fri, Aug 2, 2019 at 11:03 AM Joel M. Halpern
>      >      > <mailto:[email-masked] <mailto:[email-masked]>
>      >      >     If I am reading your various emails correctly Dinesh
>     (and I
>      >     may have
>      >      >     missed something) you are trying to use the MAC address
>      >     because you
>      >      >     want
>      >      >     to be able to send these BFD packets over arbitrary VNI to
>      >     monitor the
>      >      >     VNI.  That is not a requirement identified in the
>     document.
>      >     It is not
>      >      >     even a problem I understand, since all the VNI between an
>      >     ingress and
>      >      >     egress VTEP share fate.
>      >      >
>      >      >     Yours,
>      >      >     Joel
>      >      >
>      >      >     On 8/2/2019 1:44 PM, Dinesh Dutt wrote:
>      >      >      > Thanks for verifying this. On Linux and hardware
>     routers
>      >     that I'm
>      >      >     aware
>      >      >      > of (Cisco circa 2012 and Cumulus), the physical MAC
>     address is
>      >      >     reused
>      >      >      > across the VNIs on the VTEP. Did you check on a non-VMW
>      >     device?
>      >      >     This is
>      >      >      > more for my own curiosity.
>      >      >      >
>      >      >      > To address the general case, can we not define a
>      >     well-known (or
>      >      >     reserve
>      >      >      > one) unicast MAC address for use with VTEP? If the MAC
>      >     address is
>      >      >      > configurable in BFD command, this can be moot.
>      >      >      >
>      >      >      > Dinesh
>      >      >      >
>      >      >      > On Fri, Aug 2, 2019 at 10:27 AM Santosh P K
>      >      >      > <[email-masked]
>      >      >     <mailto:[email-masked]
>      >      >     <mailto:[email-masked]
>      >      >     <mailto:[email-masked]
>     <mailto:[email-masked]>>>>> wrote:
>      >      >      >
>      >      >      >     I have cross checked point raised about MAC address
>      >     usage. It is
>      >      >      >     possible that tenant could be using physical MAC
>      >     address and
>      >      >     when a
>      >      >      >     packet comes with valid VNI with a MAC address
>     that is
>      >     being
>      >      >     used by
>      >      >      >     tenant then packet will be sent to that tenant.
>     This rules
>      >      >     out the
>      >      >      >     fact that we could use physical MAC address as
>     inner
>      >     MAC to
>      >      >     ensure
>      >      >      >     packets get terminated at VTEP itself.
>      >      >      >
>      >      >      >     Thanks
>      >      >      >     Santosh P K
>      >      >      >
>      >      >      >     On Wed, Jul 31, 2019 at 11:00 AM Santosh P K
>      >      >      >     <[email-masked]
>      >      >     <mailto:[email-masked]
>      >      >     <mailto:[email-masked]
>      >      >     <mailto:[email-masked]
>     <mailto:[email-masked]>>>>>
>      >      >      >     wrote:
>      >      >      >
>      >      >      >         Joel,
>      >      >      >             Thanks for your inputs. I checked
>      >     implementation within
>      >      >      >         Vmware. Perhaps I should have been more clear
>      >     about MAC
>      >      >     address
>      >      >      >         space while checking internally. I will cross
>      >     check again for
>      >      >      >         the same and get back on this list.
>      >      >      >
>      >      >      >         Thanks
>      >      >      >         Santosh P K
>      >      >      >
>      >      >      >         On Wed, Jul 31, 2019 at 10:54 AM Joel M.
>     Halpern
>      >      >      >         <[email-masked]
>     <mailto:[email-masked]> <mailto:[email-masked]
>     <mailto:[email-masked]>>
>     <mailto:[email-masked] <mailto:[email-masked]>>>
>      >      >     <mailto:[email-masked]
>     <mailto:[email-masked]> <mailto:[email-masked]
>     <mailto:[email-masked]>>
>     <mailto:[email-masked] <mailto:[email-masked]>>>>> wrote:
>      >      >      >
>      >      >      >             Sorry to ask a stupid question.  Whose
>      >     implementation?
>      >      >      >
>      >      >      >             The reason I ask is that as far as I
>     can tell,
>      >     since the
>      >      >      >             tenant does not
>      >      >      >             have any control access to the VTEP,
>     there is no
>      >      >     reason for
>      >      >      >             the VTEP to
>      >      >      >             have a MAC address in the tenant
>     space.  Yes, the
>      >      >     device has
>      >      >      >             a physical
>      >      >      >             MAC address.  But the tenant could well be
>      >     using that MAC
>      >      >      >             address.  Yes,
>      >      >      >             they would be violating the Ethernet spec.
>      >     But the whole
>      >      >      >             point of
>      >      >      >             segregation is not to care about such
>     issues.
>      >      >      >
>      >      >      >             On the other hand, if you tell me that
>     the VMWare
>      >      >      >             implementation has an
>      >      >      >             Ethernet address that is part of the tenant
>      >     space, well,
>      >      >      >             they made up
>      >      >      >             this particular game.
>      >      >      >
>      >      >      >             Yours,
>      >      >      >             Joel
>      >      >      >
>      >      >      >             On 7/31/2019 1:44 PM, Santosh P K wrote:
>      >      >      >              > I have checked with implementation
>     in data
>      >     path.
>      >      >     When we
>      >      >      >             receive a
>      >      >      >              > packet with valid VNI then lookup
>     for MAC will
>      >      >     happen and
>      >      >      >             it is VTEP own
>      >      >      >              > MAC then it will be trapped to control
>      >     plane for
>      >      >      >             processing. I think we
>      >      >      >              > can have following options
>      >      >      >              > 1. Optional managment VNI
>      >      >      >              > 2. Mandatory inner MAC set to VTEP mac
>      >      >      >              > 3. Inner IP TTL set to 1 to avoid
>      >     forwarding of packet
>      >      >      >             via inner IP
>      >      >      >              > address.
>      >      >      >              >
>      >      >      >              >
>      >      >      >              > Thoughts?
>      >      >      >              >
>      >      >      >              > Thansk
>      >      >      >              > Santosh P K
>      >      >      >              >
>      >      >      >              > On Wed, Jul 31, 2019 at 9:20 AM Greg
>     Mirsky
>     <mailto:[email-masked] <mailto:[email-masked]>>>>
>      >      >      >              > <mailto:[email-masked]
>     <mailto:[email-masked]>>>
>     <mailto:[email-masked]>>>>>> wrote:
>      >      >      >              >
>      >      >      >              >     Hi Dinesh,
>      >      >      >              >     thank you for your consideration
>     of the
>      >      >     proposal and
>      >      >      >             questions. What
>      >      >      >              >     would you see as the scope of
>     testing the
>      >      >      >             connectivity for the
>      >      >      >              >     specific VNI? If it is
>      >     tenant-to-tenant, then
>      >      >     VTEPs
>      >      >      >             will treat these
>      >      >      >              >     packets as regular user frames. More
>      >     likely, these
>      >      >      >             could be Layer 2
>      >      >      >              >     OAM, e.g. CCM frames. The reason
>     to use
>      >     127/8 for
>      >      >      >             IPv4, and
>      >      >      >              >     0:0:0:0:0:FFFF:7F00:0/104 for
>     IPv6 is
>      >     to safeguard
>      >      >      >             from leaking
>      >      >      >              >     Ethernet frames with BFD Control
>     packet
>      >     to a
>      >      >     tenant.
>      >      >      >              >     You've suggested using a MAC
>     address to
>      >     trap the
>      >      >      >             control packet at
>      >      >      >              >     VTEP. What that address could be? We
>      >     had proposed
>      >      >      >             using the
>      >      >      >              >     dedicated MAC and VTEP's MAC and
>     both
>      >     raised
>      >      >     concerns
>      >      >      >             among VXLAN
>      >      >      >              >     experts. The idea of using
>     Management
>      >     VNI may
>      >      >     be more
>      >      >      >             acceptable
>      >      >      >              >     based on its similarity to the
>     practice
>      >     of using
>      >      >      >             Management VLAN.
>      >      >      >              >
>      >      >      >              >     Regards,
>      >      >      >              >     Greg
>      >      >      >              >
>      >      >      >              >     On Wed, Jul 31, 2019 at 12:03 PM
>     Dinesh
>      >     Dutt
>      >      >      >             <[email-masked]
>     <mailto:[email-masked]>>
>      >      >     <mailto:[email-masked] <mailto:[email-masked]>
>     <mailto:[email-masked] <mailto:[email-masked]>>>>
>      >      >      >              >     <mailto:[email-masked]
>      >      >     <mailto:[email-masked] <mailto:[email-masked]>
>      >      >     <mailto:[email-masked] <mailto:[email-masked]>
>     <mailto:[email-masked] <mailto:[email-masked]>>>>>>
>      >      >      >             wrote:
>      >      >      >              >
>      >      >      >              >         Hi Greg,
>      >      >      >              >
>      >      >      >              >         As long as the inner MAC
>     address is
>      >     such
>      >      >     that the
>      >      >      >             packet is
>      >      >      >              >         trapped to the CPU, it should be
>      >     fine for
>      >      >     use as
>      >      >      >             an inner MAC is
>      >      >      >              >         it not? Stating that is
>     better than
>      >     trying to
>      >      >      >             force a management
>      >      >      >              >         VNI. What if someone wants
>     to test
>      >      >     connectivity
>      >      >      >             on a specific
>      >      >      >              >         VNI? I would not pick a
>     loopback IP
>      >      >     address for
>      >      >      >             this since that
>      >      >      >              >         address range is host/node local
>      >     only. Is
>      >      >     there a
>      >      >      >             reason you're
>      >      >      >              >         not using the VTEP IP as the
>     inner IP
>      >      >     address ?
>      >      >      >              >
>      >      >      >              >         Dinesh
>      >      >      >              >
>      >      >      >              >         On Wed, Jul 31, 2019 at 5:48 AM
>      >     Greg Mirsky
>      >      >      >              >         <[email-masked]
>     <mailto:[email-masked]>>>
>     <mailto:[email-masked]>>>> <mailto:[email-masked]
>     <mailto:[email-masked]>>>
>     <mailto:[email-masked]>>>>>> wrote:
>      >      >      >              >
>      >      >      >              >             Dear All,
>      >      >      >              >             thank you for your comments,
>      >      >     suggestions on
>      >      >      >             this issue,
>      >      >      >              >             probably the most
>     challenging
>      >     for this
>      >      >      >             specification. In the
>      >      >      >              >             course of our discussions,
>      >     we've agreed to
>      >      >      >             abandon the
>      >      >      >              >             request to allocate the
>      >     dedicated MAC
>      >      >     address
>      >      >      >             to be used as
>      >      >      >              >             the destination MAC
>     address in
>      >     the inner
>      >      >      >             Ethernet frame.
>      >      >      >              >             Also, earlier using VNI
>     0 was
>      >     changed from
>      >      >      >             mandatory to one
>      >      >      >              >             of the options an
>      >     implementation may
>      >      >     offer to
>      >      >      >             an operator.
>      >      >      >              >             The most recent
>     discussion was
>      >     whether
>      >      >     VTEP's
>      >      >      >             MAC address
>      >      >      >              >             might be used as the
>      >     destination MAC
>      >      >     address
>      >      >      >             in the inner
>      >      >      >              >             Ethernet frame. As I
>     recall it, the
>      >      >     comments
>      >      >      >             from VXLAN
>      >      >      >              >             experts equally split
>     with one
>      >     for it
>      >      >     and one
>      >      >      >             against. Hence
>      >      >      >              >             I would like to propose
>     a new
>      >     text to
>      >      >     resolve
>      >      >      >             the issue. The
>      >      >      >              >             idea is to let an
>     operator select
>      >      >     Management
>      >      >      >             VNI and use
>      >      >      >              >             that VNI in VXLAN
>     encapsulation
>      >     of BFD
>      >      >      >             Control packets:
>      >      >      >              >             NEW TEXT:
>      >      >      >              >
>      >      >      >              >                 An operator MUST
>     select a VNI
>      >      >     number to
>      >      >      >             be used as
>      >      >      >              >                 Management VNI. VXLAN
>      >     packet for
>      >      >      >             Management VNI MUST NOT
>      >      >      >              >                 be sent to a tenant. VNI
>      >     number 1 is
>      >      >      >             RECOMMENDED as the
>      >      >      >              >                 default for
>     Management VNI.
>      >      >      >              >
>      >      >      >              >             With that new text, what
>     can be the
>      >      >     value of
>      >      >      >             the destination
>      >      >      >              >             MAC in the inner Ethernet? I
>      >     tend to
>      >      >     believe
>      >      >      >             that it can be
>      >      >      >              >             anything and ignored by the
>      >     reciever VTEP.
>      >      >      >             Also, if the
>      >      >      >              >             trapping is based on VNI
>      >     number, the
>      >      >      >             destination IP address
>      >      >      >              >             of the inner IP packet
>     can from
>      >     the range
>      >      >      >             127/8 for IPv4,
>      >      >      >              >             and for IPv6 from the range
>      >      >      >             0:0:0:0:0:FFFF:7F00:0/104. And
>      >      >      >              >             lastly, the TTL to be
>     set to 1 (no
>      >      >     change here).
>      >      >      >              >
>      >      >      >              >             Much appreciate your
>     comments,
>      >      >     questions, and
>      >      >      >             suggestions.
>      >      >      >              >
>      >      >      >              >             Best regards,
>      >      >      >              >             Greg
>      >      >      >              >
>      >      >      >
>      >      >
>      >
>

---

From: Dinesh Dutt <[email-masked]>
Date: Fri, 2 Aug 2019 17:30:27 -0700

What is the stated purpose of this BFD session? The VTEP reachability is
determined by the underlay, I don't need VXLAN-encaped packet for that. Do
we agree?

If I want to test the VXLAN encap/decap functionality alone, picking any
single VNI maybe fine. But is this all any network operator wants? Why? In
what situations has this been a problem? I suspect operators also want to
verify path continuity over a specific VNI. If you say this is not defined
by the document, I disagree because the current version talks about
controlling the number of BFD sessions between the VTEPs (see section 3).
More importantly, this is a real problem that operators like to verify.

Dinesh

On Fri, Aug 2, 2019 at 5:08 PM Joel M. Halpern <[email-masked]> wrote:

> >      >      > Thanks for verifying this. On Linux and hardware routers
> >     that I'm
> >      >     aware
> >      >      > of (Cisco circa 2012 and Cumulus), the physical MAC
> address is
> >      >     reused
> >      >      > across the VNIs on the VTEP. Did you check on a non-VMW
> >     device?
> >      >     This is
> >      >      > To address the general case, can we not define a
> >     well-known (or
> >      >     reserve
> >      >      >     possible that tenant could be using physical MAC
> >     address and
> >      >     when a
> >      >      >     packet comes with valid VNI with a MAC address that is
> >     being
> >      >     used by
> >      >      >     tenant then packet will be sent to that tenant. This
> rules
> >      >     out the
> >      >      >     fact that we could use physical MAC address as inner
> >     MAC to
> >      >     ensure
> >      >      >         Vmware. Perhaps I should have been more clear
> >     about MAC
> >      >     address
> >      >      >             The reason I ask is that as far as I can tell,
> >     since the
> >      >      >             tenant does not
> >      >      >             MAC address.  But the tenant could well be
> >     using that MAC
> >      >      >             address.  Yes,
> >      >      >             they would be violating the Ethernet spec.
> >     But the whole
> >      >      >             point of
> >      >      >             On the other hand, if you tell me that the
> VMWare
> >      >      >             implementation has an
> >      >      >             Ethernet address that is part of the tenant
> >     space, well,
> >      >      >             they made up
> >      >      >             this particular game.
> >      >      >
> >      >      >             Yours,
> >      >      >             Joel
> >      >      >
> >      >      >              > packet with valid VNI then lookup for MAC
> will
> >      >     happen and
> >      >      >             it is VTEP own
> >      >      >              > MAC then it will be trapped to control
> >     plane for
> >      >      >             processing. I think we
> >      >      >              > 3. Inner IP TTL set to 1 to avoid
> >     forwarding of packet
> >      >      >             via inner IP
> >      >      >              > address.
> >      >      >              >
> >      >      >              >
> >      >      >              > Thoughts?
> >      >      >              >
> >      >      >              > Thansk
> >      >      >              > Santosh P K
> >      >      >              >
> >      >      >              >     would you see as the scope of testing
> the
> >      >      >             connectivity for the
> >      >      >              >     specific VNI? If it is
> >     tenant-to-tenant, then
> >      >     VTEPs
> >      >      >             will treat these
> >      >      >              >     packets as regular user frames. More
> >     likely, these
> >      >      >             could be Layer 2
> >      >      >              >     OAM, e.g. CCM frames. The reason to use
> >     127/8 for
> >      >      >             IPv4, and
> >      >      >              >     0:0:0:0:0:FFFF:7F00:0/104 for IPv6 is
> >     to safeguard
> >      >      >             from leaking
> >      >      >              >     Ethernet frames with BFD Control packet
> >     to a
> >      >     tenant.
> >      >      >              >     You've suggested using a MAC address to
> >     trap the
> >      >      >             control packet at
> >      >      >              >     VTEP. What that address could be? We
> >     had proposed
> >      >      >             using the
> >      >      >              >     based on its similarity to the practice
> >     of using
> >      >      >             Management VLAN.
> >      >      >              >
> >      >      >              >     Regards,
> >      >      >              >     Greg
> >      >      >              >
> >      >      >              >         it not? Stating that is better than
> >     trying to
> >      >      >             force a management
> >      >      >              >         not using the VTEP IP as the inner
> IP
> >      >     address ?
> >      >      >              >
> >      >      >              >         Dinesh
> >      >      >              >
> >      >      >              >             probably the most challenging
> >     for this
> >      >      >             specification. In the
> >      >      >              >             course of our discussions,
> >     we've agreed to
> >      >      >             abandon the
> >      >      >              >             request to allocate the
> >     dedicated MAC
> >      >     address
> >      >      >             to be used as
> >      >      >              >             the destination MAC address in
> >     the inner
> >      >      >             Ethernet frame.
> >      >      >              >             Also, earlier using VNI 0 was
> >     changed from
> >      >      >             mandatory to one
> >      >      >              >             of the options an
> >     implementation may
> >      >     offer to
> >      >      >             an operator.
> >      >      >              >             that VNI in VXLAN encapsulation
> >     of BFD
> >      >      >             Control packets:
> >      >      >              >             NEW TEXT:
> >      >      >              >
> >      >      >              >                 Management VNI. VXLAN
> >      >      >              >                 be sent to a tenant. VNI
> >      >      >              >             anything and ignored by the
> >     reciever VTEP.
> >      >      >             Also, if the
> >      >      >              >             trapping is based on VNI
> >     number, the
> >      >      >             destination IP address
> >      >      >              >             of the inner IP packet can from
> >     the range
> >      >      >             127/8 for IPv4,
> >      >      >              >             lastly, the TTL to be set to 1
> (no
> >      >     change here).
> >      >      >              >

---

From: Dinesh Dutt <[email-masked]>
Date: Fri, 2 Aug 2019 17:41:40 -0700

The abstract reads this: "

This document describes the use of the Bidirectional Forwarding
   Detection (BFD) protocol in point-to-point Virtual eXtensible Local
   Area Network (VXLAN) tunnels forming up an overlay network."

How do you infer what you said?

Dinesh

On Fri, Aug 2, 2019 at 5:38 PM Joel M. Halpern <[email-masked]> wrote:

> >      >      > suspect someone will want to use BFD for that because its
> >     about
> >      >     checking
> >      >      > path continuity as stated by the draft. As long as there's
> a
> >      >     valid IP
> >      >      > that VNI. Thats all that you need to dictate.  That IP
> address
> >      >     has a MAC
> >      >      >     If I am reading your various emails correctly Dinesh
> >     (and I
> >      >     may have
> >      >      >     to be able to send these BFD packets over arbitrary
> VNI to
> >      >     monitor the
> >      >      >     VNI.  That is not a requirement identified in the
> >     document.
> >      >     It is not
> >      >      >     even a problem I understand, since all the VNI between
> an
> >      >     ingress and
> >      >      >     egress VTEP share fate.
> >      >      >
> >      >      >     Yours,
> >      >      >     Joel
> >      >      >
> >      >      >      > one) unicast MAC address for use with VTEP? If the
> MAC
> >      >     address is
> >      >      >      >     I have cross checked point raised about MAC
> address
> >      >     usage. It is
> >      >      >      >     tenant then packet will be sent to that tenant.
> >     This rules
> >      >      >     out the
> >      >      >      >         space while checking internally. I will
> cross
> >      >     check again for
> >      >      >      >         <[email-masked]
> >      >      >     <mailto:[email-masked]
> >      >      >      >             On the other hand, if you tell me that
> >     the VMWare
> >      >      >      >             implementation has an
> >      >      >      >              > I have checked with implementation
> >     in data
> >      >     path.
> >      >      >     When we
> >      >      >      >             receive a
> >      >      >      >              > packet with valid VNI then lookup
> >     for MAC will
> >      >      >     happen and
> >      >      >      >             it is VTEP own
> >      >     <mailto:[email-masked] <mailto:[email-masked]
> >>>
> >      >      >      >              >     would you see as the scope of
> >     testing the
> >      >      >      >             connectivity for the
> >      >      >      >              >     specific VNI? If it is
> >      >     tenant-to-tenant, then
> >      >      >     VTEPs
> >      >      >      >             will treat these
> >      >      >      >              >     dedicated MAC and VTEP's MAC and
> >     both
> >      >     raised
> >      >      >     concerns
> >      >      >      >             among VXLAN
> >      >      >      >              >     experts. The idea of using
> >     Management
> >      >     VNI may
> >      >      >     be more
> >      >      >      >             acceptable
> >      >      >      >              >     On Wed, Jul 31, 2019 at 12:03 PM
> >     Dinesh
> >      >     Dutt
> >      >      >      >             <[email-masked]
> >      >      >      >              >         As long as the inner MAC
> >      >      >      >              >         trapped to the CPU, it
> should be
> >      >      >      >              >         address range is host/node
> local
> >      >     only. Is
> >      >      >     there a
> >      >      >      >             reason you're
> >      >      >      >              >         not using the VTEP IP as the
> >     inner IP
> >      >      >     address ?
> >      >      >      >              >
> >      >      >      >              >         Dinesh
> >      >      >      >              >
> >      >      >      >              >         On Wed, Jul 31, 2019 at 5:48
> AM
> >      >     Greg Mirsky
> >      >      >      >              >         <[email-masked]
> >      >      >      >              >             The most recent
> >      >      >      >              >             experts equally split
> >      >      >      >              >             I would like to propose
> >      >      >      >              >                 be sent to a tenant.
> VNI
> >      >      >      >              >             MAC in the inner
> Ethernet? I
> >      >      >      >              >             and for IPv6 from the
> >      >      >      >              >             lastly, the TTL to be

---

From: Dinesh Dutt <[email-masked]>
Date: Fri, 2 Aug 2019 17:43:38 -0700

What I mean is "How do you infer that it excludes the case I'm talking
about?".

Dinesh

On Fri, Aug 2, 2019 at 5:41 PM Dinesh Dutt <[email-masked]> wrote:

>> >      >      > path continuity as stated by the draft. As long as
>> there's a
>> >      >     valid IP
>> >      >      >     even a problem I understand, since all the VNI
>> between an
>> >      >     ingress and
>> >      >      >     egress VTEP share fate.
>> >      >      >
>> >      >      >     Yours,
>> >      >      >     Joel
>> >      >      >
>> >      >      >      >              >         On Wed, Jul 31, 2019 at
>> 5:48 AM
>> VXLAN
>> >      >      >      >              >                 be sent to a
>> tenant. VNI

---

From: Greg Mirsky <[email-masked]>
Date: Sat, 3 Aug 2019 10:33:48 -0700

Hi Dinesh,
many thanks for your detailed updates on how some implementations process
VXLAN header and the inner Ethernet frame. These are very helpful in
achieving the workable solution for the problem at hand.
You've noted that a path between VTEPs may be monitored in the underlay
network by merely establishing a BFD session. That is true, but by using
BFD with VXLAN encapsulation between the pair of VTEPs we are extending the
OAM domain by including, to some extent, VXLAN forwarding engine. Abstract
in RFC 5880 defines the goal and the domain in which BFD protocol can
detect a fault as:
   This document describes a protocol intended to detect faults in the
   bidirectional path between two forwarding engines, including
   interfaces, data link(s), and to the extent possible the forwarding
   engines themselves, with potentially very low latency.
Thus, BFD in the underlay will exercise a part of IP forwarding engine
while BFD with VXLAN encapsulation, ran between the same pair of VTEPs,
extends the OAM domain. At the same time, defining BFD between tenant
systems in outside the goal of this specification. But VXLAN BFD session
between VTEPs may be useful in monitoring e2e path between tenants, as
described in the update to -07:
   At the same time, a service layer BFD session may be used between the
   tenants of VTEPs IP1 and IP2 to provide end-to-end fault management.
   In such case, for VTEPs BFD control packets of that session are
   indistinguishable from data packets.  If end-to-end defect detection
   is realized as the set of concatenated OAM domains, e.g., VM1-1 - IP1
   -- IP2 - VM2-1, then the BFD session over VXLAN between VTEPs SHOULD
   follow the procedures described in Section 6.8.17 [RFC5880].
I've attached the current working version of the draft.

Regards,
Greg

On Fri, Aug 2, 2019 at 5:43 PM Dinesh Dutt <[email-masked]> wrote:

>>> >     which IP addresses or MAC adddresses teh tenant is using or may
>>> plan
>>> >     to use.
>>> >
>>> >     Yours,
>>> >     Joel
>>> >
>>> >      >     First, the problem you describe is not what the document
>>> says
>>> >     it is
>>> >      >      > The VNIs may not share fate due to misconfiguration. And
>>> I
>>> >     strongly
>>> >      >      >      > one) unicast MAC address for use with VTEP? If
>>> the MAC
>>> >      >     address is
>>> >      >      >      >              > On Wed, Jul 31, 2019 at 9:20 AM
>>> Greg
>>> >     Mirsky
>>> >      >      >      >             <[email-masked]
>>> >      >      >      >              >     thank you for your
>>> consideration
>>> >      >      >      >              >     OAM, e.g. CCM frames. The
>>> reason
>>> >     to use
>>> >      >     127/8 for
>>> >      >      >      >             IPv4, and
>>> >      >      >      >              >     Ethernet frames with BFD
>>> Control
>>> >      >      >      >              >             With that new text,
>>> what
>>> >      >      >      >              >             trapping is based on

---

From: Dinesh Dutt <[email-masked]>
Date: Sun, 4 Aug 2019 09:07:23 -0700

Hi Greg,

Thanks for your clarifications. I agree with your sentiment on why you're
running BFD over VXLAN between VTEPs. I wasn't arguing against it at all.
All I was saying was pointing to the limitations of the use of management
VNI. I spoke to some operators who're running EVPN and mentioned the
discussion on this thread. They concur that they're using specific VNIs to
test connectivity over that VNI between VTEPs to ensure misconfiguration
doesn't lead to blackholes. My statements are based in real world operator
experience. And I was providing language that ensured packets didn't leak
across to tenants when they were destined to VTEPs.

Dinesh

On Sat, Aug 3, 2019 at 10:34 AM Greg Mirsky <[email-masked]> wrote:

>>>> >      >     most cases, a single BFD session is sufficient for the
>>>> given
>>>> >     VTEP to
>>>> >      >     monitor the reachability of a remote VTEP, regardless of
>>>> the
>>>> >     number of
>>>> >      >     VNIs in common. "
>>>> >      >
>>>> >      >     Second, you assume the existence of an IP address for a
>>>> VTEP
>>>> >     within a
>>>> >      >      > The VNIs may not share fate due to misconfiguration.
>>>> And I
>>>> >     strongly
>>>> >      >      > (because it's BFD) owned by the VTEP in that VNI, you
>>>> can
>>>> >     use BFD in
>>>> >      >      > <mailto:[email-masked] <mailto:[email-masked]
>>>> >
>>>> >      >      >      >     packet comes with valid VNI with a MAC
>>>> address
>>>> >     that is
>>>> >      >     being
>>>> >      >      >     used by
>>>> >      >      >      >     fact that we could use physical MAC address
>>>> as
>>>> >     inner
>>>> >      >     MAC to
>>>> >      >      >     ensure
>>>> >      >      >      >             Sorry to ask a stupid question.
>>>> Whose
>>>> >      >     implementation?
>>>> >      >      >      >
>>>> >      >      >      >              >         address range is
>>>> host/node local
>>>> >      >      >      >              >             request to allocate
>>>> >      >      >      >              >             Also, earlier using
>>>> >      >      >      >              >             I would like to
>>>> propose

---

From: Greg Mirsky <[email-masked]>
Date: Mon, 5 Aug 2019 15:04:34 -0700

Hi Dinesh,
what do you see as the way forward? I agree, that the proposed text doesn't
work for multi-VNI concurrent monitoring because these VNIs are tenant's
VNIs. And in that case, we need to specify another mechanism to trap the
BFD Control packet at VTEP. It seems that VTEP's Ethernet address must be
used as the destination MAC address in the inner Ethernet frame. The
destination IP address may be either VTEP's address of martian (I do prefer
martian). Let me give it  try:
NEW TEXT:

To monitor continuity of the path between two VTEPs, an operator MUST
select a VNI number to be used as Management VNI. Management VNI number
MUST NOT be one of the tenant's VNIs to prevent sending VXLAN packets
received on Management VNI to a tenant. VNI number 1 is RECOMMENDED as the
default for Management VNI. [Ed.note: What we set the Destination MAC to?
Can it be invalid MAC that MUST be ignored on receipt?]

If an implementation supports concurrent monitoring of multiple VNIs, then
the value of VNI number MAY be one of tenant's VNIs. The destination MAC
address in the inner Ethernet frame encapsulating BFD Control packet MUST
be MAC associated with the remote VTEP.
The destination IP address of the inner IP packet MUST be selected from the
range 127/8 for IPv4, and for IPv6 from the range
0:0:0:0:0:FFFF:7F00:0/104. The TTL value in the inner IP header MUST be set
to 1.

Regards,
Greg

On Sun, Aug 4, 2019 at 9:07 AM Dinesh Dutt <[email-masked]> wrote:

>>>>> >      >     VNI.  As with the MAC address, the VTEP does not have an
>>>>> IP
>>>>> >     address
>>>>> >      >      > address and you can use that on the inner frame. That
>>>>> is
>>>>> >     all normal
>>>>> >      >      >     to be able to send these BFD packets over
>>>>> arbitrary VNI to
>>>>> >      >     monitor the
>>>>> >      >      >      >             segregation is not to care about
>>>>> such
>>>>> >     issues.
>>>>> >      >      >      >
>>>>> >      >      >      >              >     based on its similarity to
>>>>> the
>>>>> >     practice
>>>>> >      >     of using
>>>>> >      >      >      >             Management VLAN.
>>>>> >      >      >      >              >
>>>>> >      >      >      >              >     Regards,
>>>>> >      >      >      >              >     Greg
>>>>> >      >      >      >              >
>>>>> >      >      >      >              >     On Wed, Jul 31, 2019 at
>>>>> 12:03 PM
>>>>> >     Dinesh
>>>>> >      >     Dutt
>>>>> >      >      >      >             <[email-masked]
>>>>> >      >      >      >              >         VNI. What if someone
>>>>> wants
>>>>> >      >      >      >              >             of the inner IP
>>>>> >      >      >      >              >             and for IPv6 from

---

From: Dinesh Dutt <[email-masked]>
Date: Mon, 5 Aug 2019 15:59:09 -0700

Hi Greg,

That we agree on the problem definition is the first step forward. Your
original document had my cases covered and so I was surprised by the track
this thread took. It doesn't matter, we're back on track.

My recommendation is to not worry about specifying the precise MAC/IP
address used in the inner header. The addresses chosen MUST ensure that the
packet is trapped to the control plane of the VTEP and not escape to the
tenant if the BFD is to the VTEP. Any solution MUST also not preclude the
use of the BFD by tenant systems for that VNI. There are many ways an
implementer can choose to implement this. For example, the inner MAC
address is whatever the VTEP implementer would return if ARP'd for the IP
address used in the inner header in the given VNI. The implementer can pick
a fixed MAC address, one that they own etc. Multiple BFD sessions can be
run for testing path connectivity on more than one VNIs. Limits should be
in place to avoid overwhelming the receiver with BFD messages (you had
words about this in your currently published draft).  If the VNI is
irrelevant in the test i.e. only the VXLAN pipe at the VTEP is being
tested. the user can use any VNI active on the VTEP on which the VTEP owns
an IP address.

I'm concerned about the use of 127/8 address only because of firewalls or
implementations that drop packets with these addresses as either the source
or destination. For example, on many merchant silicon, I don't believe you
can turn off martian checking and drops *only* for VXLAN-encapsulated BFD
packets. I don't know what the Linux kernel does today on such packets, for
example (or Hyper-V). I'd like a solution that doesn't demand additional or
new chip functionality or require additional middle-box hole punch.

Why do you feel you MUST to specify the MAC/IP address on the inner packet?
What am I missing here?

Dinesh

On Mon, Aug 5, 2019 at 3:04 PM Greg Mirsky <[email-masked]> wrote:

> Hi Dinesh,
> what do you see as the way forward? I agree, that the proposed text
> doesn't work for multi-VNI concurrent monitoring because these VNIs are
> tenant's VNIs. And in that case, we need to specify another mechanism to
> trap the BFD Control packet at VTEP. It seems that VTEP's Ethernet address
> must be used as the destination MAC address in the inner Ethernet frame.
> The destination IP address may be either VTEP's address of martian (I do
> prefer martian). Let me give it  try:
> NEW TEXT:
>
> The destination IP address of the inner IP packet MUST be selected from
> the range 127/8 for IPv4, and for IPv6 from the range
>>> many thanks for your detailed updates on how some implementations
>>> process VXLAN header and the inner Ethernet frame. These are very helpful
>>> in achieving the workable solution for the problem at hand.
>>>>>> >      >      > that VNI. Thats all that you need to dictate.  That
>>>>>> IP address
>>>>>> >      >     has a MAC
>>>>>> >      >      >      > Thanks for verifying this. On Linux and
>>>>>> hardware
>>>>>> >     routers
>>>>>> >      >     that I'm
>>>>>> >      >      >     aware
>>>>>> >      >      >      >     I have cross checked point raised about
>>>>>> MAC address
>>>>>> >      >     usage. It is
>>>>>> >      >      >      >     fact that we could use physical MAC
>>>>>> address as
>>>>>> >     inner
>>>>>> >      >     MAC to
>>>>>> >      >      >     ensure
>>>>>> >      >      >      >         space while checking internally. I
>>>>>> will cross
>>>>>> >      >     check again for
>>>>>> >      >      >      >         On Wed, Jul 31, 2019 at 10:54 AM Joel
>>>>>> M.
>>>>>> >     Halpern
>>>>>> >      >      >      >         <[email-masked]
>>>>>> >      >      >      >             The reason I ask is that as far as
>>>>>> I
>>>>>> >     can tell,
>>>>>> >      >     since the
>>>>>> >      >      >      >             tenant does not
>>>>>> >      >      >      >             have any control access to the
>>>>>> VTEP,
>>>>>> >     there is no
>>>>>> >      >      >     reason for
>>>>>> >      >      >      >             the VTEP to
>>>>>> >      >      >      >              >     0:0:0:0:0:FFFF:7F00:0/104
>>>>>> for
>>>>>> >     IPv6 is
>>>>>> >      >     to safeguard
>>>>>> >      >      >      >             from leaking
>>>>>> >      >      >      >              >     Ethernet frames with BFD
>>>>>> Control
>>>>>> >     packet
>>>>>> >      >     to a
>>>>>> >      >      >     tenant.
>>>>>> >      >      >     <mailto:[email-masked] <mailto:[email-masked]
>>>>>> >
>>>>>> >      >      >     <mailto:[email-masked] <mailto:[email-masked]
>>>>>> >
>>>>>> >      >      >     <mailto:[email-masked] <mailto:[email-masked]
>>>>>> >
>>>>>> >      >      >      >              >             experts equally
>>>>>> split
>>>>>> >      >      >      >              >             trapping is based
>>>>>> on VNI
>>>>>> >      >      >      >              >             lastly, the TTL to
>>>>>> be

---

From: Greg Mirsky <[email-masked]>
Date: Mon, 5 Aug 2019 17:56:06 -0700

Hi Dinesh,
thank you for your expedient detailed response.
I believe that the ability to run BFD session up to a tenant
(VTEP-VTEP-tenant or tenant-tenant) was never in jeopardy from this
specification.
I'm trying to provide precise specification on what can be used ad the
destination MAC and IP addresses in the inner frame/packet as I believe
that likely will help to avoid interoperability issues.
I'm interested to learn some more about the "martian checking" function. As
you know, martian addresses have been used as destination IP address in LSP
Ping and BFD over MPLS LSP and PW. I haven't heard that any silicon feature
caused problems for operators using these tools.

Regards,
Greg

On Mon, Aug 5, 2019 at 3:59 PM Dinesh Dutt <[email-masked]> wrote:

> Why do you feel you MUST to specify the MAC/IP address on the inner
> packet? What am I missing here?
>
> Dinesh
>
>> If an implementation supports concurrent monitoring of multiple VNIs,
>> then the value of VNI number MAY be one of tenant's VNIs. The destination
>> MAC address in the inner Ethernet frame encapsulating BFD Control packet
>> MUST be MAC associated with the remote VTEP.
>>> Thanks for your clarifications. I agree with your sentiment on why
>>> you're running BFD over VXLAN between VTEPs. I wasn't arguing against it at
>>> all. All I was saying was pointing to the limitations of the use of
>>> management VNI. I spoke to some operators who're running EVPN and mentioned
>>> the discussion on this thread. They concur that they're using specific VNIs
>>> to test connectivity over that VNI between VTEPs to ensure misconfiguration
>>>>>>> >     which IP addresses or MAC adddresses teh tenant is using or
>>>>>>> may plan
>>>>>>> >     to use.
>>>>>>> >
>>>>>>> >     Yours,
>>>>>>> >     Joel
>>>>>>> >
>>>>>>> >      >     First, the problem you describe is not what the
>>>>>>> document says
>>>>>>> >     it is
>>>>>>> >      >     monitor the reachability of a remote VTEP, regardless
>>>>>>> of the
>>>>>>> >     number of
>>>>>>> >      >     VNIs in common. "
>>>>>>> >      >
>>>>>>> >      >     VNI.  As with the MAC address, the VTEP does not have
>>>>>>> an IP
>>>>>>> >     address
>>>>>>> >      >      > (because it's BFD) owned by the VTEP in that VNI,
>>>>>>> you can
>>>>>>> >     use BFD in
>>>>>>> >      >      > address and you can use that on the inner frame.
>>>>>>> That is
>>>>>>> >     all normal
>>>>>>> >      >      >      > one) unicast MAC address for use with VTEP?
>>>>>>> If the MAC
>>>>>>> >      >     address is
>>>>>>> >      >      >      >             The reason I ask is that as far
>>>>>>> as I
>>>>>>> >     can tell,
>>>>>>> >      >     since the
>>>>>>> >      >      >      >             tenant does not
>>>>>>> >      >      >      >              > On Wed, Jul 31, 2019 at 9:20
>>>>>>> AM Greg
>>>>>>> >     Mirsky
>>>>>>> >      >      >      >             <[email-masked]
>>>>>>> >      >      >      >              >     You've suggested using a
>>>>>>> >      >      >      >              >         As long as the inner
>>>>>>> >      >      >      >              >         On Wed, Jul 31, 2019
>>>>>>> at 5:48 AM
>>>>>>> >      >      >      >              >             request to
>>>>>>> allocate the
>>>>>>> >      >      >      >              >             Also, earlier
>>>>>>> using VNI
>>>>>>> >      >      >      >              >             might be used as
>>>>>>> >      >      >      >              >             Ethernet frame. As
>>>>>>> >      >      >      >              >                 An operator
>>>>>>> MUST
>>>>>>> VNI. VXLAN
>>>>>>> >      >      >      >              >             With that new
>>>>>>> text, what
>>>>>>> >      >      >      >              >             Much appreciate
>>>>>>> your

---

From: Dinesh Dutt <[email-masked]>
Date: Mon, 5 Aug 2019 18:03:28 -0700

On Mon, Aug 5, 2019 at 5:56 PM Greg Mirsky <[email-masked]> wrote:

> I'm interested to learn some more about the "martian checking" function.
> As you know, martian addresses have been used as destination IP address in
> LSP Ping and BFD over MPLS LSP and PW. I haven't heard that any silicon
> feature caused problems for operators using these tools.
>

Interesting. I didn't know this aspect of use with MPLS ping. Did those
packets ever go through a firewall though? In any case, maybe suggest the
use of those addresses with a statement that this is how LSP does it, but
that other MAC/IP pairs are possible as long as the conditions of the
endpoint owning the MAC/IP was honored.

Dinesh

>>>>> You've noted that a path between VTEPs may be monitored in the
>>>>> underlay network by merely establishing a BFD session. That is true, but by
>>>>> using BFD with VXLAN encapsulation between the pair of VTEPs we are
>>>>> extending the OAM domain by including, to some extent, VXLAN forwarding
>>>>> engine. Abstract in RFC 5880 defines the goal and the domain in which BFD
>>>>> protocol can detect a fault as:
>>>>>> What I mean is "How do you infer that it excludes the case I'm
>>>>>> talking about?".
>>>>>>
>>>>>> Dinesh
>>>>>>
>>>>>>>> >      >     Second, you assume the existence of an IP address for
>>>>>>>> a VTEP
>>>>>>>> >     within a
>>>>>>>> >      >     In short, you are requiring a behavior that violates
>>>>>>>> the
>>>>>>>> >     architectural
>>>>>>>> >      >      > The VNIs may not share fate due to
>>>>>>>> misconfiguration. And I
>>>>>>>> >     strongly
>>>>>>>> >      >      >      >         On Wed, Jul 31, 2019 at 10:54 AM
>>>>>>>> Joel M.
>>>>>>>> >     Halpern
>>>>>>>> >      >      >      >         <[email-masked]
>>>>>>>> >      >      >      >              >     based on its similarity
>>>>>>>> to the
>>>>>>>> >      >      >      >              >         trapped to the CPU,
>>>>>>>> it should be
>>>>>>>> >      >      >      >              >         it not? Stating that
>>>>>>>> >      >      >      >              >         VNI? I would not pick
>>>>>>>> >      >      >      >              >         <
>>>>>>>> [email-masked]
>>>>>>>> As I
>>>>>>>> >      >      >      >              >             lastly, the TTL
>>>>>>>> to be

---

From: Greg Mirsky <[email-masked]>
Date: Wed, 7 Aug 2019 09:24:46 -0700

Hi Dinesh, Joel, Sridhar, et al.,
much appreciate the help you've given me sharing your expertise. I hope
that the updates you will find in the attached diff and the working copy of
the draft be closer to the acceptable solution for VTEP-VTEP BFD. Please
note, that I'll shortly start a new discussion thread to address one of
Carlos's questions on the ambiguity of the text on multiple concurrent
sessions between the same pair of VTEPs.
Please review the changes to Sections 4 and 6 and share your feedback,
suggestions, and questions.

Regards,
Greg

On Mon, Aug 5, 2019 at 6:03 PM Dinesh Dutt <[email-masked]> wrote:

>>> I'm concerned about the use of 127/8 address only because of firewalls
>>> or implementations that drop packets with these addresses as either the
>>> source or destination. For example, on many merchant silicon, I don't
>>> believe you can turn off martian checking and drops *only* for
>>> VXLAN-encapsulated BFD packets. I don't know what the Linux kernel does
>>> today on such packets, for example (or Hyper-V). I'd like a solution that
>>> doesn't demand additional or new chip functionality or require additional
>>> middle-box hole punch.
>>>
>>>>>> Thus, BFD in the underlay will exercise a part of IP forwarding
>>>>>> engine while BFD with VXLAN encapsulation, ran between the same pair of
>>>>>> VTEPs, extends the OAM domain. At the same time, defining BFD between
>>>>>> tenant systems in outside the goal of this specification. But VXLAN BFD
>>>>>> session between VTEPs may be useful in monitoring e2e path between tenants,
>>>>>> as described in the update to -07:
>>>>>>>>> >      >     most cases, a single BFD session is sufficient for
>>>>>>>>> the given
>>>>>>>>> >     VTEP to
>>>>>>>>> >      >      > that VNI. Thats all that you need to dictate.
>>>>>>>>> That IP address
>>>>>>>>> >      >     has a MAC
>>>>>>>>> >      >      >      >             Thanks for your inputs. I
>>>>>>>>> checked
>>>>>>>>> >      >     implementation within
>>>>>>>>> >      >      >      >             Sorry to ask a stupid
>>>>>>>>> question.  Whose
>>>>>>>>> >      >      >      >             segregation is not to care
>>>>>>>>> about such
>>>>>>>>> >     issues.
>>>>>>>>> >      >      >      >
>>>>>>>>> >      >      >      >              >     OAM, e.g. CCM frames.
>>>>>>>>> The reason
>>>>>>>>> >      >      >      >              >         VNI? I would not
>>>>>>>>> pick a
>>>>>>>>> >      >      >      >              >             trapping is
>>>>>>>>> based on VNI
>>>>>>>>> >      >      >      >              >             and for IPv6
>>>>>>>>> from the range

---

From: Dinesh Dutt <[email-masked]>
Date: Wed, 7 Aug 2019 14:25:24 -0700

Looks god to me Greg. Thank you for your hard work in this,

Dinesh

On Wed, Aug 7, 2019 at 9:25 AM Greg Mirsky <[email-masked]> wrote:

>>>>> The destination IP address of the inner IP packet MUST be selected
>>>>> from the range 127/8 for IPv4, and for IPv6 from the range
>>>>>>>>>> >      >     monitor the reachability of a remote VTEP,
>>>>>>>>>> regardless of the
>>>>>>>>>> >     number of
>>>>>>>>>> >      >     VNIs in common. "
>>>>>>>>>> >      >
>>>>>>>>>> >      >     Second, you assume the existence of an IP address
>>>>>>>>>> for a VTEP
>>>>>>>>>> >     within a
>>>>>>>>>> >      >     VNI.  As with the MAC address, the VTEP does not
>>>>>>>>>> have an IP
>>>>>>>>>> >     address
>>>>>>>>>> >      >     structure of overlay / underlay separation, and
>>>>>>>>>> common
>>>>>>>>>> >     usage.  And you
>>>>>>>>>> >      >      > path continuity as stated by the draft. As long
>>>>>>>>>> as there's a
>>>>>>>>>> >      >     valid IP
>>>>>>>>>> >      >      >     even a problem I understand, since all the
>>>>>>>>>> VNI between an
>>>>>>>>>> >      >     ingress and
>>>>>>>>>> >      >      >     egress VTEP share fate.
>>>>>>>>>> >      >      >
>>>>>>>>>> >      >      >     Yours,
>>>>>>>>>> >      >      >     Joel
>>>>>>>>>> >      >      >
>>>>>>>>>> >      >      >      > one) unicast MAC address for use with
>>>>>>>>>> VTEP? If the MAC
>>>>>>>>>> >      >     address is
>>>>>>>>>> >      >      >      >     I have cross checked point raised
>>>>>>>>>> about MAC address
>>>>>>>>>> >      >     usage. It is
>>>>>>>>>> >      >      >      >             The reason I ask is that as
>>>>>>>>>> far as I
>>>>>>>>>> >     can tell,
>>>>>>>>>> >      >     since the
>>>>>>>>>> >      >      >      >             tenant does not
>>>>>>>>>> >      >      >      >             have a MAC address in the
>>>>>>>>>> tenant
>>>>>>>>>> >     space.  Yes, the
>>>>>>>>>> >      >      >     device has
>>>>>>>>>> >      >      >      >             a physical
>>>>>>>>>> 9:20 AM Greg
>>>>>>>>>> >      >      >      >              >     Ethernet frames with
>>>>>>>>>> BFD Control
>>>>>>>>>> >      >      >      >              >     You've suggested using
>>>>>>>>>> a MAC
>>>>>>>>>> >      >      >      >              >     <mailto:
>>>>>>>>>> [email-masked]
>>>>>>>>>> >      >      >      >              >         As long as the
>>>>>>>>>> inner MAC
>>>>>>>>>> >      >      >      >              >         it not? Stating
>>>>>>>>>> >      >      >      >              >         VNI. What if
>>>>>>>>>> someone wants
>>>>>>>>>> >      >      >      >              >         On Wed, Jul 31,
>>>>>>>>>> 2019 at 5:48 AM
>>>>>>>>>> >      >      >      >              >             probably the
>>>>>>>>>> most
>>>>>>>>>> >      >      >      >              >             of the options
>>>>>>>>>> >      >      >      >              >             might be used
>>>>>>>>>> as the
>>>>>>>>>> >      >      >      >              >             idea is to let
>>>>>>>>>> >      >      >      >              >             that VNI in
>>>>>>>>>> >      >      >      >              >                 be sent to
>>>>>>>>>> a tenant. VNI
>>>>>>>>>> >      >      >      >              >             MAC in the
>>>>>>>>>> inner Ethernet? I

---

From: Greg Mirsky <[email-masked]>
Date: Thu, 8 Aug 2019 15:56:58 -0700

Dinesh, thank you for your help, much appreciated.

Hi Joel and Sridhar,
could you please check if the updated text on the inner Ethernet frame
addressed your concern.

On Wed, Aug 7, 2019 at 2:25 PM Dinesh Dutt <[email-masked]> wrote:

>>>> I'm interested to learn some more about the "martian checking"
>>>> function. As you know, martian addresses have been used as destination IP
>>>> address in LSP Ping and BFD over MPLS LSP and PW. I haven't heard that any
>>>> silicon feature caused problems for operators using these tools.
>>>>
>>>
>>>>> That we agree on the problem definition is the first step forward.
>>>>> Your original document had my cases covered and so I was surprised by the
>>>>> track this thread took. It doesn't matter, we're back on track.
>>>>>
>>>>>>>>>>> >     which IP addresses or MAC adddresses teh tenant is using
>>>>>>>>>>> or may plan
>>>>>>>>>>> >     to use.
>>>>>>>>>>> >
>>>>>>>>>>> >     Yours,
>>>>>>>>>>> >     Joel
>>>>>>>>>>> >
>>>>>>>>>>> >      >     In short, you are requiring a behavior that
>>>>>>>>>>> violates the
>>>>>>>>>>> >     architectural
>>>>>>>>>>> >      >      > (because it's BFD) owned by the VTEP in that
>>>>>>>>>>> VNI, you can
>>>>>>>>>>> >     use BFD in
>>>>>>>>>>> >      >      >      >     packet comes with valid VNI with a
>>>>>>>>>>> MAC address
>>>>>>>>>>> >     that is
>>>>>>>>>>> >      >     being
>>>>>>>>>>> >      >      >     used by
>>>>>>>>>>> >      >      >      >         space while checking internally.
>>>>>>>>>>> I will cross
>>>>>>>>>>> >      >     check again for
>>>>>>>>>>> >      >      >      >             have any control access to
>>>>>>>>>>> the VTEP,
>>>>>>>>>>> >     there is no
>>>>>>>>>>> >      >      >     reason for
>>>>>>>>>>> >      >      >      >             the VTEP to
>>>>>>>>>>> >      >      >      >              >     based on its
>>>>>>>>>>> similarity to the
>>>>>>>>>>> at 12:03 PM
>>>>>>>>>>> >      >      >      >              >         trapped to the
>>>>>>>>>>> CPU, it should be
>>>>>>>>>>> >      >      >      >              >             Ethernet
>>>>>>>>>>> frame. As I
>>>>>>>>>>> >      >      >      >              >             experts
>>>>>>>>>>> equally split
>>>>>>>>>>> >      >      >      >              >             I would like
>>>>>>>>>>> to propose
>>>>>>>>>>> >      >      >      >              >                 An
>>>>>>>>>>> operator MUST
>>>>>>>>>>> >      >      >      >              >             of the inner
>>>>>>>>>>> IP packet
>>>>>>>>>>> >      >      >      >              >             lastly, the
>>>>>>>>>>> TTL to be
>>>>>>>>>>> >      >      >      >              >             Much
>>>>>>>>>>> appreciate your

---

From: "T. Sridhar" <[email-masked]>
Date: Fri, 9 Aug 2019 06:46:42 +0000

My apologies for the delayed response due to my being on PTO.

Greg – while thinking about some of the comments below, I am wondering if we should have more text as to the problem we are addressing here. (maybe this will be addressed in the new clarification text to address Carlos’s comment).  I was specifically wondering about Dinesh’s points below and how to add text to clarify what is being solved:

If the VNI is irrelevant in the test i.e. only the VXLAN pipe at the VTEP is being tested. the user can use any VNI active on the VTEP on which the VTEP owns an IP address.

Any solution MUST also not preclude the use of the BFD by tenant systems for that VNI.

Section 3 indicates “BFD packets intended for a hypervisor VTEP MUST NOT be forwarded” –  I would have preferred more detailed text as to the targets of the BFD packets before it appears in Section 3 and 4 – per VNI (“tenant”) and in some cases where the tenant is housed within the VTEP itself (management VNI?).

I will wait for the more detailed discussion related to multiple BFD sessions and a follow on updated draft which could address these questions.

Thanks,
Sridhar

From: Greg Mirsky <[email-masked]>
Date: Thursday, August 8, 2019 at 3:57 PM
To: Dinesh Dutt <[email-masked]>
Cc: "Joel M. Halpern" <[email-masked]>, Santosh P K <[email-masked]>, rtg-bfd WG <[email-masked]>, "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>, Martin Vigoureux <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: BFD over VXLAN: Trapping BFD Control packet at VTEP

Hi Joel and Sridhar,
could you please check if the updated text on the inner Ethernet frame addressed your concern.

On Wed, Aug 7, 2019 at 2:25 PM Dinesh Dutt <[email-masked]<mailto:[email-masked]>> wrote:
Looks god to me Greg. Thank you for your hard work in this,

Dinesh

On Wed, Aug 7, 2019 at 9:25 AM Greg Mirsky <[email-masked]<mailto:[email-masked]>> wrote:
Hi Dinesh, Joel, Sridhar, et al.,
much appreciate the help you've given me sharing your expertise. I hope that the updates you will find in the attached diff and the working copy of the draft be closer to the acceptable solution for VTEP-VTEP BFD. Please note, that I'll shortly start a new discussion thread to address one of Carlos's questions on the ambiguity of the text on multiple concurrent sessions between the same pair of VTEPs.
Please review the changes to Sections 4 and 6 and share your feedback, suggestions, and questions.

Regards,
Greg

On Mon, Aug 5, 2019 at 6:03 PM Dinesh Dutt <[email-masked]<mailto:[email-masked]>> wrote:

On Mon, Aug 5, 2019 at 5:56 PM Greg Mirsky <[email-masked]<mailto:[email-masked]>> wrote:
Hi Dinesh,
thank you for your expedient detailed response.
I believe that the ability to run BFD session up to a tenant (VTEP-VTEP-tenant or tenant-tenant) was never in jeopardy from this specification.
I'm trying to provide precise specification on what can be used ad the destination MAC and IP addresses in the inner frame/packet as I believe that likely will help to avoid interoperability issues.
I'm interested to learn some more about the "martian checking" function. As you know, martian addresses have been used as destination IP address in LSP Ping and BFD over MPLS LSP and PW. I haven't heard that any silicon feature caused problems for operators using these tools.

Interesting. I didn't know this aspect of use with MPLS ping. Did those packets ever go through a firewall though? In any case, maybe suggest the use of those addresses with a statement that this is how LSP does it, but that other MAC/IP pairs are possible as long as the conditions of the endpoint owning the MAC/IP was honored.

Dinesh

Regards,
Greg

On Mon, Aug 5, 2019 at 3:59 PM Dinesh Dutt <[email-masked]<mailto:[email-masked]>> wrote:
Hi Greg,

That we agree on the problem definition is the first step forward. Your original document had my cases covered and so I was surprised by the track this thread took. It doesn't matter, we're back on track.

My recommendation is to not worry about specifying the precise MAC/IP address used in the inner header. The addresses chosen MUST ensure that the packet is trapped to the control plane of the VTEP and not escape to the tenant if the BFD is to the VTEP. Any solution MUST also not preclude the use of the BFD by tenant systems for that VNI. There are many ways an implementer can choose to implement this. For example, the inner MAC address is whatever the VTEP implementer would return if ARP'd for the IP address used in the inner header in the given VNI. The implementer can pick a fixed MAC address, one that they own etc. Multiple BFD sessions can be run for testing path connectivity on more than one VNIs. Limits should be in place to avoid overwhelming the receiver with BFD messages (you had words about this in your currently published draft).  If the VNI is irrelevant in the test i.e. only the VXLAN pipe at the VTEP is being tested. the user can use any VNI active on the VTEP on which the VTEP owns an IP address.

I'm concerned about the use of 127/8 address only because of firewalls or implementations that drop packets with these addresses as either the source or destination. For example, on many merchant silicon, I don't believe you can turn off martian checking and drops *only* for VXLAN-encapsulated BFD packets. I don't know what the Linux kernel does today on such packets, for example (or Hyper-V). I'd like a solution that doesn't demand additional or new chip functionality or require additional middle-box hole punch.

Why do you feel you MUST to specify the MAC/IP address on the inner packet? What am I missing here?

Dinesh

On Mon, Aug 5, 2019 at 3:04 PM Greg Mirsky <[email-masked]<mailto:[email-masked]>> wrote:
Hi Dinesh,
what do you see as the way forward? I agree, that the proposed text doesn't work for multi-VNI concurrent monitoring because these VNIs are tenant's VNIs. And in that case, we need to specify another mechanism to trap the BFD Control packet at VTEP. It seems that VTEP's Ethernet address must be used as the destination MAC address in the inner Ethernet frame. The destination IP address may be either VTEP's address of martian (I do prefer martian). Let me give it  try:
NEW TEXT:
To monitor continuity of the path between two VTEPs, an operator MUST select a VNI number to be used as Management VNI. Management VNI number MUST NOT be one of the tenant's VNIs to prevent sending VXLAN packets received on Management VNI to a tenant. VNI number 1 is RECOMMENDED as the default for Management VNI. [Ed.note: What we set the Destination MAC to? Can it be invalid MAC that MUST be ignored on receipt?]
If an implementation supports concurrent monitoring of multiple VNIs, then the value of VNI number MAY be one of tenant's VNIs. The destination MAC address in the inner Ethernet frame encapsulating BFD Control packet MUST be MAC associated with the remote VTEP.
The destination IP address of the inner IP packet MUST be selected from the range 127/8 for IPv4, and for IPv6 from the range 0:0:0:0:0:FFFF:7F00:0/104. The TTL value in the inner IP header MUST be set to 1.

Regards,
Greg
On Sun, Aug 4, 2019 at 9:07 AM Dinesh Dutt <[email-masked]<mailto:[email-masked]>> wrote:
Hi Greg,

Thanks for your clarifications. I agree with your sentiment on why you're running BFD over VXLAN between VTEPs. I wasn't arguing against it at all. All I was saying was pointing to the limitations of the use of management VNI. I spoke to some operators who're running EVPN and mentioned the discussion on this thread. They concur that they're using specific VNIs to test connectivity over that VNI between VTEPs to ensure misconfiguration doesn't lead to blackholes. My statements are based in real world operator experience. And I was providing language that ensured packets didn't leak across to tenants when they were destined to VTEPs.

Dinesh

On Sat, Aug 3, 2019 at 10:34 AM Greg Mirsky <[email-masked]<mailto:[email-masked]>> wrote:
Hi Dinesh,
many thanks for your detailed updates on how some implementations process VXLAN header and the inner Ethernet frame. These are very helpful in achieving the workable solution for the problem at hand.
You've noted that a path between VTEPs may be monitored in the underlay network by merely establishing a BFD session. That is true, but by using BFD with VXLAN encapsulation between the pair of VTEPs we are extending the OAM domain by including, to some extent, VXLAN forwarding engine. Abstract in RFC 5880 defines the goal and the domain in which BFD protocol can detect a fault as:
   This document describes a protocol intended to detect faults in the
   bidirectional path between two forwarding engines, including
   interfaces, data link(s), and to the extent possible the forwarding
   engines themselves, with potentially very low latency.
Thus, BFD in the underlay will exercise a part of IP forwarding engine while BFD with VXLAN encapsulation, ran between the same pair of VTEPs, extends the OAM domain. At the same time, defining BFD between tenant systems in outside the goal of this specification. But VXLAN BFD session between VTEPs may be useful in monitoring e2e path between tenants, as described in the update to -07:
   At the same time, a service layer BFD session may be used between the
   tenants of VTEPs IP1 and IP2 to provide end-to-end fault management.
   In such case, for VTEPs BFD control packets of that session are
   indistinguishable from data packets.  If end-to-end defect detection
   is realized as the set of concatenated OAM domains, e.g., VM1-1 - IP1
   -- IP2 - VM2-1, then the BFD session over VXLAN between VTEPs SHOULD
   follow the procedures described in Section 6.8.17 [RFC5880].
I've attached the current working version of the draft.

Regards,
Greg

On Fri, Aug 2, 2019 at 5:43 PM Dinesh Dutt <[email-masked]<mailto:[email-masked]>> wrote:
What I mean is "How do you infer that it excludes the case I'm talking about?".

Dinesh

On Fri, Aug 2, 2019 at 5:41 PM Dinesh Dutt <[email-masked]<mailto:[email-masked]>> wrote:
The abstract reads this: "

Dinesh

On Fri, Aug 2, 2019 at 5:38 PM Joel M. Halpern <[email-masked]<mailto:[email-masked]>> wrote:

Yours,
Joel

On 8/2/2019 8:30 PM, Dinesh Dutt wrote:
> On Fri, Aug 2, 2019 at 5:08 PM Joel M. Halpern <[email-masked]<mailto:[email-masked]>
> <mailto:[email-masked]<mailto:[email-masked]>>> wrote:
>
>     <[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>
>      > <mailto:[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>>> wrote:
>      >
>      >     <[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>
>     <mailto:[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>>
>      >      > <mailto:[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>
>     <mailto:[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>>>> wrote:
>      >      >
>      >      >      > <[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>>
>      >     <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>>>
>      >      >     <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>>
>      >     <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>>>>
>      >      >     <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>>
>      >     <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>>>
>      >      >     <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>>
>      >     <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>>>>>> wrote:
>      >      >      >
>      >      >      >     <[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>>
>      >     <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>>>
>      >      >     <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>>
>      >     <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>>>>
>      >      >     <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>>
>      >     <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>>>
>      >      >     <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>>
>      >     <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>>>>>>
>      >      >      >     wrote:
>      >      >      >
>      >      >      >         Joel,
>      >      >      >         <[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>> <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>>>
>      >     <mailto:[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>
>     <mailto:[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>>>
>      >      >     <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>> <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>>>
>      >     <mailto:[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>
>     <mailto:[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>>>>> wrote:
>      >      >      >
>      >      >      >             <[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>>
>      >     <mailto:[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>>
>     <mailto:[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>
>      >     <mailto:[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>>>
>      >      >     <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>> <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>>>
>      >     <mailto:[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>
>     <mailto:[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>>>>
>      >      >      >              > <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>>
>      >     <mailto:[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>>
>      >      >     <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>> <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>>>>
>      >      >      >             <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>>
>      >     <mailto:[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>>
>      >      >     <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>>
>      >     <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>>>>>>> wrote:
>      >      >      >              >
>      >      >      >              >     Hi Dinesh,
>      >      >      >             <[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>> <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>>>
>      >     <mailto:[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>
>     <mailto:[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>>>
>      >      >     <mailto:[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>
>     <mailto:[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>>
>      >     <mailto:[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>
>     <mailto:[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>>>>
>      >      >      >              >     <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>>
>      >     <mailto:[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>>
>      >      >     <mailto:[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>
>     <mailto:[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>>>
>      >     <mailto:[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>
>     <mailto:[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>>
>      >      >     <mailto:[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>
>     <mailto:[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>>>>>>
>      >      >      >             wrote:
>      >      >      >              >
>      >      >      >              >         Hi Greg,
>      >      >      >              >
>      >      >      >              >         As long as the inner MAC
>     address is
>      >     such
>      >      >     that the
>      >      >      >             packet is
>      >      >      >              >         <[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>>
>      >     <mailto:[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>>
>      >      >     <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>> <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>>>>
>      >      >      >             <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>>
>      >     <mailto:[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>>
>      >      >     <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>>
>      >     <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>>>>> <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>>
>      >     <mailto:[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>>
>      >      >     <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>> <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>>>>
>      >      >      >             <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>>
>      >     <mailto:[email-masked]<mailto:[email-masked]> <mailto:[email-masked]<mailto:[email-masked]>>>
>      >      >     <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>>
>      >     <mailto:[email-masked]<mailto:[email-masked]>
>     <mailto:[email-masked]<mailto:[email-masked]>>>>>>> wrote:
>      >      >      >              >
>      >      >      >              >             Dear All,

---

From: Santosh P K <[email-masked]>
Date: Wed, 14 Aug 2019 10:54:08 +0530

Greg,
   Thanks for updated version of document. Here are few comments on new
draft.

Section 4:
Destination MAC: This MUST NOT be of one of tenant's MAC
         addresses.  The MAC address MAY be configured, or it MAY be
         learned via a control plane protocol.  The details of how the
         MAC address is obtained are outside the scope of this document.

I think we may need to give background on why we are saying MAC address
MUST not be one of tenant's MAC address. Like in this thread we have
discussed one of the tenant could have borrowed the same VTEP mac address
and we if we have to use BFD then we need to avoid that conflict to ensure
BFD packets get observed in the VTEP itself. Should we add a section before
4 to set that context so that above text makes more sense in that context?

   IP header:
         Destination IP: IP address MUST NOT be of one of tenant's IP
         addresses.  IP address MAY be selected from the range 127/8 for
         IPv4, for IPv6 - from the range 0:0:0:0:0:FFFF:7F00:0/104.

         TTL: MUST be set to 1 to ensure that the BFD packet is not
         routed within the L3 underlay network.

I think we have added some text to address Sridhar comments on why TTL MUST
be 1 and dest IP address MUST be 127/8 range. I see that text is missing
now.

Section 5.1:
For such packets, the BFD session MUST be identified
   using the following three-tuples of fields of the inner header: the
   source IP, the destination IP, and the source UDP port number present
   in the IP header carried by the payload of the packet in VXLAN
   encapsulation.  If BFD packet is received with non-zero Your
Discriminator, then BFD session MUST be demultiplexed only with Your
   Discriminator as the key.

Just with 3 tuple we will not be able to demux packet. We need to consider
VNI as well if we have multiple BFD session between same pair of VTEP.

Thanks
Santosh P K

On Fri, Aug 9, 2019 at 4:27 AM Greg Mirsky <[email-masked]> wrote:

>>>>>> I'm concerned about the use of 127/8 address only because of
>>>>>> firewalls or implementations that drop packets with these addresses as
>>>>>> either the source or destination. For example, on many merchant silicon, I
>>>>>> don't believe you can turn off martian checking and drops *only* for
>>>>>>> To monitor continuity of the path between two VTEPs, an operator
>>>>>>> MUST select a VNI number to be used as Management VNI. Management VNI
>>>>>>> number MUST NOT be one of the tenant's VNIs to prevent sending VXLAN
>>>>>>> packets received on Management VNI to a tenant. VNI number 1 is RECOMMENDED
>>>>>>> as the default for Management VNI. [Ed.note: What we set the Destination
>>>>>>> MAC to? Can it be invalid MAC that MUST be ignored on receipt?]
>>>>>>>
>>>>>>> If an implementation supports concurrent monitoring of multiple
>>>>>>> VNIs, then the value of VNI number MAY be one of tenant's VNIs. The
>>>>>>> destination MAC address in the inner Ethernet frame encapsulating BFD
>>>>>>> Control packet MUST be MAC associated with the remote VTEP.
>>>>>>>>>>>> >      >      > address and you can use that on the inner
>>>>>>>>>>>> frame. That is
>>>>>>>>>>>> >     all normal
>>>>>>>>>>>> >      >      >      >              >     You've suggested
>>>>>>>>>>>> using a MAC
>>>>>>>>>>>> options an
>>>>>>>>>>>> >      >      >      >              >             The most
>>>>>>>>>>>> recent
>>>>>>>>>>>> >      >      >      >              >             idea is to
>>>>>>>>>>>> let an
>>>>>>>>>>>> >      >      >      >              >                 be sent
>>>>>>>>>>>> to a tenant. VNI
>>>>>>>>>>>> >      >      >      >              >                 default

---

From: Greg Mirsky <[email-masked]>
Date: Thu, 15 Aug 2019 16:29:04 -0700

Hi Santosh,
thank you for your comments. Please find my notes in-lined and tagged GIM>>.

Regards,
Greg

> 4 to set that context so that above text makes more sense in that context?
GIM>> Certainly. Please share the text you'd like to add.

> I think we have added some text to address Sridhar comments on why TTL
> MUST be 1 and dest IP address MUST be 127/8 range. I see that text is
> missing now.
>
GIM>> My apologies that I've missed to include the text from another
discussion thread. I believe the following would be complete:
          TTL or Hop Limit: MUST be set to 1 to ensure that the BFD
         packet is not routed within the Layer 3 underlay network.  This
         addresses the scenario when the inner IP destination address is
         of VXLAN gateway and there is a router in underlay which
         removes the VXLAN header, then it is possible to route the
         packet as VXLAN  gateway address is routable address.
> VNI as well if we have multiple BFD session between same pair of VTEP.
GIM>> This is one of comments from Carlos we need to address. Your comment
have helped me to form the question:

What is the goal running multiple BFD sessions between the pair of VTEPs?

If the goal is to monitor per VNI, then the following text should describe
the demultiplexing of the initial BFD Control packet:
   Demultiplexing of IP BFD packet has been defined in Section 3 of
   [RFC5881].  Since multiple BFD sessions may be running between two
   VTEPs, there needs to be a mechanism for demultiplexing received BFD
   packets to the proper session.  For demultiplexing packets with Your
   Discriminator equal to 0, a BFD session MUST be identified using the
   logical link over which the BFD Control packet is received.  In the
   case of VXLAN, the VNI number identifies that logical link.  If BFD
   packet is received with non-zero Your Discriminator, then BFD session
   MUST be demultiplexed only with Your Discriminator as the key.
Would there be need to run multiple BFD sessions with the same VNI number?

>>>>>> I'm trying to provide precise specification on what can be used ad
>>>>>> the destination MAC and IP addresses in the inner frame/packet as I believe
>>>>> Interesting. I didn't know this aspect of use with MPLS ping. Did
>>>>> those packets ever go through a firewall though? In any case, maybe suggest
>>>>> the use of those addresses with a statement that this is how LSP does it,
>>>>> but that other MAC/IP pairs are possible as long as the conditions of the
>>>>>>> My recommendation is to not worry about specifying the precise
>>>>>>> MAC/IP address used in the inner header. The addresses chosen MUST ensure
>>>>>>> that the packet is trapped to the control plane of the VTEP and not escape
>>>>>>> to the tenant if the BFD is to the VTEP. Any solution MUST also not
>>>>>>> preclude the use of the BFD by tenant systems for that VNI. There are many
>>>>>>> ways an implementer can choose to implement this. For example, the inner
>>>>>>> MAC address is whatever the VTEP implementer would return if ARP'd for the
>>>>>>> IP address used in the inner header in the given VNI. The implementer can
>>>>>>> pick a fixed MAC address, one that they own etc. Multiple BFD sessions can
>>>>>>> be run for testing path connectivity on more than one VNIs. Limits should
>>>>>>> be in place to avoid overwhelming the receiver with BFD messages (you had
>>>>>>>>>>>>> >      >     most cases, a single BFD session is sufficient
>>>>>>>>>>>>> for the given
>>>>>>>>>>>>> >     VTEP to
>>>>>>>>>>>>> >      >      > path continuity as stated by the draft. As
>>>>>>>>>>>>> long as there's a
>>>>>>>>>>>>> >      >     valid IP
>>>>>>>>>>>>> >      >      >      >         space while checking
>>>>>>>>>>>>> internally. I will cross
>>>>>>>>>>>>> >      >      >      >         On Wed, Jul 31, 2019 at 10:54
>>>>>>>>>>>>> AM Joel M.
>>>>>>>>>>>>> >     Halpern
>>>>>>>>>>>>> >      >      >      >         <[email-masked]
>>>>>>>>>>>>> >      >      >      >              >     specific VNI? If it
>>>>>>>>>>>>> >      >      >      >              >     OAM, e.g. CCM
>>>>>>>>>>>>> frames. The reason
>>>>>>>>>>>>> >      >      >      >              >             Also,
>>>>>>>>>>>>> earlier using VNI
>>>>>>>>>>>>> >      >      >      >              >             might be
>>>>>>>>>>>>> used as the
>>>>>>>>>>>>> >      >      >      >              >             With that
>>>>>>>>>>>>> new text, what