Subject: BFD over VXLAN: Trapping BFD Control packet at VTEP

From: Greg Mirsky <[email-masked]>
Date: Wed, 31 Jul 2019 08:48:31 -0400

Dear All,
thank you for your comments, suggestions on this issue, probably the most
challenging for this specification. In the course of our discussions, we've
agreed to abandon the request to allocate the dedicated MAC address to be
used as the destination MAC address in the inner Ethernet frame. Also,
earlier using VNI 0 was changed from mandatory to one of the options an
implementation may offer to an operator. The most recent discussion was
whether VTEP's MAC address might be used as the destination MAC address in
the inner Ethernet frame. As I recall it, the comments from VXLAN experts
equally split with one for it and one against. Hence I would like to
propose a new text to resolve the issue. The idea is to let an operator
select Management VNI and use that VNI in VXLAN encapsulation of BFD
Control packets:
NEW TEXT:

An operator MUST select a VNI number to be used as Management VNI. VXLAN
packet for Management VNI MUST NOT be sent to a tenant. VNI number 1 is
RECOMMENDED as the default for Management VNI.

With that new text, what can be the value of the destination MAC in the
inner Ethernet? I tend to believe that it can be anything and ignored by
the reciever VTEP. Also, if the trapping is based on VNI number, the
destination IP address of the inner IP packet can from the range 127/8 for
IPv4, and for IPv6 from the range 0:0:0:0:0:FFFF:7F00:0/104. And lastly,
the TTL to be set to 1 (no change here).

Much appreciate your comments, questions, and suggestions.

Best regards,
Greg

---

From: Greg Mirsky <[email-masked]>
Date: Wed, 31 Jul 2019 12:20:15 -0400

Hi Dinesh,
thank you for your consideration of the proposal and questions. What would
you see as the scope of testing the connectivity for the specific VNI? If
it is tenant-to-tenant, then VTEPs will treat these packets as regular user
frames. More likely, these could be Layer 2 OAM, e.g. CCM frames. The
reason to use 127/8 for IPv4, and 0:0:0:0:0:FFFF:7F00:0/104 for IPv6 is to
safeguard from leaking Ethernet frames with BFD Control packet to a tenant.
You've suggested using a MAC address to trap the control packet at VTEP.
What that address could be? We had proposed using the dedicated MAC and
VTEP's MAC and both raised concerns among VXLAN experts. The idea of using
Management VNI may be more acceptable based on its similarity to the
practice of using Management VLAN.

Regards,
Greg

On Wed, Jul 31, 2019 at 12:03 PM Dinesh Dutt <[email-masked]> wrote:

> Hi Greg,
>
> As long as the inner MAC address is such that the packet is trapped to the
> CPU, it should be fine for use as an inner MAC is it not? Stating that is
> better than trying to force a management VNI. What if someone wants to test
> connectivity on a specific VNI? I would not pick a loopback IP address for
> this since that address range is host/node local only. Is there a reason
> you're not using the VTEP IP as the inner IP address ?
>
> Dinesh
>
> On Wed, Jul 31, 2019 at 5:48 AM Greg Mirsky <[email-masked]> wrote:
>
>> Dear All,

---

From: Dinesh Dutt <[email-masked]>
Date: Wed, 31 Jul 2019 09:03:34 -0700

Hi Greg,

Dinesh

---

From: Santosh P K <[email-masked]>
Date: Wed, 31 Jul 2019 10:44:13 -0700

I have checked with implementation in data path. When we receive a packet
with valid VNI then lookup for MAC will happen and it is VTEP own MAC then
it will be trapped to control plane for processing. I think we can have
following options
1. Optional managment VNI
2. Mandatory inner MAC set to VTEP mac
3. Inner IP TTL set to 1 to avoid forwarding of packet via inner IP
address.

Thoughts?

Thansk
Santosh P K

On Wed, Jul 31, 2019 at 9:20 AM Greg Mirsky <[email-masked]> wrote:

>> As long as the inner MAC address is such that the packet is trapped to
>> the CPU, it should be fine for use as an inner MAC is it not? Stating that
>> is better than trying to force a management VNI. What if someone wants to
>> test connectivity on a specific VNI? I would not pick a loopback IP address
>> for this since that address range is host/node local only. Is there a
>> reason you're not using the VTEP IP as the inner IP address ?
>>
>> Dinesh
>>
>>> thank you for your comments, suggestions on this issue, probably the
>>> most challenging for this specification. In the course of our discussions,
>>> we've agreed to abandon the request to allocate the dedicated MAC address
>>> to be used as the destination MAC address in the inner Ethernet frame.
>>> Also, earlier using VNI 0 was changed from mandatory to one of the options
>>> an implementation may offer to an operator. The most recent discussion was

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 31 Jul 2019 13:53:56 -0400

Sorry to ask a stupid question.  Whose implementation?

The reason I ask is that as far as I can tell, since the tenant does not 
have any control access to the VTEP, there is no reason for the VTEP to 
have a MAC address in the tenant space.  Yes, the device has a physical 
MAC address.  But the tenant could well be using that MAC address.  Yes, 
they would be violating the Ethernet spec.  But the whole point of 
segregation is not to care about such issues.

On the other hand, if you tell me that the VMWare implementation has an 
Ethernet address that is part of the tenant space, well, they made up 
this particular game.

Yours,
Joel

On 7/31/2019 1:44 PM, Santosh P K wrote:
> I have checked with implementation in data path. When we receive a 
> packet with valid VNI then lookup for MAC will happen and it is VTEP own 
> MAC then it will be trapped to control plane for processing. I think we 
> can have following options
> On Wed, Jul 31, 2019 at 9:20 AM Greg Mirsky <[email-masked] 
> <mailto:[email-masked]>> wrote:
> 
>     Hi Dinesh,
>     thank you for your consideration of the proposal and questions. What
>     would you see as the scope of testing the connectivity for the
>     specific VNI? If it is tenant-to-tenant, then VTEPs will treat these
>     packets as regular user frames. More likely, these could be Layer 2
>     OAM, e.g. CCM frames. The reason to use 127/8 for IPv4, and
>     0:0:0:0:0:FFFF:7F00:0/104 for IPv6 is to safeguard from leaking
>     Ethernet frames with BFD Control packet to a tenant.
>     You've suggested using a MAC address to trap the control packet at
>     VTEP. What that address could be? We had proposed using the
>     dedicated MAC and VTEP's MAC and both raised concerns among VXLAN
>     experts. The idea of using Management VNI may be more acceptable
>     based on its similarity to the practice of using Management VLAN.
> 
>     Regards,
>     Greg
> 
>     On Wed, Jul 31, 2019 at 12:03 PM Dinesh Dutt <[email-masked]
>     <mailto:[email-masked]>> wrote:
> 
>         Hi Greg,
> 
>         As long as the inner MAC address is such that the packet is
>         trapped to the CPU, it should be fine for use as an inner MAC is
>         it not? Stating that is better than trying to force a management
>         VNI. What if someone wants to test connectivity on a specific
>         VNI? I would not pick a loopback IP address for this since that
>         address range is host/node local only. Is there a reason you're
>         not using the VTEP IP as the inner IP address ?
> 
>         Dinesh
> 
>         On Wed, Jul 31, 2019 at 5:48 AM Greg Mirsky
>         <[email-masked] <mailto:[email-masked]>> wrote:
> 
>             Dear All,
>             thank you for your comments, suggestions on this issue,
>             probably the most challenging for this specification. In the
>             course of our discussions, we've agreed to abandon the
>             request to allocate the dedicated MAC address to be used as
>             the destination MAC address in the inner Ethernet frame.
>             Also, earlier using VNI 0 was changed from mandatory to one
>             of the options an implementation may offer to an operator.
>             The most recent discussion was whether VTEP's MAC address
>             might be used as the destination MAC address in the inner
>             Ethernet frame. As I recall it, the comments from VXLAN
>             experts equally split with one for it and one against. Hence
>             I would like to propose a new text to resolve the issue. The
>             idea is to let an operator select Management VNI and use
>             that VNI in VXLAN encapsulation of BFD Control packets:
>             NEW TEXT:
> 
>                 An operator MUST select a VNI number to be used as
>                 Management VNI. VXLAN packet for Management VNI MUST NOT
>                 be sent to a tenant. VNI number 1 is RECOMMENDED as the
>                 default for Management VNI.
> 
>             With that new text, what can be the value of the destination
>             MAC in the inner Ethernet? I tend to believe that it can be
>             anything and ignored by the reciever VTEP. Also, if the
>             trapping is based on VNI number, the destination IP address
>             of the inner IP packet can from the range 127/8 for IPv4,
>             and for IPv6 from the range 0:0:0:0:0:FFFF:7F00:0/104. And
>             lastly, the TTL to be set to 1 (no change here).
>

---

From: Santosh P K <[email-masked]>
Date: Wed, 31 Jul 2019 11:00:04 -0700

Joel,
   Thanks for your inputs. I checked implementation within Vmware. Perhaps
I should have been more clear about MAC address space while checking
internally. I will cross check again for the same and get back on this
list.

Thanks
Santosh P K

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 31 Jul 2019 14:05:49 -0400

I made the mistake of assuming we were in agreement.
What Greg has proposed below makes very good sense to me.

Thank you Greg.
Joel

On 7/31/2019 8:48 AM, Greg Mirsky wrote:
> most challenging for this specification. In the course of our 
> discussions, we've agreed to abandon the request to allocate the 
> dedicated MAC address to be used as the destination MAC address in the 
> inner Ethernet frame. Also, earlier using VNI 0 was changed from 
> mandatory to one of the options an implementation may offer to an 
> operator. The most recent discussion was whether VTEP's MAC address 
> might be used as the destination MAC address in the inner Ethernet 
> frame. As I recall it, the comments from VXLAN experts equally split 
> with one for it and one against. Hence I would like to propose a new 
> text to resolve the issue. The idea is to let an operator select 
> Management VNI and use that VNI in VXLAN encapsulation of BFD Control 
> packets:
> NEW TEXT:
> 
>     An operator MUST select a VNI number to be used as Management VNI.
>     VXLAN packet for Management VNI MUST NOT be sent to a tenant. VNI
>     number 1 is RECOMMENDED as the default for Management VNI.
> 
> destination IP address of the inner IP packet can from the range 127/8 
> for IPv4, and for IPv6 from the range 0:0:0:0:0:FFFF:7F00:0/104. And

---

From: Dinesh Dutt <[email-masked]>
Date: Wed, 31 Jul 2019 11:25:21 -0700

Hi Greg,

> practice of using Management VLAN.

If you use the inner IP address as the VTEP IP address, then use the MAC
address that the VTEP would respond with when replying to an ARP for that
VTEP IP address. If a VXLAN expert disagrees with this, could you kindly
tell me who it is so that I can understand their disagreement? So this
handles the case where the VNI is not a user-tenant VNI. If the VNI used in
the BFD packet is a user-tenant VNI, then the receiving VTEP MUST have an
IP address in that VNI (mapped to a VRF) else you cannot use that VNI in
the BFD packet. Why won't this combination address all the cases you've
listed? What am I missing? Define VNI 1 as a possible use, not VNI 0. I
objected to VNI 0 because there are too many switching siicon out there and
some of them will not be able to handle this scenario.

Dinesh

---

From: Greg Mirsky <[email-masked]>
Date: Wed, 31 Jul 2019 15:07:28 -0400

Hi Dinesh,
if I recall correctly, T.Sridhar has noted that VTEP's MAC must not be used
as the destination MAC address in the inner Ethernet frame. Also, I should
have been more precise in the proposed text, please see the updated version
to stress that the management VNI MUST NOT be one of the tenant's VNIs:
NEW TEXT:

An operator MUST select a VNI number to be used as Management VNI.
Management VNI number MUST NOT be one of the tenant's VNIs to prevent
sending VXLAN packets received on Management VNI to a tenant. VNI number 1
is RECOMMENDED as the default for Management VNI.

On Wed, Jul 31, 2019 at 2:25 PM Dinesh Dutt <[email-masked]> wrote:

>> would you see as the scope of testing the connectivity for the specific
>> VNI? If it is tenant-to-tenant, then VTEPs will treat these packets as
>> regular user frames. More likely, these could be Layer 2 OAM, e.g. CCM
>> frames. The reason to use 127/8 for IPv4, and 0:0:0:0:0:FFFF:7F00:0/104 for
>> IPv6 is to safeguard from leaking Ethernet frames with BFD Control packet
>> to a tenant.
>>>> VXLAN packet for Management VNI MUST NOT be sent to a tenant. VNI number 1
>>>> is RECOMMENDED as the default for Management VNI.
>>>>