Subject: Benjamin Kaduk's Discuss on draft-ietf-bfd-vxlan-09: (with DISCUSS and COMMENT)

From: Benjamin Kaduk via Datatracker <[email-masked]>
Date: Mon, 16 Dec 2019 15:43:13 -0800

Benjamin Kaduk has entered the following ballot position for
draft-ietf-bfd-vxlan-09: Discuss

When responding, please keep the subject line intact and reply to all
email addresses included in the To and CC lines. (Feel free to cut this
introductory paragraph, however.)

Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
for more information about IESG DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-bfd-vxlan/

----------------------------------------------------------------------
DISCUSS:
----------------------------------------------------------------------

I have a few points that I think merit IESG discussion.

(1) I see that several directorate reviewers expressed unease at the
destination (IP and) MAC address assignment procedure for the inner
VXLAN headers, and appreciate that there was extensive on-list
discussion (more than I could follow).  That said, I failed to find a
clear statement of why the current text is believed to be safe, and in
fact my reading of the current text is that the described procedure is
*not* safe.  Pointers to key parts of the WG discusison would be more
than welcome!

To take something of a high-level view of my concerns, if we think of
the VXLAN as being a tunnel between VTEPs that carry encapsulated tenant
traffic, then what we're trying to do is roughly like BFD between VTEPs,
but we want to get fault-detection over as broad a coverage as we can
(the "outermost part of the tunnel"), so we want to have the option of
per-VNI BFD instead of just endpoint-to-endpoint (VTEP-to-VTEP).
However, we end up having to do this by trying to insert a thin filter
into the tenant's address space (i.e., the inner VXLAN header) and pick
out the specific stream of BFD traffic that we're introducing.  This is,
in some sense, a namespace grab in what is conceptually the tenant's
namespace, and we have to be careful that what we do is either
guaranteed to not impact the tenant or well-documented and
compartmentalized (akin to the "well-known URIs").

I've made comments at several places in the document that are more
directly tied to specific pieces of text, but in general, if we assume
that the tenant can add/remove new addresses at will within their VXLAN
abstration, then any attempt to preconfigure by mutual agreement the BFD
addresses to use at the VTEPs or to use the VTEP's normal (outer)
address as the sentinel value seems subject to the tenant coming in and
subsequently trying to use that address, leading to (some of) the
tenant's traffic getting silently filtered and interpreted by the VTEP.
If we were using domain names as identifiers, we could allocate
something under .arpa or similar, but I think our options are more
limited when numerical addresses are used.

The option suggested by the rtg-dir reviewer of always using the
management VNI does not suffer from this namespacing issue, though I
recognize that it does reduce the scope over which fault-detection is
available, for the cases when different VNIs' traffic are routed or
handled differently.

(2) Section 6 says:

                                                         The selection
   of the VNI number of the Management VNI MUST be controlled through
   management plane.  An implementation MAY use VNI number 1 as the
   default value for the Management VNI.  All VXLAN packets received on
   the Management VNI MUST be processed locally and MUST NOT be
   forwarded to a tenant.

It seems like the management VNI concept is something that would apply
to the entire VXLAN deployment and not just to the BFD-using portions;
is this already defined somewhere (in which case we should reference
it), or is it new with this document?  In the latter case wouldn't it be
an update to the core VXLAN spec?  (I note that there are some
procedural hoops to jump through for an IETF-stream document to update
an ISE-stream document...)

----------------------------------------------------------------------
COMMENT:
----------------------------------------------------------------------

Section 1

   In the case where a Multicast Service Node (MSN) (as described in
   Section 3.3 of [RFC8293]) resides behind a Network Virtualization
   Endpoint (NVE), the mechanisms described in this document apply and
   can, therefore, be used to test the connectivity from the source NVE
   to the MSN.

I'm not sure that I'm parsing "resides behind" properly.  Is the idea
that the multicast traffic starts off at a tenant-system source, hits a
NVE gateway to enter the VXLAN, traverses the VXLAN a bit before getting
to the MSN, and is replicated from the MSN to various NVE termini?  I
think I'd be less confused if this was described as "participates in the
VXLAN" or "is part of the virtualized environment", as the current
"behind" wording makes me think of a firewall-like topology where the
NVE behind which the MSN resides will be decapsulating traffic.

   This document describes the use of Bidirectional Forwarding Detection
   (BFD) protocol to enable monitoring continuity of the path between
   VXLAN VTEPs, performing as Network Virtualization Endpoints, and/or
   availability of a replicator multicast service node.

All the commas here potentially make the parsing ambiguous; assuming
that the "performing as Network Virtualization Endpoints" is just
describing the VXLAN VTEPs, I'd suggest do drop the first comma and
instead join those clauses with "that are".

Section 3

   between the same pair of VTEPs.  BFD packets intended for a VTEP MUST
   NOT be forwarded to a VM as a VM may drop BFD packets leading to a
   false negative.  This method is applicable whether the VTEP is a

[This "MUST NOT" is a very strict requirement, so we have to be sure that
it's achievable without disruption to tenant traffic, per the Discuss
point]

   At the same time, a service layer BFD session may be used between the
   tenants of VTEPs IP1 and IP2 to provide end-to-end fault management.
   In such case, for VTEPs BFD Control packets of that session are
   indistinguishable from data packets.

nit(?): I suggest s/indistinguishable from/regular/ -- the tenants' BFD
sessions are just regular data to the VXLAN infrastructure, though IIUC
a VTEP could, if so inclined, peek inside and "distinguish" them from
non-BFD tenant data based on on heuristics and packet format.

   0:0:0:0:0:FFFF:7F00:0/104 range for IPv6).  There could be a firewall
   configured on VTEP to block loopback addresses if set as the
   destination IP in the inner IP header.  It is RECOMMENDED to allow
   addresses from the loopback range through a firewall only if it is
   used as the destination IP address in the inner IP header, and the
   destination UDP port is set to 3784 [RFC5881].

I think we should reword this to make it clear that the default behavior
is still "block all incoming traffic with loopback destination" and that
the exception is tightly scoped to the encapsulated VXLAN traffic
discussed in this document and the specific destination port *and when
BFD has been configured for the VTEP*.  I note that well-known ports are
not reserved ports, and we have no guarangee that only a BFD
implementation would be listening on port 3784.
I think the rewording would include some phrasing like "RECOMMENDED that
the only firewall exception to allow incoming traffic with destination
address from the loopback range is when [...]", and of course, mention
the need to have BFD configured.

Section 4

   VXLAN packet.  The choice of Destination MAC and Destination IP
   addresses for the inner Ethernet frame MUST ensure that the BFD
   Control packet is not forwarded to a tenant but is processed locally
   at the remote VTEP.  [...]

This has to be 100% reliable, and I think we need to provide some
example mechanism that has that property even if we don't mandate that
it be the only allowed mechanism.

         Destination MAC: This MUST NOT be of one of tenant's MAC
         addresses.  The destination MAC address MAY be the address

But the tenant can start using new MAC addresses at any time!  How is
BFD-over-VXLAN going to dynamically detect and avoid that?

         associated with the destination VTEP.  The MAC address MAY be
         configured, or it MAY be learned via a control plane protocol.
         The details of how the MAC address is obtained are outside the
         scope of this document.

This all talks about the MAC address being relatively static
configuration, but per above, I don't think that's safe in the face of a
MUST-level requirement to avoid conflicting with tenant MAC addresses.

      IP header:

         Destination IP: IP address MUST NOT be of one of tenant's IP
         addresses.  The IP address SHOULD be selected from the range
         127/8 for IPv4, for IPv6 - from the range
         0:0:0:0:0:FFFF:7F00:0/104.  Alternatively, the destination IP
         address MAY be set to VTEP's IP address.

As for MAC addresses, can't the tenant start using new ones at any time?
Loopback is mostly safe in that the tenant generally shouldn't expect
incoming traffic to that destination address ... but what if the tenant
is also using a BFD scheme that expects incoming (single-hop) packets to
loopback as an exception to RFC 1122?
nit: please use a parallel grammatical construction for describing the
IPv4 and IPv6 recommended behavior.

         TTL or Hop Limit: MUST be set to 1 to ensure that the BFD
         packet is not routed within the Layer 3 underlay network.  This
         addresses the scenario when the inner IP destination address is
         of VXLAN gateway and there is a router in underlay which
         removes the VXLAN header, then it is possible to route the
         packet as VXLAN  gateway address is routable address.

nit: the grammar here is a bit wonky; I think the following preserves
the meaning with better grammar:

%        TTL or Hop Limit: MUST be set to 1 to ensure that the BFD
%        packet is not routed within the Layer 3 underlay network.  This
%        addresses the scenario where the inner IP destination address is
%        that of a VXLAN gateway and there is a router in the underlay
%        that removes the VXLAN header; in such cases it is possible for
%        the packet to be routed, as the VXLAN gateway's address is a
%        routable address.

Section 5

   Once a packet is received, VTEP MUST validate the packet.  If the
   Destination MAC of the inner Ethernet frame matches one of the MAC
   addresses associated with the VTEP the packet MUST be processed
   further.  If the Destination MAC of the inner Ethernet frame doesn't

What prevents the scenario where the MAC address associated with the
VTEP is also in use by the tenant?

   match any of VTEP's MAC addresses, then the processing of the
   received VXLAN packet MUST follow the procedures described in
   Section 4.1 [RFC7348].  If the BFD session is using the Management
   VNI (Section 6), BFD Control packets with unknown MAC address MUST
   NOT be forwarded to VMs.

nit: either "an unknown" or "MAC addresses"

   The UDP destination port and the TTL of the inner IP packet MUST be
   validated to determine if the received packet can be processed by
   BFD.

Can you give a pointer to or description of what this validation
consists of?

Section 5.1

   case of VXLAN, the VNI number identifies that logical link.  If BFD
   packet is received with non-zero Your Discriminator, then BFD session
   MUST be demultiplexed only with Your Discriminator as the key.

nits: "If a BFD packet", "then the BFD session"

Section 6

   In most cases, a single BFD session is sufficient for the given VTEP
   to monitor the reachability of a remote VTEP, regardless of the
   number of VNIs.  When the single BFD session is used to monitor the
   reachability of the remote VTEP, an implementation SHOULD choose any
   of the VNIs.  An implementation MAY support the use of the Management

nit: I feel like this is trying to say that the choice is arbitrary and it
doesn't matter which one is picked, but "SHOULD choose any of" is more
of a recommendation to make a choice than guidance on how to make that
choice, as written.

Section 9

I think we need to discuss the risk/potential consequences of a VTEP
failing to properly filter BFD traffic and incorrectly passing it
through to the tenant.

Relatedly, I'd also consider discussing the case of a mixed deployment
where one peer attempts to speak BFD-VXLAN to a peer that does not
implement that mechanism.

   The document requires setting the inner IP TTL to 1, which could be
   used as a DDoS attack vector.  Thus the implementation MUST have

An attack vector on what part of the system?

---

From: Jeffrey Haas <[email-masked]>
Date: Wed, 18 Dec 2019 15:24:48 -0500

Benjamin,

> Benjamin Kaduk has entered the following ballot position for

One high level point that likely didn't survive the rather verbose comment
chain is there are two implementations of this draft.  Some of the
considerations covered in the guidance here is "please don't break shipping
code".

While this is IETF, and shipping code isn't always a blocking point to
document changes, I'd suggest that as a consideration.

> per-VNI BFD instead of just endpoint-to-endpoint (VTEP-to-VTEP).

You've summarized this clearly.  Joel Halpern, in particular, raised this
point multiple times.  Effectively, "what are we testing?"  And the response
not clearly converging on exactly one of the two possibilities.

As is noted in the various IESG discussion, each of the two test points
raise slightly different considerations.

> compartmentalized (akin to the "well-known URIs").

Possibly, and it's certainly a consideration.  However, I think I'm less
convinced of it being quite the level of violation that seems to be
reflected in the rest of the IESG comments in the various other threads.
I'll respond to that detail a bit below.

> available, for the cases when different VNIs' traffic are routed or

This is a clean summary of the considerations.  At least a portion of the WG
seems to be comfortable with "test to the management VNI".  However, another
(smaller, I believe) portion were wanting to test one layer further in.

> an ISE-stream document...)

The relevant portion of the archive will have the Subject: line text
including:
"Trapping BFD Control packet at VTEP"

A portion of the discussion relating to the magic number of the management
VNI suggested '1', instead of '0'.

At least some implementations already use '0':
https://mailarchive.ietf.org/arch/msg/rtg-bfd/6WfSATmfoPv4AD6RmD-Xb7zz4CE

The argument to not use '0' starts roughly here:
https://mailarchive.ietf.org/arch/msg/rtg-bfd/z8E_a5k_r4pLLs5YfNsL_Xm9_Us

You're correct, IMO, that there's no standard practice and the above seems
to support this.  I believe this leaves the document authors in the position
of being requested to make a recommendation for the default value of this
field and knowing that the default would be invalid on some platforms.

The alternative is requiring implementations to always configure this value.

I suggest the IESG determine whether it wants a default value here or not.
If not, the text should be adjusted to require configuration.  If yes, the
IESG should consider whether the nvo3 group should produce some document
that covers current operational practices.

I don't think this consideration is necessarily critical.

BFD implementations residing in the related instance communicating to other
instances across the vxlan environment would be using RFC 5881 or RFC 5883
style BFD.  Since this isn't a tunneled BFD, the IP endpoints of the BFD
control traffic will be unicast addresses rather than the reserved
"loopback" ranges; i.e. 127/8 ::FFFF:127.0.0.0/104.  In order for those
ranges to be problematic, it'd be necessary for the client to be able to
manually encapsulate a vxlan packet - a security issue of its own.

A related point in this discussion is "we're hijacking an address managed by
the local tenant".  While true, it's in the above ranges and thus somewhat
under the auspice of the host OS to assert control.  I'm aware of some
unusual applications that make use of configured addresses in those ranges
for on-box communications, but they're also on the unusual end of things.

What sort of text would you want to cover the case that when BFD is run
up-to-the-tenant mode in this circumstance that an address MUST be reserved
for the BFD over vxlan application and that this address SHALL NOT be
available to the tenant for its own use?

> it be the only allowed mechanism.

The consideration here, I believe, is that there's currently too much
latitude by implementations as to what MAC addresses they use here.
Restrict one case, you may break some implementation.

The missing element is how a pair of implementations of BFD for vxlan
discover the necessary information?  As far as BFD is concerned, "tell me!"
This seems like work that belongs in nvo3.

> BFD-over-VXLAN going to dynamically detect and avoid that?

See above.  Either it's coordinated with the ability to prevent the tenant
from using it or the underlying vxlan environment needs to provide some
mechanism to discover what's been provisioned.

> MUST-level requirement to avoid conflicting with tenant MAC addresses.

But is it BFD's responsibility to figure this out?  This is what the
document is suggesting - a higher level with access to the implementation
specifics should be supplying the BFD provisioning information.  Or manual
provisioning in the absence thereof.

-- Jeff

---

From: Benjamin Kaduk <[email-masked]>
Date: Wed, 18 Dec 2019 14:02:46 -0800

Hi Jeff,

I think I can only touch on a few points before telechat-time rolls around,
and will finish off afterwards.

> document changes, I'd suggest that as a consideration.

It did indeed not survive (at least my pass through) the comment chain, so
thank you for calling it out.  It is indeed a consideration, and I expect
some actual discussion on the call tomorrow.

> (smaller, I believe) portion were wanting to test one layer further in.

It is reassuring that I at least managed to summarize the situation
tolerably.  Is it fair to say that testing "one layer further in" is a
superset of what "test to the managemenet VNI" can do?

> that covers current operational practices.

That does sound like something we should try to talk about on the telechat
as well; thanks for raising it so clearly.

> I don't think this consideration is necessarily critical.

I think I'm in agreement about its criticality, and will see if I can come
up with some actual text ... later.

-Ben

---

From: Jeffrey Haas <[email-masked]>
Date: Wed, 18 Dec 2019 17:18:24 -0500

Ben,

> superset of what "test to the managemenet VNI" can do?

Fundamentally, this is all BFD.  The issue is almost always considerations
related to encapsulation.

The meta concern here is that if you test to the management VNI, the
operator has a lot of control over things that are clean from a security
perspective.

The minute you test one layer deeper, it's still the same thing... but you
now have a lot of sharp edges you have to worry about.

In all of these situations, the main consideration from a security
perspective and an encapsulation perspective is "don't step on the toes of
the user".  But that said, vxlan environments are provided to contain
tenants, have their own provisioning ecosystems, and security considerations
in how they are provisioned and operated.  As long as the security and
operational considerations are understood by the operator, they can decide
whether "testing one layer further in" gives them good benefit vs. the
additional security considerations.

And that said, two fundamental portions of BFD operations and security still
apply here:
- Discriminators need to be known to mess with existing sessions.  This
  means the main consideration for someone not in the tenant environment is
  privacy.  Such privacy is an overall consideration for vxlan environments.
- Authentication mechanisms in BFD may still be deployed which further
  reduce the attack space.

Basically, if your vxlan environment is appropriately operated and secured,
the main attacker of this session is the tenant itself.  And they have all
sorts of bad things they can do to knock down their own reachability from
one VTEP to another.  I.e. it's a stupid attack. :-)

-- Jeff

---

From: Benjamin Kaduk <[email-masked]>
Date: Thu, 19 Dec 2019 17:45:52 -0800

Hi Jeff,

> one VTEP to another.  I.e. it's a stupid attack. :-)

I think in this message you've managed to get a pithy form of both main
flavors of my concern: "don't step on the toes of the user" and "[the
tenant has] all sorts of bad things they can do to knock down their own
reachability" :)

Less pithily, to keep the isolation between tenant and infrastructure, the
tenant can't send stuff that gets interpreted by the infrastructure [as
directed at the infrastructure], and the infrastructure can't intercept and
parse as directed at itself tenant traffic.  As you note, the former is
likely not a terribly interesting attack, as the scope ought to be limited
to just knocking out the tenant's own connection.  So I'm more concerned
about the latter case -- making sure the infrastructure doesn't eat up
something the tenant wanted to send over the tunnel.  That includes all
sorts of weird things tenants might try to do, that might not even be fully
in spec (depending on what the contract between tenant and provider looks
like, I suppose).  There's several technologies involved here that I don't
have a great handle on, so I'm hoping to see an explanation of how that's
prevented, in a simple enough fashion for my tiny brain to take in.

-Ben

---

From: Benjamin Kaduk <[email-masked]>
Date: Thu, 19 Dec 2019 18:00:44 -0800

> and will finish off afterwards.

as promised...

> as well; thanks for raising it so clearly.

We did get to talk about it briefly, though without a clear conclusion.
Since there's not much of a clear preexisting description of the management
VNI that we can cite, I'm leaning towards just noting that (most?)
implementations offer a concept of "management VNI", that the management
VNI can be useful for giving confidence that BFD traffic does not interfere
with tenant traffic, and the details of how to use the management VNI are
implementation-specific.  But it would not be hard to convince me that
there is a better path to take!

> up with some actual text ... later.

I think the sense I have is something like:

% It is common to have a firewall configured on the VTEP (akin to general
% common practice for all machines) to drop incoming traffic where the
% inner IP header contains a loopback address as the destination address.
% In general, such traffic would be the result of misconfiguration, and
% such a policy improves network safety.  However, the procedures specified
% in this document can result in such traffic, so it is RECOMMENDED to only
% allow incoming traffic with loopback-range inner destination IP address
% when the destination UDP port is set to 3784 and BFD has been configured
% on the VTEP.

But if that's not correct, no need to spend more time on it.

> > available to the tenant for its own use?

I'd like to hold off on proposing text until some of the other threads come
to a conclusion; what mechanism we want to use (and thus write about) may
depend on what scenarios we're trying to prevent.

> > This seems like work that belongs in nvo3.

That seems likely.
I suspect that what we'll end up with here is something akin to "here's a
procedure you can use that will work, but if you want to use some other
procedure that works, we aren't stopping you".

> > provisioning in the absence thereof.

I don't really disagree; I'm more trying to get a sense of which parts are
dynamic and which parts (relatively) static -- my current sense is that
everything in the tenant space can be very dynamic, but that the BFD
provisioning is fairly static.  That would lead to something of an
impedance mismatch if the dynamic stuff the tenant is doing could generate
a conflict with the static configuration -- it would be a sign that we're
doing something at the wrong level.  But, as implied earlier, I'm still
trying to wrap my head around the system as a whole.

Thanks,

Ben