Subject: Re: Tsvart last call review of draft-ietf-bfd-vxlan-07

From: Greg Mirsky <[email-masked]>
Date: Mon, 17 Jun 2019 13:58:05 +0900

Hi Oliver,
thank you for your thorough review, clear and detailed questions. My
apologies for the delay to respond. Please find my answers below in-line
tagged GIM>>.

Regards,
Greg

On Fri, May 31, 2019 at 12:38 PM Olivier Bonaventure via Datatracker <
[email-masked]> wrote:

> Reviewer: Olivier Bonaventure
> Review result: Ready with Issues
>
> This document has been reviewed as part of the transport area review team's
> ongoing effort to review key IETF documents. These comments were written
> primarily for the transport area directors, but are copied to the
> document's
> authors and WG to allow them to address any issues raised and also to the
> IETF
> discussion list for information.
>
> When done at the time of IETF Last Call, the authors should consider this
> review as part of the last-call comments they receive. Please always CC
> [email-masked] if you reply to or forward this review.
>
> I have only limited knowledge of VXLAN and do not know all subtleties of
> BFD.
> This review is thus more from a generalist than a specialist in this topic.
>
> Major issues
>
> Section 4 requires that " Implementations SHOULD ensure that the BFD
>    packets follow the same lookup path as VXLAN data packets within the
>    sender system."
>
> Why is this requirement only relevant for the lookup path on the sender
> system
> ? What does this sentence really implies ?
>
GIM>> RFC 5880 set the scope of the fault detection of BFD protocol as
   ... the bidirectional path between two forwarding engines, including
   interfaces, data link(s), and to the extent possible the forwarding
   engines themselves ...
The requirement aimed to the forwarding engine of a BFD system that
transmits BFD control packets over VXLAN tunnel.

>
> Is it a requirement that the BFD packets follow the same path as the data
> packet for a given VXLAN ? I guess so. In this case, the document should
> discuss how Equal Cost Multipath could affect this.
>
GIM>> I think that ECMP environment is more likely to be experienced by a
transit node in the underlay. If the BFD session is used to monitor the
specific underlay path, then, I agree, we should explain that using the
VXLAN payload information to draw path entropy may cause data and BFD
packets following different underlay routes. But, on the other hand, that
is the case for OAM and fault detection in all overlay networks in general.

>
> Minor issues
>
> Section 1
>
> You write "The asynchronous mode of BFD, as defined in [RFC5880],
>  can be used to monitor a p2p VXLAN tunnel."
>
> Why do you use the word can ? It is a possibility or a requirement ?
>
GIM>> In principle, BFD Demand mode may be used to monitor p2p paths as
well, I agree, will re-word to more assertive:
 The asynchronous mode of BFD, as defined in [RFC5880],
 is used to monitor a p2p VXLAN tunnel.
>
>
> NVE has not been defined before and is not in the terminology.
>
GIM>> Will add to the Terminology and expand as:
NVE        Network Virtualization Endpoint

>
> This entire section is not easy to read for an outsider.
>
> Section 3
>
> VNI has not been defined
>
GIM>> Will add to the Terminology section:
VNI    VXLAN Network Identifier (or VXLAN Segment ID)

>
> Figure 1 could take less space
>
GIM>> Yes, can make it bit denser. Would the following be an improvement?

      +------------+-------------+
      |        Server 1          |
      | +----+----+  +----+----+ |
      | |VM1-1    |  |VM1-2    | |
      | |VNI 100  |  |VNI 200  | |
      | |         |  |         | |
      | +---------+  +---------+ |
      | Hypervisor VTEP (IP1)    |
      +--------------------------+
                            |
                            |   +-------------+
                            |   |   Layer 3   |
                            +---|   Network   |
                                +-------------+
                                    |
                                    +-----------+
                                                |
                                         +------------+-------------+
                                         |    Hypervisor VTEP (IP2) |
                                         | +----+----+  +----+----+ |
                                         | |VM2-1    |  |VM2-2    | |
                                         | |VNI 100  |  |VNI 200  | |
                                         | |         |  |         | |
                                         | +---------+  +---------+ |
                                         |      Server 2            |
                                         +--------------------------+

> Section 4
>
> I do not see the benefits of having one paragraph in Section 4 followed by
> only
> Section 4.1
>
GIM>> Will merge Section 4.1 into 4 with minor required re-wording:
4.  BFD Packet Transmission over VXLAN Tunnel

   BFD packet MUST be encapsulated and sent to a remote VTEP as
   explained in this section.  Implementations SHOULD ensure that the
   BFD packets follow the same lookup path as VXLAN data packets within
   the sender system.

   BFD packets are encapsulated in VXLAN as described below.  The VXLAN
   packet format is defined in Section 5 of [RFC7348].  The Outer IP/UDP
   and VXLAN headers MUST be encoded by the sender as defined in
   [RFC7348].

>
> Section 4.1
>
> The document does not specify when a dedicated MAC address or the MAC
> address
> of the destination VTEP must be used. This could affect the
> interoperability of
> implementations. Should all implementations support both the dedicated MAC
> address and the destination MAC address ?
>
GIM>> After further discussion, authors decided to remove the request for
the dedicated MAC address allocation. Only the MAC address of the remote
VTEP must be used as the destination MAC address in the inner Ethernet
frame. Please check the attached diff between the -07 and the working
versions or the working version of the draft.

>
> It is unclear from this section whether IPv4 inside IPv6 and the opposite
> should be supported or not.
>
GIM>> Any combination of outer IPvX and inner IPvX is possible.

>
> Section 5.
>
> If the received packet does not match the dedicated MAC address nor the MAC
> address of the VTEP, should the packet be silently discarded or treated
> differently ?
>
GIM>> As I've mentioned earlier, authors have decided to remove the use of
the dedicated MAC address for BFD over VXLAN.

>
> Section 5.1
>
> Is this a modification to section 6.3 of RFC5880 ? This is not clear
>
GIM>> I think that this section is not modification but the definition of
the application-specific procedure that is outside the scope of RFC 5880:
   The method of demultiplexing the initial packets (in which Your
   Discriminator is zero) is application dependent, and is thus outside
   the scope of this specification.

>
> Section 9
>
> The sentence " Throttling MAY be relaxed for BFD packets
>    based on port number." is unclear.
>
GIM>> Yes, thank you for pointing to this. The updated text, in the whole
paragraph, is as follows:
NEW TEXT:
   The document requires setting the inner IP TTL to 1, which could be
   used as a DDoS attack vector.  Thus the implementation MUST have
   throttling in place to control the rate of BFD control packets sent
   to the control plane.  On the other hand, over aggressive throttling
   of BFD control packets may become the cause of the inability to form
   and maintain BFD session at scale.  Hence, throttling of BFD control
   packets SHOULD be adjusted to permit BFD to work according to its
   procedures.

---

From: Greg Mirsky <[email-masked]>
Date: Thu, 20 Jun 2019 10:09:20 +0900

Hi Carlos,
thank you for reminding of our continued discussion with Joel. We are
seeking comments from VXLAN experts and much appreciate if you have
insights on VXLAN to share.
I've got some clarifying questions before I can respond to you. To which
stage of the three-way handshake you refer as "initial demultiplexing"? I
couldn't find this term in RFC 5880.
Regarding the applicability of the Echo mode, thank you for pointing to the
need for stricter terminology, the Echo mode, as defined in RFC 5880, is
underspecified and it will require additional standardization. Future
drafts may explore and define how the Echo mode of BFD is used over VXLAN
tunnels.

Will review and respond to the remaining questions soon.

Regards,
Greg

On Thu, Jun 20, 2019 at 9:14 AM Carlos Pignataro (cpignata) <
[email-masked]> wrote:

> Hi,
>
> I have not reviewed this draft before, but triggered by this email, and
> briefly scanning through a couple of sections, it is unclear to me how some
> of the mechanics work.
>
> There are some major issues with the Mac usage and association, as Joel
> Halpern mentioned in his Rtg Dir review.
>
> And, additionally, please consider the following comments and questions:
>
>
> 1. Underspecification for initialization and initial demultiplexing.
>
> This document allows multiple BFD sessions between a single pair of VTEPs:
>
>    An
>    implementation that supports this specification MUST be able to
>    control the number of BFD sessions that can be created between the
>    same pair of VTEPs.
>
> The implication of this is that BFD single-hop initialization procedures
> will not work. Instead, there is a need to map the initial demultiplexing.
>
> This issue is explained in RFCs 5882 and 5883:
> https://tools.ietf.org/html/rfc5883#section-4 and
> https://tools.ietf.org/html/rfc5882#section-6
>
> Section 5.1 says:
>
>    For such packets, the BFD session MUST be identified
>    using the inner headers, i.e., the source IP, the destination IP, and
>    the source UDP port number present in the IP header carried by the
>    payload of the VXLAN encapsulated packet.  The VNI of the packet
>    SHOULD be used to derive interface-related information for
>    demultiplexing the packet.
>
> But this does not really explain how to do the initial demultiplexing.
> Does each BFD session need to have a separate inner source IP address? Or
> source UDP port? And how ofter are they recycled or kept as state? How are
> these mapped?
> Equally importantly, which side is Active?
> And what if there’s a race condition with both sides being Active and
> setting up redundant sessions?
>
> 1.b. By the way, based on this, using S-BFD [RFC 7880] might be easier to
> demux.
>
>
> 2. Security
>
> This document says that the TTL in the inner packet carrying BFD is set to
> 1. However, RFC 5880 says to use GTSM [RFC 5082], i.e., a value of 255.
>
> Why is GTSM not used here?
>
>
> 3. ECMP and fate-sharing under-specification:
>
> Section 4.1. says:
>
>    The Outer IP/UDP
> And RFC 7348 says:
>       -  Source Port:  It is recommended that the UDP source port number
>          be calculated using a hash of fields from the inner packet --
>          one example being a hash of the inner Ethernet frame's headers.
>          This is to enable a level of entropy for the ECMP/load-
>          balancing of the VM-to-VM traffic across the VXLAN overlay.
>          When calculating the UDP source port number in this manner, it
>          is RECOMMENDED that the value be in the dynamic/private port
>          range 49152-65535 [RFC6335].
>
>
> Based on this, depending on the hashing calculation, the outer source UDP
> port can be different leading to different ECMP treatment. Does something
> else need to be specified here in regards to the outer UDP source port?
>
>
> 4. Section 7 says that “ Support for echo BFD is outside the scope of this
> document”.
>
> Assuming this means “BFD Echo mode”, why is this out of scope? If this is
> a single logical hop underneath VXLAN, what’s preventing the use of Echo?
> Echo’s benefits are huge.
>
>
> 5. Terminology
>
>    Implementations SHOULD ensure that the BFD
>    sender system.
> What is a “look up path within a sender system”?
>
>
> 6. Deployment scenarios
>
> S3 says:
>    Figure 1 illustrates the scenario with two servers, each of them
>    hosting two VMs.  The servers host VTEPs that terminate two VXLAN
> […]
>                      Figure 1: Reference VXLAN Domain
>
>
> However, RFC 7348 Figure 3 lists that as one deployment scenario, not as
> “the scenario” and “The Reference VXLAN Domain”.
>
> Best,
>
> Carlos.
>
> On Jun 17, 2019, at 12:58 AM, Greg Mirsky <[email-masked]> wrote:
>
> Hi Oliver,
>> I do not see the benefits of having one paragraph in Section 4 followed
>> by only
>> Section 4.1
>>
> <draft-ietf-bfd-vxlan-08.txt><Diff_ draft-ietf-bfd-vxlan-07.txt -
> draft-ietf-bfd-vxlan-08.txt.html>
>
>
>

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Thu, 20 Jun 2019 01:31:32 +0000

Hello, Greg,

> On Jun 19, 2019, at 9:09 PM, Greg Mirsky <[email-masked]> wrote:
> 
> Hi Carlos,
> thank you for reminding of our continued discussion with Joel. We are seeking comments from VXLAN experts and much appreciate if you have insights on VXLAN to share.
> I've got some clarifying questions before I can respond to you.

Sure.

> To which stage of the three-way handshake you refer as "initial demultiplexing"? I couldn't find this term in RFC 5880.

“Initial demultiplexing" is a well-known term in BFD, referring to the "demultiplexing of the initial packets", BFD Control packet with YourDisc being zero.

In RFC 5880, see Section 6.3.
https://tools.ietf.org/html/rfc5880#section-6.3

Since initial demultiplexing is indeed application specific, different for one-hop versus multi-hop and dependent upon whether a single or multiple sessions are allowed between a pair of endpoints, I added below two other relevant citations, from application specific BFD specs:
1. https://tools.ietf.org/html/rfc5883#section-4 
2. https://tools.ietf.org/html/rfc5882#section-6

> Regarding the applicability of the Echo mode, thank you for pointing to the need for stricter terminology, the Echo mode, as defined in RFC 5880, is underspecified and it will require additional standardization.

No. BFD Echo is not underspecified in RFC 5880.

Please read S5: https://tools.ietf.org/html/rfc5880#section-5

   BFD Echo packets are sent in an encapsulation appropriate to the
   environment.  See the appropriate application documents for the
   specifics of particular environments.

BFD Echo is application dependent. 

Therefore, for example, single-hop BFD in RFC 5881 specifies BFD Echo for that application.

Hence, my question stands: why is this draft claiming BFD Echo is out of scope for this BFD application document?

> Future drafts may explore and define how the Echo mode of BFD is used over VXLAN tunnels.
> 

See above.

> Will review and respond to the remaining questions soon.

Thank you. 

The "remaining questions" are still all the questions below :-)

Best,

Carlos.

> 
> Regards,
> Greg
> 
> 
> On Thu, Jun 20, 2019 at 9:14 AM Carlos Pignataro (cpignata) <[email-masked]> wrote:
> Hi,
> 
> I have not reviewed this draft before, but triggered by this email, and briefly scanning through a couple of sections, it is unclear to me how some of the mechanics work.
> 
> There are some major issues with the Mac usage and association, as Joel Halpern mentioned in his Rtg Dir review.
> 
> The implication of this is that BFD single-hop initialization procedures will not work. Instead, there is a need to map the initial demultiplexing.
> 
> This issue is explained in RFCs 5882 and 5883: https://tools.ietf.org/html/rfc5883#section-4 and https://tools.ietf.org/html/rfc5882#section-6
> 
> Section 5.1 says:
> 
> But this does not really explain how to do the initial demultiplexing. Does each BFD session need to have a separate inner source IP address? Or source UDP port? And how ofter are they recycled or kept as state? How are these mapped?
> And what if there’s a race condition with both sides being Active and setting up redundant sessions?
> 
> 1.b. By the way, based on this, using S-BFD [RFC 7880] might be easier to demux.
> 
> 
> 2. Security
> 
> This document says that the TTL in the inner packet carrying BFD is set to 1. However, RFC 5880 says to use GTSM [RFC 5082], i.e., a value of 255..
> 
> Based on this, depending on the hashing calculation, the outer source UDP port can be different leading to different ECMP treatment. Does something else need to be specified here in regards to the outer UDP source port?
> 
> 
> 4. Section 7 says that “ Support for echo BFD is outside the scope of this document”. 
> 
> Assuming this means “BFD Echo mode”, why is this out of scope? If this is a single logical hop underneath VXLAN, what’s preventing the use of Echo? Echo’s benefits are huge.
> 
> 
> 5. Terminology
> 
> However, RFC 7348 Figure 3 lists that as one deployment scenario, not as “the scenario” and “The Reference VXLAN Domain”.
> 
> Best,
> 
> Carlos.
> 
>> thank you for your thorough review, clear and detailed questions. My apologies for the delay to respond. Please find my answers below in-line tagged GIM>>.
>> 
>> Regards,
>> Greg
>> 
>> On Fri, May 31, 2019 at 12:38 PM Olivier Bonaventure via Datatracker <[email-masked]> wrote:
>> primarily for the transport area directors, but are copied to the document's
>> authors and WG to allow them to address any issues raised and also to the IETF
>> I have only limited knowledge of VXLAN and do not know all subtleties of BFD.
>> Why is this requirement only relevant for the lookup path on the sender system
>> The requirement aimed to the forwarding engine of a BFD system that transmits BFD control packets over VXLAN tunnel.
>> 
>> GIM>> I think that ECMP environment is more likely to be experienced by a transit node in the underlay. If the BFD session is used to monitor the specific underlay path, then, I agree, we should explain that using the VXLAN payload information to draw path entropy may cause data and BFD packets following different underlay routes. But, on the other hand, that is the case for OAM and fault detection in all overlay networks in general.
>> 
>> Minor issues
>> 
>> Section 1
>> 
>> GIM>> In principle, BFD Demand mode may be used to monitor p2p paths as well, I agree, will re-word to more assertive:
>> I do not see the benefits of having one paragraph in Section 4 followed by only
>> Section 4.1
>> The document does not specify when a dedicated MAC address or the MAC address
>> of the destination VTEP must be used. This could affect the interoperability of
>> GIM>> After further discussion, authors decided to remove the request for the dedicated MAC address allocation. Only the MAC address of the remote VTEP must be used as the destination MAC address in the inner Ethernet frame. Please check the attached diff between the -07 and the working versions or the working version of the draft.
>> 
>> GIM>> As I've mentioned earlier, authors have decided to remove the use of the dedicated MAC address for BFD over VXLAN.
>> 
>> Section 5.1
>> 
>> GIM>> I think that this section is not modification but the definition of the application-specific procedure that is outside the scope of RFC 5880:
>> GIM>> Yes, thank you for pointing to this. The updated text, in the whole paragraph, is as follows:
>> NEW TEXT:
>> <draft-ietf-bfd-vxlan-08.txt><Diff_ draft-ietf-bfd-vxlan-07.txt - draft-ietf-bfd-vxlan-08.txt.html>
>

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Thu, 20 Jun 2019 00:14:39 +0000

Hi,

   An

Section 5.1 says:

2. Security

This document says that the TTL in the inner packet carrying BFD is set to 1. However, RFC 5880 says to use GTSM [RFC 5082], i.e., a value of 255.

Section 4.1. says:

   The Outer IP/UDP

And RFC 7348 says:

5. Terminology

6. Deployment scenarios

S3 says:

Best,

Carlos.

On Jun 17, 2019, at 12:58 AM, Greg Mirsky <[email-masked]<mailto:[email-masked]>> wrote:

Hi Oliver,

Regards,
Greg

On Fri, May 31, 2019 at 12:38 PM Olivier Bonaventure via Datatracker <[email-masked]<mailto:[email-masked]>> wrote:
Reviewer: Olivier Bonaventure
Review result: Ready with Issues

When done at the time of IETF Last Call, the authors should consider this
review as part of the last-call comments they receive. Please always CC
[email-masked]<mailto:[email-masked]> if you reply to or forward this review.

Major issues

Minor issues

Section 1

Section 3

VNI has not been defined

Section 4

Section 4.1

Section 5.

Section 5.1

Section 9

---

From: Greg Mirsky <[email-masked]>
Date: Thu, 20 Jun 2019 10:58:32 +0900

Hello Carlos,
thank you for the expedient clarification.
To your questions on demultiplexing BFD control packets with the zero value
of the Your Discriminator field:

   - only BFD control packets with the zero value of the Your Discriminator
   field are demultiplexed using the information of the inner IP header. I
   believe that the text is clear and requires that all fields of the inner IP
   header must be used to demultiplex a received BFD control packet with the
   zero value in the Your Discriminator field. Which of the fields an
   implementation uses to create multiple BFD sessions between the pair of
   VTEPs is implementation specific.

To your point on the level Echo mode of BFD is specified in RFC 5880 I'll
quote the opinion of Jeffrey Haas from the discussion of comments from
Shawn Emery on behalf of the SecDir. Shawn had commented:

Echo BFD is out of scope for the document, but does not describe the reason
for this or why state

this at all?

I've responded:

GIM>> I think that the main reason is that the BFD Echo mode is
underspecified. RFC 5880 defined some of the mechanisms related to the Echo
mode, but more standardization work may be required.

And Jeffrey Haas had added:

Speaking as a BFD chair, this is the relevant observation.  BFD Echo is
underspecified to the point where claiming compliance is difficult at best.
In general, it relies on single-hop and the ability to have the remote Echo
client loop the packets.

This packet loop may not be practical for several encapsulations and thus is
out of scope for such encapsulations.  Whether this is practical for vxlan
today, or in the presence of future extensions to vxlan is left out of scope
for the core proposal.

Will respond to other questions in a separate mail.

Regards,
Greg

On Thu, Jun 20, 2019 at 10:31 AM Carlos Pignataro (cpignata) <
[email-masked]> wrote:

> > To which stage of the three-way handshake you refer as "initial
> demultiplexing"? I couldn't find this term in RFC 5880.
>
> “Initial demultiplexing" is a well-known term in BFD, referring to the
> "demultiplexing of the initial packets", BFD Control packet with YourDisc
> being zero.
>
> Since initial demultiplexing is indeed application specific, different for
> one-hop versus multi-hop and dependent upon whether a single or multiple
> sessions are allowed between a pair of endpoints, I added below two other
> relevant citations, from application specific BFD specs:
> > Regarding the applicability of the Echo mode, thank you for pointing to
> the need for stricter terminology, the Echo mode, as defined in RFC 5880,
> is underspecified and it will require additional standardization.
>
> > This document says that the TTL in the inner packet carrying BFD is set
> to 1. However, RFC 5880 says to use GTSM [RFC 5082], i.e., a value of 255..
> >
> > Based on this, depending on the hashing calculation, the outer source
> UDP port can be different leading to different ECMP treatment. Does
> something else need to be specified here in regards to the outer UDP source
> port?
> >
> >
> > Assuming this means “BFD Echo mode”, why is this out of scope? If this
> is a single logical hop underneath VXLAN, what’s preventing the use of
> Echo? Echo’s benefits are huge.
> >
> >
> > 5. Terminology
> >
> >> GIM>> I think that ECMP environment is more likely to be experienced by
> a transit node in the underlay. If the BFD session is used to monitor the
> >> GIM>> After further discussion, authors decided to remove the request
> for the dedicated MAC address allocation. Only the MAC address of the
> remote VTEP must be used as the destination MAC address in the inner
> Ethernet frame. Please check the attached diff between the -07 and the
> working versions or the working version of the draft.
> >>
> >> GIM>> As I've mentioned earlier, authors have decided to remove the use
> of the dedicated MAC address for BFD over VXLAN.
> >>
> >> Section 5.1
> >>
> >> GIM>> I think that this section is not modification but the definition
> of the application-specific procedure that is outside the scope of RFC 5880:

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Thu, 20 Jun 2019 02:09:12 +0000

Hello, Greg,

Please see inline.

On Jun 19, 2019, at 9:58 PM, Greg Mirsky <[email-masked]<mailto:[email-masked]>> wrote:

Hello Carlos,
thank you for the expedient clarification.
To your questions on demultiplexing BFD control packets with the zero value of the Your Discriminator field:

  *   only BFD control packets with the zero value of the Your Discriminator field are demultiplexed using the information of the inner IP header. I believe that the text is clear and requires that all fields of the inner IP header must be used to demultiplex a received BFD control packet with the zero value in the Your Discriminator field. Which of the fields an implementation uses to create multiple BFD sessions between the pair of VTEPs is implementation specific.

This text is repeating was is in the draft, but does not answer any of my questions.

For example:
1. "that all fields of the inner IP header must be used to demultiplex a received BFD control packet”
    -> The text does not say “all fields”, but regardless, do you mean the DSCP and the Evil Bit? IPv6 Flow Label? How *exactly*?
2. How is the mapping of IP (not UDP?) fields to BFD session done?
3. How is this state created and maintained?
4. Since this is a set of fields on which two systems need to agree (which fields from the inner IP/UDP are mapped needs to be understood by both systems), it cannot be “implementation specific”. Further, the text does not say so.

To your point on the level Echo mode of BFD is specified in RFC 5880 I'll quote the opinion of Jeffrey Haas from the discussion of comments from Shawn Emery on behalf of the SecDir. Shawn had commented:
Echo BFD is out of scope for the document, but does not describe the reason for this or why state
this at all?
I've responded:
GIM>> I think that the main reason is that the BFD Echo mode is underspecified. RFC 5880 defined some of the mechanisms related to the Echo mode, but more standardization work may be required.
And Jeffrey Haas had added:
Speaking as a BFD chair, this is the relevant observation.  BFD Echo is
underspecified to the point where claiming compliance is difficult at best.
In general, it relies on single-hop and the ability to have the remote Echo
client loop the packets.

BFD Echo cannot be specified in RFC 5880 base spec because it is application specific.

The question remains: for VXLAN encapsulation, this is like a single hop as far as BFD is concerned (single hop VXLAN tunnel).

Since RFC 5881 defines Echo for single hop, can you please elaborate (in the document) why is out of scope or how it can work?

Best,

Carlos.

Regards,
Greg

On Thu, Jun 20, 2019 at 10:31 AM Carlos Pignataro (cpignata) <[email-masked]<mailto:[email-masked]>> wrote:
Hello, Greg,

> On Jun 19, 2019, at 9:09 PM, Greg Mirsky <[email-masked]<mailto:[email-masked]>> wrote:
>
> Hi Carlos,

Sure.

See above.

Thank you.

Best,

Carlos.

>
> Regards,
> Greg
>
>
> On Thu, Jun 20, 2019 at 9:14 AM Carlos Pignataro (cpignata) <[email-masked]<mailto:[email-masked]>> wrote:
> Hi,
>

---

From: Greg Mirsky <[email-masked]>
Date: Thu, 20 Jun 2019 13:50:24 +0900

Hello Carlos,
could you please refer me to the specification of BFD that defines the
message format that is used in the Echo mode of BFD. Is it the BFD control
packet? Something else?

Regards,
Greg

On Thu, Jun 20, 2019 at 11:09 AM Carlos Pignataro (cpignata) <
[email-masked]> wrote:

> Hello, Greg,
>
> Please see inline.
>
> On Jun 19, 2019, at 9:58 PM, Greg Mirsky <[email-masked]> wrote:
>
> Hello Carlos,
>    - only BFD control packets with the zero value of the Your
>    Discriminator field are demultiplexed using the information of the inner IP
>    header. I believe that the text is clear and requires that all fields of
>    the inner IP header must be used to demultiplex a received BFD control
>    packet with the zero value in the Your Discriminator field. Which of the
>    fields an implementation uses to create multiple BFD sessions between the
>    pair of VTEPs is implementation specific.
>
>     -> The text does not say “all fields”, but regardless, do you mean the
> DSCP and the Evil Bit? IPv6 Flow Label? How *exactly*?
> 4. Since this is a set of fields on which two systems need to agree (which
> fields from the inner IP/UDP are mapped needs to be understood by both
> systems), it cannot be “implementation specific”. Further, the text does
> not say so.
>
> The question remains: for VXLAN encapsulation, this is like a single hop
> as far as BFD is concerned (single hop VXLAN tunnel).
>
> Since RFC 5881 defines Echo for single hop, can you please elaborate (in
> the document) why is out of scope or how it can work?
>
> Best,
>
> Carlos.
>
>
>> Since initial demultiplexing is indeed application specific, different
>> for one-hop versus multi-hop and dependent upon whether a single or
>> multiple sessions are allowed between a pair of endpoints, I added below
>> two other relevant citations, from application specific BFD specs:
>> > The implication of this is that BFD single-hop initialization
>> procedures will not work. Instead, there is a need to map the initial
>> demultiplexing.
>> >
>> > However, RFC 7348 Figure 3 lists that as one deployment scenario, not
>> as “the scenario” and “The Reference VXLAN Domain”.
>> >
>> > Best,
>> >
>> > Carlos.
>> >
>> >> GIM>> I think that ECMP environment is more likely to be experienced
>> by a transit node in the underlay. If the BFD session is used to monitor
>> the specific underlay path, then, I agree, we should explain that using the
>> >> GIM>> In principle, BFD Demand mode may be used to monitor p2p paths
>> as well, I agree, will re-word to more assertive:
>> >> GIM>> As I've mentioned earlier, authors have decided to remove the
>> use of the dedicated MAC address for BFD over VXLAN.
>> >>
>> >> Section 5.1
>> >>

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Thu, 20 Jun 2019 05:08:54 +0000

Hello, Greg,

Yes, happy too answer your question.

First, however, let me make two quick observations on the exchange:

  1.  You seem to be picking very specific fragments or sub-topics from my comments, and following up on those only.
  2.  I’m happy to continue answering these questions, but they are orthogonal to (and consequently a distraction from) the initial comments I raised.

Now, your question: the BFD Echo packet format is generically spec’ed and defined in Section 5 of RFC 5880:
https://tools.ietf.org/html/rfc5880#section-5

The most relevant part is this:

   The payload of a BFD Echo packet is a local matter, since only the
   sending system ever processes the content.  The only requirement is
   that sufficient information is included to demultiplex the received
   packet to the correct BFD session after it is looped back to the
   sender.

Since the remote system does not process the echo packet content, it is a local matter only. And given the fact that, as such, there is no interoperability implications, there is no need to over-specify the packet format. The only requirement is that the sending system needs to be able to map it to a session when it boomerangs back.

No, it is generally not (i.e., does not need to be, but there’s freedom to potentially be) a BFD control packet.
Yes, it can be something else.

That said, the packet format for BFD Echo has, to my analysis and understanding, no relevancy on the questions below.

Best,

Carlos.

On Jun 20, 2019, at 12:50 AM, Greg Mirsky <[email-masked]<mailto:[email-masked]>> wrote:

Hello Carlos,
could you please refer me to the specification of BFD that defines the message format that is used in the Echo mode of BFD. Is it the BFD control packet? Something else?

Regards,
Greg

On Thu, Jun 20, 2019 at 11:09 AM Carlos Pignataro (cpignata) <[email-masked]<mailto:[email-masked]>> wrote:
Hello, Greg,

Please see inline.

Hello Carlos,

For example:

Best,

Carlos.

Regards,
Greg

Sure.

See above.

Thank you.

Best,

Carlos.

---

From: Greg Mirsky <[email-masked]>
Date: Thu, 20 Jun 2019 14:30:15 +0900

Hi Carlos,
thank you for clarifying your opinion on the applicability of the BFD Echo
mode. I've split this thread to help us and others to follow the discussion
on this particular issue. I'll note that RFC 5880 does not prohibit the
remote system to perform any kind of processing on the packet received in
the BFD Echo mode. Even more, Section 6.8.8 explicitly points out that a
received Echo packet is likely to be processed:
   A means of detecting missing Echo packets
   MUST be implemented, which most likely involves processing of the
   Echo packets that are received.  The processing of received Echo
   packets is otherwise outside the scope of this specification.
Thus, interoperability is one of the underspecified issues for BFD Echo.
Secondly, the Echo BFD mode has been excluded from the scope of RFC 5884:

Further, the use of the Echo function is outside the scope of this
specification.

Would you suggest that the scope of RFC 5884 must include the BFD Echo
function?

Regards,
Greg

On Thu, Jun 20, 2019 at 2:08 PM Carlos Pignataro (cpignata) <
[email-masked]> wrote:

>    1. You seem to be picking very specific fragments or sub-topics from
>    my comments, and following up on those only.
>    2. I’m happy to continue answering these questions, but they are
>    orthogonal to (and consequently a distraction from) the initial comments I
>    raised.
>
>
> Since the remote system does not process the echo packet content, it is a
> local matter only. And given the fact that, as such, there is no
> interoperability implications, there is no need to over-specify the packet
> format. The only requirement is that the sending system needs to be able to
> map it to a session when it boomerangs back.
>
> That said, the packet format for BFD Echo has, to my analysis and
> understanding, no relevancy on the questions below.
>
> Best,
>
> Carlos.
>
> On Jun 20, 2019, at 12:50 AM, Greg Mirsky <[email-masked]> wrote:
>
> Hello Carlos,
>>     -> The text does not say “all fields”, but regardless, do you mean
>> the DSCP and the Evil Bit? IPv6 Flow Label? How *exactly*?
>> 4. Since this is a set of fields on which two systems need to agree
>> (which fields from the inner IP/UDP are mapped needs to be understood by
>> both systems), it cannot be “implementation specific”. Further, the text
>> does not say so.
>>
>>> > Regarding the applicability of the Echo mode, thank you for pointing
>>> to the need for stricter terminology, the Echo mode, as defined in RFC
>>> 5880, is underspecified and it will require additional standardization.
>>>
>>> > I have not reviewed this draft before, but triggered by this email,
>>> and briefly scanning through a couple of sections, it is unclear to me how
>>> some of the mechanics work.
>>> >
>>> > There are some major issues with the Mac usage and association, as
>>> Joel Halpern mentioned in his Rtg Dir review.
>>> >
>>> > This document says that the TTL in the inner packet carrying BFD is
>>> set to 1. However, RFC 5880 says to use GTSM [RFC 5082], i.e., a value of
>>> 255..
>>> >
>>> >> GIM>> I think that this section is not modification but the
>>> definition of the application-specific procedure that is outside the scope
>>> of RFC 5880:

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Sat, 22 Jun 2019 00:19:22 +0000

Hello, Greg,

Thank you for having split this thread. I’ll note for ease of tracking that this specific issue is issue #4 our of the six quick comments I sent.

Now, questions you ask like this one quoted from below:
> Would you suggest that the scope of RFC 5884 must include the BFD Echo function?

Are now second-order red herrings and, once again, not relevant to the issue — instead they avoid responding and prevent convergence. In my comment, I’m just asking for a technical explanation of why BFD Echo is out of scope.

Please see inline, your response is incorrect in 3 technical areas that I enumerate.

> On Jun 20, 2019, at 1:30 AM, Greg Mirsky <[email-masked]> wrote:
> 
> Hi Carlos,
> thank you for clarifying your opinion on the applicability of the BFD Echo mode.

For correctness, I have not issued an "opinion on the applicability of the BFD Echo mode.” (Again, please do not mischaracterize!) 

My initial question was “why is this out of scope?” I believe scoping should be deliberate and intentional, and hence seeking to understand.

> I've split this thread to help us and others to follow the discussion on this particular issue. I'll note that RFC 5880 does not prohibit the remote system to perform any kind of processing on the packet received in the BFD Echo mode.

This is technical mistake #1. 

RFC 5880 says:

   The Echo function has the advantage of truly testing only the
   forwarding path on the remote system. 

No other processing on the remote system — that is the benefit of Echo.

> Even more, Section 6.8.8 explicitly points out that a received Echo packet is likely to be processed:

This is technical mistake #2.

What you are quoting is referring to the reception of Echo packet by the Echo packet *sender*. That is, the local system, not the remote system. The Echo packet is received by the same system that sent it (by definition of Echo!), and thus the node itself needs to demultiplex its own context...

> Thus, interoperability is one of the underspecified issues for BFD Echo.

No. This is technical mistake #2b.

There is no interoperability spec considerations between a system and itself (unless the sender of BFD Echo has split personality :-)

> Further, the use of the Echo function is outside the scope of this specification.

This is technical mistake #3.

It is really irrelevant to the question whether RFC 5884, which talks about BFD for unidirectional MPLS LSPs, scopes out BFD Echo.

As I mentioned in my initial email, "If this is a single logical hop underneath VXLAN, what’s preventing the use of Echo?” RFC 5884 is different than (and not useful for or relevant to) the scenario in draft-ietf-bfd-vxlan.

> Would you suggest that the scope of RFC 5884 must include the BFD Echo function?

I do not understand the context of this question. But as I mentioned, these questions take the thread in a direction that diverges from resolution.

I have not suggested that any document includes anything. My comment is: what is the reason why BFD Echo is out of scope? Technically, this is a single hop at the VXLAN level for BFD.

Best,

Carlos.

> 
> Regards,
> Greg
> 
> On Thu, Jun 20, 2019 at 2:08 PM Carlos Pignataro (cpignata) <[email-masked]> wrote:
> Hello, Greg,
> 
> 	• You seem to be picking very specific fragments or sub-topics from my comments, and following up on those only.
> 	• I’m happy to continue answering these questions, but they are orthogonal to (and consequently a distraction from) the initial comments I raised.
> 
>> On Thu, Jun 20, 2019 at 11:09 AM Carlos Pignataro (cpignata) <[email-masked]> wrote:
>> Hello, Greg,
>> 
>> Please see inline.
>> 
>>> 	• only BFD control packets with the zero value of the Your Discriminator field are demultiplexed using the information of the inner IP header. I believe that the text is clear and requires that all fields of the inner IP header must be used to demultiplex a received BFD control packet with the zero value in the Your Discriminator field. Which of the fields an implementation uses to create multiple BFD sessions between the pair of VTEPs is implementation specific.
>>> On Thu, Jun 20, 2019 at 10:31 AM Carlos Pignataro (cpignata) <[email-masked]> wrote:
>>> Hello, Greg,
>>>

---

From: Greg Mirsky <[email-masked]>
Date: Wed, 26 Jun 2019 18:22:42 -0700

Hello Carlos,
I hope that you can help me to answer a question related to the scope of
RFC 5885. In RFC 5885 is stated:
   This specification describes procedures only for BFD asynchronous mode.
Neither demand mode, nor Echo function of BFD is explicitly mentioned in
RFC 5885. Hence my question If BFD over VXLAN changes from explicitly
excluding the Echo function from the scope of the document to the
definition of the scope like used in RFC 5885, would that address your
concern?

Regards,
Greg

On Fri, Jun 21, 2019 at 5:19 PM Carlos Pignataro (cpignata) <
[email-masked]> wrote:

> Hello, Greg,
>
> Thank you for having split this thread. I’ll note for ease of tracking
> that this specific issue is issue #4 our of the six quick comments I sent.
>
> Are now second-order red herrings and, once again, not relevant to the
> issue — instead they avoid responding and prevent convergence. In my
> comment, I’m just asking for a technical explanation of why BFD Echo is out
> of scope.
>
>
> For correctness, I have not issued an "opinion on the applicability of the
> BFD Echo mode.” (Again, please do not mischaracterize!)
>
> My initial question was “why is this out of scope?” I believe scoping
> should be deliberate and intentional, and hence seeking to understand.
>
>
> > I've split this thread to help us and others to follow the discussion on
> this particular issue. I'll note that RFC 5880 does not prohibit the remote
> system to perform any kind of processing on the packet received in the BFD
> Echo mode.
>
> What you are quoting is referring to the reception of Echo packet by the
> Echo packet *sender*. That is, the local system, not the remote system. The
> Echo packet is received by the same system that sent it (by definition of
> Echo!), and thus the node itself needs to demultiplex its own context...
>
>
> There is no interoperability spec considerations between a system and
> itself (unless the sender of BFD Echo has split personality :-)
>
>
>
> It is really irrelevant to the question whether RFC 5884, which talks
> about BFD for unidirectional MPLS LSPs, scopes out BFD Echo.
>
> As I mentioned in my initial email, "If this is a single logical hop
> underneath VXLAN, what’s preventing the use of Echo?” RFC 5884 is different
> than (and not useful for or relevant to) the scenario in
> draft-ietf-bfd-vxlan.
>
>
>
> I do not understand the context of this question. But as I mentioned,
> these questions take the thread in a direction that diverges from
> resolution.
>
> I have not suggested that any document includes anything. My comment is:
> what is the reason why BFD Echo is out of scope? Technically, this is a
> single hop at the VXLAN level for BFD.
>
>
> Best,
>
> Carlos.
>
> >
> > Regards,
> > Greg
> >
> >       • You seem to be picking very specific fragments or sub-topics
> from my comments, and following up on those only.
> >       • I’m happy to continue answering these questions, but they are
> > Since the remote system does not process the echo packet content, it is
> a local matter only. And given the fact that, as such, there is no
> >>>     • only BFD control packets with the zero value of the Your
> >>> To your point on the level Echo mode of BFD is specified in RFC 5880
> I'll quote the opinion of Jeffrey Haas from the discussion of comments from
> >> The question remains: for VXLAN encapsulation, this is like a single
> hop as far as BFD is concerned (single hop VXLAN tunnel).
> >>
> >> Since RFC 5881 defines Echo for single hop, can you please elaborate
> (in the document) why is out of scope or how it can work?
> >>
> >> Best,
> >>
> >> Carlos.
> >>
> >>>
> >>> > thank you for reminding of our continued discussion with Joel. We
> are seeking comments from VXLAN experts and much appreciate if you have
> >>> Hence, my question stands: why is this draft claiming BFD Echo is out
> of scope for this BFD application document?
> >>>
> >>>
> >>> > But this does not really explain how to do the initial
> demultiplexing. Does each BFD session need to have a separate inner source
> IP address? Or source UDP port? And how ofter are they recycled or kept as
> state? How are these mapped?
> >>> > Based on this, depending on the hashing calculation, the outer
> source UDP port can be different leading to different ECMP treatment. Does
> >>> > Assuming this means “BFD Echo mode”, why is this out of scope? If
> this is a single logical hop underneath VXLAN, what’s preventing the use of
> >>> > However, RFC 7348 Figure 3 lists that as one deployment scenario,
> not as “the scenario” and “The Reference VXLAN Domain”.
> >>> >
> >>> > Best,
> >>> >
> >>> > Carlos.
> >>> >
> >>> >> thank you for your thorough review, clear and detailed questions.
> My apologies for the delay to respond. Please find my answers below in-line
> tagged GIM>>.
> >>> >>
> >>> >> Regards,
> >>> >> Greg
> >>> >>
> >>> >> GIM>> I think that ECMP environment is more likely to be
> experienced by a transit node in the underlay. If the BFD session is used
> to monitor the specific underlay path, then, I agree, we should explain
> that using the VXLAN payload information to draw path entropy may cause
> data and BFD packets following different underlay routes. But, on the other
> hand, that is the case for OAM and fault detection in all overlay networks
> in general.
> >>> >>
> >>> >> Minor issues
> >>> >>
> >>> >> Section 1
> >>> >>
> >>> >> GIM>> In principle, BFD Demand mode may be used to monitor p2p
> paths as well, I agree, will re-word to more assertive:
> >>> >>                                          |    Hypervisor VTEP (IP2)
> >>> >>                                          | |VM2-1    |  |VM2-2    |
> >>> >>                                          | |VNI 100  |  |VNI 200  |
> >>> >>                                          | |         |  |         |
> >>> >>                                          |      Server 2
> >>> >> GIM>> After further discussion, authors decided to remove the
> request for the dedicated MAC address allocation. Only the MAC address of
> the remote VTEP must be used as the destination MAC address in the inner