Subject: RE: BFD stability follow-up from IETF-91

From: Marc Binderberger <[email-masked]>
Date: Mon, 1 Dec 2014 02:28:35 -0800

Hello Gregory,

the test of "do we _really_ need this" is always a good idea :-)

>> I'm convinced that overloading BFD with performance measurement provisions 
>> is counter-productive and is inappropriate.

BFD is successful because it is so simple. And because we focused on doing 
this one thing right. Agree with your comment. It does not exclude to think 
about debug improvements though.

And as I claimed in several emails ;-), I think that at least some aspects of 
the discussion can be covered by "implementation", e.g. local time stamps and 
some common set of information collected in implementations. I.e. an informal 
draft could cover this.

To identify a packet though, so we can correlate Tx and Rx debug data, I 
don't see a good alternative to a sequence number yet. Which has the 
additional benefit that it tells us about packet loss (but for me the main 
reason is identification). But what is the alternative to introducing an 
extra field - reusing existing fields instead? (*)

Nevertheless, I prefer an "70%" debug solution with small BFD changes to a 
more complex "100%" solution.

My $0.02

Regards, Marc

P.S.: well, you know a v2 header would offer more options ;-) but looking at 
the v1 packet:

Local discriminator: some RFCs don't allow this to change (e.g. RFC5884). 
Otherwise it would be a nice identifier not only of a session but even of a 
single packet.

Required Min Echo RX Interval: my favourite as this field is effectively 
wasted for async BFD. The top bits translate into very large echo intervals 
(read: not very useful) and could be used as flags instead. The least 
significant bits translate into usec values, much faster than any 
implementation (yet?) and could also be used as flags instead. 

E.g. values of 0x80000000 - 0xFFFFFFFF could translate into an echo interval 
of zero and a 31 bit sequence number. But I don't see how to guarantee there 
won't be any interop problems with older implementations :-)

On Sat, 29 Nov 2014 03:17:54 +0000, Gregory Mirsky wrote:
> Hi Jeff,
> I absolutely agree with continuing discussion and sharing experiences from 
> real-life deployments and interoperability of BFD in IP and MPLS networks, 
> single-, multi-hop and over LAG constituents. From that we're learning and 
> that helps us make our implementations more robust, reliable, and 
> interoperable. And certainly, where it benefits the community informational 
> RFCs, like BFD intervals or RFC 7325 MPLS Forwarding Compliance and 
> Performance Requirements, to document the cases and our recommendations. 
> But I'm somewhat concerned with anything that targets standard status, even 
> as optional functionality, even though it is not proven that the problem is 
> in the standard, not in an implementation.
> 
> 	Regards,
> 		Greg
> 
> -----Original Message-----
> From: Jeffrey Haas [mailto:[email-masked]] 
> Sent: Friday, November 28, 2014 11:56 AM
> To: Gregory Mirsky
> Cc: Fan, Peng; 'MALLIK MUDIGONDA (mmudigon)'; [email-masked]
> Subject: Re: BFD stability follow-up from IETF-91
> 
> [Speaking as an individual contributor...]
> 
> On Fri, Nov 28, 2014 at 07:36:30PM +0000, Gregory Mirsky wrote:
>> this is very interesting scenario. I think that if BFD experiences ~30% 
>> packet loss, then highly likely so are affected other applications. Then 
>> it is not just BFD issue but condition that should be detected  by 
>> performance measurement method, whether active or passive packet loss 
>> measurement.
>> I'm convinced that overloading BFD with performance measurement provisions 
>> is counter-productive and is inappropriate.
> 
> My opinion is about halfway between your opinion, Greg.
> 
> I agree that we wish to be very cautious about overloading BFD with 
> components that are in other OAM mechanisms.  Among my desire for such 
> caution is that IEEE has expressed interest in not having us step on their 
> technologies and this would create paperwork for the chairs. :-)
> 
> But where I think we diverge slightly comes from experience in helping the 
> working group and vendors wend their way through implementing BFD for LAG.
> During that discussion, it was very clear that depending on the vendor, the 
> architecture and sometimes specific chipsets that "BFD" lived in very 
> different pieces of underlying architecture.
> 
> What this means is that trying to do very tight timing things will run into 
> practical issues in having to figure out what the perspective of the 
> timings are.  Is it some underlying L2? L3? Something between?  At what 
> point do you realize you are measuring contradictory things?
> 
> But similarly, when trying to measure and account for loss, having some 
> data is useful simply because it helps you determine that the component 
> that is *responsible for BFD* may be experiencing loss.  Depending on your 
> architecture, this may be the underlying layer-1, layer-2 or something else.
> In such cases, the lower-layer OAM is better to troubleshoot.  But in cases 
> where your lower-layer OAM doesn't indicate the loss, you still need to 
> understand that there is BFD-level loss.
> 
> I encourage participants in this discussion to remember this detail: We are 
> trying to help measure BFD loss.  Trying to read too much detail into what 
> that means outside of BFD may lead you to erroneous conclusions depending 
> on a given implementation.
> 
> Thus, consider what is best for BFD.
> 
> -- Jeff
>

---

From: "Fan, Peng" <[email-masked]>
Date: Mon, 1 Dec 2014 20:43:51 +0800

Hi Gregory,

I was just giving an example :) Application traffic usually cannot stand
small packet loss, not to say 30% loss.

I am actually asking for a debug function that could give us some useful
hints of poor connection with small protocol change, besides the basic
connectivity information. If it measures something, it measures packets of
BFD itself. So I don't expect it to be considered as a performance
measurement tool.

Best regards,

Peng

From: Gregory Mirsky [mailto:[email-masked]] 
Sent: Saturday, November 29, 2014 3:37 AM
To: Fan, Peng; 'MALLIK MUDIGONDA (mmudigon)'; [email-masked]
Subject: RE: BFD stability follow-up from IETF-91

Hi Peng,

this is very interesting scenario. I think that if BFD experiences ~30%
packet loss, then highly likely so are affected other applications. Then it
is not just BFD issue but condition that should be detected  by performance
measurement method, whether active or passive packet loss measurement.

                Regards,

                                Greg

From: Rtg-bfd [mailto:[email-masked]] On Behalf Of Fan, Peng
Sent: Friday, November 28, 2014 4:34 AM
To: 'MALLIK MUDIGONDA (mmudigon)'; [email-masked]
Subject: RE: BFD stability follow-up from IETF-91

Hi Mallik,

Exactly. Packets may be experiencing slight loss, but the link can hardly be
regarded as connected. More importantly, the experience of upper-level
applications can be degraded severely (e.g. TCP traffic is not able to go
fast in face of even small continuous loss). But what if one BFD frame is
lost every three frames? Then the loss rate is 30% on average, which is
already a very severe value.

Best regards,

Peng

From: MALLIK MUDIGONDA (mmudigon) [mailto:[email-masked]] 
Sent: Friday, November 28, 2014 7:53 PM
To: Fan, Peng; [email-masked]
Subject: Re: BFD stability follow-up from IETF-91

Hi Peng,

If the BFD packets are lost, doesn't the BFD session go DOWN? Are you saying
that packet loss is not big enough to make BFD session go DOWN?

Thanks

Regards

Mallik

From: <Fan>, Peng <[email-masked]>
Date: Friday, 28 November 2014 4:20 pm
To: "[email-masked]" <[email-masked]>
Subject: RE: BFD stability follow-up from IETF-91

Hi Jeff, all,

I have been following this stability extension from the beginning, and as an

operator I would like to express that this draft enables the "advanced

feature" we desire for BFD to provide additional useful information that

helps operators understand network issues. A relevant use case is detecting

lossy or "quasi-disconnected" links or member LAG links. An example of such

situation we experienced was a loosely connected fiber link resulting in

continuous, small amount of packet loss. BFD could get the information of

lost BFD frames on such unstable link, and probably report when a target

level is reached, say a certain number of frames are lost over a period or

among a total number of frames.

Best regards,

Peng

---

From: Gregory Mirsky <[email-masked]>
Date: Mon, 1 Dec 2014 17:38:06 +0000

Hi Peng,
and still, you're looking for a tool to measure BFD performance. Then you'll be looking for a tool to verify the BFD performance measurement, and on, and on. Operators do need complete set of FCAPS tools, including performance measurement. Note that passive performance measurement through marking method that Mach Chen referred to can monitor BFD flow(s) and be used to do Loss and/or Delay Measurement. And active Synthetic Loss Measurement may simulate flow of small packets as well as relatively large packets. And the same goes for active measurement method of Delay Measurement. I like Swiss Army knives but let us not turn BFD into one.

                Regards,
                                Greg

From: Fan, Peng [mailto:[email-masked]]
Sent: Monday, December 01, 2014 4:44 AM
To: Gregory Mirsky; 'MALLIK MUDIGONDA (mmudigon)'; [email-masked]
Subject: RE: BFD stability follow-up from IETF-91

Hi Gregory,

I was just giving an example :) Application traffic usually cannot stand small packet loss, not to say 30% loss.

I am actually asking for a debug function that could give us some useful hints of poor connection with small protocol change, besides the basic connectivity information. If it measures something, it measures packets of BFD itself. So I don't expect it to be considered as a performance measurement tool.

Best regards,
Peng

From: Gregory Mirsky [mailto:[email-masked]]
Sent: Saturday, November 29, 2014 3:37 AM
To: Fan, Peng; 'MALLIK MUDIGONDA (mmudigon)'; [email-masked]<mailto:[email-masked]>
Subject: RE: BFD stability follow-up from IETF-91

Hi Peng,
this is very interesting scenario. I think that if BFD experiences ~30% packet loss, then highly likely so are affected other applications. Then it is not just BFD issue but condition that should be detected  by performance measurement method, whether active or passive packet loss measurement.
I'm convinced that overloading BFD with performance measurement provisions is counter-productive and is inappropriate.

                Regards,
                                Greg

From: Rtg-bfd [mailto:[email-masked]] On Behalf Of Fan, Peng
Sent: Friday, November 28, 2014 4:34 AM
To: 'MALLIK MUDIGONDA (mmudigon)'; [email-masked]<mailto:[email-masked]>
Subject: RE: BFD stability follow-up from IETF-91

Hi Mallik,

Exactly. Packets may be experiencing slight loss, but the link can hardly be regarded as connected. More importantly, the experience of upper-level applications can be degraded severely (e.g. TCP traffic is not able to go fast in face of even small continuous loss). But what if one BFD frame is lost every three frames? Then the loss rate is 30% on average, which is already a very severe value.

Best regards,
Peng

From: MALLIK MUDIGONDA (mmudigon) [mailto:[email-masked]]
Sent: Friday, November 28, 2014 7:53 PM
To: Fan, Peng; [email-masked]<mailto:[email-masked]>
Subject: Re: BFD stability follow-up from IETF-91

Hi Peng,

If the BFD packets are lost, doesn't the BFD session go DOWN? Are you saying that packet loss is not big enough to make BFD session go DOWN?

Thanks

Regards
Mallik

From: <Fan>, Peng <[email-masked]<mailto:[email-masked]>>
Date: Friday, 28 November 2014 4:20 pm
To: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: RE: BFD stability follow-up from IETF-91

Hi Jeff, all,

Best regards,
Peng

---

From: Mahesh Jethanandani <[email-masked]>
Date: Tue, 2 Dec 2014 20:46:13 -0800

Greg,

What is Peng referring to is a way to figure out why a particular BFD session flapped, particularly if the packet(s) for that session arrive late. I do not see how that can be performance measurement. It is basic BFD debug ability. Running a separate DM does tell you why a particular BFD session flapped.

Now we can debate what methods can be employed to measure that delay and I am open to ways to doing it, including local loopback to measure transmit delays or time stamping of packets in hardware. But in cases, where there is no support for either of the capabilities, one of the suggested solutions is to use the time stamps carried in the BFD payload. 

Cheers.

> On Dec 1, 2014, at 9:38 AM, Gregory Mirsky <[email-masked]> wrote:
> 
> Hi Peng,
> and still, you’re looking for a tool to measure BFD performance. Then you’ll be looking for a tool to verify the BFD performance measurement, and on, and on. Operators do need complete set of FCAPS tools, including performance measurement. Note that passive performance measurement through marking method that Mach Chen referred to can monitor BFD flow(s) and be used to do Loss and/or Delay Measurement. And active Synthetic Loss Measurement may simulate flow of small packets as well as relatively large packets. And the same goes for active measurement method of Delay Measurement. I like Swiss Army knives but let us not turn BFD into one.
>  
>                 Regards,
>                                 Greg
>  
> I am actually asking for a debug function that could give us some useful hints of poor connection with small protocol change, besides the basic connectivity information. If it measures something, it measures packets of BFD itself. So I don’t expect it to be considered as a performance measurement tool.
>  
> Best regards,
> Peng
>  
> From: Gregory Mirsky [mailto:[email-masked] <mailto:[email-masked]>] 
> To: Fan, Peng; 'MALLIK MUDIGONDA (mmudigon)'; [email-masked] <mailto:[email-masked]>
> I’m convinced that overloading BFD with performance measurement provisions is counter-productive and is inappropriate.
>  
>                 Regards,
>                                 Greg
>  
> From: Rtg-bfd [mailto:[email-masked] <mailto:[email-masked]>] On Behalf Of Fan, Peng
> To: 'MALLIK MUDIGONDA (mmudigon)'; [email-masked] <mailto:[email-masked]>
> From: MALLIK MUDIGONDA (mmudigon) [mailto:[email-masked] <mailto:[email-masked]>] 
> To: Fan, Peng; [email-masked] <mailto:[email-masked]>
> If the BFD packets are lost, doesn’t the BFD session go DOWN? Are you saying that packet loss is not big enough to make BFD session go DOWN?
>  
> Thanks
>  
> Regards
> Mallik
>  
> From: <Fan>, Peng <[email-masked] <mailto:[email-masked]>>
> To: "[email-masked] <mailto:[email-masked]>" <[email-masked] <mailto:[email-masked]>>

Mahesh Jethanandani
Co-chair, NETCONF WG
[email-masked]

---

From: Gregory Mirsky <[email-masked]>
Date: Wed, 3 Dec 2014 23:25:25 +0000

Hi Mahesh,
I consider issues of debugability, not of just BFD but any other standardized protocol, to be outside of Standard track, at most to be suitable for Informational or Experimental track. If we agree on that, then we can discuss scenarios that present problem and investigate whether anything in the protocol requires clarification to help vendors in building well-performing, scalable and interoperable implementations and provide operational guidelines for operators.

                Regards,
                                Greg

From: Mahesh Jethanandani [mailto:[email-masked]]
Sent: Tuesday, December 02, 2014 8:46 PM
To: Gregory Mirsky
Cc: Fan, Peng; MALLIK MUDIGONDA (mmudigon); [email-masked]
Subject: Re: BFD stability follow-up from IETF-91

Greg,

Cheers.

On Dec 1, 2014, at 9:38 AM, Gregory Mirsky <[email-masked]<mailto:[email-masked]>> wrote:

Hi Peng,

                Regards,
                                Greg

From: Fan, Peng [mailto:[email-masked]]
Sent: Monday, December 01, 2014 4:44 AM
To: Gregory Mirsky; 'MALLIK MUDIGONDA (mmudigon)'; [email-masked]<mailto:[email-masked]>
Subject: RE: BFD stability follow-up from IETF-91

Hi Gregory,

Best regards,
Peng

Hi Peng,

                Regards,
                                Greg

Hi Mallik,

Best regards,
Peng

Hi Peng,

Thanks

Regards
Mallik

Hi Jeff, all,

Best regards,
Peng

Mahesh Jethanandani
Co-chair, NETCONF WG
[email-masked]<mailto:[email-masked]>

---

From: Mahesh Jethanandani <[email-masked]>
Date: Wed, 3 Dec 2014 22:22:57 -0800

Greg,

I believe we have a disagreement here. I do not believe that issue of debug ability are outside the scope of a standardized protocol.

Look at MPLS ping and traceroute (RFC 4379) . They are ultimately debug tools used to establish viability of a path and they are very much part of the standardized protocol.

> On Dec 3, 2014, at 3:25 PM, Gregory Mirsky <[email-masked]> wrote:
> 
> Hi Mahesh,
> On Dec 1, 2014, at 9:38 AM, Gregory Mirsky <[email-masked] <mailto:[email-masked]>> wrote:
>  
> Hi Peng,
> From: Fan, Peng [mailto:[email-masked] <mailto:[email-masked]>] 
> To: Gregory Mirsky; 'MALLIK MUDIGONDA (mmudigon)'; [email-masked] <mailto:[email-masked]>
> [email-masked] <mailto:[email-masked]>
Mahesh Jethanandani
Co-chair, NETCONF WG
[email-masked]

---

From: Gregory Mirsky <[email-masked]>
Date: Thu, 4 Dec 2014 06:41:48 +0000

Hi Mahesh,
indeed, LSP Ping is part of MPLS OAM tool set as BFD itself that intended to monitor operational state of the network, path continuity between two nodes. And LSP Ping, as primarily on-demand troubleshooting tool, helps localize and, to certain degree, diagnose the problem. But the ultimate debugging is proprietary. This proposal, in my view, helps not monitor behavior of the network but BFD itself, quality of BFD implementation. I’m not saying that it is not useful for implementers and operators, one can find that too many BFD sessions or at too short intervals being  ran. I don’t agree to loading this as extension of the widely used standard. Perhaps we can look into using BFD Echo as self-debugging instrument.

                Regards,
                                Greg

From: Mahesh Jethanandani [mailto:[email-masked]]
Sent: Wednesday, December 03, 2014 10:23 PM
To: Gregory Mirsky
Cc: Fan, Peng; MALLIK MUDIGONDA (mmudigon); [email-masked]
Subject: Re: BFD stability follow-up from IETF-91

Greg,

On Dec 3, 2014, at 3:25 PM, Gregory Mirsky <[email-masked]<mailto:[email-masked]>> wrote:

Hi Mahesh,

                Regards,
                                Greg

From: Mahesh Jethanandani [mailto:[email-masked]]
Sent: Tuesday, December 02, 2014 8:46 PM
To: Gregory Mirsky
Cc: Fan, Peng; MALLIK MUDIGONDA (mmudigon); [email-masked]<mailto:[email-masked]>
Subject: Re: BFD stability follow-up from IETF-91

Greg,

Cheers.

Hi Peng,

                Regards,
                                Greg

Hi Gregory,

Best regards,
Peng

Hi Peng,

                Regards,
                                Greg

Hi Mallik,

Best regards,
Peng

Hi Peng,

Thanks

Regards
Mallik

Hi Jeff, all,

Best regards,
Peng

Mahesh Jethanandani
Co-chair, NETCONF WG

Mahesh Jethanandani
Co-chair, NETCONF WG

---

From: Santosh P K <[email-masked]>
Date: Thu, 4 Dec 2014 15:01:32 +0000

Hello Greg,
  Debugging BFD is one of the use case. I also want to bring up one of the use case that Jeff suggested in his earlier  mail. Operator might NOT want to run OAM which does loss and delay measurement all the time due to its overhead. With the extension to BFD (sequence number) we can detect if there is any loss but BFD still stays up. This loss detection can be used as a trigger for loss and delay measurement. Echo can be used only in case of singlehop and in one direction only.

Thanks
Santosh P K

From: Rtg-bfd [mailto:[email-masked]] On Behalf Of Gregory Mirsky
Sent: Thursday, December 04, 2014 12:12 PM
To: Mahesh Jethanandani
Cc: [email-masked]
Subject: RE: BFD stability follow-up from IETF-91

Hi Mahesh,

                Regards,
                                Greg

Greg,

Hi Mahesh,

                Regards,
                                Greg

Greg,

Cheers.

Hi Peng,

                Regards,
                                Greg

Hi Gregory,

Best regards,
Peng

Hi Peng,

                Regards,
                                Greg

Hi Mallik,

Best regards,
Peng

Hi Peng,

Thanks

Regards
Mallik

Hi Jeff, all,

Best regards,
Peng

Mahesh Jethanandani
Co-chair, NETCONF WG

Mahesh Jethanandani
Co-chair, NETCONF WG

---

From: Gregory Mirsky <[email-masked]>
Date: Thu, 4 Dec 2014 15:39:58 +0000

Hi Santosh,
but that is what can be called “feature creep”. BFD is continuity check mechanism and for active performance measurement, even occasional, there are TWAMP in IP and RFC 6374/6375 in MPLS/MPLS-TP. It may be tempting to expand scope of BFD but, I believe, it is successful exactly because it was simple, light-weight and designed to do exactly one thing – continuity check.

                Regards,
                                Greg

From: Santosh P K [mailto:[email-masked]]
Sent: Thursday, December 04, 2014 7:02 AM
To: Gregory Mirsky; Mahesh Jethanandani
Cc: [email-masked]
Subject: RE: BFD stability follow-up from IETF-91

Hello Greg,

Thanks
Santosh P K

From: Rtg-bfd [mailto:[email-masked]] On Behalf Of Gregory Mirsky
Sent: Thursday, December 04, 2014 12:12 PM
To: Mahesh Jethanandani
Cc: [email-masked]<mailto:[email-masked]>
Subject: RE: BFD stability follow-up from IETF-91

Hi Mahesh,

                Regards,
                                Greg

Greg,

Hi Mahesh,

                Regards,
                                Greg

Greg,

Cheers.

Hi Peng,

                Regards,
                                Greg

Hi Gregory,

Best regards,
Peng

Hi Peng,

                Regards,
                                Greg

Hi Mallik,

Best regards,
Peng

Hi Peng,

Thanks

Regards
Mallik

Hi Jeff, all,

Best regards,
Peng

Mahesh Jethanandani
Co-chair, NETCONF WG

Mahesh Jethanandani
Co-chair, NETCONF WG

---

From: Manav Bhatia <[email-masked]>
Date: Thu, 4 Dec 2014 21:59:07 +0530

I am not sure what the confusion is Greg.

Assume i have a BFD session thats up. At some point in time it flaps. Now i
need to know whether the issue was at the TX or the RX.

Please tell me how TWAMP can help me here. Also tell me how what tool i can
use if its a uBFD session that flapped.

Cheers, Manav

On Thu, Dec 4, 2014 at 9:09 PM, Gregory Mirsky <[email-masked]>
wrote:

>  Hi Santosh,
>
> but that is what can be called “feature creep”. BFD is continuity check
> mechanism and for active performance measurement, even occasional, there
> are TWAMP in IP and RFC 6374/6375 in MPLS/MPLS-TP. It may be tempting to
> expand scope of BFD but, I believe, it is successful exactly because it was
> simple, light-weight and designed to do exactly one thing – continuity
> check.
>
>
>
>                 Regards,
>
>                                 Greg
>
>
>

---

From: Manav Bhatia <[email-masked]>
Date: Thu, 4 Dec 2014 21:46:04 +0530

Cheers, Manav

On Thu, Dec 4, 2014 at 9:09 PM, Gregory Mirsky <[email-masked]>
wrote:

> *From:* Santosh P K [mailto:[email-masked]]
> *Sent:* Thursday, December 04, 2014 7:02 AM
> *To:* Gregory Mirsky; Mahesh Jethanandani
>
> *Cc:* [email-masked]
> *Subject:* RE: BFD stability follow-up from IETF-91
>
>
>
> Hello Greg,
>
>   Debugging BFD is one of the use case. I also want to bring up one of the
> use case that Jeff suggested in his earlier  mail. Operator might NOT want
> to run OAM which does loss and delay measurement all the time due to its
> overhead. With the extension to BFD (sequence number) we can detect if
> there is any loss but BFD still stays up. This loss detection can be used
> as a trigger for loss and delay measurement. Echo can be used only in case
> of singlehop and in one direction only.
>
>
>
> Thanks
>
> Santosh P K
>
>
>
> *From:* Rtg-bfd [mailto:[email-masked]
> <[email-masked]>] *On Behalf Of *Gregory Mirsky
> *Sent:* Thursday, December 04, 2014 12:12 PM
> *To:* Mahesh Jethanandani
> *Cc:* [email-masked]
> *Subject:* RE: BFD stability follow-up from IETF-91
>
>
>
> Hi Mahesh,
>
> indeed, LSP Ping is part of MPLS OAM tool set as BFD itself that intended
> to monitor operational state of the network, path continuity between two
> nodes. And LSP Ping, as primarily on-demand troubleshooting tool, helps
> localize and, to certain degree, diagnose the problem. But the ultimate
> debugging is proprietary. This proposal, in my view, helps not monitor
> behavior of the network but BFD itself, quality of BFD implementation. I’m
> not saying that it is not useful for implementers and operators, one can
> find that too many BFD sessions or at too short intervals being  ran. I
> don’t agree to loading this as extension of the widely used standard.
> Perhaps we can look into using BFD Echo as self-debugging instrument.
>
>
>
>                 Regards,
>
>                                 Greg
>
>
>
> *From:* Mahesh Jethanandani [mailto:[email-masked]
> <[email-masked]>]
> *Sent:* Wednesday, December 03, 2014 10:23 PM
> *To:* Gregory Mirsky
> *Cc:* Fan, Peng; MALLIK MUDIGONDA (mmudigon); [email-masked]
> *Subject:* Re: BFD stability follow-up from IETF-91
>
>
>
> Greg,
>
>
>
> I believe we have a disagreement here. I do not believe that issue of
> debug ability are outside the scope of a standardized protocol.
>
>
>
> Look at MPLS ping and traceroute (RFC 4379) . They are ultimately debug
> tools used to establish viability of a path and they are very much part of
> the standardized protocol.
>
>
>
> I consider issues of debugability, not of just BFD but any other
> standardized protocol, to be outside of Standard track, at most to be
> suitable for Informational or Experimental track. If we agree on that, then
> we can discuss scenarios that present problem and investigate whether
> anything in the protocol requires clarification to help vendors in building
> well-performing, scalable and interoperable implementations and provide
> operational guidelines for operators.
>
>
>
>                 Regards,
>
>                                 Greg
>
>
>
> *From:* Mahesh Jethanandani [mailto:[email-masked]
> <[email-masked]>]
> *Sent:* Tuesday, December 02, 2014 8:46 PM
> *To:* Gregory Mirsky
> *Cc:* Fan, Peng; MALLIK MUDIGONDA (mmudigon); [email-masked]
> *Subject:* Re: BFD stability follow-up from IETF-91
>
>
>
> Greg,
>
>
>
> What is Peng referring to is a way to figure out why a particular BFD
> session flapped, particularly if the packet(s) for that session arrive
> late. I do not see how that can be performance measurement. It is basic BFD
> debug ability. Running a separate DM does tell you why a particular BFD
> session flapped.
>
>
>
> Now we can debate what methods can be employed to measure that delay and I
> am open to ways to doing it, including local loopback to measure transmit
> delays or time stamping of packets in hardware. But in cases, where there
> is no support for either of the capabilities, one of the suggested
> solutions is to use the time stamps carried in the BFD payload.
>
>
>
> Cheers.
>
>
>
> and still, you’re looking for a tool to measure BFD performance. Then
> you’ll be looking for a tool to verify the BFD performance measurement, and
> on, and on. Operators do need complete set of FCAPS tools, including
> performance measurement. Note that passive performance measurement through
> marking method that Mach Chen referred to can monitor BFD flow(s) and be
> used to do Loss and/or Delay Measurement. And active Synthetic Loss
> Measurement may simulate flow of small packets as well as relatively large
> packets. And the same goes for active measurement method of Delay
> Measurement. I like Swiss Army knives but let us not turn BFD into one.
>
>
>
>                 Regards,
>
>                                 Greg
>
>
>
> *From:* Fan, Peng [mailto:[email-masked]
> <[email-masked]>]
> *Sent:* Monday, December 01, 2014 4:44 AM
> *To:* Gregory Mirsky; 'MALLIK MUDIGONDA (mmudigon)'; [email-masked]
> *Subject:* RE: BFD stability follow-up from IETF-91
>
>
>
> Hi Gregory,
>
>
>
> BFD itself. So I don’t expect it to be considered as a performance
> measurement tool.
>
>
>
> Best regards,
>
> Peng
>
>
>
> *From:* Gregory Mirsky [mailto:[email-masked]
> <[email-masked]>]
> *Sent:* Saturday, November 29, 2014 3:37 AM
> *To:* Fan, Peng; 'MALLIK MUDIGONDA (mmudigon)'; [email-masked]
> *Subject:* RE: BFD stability follow-up from IETF-91
>
>
>
> Hi Peng,
>
> I’m convinced that overloading BFD with performance measurement provisions
> *From:* Rtg-bfd [mailto:[email-masked]
> <[email-masked]>] *On Behalf Of *Fan, Peng
> *Sent:* Friday, November 28, 2014 4:34 AM
> *To:* 'MALLIK MUDIGONDA (mmudigon)'; [email-masked]
> *Subject:* RE: BFD stability follow-up from IETF-91
>
>
>
> Hi Mallik,
>
>
>
> Exactly. Packets may be experiencing slight loss, but the link can hardly
> be regarded as connected. More importantly, the experience of upper-level
> *From:* MALLIK MUDIGONDA (mmudigon) [mailto:[email-masked]
> <[email-masked]>]
> *Sent:* Friday, November 28, 2014 7:53 PM
> *To:* Fan, Peng; [email-masked]
> *Subject:* Re: BFD stability follow-up from IETF-91
>
>
>
> Hi Peng,
>
>
>
> If the BFD packets are lost, doesn’t the BFD session go DOWN? Are you
> saying that packet loss is not big enough to make BFD session go DOWN?
>
>
>
> Thanks
>
>
>
> Regards
>
> Mallik
>
>
>
> *From: *<Fan>, Peng <[email-masked]>
> *Date: *Friday, 28 November 2014 4:20 pm
> *To: *"[email-masked]" <[email-masked]>
> *Subject: *RE: BFD stability follow-up from IETF-91
>
>
>
> Hi Jeff, all,
>
>
>

---

From: Gregory Mirsky <[email-masked]>
Date: Thu, 4 Dec 2014 16:33:17 +0000

Hi Manav,
I hope you don’t expect me to give a lecture on how to design and implement debugable implementation using logging and packet tracing.

                Regards,
                                Greg

From: Manav Bhatia [mailto:[email-masked]]
Sent: Thursday, December 04, 2014 8:16 AM
To: Gregory Mirsky
Cc: Santosh P K; Mahesh Jethanandani; [email-masked]
Subject: Re: BFD stability follow-up from IETF-91

Assume i have a BFD session thats up. At some point in time it flaps. Now i need to know whether the issue was at the TX or the RX.

Please tell me how TWAMP can help me here. Also tell me how what tool i can use if its a uBFD session that flapped.

Cheers, Manav

On Thu, Dec 4, 2014 at 9:09 PM, Gregory Mirsky <[email-masked]<mailto:[email-masked]>> wrote:
Hi Santosh,
but that is what can be called “feature creep”. BFD is continuity check mechanism and for active performance measurement, even occasional, there are TWAMP in IP and RFC 6374/6375 in MPLS/MPLS-TP. It may be tempting to expand scope of BFD but, I believe, it is successful exactly because it was simple, light-weight and designed to do exactly one thing – continuity check.

                Regards,
                                Greg

From: Santosh P K [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Thursday, December 04, 2014 7:02 AM
To: Gregory Mirsky; Mahesh Jethanandani

Hello Greg,

Thanks
Santosh P K

Hi Mahesh,

                Regards,
                                Greg

Greg,

Hi Mahesh,

                Regards,
                                Greg

Greg,

Cheers.

Hi Peng,

                Regards,
                                Greg

Hi Gregory,

Best regards,
Peng

Hi Peng,

                Regards,
                                Greg

Hi Mallik,

Best regards,
Peng

Hi Peng,

Thanks

Regards
Mallik

Hi Jeff, all,

Best regards,
Peng

Mahesh Jethanandani
Co-chair, NETCONF WG

Mahesh Jethanandani
Co-chair, NETCONF WG

---

From: Manav Bhatia <[email-masked]>
Date: Fri, 5 Dec 2014 08:32:41 +0530

Hi Greg,

I am sure the WG would appreciate such a lecture since that would obviate
the need for such an ID. Are you suggesting that i turn on logging and
packet tracing for *each* incoming BFD packet for all the sessions that i
have? Trying doing that for 25 BFD sessions where few are running at 50ms
and 100ms TX intervals. Now trying combing through the logs when 1 BFD
session flaps to understand where the issue was.

Cheers, Manav

On Thu, Dec 4, 2014 at 10:03 PM, Gregory Mirsky <[email-masked]

>  Hi Manav,
>
> I hope you don’t expect me to give a lecture on how to design and
> implement debugable implementation using logging and packet tracing.
>
>
>
>                 Regards,
>
>                                 Greg
>
>
>
> *From:* Manav Bhatia [mailto:[email-masked]]
> *Sent:* Thursday, December 04, 2014 8:16 AM
> *To:* Gregory Mirsky
> *Cc:* Santosh P K; Mahesh Jethanandani; [email-masked]
>
> Assume i have a BFD session thats up. At some point in time it flaps. Now
> i need to know whether the issue was at the TX or the RX.
>
>
>
> Please tell me how TWAMP can help me here. Also tell me how what tool i
> can use if its a uBFD session that flapped.
>
>
>
> Cheers, Manav
>
>
>
>
>

---

From: Gregory Mirsky <[email-masked]>
Date: Fri, 5 Dec 2014 05:22:03 +0000

Hi Manav,
if WG chairs feel that BFD implementation guide would be good to write we may find willing contributors.
As for granularity of logging, implementation I’m familiar with allows do that per BFD session without context switching.

                Regards,
                                Greg

PS. our discussion grew beyond tolerance level of mail-host, so I’ve snipped part.

From: Manav Bhatia [mailto:[email-masked]]
Sent: Thursday, December 04, 2014 7:02 PM
To: Gregory Mirsky
Cc: Santosh P K; Mahesh Jethanandani; [email-masked]
Subject: Re: BFD stability follow-up from IETF-91

Hi Greg,

I am sure the WG would appreciate such a lecture since that would obviate the need for such an ID. Are you suggesting that i turn on logging and packet tracing for *each* incoming BFD packet for all the sessions that i have? Trying doing that for 25 BFD sessions where few are running at 50ms and 100ms TX intervals. Now trying combing through the logs when 1 BFD session flaps to understand where the issue was.

Cheers, Manav

On Thu, Dec 4, 2014 at 10:03 PM, Gregory Mirsky <[email-masked]<mailto:[email-masked]>> wrote:
Hi Manav,
I hope you don’t expect me to give a lecture on how to design and implement debugable implementation using logging and packet tracing.

                Regards,
                                Greg

From: Manav Bhatia [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Thursday, December 04, 2014 8:16 AM
To: Gregory Mirsky
Cc: Santosh P K; Mahesh Jethanandani; [email-masked]<mailto:[email-masked]>

Cheers, Manav

                Regards,
                                Greg

Hello Greg,

Thanks
Santosh P K

Hi Mahesh,

                Regards,
                                Greg

---

From: Manav Bhatia <[email-masked]>
Date: Fri, 5 Dec 2014 08:31:33 +0530

Hi Greg,

Cheers, Manav

> On Thu, Dec 4, 2014 at 9:09 PM, Gregory Mirsky <
> [email-masked]> wrote:
>
> Hi Santosh,
>

---

From: Marc Binderberger <[email-masked]>
Date: Sun, 7 Dec 2014 20:07:16 -0800

Hello Manav,

> session flaps to understand where the issue was.

the problem I have here is we move deep into the implementation area. 
"Problem" for 2 reasons:

* it's telling my competitors what tricks my company has developed for better 
debugging. Wait ... I want to sell this advantage! :-)

* if we try to "standardize" this too far we choke implementor's phantasy. At 
least I fear so.

Manav, if you think about some kind of light, integrated log/tracing, then 
yes, you could do this for thousands of sessions with even faster timers. I'm 
of course speaking purely hypothetical here *cough* :-)

This is why I'm a fan of some minimal agreement plus room for local (i.e. 
per-vendor) data to be collected. For debugging you want both sides/vendors 
being involved anyway as data needs to be interpreted.

Regards, Marc

On Fri, 5 Dec 2014 08:31:33 +0530, Manav Bhatia wrote:
> On Thu, Dec 4, 2014 at 10:03 PM, Gregory Mirsky 
> <[email-masked]> wrote:
>> Hi Manav,
>> On Thu, Dec 4, 2014 at 9:09 PM, Gregory Mirsky 
>> <[email-masked]> wrote:
>> Hi Santosh,
>> expand scope of BFD but, I believe, it is successful exactly because it 
>> was simple, light-weight and designed to do exactly one thing – 
>> continuity check.
>>  
>>                 Regards,
>>                                 Greg
>>  
>> behavior of the network but BFD itself, quality of BFD implementation. I’
>> m not saying that it is not useful for implementers and operators, one can 
>> find that too many BFD sessions or at too short intervals being  ran. I don
>> ’t agree to loading this as extension of the widely used standard. 
>>> suitable for Informational or Experimental track. If we agree on that, 
>>> then we can discuss scenarios that present problem and investigate 
>>> whether anything in the protocol requires clarification to help vendors 
>>> in building well-performing, scalable and interoperable implementations 
>>> and provide operational guidelines for operators.
>>>  
>>>                 Regards,
>>>                                 Greg
>>>  
>>> late. I do not see how that can be performance measurement. It is basic 
>>> BFD debug ability. Running a separate DM does tell you why a particular 
>>> BFD session flapped.
>>>  
>>> Now we can debate what methods can be employed to measure that delay and 
>>> I am open to ways to doing it, including local loopback to measure 
>>> transmit delays or time stamping of packets in hardware. But in cases, 
>>> where there is no support for either of the capabilities, one of the 
>>> suggested solutions is to use the time stamps carried in the BFD payload. 
>>>  
>>> Cheers.
>>>  
>>>> you’ll be looking for a tool to verify the BFD performance measurement, 
>>>> and on, and on. Operators do need complete set of FCAPS tools, including 
>>>> performance measurement. Note that passive performance measurement 
>>>> through marking method that Mach Chen referred to can monitor BFD 
>>>> flow(s) and be used to do Loss and/or Delay Measurement. And active 
>>>> Synthetic Loss Measurement may simulate flow of small packets as well as 
>>>> relatively large packets. And the same goes for active measurement 
>>>> method of Delay Measurement. I like Swiss Army knives but let us not 
>>>> turn BFD into one.
>>>>  
>>>>                 Regards,
>>>>                                 Greg
>>>>  
>>>> connectivity information. If it measures something, it measures packets 
>>>> of BFD itself. So I don’t expect it to be considered as a performance 
>>>> measurement tool.
>>>>  
>>>> Best regards,
>>>> Peng
>>>>  
>>>> I’m convinced that overloading BFD with performance measurement 
>>>> provisions is counter-productive and is inappropriate.
>>>>  
>>>>                 Regards,
>>>>                                 Greg
>>>>  
>>>> Exactly. Packets may be experiencing slight loss, but the link can 
>>>> hardly be regarded as connected. More importantly, the experience of 
>>>> upper-level applications can be degraded severely (e.g. TCP traffic is 
>>>> not able to go fast in face of even small continuous loss). But what if 
>>>> one BFD frame is lost every three frames? Then the loss rate is 30% on 
>>>> average, which is already a very severe value.
>>>>  
>>>> Best regards,
>>>> Peng
>>>>

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 19 Dec 2014 16:02:22 -0500

[I'm running behind in discussions as usual]

On Thu, Dec 04, 2014 at 04:33:17PM +0000, Gregory Mirsky wrote:
> Hi Manav,
> I hope you don???t expect me to give a lecture on how to design and implement debugable implementation using logging and packet tracing.

For what it's worth, I always find such "drive-by" comments about "well,
there already exists something that does <foo>, it's here - go do your
homework!" to be a bit frustrating myself. :-)  

Not being personally familiar with TWAMP, I spend a few minutes digging
through the spec for TWAMP and OWAMP for some details about the control and
test traffic.  It has some properties that are problematic for the pieces of
the ecosystem covered by various portions of BFD deployment:

TCP control channel.  The implication is not only bidirectional
reachability, but a control IP stack that may not involve the nodes being
tested.  (Nodes in this case potentially being anything from host systems
all the way down to line card components.)

The data channel is UDP, which is good.  The problem though is there's no
guarantee it'll cover the layers of the hardware stack covered by the BFD
sessions.  As one example, BFD on LAG, we not only may not have a full IP
stack running at the point it's coming up, there's a chance that the
receiver isn't really much of an IP speaker.  It may be effectively using
the IP+UDP framing as dumb framing rather than protocol in order to
bootstrap the session.  IP may not come up until constituent LAG elements
have been put into service.

There's also the matter that to get TWAMP running at the appropriate layer,
it would be similarly necessary to embed it in the same general paths that
BFD covers.

Thus, at first very coarse analysis, TWAMP neither seems to share enough of
the forwarding fate on a guaranteed basis as we'd want to supplement BFD
debugging.  The control channel also seems a bit heavy-weight, but it's
potentially no worse than architecture of some LACP implementations I'm
aware of.

Is there something missing or incorrect in the above thinking?

-- Jeff

---

From: Gregory Mirsky <[email-masked]>
Date: Fri, 19 Dec 2014 22:52:10 +0000

Hi Jeff,
thank you for your interest in this discussion. If you agree, we can add IPPM WG to it.
I agree with your analysis of TWAMP and I would not suggest that TWAMP-Test is well suited to debug BFD specific issues. But TWAMP-Test, as other active performance measurement mechanisms, i.e. Y.1731 and MPLS PM based on RFC 6374, may be used to measure latency and jitter of a network or its segment. And even more than active, passive measurement methods may provide helpful information to troubleshoot network or BFD. That may be done using IPFIX on certain nodes in the network with the data analysis by a data collector. Would like to note that IPPM WG is in discussion of IP flow measurement method that uses marking method that I consider operationally more useful comparing to straight IPFIX.
I agree that debugging and troubleshooting is often an art and operators and vendors  need to have and use all tools that are available. We probably can do better job of instrumenting BFD debug tools but, I believe, not by changing, not by overloading its main functions of monitoring continuity and fast detection of the Loss of Continuity defect. Whether Down state is indication of the real defect or false negative, that is the question to be answered through analysis of available information and follow-up with debugging and troubleshooting the BFD itself rather than the network if there are concerns that it was not a real defect.

	Regards,
		Greg

-----Original Message-----
From: Jeffrey Haas [mailto:[email-masked]] 
Sent: Friday, December 19, 2014 1:02 PM
To: Gregory Mirsky
Cc: Manav Bhatia; [email-masked]
Subject: TWAMP analysis for assisting BFD debuggin (was Re: BFD stability follow-up from IETF-91)

> I hope you don???t expect me to give a lecture on how to design and implement debugable implementation using logging and packet tracing.

For what it's worth, I always find such "drive-by" comments about "well, there already exists something that does <foo>, it's here - go do your homework!" to be a bit frustrating myself. :-)  

Not being personally familiar with TWAMP, I spend a few minutes digging through the spec for TWAMP and OWAMP for some details about the control and test traffic.  It has some properties that are problematic for the pieces of the ecosystem covered by various portions of BFD deployment:

TCP control channel.  The implication is not only bidirectional reachability, but a control IP stack that may not involve the nodes being tested.  (Nodes in this case potentially being anything from host systems all the way down to line card components.)

The data channel is UDP, which is good.  The problem though is there's no guarantee it'll cover the layers of the hardware stack covered by the BFD sessions.  As one example, BFD on LAG, we not only may not have a full IP stack running at the point it's coming up, there's a chance that the receiver isn't really much of an IP speaker.  It may be effectively using the IP+UDP framing as dumb framing rather than protocol in order to bootstrap the session.  IP may not come up until constituent LAG elements have been put into service.

There's also the matter that to get TWAMP running at the appropriate layer, it would be similarly necessary to embed it in the same general paths that BFD covers.

Thus, at first very coarse analysis, TWAMP neither seems to share enough of the forwarding fate on a guaranteed basis as we'd want to supplement BFD debugging.  The control channel also seems a bit heavy-weight, but it's potentially no worse than architecture of some LACP implementations I'm aware of.

-- Jeff

---

From: Mahesh Jethanandani <[email-masked]>
Date: Sun, 21 Dec 2014 20:17:05 -0800

Greg,

I do not think the issue we are trying to debug is of whether a BFD flap is a real defect or a false negative, but rather why did BFD flap. And that is the information that is not available by running other PM tools.

If you go back to my email from a few weeks ago, when I gave a customer scenario, you would remember I was talking about a BFD flap causing tunnel to switch over. This draft is attempting to explain to the customer why the tunnel switched over when it did. How would you explain to the customer the reason for a failover?

Mahesh Jethanandani
[email-masked]

> On Dec 19, 2014, at 2:52 PM, Gregory Mirsky <[email-masked]> wrote:
> 
> Whether Down state is indication of the real defect or false negative, that is the question to be answered through analysis of available information and follow-up with debugging and troubleshooting the BFD itself rather than the network if there are concerns that it was not a real defect.

---

From: Gregory Mirsky <[email-masked]>
Date: Mon, 22 Dec 2014 06:46:26 +0000

Hi Mahesh,
if you assume that BFD flap is more likely because of implementation or overloading system rather than because of network condition, then I’d like to understand how intermittent congestion in a source end-node can be differentiated from intermittent congestion in the network without PM. I’m not questioning value of improved debugability of an implementation, BFD in this case, but I’d prefer it not to have adverse impact on the protocol, on its ability to perform its main task most efficiently. Debugging, I believe, been and still is, proprietary, implementation specific. Problems you’ve described, I believe, unlikely to be experienced in HW based BFD implementation. Thus, what benefit, changes you propose, would bring for such implementations?

                Regards,
                                Greg

From: Mahesh Jethanandani [mailto:[email-masked]]
Sent: Sunday, December 21, 2014 8:17 PM
To: Gregory Mirsky
Cc: Jeffrey Haas; [email-masked]
Subject: Re: TWAMP analysis for assisting BFD debuggin (was Re: BFD stability follow-up from IETF-91)

Greg,

Mahesh Jethanandani

On Dec 19, 2014, at 2:52 PM, Gregory Mirsky <[email-masked]<mailto:[email-masked]>> wrote:
Whether Down state is indication of the real defect or false negative, that is the question to be answered through analysis of available information and follow-up with debugging and troubleshooting the BFD itself rather than the network if there are concerns that it was not a real defect.

---

From: Jeffrey Haas <[email-masked]>
Date: Mon, 22 Dec 2014 10:15:24 -0500

Greg,

On Mon, Dec 22, 2014 at 06:46:26AM +0000, Gregory Mirsky wrote:
> if you assume that BFD flap is more likely because of implementation or
> overloading system rather than because of network condition, then I???d
> like to understand how intermittent congestion in a source end-node can be
> differentiated from intermittent congestion in the network without PM.

Congestion is one of the scenarios covered by tracking things at the BFD
level.  Another is simply whether packets are being generated or not or even
the simple timings within protocol packet generation.

While very lightweight, BFD still has implementation scaling choices that
can lead to unusual behaviors.  Packets are inherently jittered within the
context of the protocol, but to some extent it's simply a practical
consideration that maintaining tight timings is difficult.  Similarly, if a
packet simply is not sent because an internal timer ran late and it decided
to send the next packet instead, that's a consideration.

Being able to tie BFD issues to either issues within implementation vs.
issues happening on the wire is useful, but the tool sets will overlap in
some cases.

> I???m not questioning value of improved debugability of an implementation,
> BFD in this case, but I???d prefer it not to have adverse impact on the
> protocol, on its ability to perform its main task most efficiently.
> Debugging, I believe, been and still is, proprietary, implementation
> specific. Problems you???ve described, I believe, unlikely to be
> experienced in HW based BFD implementation. 

Unlikely to be experienced in HW?  I suspect you'll get a much different set
of feedback off-list where people think they can speak a bit more bluntly
than on-list.

> Thus, what benefit, changes
> you propose, would bring for such implementations?

Even in a situation where a given implementation was perfect, we care about
any pair of implementations.  When services flap due to BFD, you now have
two devices to troubleshoot and blame.

-- Jeff

---

From: Jeffrey Haas <[email-masked]>
Date: Mon, 22 Dec 2014 10:27:52 -0500

> thank you for your interest in this discussion. If you agree, we can add IPPM WG to it.

I think that would be excessively overloading the conversation at this time.
My primary point was that a performance measuring tech wasn't a fully
appropriate tool for this situation.  At some point if IPPM would like to
have BFD devs visit their session and provide feedback about where
implementation layering issues bias measurement, I suspect we can find
volunteers.

> I agree with your analysis of TWAMP and I would not suggest that
> TWAMP-Test is well suited to debug BFD specific issues. But TWAMP-Test, as
> other active performance measurement mechanisms, i.e. Y.1731 and MPLS PM
> based on RFC 6374, may be used to measure latency and jitter of a network
> or its segment.

But such measurements will depend on where in the implementation the
injectiosn and measurement are done.  The closer you get to IEEE work, the
more the measurements are likely to be implemented in HW as part of L2.  The
closer to IETF, the more likely it will be a layer just above HW L2.  Each
kind of measurement is valuable, and the results are ideally congruent.

>  And even more than active, passive measurement methods may
> provide helpful information to troubleshoot network or BFD. That may be
> done using IPFIX on certain nodes in the network with the data analysis by
> a data collector.

IPFIX typically will give you biased statistics either due to binning
effects or the fact that it's statistically sampled.
(Admittedly, I don't follow current work, only work I've done in the last 5
years with flow.)

To some extent, that observation highlights a difference in what is being
observed.  When you are concerned about flow-level impacts in your network,
the aggregates are the item of concern.  For BFD, and to a lesser extent
things like the other control plane protocols, a small number of lost
packets at inopportune times has significantly negative impacts.  The
granularities become much tighter.

> I agree that debugging and troubleshooting is often an art and operators
> and vendors  need to have and use all tools that are available. We
> probably can do better job of instrumenting BFD debug tools but, I
> believe, not by changing, not by overloading its main functions of
> monitoring continuity and fast detection of the Loss of Continuity defect.
> Whether Down state is indication of the real defect or false negative,
> that is the question to be answered through analysis of available
> information and follow-up with debugging and troubleshooting the BFD
> itself rather than the network if there are concerns that it was not a
> real defect.

That is the fundamental point: What tools do we need in BFD to troubleshoot
BFD especially given the layering issues that are somewhat distinct for its
implementations?  Sequencing information, minimally.  I am becoming more
convinced that the timing information is helpful - if tricky to implement in
a cross-vendor fashion.  I don't believe that BFD with such extensions
should replace mechanisms such as CFM.

-- Jeff