Subject: BFD stability follow-up from IETF-91

From: Jeffrey Haas <[email-masked]>
Date: Tue, 25 Nov 2014 19:19:31 -0500

draft-ashesh-bfd-stability-01 was presented again during IETF-91 in
Honolulu.  The slides can be viewed here:

http://www.ietf.org/proceedings/91/slides/slides-91-bfd-4.pptx

To attempt to simplify the presentation, the contentious portion of the
timers were removed from the proposal, leaving only the sequence numbering
for detecting loss of BFD async packets.

When the room was polled to see whether the draft should be adopted as a WG
item, the sense of the room was very quiet.  As promised, this is to inquire
for support for this draft on the WG mailing list to make sure the whole
group has a voice.

It should be noted that post-meeting discussion on the fate of this draft
noted that BFD authentication code points are plentiful and are available
with expert review.  Should the draft authors wish to continue this work as
Experimental, that is an option.

-- Jeff

---

From: Manav Bhatia <[email-masked]>
Date: Wed, 26 Nov 2014 12:26:41 +0530

Hi Jeff,

I vividly remember the original intent of the stability draft was to
help debug BFD failures -- to isolate the issue at the RX or the TX
side Time stamping would have helped in debugging whether the BFD
packet was sent late, or whether the packet was sent on time and also
arrived on time but was delayed when passing it up the BFD
stack/processor (lay in the RX buffer for tad too long), etc. But then
time stamping came with its own set of issues, and was hence dropped
from the original draft.

Can the authors send a summary on the list on why time stamping was
dropped so that we're all clear on that one.

The current proposal does help but is not complete.

Assume that the RX end loses a BFD session and learns later that it
did eventually receive the missing BFD packets (based on the seq #).
How would it know which end was misbehaving? Was it a delay at the TX
side, or was it the RX that delayed passing the packets to the BFD
process(or). This is usually what we want to debug and i want to
understand how this draft with sequence numbers can unequivocally tell
me that.

I believe the work is important and addresses something thats really
required (spent too much time debugging why BFD flapped!). Clearly
what would help is putting a small section that describes how we can
use the sequence numbers to debug what and where things went wrong.

Cheers, Manav

---

From: Marc Binderberger <[email-masked]>
Date: Wed, 26 Nov 2014 00:50:23 -0800

Hello Manav,

> required (spent too much time debugging why BFD flapped!).

agree :-) we should keep the discussion alive.

> stack/processor (lay in the RX buffer for tad too long)

well, I can see a point in having the Tx timestamps in the packet mainly for 
the purpose of knowing "this" packet was okay/not okay on the Tx side and to 
correlate it with your local Rx measurement.

And even this point is less relevant with sequence numbers as this number 
allows the identification of packets and thus the correlation of information 
from the Tx and Rx system.

Regards, Marc

---

From: Mach Chen <[email-masked]>
Date: Wed, 26 Nov 2014 09:17:32 +0000

Hi Marc and Manav,

> -----Original Message-----
> From: Rtg-bfd [mailto:[email-masked]] On Behalf Of Marc
> Binderberger
> Sent: Wednesday, November 26, 2014 4:50 PM
> To: Manav Bhatia
> Cc: [email-masked]
> Subject: Re: BFD stability follow-up from IETF-91
> 
> Hello Manav,
> 
> well, I can see a point in having the Tx timestamps in the packet mainly for the
> purpose of knowing "this" packet was okay/not okay on the Tx side and to

Yes, this is one solution if people think BFD delay is needed. If allow to have Tx timestamps to be carried in the packets, seems it should be no problem to leave a seat for the Rx timestamps as well :-). After all, with both Tx and Rx timestamp, it may simplify the implementation. 

> allows the identification of packets and thus the correlation of information from
> the Tx and Rx system.

Indeed, the sequence number helps a lot for the correlation between the Tx and Rx system. 

This triggers me think out there should be another solution for getting the Tx and Rx timestamps without encoding the timestamps in the BFD packets. For example, the Tx and Rx systems could just save timestamps locally or send them to a centralized entity and then use the sequence numbers to correlate them for further analyzing.

Best regards,
Mach

> >> To attempt to simplify the presentation, the contentious portion of
> >> the timers were removed from the proposal, leaving only the sequence
> >> numbering for detecting loss of BFD async packets.
> >>
> >> When the room was polled to see whether the draft should be adopted
> >> as a WG item, the sense of the room was very quiet.  As promised,
> >> this is to inquire for support for this draft on the WG mailing list
> >> to make sure the whole group has a voice.
> >>
> >> It should be noted that post-meeting discussion on the fate of this
> >> draft noted that BFD authentication code points are plentiful and are
> >> available with expert review.  Should the draft authors wish to
> >> continue this work as Experimental, that is an option.
> >>
> >> -- Jeff
> >>
> >

---

From: Marc Binderberger <[email-masked]>
Date: Wed, 26 Nov 2014 09:42:42 -0800

Hello Mach,

> This triggers me think out there should be another solution for getting the 
> Tx and Rx timestamps without encoding the timestamps in the BFD packets. 
> For example, the Tx and Rx systems could just save timestamps locally or 
> send them to a centralized entity and then use the sequence numbers to 
> correlate them for further analyzing.

I remember some discussion on NVO3 about how many bits it takes ;-) - could 
you send the links/draft names you are working on to this list? May be useful 
for further discussions.

Thanks & Regards,
Marc

On Wed, 26 Nov 2014 09:17:32 +0000, Mach Chen wrote:
> Yes, this is one solution if people think BFD delay is needed. If allow to 
> have Tx timestamps to be carried in the packets, seems it should be no 
> problem to leave a seat for the Rx timestamps as well :-). After all, with 
> both Tx and Rx timestamp, it may simplify the implementation. 
> 
>> 
> This triggers me think out there should be another solution for getting the 
> Tx and Rx timestamps without encoding the timestamps in the BFD packets. 
> For example, the Tx and Rx systems could just save timestamps locally or 
> send them to a centralized entity and then use the sequence numbers to 
> correlate them for further analyzing.
> 
> Best regards,
> Mach
> 
>> 
>> 
>> Regards, Marc
>> 
>> 
>> 
>> 
>> 
>>

---

From: Gregory Mirsky <[email-masked]>
Date: Wed, 26 Nov 2014 18:19:46 +0000

Dear All,
are we turning BFD into 1588? It does seem like that by the extent of our discussion of timestamping issue.
The PTP has two methods of measuring Resident Time, time a packet spent in a node (I feel that is what we're trying to measure by collecting timestamps):
*       one-step - resident time measured on a packet and reported in the same packet;
*       two-step - more elaborate as it requires source port identification, sequence number and proper type of a packet. Residence time measured on one packet but reported in the follow-up of appropriate type.
Does that sounds like what we're discussing?

        Regards,
                Greg

-----Original Message-----
From: Rtg-bfd [mailto:[email-masked]] On Behalf Of Mach Chen
Sent: Wednesday, November 26, 2014 1:18 AM
To: Marc Binderberger; Manav Bhatia
Cc: [email-masked]
Subject: RE: BFD stability follow-up from IETF-91

Hi Marc and Manav,

> Cc: [email-masked]<mailto:[email-masked]>
> > packet was sent late, or whether the packet was sent on time and
> > also arrived on time but was delayed when passing it up the BFD
> well, I can see a point in having the Tx timestamps in the packet
> mainly for the purpose of knowing "this" packet was okay/not okay on
> the Tx side and to correlate it with your local Rx measurement.

>
> And even this point is less relevant with sequence numbers as this
> number allows the identification of packets and thus the correlation
> of information from the Tx and Rx system.

Best regards,
Mach

> > packet was sent late, or whether the packet was sent on time and
> > also arrived on time but was delayed when passing it up the BFD
> > stack/processor (lay in the RX buffer for tad too long), etc. But
> > then time stamping came with its own set of issues, and was hence
> > dropped from the original draft.
> >
> > How would it know which end was misbehaving? Was it a delay at the
> > TX side, or was it the RX that delayed passing the packets to the
> > BFD process(or). This is usually what we want to debug and i want to
> > understand how this draft with sequence numbers can unequivocally
> > tell me that.
> >
> > On Wed, Nov 26, 2014 at 5:49 AM, Jeffrey Haas <[email-masked]<mailto:[email-masked]>> wrote:
> >> the timers were removed from the proposal, leaving only the
> >> sequence numbering for detecting loss of BFD async packets.
> >>
> >> this is to inquire for support for this draft on the WG mailing
> >> list to make sure the whole group has a voice.
> >>
> >> draft noted that BFD authentication code points are plentiful and
> >> are available with expert review.  Should the draft authors wish to

---

From: Manav Bhatia <[email-masked]>
Date: Thu, 27 Nov 2014 05:12:35 +0530

Hi Marc,

(and we meet again! :-))

My claim is that just using sequence numbers may NOT help.

A BFD session with an interval of 33ms on router A will flap if it
does not see any BFD packet for 100ms.

Assume that the last seq number that A sees from the remote end is,
say 100. A will bring down the BFD session if it now does not see 101,
102 and 103 in the next 100ms.

Further assume that these packets were not seen by A and the session
flaps. However, we get these 3 BFD packets immediately after this flap
-- at 100ms + some_delta.

Now given just the sequence numbers its almost impossible for A to
know whether the issue was at the RX or the TX side.

Am i missing something?

Cheers, Manav

---

From: Mach Chen <[email-masked]>
Date: Thu, 27 Nov 2014 08:43:12 +0000

Hi Marc,

> From: Marc Binderberger [mailto:[email-masked]]
> Sent: Thursday, November 27, 2014 1:43 AM
> To: Mach Chen
> Cc: Manav Bhatia; [email-masked]
> > This triggers me think out there should be another solution for
> > getting the Tx and Rx timestamps without encoding the timestamps in the BFD
> packets.
> > For example, the Tx and Rx systems could just save timestamps locally
> > or send them to a centralized entity and then use the sequence numbers
> > to correlate them for further analyzing.
> 
> you send the links/draft names you are working on to this list? May be useful for
> further discussions.

Sure, here is the link(http://tools.ietf.org/html/draft-chen-ippm-coloring-based-ipfpm-framework-02) for the reference.

But here I want to say is that since we have sequence number, we may not need the marking based solution. Suppose that someone want to monitor the delay of a BFD packet , just record and save the timestamp at the Tx side, which indexed by the sequence number. Similarly, do the same at the Rx side. Then based on the timestamps from both Tx and Rx, and using the sequence number to correlate the timestamps, it can also provide a way to monitor the delay of the BFD packet. 

That means, only if there is sequence number, even if without carrying the timestamp in the BFD packet, BFD packet delay can be measured. 

Best regards,
Mach

> > Yes, this is one solution if people think BFD delay is needed. If
> > allow to have Tx timestamps to be carried in the packets, seems it
> > should be no problem to leave a seat for the Rx timestamps as well
> > :-). After all, with both Tx and Rx timestamp, it may simplify the
> implementation.
> >
> >>
> > This triggers me think out there should be another solution for
> > getting the Tx and Rx timestamps without encoding the timestamps in the BFD
> packets.
> > For example, the Tx and Rx systems could just save timestamps locally
> > or send them to a centralized entity and then use the sequence numbers
> > to correlate them for further analyzing.
> >
> > Best regards,
> > Mach
> >
> >>
> >>
> >> Regards, Marc
> >>
> >>
> >>
> >>
> >>
> >>

---

From: "Vengada Prasad Govindan (venggovi)" <[email-masked]>
Date: Thu, 27 Nov 2014 12:59:53 +0000

Hello Mach,
  Often (definitely more than once), we have felt the need to carry sequence numbers in BFD packets for debugging packet losses/ delayed delivery etc. This extension will increase the ease of debugging BFD flaps caused due to delayed delivery, silent drops etc.
Thanks
Prasad

-----Original Message-----
From: Rtg-bfd [mailto:[email-masked]] On Behalf Of Mach Chen
Sent: Thursday, November 27, 2014 2:13 PM
To: Marc Binderberger
Cc: [email-masked]
Subject: RE: BFD stability follow-up from IETF-91

Hi Marc,

> > For example, the Tx and Rx systems could just save timestamps 
> > locally or send them to a centralized entity and then use the 
> > sequence numbers to correlate them for further analyzing.
> 
> I remember some discussion on NVO3 about how many bits it takes ;-) - 
> could you send the links/draft names you are working on to this list? 
> May be useful for further discussions.

Best regards,
Mach

> >>> I believe the work is important and addresses something thats 
> >>> really required (spent too much time debugging why BFD flapped!).
> >>
> >> mainly for the purpose of knowing "this" packet was okay/not okay 
> >> on the Tx side and to correlate it with your local Rx measurement.
> >
> >> number allows the identification of packets and thus the 
> >> correlation of information from the Tx and Rx system.
> >
> > For example, the Tx and Rx systems could just save timestamps 
> > locally or send them to a centralized entity and then use the 
> > sequence numbers to correlate them for further analyzing.
> >
> > Best regards,
> > Mach
> >
> >>
> >>
> >> Regards, Marc
> >>
> >>
> >>
> >>
> >>
> >>
> >>> I vividly remember the original intent of the stability draft was 
> >>> to help debug BFD failures -- to isolate the issue at the RX or 
> >>> the TX side Time stamping would have helped in debugging whether 
> >>> the BFD packet was sent late, or whether the packet was sent on 
> >>> time and also arrived on time but was delayed when passing it up 
> >>> the BFD stack/processor (lay in the RX buffer for tad too long), 
> >>> etc. But then time stamping came with its own set of issues, and 
> >>> was hence dropped from the original draft.
> >>>
> >>> Can the authors send a summary on the list on why time stamping 
> >>> was dropped so that we're all clear on that one.
> >>>
> >>> Assume that the RX end loses a BFD session and learns later that 
> >>> it did eventually receive the missing BFD packets (based on the seq #).
> >>> BFD process(or). This is usually what we want to debug and i want 
> >>> to understand how this draft with sequence numbers can 
> >>> unequivocally tell me that.
> >>>
> >>> I believe the work is important and addresses something thats 
> >>> really required (spent too much time debugging why BFD flapped!). 
> >>> Clearly what would help is putting a small section that describes 
> >>> how we can use the sequence numbers to debug what and where things went wrong.
> >>>
> >>> Cheers, Manav
> >>>
> >>>
> >>>> draft-ashesh-bfd-stability-01 was presented again during IETF-91 
> >>>> in Honolulu.  The slides can be viewed here:
> >>>>
> >>>> To attempt to simplify the presentation, the contentious portion 
> >>>> of the timers were removed from the proposal, leaving only the 
> >>>> When the room was polled to see whether the draft should be 
> >>>> adopted as a WG item, the sense of the room was very quiet.  As 
> >>>> promised, this is to inquire for support for this draft on the WG 
> >>>> mailing list to make sure the whole group has a voice.
> >>>>
> >>>> It should be noted that post-meeting discussion on the fate of 
> >>>> this draft noted that BFD authentication code points are 
> >>>> plentiful and are available with expert review.  Should the draft 
> >>>> authors wish to continue this work as Experimental, that is an option.
> >>>>
> >>>> -- Jeff
> >>>>
> >>>
> >

---

From: Manav Bhatia <[email-masked]>
Date: Thu, 27 Nov 2014 18:46:30 +0530

Hi Vengada,

Let me at the outset say that i am in favor of such a draft. I think
its an important and a useful work that should be taken up by the WG.

However, i have issues in how it current stands. I think it needs a
wee bit more polishing.

>   Often (definitely more than once), we have felt the need to carry sequence numbers in
> BFD packets for debugging packet losses/ delayed delivery etc. This extension will
> increase the ease of debugging BFD flaps caused due to delayed delivery, silent drops etc.

There are ways to detect BFD drops but then those are implementation
specific and i will refrain from going there. So i concede that it
would help in packet losses. However i dont see this helping much in
case of delayed delivery (please see my earlier email).

Can the authors add some text on how this debugging mechanism would
work if somebody employs BFD authentication?

Cheers, Manav

---

From: Manav Bhatia <[email-masked]>
Date: Thu, 27 Nov 2014 18:53:03 +0530

Ok so this helps. Can the authors add this to the draft?

Cheers, Manav

---

From: Santosh P K <[email-masked]>
Date: Thu, 27 Nov 2014 14:49:26 +0000

Manav,
   Thanks for your comments and sorry for late response. Please see inline. 

> I vividly remember the original intent of the stability draft was to help debug
> BFD failures -- to isolate the issue at the RX or the TX side Time stamping
> would have helped in debugging whether the BFD packet was sent late, or
> whether the packet was sent on time and also arrived on time but was
> delayed when passing it up the BFD stack/processor (lay in the RX buffer for
> tad too long), etc. But then time stamping came with its own set of issues,
> and was hence dropped from the original draft.

Timestamp will  help but there were comments about BFD stepping in to other protocols which are already doing it. Also there were comments about timestamping for aggressive protocol like BFD would be feasible. We can make timestamping as optional, but we should be careful not step in to other protocols?

> The current proposal does help but is not complete.

For loss detection it will help but not for delay. I agree. 

> required (spent too much time debugging why BFD flapped!). Clearly what
> would help is putting a small section that describes how we can use the
> sequence numbers to debug what and where things went wrong.

I agree, we are planning to add use case and also how we sequence number can be useful. 

Thanks
Santosh P K

> > When the room was polled to see whether the draft should be adopted as
> > a WG item, the sense of the room was very quiet.  As promised, this is
> > to inquire for support for this draft on the WG mailing list to make
> > sure the whole group has a voice.
> >

---

From: Santosh P K <[email-masked]>
Date: Thu, 27 Nov 2014 14:52:17 +0000

Mach,
   Thanks for your comments. Please see inline. 

> 
> This triggers me think out there should be another solution for getting the Tx
> and Rx timestamps without encoding the timestamps in the BFD packets. For
> example, the Tx and Rx systems could just save timestamps locally or send
> them to a centralized entity and then use the sequence numbers to correlate
> them for further analyzing.

This will help but it won't really tell where was the delay right? And it looks more like a implementation? 

Thanks
Santosh P K

---

From: Santosh P K <[email-masked]>
Date: Thu, 27 Nov 2014 14:59:08 +0000

Manav and Mach,
    Thanks a lot for reference and explanation. So with this we don’t really need time stamping to be carried with in the BFD packet.  Yes, we can add this text in the draft. 

Thanks
Santos h P K 

> From: Rtg-bfd [mailto:[email-masked]] On Behalf Of Manav Bhatia
> Sent: Thursday, November 27, 2014 6:53 PM
> To: Mach Chen
> Cc: [email-masked]
> > Sure, here is the link(http://tools.ietf.org/html/draft-chen-ippm-coloring-
> based-ipfpm-framework-02) for the reference.
> >
> > But here I want to say is that since we have sequence number, we may not
> need the marking based solution. Suppose that someone want to monitor
> the delay of a BFD packet , just record and save the timestamp at the Tx side,
> which indexed by the sequence number. Similarly, do the same at the Rx
> side. Then based on the timestamps from both Tx and Rx, and using the
> sequence number to correlate the timestamps, it can also provide a way to
> monitor the delay of the BFD packet.
> >
> > That means, only if there is sequence number, even if without carrying the
> timestamp in the BFD packet, BFD packet delay can be measured.
> >
> > Best regards,
> > Mach
> >
> >>
> >>
> >> Thanks & Regards,
> >> Marc
> >>
> >>
> >>
> >> >>> How would it know which end was misbehaving? Was it a delay at
> >> >>> the TX side, or was it the RX that delayed passing the packets to
> >> >>> the BFD process(or). This is usually what we want to debug and i
> >> >>> want to understand how this draft with sequence numbers can
> >> >>>> promised, this is to inquire for support for this draft on the
> >> >>>> WG mailing list to make sure the whole group has a voice.
> >> >>>>
> >> >>>> plentiful and are available with expert review.  Should the
> >> >>>> draft authors wish to continue this work as Experimental, that is an
> option.
> >> >>>>
> >> >>>> -- Jeff
> >> >>>>
> >> >>>
> >> >

---

From: Santosh P K <[email-masked]>
Date: Thu, 27 Nov 2014 15:02:17 +0000

Manav,
    This is good question. 

> work if somebody employs BFD authentication?

Right now we have considered without authentication (we are setting A bit). We should add some text on how we can use both Auth and de bug TLV. Is there any suggestion you have? I will get back to you on this. 

Thanks
Santosh P K

---

From: Manav Bhatia <[email-masked]>
Date: Thu, 27 Nov 2014 21:12:00 +0530

Hi Santosh,

You could use the crypto sequence numbers carried in the meticulous
cryptographic auth for detecting packet losses. However, this breaks
when you use non-meticulous crypto authentication since the sequence
number is only incremented occasionally there. This i believe is a
deal breaker since i really envision non meticulous mode to be the one
being widely deployed. In fact we were supposed to write a draft on
that and i guess it just fell through the cracks (lemme ping my
co-author on that !)

One way to solve this problem is by attaching a debug trailer that
only carries the seq numbers at the *end* of the BFD packet. This
would not be covered in the Length field carried in the BFD header but
would be accounted for in the length carried in the IP header. The
concept of attaching a trailer is documented well and is used in the
IGPs. RFC 6506 describes one such trailer for OSPFv3. The catch
however is that this debug trailer will NOT be covered by the BFD
authentication. Is this acceptable to the WG?

I think the problem of diagnosing a BFD flap becomes all the more
important with BFD authentication turned on since then we have more
points where a delay can be inserted.

Cheers, Manav

---

From: Santosh P K <[email-masked]>
Date: Thu, 27 Nov 2014 15:56:41 +0000

Manav,
   Thanks. Please see inline comments.

> cryptographic auth for detecting packet losses. However, this breaks when
> you use non-meticulous crypto authentication since the sequence number is
> only incremented occasionally there. This i believe is a deal breaker since i
> really envision non meticulous mode to be the one being widely deployed. In
> fact we were supposed to write a draft on that and i guess it just fell through
> the cracks (lemme ping my co-author on that !)
> 
> One way to solve this problem is by attaching a debug trailer that only carries
> the seq numbers at the *end* of the BFD packet. This would not be covered
> in the Length field carried in the BFD header but would be accounted for in
> the length carried in the IP header. The concept of attaching a trailer is
> documented well and is used in the IGPs. RFC 6506 describes one such trailer

You are suggesting to add the debug trailer with or without authentication right? 

> for OSPFv3. The catch however is that this debug trailer will NOT be covered
> by the BFD authentication. Is this acceptable to the WG?

I did not get this. Did you mean the length of the auth TVL will not include this length and will be only in IP header? 

> I think the problem of diagnosing a BFD flap becomes all the more important
> with BFD authentication turned on since then we have more points where a
> delay can be inserted.

I agree with this and as a developer have seen this happening. 

Thanks
Santosh P K 

> > Right now we have considered without authentication (we are setting A
> bit). We should add some text on how we can use both Auth and de bug TLV.
> Is there any suggestion you have? I will get back to you on this.
> >
> >
> > Thanks
> > Santosh P K
> >
> >> >> I remember some discussion on NVO3 about how many bits it takes
> >> >> ;-) - could you send the links/draft names you are working on to this
> list?
> >> > Sure, here is the
> >> > link(http://tools.ietf.org/html/draft-chen-ippm-coloring-
> >> the delay of a BFD packet , just record and save the timestamp at the
> >> Tx side, which indexed by the sequence number. Similarly, do the same
> >> at the Rx side. Then based on the timestamps from both Tx and Rx, and
> >> using the sequence number to correlate the timestamps, it can also
> >> provide a way to monitor the delay of the BFD packet.
> >> >
> >> >> >> From: Rtg-bfd [mailto:[email-masked]] On Behalf Of
> >> >> >> Marc Binderberger
> >> >> >>> packet was sent late, or whether the packet was sent on time
> >> >> >>> and also arrived on time but was delayed when passing it up
> >> >> >>> the BFD stack/processor (lay in the RX buffer for tad too
> >> >> >>> long)
> >> >> >>
> >> >> >> well, I can see a point in having the Tx timestamps in the
> >> >> >> packet mainly for the purpose of knowing "this" packet was
> >> >> >> okay/not okay on the Tx side and to correlate it with your local Rx
> measurement.
> >> >> >
> >> >> > Yes, this is one solution if people think BFD delay is needed.
> >> >> > If allow to have Tx timestamps to be carried in the packets,
> >> >> > seems it should be no problem to leave a seat for the Rx
> >> >> > timestamps as well :-). After all, with both Tx and Rx
> >> >> > timestamp, it may simplify the
> >> >> implementation.
> >> >> >
> >> >> >>
> >> >> >> And even this point is less relevant with sequence numbers as
> >> >> >> this number allows the identification of packets and thus the
> >> >> >>> I vividly remember the original intent of the stability draft
> >> >> >>> was to help debug BFD failures -- to isolate the issue at the
> >> >> >>> RX or the TX side Time stamping would have helped in debugging
> >> >> >>> whether the BFD packet was sent late, or whether the packet
> >> >> >>> was sent on time and also arrived on time but was delayed when
> >> >> >>> passing it up the BFD stack/processor (lay in the RX buffer
> >> >> >>> for tad too long), etc. But then time stamping came with its
> >> >> >>> own set of issues, and was hence dropped from the original draft.
> >> >> >>>
> >> >> >>> Can the authors send a summary on the list on why time
> >> >> >>> stamping was dropped so that we're all clear on that one.
> >> >> >>>
> >> >> >>> Assume that the RX end loses a BFD session and learns later
> >> >> >>> that it did eventually receive the missing BFD packets (based
> >> >> >>> on the seq
> >> #).
> >> >> >>> the TX side, or was it the RX that delayed passing the packets
> >> >> >>> to the BFD process(or). This is usually what we want to debug
> >> >> >>> and i want to understand how this draft with sequence numbers
> >> >> >>> can unequivocally tell me that.
> >> >> >>>
> >> >> >>> Clearly what would help is putting a small section that
> >> >> >>> describes how we can use the sequence numbers to debug what
> >> >> >>> and where
> >> things went wrong.
> >> >> >>>
> >> >> >>> Cheers, Manav
> >> >> >>>
> >> >> >>>
> >> >> >>>> draft-ashesh-bfd-stability-01 was presented again during
> >> >> >>>> IETF-91 in Honolulu.  The slides can be viewed here:
> >> >> >>>>
> >> >> >>>> To attempt to simplify the presentation, the contentious
> >> >> >>>> portion of the timers were removed from the proposal, leaving
> >> >> >>>> only the sequence numbering for detecting loss of BFD async
> packets.
> >> >> >>>>
> >> >> >>>> adopted as a WG item, the sense of the room was very quiet.
> >> >> >>>> As promised, this is to inquire for support for this draft on
> >> >> >>>> the WG mailing list to make sure the whole group has a voice.
> >> >> >>>>
> >> >> >>>> It should be noted that post-meeting discussion on the fate
> >> >> >>>> of this draft noted that BFD authentication code points are

---

From: Manav Bhatia <[email-masked]>
Date: Thu, 27 Nov 2014 21:47:16 +0530

Hi Santosh,

> You are suggesting to add the debug trailer with or without authentication right?

without.

> I did not get this. Did you mean the length of the auth TVL will not include this length and will be only in IP header?

Yes, it would only be in the IP header.

Cheers, Manav

---

From: Abhishek Verma <[email-masked]>
Date: Fri, 28 Nov 2014 03:13:04 +0530

Hi,

> important

Agree with Santosh. This mechanism would be most needed when authentication
is turned on.

Abhishek

---

From: "Fan, Peng" <[email-masked]>
Date: Fri, 28 Nov 2014 18:50:43 +0800

Hi Jeff, all,

I have been following this stability extension from the beginning, and as an
operator I would like to express that this draft enables the "advanced
feature" we desire for BFD to provide additional useful information that
helps operators understand network issues. A relevant use case is detecting
lossy or "quasi-disconnected" links or member LAG links. An example of such
situation we experienced was a loosely connected fiber link resulting in
continuous, small amount of packet loss. BFD could get the information of
lost BFD frames on such unstable link, and probably report when a target
level is reached, say a certain number of frames are lost over a period or
among a total number of frames.

Best regards,
Peng

---

From: "MALLIK MUDIGONDA (mmudigon)" <[email-masked]>
Date: Fri, 28 Nov 2014 11:53:20 +0000

Hi Peng,

If the BFD packets are lost, doesn’t the BFD session go DOWN? Are you saying that packet loss is not big enough to make BFD session go DOWN?

Thanks

Regards
Mallik

From: <Fan>, Peng <[email-masked]<mailto:[email-masked]>>
Date: Friday, 28 November 2014 4:20 pm
To: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: RE: BFD stability follow-up from IETF-91

Hi Jeff, all,

Best regards,
Peng

---

From: "Fan, Peng" <[email-masked]>
Date: Fri, 28 Nov 2014 20:34:08 +0800

Hi Mallik,

Exactly. Packets may be experiencing slight loss, but the link can hardly be
regarded as connected. More importantly, the experience of upper-level
applications can be degraded severely (e.g. TCP traffic is not able to go
fast in face of even small continuous loss). But what if one BFD frame is
lost every three frames? Then the loss rate is 30% on average, which is
already a very severe value.

Best regards,

Peng

From: MALLIK MUDIGONDA (mmudigon) [mailto:[email-masked]] 
Sent: Friday, November 28, 2014 7:53 PM
To: Fan, Peng; [email-masked]
Subject: Re: BFD stability follow-up from IETF-91

Hi Peng,

If the BFD packets are lost, doesn't the BFD session go DOWN? Are you saying
that packet loss is not big enough to make BFD session go DOWN?

Thanks

Regards

Mallik

From: <Fan>, Peng <[email-masked]>
Date: Friday, 28 November 2014 4:20 pm
To: "[email-masked]" <[email-masked]>
Subject: RE: BFD stability follow-up from IETF-91

Hi Jeff, all,

Best regards,

Peng

---

From: "MALLIK MUDIGONDA (mmudigon)" <[email-masked]>
Date: Fri, 28 Nov 2014 12:51:56 +0000

Thanks Peng.

Regards
Mallik

From: <Fan>, Peng <[email-masked]<mailto:[email-masked]>>
Date: Friday, 28 November 2014 6:04 pm
To: Cisco Employee <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: RE: BFD stability follow-up from IETF-91

Hi Mallik,

Exactly. Packets may be experiencing slight loss, but the link can hardly be regarded as connected. More importantly, the experience of upper-level applications can be degraded severely (e.g. TCP traffic is not able to go fast in face of even small continuous loss). But what if one BFD frame is lost every three frames? Then the loss rate is 30% on average, which is already a very severe value.

Best regards,
Peng

From: MALLIK MUDIGONDA (mmudigon) [mailto:[email-masked]]
Sent: Friday, November 28, 2014 7:53 PM
To: Fan, Peng; [email-masked]<mailto:[email-masked]>
Subject: Re: BFD stability follow-up from IETF-91

Hi Peng,

Thanks

Regards
Mallik

Hi Jeff, all,

Best regards,
Peng

---

From: Santosh P K <[email-masked]>
Date: Fri, 28 Nov 2014 13:11:36 +0000

Hi Manav,
      Thanks. Please see inline. 

> >> the seq numbers at the *end* of the BFD packet. This would not be
> >> covered in the Length field carried in the BFD header but would be
> >> accounted for in the length carried in the IP header. The concept of
> >> attaching a trailer is documented well and is used in the IGPs. RFC
> >> 6506 describes one such trailer
> >

If we can go with trailer method then it can be applicable to both with and without Auth right? I was thinking having two solution might complicate things :)? 

Thanks
Santosh P K 

> >> >> Can the authors add some text on how this debugging mechanism
> >> >> would work if somebody employs BFD authentication?
> >> >
> >> >> > Mach Chen
> >> >> >> > This triggers me think out there should be another solution
> >> >> >> > for getting the Tx and Rx timestamps without encoding the
> >> >> >> > timestamps in the BFD
> >> >> >> packets.
> >> >> need the marking based solution. Suppose that someone want to
> >> >> monitor the delay of a BFD packet , just record and save the
> >> >> timestamp at the Tx side, which indexed by the sequence number.
> >> >> Similarly, do the same at the Rx side. Then based on the
> >> >> timestamps from both Tx and Rx, and using the sequence number to
> >> >> correlate the timestamps, it can also provide a way to monitor the
> delay of the BFD packet.
> >> >> >
> >> >> >> >>> I believe the work is important and addresses something
> >> >> >> >>> thats really required (spent too much time debugging why
> >> >> >> >>> BFD
> >> flapped!).
> >> >> >> >>
> >> >> >> >>> side Time stamping would have helped in debugging whether
> >> >> >> >>> the
> >> >> BFD
> >> >> >> >>> packet was sent late, or whether the packet was sent on
> >> >> >> >>> time and also arrived on time but was delayed when passing
> >> >> >> >>> it up the BFD stack/processor (lay in the RX buffer for tad
> >> >> >> >>> too
> >> >> >> >>> long)
> >> >> >> >>
> >> >> >> >> And even this point is less relevant with sequence numbers
> >> >> >> >> as this number allows the identification of packets and thus
> >> >> >> >> the correlation of information from the Tx and Rx system.
> >> >> >> >
> >> >> >> > This triggers me think out there should be another solution
> >> >> >> > for getting the Tx and Rx timestamps without encoding the
> >> >> >> > timestamps in the BFD
> >> >> >> packets.
> >> >> >> >>> I vividly remember the original intent of the stability
> >> >> >> >>> draft was to help debug BFD failures -- to isolate the
> >> >> >> >>> issue at the RX or the TX side Time stamping would have
> >> >> >> >>> helped in debugging whether the BFD packet was sent late,
> >> >> >> >>> or whether the packet was sent on time and also arrived on
> >> >> >> >>> time but was delayed when passing it up the BFD
> >> >> >> >>> stack/processor (lay in the RX buffer for tad too long),
> >> >> >> >>> etc. But then time stamping came with its own set of issues,
> >> >> >> >>> that it did eventually receive the missing BFD packets
> >> >> >> >>> (based on the seq
> >> >> #).
> >> >> >> >>> How would it know which end was misbehaving? Was it a delay
> >> >> >> >>> at the TX side, or was it the RX that delayed passing the
> >> >> >> >>> packets to the BFD process(or). This is usually what we
> >> >> >> >>> want to debug and i want to understand how this draft with
> >> >> >> >>> sequence numbers can unequivocally tell me that.
> >> >> >> >>>
> >> >> >> >>> I believe the work is important and addresses something
> >> >> >> >>> thats really required (spent too much time debugging why
> >> >> >> >>> BFD
> >> flapped!).
> >> >> >> >>> On Wed, Nov 26, 2014 at 5:49 AM, Jeffrey Haas
> >> >> >> >>> <[email-masked]>
> >> >> wrote:
> >> >> >> >>>> http://www.ietf.org/proceedings/91/slides/slides-91-bfd-4.
> >> >> >> >>>> ppt
> >> >> >> >>>> x
> >> >> >> >>>>
> >> >> >> >>>> portion of the timers were removed from the proposal,
> >> >> >> >>>> leaving only the sequence numbering for detecting loss of
> >> >> >> >>>> BFD async
> >> packets.
> >> >> >> >>>>
> >> >> >> >>>> When the room was polled to see whether the draft should
> >> >> >> >>>> be adopted as a WG item, the sense of the room was very
> quiet.
> >> >> >> >>>> As promised, this is to inquire for support for this draft
> >> >> >> >>>> on the WG mailing list to make sure the whole group has a
> voice.
> >> >> >> >>>>
> >> >> >> >>>> It should be noted that post-meeting discussion on the
> >> >> >> >>>> fate of this draft noted that BFD authentication code
> >> >> >> >>>> points are plentiful and are available with expert review.
> >> >> >> >>>> Should the draft authors wish to continue this work as
> >> >> >> >>>> Experimental, that is an
> >> >> option.
> >> >> >> >>>>
> >> >> >> >>>> -- Jeff
> >> >> >> >>>>
> >> >> >> >>>
> >> >> >> >
> >> >> >
> >> >

---

From: Manav Bhatia <[email-masked]>
Date: Fri, 28 Nov 2014 18:51:20 +0530

Yes, it will work for both with and without Auth.

Cheers, Manav

---

From: Gregory Mirsky <[email-masked]>
Date: Fri, 28 Nov 2014 19:36:30 +0000

Hi Peng,
this is very interesting scenario. I think that if BFD experiences ~30% packet loss, then highly likely so are affected other applications. Then it is not just BFD issue but condition that should be detected  by performance measurement method, whether active or passive packet loss measurement.
I'm convinced that overloading BFD with performance measurement provisions is counter-productive and is inappropriate.

                Regards,
                                Greg

From: Rtg-bfd [mailto:[email-masked]] On Behalf Of Fan, Peng
Sent: Friday, November 28, 2014 4:34 AM
To: 'MALLIK MUDIGONDA (mmudigon)'; [email-masked]
Subject: RE: BFD stability follow-up from IETF-91

Hi Mallik,

Best regards,
Peng

Hi Peng,

If the BFD packets are lost, doesn't the BFD session go DOWN? Are you saying that packet loss is not big enough to make BFD session go DOWN?

Thanks

Regards
Mallik

Hi Jeff, all,

Best regards,
Peng

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 28 Nov 2014 14:55:37 -0500

[Speaking as an individual contributor...]

> I'm convinced that overloading BFD with performance measurement provisions is counter-productive and is inappropriate.

My opinion is about halfway between your opinion, Greg.

I agree that we wish to be very cautious about overloading BFD with
components that are in other OAM mechanisms.  Among my desire for such
caution is that IEEE has expressed interest in not having us step on their
technologies and this would create paperwork for the chairs. :-)

But where I think we diverge slightly comes from experience in helping the
working group and vendors wend their way through implementing BFD for LAG.
During that discussion, it was very clear that depending on the vendor, the
architecture and sometimes specific chipsets that "BFD" lived in very
different pieces of underlying architecture.

What this means is that trying to do very tight timing things will run into
practical issues in having to figure out what the perspective of the timings
are.  Is it some underlying L2? L3? Something between?  At what point do you
realize you are measuring contradictory things?

But similarly, when trying to measure and account for loss, having some data
is useful simply because it helps you determine that the component that is
*responsible for BFD* may be experiencing loss.  Depending on your
architecture, this may be the underlying layer-1, layer-2 or something else.
In such cases, the lower-layer OAM is better to troubleshoot.  But in cases
where your lower-layer OAM doesn't indicate the loss, you still need to
understand that there is BFD-level loss.

I encourage participants in this discussion to remember this detail: We are
trying to help measure BFD loss.  Trying to read too much detail into what
that means outside of BFD may lead you to erroneous conclusions depending on
a given implementation.

Thus, consider what is best for BFD.

-- Jeff

---

From: Gregory Mirsky <[email-masked]>
Date: Sat, 29 Nov 2014 03:17:54 +0000

Hi Jeff,
I absolutely agree with continuing discussion and sharing experiences from real-life deployments and interoperability of BFD in IP and MPLS networks, single-, multi-hop and over LAG constituents. From that we're learning and that helps us make our implementations more robust, reliable, and interoperable. And certainly, where it benefits the community informational RFCs, like BFD intervals or RFC 7325 MPLS Forwarding Compliance and Performance Requirements, to document the cases and our recommendations. But I'm somewhat concerned with anything that targets standard status, even as optional functionality, even though it is not proven that the problem is in the standard, not in an implementation.

	Regards,
		Greg

-----Original Message-----
From: Jeffrey Haas [mailto:[email-masked]] 
Sent: Friday, November 28, 2014 11:56 AM
To: Gregory Mirsky
Cc: Fan, Peng; 'MALLIK MUDIGONDA (mmudigon)'; [email-masked]
Subject: Re: BFD stability follow-up from IETF-91

I agree that we wish to be very cautious about overloading BFD with components that are in other OAM mechanisms.  Among my desire for such caution is that IEEE has expressed interest in not having us step on their technologies and this would create paperwork for the chairs. :-)

But where I think we diverge slightly comes from experience in helping the working group and vendors wend their way through implementing BFD for LAG.
During that discussion, it was very clear that depending on the vendor, the architecture and sometimes specific chipsets that "BFD" lived in very different pieces of underlying architecture.

What this means is that trying to do very tight timing things will run into practical issues in having to figure out what the perspective of the timings are.  Is it some underlying L2? L3? Something between?  At what point do you realize you are measuring contradictory things?

But similarly, when trying to measure and account for loss, having some data is useful simply because it helps you determine that the component that is *responsible for BFD* may be experiencing loss.  Depending on your architecture, this may be the underlying layer-1, layer-2 or something else.
In such cases, the lower-layer OAM is better to troubleshoot.  But in cases where your lower-layer OAM doesn't indicate the loss, you still need to understand that there is BFD-level loss.

I encourage participants in this discussion to remember this detail: We are trying to help measure BFD loss.  Trying to read too much detail into what that means outside of BFD may lead you to erroneous conclusions depending on a given implementation.

-- Jeff