Subject: Re: BFD stability follow-up from IETF-91

From: Marc Binderberger <[email-masked]>
Date: Mon, 1 Dec 2014 00:54:54 -0800

Hello Manav!

quite some emails about this topic :-) Let me nevertheless reply to this one:

> Now given just the sequence numbers its almost impossible for A to
> know whether the issue was at the RX or the TX side.

... or in between (the link/Internet).

For the Rx side: you timestamp the packets when the hardware receives them. 
This stamp could be in some additional "pak header", something that 
implementations probably have (but yes, it's an implementation detail for me, 
not an aspect of the BFD packet).

Then you timestamp the packet whenever you receive it in your software path, 
assuming the BFD implementation is in software. You may even have additional 
time stamps while the packet proceeds through the various functions.

You have a rotating buffer for these information and save the last N packet 
informations of the session when it goes down. Plus unexpected packets like 
the delayed "Up" packets while you are already in Down.

If the sender does something equivalent on the Tx side and also saving N 
packet information after receiving the first "Down" packet back from the Rx 
node you have Tx and Rx delta times and you can correlate them based on the 
sequence number. If N is large enough, e.g. N = 10 or 50 or such, then you 
should be able to debug the session how it went down.

For your example the Rx timestamps would tell that "101", "102" and "103" 
arrived at your Rx hardware with a delay. If the Tx timestamps show no delay 
in the Tx path then you would conclude it's the link/Internet in between.

If you manage to synchronize the systems down to a few milliseconds then the 
timestamps would even allow to explicitly measure the link delay, that Tx 
packet generation happened without a delay etc..

True, this requires you have the information from both, the Rx and the Tx 
node.

This sounds clumsy, compared to the elegant all-in-one-packet but at the end 
both the Tx and Rx side may need multiple timestamps and some history anyway 
to debug their system if a problem shows up.

Regards, Marc

> My claim is that just using sequence numbers may NOT help.
> 
> A BFD session with an interval of 33ms on router A will flap if it
> does not see any BFD packet for 100ms.
> 
> Assume that the last seq number that A sees from the remote end is,
> say 100. A will bring down the BFD session if it now does not see 101,
> 102 and 103 in the next 100ms.
> 
> Further assume that these packets were not seen by A and the session
> flaps. However, we get these 3 BFD packets immediately after this flap
> -- at 100ms + some_delta.
> 
> Now given just the sequence numbers its almost impossible for A to
> know whether the issue was at the RX or the TX side.
> 
> Am i missing something?

> 
> Cheers, Manav
> 
> 
> On Wed, Nov 26, 2014 at 2:20 PM, Marc Binderberger <[email-masked]> wrote:
>> Hello Manav,
>> 
>>> I believe the work is important and addresses something thats really
>>> required (spent too much time debugging why BFD flapped!).
>> 
>> agree :-) we should keep the discussion alive.
>> 
>> 
>>> side Time stamping would have helped in debugging whether the BFD
>>> packet was sent late, or whether the packet was sent on time and also
>>> arrived on time but was delayed when passing it up the BFD
>>> stack/processor (lay in the RX buffer for tad too long)
>> 
>> well, I can see a point in having the Tx timestamps in the packet mainly 
>> for
>> the purpose of knowing "this" packet was okay/not okay on the Tx side and 
>> to
>> correlate it with your local Rx measurement.
>> 
>> And even this point is less relevant with sequence numbers as this number
>> allows the identification of packets and thus the correlation of 
>> information
>> from the Tx and Rx system.
>> 
>> 
>> Regards, Marc
>> 
>> 
>> 
>> 
>> 
>> 
>> On Wed, 26 Nov 2014 12:26:41 +0530, Manav Bhatia wrote:
>>> Hi Jeff,
>>> 
>>> I vividly remember the original intent of the stability draft was to
>>> help debug BFD failures -- to isolate the issue at the RX or the TX
>>> side Time stamping would have helped in debugging whether the BFD
>>> packet was sent late, or whether the packet was sent on time and also
>>> arrived on time but was delayed when passing it up the BFD
>>> stack/processor (lay in the RX buffer for tad too long), etc. But then
>>> time stamping came with its own set of issues, and was hence dropped
>>> from the original draft.
>>> 
>>> Can the authors send a summary on the list on why time stamping was
>>> dropped so that we're all clear on that one.
>>> 
>>> The current proposal does help but is not complete.
>>> 
>>> Assume that the RX end loses a BFD session and learns later that it
>>> did eventually receive the missing BFD packets (based on the seq #).
>>> How would it know which end was misbehaving? Was it a delay at the TX
>>> side, or was it the RX that delayed passing the packets to the BFD
>>> process(or). This is usually what we want to debug and i want to
>>> understand how this draft with sequence numbers can unequivocally tell
>>> me that.
>>> 
>>> I believe the work is important and addresses something thats really
>>> required (spent too much time debugging why BFD flapped!). Clearly
>>> what would help is putting a small section that describes how we can
>>> use the sequence numbers to debug what and where things went wrong.
>>> 
>>> Cheers, Manav
>>> 
>>> 
>>> On Wed, Nov 26, 2014 at 5:49 AM, Jeffrey Haas <[email-masked]> wrote:
>>>> draft-ashesh-bfd-stability-01 was presented again during IETF-91 in
>>>> Honolulu.  The slides can be viewed here:
>>>> 
>>>> http://www.ietf.org/proceedings/91/slides/slides-91-bfd-4.pptx
>>>> 
>>>> To attempt to simplify the presentation, the contentious portion of the
>>>> timers were removed from the proposal, leaving only the sequence 
>>>> numbering
>>>> for detecting loss of BFD async packets.
>>>> 
>>>> When the room was polled to see whether the draft should be adopted as a 
>>>> WG
>>>> item, the sense of the room was very quiet.  As promised, this is to
>>>> inquire
>>>> for support for this draft on the WG mailing list to make sure the whole
>>>> group has a voice.
>>>> 
>>>> It should be noted that post-meeting discussion on the fate of this draft
>>>> noted that BFD authentication code points are plentiful and are available
>>>> with expert review.  Should the draft authors wish to continue this work 
>>>> as
>>>> Experimental, that is an option.
>>>> 
>>>> -- Jeff
>>>> 
>>> 
>

---

From: Marc Binderberger <[email-masked]>
Date: Mon, 1 Dec 2014 01:38:41 -0800

Hello Manav,

> One way to solve this problem is by attaching a debug trailer that
> only carries the seq numbers at the *end* of the BFD packet. This
> would not be covered in the Length field carried in the BFD header but
> would be accounted for in the length carried in the IP header.

BFD itself is not related to IP, i.e. there is not always an IP header.
Sure, the encapsulating "frame" may provide a length but actually, why not 
covering the debug trailer with the BFD length?

If this is solely for debug purpose than this may work. For simple 
copying-out into e.g. a packet trace buffer it would be even simpler to have 
the BFD length covering the trailer.
If hardware is supposed to process the trailer information (other than 
copying out) then it's getting ugly - having fixed position, fixed length 
extension headers would be preferable for simple access.

Another idea is to use the 0x80 bit of the auth type to distinguish between a 
"normal" authentication header and a "sequence + authentication".

Regards, Marc

On Thu, 27 Nov 2014 21:12:00 +0530, Manav Bhatia wrote:
> Hi Santosh,
> 
> You could use the crypto sequence numbers carried in the meticulous
> cryptographic auth for detecting packet losses. However, this breaks
> when you use non-meticulous crypto authentication since the sequence
> number is only incremented occasionally there. This i believe is a
> deal breaker since i really envision non meticulous mode to be the one
> being widely deployed. In fact we were supposed to write a draft on
> that and i guess it just fell through the cracks (lemme ping my
> co-author on that !)
> 
> One way to solve this problem is by attaching a debug trailer that
> only carries the seq numbers at the *end* of the BFD packet. This
> would not be covered in the Length field carried in the BFD header but
> would be accounted for in the length carried in the IP header. The
> concept of attaching a trailer is documented well and is used in the
> IGPs. RFC 6506 describes one such trailer for OSPFv3. The catch
> however is that this debug trailer will NOT be covered by the BFD
> authentication. Is this acceptable to the WG?
> 
> I think the problem of diagnosing a BFD flap becomes all the more
> important with BFD authentication turned on since then we have more
> points where a delay can be inserted.
> 
> Cheers, Manav
> 
> 
> On Thu, Nov 27, 2014 at 8:32 PM, Santosh P K <[email-masked]> wrote:
>> Manav,
>>     This is good question.
>> 
>>> Can the authors add some text on how this debugging mechanism would
>>> work if somebody employs BFD authentication?
>> 
>> Right now we have considered without authentication (we are setting A 
>> bit). We should add some text on how we can use both Auth and de bug TLV. 
>> Is there any suggestion you have? I will get back to you on this.
>> 
>> 
>> Thanks
>> Santosh P K
>> 
>>>> -----Original Message-----
>>>> From: Rtg-bfd [mailto:[email-masked]] On Behalf Of Mach Chen
>>>> Sent: Thursday, November 27, 2014 2:13 PM
>>>> To: Marc Binderberger
>>>> Cc: [email-masked]
>>>> Subject: RE: BFD stability follow-up from IETF-91
>>>> 
>>>> Hi Marc,
>>>> 
>>>>> -----Original Message-----
>>>>> From: Marc Binderberger [mailto:[email-masked]]
>>>>> Sent: Thursday, November 27, 2014 1:43 AM
>>>>> To: Mach Chen
>>>>> Cc: Manav Bhatia; [email-masked]
>>>>> Subject: RE: BFD stability follow-up from IETF-91
>>>>> 
>>>>> Hello Mach,
>>>>> 
>>>>>> This triggers me think out there should be another solution for
>>>>>> getting the Tx and Rx timestamps without encoding the timestamps in
>>>>>> the BFD
>>>>> packets.
>>>>>> For example, the Tx and Rx systems could just save timestamps
>>>>>> locally or send them to a centralized entity and then use the
>>>>>> sequence numbers to correlate them for further analyzing.
>>>>> 
>>>>> I remember some discussion on NVO3 about how many bits it takes ;-) -
>>>>> could you send the links/draft names you are working on to this list?
>>>>> May be useful for further discussions.
>>>> 
>>>> Sure, here is the 
>>>> link(http://tools.ietf.org/html/draft-chen-ippm-coloring-
>>> based-ipfpm-framework-02) for the reference.
>>>> 
>>>> But here I want to say is that since we have sequence number, we may not
>>> need the marking based solution. Suppose that someone want to monitor
>>> the delay of a BFD packet , just record and save the timestamp at the Tx 
>>> side,
>>> which indexed by the sequence number. Similarly, do the same at the Rx
>>> side. Then based on the timestamps from both Tx and Rx, and using the
>>> sequence number to correlate the timestamps, it can also provide a way to
>>> monitor the delay of the BFD packet.
>>>> 
>>>> That means, only if there is sequence number, even if without carrying 
>>>> the
>>> timestamp in the BFD packet, BFD packet delay can be measured.
>>>> 
>>>> Best regards,
>>>> Mach
>>>> 
>>>>> 
>>>>> 
>>>>> Thanks & Regards,
>>>>> Marc
>>>>> 
>>>>> 
>>>>> 
>>>>> On Wed, 26 Nov 2014 09:17:32 +0000, Mach Chen wrote:
>>>>>> Hi Marc and Manav,
>>>>>> 
>>>>>>> -----Original Message-----
>>>>>>> From: Rtg-bfd [mailto:[email-masked]] On Behalf Of Marc
>>>>>>> Binderberger
>>>>>>> Sent: Wednesday, November 26, 2014 4:50 PM
>>>>>>> To: Manav Bhatia
>>>>>>> Cc: [email-masked]
>>>>>>> Subject: Re: BFD stability follow-up from IETF-91
>>>>>>> 
>>>>>>> Hello Manav,
>>>>>>> 
>>>>>>>> I believe the work is important and addresses something thats
>>>>>>>> really required (spent too much time debugging why BFD flapped!).
>>>>>>> 
>>>>>>>> packet was sent late, or whether the packet was sent on time and
>>>>>>>> also arrived on time but was delayed when passing it up the BFD
>>>>>>> well, I can see a point in having the Tx timestamps in the packet
>>>>>>> mainly for the purpose of knowing "this" packet was okay/not okay
>>>>>>> on the Tx side and to correlate it with your local Rx measurement.
>>>>>> 
>>>>>> Yes, this is one solution if people think BFD delay is needed. If
>>>>>> allow to have Tx timestamps to be carried in the packets, seems it
>>>>>> should be no problem to leave a seat for the Rx timestamps as well
>>>>>> :-). After all, with both Tx and Rx timestamp, it may simplify the
>>>>> implementation.
>>>>>> 
>>>>>>> 
>>>>>>> And even this point is less relevant with sequence numbers as this
>>>>>>> number allows the identification of packets and thus the
>>>>>>> correlation of information from the Tx and Rx system.
>>>>>> 
>>>>>> Indeed, the sequence number helps a lot for the correlation between
>>>>>> the Tx and Rx system.
>>>>>> 
>>>>>> This triggers me think out there should be another solution for
>>>>>> getting the Tx and Rx timestamps without encoding the timestamps in
>>>>>> the BFD
>>>>> packets.
>>>>>> For example, the Tx and Rx systems could just save timestamps
>>>>>> locally or send them to a centralized entity and then use the
>>>>>> sequence numbers to correlate them for further analyzing.
>>>>>> 
>>>>>> Best regards,
>>>>>> Mach
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Regards, Marc
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> I vividly remember the original intent of the stability draft was
>>>>>>>> to help debug BFD failures -- to isolate the issue at the RX or
>>>>>>>> the TX side Time stamping would have helped in debugging whether
>>>>>>>> the BFD packet was sent late, or whether the packet was sent on
>>>>>>>> time and also arrived on time but was delayed when passing it up
>>>>>>>> the BFD stack/processor (lay in the RX buffer for tad too long),
>>>>>>>> etc. But then time stamping came with its own set of issues, and
>>>>>>>> was hence dropped from the original draft.
>>>>>>>> 
>>>>>>>> Can the authors send a summary on the list on why time stamping
>>>>>>>> was dropped so that we're all clear on that one.
>>>>>>>> 
>>>>>>>> Assume that the RX end loses a BFD session and learns later that
>>>>>>>> it did eventually receive the missing BFD packets (based on the seq
>>> #).
>>>>>>>> How would it know which end was misbehaving? Was it a delay at
>>>>>>>> the TX side, or was it the RX that delayed passing the packets to
>>>>>>>> the BFD process(or). This is usually what we want to debug and i
>>>>>>>> want to understand how this draft with sequence numbers can
>>>>>>>> unequivocally tell me that.
>>>>>>>> 
>>>>>>>> I believe the work is important and addresses something thats
>>>>>>>> really required (spent too much time debugging why BFD flapped!).
>>>>>>>> Clearly what would help is putting a small section that describes
>>>>>>>> how we can use the sequence numbers to debug what and where
>>> things went wrong.
>>>>>>>> 
>>>>>>>> Cheers, Manav
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> draft-ashesh-bfd-stability-01 was presented again during IETF-91
>>>>>>>>> in Honolulu.  The slides can be viewed here:
>>>>>>>>> 
>>>>>>>>> To attempt to simplify the presentation, the contentious portion
>>>>>>>>> of the timers were removed from the proposal, leaving only the
>>>>>>>>> sequence numbering for detecting loss of BFD async packets.
>>>>>>>>> 
>>>>>>>>> When the room was polled to see whether the draft should be
>>>>>>>>> adopted as a WG item, the sense of the room was very quiet.  As
>>>>>>>>> promised, this is to inquire for support for this draft on the
>>>>>>>>> WG mailing list to make sure the whole group has a voice.
>>>>>>>>> 
>>>>>>>>> It should be noted that post-meeting discussion on the fate of
>>>>>>>>> this draft noted that BFD authentication code points are
>>>>>>>>> plentiful and are available with expert review.  Should the
>>>>>>>>> draft authors wish to continue this work as Experimental, that is an
>>> option.
>>>>>>>>> 
>>>>>>>>> -- Jeff
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>> 
>> 
>

---

From: Santosh P K <[email-masked]>
Date: Wed, 3 Dec 2014 13:39:29 +0000

Hello Manav and Marc,

> If this is solely for debug purpose than this may work. For simple copying-out
> into e.g. a packet trace buffer it would be even simpler to have the BFD
> length covering the trailer.

Fixed length would be easy to process in hardware. Problem is when we have many have extensions in future. If we want to use only one extension that is at the last then I will be forced to pad all the other extension ahead of it? This might not be a problem if we have fewer extensions but might become problem when there are too many extensions. 

> "normal" authentication header and a "sequence + authentication".

I think this is good. In the BFD extension TLV we still have many reserved bits that can be used as well?

Thanks
Santosh P K

---

From: Gregory Mirsky <[email-masked]>
Date: Wed, 3 Dec 2014 20:52:25 +0000

Dear All,
had authors of the proposal or we already dismissed use of BFD Echo? I've scanned the thread and couldn't find trace of us discussing BFD Echo mode. I think that it is more suitable for experimentation and unorthodox use.

	Regards,
		Greg

-----Original Message-----
From: Rtg-bfd [mailto:[email-masked]] On Behalf Of Santosh P K
Sent: Wednesday, December 03, 2014 5:39 AM
To: Marc Binderberger; Manav Bhatia
Cc: [email-masked]
Subject: RE: BFD stability follow-up from IETF-91

Hello Manav and Marc,

> > would not be covered in the Length field carried in the BFD header 
> > but would be accounted for in the length carried in the IP header.
> 
> Sure, the encapsulating "frame" may provide a length but actually, why 
> not covering the debug trailer with the BFD length?
> 
> copying-out into e.g. a packet trace buffer it would be even simpler 
> to have the BFD length covering the trailer.
> copying out) then it's getting ugly - having fixed position, fixed 
> length extension headers would be preferable for simple access.

> 
> Another idea is to use the 0x80 bit of the auth type to distinguish 
> between a "normal" authentication header and a "sequence + authentication".
> 

Thanks
Santosh P K 

> > deal breaker since i really envision non meticulous mode to be the 
> > one being widely deployed. In fact we were supposed to write a draft 
> > on that and i guess it just fell through the cracks (lemme ping my 
> > co-author on that !)
> >
> > would not be covered in the Length field carried in the BFD header 
> > but would be accounted for in the length carried in the IP header. 
> > The concept of attaching a trailer is documented well and is used in 
> > the IGPs. RFC 6506 describes one such trailer for OSPFv3. The catch 
> >>> Can the authors add some text on how this debugging mechanism 
> >>> would work if somebody employs BFD authentication?
> >>
> >> Right now we have considered without authentication (we are setting 
> >> A bit). We should add some text on how we can use both Auth and de 
> >> bug
> TLV.
> >>>>>> getting the Tx and Rx timestamps without encoding the 
> >>>>>> timestamps
> in
> >>>>>> the BFD
> >>>>> packets.
> >>>>> I remember some discussion on NVO3 about how many bits it takes 
> >>>>> ;-
> ) -
> >>>> But here I want to say is that since we have sequence number, we 
> >>>> may
> not
> >>> the delay of a BFD packet , just record and save the timestamp at 
> >>> the Tx side, which indexed by the sequence number. Similarly, do 
> >>> the same at the Rx side. Then based on the timestamps from both Tx 
> >>> and Rx, and using the sequence number to correlate the timestamps, 
> >>> it can also provide a way
> to
> >>>> That means, only if there is sequence number, even if without 
> >>>> carrying the
> >>>>>>>> packet was sent late, or whether the packet was sent on time 
> >>>>>>>> and also arrived on time but was delayed when passing it up 
> >>>>>>>> the BFD stack/processor (lay in the RX buffer for tad too 
> >>>>>>>> long)
> >>>>>>>
> >>>>>>> well, I can see a point in having the Tx timestamps in the 
> >>>>>>> packet mainly for the purpose of knowing "this" packet was 
> >>>>>>> okay/not okay on the Tx side and to correlate it with your local Rx measurement.
> >>>>>>
> >>>>>> Yes, this is one solution if people think BFD delay is needed. 
> >>>>>> If allow to have Tx timestamps to be carried in the packets, 
> >>>>>> seems it should be no problem to leave a seat for the Rx 
> >>>>>> timestamps as well :-). After all, with both Tx and Rx 
> >>>>>> timestamp, it may simplify the
> >>>>> implementation.
> >>>>>>
> >>>>>>>
> >>>>>>> And even this point is less relevant with sequence numbers as 
> >>>>>>> this number allows the identification of packets and thus the 
> >>>>>> getting the Tx and Rx timestamps without encoding the 
> >>>>>> timestamps
> in
> >>>>>> the BFD
> >>>>> packets.
> >>>>>>>> I vividly remember the original intent of the stability draft 
> >>>>>>>> was to help debug BFD failures -- to isolate the issue at the 
> >>>>>>>> RX or the TX side Time stamping would have helped in 
> >>>>>>>> debugging
> whether
> >>>>>>>> the BFD packet was sent late, or whether the packet was sent 
> >>>>>>>> on time and also arrived on time but was delayed when passing 
> >>>>>>>> it up the BFD stack/processor (lay in the RX buffer for tad 
> >>>>>>>> too long), etc. But then time stamping came with its own set 
> >>>>>>>> of issues, and was hence dropped from the original draft.
> >>>>>>>>
> >>>>>>>> Can the authors send a summary on the list on why time 
> >>>>>>>> stamping was dropped so that we're all clear on that one.
> >>>>>>>>
> >>>>>>>> Assume that the RX end loses a BFD session and learns later 
> >>>>>>>> that it did eventually receive the missing BFD packets (based 
> >>>>>>>> on the
> seq
> >>> #).
> >>>>>>>> How would it know which end was misbehaving? Was it a delay 
> >>>>>>>> at the TX side, or was it the RX that delayed passing the 
> >>>>>>>> packets to the BFD process(or). This is usually what we want 
> >>>>>>>> to debug and i want to understand how this draft with 
> >>>>>>>> sequence numbers can unequivocally tell me that.
> >>>>>>>>
> >>>>>>>> Clearly what would help is putting a small section that 
> >>>>>>>> describes how we can use the sequence numbers to debug what 
> >>>>>>>> and
> where
> >>> things went wrong.
> >>>>>>>>
> >>>>>>>> Cheers, Manav
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Wed, Nov 26, 2014 at 5:49 AM, Jeffrey Haas 
> >>>>>>>> <[email-masked]>
> >>> wrote:
> >>>>>>>>> draft-ashesh-bfd-stability-01 was presented again during 
> >>>>>>>>> IETF-91 in Honolulu.  The slides can be viewed here:
> >>>>>>>>>
> >>>>>>>>> To attempt to simplify the presentation, the contentious 
> >>>>>>>>> portion of the timers were removed from the proposal, 
> >>>>>>>>> leaving only the sequence numbering for detecting loss of BFD async packets.
> >>>>>>>>>
> >>>>>>>>> adopted as a WG item, the sense of the room was very quiet.  
> >>>>>>>>> As promised, this is to inquire for support for this draft 
> >>>>>>>>> on the WG mailing list to make sure the whole group has a voice.
> >>>>>>>>>
> >>>>>>>>> It should be noted that post-meeting discussion on the fate 
> >>>>>>>>> of this draft noted that BFD authentication code points are 
> >>>>>>>>> draft authors wish to continue this work as Experimental, 
> >>>>>>>>> that is
> an
> >>> option.
> >>>>>>>>>
> >>>>>>>>> -- Jeff
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> >

---

From: Santosh P K <[email-masked]>
Date: Thu, 4 Dec 2014 05:53:37 +0000

Greg,
   I don't think we have discussed about echo in this context. Echo is good thing but payload of BFD echo packet is decided by local system. Did you mean to add suggestion on how echo packet should look like? Or how echo can help in BFD loss/delay issue? 

Thanks
Santosh P K
 
> From: Gregory Mirsky [mailto:[email-masked]]
> Sent: Thursday, December 04, 2014 2:22 AM
> To: Santosh P K; Marc Binderberger; Manav Bhatia
> Cc: [email-masked]
> had authors of the proposal or we already dismissed use of BFD Echo? I've
> scanned the thread and couldn't find trace of us discussing BFD Echo mode. I
> think that it is more suitable for experimentation and unorthodox use.
> 
> 	Regards,
> 		Greg
> 
> Fixed length would be easy to process in hardware. Problem is when we
> have many have extensions in future. If we want to use only one extension
> that is at the last then I will be forced to pad all the other extension ahead of
> it? This might not be a problem if we have fewer extensions but might
> become problem when there are too many extensions.
> 
> 
> >
> > >>>>>>>> side Time stamping would have helped in debugging whether
> the
> > >>> BFD

---

From: Gregory Mirsky <[email-masked]>
Date: Thu, 4 Dec 2014 06:13:39 +0000

Hi Santosh,
yes, the format and thus interpretation of payload in Echo mode is not defined and that, in my view, is what we need - some open space. And Echo well could be exactly that - no legacy, no backward compatibility (addressee that doesn't support "extended Echo" will simply loop the packet back to sender). Perhaps that will be direction we can discuss and, hopefully, agree on.

	Regards,
		Greg

-----Original Message-----
From: Santosh P K [mailto:[email-masked]] 
Sent: Wednesday, December 03, 2014 9:54 PM
To: Gregory Mirsky; Marc Binderberger; Manav Bhatia
Cc: [email-masked]
Subject: RE: BFD stability follow-up from IETF-91

Greg,

Thanks
Santosh P K
 
> had authors of the proposal or we already dismissed use of BFD Echo? 
> I've scanned the thread and couldn't find trace of us discussing BFD 
> Echo mode. I think that it is more suitable for experimentation and unorthodox use.
> 
> 	Regards,
> 		Greg
> 
> > Sure, the encapsulating "frame" may provide a length but actually, 
> > why not covering the debug trailer with the BFD length?
> >
> > If hardware is supposed to process the trailer information (other 
> > than copying out) then it's getting ugly - having fixed position, 
> > fixed length extension headers would be preferable for simple access.
> 
> have many have extensions in future. If we want to use only one 
> extension that is at the last then I will be forced to pad all the 
> other extension ahead of it? This might not be a problem if we have 
> fewer extensions but might become problem when there are too many extensions.
> 
> 
> >
> > > You could use the crypto sequence numbers carried in the 
> > > meticulous cryptographic auth for detecting packet losses. 
> > > However, this breaks when you use non-meticulous crypto 
> > > authentication since the sequence number is only incremented 
> > > occasionally there. This i believe is a deal breaker since i 
> > > really envision non meticulous mode to be the one being widely 
> > > deployed. In fact we were supposed to write a draft on that and i 
> > > guess it just fell through the cracks (lemme ping my co-author on 
> > > that !)
> > >
> > > The concept of attaching a trailer is documented well and is used 
> > > in the IGPs. RFC 6506 describes one such trailer for OSPFv3. The 
> > > catch however is that this debug trailer will NOT be covered by 
> > > the BFD authentication. Is this acceptable to the WG?
> > >
> > > important with BFD authentication turned on since then we have 
> > > more points where a delay can be inserted.
> > >
> > > Cheers, Manav
> > >
> > >
> > > On Thu, Nov 27, 2014 at 8:32 PM, Santosh P K 
> > > <[email-masked]>
> > wrote:
> > >> Manav,
> > >>     This is good question.
> > >>
> > >> Right now we have considered without authentication (we are 
> > >> setting A bit). We should add some text on how we can use both 
> > >> Auth and de bug
> > TLV.
> > >>>> From: Rtg-bfd [mailto:[email-masked]] On Behalf Of 
> > >>>> Mach
> > Chen
> > >>>>>> This triggers me think out there should be another solution 
> > >>>>>> for getting the Tx and Rx timestamps without encoding the 
> > >>>>>> timestamps
> > in
> > >>>>>> the BFD
> > >>>>> packets.
> > >>>> But here I want to say is that since we have sequence number, 
> > >>>> we may
> > not
> > >>> the delay of a BFD packet , just record and save the timestamp 
> > >>> at the Tx side, which indexed by the sequence number. Similarly, 
> > >>> do the same at the Rx side. Then based on the timestamps from 
> > >>> both Tx and Rx, and using the sequence number to correlate the 
> > >>> timestamps, it can also provide a way
> > to
> > >>>>>>>> I believe the work is important and addresses something 
> > >>>>>>>> thats really required (spent too much time debugging why 
> > >>>>>>>> BFD
> > flapped!).
> > >>>>>>>
> > >>>>>>>> packet was sent late, or whether the packet was sent on 
> > >>>>>>>> time and also arrived on time but was delayed when passing 
> > >>>>>>>> too
> > >>>>>>> okay/not okay on the Tx side and to correlate it with your 
> > >>>>>>> local Rx
> measurement.
> > >>>>>>
> > >>>>>>> And even this point is less relevant with sequence numbers 
> > >>>>>>> as this number allows the identification of packets and thus 
> > >>>>>>> the correlation of information from the Tx and Rx system.
> > >>>>>>
> > >>>>>> This triggers me think out there should be another solution 
> > >>>>>> for getting the Tx and Rx timestamps without encoding the 
> > >>>>>> timestamps
> > in
> > >>>>>> the BFD
> > >>>>> packets.
> > >>>>>>>> I vividly remember the original intent of the stability 
> > >>>>>>>> draft was to help debug BFD failures -- to isolate the 
> > >>>>>>>> issue at the RX or the TX side Time stamping would have 
> > >>>>>>>> helped in debugging
> > whether
> > >>>>>>>> the BFD packet was sent late, or whether the packet was 
> > >>>>>>>> sent on time and also arrived on time but was delayed when 
> > >>>>>>>> passing it up the BFD stack/processor (lay in the RX buffer 
> > >>>>>>>> for tad too long), etc. But then time stamping came with 
> > >>>>>>>> its own set of issues, and was hence dropped from the original draft.
> > >>>>>>>>
> > >>>>>>>> that it did eventually receive the missing BFD packets 
> > >>>>>>>> (based on the
> > seq
> > >>> #).
> > >>>>>>>> packets to the BFD process(or). This is usually what we 
> > >>>>>>>> want to debug and i want to understand how this draft with 
> > >>>>>>>> I believe the work is important and addresses something 
> > >>>>>>>> thats really required (spent too much time debugging why 
> > >>>>>>>> BFD
> > flapped!).
> > >>>>>>>>> http://www.ietf.org/proceedings/91/slides/slides-91-bfd-4.
> > >>>>>>>>> pp
> > >>>>>>>>> tx
> > >>>>>>>>>
> > >>>>>>>>> leaving only the sequence numbering for detecting loss of 
> > >>>>>>>>> BFD
> async packets.
> > >>>>>>>>>
> > >>>>>>>>> When the room was polled to see whether the draft should 
> > >>>>>>>>> be adopted as a WG item, the sense of the room was very quiet.
> > >>>>>>>>> It should be noted that post-meeting discussion on the 
> > >>>>>>>>> fate of this draft noted that BFD authentication code 
> > >>>>>>>>> points are plentiful and are available with expert review.  
> > >>>>>>>>> Should the draft authors wish to continue this work as 
> > >>>>>>>>> Experimental, that is
> > an
> > >>> option.
> > >>>>>>>>>
> > >>>>>>>>> -- Jeff
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>
> > >

---

From: "Vengada Prasad Govindan (venggovi)" <[email-masked]>
Date: Thu, 4 Dec 2014 11:15:06 +0000

Hello Greg/ Santosh,
  It was my understanding that BFD Async was chosen for stability since there are configurations where BFD echo mode is not supported (e.g. Multi-hop, BFD on LAGs and BFD over MPLS). Am I missing something here, please let me know.
Thanks
Prasad

-----Original Message-----
From: Rtg-bfd [mailto:[email-masked]] On Behalf Of Gregory Mirsky
Sent: Thursday, December 04, 2014 11:44 AM
To: Santosh P K; Marc Binderberger; Manav Bhatia
Cc: [email-masked]
Subject: RE: BFD stability follow-up from IETF-91

Hi Santosh,

	Regards,
		Greg

-----Original Message-----
From: Santosh P K [mailto:[email-masked]]
Sent: Wednesday, December 03, 2014 9:54 PM
To: Gregory Mirsky; Marc Binderberger; Manav Bhatia
Cc: [email-masked]
Subject: RE: BFD stability follow-up from IETF-91

Greg,

Thanks
Santosh P K

---

From: Santosh P K <[email-masked]>
Date: Thu, 4 Dec 2014 11:44:15 +0000

Hi Greg,
   Since packet is processed in local system and remote system will just loop it back, packet can be anything right? I mean do we need to standardized how a packet should look like? There won't be any backward compatibility issues with this. More over on some system it may be possible that looping back packets on same interface might not be allowed.   

Thanks
Santosh P K 

> yes, the format and thus interpretation of payload in Echo mode is not
> defined and that, in my view, is what we need - some open space. And Echo
> well could be exactly that - no legacy, no backward compatibility (addressee
> that doesn't support "extended Echo" will simply loop the packet back to
> sender). Perhaps that will be direction we can discuss and, hopefully, agree
> on.
> 
> 	Regards,
> 		Greg
> 
>    I don't think we have discussed about echo in this context. Echo is good
> thing but payload of BFD echo packet is decided by local system. Did you
> mean to add suggestion on how echo packet should look like? Or how echo
> can help in BFD loss/delay issue?
> 
> Thanks
> Santosh P K
>

---

From: Santosh P K <[email-masked]>
Date: Thu, 4 Dec 2014 11:45:34 +0000

Hi Prasad,
  That's right. We echo has limitations but for single hop we can still use it. I was trying to understand even for single hop do we need to define any packet format for echo?

Thanks
Santosh P K 

> From: Vengada Prasad Govindan (venggovi) [mailto:[email-masked]]
> Sent: Thursday, December 04, 2014 4:45 PM
> To: Gregory Mirsky; Santosh P K; Marc Binderberger; Manav Bhatia
> Cc: [email-masked]
>   It was my understanding that BFD Async was chosen for stability since there
> are configurations where BFD echo mode is not supported (e.g. Multi-hop,
> BFD on LAGs and BFD over MPLS). Am I missing something here, please let
> me know.
> Thanks
> Prasad
>

---

From: Manav Bhatia <[email-masked]>
Date: Thu, 4 Dec 2014 18:02:05 +0530

Greg,

This is not what we want.

We wanted some additional data carried in the BFD packets that can help us
diagnose BFD failures. We dont want an out-of-band mechanism to tell us
whether BFD packets are getting dropped or not (there are other OAM tools
for that). We're not even interested in that. What we want to know is why a
particular BFD session flapped -- whether the issue was at the TX or the
RX. Sending ECHOs will not help you there -- You need to send some data
along with your regular BFD packet for it to be meaningful.

Cheers, Manav

On Thu, Dec 4, 2014 at 11:43 AM, Gregory Mirsky <[email-masked]

> well could be exactly that - no legacy, no backward compatibility
> (addressee that doesn't support "extended Echo" will simply loop the packet
> back to sender). Perhaps that will be direction we can discuss and,
> hopefully, agree on.
>
>         Regards,
>                 Greg
>
>    I don't think we have discussed about echo in this context. Echo is
> good thing but payload of BFD echo packet is decided by local system. Did
> you mean to add suggestion on how echo packet should look like? Or how echo

---

From: "MALLIK MUDIGONDA (mmudigon)" <[email-masked]>
Date: Thu, 4 Dec 2014 11:45:22 +0000

Hi Greg,

The other point is, if echo mode is used, how do you verify that the receive path is ok. Echo packets are going to be sent back in the forwarding path. If there is a problem in the receive path we will not be able to detect that with echo.

Thanks

Regards
Mallik

From: "Vengada Prasad Govindan (venggovi)" <[email-masked]<mailto:[email-masked]>>
Date: Thursday, 4 December 2014 4:45 pm
To: Gregory Mirsky <[email-masked]<mailto:[email-masked]>>, Santosh P K <[email-masked]<mailto:[email-masked]>>, Marc Binderberger <[email-masked]<mailto:[email-masked]>>, Manav Bhatia <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: RE: BFD stability follow-up from IETF-91

Hello Greg/ Santosh,

-----Original Message-----
From: Rtg-bfd [mailto:[email-masked]] On Behalf Of Gregory Mirsky
Sent: Thursday, December 04, 2014 11:44 AM
To: Santosh P K; Marc Binderberger; Manav Bhatia
Cc: [email-masked]<mailto:[email-masked]>
Subject: RE: BFD stability follow-up from IETF-91

Hi Santosh,

Regards,
Greg

-----Original Message-----
From: Santosh P K [mailto:[email-masked]]
Sent: Wednesday, December 03, 2014 9:54 PM
To: Gregory Mirsky; Marc Binderberger; Manav Bhatia
Cc: [email-masked]<mailto:[email-masked]>
Subject: RE: BFD stability follow-up from IETF-91

Greg,

Thanks
Santosh P K
-----Original Message-----
From: Gregory Mirsky [mailto:[email-masked]]
Sent: Thursday, December 04, 2014 2:22 AM
To: Santosh P K; Marc Binderberger; Manav Bhatia
Cc: [email-masked]<mailto:[email-masked]>
Subject: RE: BFD stability follow-up from IETF-91
Dear All,
-----Original Message-----
From: Rtg-bfd [mailto:[email-masked]] On Behalf Of Santosh P
K
Sent: Wednesday, December 03, 2014 5:39 AM
To: Marc Binderberger; Manav Bhatia
Cc: [email-masked]<mailto:[email-masked]>
Subject: RE: BFD stability follow-up from IETF-91
Hello Manav and Marc,
> between a "normal" authentication header and a "sequence +
authentication".
I think this is good. In the BFD extension TLV we still have many
reserved bits that can be used as well?
Thanks
Santosh P K
> > <[email-masked]<mailto:[email-masked]>>
> wrote:
> >> Manav,
> >>     This is good question.
> >>
> >>>> Cc: [email-masked]<mailto:[email-masked]>
> >>>>> Cc: Manav Bhatia; [email-masked]<mailto:[email-masked]>
> >>>>>>> Cc: [email-masked]<mailto:[email-masked]>
the
measurement.
> >>>>>>>> <[email-masked]<mailto:[email-masked]>>
> >>> wrote:
async packets.

---

From: "MALLIK MUDIGONDA (mmudigon)" <[email-masked]>
Date: Thu, 4 Dec 2014 11:49:02 +0000

Hi Santosh,

With Echo, the receive side is not going to look into the packet. It is just going to send it back and it is the responsibility of the sender to interpret its contents.

Regards
Mallik

From: Santosh P K <[email-masked]<mailto:[email-masked]>>
Date: Thursday, 4 December 2014 5:15 pm
To: "Vengada Prasad Govindan (venggovi)" <[email-masked]<mailto:[email-masked]>>, Gregory Mirsky <[email-masked]<mailto:[email-masked]>>, Marc Binderberger <[email-masked]<mailto:[email-masked]>>, Manav Bhatia <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: RE: BFD stability follow-up from IETF-91

Hi Prasad,

Thanks
Santosh P K

-----Original Message-----
From: Vengada Prasad Govindan (venggovi) [mailto:[email-masked]]
Sent: Thursday, December 04, 2014 4:45 PM
To: Gregory Mirsky; Santosh P K; Marc Binderberger; Manav Bhatia
Cc: [email-masked]<mailto:[email-masked]>
Subject: RE: BFD stability follow-up from IETF-91
Hello Greg/ Santosh,
-----Original Message-----
From: Rtg-bfd [mailto:[email-masked]] On Behalf Of Gregory
Mirsky
-----Original Message-----
From: Santosh P K [mailto:[email-masked]]
Sent: Wednesday, December 03, 2014 9:54 PM
To: Gregory Mirsky; Marc Binderberger; Manav Bhatia
Cc: [email-masked]<mailto:[email-masked]>
Subject: RE: BFD stability follow-up from IETF-91
Greg,

---

From: Santosh P K <[email-masked]>
Date: Thu, 4 Dec 2014 11:52:17 +0000

Mallaik,
  That's correct and that is why I asked if we really need to define any packet format as Greg suggested?

Thanks
Santosh P K

From: MALLIK MUDIGONDA (mmudigon) [mailto:[email-masked]]
Sent: Thursday, December 04, 2014 5:19 PM
To: Santosh P K; Vengada Prasad Govindan (venggovi); Gregory Mirsky; Marc Binderberger; Manav Bhatia
Cc: [email-masked]
Subject: Re: BFD stability follow-up from IETF-91

Hi Santosh,

Regards
Mallik

Hi Prasad,

Thanks
Santosh P K

-----Original Message-----
From: Vengada Prasad Govindan (venggovi) [mailto:[email-masked]]
Sent: Thursday, December 04, 2014 4:45 PM
To: Gregory Mirsky; Santosh P K; Marc Binderberger; Manav Bhatia
Cc: [email-masked]<mailto:[email-masked]>
Subject: RE: BFD stability follow-up from IETF-91
Hello Greg/ Santosh,
-----Original Message-----
From: Santosh P K [mailto:[email-masked]]
Sent: Wednesday, December 03, 2014 9:54 PM
To: Gregory Mirsky; Marc Binderberger; Manav Bhatia
Cc: [email-masked]<mailto:[email-masked]>
Subject: RE: BFD stability follow-up from IETF-91
Greg,

---

From: "Nobo Akiya (nobo)" <[email-masked]>
Date: Thu, 4 Dec 2014 14:00:39 +0000

A quick question to understand where we are.

If we had:

1.      Standardization of Null Authentication (i.e., sequence numbers)

2.      Implementation of local TX/RX timestamp mechanism described by Marc below

What are the core requirements which have not been satisfied?

Thanks!

-Nobo

P.S. No, there is no need to standardize BFD echo contents.

From: Rtg-bfd [mailto:[email-masked]] On Behalf Of Santosh P K
Sent: Thursday, December 04, 2014 6:52 AM
To: MALLIK MUDIGONDA (mmudigon); Vengada Prasad Govindan (venggovi); Gregory Mirsky; Marc Binderberger; Manav Bhatia
Cc: [email-masked]
Subject: RE: BFD stability follow-up from IETF-91

Mallaik,

Thanks
Santosh P K

Hi Santosh,

Regards
Mallik

Hi Prasad,

Thanks
Santosh P K

-----Original Message-----
From: Vengada Prasad Govindan (venggovi) [mailto:[email-masked]]
Sent: Thursday, December 04, 2014 4:45 PM
To: Gregory Mirsky; Santosh P K; Marc Binderberger; Manav Bhatia
Cc: [email-masked]<mailto:[email-masked]>
Subject: RE: BFD stability follow-up from IETF-91
Hello Greg/ Santosh,
-----Original Message-----
From: Santosh P K [mailto:[email-masked]]
Sent: Wednesday, December 03, 2014 9:54 PM
To: Gregory Mirsky; Marc Binderberger; Manav Bhatia
Cc: [email-masked]<mailto:[email-masked]>
Subject: RE: BFD stability follow-up from IETF-91
Greg,

---

From: Manav Bhatia <[email-masked]>
Date: Thu, 4 Dec 2014 20:23:21 +0530

Nobo - Locally storing TX/RX timestamps is not interoperable.

Cheers, Manav

---

From: "Nobo Akiya (nobo)" <[email-masked]>
Date: Thu, 4 Dec 2014 15:14:50 +0000

[no hat on, btw]

Hi Manav,

If what you say is the only requirement not met, one approach may be to pursue a non-standard-track document describing some suggested implementation techniques to locally store TX/RX timestamp.

Given that echo approach will be less accurate and given that we seem to be having difficulty converging, I thought I’ll throw out another idea.

Thanks!

-Nobo

From: Manav Bhatia [mailto:[email-masked]]
Sent: Thursday, December 04, 2014 9:53 AM
To: Nobo Akiya (nobo)
Cc: Santosh P K; MALLIK MUDIGONDA (mmudigon); Vengada Prasad Govindan (venggovi); Gregory Mirsky; Marc Binderberger; [email-masked]
Subject: Re: BFD stability follow-up from IETF-91

Cheers, Manav

On Thu, Dec 4, 2014 at 7:30 PM, Nobo Akiya (nobo) <[email-masked]<mailto:[email-masked]>> wrote:
A quick question to understand where we are.

If we had:

Thanks!

-Nobo

---

From: Jeffrey Haas <[email-masked]>
Date: Thu, 4 Dec 2014 10:17:08 -0500

On Thu, Dec 04, 2014 at 03:14:50PM +0000, Nobo Akiya (nobo) wrote:
> Given that echo approach will be less accurate and given that we seem to be having difficulty converging, I thought I???ll throw out another idea.

I think my biggest concern is that the echo approach has bidirectional
packet loss possibilities.  Async at least lets the receiver know about
unidirectional packet loss.

Of course, if your goal is to notify the sender that their packets are being
lost, you need a backchannel anyway.  I just don't know if we want that back
channel to be bfd.

- Jeff

---

From: Manav Bhatia <[email-masked]>
Date: Thu, 4 Dec 2014 20:52:17 +0530

Hi Nobo,

The echo proposal doesnt make any sense. I thought the one with carrying
some info as part of the the UDP or the IP payload made some.

Having spent countless number of hours debugging a "BFD flap" i would
definitely like to have a std-track mechanism that aids me in debugging the
root cause.

Cheers, Manav

On Thu, Dec 4, 2014 at 8:44 PM, Nobo Akiya (nobo) <[email-masked]> wrote:

>  [no hat on, btw]
>
>
>
> Hi Manav,
>
>
>
> If what you say is the only requirement not met, one approach may be to
> pursue a non-standard-track document describing some suggested
> implementation techniques to locally store TX/RX timestamp.
>
>
>
> Given that echo approach will be less accurate and given that we seem to
> be having difficulty converging, I thought I’ll throw out another idea.
>
>
>
> Thanks!
>
>
>
> -Nobo
>
>
>
> *From:* Manav Bhatia [mailto:[email-masked]]
> *Sent:* Thursday, December 04, 2014 9:53 AM
> *To:* Nobo Akiya (nobo)
> *Cc:* Santosh P K; MALLIK MUDIGONDA (mmudigon); Vengada Prasad Govindan
> (venggovi); Gregory Mirsky; Marc Binderberger; [email-masked]
> *Subject:* Re: BFD stability follow-up from IETF-91
>
>
>

---

From: Gregory Mirsky <[email-masked]>
Date: Thu, 4 Dec 2014 15:28:05 +0000

Hi Prasad,
true, though I'd say that BFD Echo is not defined in these configurations, scenarios.

	Regards,
		Greg

-----Original Message-----
From: Vengada Prasad Govindan (venggovi) [mailto:[email-masked]] 
Sent: Thursday, December 04, 2014 3:15 AM
To: Gregory Mirsky; Santosh P K; Marc Binderberger; Manav Bhatia
Cc: [email-masked]
Subject: RE: BFD stability follow-up from IETF-91

Hello Greg/ Santosh,

-----Original Message-----
From: Rtg-bfd [mailto:[email-masked]] On Behalf Of Gregory Mirsky
Sent: Thursday, December 04, 2014 11:44 AM
To: Santosh P K; Marc Binderberger; Manav Bhatia
Cc: [email-masked]
Subject: RE: BFD stability follow-up from IETF-91

Hi Santosh,

	Regards,
		Greg

-----Original Message-----
From: Santosh P K [mailto:[email-masked]]
Sent: Wednesday, December 03, 2014 9:54 PM
To: Gregory Mirsky; Marc Binderberger; Manav Bhatia
Cc: [email-masked]
Subject: RE: BFD stability follow-up from IETF-91

Greg,

Thanks
Santosh P K

---

From: Gregory Mirsky <[email-masked]>
Date: Thu, 4 Dec 2014 15:47:27 +0000

Hi Jeff,
I can reference RFC 5357 here. The Appendix describes what is called TWAMP-Light mode with Stateless Reflector. About year and a half the Errata been accepted that describes Stateful Reflector, which supports measurement of one-way latency/jitter and packet loss metrics.

	Regards,
		Greg

-----Original Message-----
From: Rtg-bfd [mailto:[email-masked]] On Behalf Of Jeffrey Haas
Sent: Thursday, December 04, 2014 7:17 AM
To: Nobo Akiya (nobo)
Cc: [email-masked]
Subject: Re: BFD stability follow-up from IETF-91

> Given that echo approach will be less accurate and given that we seem to be having difficulty converging, I thought I???ll throw out another idea.

I think my biggest concern is that the echo approach has bidirectional packet loss possibilities.  Async at least lets the receiver know about unidirectional packet loss.

Of course, if your goal is to notify the sender that their packets are being lost, you need a backchannel anyway.  I just don't know if we want that back channel to be bfd.

- Jeff

---

From: "Sam K. Aldrin" <[email-masked]>
Date: Thu, 4 Dec 2014 10:11:42 -0800

Way too many emails to catch up to discussion and to understand clearly, every point.

Here are my questions and concerns

- Is this primarily to debug the debugger i.e. OAM for BFD?
- Is this because the primary BFD protocol is too simple and cannot debug its failure. And this enhancement adds some sort of extension to base protocol? Say BFD 1.1?
- Every OAM tool has issues to debug itself, but is there any reference where there is OAM for OAM which got standardized?

Concern:
- With the sequence number added, one could find(?) that BFD control packets are experiencing congestion or packet delay. So, the assertion is that BFD flapping could be avoided. But if the packets in the network, including BFD, are really experiencing this, shouldn't this be the right behavior?
- I see concerns regarding timestamps and sequence numbers expressed in emails. In that case, the proposed model is still not going to identify the problem completely. Am I reading it right?

-sam

---

From: Manav Bhatia <[email-masked]>
Date: Fri, 5 Dec 2014 06:35:11 +0530

Hi Sam,

>
>
> Concern:
> - With the sequence number added, one could find(?) that BFD control
> packets are experiencing congestion or packet delay. So, the assertion is
> that BFD flapping could be avoided. But if the packets in the network,
> including BFD, are really experiencing this, shouldn't this be the right
> behavior?
>

I was looking at it to help debug scenarios where its not just the network
but implementation issues as well that cause sporadic BFD flaps. If you
have a BFD pkt lying in your buffer for too long before its gets scheduled
out, then perhaps there is a case for you to look at your pkt TX scheduler.
Similarly, you could look at issues on the RX side -- perhaps the queue
isnt being drained fast enough, or the CIR/PIR values are aggressive and
you drop certain packets when you have scaled up BFD sessions.

Ideally even if there is some bit of congestion i would like the BFD packet
to get through.

Cheers, Manav

- I see concerns regarding timestamps and sequence numbers expressed in
> emails. In that case, the proposed model is still not going to identify the
> problem completely. Am I reading it right?
>
> -sam
> > I can reference RFC 5357 here. The Appendix describes what is called
> TWAMP-Light mode with Stateless Reflector. About year and a half the Errata
> been accepted that describes Stateful Reflector, which supports measurement
> of one-way latency/jitter and packet loss metrics.
> >
> >       Regards,
> >               Greg
> >
> >> Given that echo approach will be less accurate and given that we seem
> to be having difficulty converging, I thought I???ll throw out another idea.
> >
> > Of course, if your goal is to notify the sender that their packets are
> being lost, you need a backchannel anyway.  I just don't know if we want
> that back channel to be bfd.
> >
> > - Jeff
> >
>
>

---

From: "Sam K. Aldrin" <[email-masked]>
Date: Thu, 4 Dec 2014 17:40:56 -0800

Hi Manav,

On Dec 4, 2014, at 5:05 PM, Manav Bhatia wrote:

> I was looking at it to help debug scenarios where its not just the network but implementation issues as well that cause sporadic BFD flaps. If you have a BFD pkt lying in your buffer for too long before its gets scheduled out, then perhaps there is a case for you to look at your pkt TX scheduler. Similarly, you could look at issues on the RX side -- perhaps the queue isnt being drained fast enough, or the CIR/PIR values are aggressive and you drop certain packets when you have scaled up BFD sessions.
> 
> Ideally even if there is some bit of congestion i would like the BFD packet to get through.
I understand the queuing problems but I am not clear how the ID is going to solve the problem, if there is only congestion and not packet drop. In that case just the sequence # doesn't help and timestamping is needed. Even if timestamping is to be done, realistically it cannot happen the same way across multi vendor i.e. where the timestamping should/could be done. For ex: Before it is queued or after OR LC/RP/SP/Process etc.

Secondly, if the congestion happens, the CIR/PIR should apply to data packets too. In that case BFD flap is at least a good indicator of the problem, isn't it? 

Lastly, these improvements is change from the existing BFD model/protocol. 
I do not see why it shouldn't be part of BFD v2 OR lead to v2 :D

-sam

---

From: Manav Bhatia <[email-masked]>
Date: Fri, 5 Dec 2014 08:21:13 +0530

Hi Sam,

> I understand the queuing problems but I am not clear how the ID is going
> to solve the problem, if there is only congestion and not
>

It would help.

Assume the last sequence number you saw before the flap was s1. You timed
out since you did not see s2 and s3 before the timeout. Now further assume
that you know that you did receive s2 and s3, however they arrived after
the BFD expiry interval then you know that there wasnt a drop, and the
packet arrived late because of some queing issue. Now how you determine
whether this delay was seen at the TX or RX side is open to discussion.

Without a sequence number you have no idea whether the packet was dropped
or whether it arrived/processed late.

If by some out-of-band mechanism you can figure out that the TX was done on
time and the delay was at RX end then its an implementation issue on the RX
side. If the TX was delayed then its an implementation issue at the TX side.

This helps isolating the node that needs to fix the issue, otherwise we're
only shooting in the dark.

packet drop. In that case just the sequence # doesn't help and timestamping
> is needed. Even if timestamping is to be done, realistically it cannot
> happen the same way across multi vendor i.e. where the timestamping
> should/could be done. For ex: Before it is queued or after OR
> LC/RP/SP/Process etc.
>

This isnt a new problem. Each vendor time stamps 1588 packets differently.
However, the aggregate solution works across multiple vendors.

While it may not solve all the issues because of the vagaries of how each
vendor does time stamping it would certainly help debugging large number of
BFD flaps.

> Secondly, if the congestion happens, the CIR/PIR should apply to data
> packets too. In that case BFD flap is at least a good indicator of the
> problem, isn't it?
>

There is usually a separate CIR/PIR for different CPU bound packets. So
based on some parameters you might impose a different CIR/PIR for BFD than
say, ssh and radius packets. If BFD flaps and you know you missed a few
sequence numbers and you see drops in that queue then all of this could be
co-related and you could fix the CPU queue parameters for BFD. I understand
that this is a very implementation specific issue, but then thats what i
had said earlier -- such a mechanism can help in isolating implementation
specific issues as well.

> I do not see why it shouldn't be part of BFD v2 OR lead to v2 :D

Sure, that would make Marc very happy ! :-)

I am not sure if we have enough momentum right now that can propel us
towards BFD v2.

OTOH, if the WG believes that this is an opportune time for us to start
looking at BFDv2, then i would be more than willing to participate!

Cheers, Manav

>> >> If what you say is the only requirement not met, one approach may be
>> to pursue a non-standard-track document describing some suggested

---

From: "Sam K. Aldrin" <[email-masked]>
Date: Thu, 4 Dec 2014 19:52:19 -0800

Hi Manav,

On Dec 4, 2014, at 6:51 PM, Manav Bhatia wrote:

> I understand the queuing problems but I am not clear how the ID is going to solve the problem, if there is only congestion and not
> 
> It would help.
> 
> Assume the last sequence number you saw before the flap was s1. You timed out since you did not see s2 and s3 before the timeout. Now further assume that you know that you did receive s2 and s3, however they arrived after the BFD expiry interval then you know that there wasnt a drop, and the packet arrived late because of some queing issue. Now how you determine whether this delay was seen at the TX or RX side is open to discussion.

As the draft doesn't say, what exactly one would/should do, assuming there is packet throttling happing due to various reasons, could you/authors elaborate on this? I would like to see those tangible things detailed first.

I see the problem little differently though. BFD session flapping is an indicator of the network behavior, be it device or network congestion or something else. Even if the packets arrive late, as you say, one cannot know where exactly the delay is happening.

> 
> Without a sequence number you have no idea whether the packet was dropped or whether it arrived/processed late. 
> 
> If by some out-of-band mechanism you can figure out that the TX was done on time and the delay was at RX end then its an implementation issue on the RX side. If the TX was delayed then its an implementation issue at the TX side.
Don't think so. It could be the lot more than implementation issue. Could be due to network congestion due to bursty traffic and nothing to do with the implementation.
> 
> This helps isolating the node that needs to fix the issue, otherwise we're only shooting in the dark. 
The issue you see is what I think could be the actual network behavior and not the device issue. 
Debugging Packet drops, latency and mis-ordering of packets is mostly shooting in the dark :D
Nevertheless, I do not see this as BFD specific only. 

> 
> packet drop. In that case just the sequence # doesn't help and timestamping is needed. Even if timestamping is to be done, realistically it cannot happen the same way across multi vendor i.e. where the timestamping should/could be done. For ex: Before it is queued or after OR LC/RP/SP/Process etc.
> 
> This isnt a new problem. Each vendor time stamps 1588 packets differently. However, the aggregate solution works across multiple vendors.
> 
> While it may not solve all the issues because of the vagaries of how each vendor does time stamping it would certainly help debugging large number of BFD flaps.
As timestamp is not in the ID, we could differ the discussion for later, but I definitely believe that it is a bigger issue where TS is taken, if  granularity and accuracy are important.
> There is usually a separate CIR/PIR for different CPU bound packets. So based on some parameters you might impose a different CIR/PIR for BFD than say, ssh and radius packets. If BFD flaps and you know you missed a few sequence numbers and you see drops in that queue then all of this could be co-related and you could fix the CPU queue parameters for BFD. I understand that this is a very implementation specific issue, but then thats what i had said earlier -- such a mechanism can help in isolating implementation specific issues as well.
I agree that it will be helpful. But then, when a new mechanism is introduced, it should clearly spell out the mechanisms on interpreting the problems and how to deal with it. The ID has none of it, as of now.
> I am not sure if we have enough momentum right now that can propel us towards BFD v2.
> 
> OTOH, if the WG believes that this is an opportune time for us to start looking at BFDv2, then i would be more than willing to participate!
Well, if there is real issue that existing BFD falls short of the needs, then interest automatically increases. As this ID introduces new things to existing version, it should be pursued as part of next version, rather than changing the existing model for the same version.

-sam

---

From: Gregory Mirsky <[email-masked]>
Date: Fri, 5 Dec 2014 05:06:33 +0000

Hi Manav, et. al,
you’ve said:
Ideally even if there is some bit of congestion i would like the BFD packet to get through.
Assume that the congestion is in egress queue, related to the fast forwarding path. Then wouldn’t pushing BFD control packets ahead of every other packet of the same DSCP be hiding the problem in the network, rather than detecting it, doing exactly what BFD intended to do? Perhaps that, tuning DSCP marking of BFD, is something to look into before changing BFD?

                Regards,
                                Greg

From: Manav Bhatia [mailto:[email-masked]]
Sent: Thursday, December 04, 2014 5:05 PM
To: Sam K. Aldrin
Cc: Gregory Mirsky; [email-masked]
Subject: Re: BFD stability follow-up from IETF-91

Hi Sam,

Concern:

Cheers, Manav

-sam
On Dec 4, 2014, at 7:47 AM, Gregory Mirsky wrote:

> From: Rtg-bfd [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Jeffrey Haas

---

From: Gregory Mirsky <[email-masked]>
Date: Fri, 5 Dec 2014 05:10:17 +0000

Hi Sam, et. al,
if we're starting to compile list of requirements for what may become BFDv2 - count me in.

                Regards,
                                Greg

From: Sam K. Aldrin [mailto:[email-masked]]
Sent: Thursday, December 04, 2014 7:52 PM
To: Manav Bhatia
Cc: Gregory Mirsky; [email-masked]
Subject: Re: BFD stability follow-up from IETF-91

Hi Manav,

Hi Sam,

It would help.

-sam

Cheers, Manav

-sam

Cheers, Manav

-sam

---

From: Mahesh Jethanandani <[email-masked]>
Date: Thu, 04 Dec 2014 18:39:45 +0000

> What are the core requirements which have not been satisfied?

Nobo, if we had ways to collect loss (inline) and delay (offline or inline)
information, it will go a long way to address the core requirements that we
want to address with this draft.

One thing that probably did not come out clearly from the draft was that
adding time stamps in the BFD payload was optional. As I suggested in my
other e-mail we are open to other ways of collecting time stamp
information. Sending it over in the payload is one of the options and
allows the receiver to compute, determine and react to delays in a more
timely fashion.

> *From:* Rtg-bfd [mailto:[email-masked]] *On Behalf Of *Santosh
> P K
> *Sent:* Thursday, December 04, 2014 6:52 AM
> *To:* MALLIK MUDIGONDA (mmudigon); Vengada Prasad Govindan (venggovi);
> Gregory Mirsky; Marc Binderberger; Manav Bhatia
> *Cc:* [email-masked]
> *Subject:* RE: BFD stability follow-up from IETF-91
>
>
>
> Mallaik,
>
>   That’s correct and that is why I asked if we really need to define any
> packet format as Greg suggested?
>
>
>
> Thanks
>
> Santosh P K
>
>
>
> *From:* MALLIK MUDIGONDA (mmudigon) [mailto:[email-masked]
> <[email-masked]>]
> *Sent:* Thursday, December 04, 2014 5:19 PM
> *To:* Santosh P K; Vengada Prasad Govindan (venggovi); Gregory Mirsky;
> Marc Binderberger; Manav Bhatia
> *Cc:* [email-masked]
> With Echo, the receive side is not going to look into the packet. It is
> just going to send it back and it is the responsibility of the sender to
> interpret its contents.
>
>
>
> Regards
>
> Mallik
>
>
>
> *From: *Santosh P K <[email-masked]>
> *Date: *Thursday, 4 December 2014 5:15 pm
> *To: *"Vengada Prasad Govindan (venggovi)" <[email-masked]>, Gregory
> Mirsky <[email-masked]>, Marc Binderberger <[email-masked]>,
> Manav Bhatia <[email-masked]>
> *Cc: *"[email-masked]" <[email-masked]>
> *Subject: *RE: BFD stability follow-up from IETF-91
>
>
>
> Hi Prasad,
>
>   That's right. We echo has limitations but for single hop we can still
> use it. I was trying to understand even for single hop do we need to define
> any packet format for echo?
>
>
>
> Thanks
>
> Santosh P K
>
>
>
> From: Vengada Prasad Govindan (venggovi) [mailto:[email-masked]
> <[email-masked]>]
>
> From: Rtg-bfd [mailto:[email-masked] <[email-masked]>]
> On Behalf Of Gregory
>
> Mirsky
>
> (addressee
> From: Santosh P K [mailto:[email-masked] <[email-masked]>]
>
> good
> > From: Gregory Mirsky [mailto:[email-masked]
> <[email-masked]>]
>
> > Echo mode. I think that it is more suitable for experimentation and
>
> unorthodox use.
>
> >
>
> > Regards,
>
> > Greg
>
> >
>
> > From: Rtg-bfd [mailto:[email-masked]
> <[email-masked]>] On Behalf Of Santosh P
>
> > K
>
> > fewer extensions but might become problem when there are too many
>
> extensions.
>
> >
>
> >
>
> > >
>
> > > between a "normal" authentication header and a "sequence +
>
> > authentication".
>
> > >
>
> >
>
> > > >>>> From: Rtg-bfd [mailto:[email-masked]
> <[email-masked]>] On Behalf Of
>
> > > >>>> Mach
>
> > > Chen
>
> > > >>>>> From: Marc Binderberger [mailto:[email-masked] <[email-masked]>]
>
> > > >>>>> I remember some discussion on NVO3 about how many bits it
>
> > > >>>>> takes
>
> > > >>>>> ;-
>
> > > ) -
>
> > > >>>>> could you send the links/draft names you are working on to this
>
> list?
>
> > > >>> need the marking based solution. Suppose that someone want to
>
> > > monitor
>
> > > >>>>>>> From: Rtg-bfd [mailto:[email-masked]
> <[email-masked]>] On Behalf Of
>
> > > Marc
>
> > > >>>>>>> Binderberger
>
> > > >>>>>> Indeed, the sequence number helps a lot for the correlation
>
> > > between
>
> > > >>>>>> the Tx and Rx system.
>
> > > >>>>>>
>
> > > >>>>>>>> its own set of issues, and was hence dropped from the original
>
> draft.
>
> > > >>>>>>>>
>
> > > >>>>>>>> How would it know which end was misbehaving? Was it a
>
> delay
>
> > > >>>>>>>> describes how we can use the sequence numbers to debug
>
> > what
>
> > > >>>>>>>> and
>
> > > where
>
> > > >>> things went wrong.
>
> > > >>>>>>>>
>
> > > >>>>>>>> Cheers, Manav
>
> > > >>>>>>>>
>
> > > >>>>>>>>
>
> > > >>>>>>>>> be adopted as a WG item, the sense of the room was very
>
> quiet.
>
> > > >>>>>>>>> on the WG mailing list to make sure the whole group has a
>
> > voice.
>
> > > >>>>>>>>>
>

---

From: Marc Binderberger <[email-masked]>
Date: Sun, 7 Dec 2014 19:36:10 -0800

Hello Manav,

many emails on this thread - hard to follow up :-)

Anyway, several comments:

* if such a debug supports becomes informal, experimental or standard is IMHO 
irrelevant for it being actually implemented. As it looks like we talk about 
aspects that are implementation specific, e.g. when exactly a packet is 
timestamped, by hard- or software and so on, I have my doubt about a standard 
document though.

* Greg's echo idea is of course do-able - when you think timestamping in 
hardware can be done then it can be done in the forwarding path for echos as 
well. Depends on your hardware :-) and on an agreed (minimal) format for 
echo. As mentioned BFD echo is not defined/used for multiple BFD features, 
which limits it's use though.

* the local timestamps cannot create an interop problem as they are _local_ 
and not exchanged :-) A stability draft could define the minimum requirements 
we have for what timestamps should be (locally) collected. For interop all 
that is needed is an agreement on the "ID", i.e. the sequence number, 
encoding in the packet.

Regards, Marc

On Thu, 4 Dec 2014 20:52:17 +0530, Manav Bhatia wrote:
>> Cc: Santosh P K; MALLIK MUDIGONDA (mmudigon); Vengada Prasad Govindan

---

From: Manav Bhatia <[email-masked]>
Date: Mon, 8 Dec 2014 09:33:05 +0530

Hi Marc,

> which limits it's use though.
For the echo mechanism to work, do you agree that you would have to
continuously send Echos so that you can detect the issue?

Or are you suggesting that once BFD flaps we will start sending Echoes
overloaded with debug information to detect the issue?

I'd like to understand this before the mailing list sees a barrage of
emails. Alternatively, we can also take it offline and only report the
summary of our discussion to the list.

Cheers, Manav

---

From: Marc Binderberger <[email-masked]>
Date: Sun, 7 Dec 2014 21:00:38 -0800

Hello Sam, Manav et al.,

really a long thread ... :-)

Reading the emails I had a few time the question: what are we trying to 
achieve here?

There is debugging. Obviously, having more (relevant) data helps to find the 
cause, likely by eliminating the not-causes. I don't expect anything perfect, 
as Sam stated

> Debugging Packet drops, latency and mis-ordering of packets is mostly 

But then there may be another motivation. Manav wrote

> Ideally even if there is some bit of congestion i would like the BFD 

and I wonder what do you mean?  You mean BFD should not flap if it was 
congestion?
The "stability" and some comments in the emails makes me wonder if we talk 
about "do not timeout when ...". And that would be a very different topic. It 
would mean we talk about a different trigger than the timeout defined in 5880.

In this case we would need to talk about the new trigger condition first 
before we discuss what data needs to be exchanged.

>> Sure, that would make Marc very happy ! :-)

Count me in! :-)

Seriously: if the discussion is to improve debugging of v1 then v2 maybe a 
long shot. If we talk about different triggers then a v2 header may make 
sense.

Regards, Marc

On Thu, 4 Dec 2014 19:52:19 -0800, Sam K. Aldrin wrote:
> As the draft doesn't say, what exactly one would/should do, assuming there 
> is packet throttling happing due to various reasons, could you/authors 
> elaborate on this? I would like to see those tangible things detailed first.
> 
> I see the problem little differently though. BFD session flapping is an 
> indicator of the network behavior, be it device or network congestion or 
> something else. Even if the packets arrive late, as you say, one cannot 
> know where exactly the delay is happening.
> 
>> 
>> If by some out-of-band mechanism you can figure out that the TX was done 
>> on time and the delay was at RX end then its an implementation issue on 
>> the RX side. If the TX was delayed then its an implementation issue at the 
>> TX side.
> Don't think so. It could be the lot more than implementation issue. Could 
> be due to network congestion due to bursty traffic and nothing to do with 
> the implementation.
>> 
>>> packet drop. In that case just the sequence # doesn't help and 
>>> timestamping is needed. Even if timestamping is to be done, realistically 
>>> it cannot happen the same way across multi vendor i.e. where the 
>>> timestamping should/could be done. For ex: Before it is queued or after 
>>> OR LC/RP/SP/Process etc.
>> 
>> vendor does time stamping it would certainly help debugging large number 
>> of BFD flaps.
> As timestamp is not in the ID, we could differ the discussion for later, 
> but I definitely believe that it is a bigger issue where TS is taken, if  
> granularity and accuracy are important.
>> 
>>> 
>> co-related and you could fix the CPU queue parameters for BFD. I 
>> understand that this is a very implementation specific issue, but then 
>> thats what i had said earlier -- such a mechanism can help in isolating 
>> implementation specific issues as well.
> I agree that it will be helpful. But then, when a new mechanism is 
> introduced, it should clearly spell out the mechanisms on interpreting the 
> problems and how to deal with it. The ID has none of it, as of now.
>> 
>>> 
> Well, if there is real issue that existing BFD falls short of the needs, 
> then interest automatically increases. As this ID introduces new things to 
> existing version, it should be pursued as part of next version, rather than 
> changing the existing model for the same version.
> 
> -sam
> 
>> 
>> Cheers, Manav 
>>> 
>>> -sam
>>>> 
>>>> Cheers, Manav
>>>> 
>>>>> emails. In that case, the proposed model is still not going to identify 
>>>>> the problem completely. Am I reading it right?
>>>>> 
>>>>> -sam
>>>>> TWAMP-Light mode with Stateless Reflector. About year and a half the 
>>>>> Errata been accepted that describes Stateful Reflector, which supports 
>>>>> measurement of one-way latency/jitter and packet loss metrics.
>>>>>>
>>>>>>       Regards,
>>>>>>               Greg
>>>>>>
>>>>>>> Given that echo approach will be less accurate and given that we 
>>>>> seem to be having difficulty converging, I thought I???ll throw out 
>>>>> another idea.
>>>>>>
>>>>>> I think my biggest concern is that the echo approach has 
>>>>> bidirectional packet loss possibilities.  Async at least lets the 
>>>>> receiver know about unidirectional packet loss.
>>>>>>
>>>>>> Of course, if your goal is to notify the sender that their packets 
>>>>> are being lost, you need a backchannel anyway.  I just don't know if we 
>>>>> want that back channel to be bfd.
>>>>>>
>>>>>> - Jeff
>>>>>>
>>>>> 
>>>> 
>>> 
>> 
>

---

From: Marc Binderberger <[email-masked]>
Date: Sun, 7 Dec 2014 21:21:02 -0800

Hello Manav,

> continuously send Echos so that you can detect the issue?

that's what I had in mind, yes

> overloaded with debug information to detect the issue?

interesting idea - that would be an alternative use, collecting forensic 
data. Maybe we should support that too!

My biggest problem with the echo idea is so far BFD-over-LAG. But maybe it is 
not a real problem, any echo stamping/updating in the forwarding path would 
require an hardware update (or reprogramming, if your hardware allows) and in 
this case one could boldly state that the echo packet must leave via the 
ingress port :-)

Regards, Marc

---

From: Gregory Mirsky <[email-masked]>
Date: Mon, 8 Dec 2014 05:41:27 +0000

Hi Marc,
regarding your concern on ensuring that BFD Echo reply traverses the same constituent link as crossed by corresponding BFD Echo request , if it is indeed the requirement, I'll refer to revision of IEEE-802.1AX-REV in its part related to conversation-sensitive frame collection and distribution. Alternatively, we may realize the RFC 7130 needs an update that introduces or explains behavior of micro-BFD Echo over LAG.

	Regards,
		Greg

-----Original Message-----
From: Rtg-bfd [mailto:[email-masked]] On Behalf Of Marc Binderberger
Sent: Monday, December 08, 2014 1:21 PM
To: Manav Bhatia
Cc: [email-masked]
Subject: Re: BFD stability follow-up from IETF-91

Hello Manav,

> overloaded with debug information to detect the issue?

interesting idea - that would be an alternative use, collecting forensic data. Maybe we should support that too!

Regards, Marc

---

From: Manav Bhatia <[email-masked]>
Date: Mon, 8 Dec 2014 11:12:47 +0530

> that's what I had in mind, yes

.. and i dont like this. Are you saying that for each BFD session now you
will run a parallel BFD echo session that would carry the debug data? Your
scaling just went for a toss !!

> data. Maybe we should support that too!

This would help if the flap is deterministic-ally reproducible. If its not,
then how long do you run the Echo BFD? And what happens to the original BFD
session? You run the two parallel-ly?

Cheers, Manav

> >> hardware can be done then it can be done in the forwarding path for
> echos
> >> as

---

From: Gregory Mirsky <[email-masked]>
Date: Mon, 8 Dec 2014 05:46:07 +0000

Hi Marc,
missed to add note on my intention of use of BFD Echo.
I think of it as on-demand mechanism only, not proactive. Thus it would most likely be invoked to investigate BFD sessions that already failed or unstable.

	Regards,
		Greg

-----Original Message-----
From: Rtg-bfd [mailto:[email-masked]] On Behalf Of Marc Binderberger
Sent: Monday, December 08, 2014 1:21 PM
To: Manav Bhatia
Cc: [email-masked]
Subject: Re: BFD stability follow-up from IETF-91

Hello Manav,

Regards, Marc

---

From: Santosh P K <[email-masked]>
Date: Mon, 8 Dec 2014 06:07:18 +0000

Hello Mac and Manav,
     Are we just talking about singlehop? How about MPLS BFD and multihop where echo does not work?

Thanks
Santosh P K

From: Rtg-bfd [mailto:[email-masked]] On Behalf Of Manav Bhatia
Sent: Monday, December 08, 2014 9:33 AM
To: Marc Binderberger
Cc: [email-masked]
Subject: Re: BFD stability follow-up from IETF-91

Hi Marc,

For the echo mechanism to work, do you agree that you would have to continuously send Echos so that you can detect the issue?

Or are you suggesting that once BFD flaps we will start sending Echoes overloaded with debug information to detect the issue?

I'd like to understand this before the mailing list sees a barrage of emails. Alternatively, we can also take it offline and only report the summary of our discussion to the list.

Cheers, Manav

---

From: Manav Bhatia <[email-masked]>
Date: Mon, 8 Dec 2014 11:43:08 +0530

Cool. I had multihop in my mind.

Cheers, Manav

On Mon, Dec 8, 2014 at 11:37 AM, Santosh P K <[email-masked]> wrote:

> *From:* Rtg-bfd [mailto:[email-masked]] *On Behalf Of *Manav
> Bhatia
> *Sent:* Monday, December 08, 2014 9:33 AM
> *To:* Marc Binderberger
> *Cc:* [email-masked]

---

From: Gregory Mirsky <[email-masked]>
Date: Mon, 8 Dec 2014 06:22:36 +0000

Hi Santosh,
do you envision scenario when BFD multi-hop and single-hop sessions that share the same sender behave differently? In my opinion, if multi-hop session is unstable and single-hop is stable, then the problem is likely in the network and thus is real, rather than in the BFD sender and more implementation specific. Thus running BFD Echo even without cooperation of the next hop node (no timestamps there) would give sender information about latencies BFD is subjected to in this node.

                Regards,
                                Greg

From: Rtg-bfd [mailto:[email-masked]] On Behalf Of Santosh P K
Sent: Monday, December 08, 2014 2:07 PM
To: Manav Bhatia; Marc Binderberger
Cc: [email-masked]
Subject: RE: BFD stability follow-up from IETF-91

Hello Mac and Manav,

Thanks
Santosh P K

Hi Marc,

Cheers, Manav

---

From: Manav Bhatia <[email-masked]>
Date: Mon, 8 Dec 2014 11:56:44 +0530

Hi Greg,

What if i dont have a single-hop session? Do you want the operator to set
up one between the sender and its immediate next-hop and another between
the receiver and its immediate next-hop?

Cheers, Manav

On Mon, Dec 8, 2014 at 11:52 AM, Gregory Mirsky <[email-masked]

>  Hi Santosh,
>
> do you envision scenario when BFD multi-hop and single-hop sessions that
> share the same sender behave differently? In my opinion, if multi-hop
> session is unstable and single-hop is stable, then the problem is likely in
> the network and thus is real, rather than in the BFD sender and more
> implementation specific. Thus running BFD Echo even without cooperation of
> the next hop node (no timestamps there) would give sender information about
> latencies BFD is subjected to in this node.
>
>
>
>                 Regards,
>
>                                 Greg
>
>
>
> *Sent:* Monday, December 08, 2014 2:07 PM
> *To:* Manav Bhatia; Marc Binderberger
> *Cc:* [email-masked]
> *From:* Rtg-bfd [mailto:[email-masked]
> <[email-masked]>] *On Behalf Of *Manav Bhatia

---

From: "MALLIK MUDIGONDA (mmudigon)" <[email-masked]>
Date: Mon, 8 Dec 2014 06:27:59 +0000

Hi Greg,

Are we assuming that there is a single node in-between the sender and receiver for the multi hop case? The actual problem of packet loss may be happening after the next hop node. Since we don’t have a way to run echo for multi hop sessions, how do we identify packet losses in such a case if we use echo?

Thanks

Regards
Mallik

From: Gregory Mirsky <[email-masked]<mailto:[email-masked]>>
Date: Monday, 8 December 2014 11:52 am
To: Santosh P K <[email-masked]<mailto:[email-masked]>>, Manav Bhatia <[email-masked]<mailto:[email-masked]>>, Marc Binderberger <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: RE: BFD stability follow-up from IETF-91

Hi Santosh,

                Regards,
                                Greg

Hello Mac and Manav,

Thanks
Santosh P K

Hi Marc,

Cheers, Manav

---

From: Gregory Mirsky <[email-masked]>
Date: Mon, 8 Dec 2014 06:34:09 +0000

Hi Mallik,
the packet loss in the network, in my opinion, is the demonstration of the real network problem that BFD is intended to detect. The BFD Echo may, I assume, help to rule out potential issues with the particular BFD implementation. Troubleshooting the real network problem, e.g. packet loss, is not in scope of BFD, in my opinion, and can be done with help of other OAM instruments.

                Regards,
                                Greg

From: MALLIK MUDIGONDA (mmudigon) [mailto:[email-masked]]
Sent: Monday, December 08, 2014 2:28 PM
To: Gregory Mirsky; Santosh P K; Manav Bhatia; Marc Binderberger
Cc: [email-masked]
Subject: Re: BFD stability follow-up from IETF-91

Hi Greg,

Are we assuming that there is a single node in-between the sender and receiver for the multi hop case? The actual problem of packet loss may be happening after the next hop node. Since we don't have a way to run echo for multi hop sessions, how do we identify packet losses in such a case if we use echo?

Thanks

Regards
Mallik

Hi Santosh,

                Regards,
                                Greg

Hello Mac and Manav,

Thanks
Santosh P K

Hi Marc,

Cheers, Manav

---

From: Gregory Mirsky <[email-masked]>
Date: Mon, 8 Dec 2014 06:35:35 +0000

Hi Manav,
I’d propose to take this discussion off-line. Anyone who’s interested in developing solution based on BFD Echo please feel free to contact me directly.

                Regards,
                                Greg

From: Manav Bhatia [mailto:[email-masked]]
Sent: Monday, December 08, 2014 2:27 PM
To: Gregory Mirsky
Cc: Santosh P K; Marc Binderberger; [email-masked]
Subject: Re: BFD stability follow-up from IETF-91

Hi Greg,

What if i dont have a single-hop session? Do you want the operator to set up one between the sender and its immediate next-hop and another between the receiver and its immediate next-hop?

Cheers, Manav

On Mon, Dec 8, 2014 at 11:52 AM, Gregory Mirsky <[email-masked]<mailto:[email-masked]>> wrote:
Hi Santosh,
do you envision scenario when BFD multi-hop and single-hop sessions that share the same sender behave differently? In my opinion, if multi-hop session is unstable and single-hop is stable, then the problem is likely in the network and thus is real, rather than in the BFD sender and more implementation specific. Thus running BFD Echo even without cooperation of the next hop node (no timestamps there) would give sender information about latencies BFD is subjected to in this node.

                Regards,
                                Greg

From: Rtg-bfd [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Santosh P K

Hello Mac and Manav,

Thanks
Santosh P K

Hi Marc,

Cheers, Manav

---

From: Sam Aldrin <[email-masked]>
Date: Sun, 7 Dec 2014 22:38:39 -0800

Hi Marc,

Thanks for the response.
Inline for my comments.
> On Dec 7, 2014, at 9:00 PM, Marc Binderberger <[email-masked]> wrote:
> 
> Hello Sam, Manav et al.,
> 
I have asked the same and haven’t received the response yet. The ID doesn’t provide clear or convincing answer.
But the email flood is discussing the solution already.

> before we discuss what data needs to be exchanged.

I am not sure if the email thread has reached point of no return, but, it is not effective when the problem to solve is not discussed enough, but the solution is.
> long shot. If we talk about different triggers then a v2 header may make 
I would add another caveat or put it differently, if the goal of debugging the debugger (v1) with  changes to protocol to v1, then take it as v2.

May be OAM for OAM WG is an idea </jk> :D

-sam

---

From: "MALLIK MUDIGONDA (mmudigon)" <[email-masked]>
Date: Mon, 8 Dec 2014 06:39:29 +0000

Hi Greg,

I understand your point. I am not suggesting that we use a particular mechanism for this purpose.  The original issue that was raised to support the sequence numbers in BFD packets was that there may be loss of packets but not large enough to cause BFD session flap. So though the session stays up, there may be some packets lost. Adding sequence numbers to BFD is one of the suggestion on this thread. You suggested using echo for this purpose. My question was echo is not supported over multi hop sessions and so is there a generic way to take this forward?

Thanks

Regards
Mallik

From: Gregory Mirsky <[email-masked]<mailto:[email-masked]>>
Date: Monday, 8 December 2014 12:04 pm
To: Cisco Employee <[email-masked]<mailto:[email-masked]>>, Santosh P K <[email-masked]<mailto:[email-masked]>>, Manav Bhatia <[email-masked]<mailto:[email-masked]>>, Marc Binderberger <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: RE: BFD stability follow-up from IETF-91

Hi Mallik,

                Regards,
                                Greg

Hi Greg,

Thanks

Regards
Mallik

Hi Santosh,

                Regards,
                                Greg

Hello Mac and Manav,

Thanks
Santosh P K

Hi Marc,

Cheers, Manav

---

From: Gregory Mirsky <[email-masked]>
Date: Mon, 8 Dec 2014 06:47:15 +0000

Hi Mallik,
I think I was not clear in explaining the possible use of BFD Echo. It is not to measure packet loss in the network. That, I'm certain, is real network problem that can e properly quantified only with use of performance measurement methods, active or passive.
The subject of the proposal being discussed, as I understand, detecting superficial outages of BFD caused by scaling and/or implementation issues, not by the network environment. I agree with Mark that ensuring consistent and thus informative timestamping is the challenge by itself and better be left internal. But if there's some interest in developing something that may be used for this purpose and, at the same time, would not affect the most used BFD Async mode, then BFD Echo may be, may be the right vechicle.

                Regards,
                                Greg

From: MALLIK MUDIGONDA (mmudigon) [mailto:[email-masked]]
Sent: Monday, December 08, 2014 2:39 PM
To: Gregory Mirsky; Santosh P K; Manav Bhatia; Marc Binderberger
Cc: [email-masked]
Subject: Re: BFD stability follow-up from IETF-91

Hi Greg,

Thanks

Regards
Mallik

Hi Mallik,

                Regards,
                                Greg

Hi Greg,

Thanks

Regards
Mallik

Hi Santosh,

                Regards,
                                Greg

Hello Mac and Manav,

Thanks
Santosh P K

Hi Marc,

Cheers, Manav

---

From: Mahesh Jethanandani <[email-masked]>
Date: Sun, 7 Dec 2014 23:33:16 -0800

Let me try to see if I can bring the focus back on at least what the draft is trying to achieve. And I will do this by way of a customer scenario. 

Customer configures MPLS-TP with working and protect tunnels. Customer then configures BFD on both the working and protect tunnels. BFD flaps and triggers a failover to the protect tunnel. After the fact the customer wants to know why did the failover happen. 

We do not have the option of saying that they needed to change their hardware (to perform echo or hardware based time stamping) or to run other protocols continuously without impacting their network. I am outlining these parameters not because I want the solution to be limited to this situation but because most of the explanations or solutions seemed to have missed the fact that one is dealing with an existing network with existing hardware. So while the solution can extend itself to more configurations/situations it cannot ignore the existing setup.

How do we determine why a BFD session flapped? There are really two explanations to a BFD session flapping. Either the BFD packets were lost or packets were delayed. So the first determination is to do just that.

That essentially is germane to why the draft is written.

Cheers.

p.s. And to tell the customer that we had no way to explain why one of the features used to detect problems in the network cannot explain *why* a particular problem happens also does not help.

Mahesh Jethanandani
[email-masked]

---

From: Marc Binderberger <[email-masked]>
Date: Sun, 7 Dec 2014 23:49:58 -0800

Hello Manav,

> scaling just went for a toss !!

Just by a factor of 2 ;-)

But seriously: if you run echo at hight speed you would usually avoid running 
the control packets at high speed too. Just what RFC5880 describes.

> This would help if the flap is deterministic-ally reproducible. If its not, 

Correct. Good for some problems, not good for others. As I said, interesting 
option but not what I had in mind in first place.

> session? You run the two parallel-ly?

The "original" session?  I'm talking here about an echo session that is 
controlled by the corresponding control packet exchange. As described in 
RFC5880. Echo not defined for 5884 or 7130? Ah well, one probably could.

Echo seems at least an option to me, in the context of debugging, that can be 
discussed. The freedom of defining the packet is interesting, some parameters 
like RTT are measured easily. Other aspects, like Multihop, are a problem, no 
doubt. I was simply surprised how quickly this idea was dismissed ... .

Regards, Marc

---

From: "MALLIK MUDIGONDA (mmudigon)" <[email-masked]>
Date: Mon, 8 Dec 2014 08:27:24 +0000

Hi Marc,

Nothing has been dismissed yet :-). Trying to see what can be done.

Thanks

Regards
Mallik

From: Marc Binderberger <[email-masked]<mailto:[email-masked]>>
Date: Monday, 8 December 2014 1:19 pm
To: Manav Bhatia <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: BFD stability follow-up from IETF-91

Hello Manav,

Regards, Marc

My biggest problem with the echo idea is so far BFD-over-LAG. But maybe it
is
require an hardware update (or reprogramming, if your hardware allows) and

echo. As mentioned BFD echo is not defined/used for multiple BFD
features,

Cheers, Manav

---

From: Manav Bhatia <[email-masked]>
Date: Mon, 8 Dec 2014 18:13:18 +0530

Hi Marc,

> Just by a factor of 2 ;-)

.. and thats quite bad.

> the control packets at high speed too. Just what RFC5880 describes.

You never send control plane packets at a high rate - but now you send and
receive twice as many BFD packets as you were processing earlier. I would
wager that this would be a show stopper.

Cheers, Manav

---

From: Marc Binderberger <[email-masked]>
Date: Mon, 8 Dec 2014 10:18:40 -0800

Hello Manav,

> wager that this would be a show stopper. 

that depends on implementation details how much impact this has. BFD itself 
is still sending/receiving largely the same number of packets.

The echo U-turn processing would happen in the data/forwarding plane and 
should (well, must) be simple, even with some stamping.

Regards, Marc