Subject: RE: BFD WG adoption for draft-haas-bfd-large-packets

From: "Albert Fu (BLOOMBERG/ 120 PARK)" <[email-masked]>
Date: Tue, 23 Oct 2018 15:44:32 -0000

Hi Les,

Given that it takes relative lengthy time to troubleshoot the MTU issue, and the associated impact on customer traffic, it is important to have a reliable and fast mechanism to detect the issue. 

I believe BFD, especially for single hop control-plane independent situation (btw, this covers majority of our BFD use case), is indeed an ideal and reliable solution for this purpose. It is also closely tied with the routing protocols, and enable traffic to be diverted very quickly. 

The choice of BFD timer is also one of the design tradeoffs - low BFD detection timer will cause more network churns. We do not need extremely aggressive BFD timer to achieve fast convergence. For example, with protection, we can achieve end to end sub-second convergence by using relatively high BFD interval of 150ms. 

In the case where the path will be used for a variety of encapsulations (e.g. Pure IP and L3VPN traffic), we would set the BFD padding to cater for the largest possible payload. So, in our case, our link needs to carry a mix of pure IP (1500 max payload) and MPLS traffic (1500 + 3 headers), we would set the padding so that the total padded BFD packet size is 1512 bytes. 

As you rightly pointed out, ISIS routing protocol does support hello padding, but since this is a control plane process, we can not use aggressive timer. The lowest hello interval the can be configured is 1s, so with default multiplier of 3, the best we can achieve is 3s detection time.

What we would like is a simple mechanism to validate that a link can indeed carry the expected max payload size before we put it into production. If an issue occurs where this is no longer the case (e.g. due to outages or re-routing within the Telco circuit), we would like a reliable mechanism to detect this, and also divert traffic around the link quickly. I feel BFD is a good method for this purpose.

Thanks
Albert 

From: [email-masked] At: 10/23/18 10:45:02To:  Albert Fu (BLOOMBERG/ 120 PARK ) ,  [email-masked]
Subject: RE: BFD WG adoption for draft-haas-bfd-large-packets

Albert – 
  
Please understand that I fully agree with the importance of being able to detect/report MTU issues. In my own experience this can be a difficult problem to diagnose.  You do not have to convince me that some improvement in detection/reporting is needed. The question really is whether using BFD is the best option. 
  
Could you respond to my original questions – particularly why sub-second detection of this issue is a requirement? 
  
For your convenience: 
  
<snip> 
It has been stated that there is a need for sub-second detection of this condition – but I really question that requirement.  
What I would expect is that MTU changes only occur as a result of some maintenance operation (configuration change, link addition/bringup,  insertion of a new box in the physical path etc.). The idea of using a mechanism which is specifically tailored for sub-second detection to monitor something that is only going to change occasionally seems inappropriate. It makes me think that other mechanisms  (some form of OAM, enhancements to routing protocols to do what IS-IS already does ) could be more appropriate and would still meet the operational requirements. 
  
I have listened to the Montreal recording – and I know there was discussion related to these issues (not sending padded packets  all the time, use of BFD echo, etc.) – but I would be interested in more discussion of the need for sub-second detection. 
  
Also, given that a path might be used with a variety of encapsulations, how do you see such a mechanism being used when multiple  BFD clients share the same BFD session and their MTU constraints are different? 
<end snip> 
  
Thanx. 
  
   Les

---

From: "Acee Lindem (acee)" <[email-masked]>
Date: Tue, 23 Oct 2018 16:30:52 +0000

Hi Albert, Les,

I tend to agree with Les that BFD doesn’t seem like the right protocol for this. Note that if you use OSPF as your IGP and flap the interface when the MTU changes, you’ll detect MTU mismatches immediately due to OSPF’s DB exchange MTU negotiation. Granted, control plane detection won’t detect data plane bugs resulting in MTU fluctuations but I don’t see this as a frequent event.

Thanks,
Acee

From: Rtg-bfd <[email-masked]> on behalf of "Albert Fu (BLOOMBERG/ 120 PARK)" <[email-masked]>
Reply-To: Albert Fu <[email-masked]>
Date: Tuesday, October 23, 2018 at 11:44 AM
To: "[email-masked]" <[email-masked]>, "Les Ginsberg (ginsberg)" <[email-masked]>
Subject: RE: BFD WG adoption for draft-haas-bfd-large-packets

Hi Les,

Thanks
Albert

From: [email-masked] At: 10/23/18 10:45:02
To: Albert Fu (BLOOMBERG/ 120 PARK ) <mailto:[email-masked]> , [email-masked]<mailto:[email-masked]>
Subject: RE: BFD WG adoption for draft-haas-bfd-large-packets
Albert –

For your convenience:

<snip>
It has been stated that there is a need for sub-second detection of this condition – but I really question that requirement.
What I would expect is that MTU changes only occur as a result of some maintenance operation (configuration change, link addition/bringup, insertion of a new box in the physical path etc.). The idea of using a mechanism which is specifically tailored for sub-second detection to monitor something that is only going to change occasionally seems inappropriate. It makes me think that other mechanisms (some form of OAM, enhancements to routing protocols to do what IS-IS already does •) could be more appropriate and would still meet the operational requirements.

Thanx.

   Les

---

From: Greg Mirsky <[email-masked]>
Date: Tue, 23 Oct 2018 09:53:14 -0700

Dear All,
I agree and subscribe to the proposal to discuss the problem and the
possible solution based on BFD protocol. And adoption of this document, in
my view, is a logical step. I may point out that RFC 6423 described the use
of the existing BFD session providing connectivity verification over p2p
bidirectional MPLS-TP LSP or PW (note that BFD, in fact, provides
continuity check, not checking connectivity in transport network sense).

Regards,
Greg

On Tue, Oct 23, 2018 at 9:31 AM Acee Lindem (acee) <[email-masked]> wrote:

> Hi Albert, Les,
>
>
>
> I tend to agree with Les that BFD doesn’t seem like the right protocol for
> this. Note that if you use OSPF as your IGP and flap the interface when the
> MTU changes, you’ll detect MTU mismatches immediately due to OSPF’s DB
> exchange MTU negotiation. Granted, control plane detection won’t detect
> data plane bugs resulting in MTU fluctuations but I don’t see this as a
> frequent event.
>
>
>
> Thanks,
>
> Acee
>
>
>
> *From: *Rtg-bfd <[email-masked]> on behalf of "Albert Fu
> (BLOOMBERG/ 120 PARK)" <[email-masked]>
> *Reply-To: *Albert Fu <[email-masked]>
> *Date: *Tuesday, October 23, 2018 at 11:44 AM
> *To: *"[email-masked]" <[email-masked]>, "Les Ginsberg (ginsberg)" <
> [email-masked]>
> *Subject: *RE: BFD WG adoption for draft-haas-bfd-large-packets
>
>
>
> Hi Les,
>
>
>
> Given that it takes relative lengthy time to troubleshoot the MTU issue,
> and the associated impact on customer traffic, it is important to have a
> reliable and fast mechanism to detect the issue.
>
>
>
> I believe BFD, especially for single hop control-plane independent
> situation (btw, this covers majority of our BFD use case), is indeed an
> ideal and reliable solution for this purpose. It is also closely tied with
> the routing protocols, and enable traffic to be diverted very quickly.
>
>
>
> The choice of BFD timer is also one of the design tradeoffs - low BFD
> detection timer will cause more network churns. We do not need extremely
> aggressive BFD timer to achieve fast convergence. For example, with
> protection, we can achieve end to end sub-second convergence by using
> relatively high BFD interval of 150ms.
>
>
>
> In the case where the path will be used for a variety of encapsulations
> (e.g. Pure IP and L3VPN traffic), we would set the BFD padding to cater for
> the largest possible payload. So, in our case, our link needs to carry a
> mix of pure IP (1500 max payload) and MPLS traffic (1500 + 3 headers), we
> would set the padding so that the total padded BFD packet size is 1512
> bytes.
>
>
>
> As you rightly pointed out, ISIS routing protocol does support hello
> padding, but since this is a control plane process, we can not use
> aggressive timer. The lowest hello interval the can be configured is 1s, so
> with default multiplier of 3, the best we can achieve is 3s detection time.
>
>
>
> What we would like is a simple mechanism to validate that a link can
> indeed carry the expected max payload size before we put it into
> production. If an issue occurs where this is no longer the case (e.g. due
> to outages or re-routing within the Telco circuit), we would like a
> reliable mechanism to detect this, and also divert traffic around the link
> quickly. I feel BFD is a good method for this purpose.
>
>
>
> Thanks
>
> Albert
>
>
>
> To: Albert Fu (BLOOMBERG/ 120 PARK ) <[email-masked]>,
> [email-masked]
> Please understand that I fully agree with the importance of being able to
> detect/report MTU issues. In my own experience this can be a difficult
> problem to diagnose. You do not have to convince me that some improvement
> in detection/reporting is needed. The question really is whether using BFD
> is the best option.
>
>
>
> Could you respond to my original questions – particularly why sub-second
> detection of this issue is a requirement?
>
>
>
> For your convenience:
>
>
>
> *<snip>*
>
> *It has been stated that there is a need for sub-second detection of this
> condition – but I really question that requirement. *
>
> *What I would expect is that MTU changes only occur as a result of some
> maintenance operation (configuration change, link addition/bringup,
> insertion of a new box in the physical path etc.). The idea of using a
> mechanism which is specifically tailored for sub-second detection to
> monitor something that is only going to change occasionally seems
> inappropriate. It makes me think that other mechanisms (some form of OAM,
> enhancements to routing protocols to do what IS-IS already does **J)
> could be more appropriate and would still meet the operational
> requirements.*
>
>
>
> *I have listened to the Montreal recording – and I know there was
> discussion related to these issues (not sending padded packets all the
> time, use of BFD echo, etc.) – but I would be interested in more discussion
> of the need for sub-second detection.*
>
>
>
> *Also, given that a path might be used with a variety of encapsulations,
> how do you see such a mechanism being used when multiple BFD clients share
> the same BFD session and their MTU constraints are different?*
>
> *<end snip>*
>
>
>
> Thanx.
>
>
>
>    Les
>
>
>
>
>
>

---

From: "Les Ginsberg (ginsberg)" <[email-masked]>
Date: Tue, 23 Oct 2018 23:52:50 +0000

Albert -

From: Albert Fu (BLOOMBERG/ 120 PARK) <[email-masked]>
Sent: Tuesday, October 23, 2018 8:45 AM
To: [email-masked]; Les Ginsberg (ginsberg) <[email-masked]>
Subject: RE: BFD WG adoption for draft-haas-bfd-large-packets

Hi Les,

[Les:] This is  one of the points where we are not in full agreement. I agree you need an easy and reliable way to detect the problem when it occurs.
However, I disagree that you need to do this “fast” – when fast is defined as sub-second.

You have something that we know only occurs during some maintenance event – which is planned and only occurs “once/day,week”.
Checking for this even once/second is overly aggressive.
If it came for free, then no reason not to do so.
But as this discussion has shown, there are costs/risks.

For example, if you were using IS-IS and you detected this within the default adjacency hold time (30 seconds on p2p circuits) – would that be too slow for you? If so, please explain why this is too slow.

I think the primary issue here is ease of use and reliability. Whether detection time is one second or one minute seems relatively unimportant.

Do you disagree?

   Les

---

From: Jeffrey Haas <[email-masked]>
Date: Thu, 25 Oct 2018 11:29:09 -0400

Note that I'm choosing this message out of a mixed thread to give a reply.
So, not all of this is targeted as a response to you, Acee.

> I tend to agree with Les that BFD doesn’t seem like the right protocol for this. Note that if you use OSPF as your IGP and flap the interface when the MTU changes, you’ll detect MTU mismatches imme…

Commenting specifically on the OSPF case, when you have such misconfigured
MTUs, this manifests as weird protocol hiccups.  You don't so much detect
that there's an MTU issue - you just see OSPF failing to make progress.

Commenting on the general "well, this other protocol has MTU features",
one of the meta issues is there's no guarantee that a given protocol may be
running over a given link.  Several of the scenarios Albert writes about the
links may only be running BGP as a control plane protocol.  

Should BFD be *the* protocol for this?  Possibly not.  What else instead?  A
specific MTU probing protocol?  What does it look like?  My suspicion is
that such a thing would have a large overlap with BFD state machinery.

> Could you respond to my original questions – particularly why sub-second detection of this issue is a requirement?

Sub-second detection of MTU is not specifically the requirement - at least
in Albert's use case.  I'll let others write about their own requirements.

The interesting question arises that if BFD is where we're choosing our MTU
probing, what's the intersection between BFD timeliness requirements with
its usual shorter packets vs. MTU probing?  

Note this is particularly important for standard single-hop BFD since you
only ever have a *single* session between a given set of endpoints.

> It has been stated that there is a need for sub-second detection of this
> condition – but I really question that requirement.
>
> What I would expect is that MTU changes only occur as a result of some
> insertion of a new box in the physical path etc.).

Unfortunately, that's not always the case.

An example (not necessarily Albert's) of where this is problematic is in the
case of leased circuits that are partially created using some sort of
mechanism such as L2VPNs.  While it may appear to the customer that the
circuit is a leased line of a given set of properties, it may be delivered
through the provider network in a packet-switched environment.  This means
that MTU issues through the provider network, perhaps the result of a
topology change, may result in issues.  

So, this work - and then stop working.

> The idea of using a
> inappropriate.

For Albert's specific use case, I think there's some level of tolerance for
detection timing for a down link vs. bad MTU detection.  However, I don't
think we pretend to speak for all possible users of such a mechanism.  And
thus, this discussion. :-)

>  It makes me think that other mechanisms (some form of OAM,
> enhancements to routing protocols to do what IS-IS already does •) could
> be more appropriate and would still meet the operational requirements.

As noted above, I'm not sure layering this onto a full control plane
protocol as the primary mechanism is the best idea.  (You want MTU
detection?  You have to build a dummy IS-IS session?  Ugh.)

Certainly documenting existing mechanisms and when they can deal with the
use case is worth noting - perhaps even in the BFD Large draft itself.
Please feel free to suggest text.

But we have other situations where an IGP-like MTU mechanism won't help.
Two simple examples being BGP and static routes.

Also, certainly some form of lower layer mechanism could be used,if
applicable.  Given that one form of known issue is when a LAG is built from
component elements where a given element may have smaller MTU than their
siblings, it'd be nice if this was solved in something like LACP as well.
(And we note that BFD for LAG exists, and is one of the possible
applications for BFD-large as well.)

-- Jeff

---

From: "Acee Lindem (acee)" <[email-masked]>
Date: Thu, 25 Oct 2018 21:18:16 +0000

Hi Jeff, 

Although technically BFD doesn’t seem like the right hammer, I can see the advantages of using it in specific use cases since all the notification machinery is already in place. See one inline. 

> On Oct 25, 2018, at 11:29 AM, Jeffrey Haas <[email-masked]> wrote:
> 

However, when implementations start supporting ietf-ospf.yang, there’ll be an if-config-err notification specifically indicating an MTU-mismatch. ;^) 

Thanks,
Acee