Subject: Re: BFD WG adoption for draft-haas-bfd-large-packets

From: "Albert Fu (BLOOMBERG/ 120 PARK)" <[email-masked]>
Date: Tue, 23 Oct 2018 17:21:09 -0000

Hi Acee,

Please see comments in-line. 

Thanks,

Albert

From: [email-masked] At: 10/23/18 13:02:49To:  Albert Fu (BLOOMBERG/ 120 PARK ) ,  [email-masked],  [email-masked]
Subject: Re: BFD WG adoption for draft-haas-bfd-large-packets

Hi Albert,  

From: "Albert Fu (BLOOMBERG/ 120 PARK)" <[email-masked]>
Reply-To: Albert Fu <[email-masked]>
Date: Tuesday, October 23, 2018 at 12:45 PM
To: "[email-masked]" <[email-masked]>, "Les Ginsberg (ginsberg)" <[email-masked]>, Acee Lindem <[email-masked]>
Subject: Re: BFD WG adoption for draft-haas-bfd-large-packets 

Hi Acee,  

You are right in that this issue does not happen frequently, but when it does, it is time consuming to troubleshoot and causes unnecessary network downtime  to some applications (e.g. between two end hosts, some applications worked fine, but others would intermittently fail when they tried to send large size packets over the failing ECMP path). 
  
So you’re saying there is a problem where the data plane interfaces do not support the configured MTU due to a SW bug? I hope these are not our routers 😉

AF> There's no bug.  

1) The issue we have seen is with the Telco network. The router can happily transmit and receive up to configured interface MTU, but the Telco circuit fails to support it. One example is when Telco uses L2VPN to deliver the P2P service to us, but due to some faults, traffic was re-routed to a mis-configured path that did not support our MTU size (e.g. MTU on Telco PE router was not increased to account for MPLS headers for the L2VPN service).

2) AFAIK, the OSPF MTU detection is based on checking MTU value in the DBD packet, The actual OSPF packet size may be smaller and may not detect data plane issue in Telco network during OSPF session establishment.

I believe the OSPF MTU detection is a control plane mechanism to check config, and may not necessary detect a data plane MTU issue (since OSPF does not support  padding). Also, most of our issues occurred after routing adjacency had been established, and without any network alarms. 
  
Right. However, if the interface is flapped when the MTU changes, OSPF would detect dynamic MTU changes (e.g., configuration), that the control plane is aware of. 

AF> We have encountered the MTU issue without any interface flaps on our routers (no config change on our routers). The MTU issue occurred within the Telco network. Note also some Telco providers that provide WAN circuit spanning several countries may make use of smaller local providers to provide the last mile.  We have seen issues with the smaller providers.

  
Thanks, 
Acee  

Thanks 

Albert 

From: [email-masked] At: 10/23/18 12:30:55 
To: Albert Fu (BLOOMBERG/ 120 PARK ) , [email-masked], [email-masked]
Subject: Re: BFD WG adoption for draft-haas-bfd-large-packets 

Hi Albert, Les,  
  
I tend to agree with Les that BFD doesn’t seem like the right protocol for this. Note that if you use OSPF as your IGP and flap the interface when the MTU changes, you’ll detect MTU mismatches  immediately due to OSPF’s DB exchange MTU negotiation. Granted, control plane detection won’t detect data plane bugs resulting in MTU fluctuations but I don’t see this as a frequent event. 
  
Thanks, 
Acee 

From: Rtg-bfd <[email-masked]> on behalf of "Albert Fu (BLOOMBERG/ 120 PARK)" <[email-masked]>
Reply-To: Albert Fu <[email-masked]>
Date: Tuesday, October 23, 2018 at 11:44 AM
To: "[email-masked]" <[email-masked]>, "Les Ginsberg (ginsberg)" <[email-masked]>
Subject: RE: BFD WG adoption for draft-haas-bfd-large-packets 

Hi Les,   

Given that it takes relative lengthy time to troubleshoot the MTU issue, and the associated impact on customer traffic, it is important to have a reliable and fast mechanism to detect the  issue.  

I believe BFD, especially for single hop control-plane independent situation (btw, this covers majority of our BFD use case), is indeed an ideal and reliable solution for this purpose. It  is also closely tied with the routing protocols, and enable traffic to be diverted very quickly.  

The choice of BFD timer is also one of the design tradeoffs - low BFD detection timer will cause more network churns. We do not need extremely aggressive BFD timer to achieve fast convergence.  For example, with protection, we can achieve end to end sub-second convergence by using relatively high BFD interval of 150ms.  

In the case where the path will be used for a variety of encapsulations (e.g. Pure IP and L3VPN traffic), we would set the BFD padding to cater for the largest possible payload. So, in our  case, our link needs to carry a mix of pure IP (1500 max payload) and MPLS traffic (1500 + 3 headers), we would set the padding so that the total padded BFD packet size is 1512 bytes.  

As you rightly pointed out, ISIS routing protocol does support hello padding, but since this is a control plane process, we can not use aggressive timer. The lowest hello interval the can  be configured is 1s, so with default multiplier of 3, the best we can achieve is 3s detection time. 

What we would like is a simple mechanism to validate that a link can indeed carry the expected max payload size before we put it into production. If an issue occurs where this is no longer  the case (e.g. due to outages or re-routing within the Telco circuit), we would like a reliable mechanism to detect this, and also divert traffic around the link quickly. I feel BFD is a good method for this purpose. 

Thanks 

Albert   

From: [email-masked] At: 10/23/18 10:45:02 
To: Albert Fu (BLOOMBERG/ 120 PARK ) , [email-masked]
Subject: RE: BFD WG adoption for draft-haas-bfd-large-packets 

Albert – 
  
Please understand that I fully agree with the importance of being able to detect/report MTU issues. In my own experience this can be a difficult problem to diagnose. You do not have  to convince me that some improvement in detection/reporting is needed. The question really is whether using BFD is the best option. 
  
Could you respond to my original questions – particularly why sub-second detection of this issue is a requirement? 
  
For your convenience: 
  
<snip> 
It has been stated that there is a need for sub-second detection of this condition – but I really question that requirement.  
What I would expect is that MTU changes only occur as a result of some maintenance operation (configuration change, link addition/bringup, insertion of a new box in the physical  path etc.). The idea of using a mechanism which is specifically tailored for sub-second detection to monitor something that is only going to change occasionally seems inappropriate. It makes me think that other mechanisms (some form of OAM, enhancements to  routing protocols to do what IS-IS already does J) could be more appropriate  and would still meet the operational requirements. 
  
I have listened to the Montreal recording – and I know there was discussion related to these issues (not sending padded packets all the time, use of BFD echo, etc.) – but I would  be interested in more discussion of the need for sub-second detection. 
  
Also, given that a path might be used with a variety of encapsulations, how do you see such a mechanism being used when multiple BFD clients share the same BFD session and their  MTU constraints are different? 
<end snip> 
  
Thanx. 
  
   Les

---

From: "Albert Fu (BLOOMBERG/ 120 PARK)" <[email-masked]>
Date: Tue, 23 Oct 2018 21:44:41 -0000

Hi Naiming,

I agree with you that BFD padding should be an option, and on per neighbor/interface basis. 

For internal infrastructure consisting of mainly back-to-back links, network designers can choose to use default BFD behavior without padding (unless they want to guard against potential config error or data plane error due to HW issues). In our case, most of our links are on WAN circuits - we would like to use BFD padding to guard against Telco MTU issue.

Thanks
Albert

From: [email-masked] At: 10/23/18 17:04:27To:  [email-masked]
Cc:  Albert Fu (BLOOMBERG/ 120 PARK ) ,  [email-masked],  [email-masked]
Subject: Re: BFD WG adoption for draft-haas-bfd-large-packets

 I understand in some networks (such as in the network Albert mentioned about) 
it may need sub-second detection if the MTU has been impacted on the path, 
but that should only be an implementation/operational option, I would think most 
of the networks will not use this way.  The draft should handle it in 
a general way, or are we saying other cases are not valid or we need a 
different draft to do the other ways? 

Regards, 
- Naiming 

On Oct 23, 2018, at 10:02 AM, Acee Lindem (acee) <[email-masked]> wrote: 
Hi Albert,  

Hi Acee, 

  

Thanks, 
Acee  
  
Thanks 
Albert

---

From: "Albert Fu (BLOOMBERG/ 120 PARK)" <[email-masked]>
Date: Wed, 24 Oct 2018 14:48:20 -0000

Hi Les,

All the MTU issues we have seen were on Telco WAN circuits. They were not planned events, and there were no alarms on our side. If there had been alarms, it would have made troubleshooting that much easier as we would know where to focus troubleshooting efforts. 

ISIS does support padding, but 30s outage is not an acceptable outage in our network. Additionally, we have WAN circuits that run other routing protocols such as OSPF and eBGP which do not support hello padding. Even if the routing protocol does support padding, we may not want to use aggressive timer as it is a control plane activity. Also, as mentioned previously, if we use minimum 1s protocol timer, we still have about 3s of outage. As per common practice, we leave our protocol timers default and leverage BFD for fast failure detection. 

Hence, I believe BFD is a very good mechanism to address this issue. I understand some customers want to run very aggressive BFD timers to detect failures quickly (at the expense of higher network churns). We found that we can achieve sub-second convergence with protection using relatively conservative BFD interval of 150msec. Also, as mentioned previously, depending on implementation, the BFD padding support may have very small impact on performance.

I would also add that the BFD padding support will be an option on a per interface/neighbor basis. Network Designer who does not have to deal with MTU issue can choose to use the default behavior. They can also enable padding on WAN circuits, and use default for back-back intra-site links.

Thanks
Albert

From: [email-masked] At: 10/23/18 19:52:53To:  Albert Fu (BLOOMBERG/ 120 PARK ) ,  [email-masked]
Subject: RE: BFD WG adoption for draft-haas-bfd-large-packets

Albert - 

From: Albert Fu (BLOOMBERG/ 120 PARK) <[email-masked]> 
Sent: Tuesday, October 23, 2018 8:45 AM
To: [email-masked]; Les Ginsberg (ginsberg) <[email-masked]>
Subject: RE: BFD WG adoption for draft-haas-bfd-large-packets 

Hi Les, 

[Les:] This is  one of the points where we are not in full agreement. I agree you need an easy and reliable way to detect the problem when it occurs. 
However, I disagree that you need to do this “fast” – when fast is defined as sub-second. 
  
You have something that we know only occurs during some maintenance event – which is planned and only occurs “once/day,week”. 
Checking for this even once/second is overly aggressive. 
If it came for free, then no reason not to do so. 
But as this discussion has shown, there are costs/risks. 
  
For example, if you were using IS-IS and you detected this within the default adjacency hold time (30 seconds on p2p circuits) – would that be too slow  for you? If so, please explain why this is too slow. 
  
I think the primary issue here is ease of use and reliability. Whether detection time is one second or one minute seems relatively unimportant. 
  
Do you disagree? 
  
   Les

---

From: "Albert Fu (BLOOMBERG/ 120 PARK)" <[email-masked]>
Date: Fri, 26 Oct 2018 17:06:27 -0000

Hi Acee,

>> Commenting specifically on the OSPF case, when you have such misconfigured
>> MTUs, this manifests as weird protocol hiccups.B  You don't so much detect
>> that there's an MTU issue - you just see OSPF failing to make progress.
>
> However, when implementations start supporting ietf-ospf.yang, there?ll be an
> if-config-err notification specifically indicating an MTU-mismatch. ;^) 

As mentioned previously, the issue that we are trying to address is where there's
no issue with OSPF MTU config, but the WAN circuit fails to carry traffic up to
the configured MTU. 

For example, in last 2 weeks, we have 2 such issues, one with major Tier 1
provider in US and another one with Tier 1 provider in Europe, where OSPF was humming 
along fine, the circuits could carry 1500 byte payload but not 1512 byte payload.
This caused intermittent issue to our end user traffic that traversed these links and
not desirable. This is a fault situation - the circuit should not be put
in production. We have redundant circuits and would want to divert traffic from 
these links automatically (hence this draft).

Thanks
Albert.

---

From: "Albert Fu (BLOOMBERG/ 120 PARK)" <[email-masked]>
Date: Fri, 26 Oct 2018 17:52:40 -0000

(* sorry, resend ING with updated subject *) 

Hi Les,

> [Les:] I have read this draft - not sure how relevant it is.
> 
> Naiming had suggested that MTU sized packets need not be sent all the time but only occasionally - 
> and that a failure might not be used to take the BFD session down - 
> rather it would be seen as a "soft-failure" and reported separately from the BFD session state.
> My response was in that context - which it seems was also in your mind in your BFD Echo proposal.
> 
> This, however, seems not to be what Albert has in mind - as he has since commented that he really wants to have sub-second detection of MTU issues and 
> he wants traffic rerouted "immediately".
> 
>  Les

The MTU issue is a "hard" failure - e.g. any TCP application that's 
trying to send traffic with the MTU sized payload over that link 
will time out, and fail. Applications (between the same end stations)
that send small packets over that link will continue to work fine. 
Hence, this makes troubleshooting that much harder especially in 
network with lots of ECMP paths.

Thanks
Albert