Subject: [sfc] NSH Loop Prevention

From: "Reinaldo Penno (repenno)" <[email-masked]>
Date: Thu, 22 Oct 2015 15:44:55 +0000

Hello fellow SFCers,

wanted to share implementation experience on NSH Loop prevention topic. I might write a draft about this during/after the IETF but a presentation seems fine at this point. This is not really exhaustive, just a summary to gauge interest on the topic.

https://www.dropbox.com/s/ls8utofr1oddqon/nsh_loop_prevention.pptx?dl=0

Thanks,

-RP

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 22 Oct 2015 13:35:55 -0400

The description of the problem makes me wonder if we are decrementing 
the SFP ID in the wrong place.  In terms of my implementation, it is 
helpful to have the SF do the decrementing.  But most of these problems 
disappear if the SFF is responsible for the decrementing.

I also have concerns with the material on slides 3 and 5.  The 
disucssion in slide 3 assumes that the outer encaps is IP, and it 
assumes that the TTL on the outgoing transport header is related to the 
TTL on the incoming transport header.  But there is no reason for there 
to be a relationship.  A perfectly valid implementation strips 
transport, looks at NSH, then generates transport.  If SFF do not 
decrement SFP ID, and if the SFF are misconfigured, a non-terminating 
loop results.

Slide 5 assumes that the SFF knows, when processing a received packet 
from an SF, what the index was when it sent the packet to the SF.  Even 
in the case where the outbound processing SFF is the inbound processing 
SFF, there is no reason it should have saved that state, and the assumed 
packet matching is neither obvious nor simple.

Yours,
Joel

On 10/22/15 11:44 AM, Reinaldo Penno (repenno) wrote:
> Hello fellow SFCers,
>
> wanted to share implementation experience on NSH Loop prevention topic.
> I might write a draft about this during/after the IETF but a
> presentation seems fine at this point. This is not really exhaustive,
> just a summary to gauge interest on the topic.
>
> _______________________________________________
> sfc mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sfc
>

---

From: "Reinaldo Penno (repenno)" <[email-masked]>
Date: Thu, 22 Oct 2015 17:48:16 +0000

Joel,

Thanks for the comments, inline with [RP]

On 10/22/15, 10:35 AM, "Joel M. Halpern" <[email-masked]> wrote:

>disucssion in slide 3 assumes that the outer encaps is IP,

[RP] yes, that is the case. Other encaps will most likely solve this in a
different way. 
For example if the outer encap is L2 (ethernet) then STP will probably
need to be used. 

>and it 

[RP] Yes, that is a valid implementation but what I¹m suggesting is that
if the TTL is not 
decremented across VXLAN segments than serious problems might happen.
Whether the developer
want to heed to the advice or not is his(er) choice.

>packet matching is neither obvious nor simple.

[RP] yep, correct. Other solutions might be be even more intense in terms
of data plane. This is one suggestion.

---

From: Dave Dolson <[email-masked]>
Date: Thu, 22 Oct 2015 18:06:08 +0000

To add to Joel's comments, (which I agree with),

1. This isn't a solution for NSH carried directly in an Ethernet frame or other non-IP encapsulations.

2. If I understand correctly, there is a blatant layering violation. In fact, the IP TTL will be decremented by both SFF functions and also by the routers delivering the IP/VxLAN packets between IP end-points. E.g., a single SFF-to-SFF hop could be up to 255 IP hops.

3. If an SF does not decrement the path index, I see that as a bug in SF implementation, not a misconfiguration.
So I don't think the slides 5 or 6 cases need to be handled.

I do see that loops between SFFs (slide 3) are a real problem caused by misconfiguration of SFF forwarding tables, due to buggy orchestration.

What is a best-practice for solving such problems with an SDN mind-set?
Clearly it is possible to (mis)configure OpenFlow switches to create an infinite packet loop.
Is it expected to drop a packet passing too many hops, or merely to detect a problem?

-Dave

-----Original Message-----
From: sfc [mailto:[email-masked]] On Behalf Of Joel M. Halpern
Sent: Thursday, October 22, 2015 1:36 PM
To: Reinaldo Penno (repenno); [email-masked]
Subject: Re: [sfc] NSH Loop Prevention

Yours,
Joel

---

From: "Reinaldo Penno (repenno)" <[email-masked]>
Date: Thu, 22 Oct 2015 18:17:50 +0000

>other non-IP encapsulations.

[RP] Answered that one to Joel.

>
>2. If I understand correctly, there is a blatant layering violation. In
>fact, the IP TTL will be decremented by both SFF functions and also by
>the routers delivering the IP/VxLAN packets between IP end-points. E.g.,
>a single SFF-to-SFF hop could be up to 255 IP hops.

[RP] This is not clear but a good point for more discussion.

>So I don't think the slides 5 or 6 cases need to be handled.

[RP] Yes, it is a bug but should you protect the network or not? We know
that a high speed loop will make the network crawl and crash.

>
>
>I do see that loops between SFFs (slide 3) are a real problem caused by
>misconfiguration of SFF forwarding tables, due to buggy orchestration.
>

[RP] All good points, the purpose of the presentation was to foster this
discussion and see what we can do (or suggest should be done) about it

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 22 Oct 2015 14:25:25 -0400

In case it was unclear, I very much appreciate your raising the points.

I am concerned that we probably need a systemic approach to solving the 
SFF forwarding loop problem.  Solutions which depend upon the 
combination of transport protocol behaviors and unspecified mechanisms 
in the SFF (such as relating incoming and outgoing transport TTLs) would 
seem to indicate that we have an incomplete specification.

Yours,
Joel

---

From: Dave Dolson <[email-masked]>
Date: Thu, 22 Oct 2015 18:51:01 +0000

It occurs to me that *both* SFFs and SFs could decrement the Path Index.
This makes the longest path to be 127 vs. 255.
But the packet would never be placed on the network with the same index twice.

I came to my point about not handling bugs by analogy: 
- if we have routers that do not decrement TTL, there can be loops. But we don't add another mechanism on top to detect if such buggy routers exist.

-----Original Message-----
From: Joel M. Halpern [mailto:[email-masked]] 
Sent: Thursday, October 22, 2015 2:25 PM
To: Reinaldo Penno (repenno); Dave Dolson; [email-masked]
Subject: Re: [sfc] NSH Loop Prevention

Yours,
Joel

---

From: "Reinaldo Penno (repenno)" <[email-masked]>
Date: Thu, 22 Oct 2015 19:07:00 +0000

Yep, it would simplify things, but requires a change to the current
proposed protocol. 

On 10/22/15, 11:51 AM, "Dave Dolson" <[email-masked]> wrote:

>- if we have routers that do not decrement TTL, there can be loops. But
>we don't add another mechanism on top to detect if such buggy routers
>exist.
>
>
>
>>> 1. This isn't a solution for NSH carried directly in an Ethernet frame
>>>or
>>> other non-IP encapsulations.
>>
>>> 3. If an SF does not decrement the path index, I see that as a bug in
>>>SF
>>> implementation, not a misconfiguration.

---

From: "Richard Johnson (raj)" <[email-masked]>
Date: Thu, 22 Oct 2015 19:10:26 +0000

Having the SFF decrement the index when passing to another SFF would seem to be a non-backward-compatible change to the protocol, but it would be useful for detecting loops in this situation.

Another thing it would allow would be the creation of one service-chain, which could then be downloaded to all SFFs.

Decrementing when passing to SFF:

Index    Node
255      SF-a
254      SF-b
253      SFF-2
252      SF-c
251      SFF-3
250      SF-d
249      terminate

Not decrementing when passing to SFF requires separate tables:

Index    Node
SFF-1:
255      SF-a
254      SF-b
253      SFF-2

SFF-2:
253      SF-c
252      SFF-3

SFF-3:
252      SF-d
250      terminate

/raj

> On Oct 22, 2015, at 11:51 AM, Dave Dolson <[email-masked]> wrote:
>

---

From: Dave Dolson <[email-masked]>
Date: Thu, 22 Oct 2015 19:21:07 +0000

I think it is backwards compatible at the NSH specification;  however the control-plane would need to know which SFFs do the decrement.
- the SFs are unaffected
- SFFs look up incoming coordinates in table and forward (some classes of SFF might decrement).

---

From: Joel Halpern Direct <[email-masked]>
Date: Thu, 22 Oct 2015 15:23:15 -0400

Why does it matter if it is backwards compatible or not?  The working 
group still owns the specification.  There is no RFC.  And even if there 
were, if we found a problem, we would have to fix it.

Yours,
Joel

---

From: "Reinaldo Penno (repenno)" <[email-masked]>
Date: Thu, 22 Oct 2015 19:25:21 +0000

On 10/22/15, 12:21 PM, "Dave Dolson" <[email-masked]> wrote:

>I think it is backwards compatible at the NSH specification;  however the
>control-plane would need to know which SFFs do the decrement.
>- the SFs are unaffected

[RP] Oh, please, no, no. All SFFs would need to decrement otherwise we
would be worse off than when we started this discussion. We would have two
code paths, two test suites two types of chains, etc.

>>>> Clearly it is possible to (mis)configure OpenFlow switches to create
>>>>an
>>>> infinite packet loop.

---

From: "Jim Guichard (jguichar)" <[email-masked]>
Date: Thu, 22 Oct 2015 20:13:16 +0000

I disagree and for me decrementing index at SFF seriously complicates
implementation. If the SFF decrements the index then you would have to
keep track of ³received" interface as otherwise how would an SFF know
whether to forward to an SF, or decrement and forward to another SF or SFF?

Jim

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 22 Oct 2015 16:20:26 -0400

It seems to me that if SFF decrement then they should always decrement. 
  Trying for hybrids seems more complex.  At which point I do not see 
value in the SF decrementing.

If the SFF always decrement, does that complicate your implementation? 
It does cause some difficulty for mine, but I am having trouble finding 
a better answer.

Yours,
Joel

---

From: Dave Dolson <[email-masked]>
Date: Thu, 22 Oct 2015 20:20:36 +0000

Obviously it would have to be clearly defined as to whether the
incrementing happens before or after the next-hop lookup.
But once that is clear, the next-hop of any received packet
is precisely defined by the look-up table.

And also, I think it is a solution to the loop prevention, solved
within the NSH layer only.

-----Original Message-----
From: Jim Guichard (jguichar) [mailto:[email-masked]] 
Sent: Thursday, October 22, 2015 4:13 PM
To: Dave Dolson; Joel M. Halpern; Reinaldo Penno (repenno); [email-masked]
Subject: Re: [sfc] NSH Loop Prevention

Jim

---

From: "Reinaldo Penno (repenno)" <[email-masked]>
Date: Thu, 22 Oct 2015 21:14:41 +0000

I agree with Jim. This is much easier said than done.  Just look around
how routers implement decrementing TTL.

Some routers that decrement on interface ingress,  others after route
lookup, or interface egress, and sometimes even more than once. All
possible variations.

Thanks,

-RP

---

From: Dave Dolson <[email-masked]>
Date: Thu, 22 Oct 2015 21:34:41 +0000

Don't the same questions (and more) exist ‎with the VxLAN TTL approach?

  Original Message
From: Reinaldo Penno (repenno)
Sent: Thursday, October 22, 2015 5:15 PM
To: Dave Dolson; Jim Guichard (jguichar); Joel M. Halpern; [email-masked]
Subject: Re: [sfc] NSH Loop Prevention

Thanks,

-RP

---

From: "Jim Guichard (jguichar)" <[email-masked]>
Date: Thu, 22 Oct 2015 21:34:58 +0000

I would argue Joel that it complicates any implementation. For an SFF
forwarding loop use outer transport loop detection mechanism that already
exists.

Jim

---

From: "Reinaldo Penno (repenno)" <[email-masked]>
Date: Thu, 22 Oct 2015 22:34:05 +0000

not exactly because TTL is _only_ used for loop detection and not
forwarding, and that is key. So if it is decremented during ingress, after
route lookup or egress it does not matter. Therefore TTL takes care of
loop between SFFs as a transport problem (VXLAN).

On 10/22/15, 2:34 PM, "Dave Dolson" <[email-masked]> wrote:

>Don't the same questions (and more) exist with the VxLAN TTL approach?
>
>
>  Original Message
>>>>>>https://www.dropbox.com/s/ls8utofr1oddqon/nsh_loop_prevention.pptx?dl
>>>>>>=
>>>>>>0
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> -RP
>>>>>>
>>>>>>
>>>>>>

---

From: Dave Dolson <[email-masked]>
Date: Thu, 22 Oct 2015 22:43:18 +0000

Another solution is to add a TTL to the NSH header.
Decremented by all nodes, not used for forwarding.
Works on all transports.

  Original Message
From: Reinaldo Penno (repenno)
Sent: Thursday, October 22, 2015 6:34 PM
To: Dave Dolson; Jim Guichard (jguichar); Joel M. Halpern; [email-masked]
Subject: Re: [sfc] NSH Loop Prevention

---

From: "Surendra Kumar (smkumar)" <[email-masked]>
Date: Fri, 23 Oct 2015 16:06:43 +0000

Thanks for shining light on NSH service index, it is an important consideration and one that needs to be addressed.

Slide3, about SFs decrementing SI is in itself a problem. Having SFs interpret SPIDs or decrement the index is fraught with problems.
Firstly, the SFs now need to be aware of not only the Service Chains but also where they are in the service chain.
* This blurs the functional lines between SFF and SFs.
* Why should a video-optimizer care if it is before or after a firewall or DPI ?

Control plane has to now program SPIDs and SPID/SI specific policies into all the SFs, even though this has nothing to do with the service being delivered by that SF
* This again devalues the SFF functionality

There is a security implication in allowing SFs decrement the SI.
* If an SF decrements more than it should - think of bypassing a firewall, what good is having an SFF.
* This brings up the question of trust domains. Is every SF, third party for instance, part of this trust domain ?
* Do we have a mechanism to enforce this ?

If the fix, as indicated on slide5/6, is to check the previous index or perform additional checks, why even allow the SFs to decrement in the first place ?

Having the SFC infra, manage the SPIDs and Service Indices, keeps the boundaries and the architecture simple. It further allows invariants to be enforced at the infra level while isolating the trust domains.

My $.02
Surendra.

-----Original Message-----
From: sfc [mailto:[email-masked]] On Behalf Of Joel M. Halpern
Sent: Thursday, October 22, 2015 1:20 PM
To: Jim Guichard (jguichar) <[email-masked]>; Dave Dolson <[email-masked]>; Reinaldo Penno (repenno) <[email-masked]>; [email-masked]
Subject: Re: [sfc] NSH Loop Prevention

It seems to me that if SFF decrement then they should always decrement. 
  Trying for hybrids seems more complex.  At which point I do not see value in the SF decrementing.

If the SFF always decrement, does that complicate your implementation? 
It does cause some difficulty for mine, but I am having trouble finding a better answer.

Yours,
Joel

On 10/22/15 4:13 PM, Jim Guichard (jguichar) wrote:
>> - if we have routers that do not decrement TTL, there can be loops. 
>> But we don't add another mechanism on top to detect if such buggy 
>> routers exist.
>>
>>
>>
>> I am concerned that we probably need a systemic approach to solving 
>> the SFF forwarding loop problem.  Solutions which depend upon the 
>> combination of transport protocol behaviors and unspecified 
>> mechanisms in the SFF (such as relating incoming and outgoing 
>> transport TTLs) would seem to indicate that we have an incomplete specification.
>>
>> Yours,
>> Joel
>>
>>>> 2. If I understand correctly, there is a blatant layering 
>>>> violation. In fact, the IP TTL will be decremented by both SFF 
>>>> functions and also by the routers delivering the IP/VxLAN packets between IP end-points.
>>>> E.g.,
>>>> 3. If an SF does not decrement the path index, I see that as a bug 
>>>> in SF implementation, not a misconfiguration.

---

From: "Bottorff, Paul" <[email-masked]>
Date: Fri, 23 Oct 2015 17:24:05 +0000

This is discussion is enlightening on the use of the service index.

It appears we are overloading the use of the service index to both express the hop count and indicate SF completion.

As Surendra has pointed out, using an incrementing index for SF completion creates both a security problems and control plane problems. In addition, the creation of a path as series of index hops makes modification of the chain more complex since the entire chain must be re-written to insert or delete a SF hop.

I believe the SF itself does not need to increment the service index, however it does need to be incremented in the hop to the SF. For instance it could be incremented by the guest OS or at the vSwitch port (connection point between vSwitch and vNIC) or at a SFF virtual port.

I also agree Dave that the use of IP layer TTL is a layer violation.

The next hop and the hop count should be separate fields. It would be better to use the service index strictly as a hop count which should be incremented at both SF(or along the path to SF) and SFF.

Cheers,

Paul

-----Original Message-----
From: sfc [mailto:[email-masked]] On Behalf Of Surendra Kumar (smkumar)
Sent: Friday, October 23, 2015 9:07 AM
To: Joel M. Halpern; Jim Guichard (jguichar); Dave Dolson; Reinaldo Penno (repenno); [email-masked]
Subject: Re: [sfc] NSH Loop Prevention

My $.02
Surendra.

-----Original Message-----
From: sfc [mailto:[email-masked]] On Behalf Of Joel M. Halpern
Sent: Thursday, October 22, 2015 1:20 PM
To: Jim Guichard (jguichar) <[email-masked]>; Dave Dolson <[email-masked]>; Reinaldo Penno (repenno) <[email-masked]>; [email-masked]
Subject: Re: [sfc] NSH Loop Prevention

Yours,
Joel

On 10/22/15 4:13 PM, Jim Guichard (jguichar) wrote:
>>> [RP] Yes, it is a bug but should you protect the network or not? We 
>>> know that a high speed loop will make the network crawl and crash.
>>>
>>>>
>>>>
>>>> I do see that loops between SFFs (slide 3) are a real problem 
>>>> caused by misconfiguration of SFF forwarding tables, due to buggy orchestration.
>>>>
>>> [RP] All good points, the purpose of the presentation was to foster 
>>> this discussion and see what we can do (or suggest should be done) 
>>> about it
>>>
>>>>
>>>>
>>>> -Dave
>>>>
>>>>
>>>>
>>>> The description of the problem makes me wonder if we are 
>>>> decrementing the SFP ID in the wrong place.  In terms of my 
>>>> implementation, it is helpful to have the SF do the decrementing.  
>>>> But most of these problems disappear if the SFF is responsible for the decrementing.
>>>>
>>>> assumes that the TTL on the outgoing transport header is related to 
>>>> the TTL on the incoming transport header.  But there is no reason 
>>>> for there to be a relationship.  A perfectly valid implementation 
>>>> strips transport, looks at NSH, then generates transport.  If SFF 
>>>> do not decrement SFP ID, and if the SFF are misconfigured, a 
>>>> non-terminating loop results.
>>>>
>>>> in the case where the outbound processing SFF is the inbound 
>>>> processing SFF, there is no reason it should have saved that state, 
>>>> and the assumed packet matching is neither obvious nor simple.
>>>>
>>>> Yours,
>>>> Joel
>>>>
>>>>> presentation seems fine at this point. This is not really 
>>>>> exhaustive, just a summary to gauge interest on the topic.
>>>>>
>>>>>