Subject: Re: [sfc] decrementing service function path index

From: "jmh.direct" <[email-masked]>
Date: Fri, 11 Nov 2016 09:04:02 -0800

Actually, I had presumed that the particular liaden balancing you describe was not expected to work.
One of my concerns with the proposed BGP encoding is that it can represent many things that will not work.  This seems to make it fragile.
Yours,Joel

Sent via the Samsung Galaxy S® 6, an AT&T 4G LTE smartphone
-------- Original message --------From: Eric C Rosen <[email-masked]> Date: 11/11/16  08:58  (GMT-08:00) To: Dave Dolson <[email-masked]>, [email-masked], "'Joel M. Halpern'" <[email-masked]>, [email-masked] Subject: Re: [sfc] decrementing service function path index 
On 11/1/2016 4:26 PM, Dave Dolson wrote:
> Yes, I'm saying that the current (as I understand it) design permits the SFF to be a simple forwarder,
> not needing to discriminate between packets from another SFF or returning from an SF.
> A single forwarding table handles both cases.
> E.g., at SFF1, when SI=x, send to SF1; when SI=x+1, send to SFF2
I wonder what the WG has decided about the following case.

Suppose Node 1 contains SFF1 and SF1-1, while Node 2 contains SFF2 and 
SF1-2, where SF1-1 and SF1-2 are two instances of of the same service 
function (SF1).

So SFF1 is attached locally to SF1-1, and SFF2 is attached locally to SF1-2,

Now suppose that the SPI/SI value "x/y" means "SF1".

When SFF1 sees "x/y" in the NSH, it may choose to load balance between 
SF1-1 and SF1-2.  This means it may forward either locally to SF1-1 or 
over the network to SF1-2.

Suppose SFF1 chooses SF1-2.  It then needs to tunnel the packet through 
the underlay to Node 2.

What happens when the packet arrives at Node 2?  If the packet is seen 
by SFF2, SFF2 will see "x/y" in the NSH, and may choose to forward 
either locally to SF1-2 or over the network to SF1-1. Since SFF2 doesn't 
know that the packet came from another SFF, nothing stops it from 
choosing SF1-1.

That is,  SFF1 chooses to use SF1-2, and SFF2 chooses to use SF1-1, thus 
forming a loop.  Since this loop is between the two SFFs, the SI never 
gets decremented.

Presumably this type of load-balancing is allowed, but the loop must be 
prevented.

One way to prevent the loop is for SFF2 to know whether a packet with 
"x/y" in the NSH has already been seen by another forwarder or not.

> In some cases, an SFF can be a single-interface device, so "from SFF" or "from SF" is not as simple
> as checking ingress interface, for example.

Presumably there's a tunnel of some sort, through the underlay, that 
leads from SFF1 to SFF2.   This tunnel could be regarded as a virtula 
ingress interface.

Alternatively, SFF1 could tunnel a packet directly to SF1-2, bypassing 
SFF2 entirely.  SFF2 then wouldn't actually process the packet until 
SF1-2 has finished with it.  The load balancing decision would be made 
only once (by SFF1).  in this case, the SFFs don't need to know about a 
packet's ingress "interface" because SFFs don't get packets from each 
other directly.

This requires only that the tunnel through the underlay contain 
demultiplexing information that will cause the packet, once it arrives 
at Node 2, to be delivered directly to SF1-2, bypassing SFF2.

This latter model seems to be what is required by section 5.5 of 
RFC7665, but the RFC is not very explicit about it.

---

From: Ron Parker <[email-masked]>
Date: Fri, 11 Nov 2016 17:35:45 +0000

Joel,

If you recall our discussions around Rendered Service Paths (RSPs), I had a full expectation that this type of load balancing would work.   Further, we had some discussion of whether the NSH header should communicated SFP ID or RSP ID where the former is non-specific around the actual load balanced selection of SFIs and the latter is completely concrete in terms of the actual selected SFIs.

   Ron

From: sfc [mailto:[email-masked]] On Behalf Of jmh.direct
Sent: Friday, November 11, 2016 12:04 PM
To: Eric C Rosen <[email-masked]>; Dave Dolson <[email-masked]>; [email-masked]; 'Joel M. Halpern' <[email-masked]>; [email-masked]
Subject: Re: [sfc] decrementing service function path index

Yours,
Joel

---

From: Eric C Rosen <[email-masked]>
Date: Fri, 11 Nov 2016 15:49:50 -0500

On 11/11/2016 12:04 PM, jmh.direct wrote:
> Actually, I had presumed that the particular load balancing you 
> describe was not expected to work.

I can't imagine why this would not be expected to work.

However, it's nice to have one WG chair say "that is not expected to 
work" and another say "it's obvious how that works".  ;-)

> One of my concerns with the proposed BGP encoding is that it can 
> represent many things that will not work. This seems to make it fragile.

I guess I have a very different perspective.  As long as the protocol 
supports the SFC/NSH architecture, it's good if it also allows for 
generalizations of the architecture and for additional features.  That 
doesn't make it fragile, that makes it robust. "Fragile" is when every 
new feature requires a redesign of the protocol.

Another way to make a protocol fragile is to overload too many functions 
on any one field.  For example, imagine a protocol that had one 
eight-bit field that was used (a) as part of an address, (b) as a TTL, 
and (c) as a "I'm done with this packet" flag.  And suppose this field 
could also be increased in value (say, by a "reclassifier") as a packet 
traverses the network.  Now, that would be a fragile protocol ;-)

---

From: "Joel M. Halpern" <[email-masked]>
Date: Sat, 12 Nov 2016 06:22:38 -0500

There are at least two forms of confusion I can see here.

1) I was speaking about what I had observed as a participant.  I am 
sorry if it appeared I was speaking as chair.  The discussions in the WG 
occured before I was chair, and I did not attempt to review the archives 
as would be needed to make a chair assertion.

2) "this" actually covers a range of behaviors.

As a participant, and as an author on the SFC architecture, I understood 
the request for delegation to the data plane to cover a number of 
different cases.  However, I at least did not understand it to cover the 
case I see Eric asking for.

Let me describe what I understood, and what I think is being requested. 
It is clearly up to the WG if there is actual support for changing what 
we have3.

First, structurally, the SFF form an overlay connected by transport 
mechanisms.  Whether those are tunnels or not depends upon the 
particular transport.  Many of the solutions use tunnels.
Also, structurally, the mechanism for connecting service function 
instances to SFF is out of scope of the work, except that it has to 
preserve the NSH header.  (The SFC Proxy is for the case where the 
actual SF does not understand / preserve the NSH header.)

Given that, I believe we are discussing a situation where we have SFF-A 
and SFF-B. SFF-A has access to SF instance of two different functions, 
SF_1 and SF-2.  SFF-B also has access to SF instance of two different 
functions, one of them the same functionality SF-2, and one some other 
SF-3.  Whether the two SF-2s are actually the same instance, different 
instances, or two different front end load balancers for the same 
cluster is, for this discussion, immaterial.

One of the cases we were specifically asked to allow is that when an SFP 
needs to get to an instance of SF-2, it should be allowed than an 
earlier SFF could pick between SFF-A and SFF-B.

Another case that we I understood to be requested was when SF-2 needed 
to be reached after SF-1, that SFF-A was free to decide whether to use 
its local SF-2 instance, or to pass the packet to SFF-B to deliver to 
his instance.
As a side-effect of this, on a different SFP, one could have the 
situation where SF-2 needed to be reached after SF-3, and the packet 
went first to SFF-B and then SFF-B could decide whether to hand the 
packet to the local SF-2 or hand it to SFF-A to delvier to its SF-2.

What I heard Eric asking for was for the delegation to allow for a 
packet which needed to be delviered to SF-2 to be delivered by the 
preceding SFF to either SFF-A or SFF-B, and then for the SFF it got 
delivered to to decide whether to delvier it to a local SF-2 or to the 
other SFF.

I do not believe this case was ever suggested to the WG.  The logic 
described in teh SFC archtiecture and the NSH draft will not allow that. 
  In particular, the NSH document is very clear that the SFF can make 
its forwarding decision based solely on the SFP-ID and SFP Index.  As 
such, if the path and index allows A to hand the packet to B and allows 
B to hand the packet to A, we would have a major problem.
There has been significant discussion on the email list, particularly in 
the context of who modifies the SFP-Index, about the desire to have the 
NSH processing be independent of the incoming interface.  We also 
discussed this in the context of possible loops between SFF if there 
were situations in which SFF forwarded to each other without handing the 
packet to an SF.  As a participant trying to understand the logic, I 
thought we were clear that even when SFF considered additional (e.g. 
metadata) information, it did not consider the incoming port.

One of the problems we discussed with using incoming port to distingusih 
things is that depending upon what kinds of transport are used, there is 
not always port information.  There are sometimes separate interfaces. 
There are sometimes separate source addresses, and sometimes there is 
simply no information provided by the transport as to who sent the 
packet to the SFF.  Yes, we could as a working group declare that there 
must always be distinct sender information.  That would be a new 
requirement on the transports, and would complicate cases such as using 
MPLS as a transport that some members have indicated they consider 
important.

Yours,
Joel

---

From: Ron Parker <[email-masked]>
Date: Sat, 12 Nov 2016 12:47:31 +0000

I think that the view that CP communicates the forwarding table (I believe this view to be broadly held) is too limiting.  If, instead, it communicated forwarding policy/rules, the actual forwarding table would be constructed locally, accounting for any local load balancing decisions.   For SFPs that support some or all of their SFs to be load balanced in this manner, the SI as interpreted by CP now means a class of SFs and not a specific SF (in positions of the SFP where the load balancing is allowed in this manner). 
 
   Ron

> On Nov 12, 2016, at 6:22 AM, Joel M. Halpern <[email-masked]> wrote:
> 
> 1) I was speaking about what I had observed as a participant.  I am sorry if it appeared I was speaking as chair.  The discussions in the WG occured before I was chair, and I did not attempt to review the archives as would be needed to make a chair assertion.
> 
> As a participant, and as an author on the SFC architecture, I understood the request for delegation to the data plane to cover a number of different cases.  However, I at least did not understand it to cover the case I see Eric asking for.
> 
> Let me describe what I understood, and what I think is being requested. It is clearly up to the WG if there is actual support for changing what we have3.
> 
> First, structurally, the SFF form an overlay connected by transport mechanisms.  Whether those are tunnels or not depends upon the particular transport.  Many of the solutions use tunnels.
> Also, structurally, the mechanism for connecting service function instances to SFF is out of scope of the work, except that it has to preserve the NSH header.  (The SFC Proxy is for the case where the actual SF does not understand / preserve the NSH header.)
> 
> Given that, I believe we are discussing a situation where we have SFF-A and SFF-B. SFF-A has access to SF instance of two different functions, SF_1 and SF-2.  SFF-B also has access to SF instance of two different functions, one of them the same functionality SF-2, and one some other SF-3.  Whether the two SF-2s are actually the same instance, different instances, or two different front end load balancers for the same cluster is, for this discussion, immaterial.
> 
> One of the cases we were specifically asked to allow is that when an SFP needs to get to an instance of SF-2, it should be allowed than an earlier SFF could pick between SFF-A and SFF-B.
> 
> Another case that we I understood to be requested was when SF-2 needed to be reached after SF-1, that SFF-A was free to decide whether to use its local SF-2 instance, or to pass the packet to SFF-B to deliver to his instance.
> As a side-effect of this, on a different SFP, one could have the situation where SF-2 needed to be reached after SF-3, and the packet went first to SFF-B and then SFF-B could decide whether to hand the packet to the local SF-2 or hand it to SFF-A to delvier to its SF-2.
> 
> What I heard Eric asking for was for the delegation to allow for a packet which needed to be delviered to SF-2 to be delivered by the preceding SFF to either SFF-A or SFF-B, and then for the SFF it got delivered to to decide whether to delvier it to a local SF-2 or to the other SFF.
> 
> I do not believe this case was ever suggested to the WG.  The logic described in teh SFC archtiecture and the NSH draft will not allow that.  In particular, the NSH document is very clear that the SFF can make its forwarding decision based solely on the SFP-ID and SFP Index.  As such, if the path and index allows A to hand the packet to B and allows B to hand the packet to A, we would have a major problem.
> There has been significant discussion on the email list, particularly in the context of who modifies the SFP-Index, about the desire to have the NSH processing be independent of the incoming interface.  We also discussed this in the context of possible loops between SFF if there were situations in which SFF forwarded to each other without handing the packet to an SF.  As a participant trying to understand the logic, I thought we were clear that even when SFF considered additional (e.g. metadata) information, it did not consider the incoming port.
> 
> One of the problems we discussed with using incoming port to distingusih things is that depending upon what kinds of transport are used, there is not always port information.  There are sometimes separate interfaces. There are sometimes separate source addresses, and sometimes there is simply no information provided by the transport as to who sent the packet to the SFF.  Yes, we could as a working group declare that there must always be distinct sender information.  That would be a new requirement on the transports, and would complicate cases such as using MPLS as a transport that some members have indicated they consider important.
> 
> Yours,
> Joel
> 
> _______________________________________________
> sfc mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sfc

---

From: "Joel M. Halpern" <[email-masked]>
Date: Sat, 12 Nov 2016 19:23:30 -0500

Since we are not defining the control protocol, we do not need to decide 
whether control provides policy or SF information.  As I understand it 
from working on the drafts, we have agreed that even if control provides 
SF identification, it is allowed to provide a list of choices.  Which is 
why the three different load balancing cases I cited are, as far as I 
can tell, supported by the current definitions.

Yours,
Joel

---

From: Eric C Rosen <[email-masked]>
Date: Mon, 14 Nov 2016 15:26:58 -0500

On 11/12/2016 6:22 AM, Joel M. Halpern wrote:
> One of the problems we discussed with using incoming port to 
> distingusih things is that depending upon what kinds of transport are 
> used, there is not always port information.  There are sometimes 
> separate interfaces. There are sometimes separate source addresses, 
> and sometimes there is simply no information provided by the transport 
> as to who sent the packet to the SFF. Yes, we could as a working group 
> declare that there must always be distinct sender information.  That 
> would be a new requirement on the transports, and would complicate 
> cases such as using MPLS as a transport that some members have 
> indicated they consider important.

You don't need to know the incoming interface or the source address of 
the previous hop.  The only information that the transport needs to 
provide is an identifier of the transport connection.  (Or in the case 
of MPLS, an incoming MPLS label.)

All an SFF would need to know is that a particular packet arrived on a 
transport connection originating at another SFF.  In the case of MPLS 
this is particularly easy, as an SFF could advertise two labels: one 
that is only to be pushed on a packet by another SFF, and one that is 
only to be pushed on a packet by an SFI.

I don't really understand the difference between the load balancing use 
case I raised and the ones Joel has described.  If SFF1 sends a packet 
to SFF2, then SFF1 has to be responsible for making the load balancing 
decision, and SFF2 has to know that the load balancing decision has 
already been made.

---

From: Dave Dolson <[email-masked]>
Date: Mon, 14 Nov 2016 21:06:37 +0000

Eric,
As I understand your solution, it requires examining multiple protocol layers to make a next-hop decision. 
When the SF decrements the index, only the NSH layer needs to be examined to route packets to the next SF, at least in simple deployments.

In more complex deployments, lower layers may indicate a domain and higher layers might be used for load-balancing selection.

But from what you are saying, I recommend keeping the SFs at a single NSH-hop away from the load-balancer. Don't let an intermediate SFF second-guess the original decision.

-Dave

-----Original Message-----
From: Eric C Rosen [mailto:[email-masked]] 
Sent: Tuesday, November 15, 2016 5:27 AM
To: Joel M. Halpern; Dave Dolson; [email-masked]; [email-masked]
Subject: Re: [sfc] decrementing service function path index

> indicated they consider important.

You don't need to know the incoming interface or the source address of the previous hop.  The only information that the transport needs to provide is an identifier of the transport connection.  (Or in the case of MPLS, an incoming MPLS label.)

All an SFF would need to know is that a particular packet arrived on a transport connection originating at another SFF.  In the case of MPLS this is particularly easy, as an SFF could advertise two labels: one that is only to be pushed on a packet by another SFF, and one that is only to be pushed on a packet by an SFI.

I don't really understand the difference between the load balancing use case I raised and the ones Joel has described.  If SFF1 sends a packet to SFF2, then SFF1 has to be responsible for making the load balancing decision, and SFF2 has to know that the load balancing decision has already been made.