Subject: Re: [bess] draft-rosen-mpls-rfc3107bis

From: Eric C Rosen <[email-masked]>
Date: Fri, 1 Apr 2016 12:44:22 -0400

On 3/25/2016 7:25 AM, [email-masked] wrote:
>> I'm quite sure you have deployed  implementations, from several
>> prominent vendors, that will not properly handle this case.
> I'm waiting for this/these implementation(s) to make a public statement in this thread / IETF WGs. Then we can discuss whether the issue comes from RFCF3107 or from the implementation.
> If none make a public statement, we should assume that all implementations are capable of receiving multiple labels, as per RFC 3107.
I strongly disagree with this.  We should not ignore the facts just 
because you don't like the way the facts were gathered.

A better approach would be to have operators state whether they have any 
deployments in which the "multiple labels" feature is used in a 
multi-vendor environment.  It is very useful when working on a "bis" 
draft to determine which features have been proven to work in a 
multi-vendor environment and which have not.

> Any non-compliant implementation may create interoperability issues and unpredictable results.
>  From an IETF standpoint, the question is whether a RFC 3107 implementation would create interoperability issues, up to shutting down the BGP session.

There are deployed 3107 implementations which always assume that the 
NLRI contains a single label.  If you tried to interwork these with 3107 
implementations that send multiple labels , you will experience the kind 
of disruption.  3107bis tries to allow the use of multiple labels while 
preventing this sort of disruption from occurring.

> If you mean that some non-compliant implementation do not work, well let's fix them.

The situation is that there is a commonly deployed "bug" in old 
implementations, but it is not seen because the bug is in a feature that 
no one has been using.  If new implementations use that feature, the bug 
will be seen, and network disruption will occur. One could say "fix all 
the old implementations", but it seems wiser to have new implementations 
avoid tickling the bug.   The Capability is not proposed  for the 
purpose of helping the vendors, it's there to help the operators.

I'm not sure why you think there would be BGP session drops due to 
3107bis; if a 3107 implementation sends multiple labels to a 3107bis 
implementation, I think the 3107bis implementation would do 
"treat-as-withdraw" rather than "drop the session".

Perhaps a reasonable approach for 3107bis would be the following:

- A 3107bis implementation will not send multiple labels to a peer 
unless the Capability has been received from that peer.  (This prevents 
3107bis implementations from tickling the 'bug' in 3107 implementations.)

- A 3107bis implementation will accept multiple labels from a peer even 
in the absence of the Capability.

Another approach would be to have a knob that determines whether the 
Capability needs to be used before multiple labels are advertised.

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 1 Apr 2016 22:23:41 +0200

Hi Eric,

I have read your proposed draft as well as watched this thread with a bit
of an interest.

To me the best compromise - which is to agree with Bruno's points as well
as address your intentions is simply to request new SAFI for 3107bis.

>From the draft you are really not updating 3107 base spec but obsoleting it
which to me looks like a bad idea.

You are even requesting to remove IANA reference to original spec. How
would IANA know when is it safe to do that .. meaning when all
implementations will not suddenly support and all deployments will enable
3107bis ?

New SAFI requires a new capability which you are asking for anyway.

As far as implementations please keep in mind very important point that
some implementations treat SAFI 1 & 4 in single table and some in separate
tables. That when mixed with 3107bis may just explode if not in new set of
bugs then with operational nightmare. While we are at this it would be much
cleaner to mandate in the new spec to have 3107bis always to use separate
tables as compared with from SAFI 1.

Thx,
Robert.

PS.

As we all know 3107(bis) tries to add NNI to MPLS. However it must be very
well stated that this is only one deployment option for interdomain
encapsulation. I would very much like to see a section indicating that IPv6
or/and IPv4 be used as an alternative encap for those applications which
require it and when needed provide local bindings between intradomain MPLS
and interdomain IP.

On Fri, Apr 1, 2016 at 6:44 PM, Eric C Rosen <[email-masked]> wrote:

>> I'm waiting for this/these implementation(s) to make a public statement
>> in this thread / IETF WGs. Then we can discuss whether the issue comes from
>> RFCF3107 or from the implementation.
>> If none make a public statement, we should assume that all
>> implementations are capable of receiving multiple labels, as per RFC 3107.
>>
> multi-vendor environment.  It is very useful when working on a "bis" draft
> to determine which features have been proven to work in a multi-vendor
> environment and which have not.
>
>>  From an IETF standpoint, the question is whether a RFC 3107
>> implementation would create interoperability issues, up to shutting down
>> the BGP session.
>>
>
> There are deployed 3107 implementations which always assume that the NLRI
> contains a single label.  If you tried to interwork these with 3107
> implementations that send multiple labels , you will experience the kind of
> disruption.  3107bis tries to allow the use of multiple labels while
> implementations, but it is not seen because the bug is in a feature that no
> one has been using.  If new implementations use that feature, the bug will
> be seen, and network disruption will occur. One could say "fix all the old
> implementations", but it seems wiser to have new implementations avoid
> tickling the bug.   The Capability is not proposed  for the purpose of
> helping the vendors, it's there to help the operators.
>
> - A 3107bis implementation will not send multiple labels to a peer unless
> the Capability has been received from that peer.  (This prevents 3107bis
> implementations from tickling the 'bug' in 3107 implementations.)
>
> - A 3107bis implementation will accept multiple labels from a peer even in
> the absence of the Capability.
>
> _______________________________________________
> BESS mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/bess
>

---

From: "Acee Lindem (acee)" <[email-masked]>
Date: Fri, 1 Apr 2016 21:29:10 +0000

Hi Robert,

I think this would defeat the purpose of clarifying RFC 3101 multi-label behavior in a BIS draft. Let’s see if we can reach consensus first.

Thanks,
Acee

From: Idr <[email-masked]<mailto:[email-masked]>> on behalf of Robert Raszuk <[email-masked]<mailto:[email-masked]>>
Date: Friday, April 1, 2016 at 4:23 PM
To: Eric C Rosen <[email-masked]<mailto:[email-masked]>>
Cc: Bruno Decraene <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, BESS <[email-masked]<mailto:[email-masked]>>, IDR List <[email-masked]<mailto:[email-masked]>>
Subject: Re: [Idr] [bess] draft-rosen-mpls-rfc3107bis

Hi Eric,

I have read your proposed draft as well as watched this thread with a bit of an interest.

To me the best compromise - which is to agree with Bruno's points as well as address your intentions is simply to request new SAFI for 3107bis.

From the draft you are really not updating 3107 base spec but obsoleting it which to me looks like a bad idea.

You are even requesting to remove IANA reference to original spec. How would IANA know when is it safe to do that .. meaning when all implementations will not suddenly support and all deployments will enable 3107bis ?

As far as implementations please keep in mind very important point that some implementations treat SAFI 1 & 4 in single table and some in separate tables. That when mixed with 3107bis may just explode if not in new set of bugs then with operational nightmare. While we are at this it would be much cleaner to mandate in the new spec to have 3107bis always to use separate tables as compared with from SAFI 1.

Thx,
Robert.

PS.

As we all know 3107(bis) tries to add NNI to MPLS. However it must be very well stated that this is only one deployment option for interdomain encapsulation. I would very much like to see a section indicating that IPv6 or/and IPv4 be used as an alternative encap for those applications which require it and when needed provide local bindings between intradomain MPLS and interdomain IP.

On Fri, Apr 1, 2016 at 6:44 PM, Eric C Rosen <[email-masked]<mailto:[email-masked]>> wrote:
On 3/25/2016 7:25 AM, [email-masked]<mailto:[email-masked]> wrote:
I'm quite sure you have deployed  implementations, from several
prominent vendors, that will not properly handle this case.
I'm waiting for this/these implementation(s) to make a public statement in this thread / IETF WGs. Then we can discuss whether the issue comes from RFCF3107 or from the implementation.
If none make a public statement, we should assume that all implementations are capable of receiving multiple labels, as per RFC 3107.
I strongly disagree with this.  We should not ignore the facts just because you don't like the way the facts were gathered.

A better approach would be to have operators state whether they have any deployments in which the "multiple labels" feature is used in a multi-vendor environment.  It is very useful when working on a "bis" draft to determine which features have been proven to work in a multi-vendor environment and which have not.

There are deployed 3107 implementations which always assume that the NLRI contains a single label.  If you tried to interwork these with 3107 implementations that send multiple labels , you will experience the kind of disruption.  3107bis tries to allow the use of multiple labels while preventing this sort of disruption from occurring.

The situation is that there is a commonly deployed "bug" in old implementations, but it is not seen because the bug is in a feature that no one has been using.  If new implementations use that feature, the bug will be seen, and network disruption will occur. One could say "fix all the old implementations", but it seems wiser to have new implementations avoid tickling the bug.   The Capability is not proposed  for the purpose of helping the vendors, it's there to help the operators.

I'm not sure why you think there would be BGP session drops due to 3107bis; if a 3107 implementation sends multiple labels to a 3107bis implementation, I think the 3107bis implementation would do "treat-as-withdraw" rather than "drop the session".

- A 3107bis implementation will not send multiple labels to a peer unless the Capability has been received from that peer.  (This prevents 3107bis implementations from tickling the 'bug' in 3107 implementations.)

- A 3107bis implementation will accept multiple labels from a peer even in the absence of the Capability.

Another approach would be to have a knob that determines whether the Capability needs to be used before multiple labels are advertised.

_______________________________________________
BESS mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/bess

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 1 Apr 2016 23:35:33 +0200

Hi AC,

I am much less concern if we must be stuck to 3107 till retirement.

I think it would be much smoother on many levels to leave 3107 as is and
propose better solution for interdomain label exchange with BGP in new RFC.

With time we can obsolete 3107.

Such model has been done in the past and worked pretty well AFAIK.

Best,
r.

On Fri, Apr 1, 2016 at 11:29 PM, Acee Lindem (acee) <[email-masked]> wrote:

> Hi Robert,
>
> I think this would defeat the purpose of clarifying RFC 3101 multi-label
> behavior in a BIS draft. Let’s see if we can reach consensus first.
>
> Thanks,
> Acee
>
> From: Idr <[email-masked]> on behalf of Robert Raszuk <
> [email-masked]>
> To: Eric C Rosen <[email-masked]>
> Cc: Bruno Decraene <[email-masked]>, "[email-masked]" <
> [email-masked]>, BESS <[email-masked]>, IDR List <[email-masked]>

---

From: Eric C Rosen <[email-masked]>
Date: Fri, 1 Apr 2016 17:45:16 -0400

On 4/1/2016 4:23 PM, Robert Raszuk wrote:
> To me the best compromise - which is to agree with Bruno's points as 
> well as address your intentions is simply to request new SAFI for 
> 3107bis.

I don't think that makes any sense at all.  The whole point is to ensure 
that 3107bis interoperates with 3107 for those features that are already 
deployed and are already multi-vendor interoperable.

>
> From the draft you are really not updating 3107 base spec but 
> obsoleting it which to me looks like a bad idea.

That is the nature of a "bis" draft.

>
> You are even requesting to remove IANA reference to original spec.

That is the nature of a "bis" draft.

> How would IANA know when is it safe to do that .. meaning when all 
> implementations will not suddenly support and all deployments will 
> enable 3107bis ?

I don't understand the issue you are raising.  I don't see any issue of 
"safety".

> New SAFI requires a new capability which you are asking for anyway.

I don't understand the point you are making here.

>
> As far as implementations please keep in mind very important point 
> that some implementations treat SAFI 1 & 4 in single table and some in 
> separate tables.

Yes, and these implementation differences have consequences that are 
discussed in the draft.

> That when mixed with 3107bis may just explode if not in new set of 
> bugs then with operational nightmare.

I don't understand the issue you are raising here.

> While we are at this it would be much cleaner to mandate in the new 
> spec to have 3107bis always to use separate tables as compared with 
> from SAFI 1.

Two goals of this draft are (a) to document the consequences of the 
implementation differences, and (b) to avoid invalidating any particular 
implementation.  Obviously these goals would not be met if the spec 
mandated a particular implementation method.

> As we all know 3107(bis) tries to add NNI to MPLS. However it must be 
> very well stated that this is only one deployment option for 
> interdomain encapsulation. I would very much like to see a section 
> indicating that IPv6 or/and IPv4 be used as an alternative encap for 
> those applications which require it and when needed provide local 
> bindings between intradomain MPLS and interdomain IP.

This is entirely out of scope.

---

From: <[email-masked]>
Date: Mon, 4 Apr 2016 21:28:37 +0000

> From: Eric C Rosen [mailto:[email-masked]]
> Sent: Friday, April 01, 2016 1:44 PM
> 
> in this thread / IETF WGs. Then we can discuss whether the issue comes
> from RFCF3107 or from the implementation.

Asking operators or vendors is equally fine for me.
My issue is how do we prove that _nobody_ is using it? Proving the negative is hard, and silence is not part of the proof. To prove the negative, we would need explicit statement from everyone, which looks impossible.
 
> of disruption. 

Agreed. But between 2 3107 speakers, we can determine which implementation has a bug. Whereas between a 3107 speaker and 3107bis speaker, both implementations may be compliant, but still the BGP session would go done.

> 3107bis tries to allow the use of multiple labels while

Agreed. I support this.
 
> purpose of helping the vendors, it's there to help the operators.

I support the capability.
 
> "treat-as-withdraw" rather than "drop the session".

"treat-as-withdraw" would be fine for me. But this requires the 3107bis implementation to be able to parse the stack of (multiple) labels, in order to identify the IP prefix and treat it as withdraw. However, by hypothesis, this speaker is not capable of receiving multiple labels (in short skipping labels until it founds the S bit set)

> 3107bis implementations from tickling the 'bug' in 3107 implementations.)

Good for me.

> in the absence of the Capability.

Good for me.
Note that I'm not asking for this route to be installed: I'm fine with treat-as-withdraw. But this does imply that the 3107bis speaker be always capable of parsing a stack of multiple labels, in order to skip the MPLS stack, identify the IP prefix, and treat it as withdraw.

Your approach seems along the line of my original email where I was proposing the following change:
"- Even if the capability is not advertised by both peers, and hence a single label is expected, all implementations MUST check that the "S" bit (in this first label) is set to 1. If the bit is cleared, the Prefix MUST be identified as per RFC 3107/this document and treated as withdraw as defined in RFC 7606."
https://mailarchive.ietf.org/arch/msg/mpls/XNhUSSfwTpnOjNZGlpc7O2j3G1g

> Capability needs to be used before multiple labels are advertised.

I'm may be missing what you mean, but that alternative approach does not seem to address my concern.

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 5 Apr 2016 08:05:04 -0400

Eric,

If as it turns out if the primary motivation for 3107bis is to distribute
label stack for segment routing that I do not think per destination prefix
is a sufficient granularity.

How with 3107(bis) you can match on the src of the packets ?

How you can match on the more granular information to steer packets
differently depending on the application ?

Wouldn't it be more flexible to simply define new attribute to carry the
stack within any SAFI as an opaque to BGP data ? That way one can easily
use it in unicast SAFI or even in FlowSpec.

Thx,
R.

---

From: Eric C Rosen <[email-masked]>
Date: Tue, 5 Apr 2016 09:28:42 -0400

On 4/5/2016 8:05 AM, Robert Raszuk wrote:
> Wouldn't it be more flexible to simply define new attribute to carry  > the stack within any SAFI as an opaque to BGP data ? That way one 
can > easily use it in unicast SAFI or even in FlowSpec.

That can be done with the Tunnel Encapsulation attribute.  Look at, for 
example, at draft-previdi-idr-segment-routing-te-policy (though I think 
that document still needs quite a bit of work).

> If as it turns out if the primary motivation for 3107bis is to  > distribute label stack for segment routing

3107bis fixes a number of problems in 3107, and it also specifies how to 
use the "multiple labels" feature without tickling known bugs in 
existing deployments.   One could argue about whether this is the best 
tool for various segment routing applications, but that's outside the 
scope of the draft.  And there are use cases that have nothing to do 
with segment routing; see, e.g., the discussion of context labels in 
section 4.

---

From: Eric C Rosen <[email-masked]>
Date: Tue, 5 Apr 2016 10:25:48 -0400

On 4/4/2016 5:28 PM, [email-masked] wrote:
> My issue is how do we prove that_nobody_  is using it? Proving the negative is hard, and silence is not part of the proof. To prove the negative, we would need explicit statement from everyone, which looks impossible.

I think you're exaggerating a little ;-)  I'm having trouble believing 
that you are unable to determine whether the 3107 "multiple labels" 
feature is in use in any of your company's deployments.

I think a more serious concern is how to avoid tickling the "day 1" bugs 
that may exist.  There may be implementations, supporting only a single 
label, that fail to set the S bit.  There may be implementations, 
supporting only a single label, that fail to check the S bit.  These 
"day 1" bugs will never have caused a problem, because the multiple 
labels feature has not been used.  We should endeavor to ensure that 
these"day 1" bugs do not cause a problem in the future if newer 
implementations begin to use the multiple labels feature.

Right now, we're arguing about which of the following two strategies is 
more likely to cause a problem:

1. In the first strategy, 3107bis requires the S bit to be set when 
there is a single label.  This will allow 3107bis to interoperate with 
3107 implementations of "multiple labels", but it will not allow 3107bis 
to interoperate with (buggy) 3107 implementations that send a single 
label, but don't set the S bit.

2. In the second strategy, 3107bis assumes, in the absence of the 
Capability, that there is only a single label, and doesn't bother to 
check the S bit.  This will allow 3107bis to interoperate with (buggy) 
implementations that send a single label but fail to set the S bit; it 
will not allow 3107bis to interoperate with (non-buggy) 3107 
implementations of multiple labels.

My argument is that the second strategy is better because it will be 
less disruptive.  This is based on my belief that the "day 1" bugs do 
exist, and that the "multiple labels" feature has yet to be deployed.

Your argument seems to be that the first strategy is better because (a)  
it only causes disruption if the 3107 implementation has a bug, in which 
case the bug can be fixed, and (b) if the second strategy is used, a 
3107-compliant implementation of multiple labels will fail to 
interoperate with a 3107bis-compliant implementation of multiple labels, 
and both implementors will claim compliance.

I think your argument is reasonable, the question is really just which 
strategy will cause less disruption.

Do other members of the WGs have opinions about this?

---

From: <[email-masked]>
Date: Wed, 13 Apr 2016 11:32:32 +0000

Please see 1 comment inline [Bruno]

[Bruno] Indeed, we had a different perspective.

On my side, I was assuming that the IETF would not introduce a non-backward compatible specification change, without first checking that none of the deployment would be affected. My point was that this is hard to prove.

On your side, you were assuming that the responsibility would be on each network operator to check whether they would have an issue. This is breaking the big problem into N smaller ones.

--Bruno

---

From: <[email-masked]>
Date: Wed, 13 Apr 2016 11:32:36 +0000

> the "day 1" bugs do exist

Do we really have implementation not setting the S bit on the sending side??
I don't see how this could be per design, as I fail to see a reason:
- sending with S=1 or S=0 has the same cost from an implementation perspective, so this is not a "simplification" issue
- sending with S=0 is a clear violation of RFC 3107 and will trigger a BGP session shutdown from a peer compliant with 3107, so this does not seem like a desirable goal from an implementation standpoint.

Now, that could be a bug, but that bug would be detected with the first interoperability test with a compliant RFC 3107 implementation...

So, how have we got into this situation? How can we be there 15 years after the publication of RFC 3107?

Thanks
-- Bruno

---

From: <[email-masked]>
Date: Wed, 13 Apr 2016 11:32:24 +0000

Sorry for the late answer and for breaking the email thread. (my laptop hard drive failed during the IETF week).

Please see 2 comments inline [Bruno]

And as already expressed, I support the adoption of this document.

[Bruno] Thank you for this good and fair summary.

I would add: (c) a 3107-compliant implementation of multiple labels sending one route with multiple labels to a 3107bis-compliant implementation of a single label, will trigger the shutdown of the BGP sessions and hence the removal of all labelled routes and most likely all routes from all AFI/SAFI sharing this BGP session. This is very disruptive. And both implementors will claim compliance.

There may be a third strategy: second strategy, plus when a BGP error is found when parsing the NLRI, search for the S bit in order to identify the IP prefix and treat it as withdraw. (rather than kill the BGP session)

To be on the extra safe side, I'd prefer the third strategy, on the basis that you never know what you can receive (cf the BGP attribute 99 incident, which nobody asked for/expected), plus the 3107 speaker sending multiple labels IS compliant.

If you don't think that this is feasible/reasonable, I'm fine with the second strategy.

[Bruno] In either case, it seems like the draft will need to document the issue and warn the network operator about it.