Subject: [OPSEC] Alvaro Retana's Discuss on draft-ietf-opsec-urpf-improvements-03: (with DISCUSS and COMMENT)

From: Alvaro Retana via Datatracker <[email-masked]>
Date: Fri, 16 Aug 2019 15:01:44 -0700

Alvaro Retana has entered the following ballot position for
draft-ietf-opsec-urpf-improvements-03: Discuss

When responding, please keep the subject line intact and reply to all
email addresses included in the To and CC lines. (Feel free to cut this
introductory paragraph, however.)

Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
for more information about IESG DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-opsec-urpf-improvements/

----------------------------------------------------------------------
DISCUSS:
----------------------------------------------------------------------

I have significant concerns about the recommendation (in §3.7) to use Algorithm
A.  I think that the considerations to select an Algorithm are not clear and
potentially impossible to verify.  Also, the selection of Algorithm A creates a
vulnerability that could result in legitimate traffic dropped.

§3.7 (Summary of Recommendations) says that "if the scenario does not involve
complexity...(see Section 3.3, Figure 4), then...Algorithm A...SHOULD be
applied on customer interfaces."

>From Figure 4, how does the operator of ISP4 know that it is not in a
"challenging scenario" (§3.3)?  How can ISP4 tell the difference between ISP2
not propagating routes vs it not even being connected to AS1?  By definition,
each autonomous system can have its own set of policies, which may not be known
by any of their upstreams.  In short, I find the guidance provided in the
document to select an algorithm not clear enough to really be actionable --
specially in a document tagged to be a BCP.

Still looking at Figure 4, if the operator of ISP4 uses Algorithm A (because
he/she thinks they may not be in a "challenging scenario"), then it opens the
door to ISP2 changing its policy so that the uRPF check in ISP4 fails, as shown
in Figure 4.  IOW, an attacker in ISP2 could stop advertising reachability to
some prefixes and cause ISP4 to fail the uRPF check and drop the traffic.  The
victim in this case could be AS1, whose traffic (through ISP2 to ISP4) was
flowing fine and then it stopped.

I am balloting DISCUSS because the guidance provided for the selection of an
Algorithm is not clear enough to be certain that the selection is the correct
one; and the election of Algorithm A creates a vulnerability (as explained in
the text) that is not mentioned.

----------------------------------------------------------------------
COMMENT:
----------------------------------------------------------------------

(1) The implementation of the EFP-uRPF method is expected at a transit AS. 
However, that AS has no control over what the origin AS and others announce.  I
find the use of rfc2119 keywords in §3.2 inappropriate because none of the
actions are under the control of the AS implementing uRPF and can't be
Normatively enforced.

(2) §3.5: "Prefixes from registered ROAs and IRR route objects that include
ASes in an ISP's customer cone SHOULD be used to augment the appropriate RPF
lists."  How?  Note that the Introduction already sets the stage by saying that
"the routes under consideration are assumed to have been vetted based on prefix
filtering [RFC7454] and possibly (in the future) origin validation [RFC6811]." 
If ROAs SHOULD be used (§3.5), why is origin validation only "possibly (in the
future)" considered (§1)?  Maybe a better statement would be that "routes
SHOULD be validated using prefix filtering, origin validation and IRR route
objects".

(3) How does EFP-uRPF work when multiple border routers are present in the AS,
some customer facing and some not (which I assume to be the normal case)?  I'm
asking because §3.1 describes the method when "prefixes with the same origin AS
were received on different interfaces (at border routers)", and the example
talks about the Local Adj-RIB-Ins.  I think there's a missing explanation of
how the routers with the customer-facing interfaces learn about *all* the
received routes if the local policy might select a single BGP best route. 
OTOH, maybe I'm missing something.

(4) The document assumes familiarity with terminology such as "customer cone"
or "lateral peer", that is not explained anywhere.  The average operator will
most likely know what those terms mean and how they apply to their network.
However, those operators are not the only people reviewing this document.  A
short explanation or reference would be nice.

(5) Please use the rfc8174 template.

(6) [For the AD/Shepherd.] I'm assuming that the intent is for this document to
be part of BCP 84, is that correct?  I'm not sure how to let the RFC Editor
know that is the intent, but it should be made clear somewhere.

---

From: "Sriram, Kotikalapudi (Fed)" <[email-masked]>
Date: Sun, 18 Aug 2019 21:48:02 +0000

Alvaro,

Thanks very much for your careful review and comments.
Please see authors’ responses inline below.

You are asking how does an ISP or AS operator know
that they have a scenario in which Algorithm A is applicable. 
Operators have communications at least with 
their immediate downstream customers. So, consider this 
scenario: Operator of AS4 knows (says), 
“I have only two direct customers AS2 (ISP2) and AS3 (ISP3).
Each of them has assured me that they have customers that
are all stub ASes and that these customers do not currently use 
NO_EXPORT on their announced routes.
My direct customers (ISP2 and ISP3) have mechanisms by which 
they would notify me if this ever changes.
They further assure me that they would never 
deliberately drop their customer routes.”
This gives the operator of AS4 the knowledge/confidence 
that Algorithm A is applicable at their AS.

It should be emphasized that Algorithm A works 
in scenarios like the above (customer cone depth is limited 
to two tiers below the ISP) provided each stub customer 
does not attach NO_EXPORT on *at least* one prefix announcement
out of however many prefixes they announce.
(We’ll add this observation in the draft.)

There can be many realistic scenarios like the above that exist
at the edges of the Internet. But just one is enough to
justify the inclusion of Algorithm A in the recommendations (Sec. 3.7).
(Note: uRPF is expected to work best at the edges of the Internet.) 

Further, the feasible path uRPF (FP-uRPF) is already a BCP (part of BCP 84).
If an operator deems their scenario fit for use of FP-uRPF,
then the EFP-uRPF with Algorithm A is also applicable in that scenario and 
performs equally or better (i.e., lower false positive rate for SA invalid) than FP-uRPF. 
Of course, the EFP-uRPF works better under less restrictive 
conditions than required for FP-uRPF as explained in the draft.  

Of course, according to Section 3.7, if an ISP is uncertain that
Algorithm A is applicable in their scenario, then they SHOULD apply Algorithm B.
We'll be happy to reword the section more carefully if you feel
that would help. You may advise us on any specific wording 
you would like to see added.

>flowing fine and then it stopped.

There are two observations that are in order here:
1. An ISP would not normally drop its own paying customer’s 
routes with malicious intent.
2. It is also hard to envision that ISP2 would attack its upstream ISP4
(in Figure 4) just to disrupt its SAV (uRPF), while that results in 
absolutely no gain but only causes unreachability for 
its (ISP2’s) own customer and self-inflicted loss of revenue/customers.    
  
>However, that AS has no control over what the origin AS and others announce.  

Please see responses above.

>I find the use of rfc2119 keywords in §3.2 inappropriate because none of the

Yes, agree. The use of rfc2119 keywords in not necessary in Section 3.2.
The recommendations here are to facilitate feasibility of uRPF
and are not about actions performed by the AS implementing EFP-uRPF.  

>SHOULD be validated using prefix filtering, origin validation and IRR route

We agree with your suggested rewording in Section 1. We’ll use that.
What is advised in Section 3.5 regarding ROAs is as follows.
In Figure 3, suppose that prefix P4 a ROA with with AS1, 
and AS4 (ISP4) may not have received a route for P4 (on a given customer interface).
But AS4 (ISP4) has received (on that customer interface) a route for P1 with 
the same origin AS (AS1) as in the ROA for P4.
Then AS4 (ISP4) should augment its corresponding RPF list with P4
to allow the possibility that AS1 could legitimately announce 
a route for P4 and potentially use a SA (in a data packet) belonging in P4.  
We'll add this explanation in Section 3.5.
  
>OTOH, maybe I'm missing something.

Yes, we perhaps need to explain this better in the document.
It is not that “customer-facing interfaces learn about *all* the
received routes” but simply that customer-facing routers 
must learn at least one route (via iBGP/eBGP) for each prefix 
originated by an AS (in its customer cone).”  This is normal in BGP.
(There is a little more to it; but that is already well explained 
in the draft so I’ll not repeat here.)  
Considering AS4 (ISP4) in Figure 3, the customer-facing router 
that is facing AS2 learns at least one route each for P2, P3 via iBGP.
That is how it includes P2 and P3 in the RPF list (in addition to P1)
for the customer-interface towards AS2 (in Algorithm A).  

>short explanation or reference would be nice.

Will do. Will provide explanation or reference or both.
 
>(5) Please use the rfc8174 template.

Yes, this is already included in my author’s draft for the next version.

>know that is the intent, but it should be made clear somewhere.

I am not sure that “to be part of BCP 84” is the intent.
The intent is that this RFC/BCP (to be published) will be 
a new BCP that updates BCP 84 (RFC  3704). 
It proposes to recommend the new EFP-uRPF which is 
an improvement over the FP-uRPF in RFC 3704. 
We’ll state that explicitly in the abstract as well as the introduction.
The draft header already states – “Updates: RFC3704 (if approved).”

Thank you once again for the detailed review and comments.

Sriram

---

From: Alvaro Retana <[email-masked]>
Date: Mon, 19 Aug 2019 15:26:55 -0400

On August 18, 2019 at 5:48:04 PM, Sriram, Kotikalapudi (Fed) (
[email-masked]) wrote:

Sriram:

Hi!

Precisely the fact that transitive assurances can’t be trusted about
networks being well behaved is that mechanisms like ingress filtering,
uRPF, RPKI/Origin Validation, BGPSec, etc…have been developed.  I’m sorry,
but it sounds ingenuous for the guidance to be to trust your neighbors…

Quoting from rfc3704: “...essentially trusts the downstream network to
behave itself, which is not the wisest course of action."

performs equally or better (i.e., lower false positive rate for SA invalid)
than FP-uRPF.

I couldn’t find much in terms of guidance in rfc3704 about when using
FP-uRPF is recommended, except for this: "Feasible Path RPF…is suitable in
all the scenarios where Strict RPF is, but multihomed or asymmetric
scenarios in particular.  However, one must remember that Feasible RPF
assumes the consistent origination and propagation of routing information
to work; the implications of this must be understood especially if a prefix
advertisement passes through third parties.”

I think that the guidance in rfc3704 is about the same as the guidance in
this document, plus the fact that some of the potential issues with
trusting other networks are at least mentioned there.  However, I don’t
think that there’s enough guidance in rfc3704 for this document to simply
point at it — the same questions would apply: how would an operator know
that "consistent origination and propagation of routing information” is
present?

Of course, according to Section 3.7, if an ISP is uncertain that
Algorithm A is applicable in their scenario, then they SHOULD apply
Algorithm B.

True, but that still doesn’t provide any tools for the ISP to know either
way.  If there’s no way to know (other than trusting your neighbors) that
the network is not in the middle of a “challenging scenario”, then why
wouldn't Algorithm B always be used?

I also doubt that an operator would take actions to harm its own paying
customers — but the routers can be compromised anyway...or the
advertisement can come from the customer itself with a NO_EXPORT community,
etc..  The main case I’m thinking of is the compromised/rogue router.    In
any of those cases, choosing Algorithm A opens the door to
vulnerabilities.  Yes, they might be the same (or at least similar) as what
exists with FP-uRPF — but they are not mentioned.

>———————————————————————————————————

...

and AS4 (ISP4) may not have received a route for P4 (on a given customer
interface).

Hmm… Except that the “possibility” that an announcement exists *and* the
use as a SA, is different from AS1 really doing it.  IOW, augmenting with
the expectation that it may happen has its own risks…

...

Updating rfc3704 and being part of the same BCP are separate
considerations.  To me, the fact that this document Updates rfc3704 makes
it a good candidate to be part of the same BCP; a BCP can contain more than
one RFC.

A good example is BCP 14, which includes rfc2119 and and the more recent
rfc8174.

Thanks!

Alvaro.

---

From: Alvaro Retana <[email-masked]>
Date: Mon, 19 Aug 2019 17:18:02 -0400

Sriram:

Hi!

...

...

Thanks!

Alvaro.

---

From: "Sriram, Kotikalapudi (Fed)" <[email-masked]>
Date: Tue, 20 Aug 2019 14:29:21 +0000

Alvaro,

Responses below inline.

>Precisely the fact that transitive assurances can’t be trusted about networks being well behaved is that mechanisms like ingress filtering, uRPF, RPKI/Origin Validation, BGPSec, etc…have been developed.  I’m sorry, but it sounds ingenuous for the guidance to be to trust your neighbors…

First, I notice that BCP 84 does not use RFC2119/RFC 8174 language. 
So, I think this document can also avoid using upper case SHOULD etc.
Having noted that, I feel there is a difference between saying, 
“ISP should trust their neighbor and should use Algorithm A” 
versus saying, “if an ISP has knowledge (by whatever mechanism they choose) 
that the scenario they are in is amenable to Algorithm A, 
then they may consider using it in their AS.” The draft clearly intends the latter.

I think we shouldn’t dismiss the possibility some ISPs may have the means
to rule out the NO_EXPORT complexity. 
The document would not specify how the determination is made; 
would not even say that it is required.
However, if an ISP finds that they have rapport with their direct customers
or that they have access to their RIB-in feeds or whatever,
and thereby able to determine that Algorithm A can be used, that is their decision.
(I am not saying that the immediate above sentence needs to be stated in the draft.)
In most cases, an ISP will likely find itself using Algorithm B.
How about the following change in the draft in Section 3.7?

OLD text:
   3.  For all other scenarios:
       *  If the scenario does not involve complexity such as NO_EXPORT
          of routes (see Section 3.3, Figure 4), then the enhanced
          feasible-path uRPF method in Algorithm A (see Section 3.1.1)
          SHOULD be applied on customer interfaces.
       *  Else, if the scenario involves the complexity then the
          enhanced feasible-path uRPF method in Algorithm B (see
          Section 3.4) SHOULD be applied on customer interfaces.

NEW text:
   3.  For all other scenarios:
       *  If the ISP has some means to ascertain that the scenario does 
          not involve complexity such as NO_EXPORT
          of routes (see Section 3.3, Figure 4), then the enhanced
          feasible-path uRPF method in Algorithm A (see Section 3.1.1)
          may be applied on customer interfaces.
       *  If there is a possibility that the scenario may involve the complexity,
          or if the ISP simply wishes to avoid any risk associated with Algorithm A
          (see Section 3.3), then the enhanced feasible-path uRPF method 
          in Algorithm B (see Section 3.4) should be applied on customer interfaces.

….
 
>I couldn’t find much in terms of guidance in rfc3704 about when using FP-uRPF is recommended, except for this: "Feasible Path RPF…is suitable in all the scenarios where Strict RPF is, but multihomed or asymmetric scenarios in particular.  However, one must remember that Feasible RPF assumes the consistent origination and propagation of routing information to work; the implications of this must be understood especially if a prefix advertisement passes through third parties.”  

>I think that the guidance in rfc3704 is about the same as the guidance in this document, plus the fact that some of the potential issues with trusting other networks are at least mentioned there.  However, I don’t think that there’s enough guidance in rfc3704 for this document to simply point at it — the same questions would apply: how would an operator know that "consistent origination and propagation of routing information” is present?

As the draft notes, EFP-uRPF (Algorithm A) is more tolerant than FP-uRPF 
regarding "consistent origination and propagation of routing information”. 
In fact, ASes not using NO_EXPORT on *all* originated prefixes 
paves the way for use of Algorithm A. 
Of course, Algorithm B is highly flexible/accommodative.   

>True, but that still doesn’t provide any tools for the ISP to know either way.  If there’s no way to know (other than trusting your neighbors) that the network is not in the middle of a “challenging scenario”, then why wouldn't Algorithm B always be used?

Yes, it may end up being true that Algorithm B is always used. 
We also want to let ISPs make their own decision about 
applicability of Algorithm A. I am hoping for agreement on the proposed 
wording change in Section 3.7 (with modifications you may suggest).

>I also doubt that an operator would take actions to harm its own paying customers — but the routers can be compromised anyway...or the advertisement can come from the customer itself with a NO_EXPORT community, etc..  The main case I’m thinking of is the compromised/rogue router.    In any of those cases, choosing Algorithm A opens the door to vulnerabilities.  Yes, they might be the same (or at least similar) as what exists with FP-uRPF — but they are not mentioned.

Even for other BGP security mechanisms, the issue of compromise 
of a router or RPKI/ROA management system exists.
Due to such compromises, malicious ROAs may get created,
AS prepend count in BGPsec may be changed maliciously, or the route leak 
protection (RLP) attribute in the route leak solution may be maliciously altered.
We hope that such malicious activity is 1% or less of the attacks,
and that we are successfully addressing the other 99% in our solution methods.
I’ll add text accordingly for EPF-uRPF/Algorithm A in the Security Considerations section.

>>In Figure 3, suppose that prefix P4 a ROA with AS1, 

>Hmm… Except that the “possibility” that an announcement exists *and* the use as a SA, is different from AS1 really doing it.  IOW, augmenting with the expectation that it may happen has its own risks…
>

Yep, there is that trade-off. Here we choose to further reduce the possibility 
of false positive (for SA invalid) at the expense of some other risk.
Yes, a malicious actor at another AS in the neighborhood 
within the customer cone might take advantage to some extent. 
But the same vulnerability exists even if the update were announced.  

>Updating rfc3704 and being part of the same BCP are separate considerations.  To me, the fact that this document Updates rfc3704 makes it a good candidate to be part of the same BCP; a BCP can contain more than one RFC.
>
>A good example is BCP 14, which includes rfc2119 and the more recent rfc8174.

Oh, that is good to know. Thanks for clarifying. I am fine with your suggestion.
Will double check with co-authors and Sandy.

Thank you.

Sriram

---

From: Alvaro Retana <[email-masked]>
Date: Tue, 20 Aug 2019 11:35:02 -0400

On August 20, 2019 at 10:29:22 AM, Sriram, Kotikalapudi (Fed) (
[email-masked]) wrote:

Sriram:

Hi!

Let’s cut to the chase…

Algorithm B.

It may be just me…in fact, I didn’t find significant discussion about the
applicability of the algorithms on the list, beyond the initial discussion
of draft-sriram-opsec-urpf-improvements-00, where the “challenging
scenario” was introduced because (in my interpretation) the original
proposal didn’t address real deployments.

Given that Algorithm B is more flexible, and that the limitations from
Algorithm A are overcome (§3.4), I would like to see B be the
recommendation.

I am ok if Algorithm A is mentioned as an alternative (not a
recommendation); one that is less flexible and has specific limitations —
even if those limitations are not expected/assumed, the vulnerability to a
change in conditions should be clearly spelled out.

customers — but the routers can be compromised anyway...or the
advertisement can come from the customer itself with a NO_EXPORT community,
etc.. The main case I’m thinking of is the compromised/rogue router. In any
of those cases, choosing Algorithm A opens the door to vulnerabilities.
Yes, they might be the same (or at least similar) as what exists with
FP-uRPF — but they are not mentioned.

Even for other BGP security mechanisms, the issue of compromise
of a router or RPKI/ROA management system exists.
Due to such compromises, malicious ROAs may get created,
AS prepend count in BGPsec may be changed maliciously, or the route leak
protection (RLP) attribute in the route leak solution may be maliciously
altered.
We hope that such malicious activity is 1% or less of the attacks,
and that we are successfully addressing the other 99% in our solution
methods.
I’ll add text accordingly for EPF-uRPF/Algorithm A in the Security
Considerations section.

We all hope that malicious activity on the Internet is kept to a
minimum…but that doesn’t mean that the risk doesn’t exist, or that the
effect of minimal activity will also be minimal.  Not all prefixes are made
the same: affecting just one could have significant consequences.

interface).

Yep, there is that trade-off. Here we choose to further reduce the
possibility

Yes…but it is important to clearly mention the known vulnerabilities of the
solutions being proposed.

Thanks!

Alvaro.

---

From: "Sriram, Kotikalapudi (Fed)" <[email-masked]>
Date: Tue, 20 Aug 2019 17:14:27 +0000

Thank you, Alvaro.
This has been a very helpful discussion.
I'll update the draft following your recommendations/suggestions.
I think I can upload a new version by tomorrow morning.

Sriram

________________________________________
From: Alvaro Retana <[email-masked]>
Sent: Tuesday, August 20, 2019 11:35 AM
To: Sriram, Kotikalapudi (Fed); The IESG
Cc: Murphy, Sandra; Sandra Murphy; [email-masked]; Jen Linkova; [email-masked]; [email-masked]
Subject: Re: Alvaro Retana's Discuss on draft-ietf-opsec-urpf-improvements-03: (with DISCUSS and COMMENT)

On August 20, 2019 at 10:29:22 AM, Sriram, Kotikalapudi (Fed) ([email-masked]<mailto:[email-masked]>) wrote:

Sriram:

Hi!

Let’s cut to the chase…

It may be just me…in fact, I didn’t find significant discussion about the applicability of the algorithms on the list, beyond the initial discussion of draft-sriram-opsec-urpf-improvements-00, where the “challenging scenario” was introduced because (in my interpretation) the original proposal didn’t address real deployments.

Given that Algorithm B is more flexible, and that the limitations from Algorithm A are overcome (§3.4), I would like to see B be the recommendation.

I am ok if Algorithm A is mentioned as an alternative (not a recommendation); one that is less flexible and has specific limitations — even if those limitations are not expected/assumed, the vulnerability to a change in conditions should be clearly spelled out.

We all hope that malicious activity on the Internet is kept to a minimum…but that doesn’t mean that the risk doesn’t exist, or that the effect of minimal activity will also be minimal.  Not all prefixes are made the same: affecting just one could have significant consequences.

Yes…but it is important to clearly mention the known vulnerabilities of the solutions being proposed.

Thanks!

Alvaro.

---

From: "Sriram, Kotikalapudi (Fed)" <[email-masked]>
Date: Fri, 30 Aug 2019 17:28:19 +0000

Alvaro,

A new version -04 has been uploaded that includes changes
based on your Discuss and other comments.
It took sometime because we (authors) thought it was best to 
incorporate all of the IESG reviewers' comments before uploading
a new version. That is done now. 

>....

These main changes per your recommendation are incorporated.
Please take a look at the revised Section 3.7, where we've added 
a new Section 3.7.1 to speak about applicability and limitations of Algorithm A.
You may take a look at the diff:
https://tools.ietf.org/rfcdiff?url2=draft-ietf-opsec-urpf-improvements-04.txt

>Yes…but it is important to clearly mention the known vulnerabilities of the solutions being proposed.

We have carefully revised the Security Considerations section.
There we mention again the limitations of Algorithm A 
and refer back to Section 3.7.1 for details.
We also discuss (in the Security Considerations) the risks
of augmenting the RPF list based on ROA data (per your suggestion). 
We also mention the need for "security hardening of    
routers and other supporting systems (e.g., Resource PKI (RPKI) and 
ROA management systems) against compromise..."

Please let us know if you have any additional comments.
Thanks very much for your help.

Sriram

---

From: Alvaro Retana <[email-masked]>
Date: Fri, 30 Aug 2019 15:41:29 -0400

Thanks Sriram!

I’m clearing my DISCUSS.

Alvaro.

On August 30, 2019 at 1:28:22 PM, Sriram, Kotikalapudi (Fed) (
[email-masked]) wrote:

Alvaro,

a new Section 3.7.1 to speak about applicability and limitations of
Algorithm A.

Sriram