Subject: Re: [Sidrops] ASPA duplicates

From: Alexander Azimov <[email-masked]>
Date: Sat, 2 May 2020 21:39:31 +0300

Thank you all for comments.

First of all, I'd like to address the reasons for that change at the side
of RTR.

And the comparison with ROA is important. The RTR documents don't say that
"you MUST apply diff after receiving EOD", instead it is applied
incrementally. So, the race in the updates for selected address space may
invalidate the correct routes. One may say that finally, all data will
become consistent and the router will make a proper marking. Unfortunately,
it's not true for two reasons:

   - You never know when all data will arrive, there may be every kind of
   network issues between cache and router (and it's TCP after all);
   - Even if all updates finally arrives some routing software will keep
   the result of original verification. I'm not sure how it is covered with
   RFC, but I know that such implementations exist.

As a result, we may end with hard to debug prefix propagation issues. These
race conditions are partially addressed in the RFC8210bis by adding order
to the ROA updates - from more specific to less specific routes. It solves
the problem with less specifics, but issues for prefixes that have multiple
records may still exist.

In the ASPA, we are trying to prevent race conditions from happening. And
the easiest way is to guarantee the atomicity of updates for each key-value
pair. In our cases - to pack all ASPA records for the selected customer AS
into one PDU. I hope this will make clear the motivation why ASPA PDU is
different from what we have now for ROA. Btw, I will vote to update ROA too
(and the update of RTR is a good time for such change, there is no
requirement for compatibility between versions).

The way new ASPA records are issued and the way they are received by RTR
(rsync?) is another place for race conditions. And it seems native to
guarantee consistency the same way - making a single ASPA object, thus
preventing possible misconfigurations.

Now the question - what should be done if RTR cache receives multiple RTR
records for ASPA? Let's discuss possible scenarios:

   - There is misfunction at the level of RPKI software that issued records;

If we face a bug in the hosting RPKI software it should be gracefully
processed, without effect on operations.

   - Tim's example with administrative domain splitting;

The administrative domain splitting in RPKI doesn't seem to be a good idea,
it significantly increases chances to get in the state with a partial set
of customer-providers pairs with hardly predictable outcomes.

   - Transfer between RIRs.

And the last seems to be on the same board - such semistate isn't safe, and
one should keep one record at any point in time.

Keeping in mind that the absence of ASPA record is less dangerous then
inconsistent state my thinking is that if RPKI cache receives multiple ASPA
records for selected customers AS it should ignore all of them. IMO looks
like a fair good compromise: prevent race conditions and also do not have
an effect on operations if something goes wrong.

ср, 29 апр. 2020 г. в 15:26, Tim Bruijnzeels <[email-masked]>:

> Hi,
>
> > On 28 Apr 2020, at 19:03, Job Snijders <[email-masked]> wrote:
> >
> > On Tue, Apr 28, 2020, at 18:53, Jay Borkenhagen wrote:
> >> The current ASPA Verification draft:
> >>
> >> https://tools.ietf.org/html/draft-ietf-sidrops-aspa-verification-04
> >>
> >> .... says in Section 3 "For a selected Customer AS MAY exist only
> >> single ASPA object."
> >>
> >> I concur that an ASPA object should list every authorized upstream ASN
> >> to avoid possible race conditions, and as such it makes sense for only
> >> a single ASPA object to exist at any point in time.
> >>
> >> But how is that uniqueness to be ensured?  What should RPs do if
> >> multiple validated ASPA objects are ever found to exist?
> >
> > Good question. What should it do?
>
> In my mind it's good to that CAs can create a single ASPA object for a
> customer ASN they hold for all their provider(*) ASNs.
>
> But, I think it may be better to cover that CAs SHOULD (or even MUST if
> you will) combine all provider ASNs on a single ASPA objects as a separate
> BCP, and leave a bit more flexibility in the profile and validation drafts.
> This makes it easier to change things, should we find that we need to
> change our minds on what is best.
>
> I don't think that RPs have to check whether the CA did indeed use a
> single ASPA object for a customer ASN. I think it should just validate all
> ASPA objects it finds, and build a list Verified Asn Relations <or insert
> alternate acronym here>. Furthermore, I think that  8210bis should then
> specify that the RP (cache) MUST exclude duplicates when sending these
> relations to routers.
>
> And veering a bit further into that document, I am not sure why those
> updates should be sent as a single PDU containing all current provider
> ASNs. I think it make sense to do the same as with ROAs where
> 'announcement' (add) or 'withdraw' VRPs (excluding duplicates) are sent
> over RTR. I read the comment about race conditions, but I don't see how
> this is an issue when such PDUs are sent as a typical exchange (section
> 8.2) between a Cache Response and End of Data PDUs.
>
> *) relations as defined in draft, not necessarily the business relation.
>
> > I expect such duplicates *will* exist if ASPA were to be deployed for
> real: in cases where an ASN is transferred from one RIR to another RIR and
> one wishes to make before break.
>
> Well in that case one would expect objects for the same customer ASN in
> different parts of the RPKI tree.
>
> But a similar issue could also come up if a parent issues the same ASN to
> multiple children. Which could be a working model if a large operation
> operates the same ASN globally, but has different teams managing it. In
> that case an organisation could chooses to run a delegated CA and issue
> certificates to child CAs in use by those teams. (I am not sure how likely
> this is, you have much more operational insight than I do, but it's
> definitely possible under the standards).
>
> In those cases any of those CAs can issue valid ASPA objects, and they
> should be combined as above.
>
> Furthermore, the security considerations (section 8) of the AS path
> verification draft can use some additional words to warn that this also
> means that any ASPA issuer can potentially invalidate other users of the
> same ASN if they exist.
>
> Maybe the idea of multiple parties operating the same ASN is ludicrous,
> but then I think it would be good to have words that discourage delegating
> the same ASN to multiple CAs, and similarly, discourage issuing ASPA for
> customer ASNs that are also delegated.
>
> Note that this issue is somewhat similar to ROAs for prefixes which are
> also (partially) delegated, or received by multiple organisations.
>
>
>
>
> >
> > Kind regards,
> >
> > Job
> >
> > _______________________________________________
> > Sidrops mailing list
> > [email-masked]
> > https://www.ietf.org/mailman/listinfo/sidrops
>
> _______________________________________________
> Sidrops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sidrops
>

-- 
Best regards,
Alexander Azimov

---

From: [email-masked]
Date: Sun, 03 May 2020 04:02:24 +0700

Mmb

Sent from Android device

On 3 May 2020 01:39, Alexander Azimov <[email-masked]> wrote:

First of all, I'd like to address the reasons for that change at the side of RTR.

And the comparison with ROA is important. The RTR documents don't say that "you MUST apply diff after receiving EOD", instead it is applied incrementally. So, the race in the updates for selected address space may invalidate the correct routes. One may say that finally, all data will become consistent and the router will make a proper marking. Unfortunately, it's not true for two reasons:

You never know when all data will arrive, there may be every kind of network issues between cache and router (and it's TCP after all);

Even if all updates finally arrives some routing software will keep the result of original verification. I'm not sure how it is covered with RFC, but I know that such implementations exist.

As a result, we may end with hard to debug prefix propagation issues. These race conditions are partially addressed in the RFC8210bis by adding order to the ROA updates - from more specific to less specific routes. It solves the problem with less specifics, but issues for prefixes that have multiple records may still exist.

In the ASPA, we are trying to prevent race conditions from happening. And the easiest way is to guarantee the atomicity of updates for each key-value pair. In our cases - to pack all ASPA records for the selected customer AS into one PDU. I hope this will make clear the motivation why ASPA PDU is different from what we have now for ROA. Btw, I will vote to update ROA too (and the update of RTR is a good time for such change, there is no requirement for compatibility between versions).

The way new ASPA records are issued and the way they are received by RTR (rsync?) is another place for race conditions. And it seems native to guarantee consistency the same way - making a single ASPA object, thus preventing possible misconfigurations.

Now the question - what should be done if RTR cache receives multiple RTR records for ASPA? Let's discuss possible scenarios:

There is misfunction at the level of RPKI software that issued records;

If we face a bug in the hosting RPKI software it should be gracefully processed, without effect on operations.

Tim's example with administrative domain splitting;

The administrative domain splitting in RPKI doesn't seem to be a good idea, it significantly increases chances to get in the state with a partial set of customer-providers pairs with hardly predictable outcomes.

Transfer between RIRs.

And the last seems to be on the same board - such semistate isn't safe, and one should keep one record at any point in time.

Keeping in mind that the absence of ASPA record is less dangerous then inconsistent state my thinking is that if RPKI cache receives multiple ASPA records for selected customers AS it should ignore all of them. IMO looks like a fair good compromise: prevent race conditions and also do not have an effect on operations if something goes wrong.

Hi,

> Good question. What should it do?

In my mind it's good to that CAs can create a single ASPA object for a customer ASN they hold for all their provider(*) ASNs.

But, I think it may be better to cover that CAs SHOULD (or even MUST if you will) combine all provider ASNs on a single ASPA objects as a separate BCP, and leave a bit more flexibility in the profile and validation drafts. This makes it easier to change things, should we find that we need to change our minds on what is best.

I don't think that RPs have to check whether the CA did indeed use a single ASPA object for a customer ASN. I think it should just validate all ASPA objects it finds, and build a list Verified Asn Relations <or insert alternate acronym here>. Furthermore, I think that  8210bis should then specify that the RP (cache) MUST exclude duplicates when sending these relations to routers.

And veering a bit further into that document, I am not sure why those updates should be sent as a single PDU containing all current provider ASNs. I think it make sense to do the same as with ROAs where 'announcement' (add) or 'withdraw' VRPs (excluding duplicates) are sent over RTR. I read the comment about race conditions, but I don't see how this is an issue when such PDUs are sent as a typical exchange (section 8.2) between a Cache Response and End of Data PDUs.

> I expect such duplicates *will* exist if ASPA were to be deployed for real: in cases where an ASN is transferred from one RIR to another RIR and one wishes to make before break.

Well in that case one would expect objects for the same customer ASN in different parts of the RPKI tree.

But a similar issue could also come up if a parent issues the same ASN to multiple children. Which could be a working model if a large operation operates the same ASN globally, but has different teams managing it. In that case an organisation could chooses to run a delegated CA and issue certificates to child CAs in use by those teams. (I am not sure how likely this is, you have much more operational insight than I do, but it's definitely possible under the standards).

In those cases any of those CAs can issue valid ASPA objects, and they should be combined as above.

Furthermore, the security considerations (section 8) of the AS path verification draft can use some additional words to warn that this also means that any ASPA issuer can potentially invalidate other users of the same ASN if they exist.

Maybe the idea of multiple parties operating the same ASN is ludicrous, but then I think it would be good to have words that discourage delegating the same ASN to multiple CAs, and similarly, discourage issuing ASPA for customer ASNs that are also delegated.

Note that this issue is somewhat similar to ROAs for prefixes which are also (partially) delegated, or received by multiple organisations.

Sidrops mailing list

[email-masked]

--

Best regards,
Alexander Azimov

---

From: Tim Bruijnzeels <[email-masked]>
Date: Mon, 4 May 2020 15:09:53 +0200

Hi,

> On 2 May 2020, at 20:39, Alexander Azimov <[email-masked]> wrote:
> 
> 	• You never know when all data will arrive, there may be every kind of network issues between cache and router (and it's TCP after all);
> 	• Even if all updates finally arrives some routing software will keep the result of original verification. I'm not sure how it is covered with RFC, but I know that such implementations exist.

Ok, understood. Makes sense now.

> 	• There is misfunction at the level of RPKI software that issued records;
> 	• Tim's example with administrative domain splitting;
> 	• Transfer between RIRs.

This might not be easy for existing RPs - with ROAs the thought has also been that they are cumulative. Now you would need to keep much more state, and even check between all configured TAs - which until now could just be processed in parallel.

I understand your reasoning, but I think this warrants at least some normative wording in the document and discussion in the security section. And most importantly I am curious to learn what RP software implementers think (nowadays I do the CA side, which is much easier in this context).

Tim

---

From: Randy Bush <[email-masked]>
Date: Mon, 04 May 2020 06:39:34 -0700

> This might not be easy for existing RPs - with ROAs the thought has
> also been that they are cumulative. Now you would need to keep much
> more state, and even check between all configured TAs - which until
> now could just be processed in parallel.
> 
> I understand your reasoning, but I think this warrants at least some
> normative wording in the document and discussion in the security
> section. And most importantly I am curious to learn what RP software
> implementers think (nowadays I do the CA side, which is much easier in
> this context).

from draft-ietf-sidrops-8210bis

11.  ROA PDU Race Minimization

   When a cache is sending ROA PDUs to the router, especially during an
   initial full load, two undesirable race conditions are possible:

   Break Before Make:  For some prefix P, an AS may announce two (or
      more) ROAs because they are in the process of changing what
      provider AS is announcing P.  This is is a case of "make before
      break."  If a cache is feeding a router and sends the one not yet
      in service a significant time before sending the one currently in
      service, then BGP data could be marked invalid during the
      interval.  To minimize that interval, the cache SHOULD announce
      all ROAs for the same prefix as close to sequentially as possible.

   Shorter Prefix First:  If an AS has issued a ROA for P0, and another
      AS (likely their customer) has issued a ROA for P1 which is a sub-
      prefix of P0, a router which receives the ROA for P0 before that
      for P1 is likely to mark a BGP prefix P1 invalid.  Therefore, the
      cache SHOULD announce the sub-prefix P1 before the covering prefix
      P0.

randy

---

From: Tim Bruijnzeels <[email-masked]>
Date: Tue, 5 May 2020 09:45:07 +0200

Hi,

> On 4 May 2020, at 15:39, Randy Bush <[email-masked]> wrote:
> 

Thanks for the pointer! It's unfortunate that there are no transactions, but I accept (already did) that this is what it is.

However, my point which you quoted was about the suggestion that Alexander made that there MUST NOT be multiple ASPA objects for the same customer ASN in the RPKI, and that if there are they should all be ignored - even if they are all valid objects.

At least that was my understanding of his intent. So that there is a simple model to deal with ASNs being present on multiple CA certificates in the RPKI (delegated or different TAs even) and the confusion and race conditions that may arise from managing ASPA objects with that ASN as customer ASN all together. Ignoring the objects provides a soft landing then.

I am not against this per se, but I think *this* needs clarification in the ASPA doc, and possibly some more discussion here. I am sure that it's possible, but it requires that RPs keep more state than they do today.

Furthermore, I see some advantage here (it's a simple model), but as a downside this provides a sort of denial attack where a parent CA can issue an ASPA object to essentially invalidate the object issued by the operational holder of the ASN. Worse, this can happen between TAs, so only one TA would need to be subverted in order to invalidate ASPA objects under any of the other TAs.

For ROAs the RPs just build a complete list of all VRPs found on all valid ROAs anywhere, and only then a set is made and duplicates are excluded when talking to the router. So VRPs appearing elsewhere do not have the power to force other VRPs to the 'ignore list'. For ROAs the advice is that parent CAs have a responsibility NOT to invalidate their children's announcements, and for inter RIR transfers make before break is at least theoretically possible. A similar model could also apply to ASPA objects.

I know this is a political minefield as much as a technical issue. Nonetheless, I think we should make a clear decision which scenario is desired under today's operational reality.

Tim

---

From: Alexander Azimov <[email-masked]>
Date: Tue, 5 May 2020 23:49:16 +0300

Hi Tim,

вт, 5 мая 2020 г. в 10:45, Tim Bruijnzeels <[email-masked]>:

> Thanks for the pointer! It's unfortunate that there are no transactions,
> but I accept (already did) that this is what it is.
>
> However, my point which you quoted was about the suggestion that Alexander
> made that there MUST NOT be multiple ASPA objects for the same customer ASN
> in the RPKI, and that if there are they should all be ignored - even if
> they are all valid objects.
>
> At least that was my understanding of his intent. So that there is a
> simple model to deal with ASNs being present on multiple CA certificates in
> the RPKI (delegated or different TAs even) and the confusion and race
> conditions that may arise from managing ASPA objects with that ASN as
> customer ASN all together. Ignoring the objects provides a soft landing
> then.
>
> I am not against this per se, but I think *this* needs clarification in
> the ASPA doc, and possibly some more discussion here. I am sure that it's
> possible, but it requires that RPs keep more state than they do today.
>
I do agree. In this thread was raised the proper question which wasn't
covered by the current draft and we may have found the proper answer.
I'll add clarification with the next update.

> Furthermore, I see some advantage here (it's a simple model), but as a
> downside this provides a sort of denial attack where a parent CA can issue
> an ASPA object to essentially invalidate the object issued by the
> operational holder of the ASN. Worse, this can happen between TAs, so only
> one TA would need to be subverted in order to invalidate ASPA objects under
> any of the other TAs.
>
If we ignore duplicate object it can't be used for DoS. The draft doesn't
suggest any actions against 'unknown' paths.
The question of what happens if one TA issues invalid records for address
space/ASNs that are operated by another TA where such records do not exist,
should be discussed separately if we admit such an attack vector. IMO it is
highly unlikely, but if it happens can already lead to global connectivity
disruption.

> For ROAs the RPs just build a complete list of all VRPs found on all valid
> ROAs anywhere, and only then a set is made and duplicates are excluded when
> talking to the router. So VRPs appearing elsewhere do not have the power to
> force other VRPs to the 'ignore list'. For ROAs the advice is that parent
> CAs have a responsibility NOT to invalidate their children's announcements,
> and for inter RIR transfers make before break is at least theoretically
> possible. A similar model could also apply to ASPA objects.
>
I believe the operation requirements for address space transfer is
different from the ASN transfer from one RIR to another. But even in the
ROA scenario, different ROAs existing in two RIRs simultaneously might
cause problems.

> I know this is a political minefield as much as a technical issue.
> Nonetheless, I think we should make a clear decision which scenario is
> desired under today's operational reality.
>
The ASN transfers do happen, but as far as I know, it doesn't represent
common day-to-day operations. IMO the security reasoning should prevail for
ASPA and might be reconsidered for ROA.

---

From: Randy Bush <[email-masked]>
Date: Tue, 05 May 2020 13:53:20 -0700

yes, the CAs could issue multiple ASPAs for an AS

the cache merges them into one ASPA PDU to feed the routers

randy