Subject: [Sidrops] ARIN RPKI Service Impact - 12 August 2020 - manifest issue - resolved

From: John Curran <[email-masked]>
Date: Thu, 13 Aug 2020 17:53:54 +0000

RPKI folks - 

	In the process of upgrading our HSM yesterday, ARIN updated its RPKI signing infrastructure incorrectly – the result was an encoding error in our manifest that caused rpki-client and FORT validators no longer consider ARIN’s RPKI data to be valid. (see attached service announcement) 

	This has been since resolved and we’re in the process of reissuing ROAs created during the time.  We are not aware of any delegated repositories impacted during this period. 

	Our thanks to the OpenBSD team - Sebastian Benoit, Theo Buehler, Joel Sing, Job Snijders, and Claudio Jeker - who were instrumental in hunting down this issue. 

	I’ll provide a more detailed post-mortem here once available.

My apologies for the service impact,
/John

John Curran
President and CEO
American Registry for Internet Numbers

===  https://www.arin.net/announcements/20200813/

RPKI Service Notice Update

Posted: Thursday, 13 August 2020 
Service Update 

After upgrading our HSM on Wednesday, August 12, 2020, Job Snijders reported to us that our RPKI repository was no longer validating using rpki-client or fort. Upon investigation, it was discovered that we had an encoding error in our new software. (Specifically, there was a mismatch in the “parameters” field between the “algorithm identifier” of the certificate and the certificate To Be Signed [TBS]. The TBS set the “parameters” as null and the certificate as empty.)

A fix has been made and there will be a pending data clean up in the next few days to fix some of the ROAs created during the interim.

We would like to thank Sebastian Benoit, Theo Buehler, Joel Sing, Job Snijders, and Claudio Jeker, from the OpenBSD project (https://openbsd.org ), as they spent considerable time working with us to identify the root cause of the issue.

Regards,

Richard Jimmerson
Chief Operating Officer
American Registry for Internet Numbers (ARIN)
===

---

From: Christopher Morrow <[email-masked]>
Date: Thu, 13 Aug 2020 22:31:21 -0400

howdy john!
1st: thanks for the pointer/RFO/report... (again, learning from others
mistakes means maybe we'll have less pain going forward? :) )
2nd: thanks for the call out to community members who helped poke the
bear here and get to a solution! :)

>         In the process of upgrading our HSM yesterday, ARIN updated its RPKI signing infrastructure incorrectly – the result was an encoding error in our manifest that caused rpki-client and FORT v…

Are there lessons learned here for the other validators and CA folk?
Are there test cases we can use in other CA deployments? (both RIR and
delegated)

-chris

> _______________________________________________
> Sidrops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sidrops

---

From: John Curran <[email-masked]>
Date: Fri, 14 Aug 2020 11:32:27 +0000

On 13 Aug 2020, at 10:31 PM, Christopher Morrow <[email-masked]> wrote:
> ...
> ...

Hello Chris!

   Short answer - see above (i.e. “I‘ll provide a more detailed post-mortem here once available.”)
	
   In the meantime, I’ll speculate a bit – warning that this is the view from 10km up by someone with only an offhand knowledge of such things – 

	1) CA operators (e.g. ARIN) should test against a larger portion of the validator ecosystem when doing major changes. 
	2) ARIN needs more diverse and coordinated test environment usage by the RPKI community 
	3) Additional stringency to specs for the more common validators would help in some cases 

  If you’re looking right now for insight of this incident sufficient for writing test cases, I’d look at Job’s OpenBSD writeup - 
  <http://sobornost.net/~job/arin-manifest-issue-2020.08.12.txt>

Best wishes (and stay safe!)
/John

John Curran
President and CEO

---

From: Randy Bush <[email-masked]>
Date: Sat, 15 Aug 2020 14:19:36 -0700

thanks for the preliminary post mortem

> 3) Additional stringency to specs for the more common validators would

fwiw, one dragon lab instance sra is running is so old it is rsync only,
and so did not see the problem.

the assorted dragon labs instances i watch did not report anything.
they quietly fell back from rrdp to rsync per spec.  this is both good
news, things worked as expected, and bad news, they knew something went
wrong and did not report it.

in general, the rpki infrastructure lacks alerts and lacks reporting
channels other than ghostbuster records.

this week our (john kristoff doing the heavy lifting) short paper was
accepted at imc.  will shout when there is camera ready.  from the
abstract

    In this short paper, we introduce a framework to observe RPKI
    relying parties (i.e., those that fetch RPKI data from the
    distributed repository) and present insights into this ecosystem
    for the first time. Our longitudinal study of data gathered from
    three RPKI certification authorities (AFRINIC, APNIC, and our own
    CA) identifies different deployment models of relying parties and
    (surprisingly) larger inconsistent fetching behavior, which might
    affect robustness in Internet routing. Our results reveal that some
    relying parties are not able to connect to specific publication
    points at all.

randy

---

From: Job Snijders <[email-masked]>
Date: Sat, 15 Aug 2020 21:59:24 +0000

>From my observations RPKI data with a signature identifier discrepancy
was distributed both via RRDP and RSYNC. This was not a transport
protocol specific event.

A copy (made during the outage window) is available here:
http://sobornost.net/~job/arin-broken-state-20200812.tar.gz

Kind regards,

Job

---

From: John Curran <[email-masked]>
Date: Sat, 15 Aug 2020 22:11:24 +0000

Hi Randy!

> thanks for the preliminary post mortem

No problem…  (fairly obligatory after precipitating production infrastructure services) 

> channels other than ghostbuster records.

Something that we all should probably spend some time thinking about – 
particularly as the dependency on this infrastructure increases. 

>    relying parties are not able to connect to specific publication

Thanks for the info – definitely some interesting work!
/John

John Curran
President and CEO

---

From: Martin Hoffmann <[email-masked]>
Date: Mon, 17 Aug 2020 16:31:34 +0200

Heisann,

I started to implement the rules laid out in section 6 "Relying
Party Processing of Manifests" of the draft and now have a few
questions and remarks. I decided to split these up into separate
messages. So, expect a few more messages ...

                             *   *   *

It is not quite clear how to deal with failed fetches. Section
6.4 dictates that invalid signed objects (but curiously not invalid
certificates or CRLs) fail a fetch. The consequence of this is that a
fetch already fails if a single ROA (or worse: a single ghost-buster
record) is expired.

Section 6.7 now states that I should keep using "cached versions of the
objects associated with this CA instance." What does that actually
mean? What are these objects associated with the instance? The objects
listed on the new manifest or the old manifest? Or does objects here
mean validated output generated earlier?

If indeed it means RPKI objects, then doesn’t the expired ROA
essentially block all updates to the other objects to the CA? That
would strike me as counter-productive in terms of robustness.

This rule also blocks skipping objects of types I don’t know or care
about. I will have to at least do signed object validation on them,
which means reading and parsing them and then do signature validation.
If that is intended, I think this should be called out explicitly in
the document.

But I’m not even sure it provides any benefit. I, say, I am validating
a resource tagged association (RTA, [0]), I don’t care about the ROAs
at all. Does the RTA become invalid because a CA somewhere in the
validation chain had an expired ROA?

Kind regards,
Martin

[0] https://datatracker.ietf.org/doc/draft-michaelson-rpki-rta/

---

From: Stephen Kent <[email-masked]>
Date: Mon, 17 Aug 2020 14:34:20 -0400

Martin,
> It is not quite clear how to deal with failed fetches.

The overall response to a failed fetch is described in the intro to 
Section 6, as well as in 6.7. the intro states:

    If a fetch fails, it is assumed that a subsequent fetch will resolve
    problems encountered during the fetch.  Until such time as a
    successful fetch is executed, an RP SHOULD use cached data from a
    previous, successful fetch.  ...

Section 6.7 reiterates this description of what to do, and provide more 
details re what "termination of processing" means.

> Section

Section 6.4 deals with acquiring all files referenced by a manifest, 
including certs and CRLs. The text says:

    The RP MUST acquire all of the files enumerated in the manifest
    (fileList) from the publication point.  This includes the CRL, each
    object containing an EE certificate issued by the CA, and any
    subordinate CA and EE certificates.

> mean validated output generated earlier?

The phrase "CA instance" is used throughout Section 6 to accommodate CA 
key or alg rollover. So, a CA instance refers to a CA cert and the 
objects issued under it, as indicated by the SIA info. This is described 
in Section 6.1, which notes to use the id-ad-rpkiManifest URI, as well 
as in the intro to Section 6.

The cached versions of objects refers to certs, CRLs, and signed objects 
that were validated during a previous fetch cycle, and which are not 
stale/expired.

> would strike me as counter-productive in terms of robustness.
Are you positing the case where the cache contains an expired ROA for a 
CA instance, and a fetch that would have replaced the expired ROA fails? 
In that case, the cache entry for that ROA would no longer be valid, but 
I don;'t think that would affect other cache entries for the CA 
instance. Yes, this re-write of 6486 trades off robustness for more 
restrictive manifest checking.
> If that is intended, I think this should be called out explicitly in
If a manifest points to objects that are not CRLs, certs, ROAs, etc., 
then it is in error. But, your question seems to be what processing has 
to be performed on the files contained in an apparently valid manifest, 
right? Section 6.4 and RFC 6488 defines the tests to be performed, and 
6.4 explicitly cites 6488. What additional info do you feel is needed here?
> validation chain had an expired ROA?

I have not examined the RTA ID, and it's an expired draft, so ...

Steve

---

From: Martin Hoffmann <[email-masked]>
Date: Tue, 18 Aug 2020 08:36:59 +0200

Stephen Kent wrote:
>   
> Are you positing the case where the cache contains an expired ROA for
> a CA instance, and a fetch that would have replaced the expired ROA
> fails?

No, I am talking about a case where the ROA can be fetched successfully
and matches the manifest hash, but its EE certificate is expired.
Section 6.4 says that "if [files] fail the validity tests specified in
[RFC6488]", the fetch has failed. Thus, the expired EE certificate in
the ROA fails the complete fetch of all objects associated with the CA.

So, not replacing an expired ROA in a publication point makes the
entire CA not update anymore. I.e., any other objects that now expire
cannot be replaced until that ROA is also replaced.

You could argue “Don’t do that, then” but this approach doesn’t make
the RPKI more robust but rather makes it break more easily on simple
oversights.

> > which means reading and parsing them and then do signature
> > validation. If that is intended, I think this should be called out
> > explicitly in the document.  
> then it is in error.

How do you introduce new object types in this case? There will always
be relying parties that run old software that doesn’t know of them. You
rather have to assume that objects of unknown types are signed objects
with unknown content. If you do that, the current draft stipulates that
you have to read, parse, and validate them -- and then throw away the
content.

This still means that all object types added to the RPKI must be signed
objects. Whether that is okay or not, I don’t quite know.

> But, your question seems to be what processing
> has to be performed on the files contained in an apparently valid
> manifest, right? Section 6.4 and RFC 6488 defines the tests to be
> performed, and 6.4 explicitly cites 6488. What additional info do you
> feel is needed here?

I would like the document to explicitly state how to deal with object
types appearing on a manifest that a replying party does not know. If
nothing else then to make the document more helpful for implementers.

> > But I’m not even sure it provides any benefit. I, say, I am
> > validating a resource tagged association (RTA, [0]), I don’t care
> > about the ROAs at all. Does the RTA become invalid because a CA
> > somewhere in the validation chain had an expired ROA?  
> 

RTA validates signed objects distributed via alternative means using
the PKI published as part of the RPKI. I.e., one of the CA
certificates published via the RPKI has issued the EE certificate used
in that signed object.

In order to validate that object, I do not need to look at any ROAs or
GBRs, only certificates, CRLs, and manifests.

Should validation of that object fail if there is an expired ROA
published by one of the CAs along the validation chain?

Kind regards,
Martin

---

From: George Michaelson <[email-masked]>
Date: Wed, 19 Aug 2020 05:50:51 +1000

FYI we fully intend resubmitting RTA, we were waiting for the second
implementation (the work Martin is doing) to have input of merit to
spin the new version. It would be plausible to resubmit for the
November IETF. Two running code!

We designed RTA to be complete outside of a repo, so it can be used
privately between B2B partners. Not that its things cannot "be" on the
manifest, but that there is no dependency on the repo framework beyond
the TA, because the passed data is the chain for validation. This
matters to people who want to conduct activity outside of the BGP
validation problem, which is an 'all data needed' problem. B2B uses of
RTA are not directly about all data. Its different. (provisioning is
typically the use case here and the intent is to show the request to
route, originate, provision is valid, because control of the resource
can be demonstrated. its inherently a one-to-one conversation)

RTA objects have an OID, are in the registry of types. So there is a
meta-question implicit here, that the set of OID which can be
referenced from a Manifest is NOT bound, and CAN increase, so
validators have to be written in a way which expect to deal with
unknown signed things: The system is not closed, and there will be
others in future. I think the validation of a manifest MUST NOT fail
simply because unknown objects exist on the Manifest, especially if
the OID lies in the arc of RPKI definitions.

Martin's question is also on the other side of the deal: If a manifest
is showing a repository is incomplete, does it cause formal
invalidation of all things in the manifest, which has potential to
make validation of the 'complete' chain we send in RTA not work
because now, an object on the RTA validation path is questioned for
other reasons.

I tend to the view that only things which invalidate the certificates
matter, ROA or other elements are not contributing to the validity of
the RTA.  This is because they are not material to the validity of the
object. If it was a 'whole repo' problem I could argue otherwise. It
is not. It is inherently constrained.

Specifically Missing CRL.. I have more concerns about. That would mean
we cannot authentically deny the objects being validated.

-G

---

From: Di Ma <[email-masked]>
Date: Wed, 19 Aug 2020 20:28:55 +0800

George,

> 2020年8月19日 03:50，George Michaelson <[email-masked]> 写道：
> 

Do you mean RTA validator has nothing to do with RPKI repository sync, given the RTA use-case just requires the INR holder to provide all the necessary certificates to construct the validation path down to TA?

It reminds me of the similar model in RFC 6494 where the RPKI is used for IPv6 SeND. 

Di

---

From: George Michaelson <[email-masked]>
Date: Thu, 20 Aug 2020 08:15:16 +1000

Yes. This is what I mean.

The fly in the ointment is the CRL. I think we need to be very sure
about how you source a CRL which invalidates things being used to
validate a bundle. Is it acceptable to use a CRL which was applicable
when the object was constructed? I think this is weaker than requiring
the CRL to be checked for applicability at the time you validate. To
do that, you have to be online and have access to all CRL from all
Repo along the validation path.

-G

---

From: Stephen Kent <[email-masked]>
Date: Fri, 21 Aug 2020 14:44:55 -0400

George,

Thanks for the background re RTA. I think your comments address the 
issue Martin raised:

the data structure is designed to be "complete outside of a repo" and, 
unlike the other objects at a pub point, this one carries a full 
validation path to some TA, not to the CA associated with the pub point.

So, I don't think an RTA should be stored in the RPKI repo system, and 
thus it ought not appear in a manifest.

You raise the "meta-question"of whether the set of data items that can 
be stored at a pub point, and thus included on a manifest, is bounded. I 
believe that new data items might be defined for storage in the repo 
system, and that, if they are, they should be covered by the manifest. 
However, such items ought to be consistent with the overall RPKI 
functionality, not just signed objects that someone wants to piggyback 
on the RPKI repo system. Let's not encourage the RPKI to be more like an 
IRR DB.

Steve

> On Tue, Aug 18, 2020 at 4:37 PM Martin Hoffmann<[email-masked]>  wrote:
>> Stephen Kent wrote:

---

From: Martin Hoffmann <[email-masked]>
Date: Wed, 26 Aug 2020 12:25:39 +0200

Hi Steve,

looks like you forgot to include the mailing list, so I’ll keep the
quote in full.

Stephen Kent wrote:
> Martin,
> > Stephen Kent wrote:  
> >>    
> >> Are you positing the case where the cache contains an expired ROA
> >> for a CA instance, and a fetch that would have replaced the
> >> expired ROA fails?  
> > No, I am talking about a case where the ROA can be fetched
> > successfully and matches the manifest hash, but its EE certificate
> > is expired. Section 6.4 says that "if [files] fail the validity
> > tests specified in [RFC6488]", the fetch has failed. Thus, the
> > expired EE certificate in the ROA fails the complete fetch of all
> > objects associated with the CA.  
> Thanks for the clarification.
> > entire CA not update anymore. I.e., any other objects that now
> > expire cannot be replaced until that ROA is also replaced.  
> "not update anymore" is not how I would state the result. This fetch 
> will fail. Because a failed fetch will be reported to the RP
> operations staff, hopefully they will contact the cognizant entity
> for the pub point in question, causing the error to be fixed. Them a
> subsequent fetch can succeed.

That seems like an overly optimistic approach to the issue. Assume the
problem is created by a bug or, worse, design oversight in the CA
software. The turnaround from discovering the issue to deploying a fix
can easily be weeks with some vendors. During all that time, not only
can no ROAs be updated and may child CA certificates slowly expire, the
entire CA’s data will not be available at all for any newly deployed
relying parties. With containerised deployment, this is quite a serious
issue.

As a consequence, this approach will make the routing system less
secure for, I’d like to argue, no actual gain.

> My sense of the WG discussion was that the majority of  folks chose
> to prioritize correctness over robustness, and I made numerous
> changes to the text to reflect that.

I disagree with the blanket assessment that this approach makes RPKI more
correct. To switch to the example I should have used in the first place:
Ignoring a broken GBR object when producing a list of VRPs does not
make the list less correct. In fact, the opposite is true: Ignoring the
CA or updates to the CA because of a broken GBR makes this list less
correct.

> >> If a manifest points to objects that are not CRLs, certs, ROAs,
> >> etc., then it is in error.  
> > How do you introduce new object types in this case? There will
> > always be relying parties that run old software that doesn’t know
> > of them. You rather have to assume that objects of unknown types
> > are signed objects with unknown content. If you do that, the
> > current draft stipulates that you have to read, parse, and validate
> > them -- and then throw away the content.
> >
> > This still means that all object types added to the RPKI must be
> > signed objects. Whether that is okay or not, I don’t quite know.  
> 
> Yes, all objects in the RPKI are always signed!
> 
> But, you first question raises a valid point, i.e., we do not have a 
> generic description of how new objects types are to be introduced in
> a graceful fashion (wrt RP processing). I am not sure that one
> can/should amend 6486bis to address this topic.

You absolutely have to deal with this issue in 6486bis in its current
strict form. Any introduction of a new object type will permanently
break CAs that use these objects when validated with a relying party
software that is not aware of this type. I don’t think this is
acceptable, as it effectively blocks the introduction of new types
pretty much forever.

> Instead I believe it
> makes sense for any new object proposed for inclusion in the RPKI
> repository system to address this question as part of its
> documentation; it's not clear that a uniform approach is appropriate,
> i.e., one size may not fit all. 6486 can be updated to reflect the
> processing approach proposed for any new objects.

It seems to me that the best approach is to simply ignore unknown
objects. We could argue whether they can be ignored completely or
whether one should at least check their manifest hash. Personally, I
think completely ignoring is the better approach as I don’t see any
benefit in rejecting a CA because someone swapped out an object I don’t
care about.

Ultimately, I feel we’ve swung the pendulum way too far to the other
side. The RPKI isn’t a single data set that needs to synchronized in
full but it consists of multiple data sets that can be treated as
independent: currently these are VRPs, router keys, and GBRs. If I use
the RPKI for route origin validation, I don’t need to synchronize the
router keys or GBRs. Why does it improve route origin validation if
available and correctly signed data is skipped because of issues with
irrelevant data?

> >> performed, and 6.4 explicitly cites 6488. What additional info do
> >> you feel is needed here?  
> > I would like the document to explicitly state how to deal with
> > object types appearing on a manifest that a replying party does not
> > know. If nothing else then to make the document more helpful for
> > implementers.  
> At this time the types of objects that may legitimately appear in a 
> manifest is small and well-defined. Any other objects would result in
> a fatch failure.
> > certificates published via the RPKI has issued the EE certificate
> > used in that signed object.
> >
> > In order to validate that object, I do not need to look at any ROAs
> > or GBRs, only certificates, CRLs, and manifests.  
> According to George, the RTA carries it own, complete cert path, so 
> certs and CRLs stored in the repository system are irrelevant, and, 
> George noted that the RTA was designed to not require distribution
> via the repository system, so it's not relevant example.
> As I noted above, an RTA is not relevant to this discussion, based on 
> George's description of its design and intent.
> 
> Steve

Kind regards,
Martin

---

From: Tim Bruijnzeels <[email-masked]>
Date: Wed, 26 Aug 2020 14:25:33 +0200

> On 26 Aug 2020, at 12:25, Martin Hoffmann <[email-masked]> wrote:
> 
> Hi Steve,
> 

I believe that manifest should tell me that I have *all* *latest* objects. If there is stuff missing then rejecting the publication point is fine by me. But I have a problem with doing the same if invalid objects are found.

In addition to what Martin said. This makes the system very brittle w.r.t. changes in resources. Suppose that I had received 10.0.0.0/24 and 192.168.0.0/24 on my certificate issued to me by my parent, and I created ROAs for both. Now my parent pro-actively removes 10.0.0.0/24 from my certificate but they have neglected to tell me in time. I see no reason why my ROA for 192.168.0.0/24 should now be considered invalid. And I remember that we had discussions in this WG advocating that separate ROAs are created for each prefix, precisely to prevent this kind of fate sharing.

Tim

---

From: John Curran <[email-masked]>
Date: Wed, 26 Aug 2020 14:54:17 +0000

On 13 Aug 2020, at 1:53 PM, John Curran <[email-masked]<mailto:[email-masked]>> wrote:
...
I’ll provide a more detailed post-mortem here once available.

RPKI Folks -

As promised, please find the "reason for outage" report for ARIN’s 12 August RPKI incident -
https://www.arin.net/announcements/20200826/  (and attached below in plaintext)

As noted, we’ll be expanding our internal RPKI testing strategy - please review and let us know if there’s additional validators/packages we should be testing against.

Thanks,
 /John

John Curran
President and CEO

===

12-13 August RPKI Outage: Update
Posted: Wednesday, 26 August 2020
Service Update

12 August 2020 at 12:21 PM to 13 August at 11:36 AM

On 12 August at 12:21 PM, ARIN deployed a new version of its RPKI system. ARIN’s repository showed no errors in both RIPE’s Validator and NLNetLab’s Routinator systems. At 12:46 PM on that same day we received a service issue notice that ARIN’s repository was not working with rpki-client. ARIN Engineering worked closely with the OpenBSD software developers to pinpoint the error within the RPKI system. Both ARIN engineering and the OpenBSD developers independently found the error within ARIN’s repository. The fix was developed and deployed on 13 August at 11:36 AM.

Here is a detailed analysis of the error:

During RPKI repository generation, ARIN creates “manifests.” A manifest is cryptographic object specific to the RPKI which is used to help guarantee the integrity of the repository. One manifest is associated with each resource certificate in the repository. The manifest, flagged by the OpenSSL-based validators, had a subtle encoding issue. The manifest in question essentially contains two copies of an AlgorithmIdentifier variable in different locations (and used for different purposes). Per RFC 5280, Section 4.1.1.2, these two instances must match completely. In ARIN’s manifest, one contained an empty string (“”) as a parameter and the other contained a NULL (pointer to nothing). The empty string parameter was incorrect and the OpenSSL-based validators were flagging this because the two definitions of AlgorithmIdentifier did not match.

Planned Corrective actions:

As a corrective action, ARIN will be broadening its testing strategy. In future releases, we will be validating not only LibreSSL-based validators (RIPE’s Validator and NLNetlab’s Routinator) but also OpenSSL-based validators such as rpki-client and Fort. The list of validators we do test against the ARIN repository will be noted within the RPKI section of ARIN’s website.

ARIN apologizes for any inconvenience that this may have cause those who run OpenSSL-based validators. We take any incompatibility seriously and are working to better service the community on this emerging but yet very important service. We would like to thank Job Snijders and the OpenBSD rpki-client team for helping assist us in solving this issue quickly.

Regards,

Mark Kosters
Chief Technology Officer

===

---

From: Job Snijders <[email-masked]>
Date: Wed, 26 Aug 2020 16:00:01 +0000

Dear John,

I'd like to offer some contributions to this document's outline of the
background of some implementations. Mostly just nitpicking.

On Wed, Aug 26, 2020 at 02:54:17PM +0000, John Curran wrote:
> On 12 August at 12:21 PM, ARIN deployed a new version of its RPKI
> system. ARIN’s repository showed no errors in both RIPE’s Validator
> and NLNetLab’s Routinator systems. 

The current versions of routinator and ripe ncc's validator have weak
(lacking) support for manifest handling, there are other issues in both
softwares that don't yield errors where they should yield errors related
to manifest handling. Neither implementation handles manifests correctly
at the moment, so neither software currently can be used to confirm the
correct publication of manifest related data. :-(

> At 12:46 PM on that same day we received a service issue notice that
> ARIN’s repository was not working with rpki-client. ARIN Engineering
> worked closely with the OpenBSD software developers to pinpoint the
> error within the RPKI system. Both ARIN engineering and the OpenBSD
> developers independently found the error within ARIN’s repository. The
> fix was developed and deployed on 13 August at 11:36 AM.

It was not just rpki-client (linked against LibreSSL), but the FORT
validator (linked against OpenSSL) also flagged the same issue and
emitted an error. LibreSSL and OpenSSL's RPKI extensions do share some
public APIs, (and have some code in common), but they are distinct
implementations from separate development communities. 

> During RPKI repository generation, ARIN creates “manifests.” A
> manifest is cryptographic object specific to the RPKI which is used to
> help guarantee the integrity of the repository. One manifest is
> associated with each resource certificate in the repository. The
> manifest, flagged by the OpenSSL-based validators, had a subtle
> encoding issue. 
>
> The manifest in question essentially contains two
> copies of an AlgorithmIdentifier variable in different locations (and
> used for different purposes). Per RFC 5280, Section 4.1.1.2, these two
> instances must match completely. In ARIN’s manifest, one contained an
> empty string (“”) as a parameter and the other contained a NULL
> (pointer to nothing). The empty string parameter was incorrect and the
> OpenSSL-based validators were flagging this because the two
> definitions of AlgorithmIdentifier did not match.

[ off topic note: from what I understand, the technical reason these
identifiers MUST match is to make certain X.509 fingerprinting
operations possible. If both identifiers had the (required) parameter
field omitted the situation would've been different. ]

> As a corrective action, ARIN will be broadening its testing strategy.
> In future releases, we will be validating not only LibreSSL-based
> validators (RIPE’s Validator and NLNetlab’s Routinator) but also
> OpenSSL-based validators such as rpki-client and Fort. The list of
> validators we do test against the ARIN repository will be noted within
> the RPKI section of ARIN’s website.

This is a good and sensible plan.

Note: Neither RIPE's Validator and routinator are not based on LibreSSL
or OpenSSL. RIPE uses a cryptographic library called "Bouncy Castle" and
routinator appears (to in part?) rely on library bindings from the rust
ecosystem (which under the hood may link LibreSSL or OpenSSL), but also
in part seem to attempt to build their own crypto. LibreSSL and OpenSSL
both reject X.509 data with this type of encoding issue, and
consequently any RPKI validator implementations linked against these
libraries will automatically consider the algorithmidentifier mismatch
an error.

It is key to ensure not only a diverse set of validator implementations
is used for testing, but also care is taken to ensure a diverse set of
cryptographic libraries are put through the test. It was not rpki-client
or FORT rejecting the data, it was libressl and openssl rejecting the
data.

FORT and rpki-client have committed to a strict (safe) interpretation of
how cryptographic data should be handled and what those design choices
mean for internet operations in the global routing table. If either of
these implementations are used to confirm the correct operating of the
ARIN Trust Anchor (in cojunction with less strict implementations), one
will not only find the lowest common denominator, but in doing also
adhere to the highest standards.
 
Kind regards,

Job

---

From: Martin Hoffmann <[email-masked]>
Date: Wed, 26 Aug 2020 20:24:42 +0200

Job Snijders wrote:
> (lacking) support for manifest handling, there are other issues in
> both softwares that don't yield errors where they should yield errors
> related to manifest handling. Neither implementation handles
> manifests correctly at the moment, so neither software currently can
> be used to confirm the correct publication of manifest related
> data. :-(

To the best of my knowledge, Routinator and the RIPE NCC RPKI Validator
handle manifests according to the specifications laid out in the
relevant standards track IETF documents. I assume that you are referring
to your assessment that all objects published by a CA should be
discarded if any inconsistencies are discovered. While such behaviour
is certainly acceptable under the current specification, not doing so
does not constitute incorrect handling of manifests.

Given that this topic is currently discussed in this very working
group and there wasn’t outright consensus on how software should behave
in these cases, it seems only prudent to delay modifications until
after such consensus has been achieved.

If you have information about other issues these software packages
have with regards to manifest handling -- as you seem to imply there
are multiple issues --, I am sure the maintainers would like to hear
about them.

Regards,
Martin

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 27 Aug 2020 14:30:13 +0200 (CEST)

On Wed, 26 Aug 2020, Martin Hoffmann wrote:

> discarded if any inconsistencies are discovered. While such behaviour is 
> certainly acceptable under the current specification, not doing so does 
> not constitute incorrect handling of manifests.

At this point in time, everybody I am aware of who are implementing RPKI 
in the routing system are doing invalid=drop, and nothing else.

The overall goal right now is to make sure the ROA is correctly validated, 
all the way. If a ROA is gone, it doesn't cause an outage. It causes lack 
of protection. It's not a competition which validator can validate the 
most ROAs, the competition should be to get things *right*.

If something doesn't validate correctly, drop it. Drop all it depends on. 
If all ROAs from a RIR are gone for an hour or a day, it's not the end of 
the world. It's not an outage.

We need to make sure the entire ecosystem gets things right, correct, and 
have procedures in place to do things right, all the time. Other parts of 
the Internet ecosystem had teething problems in the beginning, but they 
worked it out. RPKI ecosystem needs to do the same.

The goal of the validator should be to validate. It should be picky. It 
should throw things away that doesn't look right. You're advocating for 
something else, and I don't understand why.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: John Curran <[email-masked]>
Date: Thu, 27 Aug 2020 12:42:33 +0000

On 27 Aug 2020, at 8:30 AM, Mikael Abrahamsson <[email-masked]> wrote:
>> To the best of my knowledge, Routinator and the RIPE NCC RPKI Validator handle manifests according to the specifications laid out in the relevant standards track IETF documents. I assume that you are referring to your assessment that all objects published by a CA should be discarded if any inconsistencies are discovered. While such behaviour is certainly acceptable under the current specification, not doing so does not constitute incorrect handling of manifests.
> 
> At this point in time, everybody I am aware of who are implementing RPKI in the routing system are doing invalid=drop, and nothing else.
> 
> The overall goal right now is to make sure the ROA is correctly validated, all the way. If a ROA is gone, it doesn't cause an outage. It causes lack of protection. It's not a competition which validator can validate the most ROAs, the competition should be to get things *right*.
> 
> If something doesn't validate correctly, drop it. Drop all it depends on. If all ROAs from a RIR are gone for an hour or a day, it's not the end of the world. It's not an outage.
> 
> We need to make sure the entire ecosystem gets things right, correct, and have procedures in place to do things right, all the time. Other parts of the Internet ecosystem had teething problems in the beginning, but they worked it out. RPKI ecosystem needs to do the same.
> 
> The goal of the validator should be to validate. It should be picky. It should throw things away that doesn't look right. You're advocating for something else, and I don't understand why.

Mikael - 

	Did you mean a) “drop it” or b) “drop it with very clear and specific messages that allow for ready identification of the issue” ?

	(I’m unaware how we get to 'work out the teething issues’, as you suggest, unless you meant the latter and its detailed messages…) 

/John

John Curran
President and CEO

---

From: Martin Hoffmann <[email-masked]>
Date: Thu, 27 Aug 2020 15:40:45 +0200

Mikael Abrahamsson wrote:
> 
> At this point in time, everybody I am aware of who are implementing
> RPKI in the routing system are doing invalid=drop, and nothing else.
> 
> The overall goal right now is to make sure the ROA is correctly
> validated, all the way. If a ROA is gone, it doesn't cause an outage.
> It causes lack of protection. It's not a competition which validator
> can validate the most ROAs, the competition should be to get things
> *right*.
> 
> If something doesn't validate correctly, drop it. Drop all it depends
> on. If all ROAs from a RIR are gone for an hour or a day, it's not
> the end of the world. It's not an outage.
> 
> We need to make sure the entire ecosystem gets things right, correct,
> and have procedures in place to do things right, all the time. Other
> parts of the Internet ecosystem had teething problems in the
> beginning, but they worked it out. RPKI ecosystem needs to do the
> same.
> 
> The goal of the validator should be to validate. It should be picky.
> It should throw things away that doesn't look right. You're
> advocating for something else, and I don't understand why.

Two separate things.

First, you will notice that I talked about delaying modifications, not
refusing modifications. I strongly believe that a change in
invalidation strategy should be discussed in the community and the best
possible strategy found. We are talking about security here which is
never easy and the obvious answers are frequently wrong.

For instance: If a ROA is found to be invalid, should all the ROAs
published by the issuing CA and its child CAs be dropped or should not
in fact all statements made for resources owned by the CA be filtered
completely, even those made by unrelated CAs? Only the latter covers
for cases where a route might accidentally become invalid because its
prefix was authorized to two or more CAs.

Further, the current proposal suggests to reuse non-expired cached data
instead. Routinator, for example, does not in fact do that and needs
significant changes to its architecture to make that possible. Do I
want to make those changes before the community agrees that using old
objects is a good idea? Doing so may make it easier to block object
revocation. Do we prefer that over just not having certain objects at
all? I don’t think there is consensus yet.

Second and entirely independent of the first, the matter of the
validity of that ARIN manifest. Some of us have concluded that it
didn’t need rejecting because it was not in fact invalid. While it was
indeed encoded in a way that is not allowed by the respective specs for
encoders, the specs instruct decoders to be more forgiving. Such
forgiveness is a decision an implementer makes. The particular mistake
did not alter the meaning of these fields or endangered the
cryptographic integrity of the certificate as an RPKI resource
certificate, hence the decision to accept such certificates.

This assessment might be different were the certificate used in another
context and thus it may indeed need to be rejected by a general purpose
library, but a library that only and exclusively deals in RPKI resource
certificates does not.

Should it be picky nonetheless and drop anything that looks vaguely
suspicious? Would you be surprised to hear that if we indeed did that,
out of the currently 175,330 VRPs, a mere 3,571 would be left. Even if
we interpreted unclear RFCs in favour of implementers, we still would be
down to 54,258 or about a third.

Would dropping all these VRPs out of caution make the Internet more
secure? It is kind of hard to argue for that. I believe that whether a
security system in itself is more or less secure is an entirely
irrelevant question. It needs to be assessed in terms of whether
it makes the thing it serves more secure. And so need concrete
implementations and all the decisions made along the way.

Kind regards,
Martin

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 27 Aug 2020 15:47:47 +0200 (CEST)

> 	Did you mean a) “drop it” or b) “drop it with very clear and specific messages that allow for ready identification of the issue” ?

I just take for granted that if something is dropped because it was 
incorrect, there is a diagnostic message generated so it can be fault 
found. so "b)".

> 	(I’m unaware how we get to 'work out the teething issues’, as you 
> suggest, unless you meant the latter and its detailed messages…)

Of course.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 27 Aug 2020 15:50:14 +0200 (CEST)

> down to 54,258 or about a third.

That's fine, let's then fix those. What's wrong with them? Has this been 
fed back to the people responsible for them?

---

From: Job Snijders <[email-masked]>
Date: Thu, 27 Aug 2020 14:28:27 +0000

Dear all,

It pains me to write this email. It appears there is an increasingly
acrimonious situation in which RIPE NCC, Cloudflare, and NLNetLabs
representatives not only produce and publish insecure software, but also
argue towards erosion of the robustness of the object security RPKI
depends on.

I'm drawing harsh conclusions: the reality is that we are now 6 months
into /what should've been/ a simple bug report, but turned into a trench
war. Folks are digging in their heels deeper.  Attempts in this group to
bridge the knowledge gap have failed so far.

On Wed, Aug 26, 2020 at 08:24:42PM +0200, Martin Hoffmann wrote:
> To the best of my knowledge, Routinator and the RIPE NCC RPKI
> Validator handle manifests according to the specifications laid out in
> the relevant standards track IETF documents. 

The implementers of RIPE NCC's validator, Routinator, and OctoRPKI
entirely missed the point of WHY RPKI Manifests exist at all. The bigger
picture is ignored, one can't look at normative terms in a vacuum.

I quote from the INTRODUCTION of RFC6486:

    "A manifest is intended to allow an RP to detect unauthorized object
    removal or the substitution of stale versions of objects at a
    publication point."

A Manifest makes it possible for a validator software to react sanely
when data tampering is detected. Manifests exist to *protect* both the
issuing CA and the RP, failure to acknowledge the purpose of manifests
is akin to the famous quote "the operation was successful - but the
patient died". Did any CA ever wish for an incomplete view of their
routing intentions to be transformed into routing decisions? Zero CAs
want this.

One has to look further than the normative terms, one has to realize
what the implications are to routing in the global system and inevitably
the conclusion is to err on the side of caution. To be cynical about
what data is provided via an untrusted network input channel. Why
implement a virus scanner, which can detect virus files, but
subsequently doesn't do anything about it?

Manifests are the *only* mechanism to verify a publication point's
completeness and integrity. Neither Routinator nor RIPE NCC's software
attach any consequence to integrity issues at a publication point. Both
continue to emit as many VRPs as possible, regardless of whether the
publication point is complete to begin with! 

The datastructure of Route Origin Authorizations (ROAs) allows only a
single origin ASN per .roa file, this means network operators who wish
to grant permission to multiple ASNs (a common example: their own and
their customers' ASNs) to originate parts of their IP space, they *have*
the create multiple .roa files. The IP Block owner's routing intentions
can only be considered when the full bundle of .roa files is available.

Logically, when some .roa files are missing (which according to a valid
current manifest must be present), the remaining .roa files at the
publication point become useless as they represent an *incomplete*
overview of routing intentions; even worse those files flip from
'useless' to 'dangerous' when they are injected as VRPs into the
operator's routing system.

Manifests are analogous to to Debian's "Release + Release.gpg" APT
archive concepts. APT (or yum/dnf) do *not* proceed to install packages
when critical dependencies are missing, or when the SIGNED checksums do
not match the checksum of the downloaded .deb file.  An administrator
has to *explicitly* override (-y --force) to install such packages when
dependencies or checksums don't match.

Let me demonstrate what happens when I cherry-pick just a few words you
wrote, and withhold some of your other words. You wrote this email:
https://mailarchive.ietf.org/arch/msg/sidrops/7JxOCNBvYbwDHL7hcPHsfvxto0Q/

*** start of modified email ***
    On Wed, Aug 26, 2020 at 08:24:42PM +0200, Martin Hoffmann wrote:
    > Routinator and the RIPE NCC RPKI Validator have issues.
*** end of modified email ***

Do you see the issue now? I didn't even change the order of your words,
I merely withheld some of the text you wrote, and the resulting text is
entirely contradictory to what you intended to write!

Let's be honest, neither RIPE NCC nor NLNetLabs have real experience
using RPKI ROV 'invalid == reject' in their own networks. RIPE NCC so
far has refused to implement ROV in AS 3333 out of fear, and NLNetLab's
own ASN is a simple single-homed stub network. Why are both
organisations ignoring the community's pleas to fix a security issue?
Why the hubris? Do you really think you know better? Why does Alexander
Band say that fixing this is "not a priority", why is RIPE NCC refusing
to commit a one-line patch to fix their validator?

Is loss of face the issue? The longer the delay to provide a fix, the
longer NLNetLabs and RIPE NCC keep hurting their users (and dependents).
Is this what one calls 'good for the Internet'? The issue was brought to
attention MONTHS [1] ago, it should've been a few days to get it patched.

> after such consensus has been achieved.

The only ones arguing against the consensus are RIPE NCC and NLNetLabs
employees. Go figure. Staff and knowledge were exchanged between the two
software houses, a path is visible how the misconceptions continued to
proliferate. It is not too late to change course, but catch-up is
needed.

Believe it not, RIPE NCC, Cloudflare, and NLNetLabs are now at an
existential crisis: your credibility is on the line. Are you going to
produce routing security software which actually improves security, or
not? Will you attempt to absorb decades of PKI and X.509 experience, or
throw it all in the wind? 

Currently routinator + ripe ncc's validator + octorpki set their users
up for failure. Operators using these softwares ARE AT NEEDLESS RISK. 

Regards,

Job

[1]: https://github.com/NLnetLabs/routinator/issues/319
https://github.com/RIPE-NCC/rpki-validator-3/issues/232
https://github.com/RIPE-NCC/rpki-validator-3/issues/158
https://github.com/cloudflare/cfrpki/issues/38

---

From: Tim Bruijnzeels <[email-masked]>
Date: Thu, 27 Aug 2020 16:32:40 +0200

Hi John, all,

> On 26 Aug 2020, at 16:54, John Curran <[email-masked]> wrote:
> 

fwiw, we have automated tests for Krill where we set it up with an embedded TA and a CA which produces a bunch of ROAs. We then expect the same set of VRPs to come out on the other end of:

- fort 1.1.3
- octorpki v1.1.4
- rcynic buildbot-1.0.1544679302
- routinator 0.7.0-pre
- rpkiclient VERSION_0_3_0
- RIPE NCC RPKI Validator 3.1-2020.07.06.14.28

We try to use 'strict' modes where available.

The set up is based on GitHub actions, Terraform and docker. The details are probably not interesting to this mailing list, but if CA or RP implementors want to have more details please feel free to contact me directly.

Tim

---

From: Stephen Kent <[email-masked]>
Date: Thu, 27 Aug 2020 11:07:50 -0400

Martin,
>> secure for, I’d like to argue, no actual gain.

When I the WG chairs tell me that I misunderstood the WG consensus re 
strict correctness then I will revisit this topic. However, the recent 
messages from Mikael and Job suggest that your perception of WG 
consensus is not accurate. This issue is one that needs to be decided by 
network operators, not by developers of RP software. I agree with the 
observations made by Mikael and Job, i.e., that requiring strict 
conformance with manifest rules is the preferred approach.

>> CA or updates to the CA because of a broken GBR makes this list less

I suspect we disagree on what constitutes "correct." A correctly 
functioning CA does not publish objects with bad signatures or format 
errors, use certs that have expired, does not fail to replace expired or 
stale objects, does not include objects at pub points that are not on 
the manifest, etc.

>> acceptable, as it effectively blocks the introduction of new types

No, it does not. What I suggested is that when a new object is proposed, 
it is the responsibility of the those proposing the new object to 
explain how it will be incrementally deployed. That explanation belongs 
in the RFC defining the new object, and in an updated version of 6486 
will need to be generated. We have no good examples of new objects that 
provide a basis for describing how to accommodate incremental 
deployment, and thus no basis for defining such mechanisms at this time. 
It might be the case that a new object will be defined that requires the 
CA to maintain a separate pub point using some newly-defined SIA 
pointer, indicting that the new pub point contains the new object and 
thus RPs that don't know how to process the object MUST NOT follow that 
pointer. There will need to be agreement on how long a CA MUST maintain 
the old pub point, etc. But, absent a concrete example of a new object 
type that warrants this sort of effort it is premature to write a spec.

>> benefit in rejecting a CA because someone swapped out an object I don’t
In X.509 certs we mark extensions as critical, or not. An extension 
marked as critical will cause a cert to be rejected by an RP that does 
not know how to process that extension. One might revise the generic 
signed object definition (RFC 6488)  introduce a similar flag. But, 
first, we would have to figure out how to incrementally deploy the new 
signed object format, with a new version number, etc. I hope you see why 
this approach to incremental deployment of new object types probably 
would entail more that a revision of 6486.

>> available and correctly signed data is skipped because of issues with

The RPKI was designed to support origin validation first, and BGPsec 
second. The set of objects that were defined are intended to support 
these two functions. If the WG decides to extend the set of supported 
functions it needs to take a hard look at a wide range of RFCs that will 
be affected, not just 6486.

Steve

---

From: Nick Hilliard <[email-masked]>
Date: Thu, 27 Aug 2020 16:16:30 +0100

> conformance with manifest rules is the preferred approach.

I concur - strictness is generally desirable when dealing with data 
integrity.

In this case, the manifest problem was harmless and the data integrity 
was unaffected.  Speaking as an operator and rpki consumer, what I don't 
have good handle on is whether the manifest checks that would be 
required to detect the problem that happened at ARIN would also detect 
actual manifest integrity problems.  If the answer to this is yes, then 
there is no question about whether these checks should be added to the 
code bases that currently lack them.  Otherwise they are a nice-to-have, 
and I'd very much like to see them added.

Nick

---

From: Randy Bush <[email-masked]>
Date: Thu, 27 Aug 2020 08:40:50 -0700

> I agree with the observations made by Mikael and Job, i.e., that
> requiring strict conformance with manifest rules is the preferred
> approach.

otherwise, why the heck would we make them.  this non-compliance is
destructive.  i try to save civil disobedience for BLM etc.

> errors, use certs that have expired, does not fail to replace expired
> or stale objects, does not include objects at pub points that are not
> on the manifest, etc.

this would seem to be pretty obvious, CA 101.

> No, it does not. What I suggested is that when a new object is
> proposed, it is the responsibility of the those proposing the new
> object to explain how it will be incrementally deployed.

cf router keys, ASPA

Job Snijders wrote:
> representatives not only produce and publish insecure software, but
> also argue towards erosion of the robustness of the object security
> RPKI depends on.

you are too polite.  it is destructive of the internet and therefore
quite dangerous.

>     removal or the substitution of stale versions of objects at a

seemed pretty simple at the time

so we have 6486-bis in progress.  but if some vendors have the arrogance
to continue these same games, why should we bother?

> Did any CA ever wish for an incomplete view of their routing
> intentions to be transformed into routing decisions? Zero CAs want
> this.

sad to say, given the "we can do whatever the hell we want" dogma, i
have my suspicions otherwise.  or is it just lack of ops clue?

> entirely contradictory to what you intended to write!

yes.  but it is correct!  :)

> existential crisis: your credibility is on the line.

no, for some of us, it's gone.

randy

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Thu, 27 Aug 2020 17:30:54 +0000

-----Original Message-----
From: Mikael Abrahamsson
Sent: Thursday, August 27, 2020 5:30 AM

> If a ROA is gone, it doesn't cause an outage. It causes lack of protection.

Suppose a provider has a ROA for a large prefix and subdivides it into more specifics.
It allocates these more specifics to other ASes and those ASes publish ROAs for
those more specifics.

If the ROAs for the more specifics are gone, then the less specific ROA
for the larger prefix will invalidate announcements for the more specific prefixes.

That's an outage.

Regards,
Jakob.

---

From: Randy Bush <[email-masked]>
Date: Thu, 27 Aug 2020 10:49:42 -0700

> If the ROAs for the more specifics are gone, then the less specific
> ROA for the larger prefix will invalidate announcements for the more
> specific prefixes.

yep; simply stated.  and perhaps the more likely common case.

a less common case could be if i am doing a provider switch where my
upstreams do my announcements.  for "make before break" i would have
roas for both providers.  if the roa for the one which is currently
announcing is dropped, kaboom.

similarly a transfer of ip space from one AS to another.

the correctness of CA publication point data and the rigor and
reliability of RP collection and propagation to routers is critical.
and the threat of half-assed vendor software is a far bigger and more
real threat to operations than the dutch court attack boogeyperson.

randy

---

From: Benno Overeinder <[email-masked]>
Date: Thu, 27 Aug 2020 22:39:28 +0200

> proliferate. It is not too late to change course, but catch-up is

Please stick to technical arguments in the discussion on the mailing
list, and omit insinuations and false extrapolations.

Thank you.

-- Benno

-- 
Benno J. Overeinder
NLnet Labs
https://www.nlnetlabs.nl/

---

From: Tim Bruijnzeels <[email-masked]>
Date: Fri, 28 Aug 2020 08:33:04 +0200

Dear WG,

Please allow me to zoom out a bit..

Much of this discussion is caused by:
a) different interpretations of a spec
b) differing views on Postel's Law and how it should (not) be applied in a security context

@a: the spec

While I write RPKI CA code, I was not intimately involved in writing validation code related to the particular issue in ARIN's manifest. The discussion however was painful. Bug reports were created. They were not blindly followed. People double checked. People argued that it should be discussed in the IETF, and not on GitHub. People had differing interpretations on what encodings are okay. Painful as it was I believe that in the end a discussion based on the content of this particular issue did take place in this group. ARIN fixed their manifest and the RP implementers planned fixes for inclusion in upcoming releases.

@b: the robustness principle and security

Should any and all violations of spec lead to the rejection of objects?

Arguments can be made, are being made, that in the context of security the answer this question is 'yes'. It makes for a simpler world, and simple is good when it comes to security. There is no slippery slope anymore. Taking this even further one can argue (in the context of manifests) that if an RP knows that it has *all current products* of a CA, and the CA clearly messed up even one object - they should no longer be trusted to know what they are doing.

The counterarguments are that we have existing deployments, and that there are shades of gray in this world.

The bug in ARIN's manifest seemed to me to be a violation of spec, but not necessarily a critical security issue. The algorithms are all specified in the relevant RPKI RFCs - hashes and signatures providing proof of possession worked out. The dialect may have been funny, but the intent was clear.

This is not the only existing encoding issue. The bouncy castle library used by both the RIPE NCC and Lacnic code bases produces CMS that is BER, not DER, encoded. The specs say that DER MUST be used. While this is is the case OpenSSL and other implementations will happily decode BER encoded CMS objects - most other CMS implementations use BER after all. This is a violation of spec, but is this a critical security issue?

Other examples were found over the years. And I have no doubt that other deviations from the specs exist today.

Some went unnoticed for years (9 years of deployment) - and are only being found now that more independent RP tools are being built.

This is a good thing. But does this mean that we should reject all objects that do not conform to spec today? That would leave alarmingly few VRPs. Should we apply 'shades of gray' - and allow certain deviations from spec if we feel they are not harmful, indefinitely? Or should we aim for full compliance in years to come, but give CAs time to fix existing issues?

I would like to believe that it's okay to have different viewpoints on this during the discussion phase. And while I may have my own opinions of what I feel is right, I am quite willing to change my mind if we can discuss these matters openly and constructively.

Kind regards,

Tim

> On 27 Aug 2020, at 16:28, Job Snijders <[email-masked]> wrote:
> 
> Dear all,
>

---

From: Tim Bruijnzeels <[email-masked]>
Date: Fri, 28 Aug 2020 08:40:45 +0200

Hi,

> On 27 Aug 2020, at 19:49, Randy Bush <[email-masked]> wrote:
> 

Indeed, I believe that, if the publication point for a Certification Authority in the RPKI is rejected for whatever reason, then all VRPS for prefixes overlapping the resources issued to them in their CA certificate for said publication point should be filtered out.

Note that most RP software already implements SLURM - and this kind of post-validation filtering can be done in a very similar way.

It is important to only do this for VALID CA certificates with invalid publication points. This is what I said above, but I just want to be overly clear here.. because if one would start doing this for invalid over-claiming CA certificates then this would of course open a wide vector for people to DoS legitimate statements about other people's address space.

Tim

---

From: Tim Bruijnzeels <[email-masked]>
Date: Fri, 28 Aug 2020 09:45:39 +0200

Hi,

> On 27 Aug 2020, at 17:07, Stephen Kent <stkent=[email-masked]> wrote:
> 
> Martin,
>> ...
> When I the WG chairs tell me that I misunderstood the WG consensus re strict correctness then I will revisit this topic. However, the recent messages from Mikael and Job suggest that your perception of WG consensus is not accurate. This issue is one that needs to be decided by network operators, not by developers of RP software. I agree with the observations made by Mikael and Job, i.e., that requiring strict conformance with manifest rules is the preferred approach.
> 
> 
> I suspect we disagree on what constitutes "correct." A correctly functioning CA does not publish objects with bad signatures or format errors, use certs that have expired, does not fail to replace expired or stale objects, does not include objects at pub points that are not on the manifest, etc.

I agree that a correctly functioning CA will produce a sound set of objects. Also it is highly unlikely that a CA will just publish an expired ROA. The much more likely scenario is that a CA is unable to make or publish updates and the MFT and CRL go stale, and the MFT EE expires, way before any ROA or issued cert would.

That being said there are scenarios beyond the control of a CA that can lead to invalid objects:

1) rsync

If the RP does rsync only, they may get an inconsistent data set. Transactions are not supported here. They just get an old MFT and new CRL or vice versa, or they may get a new MFT but not the new object, etc. This is a race condition that happens infrequently when RPs fetch during publication. It is rare, but it does happen.

I remember that many years ago I wrote code in the, then, RIPE NCC validator 2 to catch this issue. And only process publication points if the set was consistent. This was an allowed, but not mandated, interpretation of 6486.

Note that with RRDP we have deltas. Solving this issues was one of its design goals. So, phasing out rsync could mitigate this.
(yes, I realise now that the co-chairs asked me to re-submit the I-D for this, and I still need to do it)

2) resource changes

If the parent removes any resources from the child before the child has had a chance to clean up objects then this will lead to invalid objects. Being strict that *all* objects must be valid will then lead to rejection of the CA's products. Even if it is only for a few hours.

I believe that this issue could be greatly mitigated if RFC6492 would be extended with a notion to give children a warning about planned resource removals. The length of this warning time being a function of the contact frequency of child CAs and the depth of the CA hierarchy. But, if this frequency could also be stipulated - say SHOULD every 10 minutes? Then a warning time of just a few hours would already be plenty.

However, this is a separate discussion. I do not suggest that we block progress on 6486bis for this. I am just saying that if strict checking for *all* objects is where the consensus goes then the WG must be aware of these consequences and accept them.

> No, it does not. What I suggested is that when a new object is proposed, it is the responsibility of the those proposing the new object to explain how it will be incrementally deployed. That explanation belongs in the RFC defining the new object, and in an updated version of 6486 will need to be generated. We have no good examples of new objects that provide a basis for describing how to accommodate incremental deployment, and thus no basis for defining such mechanisms at this time. It might be the case that a new object will be defined that requires the CA to maintain a separate pub point using some newly-defined SIA pointer, indicting that the new pub point contains the new object and thus RPs that don't know how to process the object MUST NOT follow that pointer. There will need to be agreement on how long a CA MUST maintain the old pub point, etc. But, absent a concrete example of a new object type that warrants this sort of effort it is premature to write a spec.
> 
> 
>>>> Instead I believe it
> In X.509 certs we mark extensions as critical, or not. An extension marked as critical will cause a cert to be rejected by an RP that does not know how to process that extension. One might revise the generic signed object definition (RFC 6488)  introduce a similar flag. But, first, we would have to figure out how to incrementally deploy the new signed object format, with a new version number, etc. I hope you see why this approach to incremental deployment of new object types probably would entail more that a revision of 6486.

The current version of 6486 allows RPs to treat individual objects as invalid. -bis is trying to change that to say that the whole set of objects must be valid. This changes the consequences for the deployment of new object types, so it is right to consider this now.

The most likely next object type would be the ASPA objects. I don't think that RPs should reject all ROAs because they do not yet understand ASPA. Given that object types in the RPKI get distinct file names, and RPKI signed objects get distinct OIDs I think it would be reasonable to say that RP software can ignore object *types* that it does not understand. I would still advocate then that the RPs do check for the presence and hashes of these objects as stipulated by the signed MFT, but not consider their content.

If we don't then the consequence will be that new object types only become deployable after a significant percentage of RP has been upgraded to version that understand the new type.

Now, if all operators say that they are fine with this: let things go unknown for everyone who did not upgrade, and force them to do so.. sure. But it is right to consider this now and make a conscious choice. Oh and BTW.. for RPKI use cases where we only have VALID/INVALID but no NOT FOUND, such as BGPSec, this would be problematic.

> The RPKI was designed to support origin validation first, and BGPsec second. The set of objects that were defined are intended to support these two functions. If the WG decides to extend the set of supported functions it needs to take a hard look at a wide range of RFCs that will be affected, not just 6486.
> 
> Steve
> 
>

---

From: Randy Bush <[email-masked]>
Date: Fri, 28 Aug 2020 05:00:34 -0700

excuse, but a bit pre caffeine here

> I believe that, if the publication point for a Certification Authority
> in the RPKI is rejected for whatever reason, then all VRPS for
> prefixes overlapping the resources issued to them in their CA
> certificate for said publication point should be filtered out.

complexity and cleverness produce fragility.

first, you may not know the prefix from the failed pp; it failed.
second, getting you to think that you do could be a nice attack.

> It is important to only do this for VALID CA certificates with invalid
> publication points.

i am not sure what an invalid pp is.  one where none of the referring
object(s) are cryptographically validatable to a TA?

> if one would start doing this for invalid over-claiming CA
> certificates then this would of course open a wide vector for people
> to DoS legitimate statements about other people's address space.

good reasone not to do it at all.

randy

---

From: Tim Bruijnzeels <[email-masked]>
Date: Fri, 28 Aug 2020 14:10:42 +0200

Hi,

> On 28 Aug 2020, at 14:00, Randy Bush <[email-masked]> wrote:
> 

Sure, but:
1) it may not be that complicated (SLURM does the same)
2) as you said, if you only reject the objects under one CA and the same resources exist elsewhere this can lead to invalids

> second, getting you to think that you do could be a nice attack.

You were looking at all the objects listed on a manifest you found for a valid CA certificate. You know which resources the valid CA certificate holds.

> object(s) are cryptographically validatable to a TA?

I thought the discussion here was that if any of the objects for a mft is invalid, then all objects should be treated as invalid. By publication point I mean the MFT and all objects listed on it,

> good reasone not to do it at all.

As above.. I just wanted to make it abundantly clear.. you start off with a validated CA certificate.

---

From: Tim Bruijnzeels <[email-masked]>
Date: Fri, 28 Aug 2020 14:14:31 +0200

> On 28 Aug 2020, at 14:10, Tim Bruijnzeels <[email-masked]> wrote:
> 
> Hi,
> 

@2: Announcements with state RPKI Invalid w.r.t. ROV (not other invalid objects)

---

From: Stephen Kent <[email-masked]>
Date: Fri, 28 Aug 2020 10:00:14 -0400

Tim,
> @b: the robustness principle and security

Jon Postel's robustness principle is appropriate for promoting 
interoperability, but it is not a good principle for security. Security 
requires discipline. In my experience, people are not very disciplined, 
and thus we have recurring security problems. Experience shows that 
without feedback, people are not motivated to become more disciplined, 
and security problems become worse. Thus I feel is is critical for the 
community to reject RPKI repository data that does not conform to the 
RFCs, as a way of providing feedback and improving discipline.

The revisions we have made to the manifest RFC are intended to remove 
ambiguity and to impose strict standards for how CAs manage the data at 
a pub point. We all seemed to agree that removing ambiguity is desirable 
and necessary, but not everyone seems to believe that strict standards 
are desirable. I defer to the WG chairs to decide what the consensus 
position is in this respect. However, at this time, suggesting that the 
manifest RFC needs to specify how new objects can be incrementally be 
introduced into the repository system is a distraction. I can envision 
one way to do that, as noted in a prior message, but that would entail 
changes to several other RFCs, and will delay, substantially, progress 
on revisions to the manifest RFC.

I am very bothered by the observation that, if were were to strictly 
enforce the requirements imposed by the RPKI RFCs, then the number of 
verified routes would substantially decrease. This seems to suggest that 
it is more important to be able to cite big numbers of verified routes, 
to convey a sense of RPKI deployment success,  than to "encourage" CAs 
to get their act together and follow the standards. That, I believe, is 
a mistake.

Steve

---

From: Stephen Kent <[email-masked]>
Date: Fri, 28 Aug 2020 10:25:40 -0400

Tim,
> Arguments can be made, are being made, that in the context of security the answer this question is 'yes'. It makes for a simpler world, and simple is good when it comes to security. There is no sli…

The issue is not whether a CA that makes some types of errors cannot be trusted to know what they are doing. Rather, the issue is that, absent feedback, a CA that persistently makes such errors will do so forever, forcing all RPs to accommodate these errors. This IS a slippery slope; it paves the way to more serious errors to be tolerated, based on an established pattern of accommodating sloppy behavior.

Steve

---

From: Job Snijders <[email-masked]>
Date: Fri, 28 Aug 2020 14:40:17 +0000

On Fri, Aug 28, 2020 at 02:10:42PM +0200, Tim Bruijnzeels wrote:
> You were looking at all the objects listed on a manifest you found for
> a valid CA certificate. You know which resources the valid CA
> certificate holds.

Suggested text, feedback welcome:

================

If a CA's publication point's RPKI data is invalid, a Relying Party MUST
add all IP address resources listed on the certificate's issuer to a
Prefix Filter. The Relying party MUST consider any VRPs derived from any
Trust Anchor to match with this Prefix Filter if the VRP prefix is equal
to or covered by any of the Prefix Filter prefix. The following prefixes
MUST NOT be added to the Prefix Filter: 0.0.0.0 and ::/0

================

In other words, if a valid current manifest at a publication point
demonstrates files are missing, the resources listed in the
AIA's sbgp-ipAddrBlock extension become an output filter.

If under RIPE's TA a CA delegates authority for 85.xx.0.0/16 to a
publication point, and a ROA file is missing, and 85.xx.0.0/16 also
exists somewhere under ARIN's TA, the RP should omit to produce VRPs
covering 85.xx.0.0/16 or more specific. This is the safest approach.

By excluding 0.0.0.0/0 and ::/0 we prevent one TA from taking down
another TA, and encourage a (hopefully) healthy approach to operational
intermediate certificates which exist to narrow the TA's blast radius.

Kind regards,

Job

---

From: Tim Bruijnzeels <[email-masked]>
Date: Fri, 28 Aug 2020 16:54:56 +0200

Steve,

> On 28 Aug 2020, at 16:00, Stephen Kent <stkent=[email-masked]> wrote:
> 
> Tim,
>> ...
>> 
> Jon Postel's robustness principle is appropriate for promoting interoperability, but it is not a good principle for security. Security requires discipline. In my experience, people are not very disciplined, and thus we have recurring security problems. Experience shows that without feedback, people are not motivated to become more disciplined, and security problems become worse. Thus I feel is is critical for the community to reject RPKI repository data that does not conform to the RFCs, as a way of providing feedback and improving discipline.

I believe that strict conformance is a laudable goal, but I also think that given that none of the RP software has been doing this until today CAs should be given time to fix things.

If we ask all CA implementors to test their repositories with strict compliance validation options, then I believe that setting a flag day for strict compliance in future can get us there without impacting current operations.

Yes, it implies that we accept that things like using the wrong type of 'String' for a Subject, and even BER vs DER encoding, are not critical security errors.

> The revisions we have made to the manifest RFC are intended to remove ambiguity and to impose strict standards for how CAs manage the data at a pub point. We all seemed to agree that removing ambiguity is desirable and necessary, but not everyone seems to believe that strict standards are desirable. I defer to the WG chairs to decide what the consensus position is in this respect.

No, I do believe that strict standards are desirable.

What I was advocating is that MFT are treated strictly by RPs to determine that they have the full set of statements made by a CA. This protects against issues between the CA and Publication Server, and between a Publication Server and RP.

Going beyond this by saying that if a CA makes any mistake we should disregard everything they say has implications that I outlined in my other email. I think there are scenarios where security is not impacted (e.g. GBR vs ROA). But I can live with it as long as the WG understands and accepts that there are at least two (temporary) failure scenarios where the issuing CA is not to 'blame': inconsistent (rsync) fetches and resource removal timing issues.

> However, at this time, suggesting that the manifest RFC needs to specify how new objects can be incrementally be introduced into the repository system is a distraction. I can envision one way to do that, as noted in a prior message, but that would entail changes to several other RFCs, and will delay, substantially, progress on revisions to the manifest RFC.

As I said in my other email, the suggested change that all listed objects in a MFT are valid makes incremental much harder than before. Therefore it is something that needs to be considered.

I am not going to repeat what I said there, but maybe you can reply to the specific points I raised there, and suggestion I made that incidentally would not cause any delay on this revision.

> I am very bothered by the observation that, if were were to strictly enforce the requirements imposed by the RPKI RFCs, then the number of verified routes would substantially decrease. This seems to suggest that it is more important to be able to cite big numbers of verified routes, to convey a sense of RPKI deployment success,  than to "encourage" CAs to get their act together and follow the standards. That, I believe, is a mistake.

See above, I think we should aim for a flag day

Tim

---

From: Randy Bush <[email-masked]>
Date: Fri, 28 Aug 2020 07:56:13 -0700

> If a CA's publication point's RPKI data is invalid

"invalid" is already an overloaded magic word in this space; not
cryptographically validatable up to a TA, and bgp announcement not
finding a matching roa.

can we come up with another term for a failed (for many reasons) fetch
of a PP's data?

> a Relying Party MUST add all IP address resources listed on the
> certificate's issuer

what certificate?

> to a Prefix Filter

what is a prefix filter?

> The Relying party MUST consider any VRPs

an RP cache knows if a PP was not fetchable, but does not have any VRPs
(yet), just ROAs.  a router has VRPs, but has no idea what happened
between the RP cache and the PPs it tried to fetch.

i suspect this whole thing is a capybara hole we just do not want to
explore, especially when we have software out there which is <bleep>.

though i would not mind looking at a concise and simple statement of
what it is you're trying to do here.

but do consider the PPs of a CA whose INRs are delegated from multiple
parents, and which, in turn, delegates to other CA(s).

randy

---

From: Job Snijders <[email-masked]>
Date: Fri, 28 Aug 2020 15:25:05 +0000

Dear Stephen,

On Fri, Aug 28, 2020 at 10:00:14AM -0400, Stephen Kent wrote:
> verified routes would substantially decrease. 

>From my observations, OpenBSD rpki-client produces 514 VRPs fewer than
some of the other validators, but still totals at 171,643 VRPs related
to the global routing system (currently 895,143 routing table entries,
ipv4 and ipv6 combined).

In the grand scheme of things those 500 VRPs to me are not 'substantial'
but rather "just out of luck", knowing that any attempts to 'repair' or
'salvage' those 500 VRPs puts the remaining 171,643 route origin
authorizations at risk.

This is a good noise level, and if we come up with additional ideas
to improve strictness that krank it from 500 to the low thousands, we
are still in great shape. Also knowing that whatever triggers further
decreases can probably easily be remedied by the relevant TA or CA.

Kind regards,

Job

---

From: Lukas Tribus <[email-masked]>
Date: Fri, 28 Aug 2020 19:36:33 +0200

Hello,

> verified routes would substantially decrease.

This I believe is at the core of the disagreement.

There is a faction that argues with points that sound a lot like "if
in doubt, consider the ROA valid, because we NEED to emit a VRP",
"CA's are too big to fail" and equivalents of "but if HTTPS fails,
people will just use HTTP so we end up being less secure" (as in
"better have a stale VRP than no VRP at all"). This is a mindset thing
and goes beyond the specific issue about manifest invalidation. For
example some RP's accept stale objects by default.

Err'ing on the side of caution seems to mean "consider the ROA valid,
so we emit the VRP" for some, which as a network operator I find
deeply troubling.

Let's be honest: there are not a lot of incentives for the CA's to fix
underlying issues if the 3 major RP's accept those ROA's.

Yes, sure, all of this is an oversimplification and the devil is in
the details, which do need clarification.

But we cannot be guided by a fear of decreasing VRP numbers when
improving validation. Immaggine if browsers would do that with WebPKI
(and that is actually causing direct outages for users, as opposed to
ROV where outages are more likely caused by the *presence* of VRPs
than by its absence).

-- lukas

---

From: Jay Borkenhagen <[email-masked]>
Date: Fri, 28 Aug 2020 16:00:29 -0400

Steve,

I would like to comment on two things you recently wrote to this list: 

Stephen Kent writes:
 > When I the WG chairs tell me that I misunderstood the WG consensus re:

I don't think viewing this as "network operators" vs "developers of RP
software" is a productive approach.

I am a professional network operator -- somebody pays me to do that.
I also happen to have some opinions on how a successful PKI should
operate, but so far no one has offered to pay me for those opinions.
If asked to use only the network operator part of my brain to say
whether strict conformance is the best approach here, I would answer
"Honestly, I don't care.  As a network operator I care only that the
VRPs I use in my network are as trustable and reliable as possible."
I certainly have opinions how _how_ to make the system as trustable
and reliable as possible, but I know others have thought about such
aspects far more than I have -- in particular folks who have
implemented RP software.

At the risk of making my comments here too personal (they are not
intended that way), my friend Job wears both of these hats: he is a
network operator, and he is involved in developing an RP
implementation.  That's great!  I value his opinions, and I am happy
to hear that you do, too.  But it would be fruitless to attempt to
determine which parts of Job's psyche most inform his opinions voiced
in this WG.  So rather than attempting to do so, I would like to ask
you to also value the opinions and contributions of other RP
implementors, particularly those who have been active in this
discussion.  Many network operators do not participate here in
SIDROPS, but presumably all RP implementors do.

 > type that warrants this sort of effort it is premature to write a spec.

The situation you describe here sounds horrible, trying to keep older
RPs from seeing newer objects they're not equipped to fully
understand.  A much more robust system would be one where RPs are not
thrown off by unknown object types retrieved from known publication
points and listed in manifests.  Clearly older RPs would not know what
use to make of newer objects, and thus would ignore them.  Network
operators interested in taking advantage of the new objects would
first need to upgrade their RPs in order to do so, but that's BAU.

Thank you.

						Jay B.

---

From: Nathalie Trenaman <[email-masked]>
Date: Sat, 29 Aug 2020 12:22:54 +0200

Dear all,

Last month, we released a “strict mode” for our validator. It takes into account 6486bis except the use of cached data, which I did not see consensus on yet (correct me if I’m wrong please). 
The reason why we did not release strict mode as default is because we first needed to have a clear picture what it was we were actually losing in terms of objects and the impact this would cause.
Let me put it very clear that none of us at RIPE NCC are objecting against any consensus. 

Where Job sees around 400 objects missing, we do see a lot more. Here you can see an instance of our validator in the default mode:
https://rpki-validator.ripe.net/trust-anchors <https://rpki-validator.ripe.net/trust-anchors>
While here you can see an instance running in strict mode:
http://alias.student.utwente.nl:9177/trust-anchors <http://alias.student.utwente.nl:9177/trust-anchors>

We have been monitoring the amount of missing objects and we see a difference of around 4500 objects. Mainly from the APNIC region. 
This is why I believe Tim’s suggestion for a flag day sounds reasonable to me. We have to inform CAs about the impact changing the behaviour of RPs has on them. 

Furthermore, I would really appreciate it if we can keep the discussion constructive. 

Regards,
Nathalie Trenaman
RIPE NCC

> Op 27 aug. 2020, om 16:28 heeft Job Snijders <[email-masked]> het volgende geschreven:
> 
> Dear all,
>

---

From: Job Snijders <[email-masked]>
Date: Sat, 29 Aug 2020 18:48:19 +0000

On Sat, Aug 29, 2020 at 12:22:54PM +0200, Nathalie Trenaman wrote:
> Last month, we released a “strict mode” for our validator. It takes
> into account 6486bis except the use of cached data, which I did not
> see consensus on yet (correct me if I’m wrong please).

I think you are wrong on the cached data consensus, but it doesn't
matter, it is a small implementation detail. If an implementer doesn't
want to take advantage of their local file cache, they don't need to.
RPs are expected to be able to function with empty caches as well.

> Where Job sees around 400 objects missing, we do see a lot more. 
> 
> We have been monitoring the amount of missing objects and we see a
> difference of around 4500 objects. Mainly from the APNIC region.

The difference between rpki-client and ripe ncc validator
'insecure-mode=off / rsync-only' seems to be about 2,500 VRPs. However,
the ripe ncc validator logs 5 only errors: https://rpki.meerval.net/trust-anchors/monitor/2

One of the errors appears to be logged at "2020-08-29 18:08:48" and
indicates: "CRL next update was expected on or before 2020-08-29T22:46:19.000Z"
It is not clear why it errors on things still in the future. The 2,500
VRP difference might be caused by a software issue rather than a
specifications issue?

For example, I can't find a reason why a VRP for Prefix:
203.163.202.0/23 Maxlength: 24, Origin: AS10085, TA: APNIC is not
emitted by the ripe ncc validator. It is also curious to see RRDP
fetches are performed while 'rpki.validator.rsync-only=true' is set.

Are you sure the current version implemented things correctly?

> This is why I believe Tim’s suggestion for a flag day sounds
> reasonable to me. We have to inform CAs about the impact changing the
> behaviour of RPs has on them. 

A flag day to release a security update? Flag day's generally are used
for other types of events.

Regards,

Job

---

From: Stephen Kent <[email-masked]>
Date: Sat, 29 Aug 2020 14:58:17 -0400

Tim,
> I agree that a correctly functioning CA will produce a sound set of objects. Also it is highly unlikely that a CA will just publish an expired ROA. The much more likely scenario is that a CA is una…
I agree that these seem to be the more likely error modes, but there 
also was a discussion of encountering more than one CRL, for example.
> If the RP does rsync only, they may get an inconsistent data set. Transactions are not supported here. They just get an old MFT and new CRL or vice versa, or they may get a new MFT but not the new…
I agree that this may happen, even if one publishes all of the objects 
in a new directory and the  switches the pointer. But, in that case, the 
RP should consider this a failed fetch and retry. I believe that's what 
the RPSTIR software did (does?), so it's not a fatal error.
> (yes, I realise now that the co-chairs asked me to re-submit the I-D for this, and I still need to do it)
I'm still waiting to see a clear, and hopefully concise, description of 
how an RP verifies that the pub point data acquired via RRDP is 
constrained in the hierarchic fashion imposed by the rsync directory 
model, but that's a separate issue.
> However, this is a separate discussion. I do not suggest that we block progress on 6486bis for this. I am just saying that if strict checking for *all* objects is where the consensus goes then the…
yes, this is a separate discussion.
> The current version of 6486 allows RPs to treat individual objects as 
> invalid. -bis is trying to change that to say that the whole set of 
> objects must be valid. This changes the consequences for the 
> deployment of new object types, so it is right to consider this now. 
6486 allows an RP to behave either way, e.g., to accept some but not all 
inconsistencies to to reject them. So, it's not quite accurate to 
suggest that RP software that operates in accordance with 6486 will 
treat individual objects as invalid- it might, or it might not.
> The most likely next object type would be the ASPA objects. I don't think that RPs should reject all ROAs because they do not yet understand ASPA. Given that object types in the RPKI get distinct f…
Allowing RPs to ignore object types they don't understand prevents a CA 
from being able to convey the notion that a new object type is important 
(to that CA). I don't think this is a good strategy. It means that RP 
behavior will be ambiguous relative to new object types.
> Now, if all operators say that they are fine with this: let things go unknown for everyone who did not upgrade, and force them to do so.. sure. But it is right to consider this now and make a consc…

If we want to have a consistent and flexible approach to accommodating 
new objects I suggest the strategy I mentioned earlier. Define an 
additional SIA URI that points to a pub point (and manifest) where we 
can introduce the next version of the signed object format, one that 
includes a critical flag, analogous to X.509v3 extensions. This allows 
each CA to decide which object types have to be processed  by an RP in 
order for the whole pub point to be accepted vs. rejected. Note that 
this will require modifying a lot of RFCs, but it is a flexible, 
extensible approach to this issue.

I agree that even if we adopt the current 6486bis , for now, that a flag 
day is appropriate, and it shoudk be part of the document.

Steve

---

From: Stephen Kent <[email-masked]>
Date: Sat, 29 Aug 2020 15:11:18 -0400

Jay,
> SIDROPS, but presumably all RP implementors do.
The crux of the issue is whether it is preferable to reject all objects 
at a pub point if an error is detected (until the error is rectified), 
and thus potentially treat many more routes as having an unknown 
verification status, or to accept bits and pieces of data from a pub 
point with the potential that some routes will be treated as valid, even 
if they are invalid. To me, that is a value judgement to be made by 
network operators, not by developers of RP software. Job can pick 
whatever hat fits him best.
> first need to upgrade their RPs in order to do so, but that's BAU.

Experience with X.509 cert extensions suggests that the approach I noted 
is preferable. Saying that older RPs can ignore objects they don't 
understand introduces ambiguity in RP behavior, which is precisely the 
concern that motivated the effort to generate 6486bis.

Steve

---

From: Di Ma <[email-masked]>
Date: Sun, 30 Aug 2020 12:48:51 +0800

> 2020年8月30日 02:58，Stephen Kent <stkent=[email-masked]> 写道：
> 
> Tim,
>> Hi,
> I agree that these seem to be the more likely error modes, but there also was a discussion of encountering more than one CRL, for example.
> I agree that this may happen, even if one publishes all of the objects in a new directory and the  switches the pointer. But, in that case, the RP should consider this a failed fetch and retry. I believe that's what the RPSTIR software did (does?), so it's not a fatal error.

Yes. RPSTIR2 does do so, retrying.

It is difficult for an RP to determine if it is a rollover or a mistake when it sees new MFT and old signed objects coexist in the repository in question.

If it is a mistake, RP is supposed to consider those old objects invalid as per 6486bis.

But if it is during rollover, RPSTIR2 now has to keep retrying until things are seems ok. 

IMHO, I suggest it is the CA that does make-before-break by setting up a new publication point with new MFT and new objects, keeping old ones still valid. Once the new publication is all ready to service, CA replaces old SIA with new one. This is what I can see by far a  compromise to solve this issue.

Di

---

From: Stephen Kent <[email-masked]>
Date: Sun, 30 Aug 2020 15:59:54 -0400

Job,
>> verified routes would substantially decrease.
You didn't copy the rest of my message, in which I indicated that I felt 
we were worrying too much about the bad PR associated with having fewer 
routes verified, as a side effect of adopting stricter rules wrt 
manifest processing.
> decreases can probably easily be remedied by the relevant TA or CA.

I agree with your point.

Steve

---

From: Tim Bruijnzeels <[email-masked]>
Date: Mon, 31 Aug 2020 10:47:39 +0200

Hi,

> On 29 Aug 2020, at 20:48, Job Snijders <[email-masked]> wrote:
> 

To be clear: I suggested a flag day for encoding issues which are not *critical*. No CA is big enough to fail if there is a critical issue.

Fact is that if you take the position that any object which is not 100% specification compliant MUST be considered invalid, then NONE of the current validators are secure.

The ARIN issue was related to wrong encoding. The relevant RFC is pretty easy to misinterpret. But while there is a mandated encoding here, the error did not change the semantics. Furthermore the signature algorithm is defined by the RPKI RFCs - and the object has a whole could be validated. There never was a chance of different interpretations. There never was a chance of the signature being valid over modified content. So, wrong as this may have been, qualifying this as a critical issue is somewhat questionable. And given that the RFC was not very clear, it was right that RP implementers questioned the issue raised before implementing.

I could make equally bold statements on how routinator with the --strict option is in fact the only secure validator, because all others accept BER (not DER) encoded MFT and ROA objects. BER encoding is more complicated to decode - one issue in particular is that you do not necessarily know the size of compound values up front, and this has led to security issues in other applications. Surely, this is at least as bad, quite possibly worse. In practice, of course, we downloaded the objects and we can check their size before ending up here. So, while it is wrong, I would actually claim that this is not *critical*.

This should not stop us from improving. On the contrary. We can all learn from each other here. It is a very good goal to aim for 100% compliance, but it's going to require work from everyone to get there. This is why I suggested a flag day. Let's just focus our energy on working with each other, shall we?

- RP compliance checking

It is not easy to get to 100% checking in RPs. You can only test for things that you can think of - or things your library developers (openssl, bouncy castle etc) have thought of..

In days long past Rob Austein and Andrew Chi (then BBN) and I (then RIPE NCC) met to discuss validation corner cases. Andrew was able to generate all kinds of broken objects that our library simply could not generate (we could generate objects that we believe to be valid only). So, I was very grateful that he could do this and we could test our validation software. These objects are still used for regression testing in the RIPE NCC validator today. You can find them (100s of corner cases) here:
https://github.com/RIPE-NCC/rpki-commons/tree/master/src/test/resources/conformance

The ARIN encoding issue is not in there. Still, maybe this kind of set up can be useful for RP implementors again. It could be revived, extended, and hopefully be of use for all.

- CA compliance

Most implementations have code bases going back to 2011 or earlier. Most have automated tests in place using rcynic and/or the ripe ncc validator. Nowadays there are many more options available. My advice would be that all CA developers start testing with all RP implementations they can get their hands on, and use 'strict mode' where available.

As for processes.. if we can get RP and CA developers to agree, I think we can move forward on this without the need for an RFC even. It's a matter of organising things for sure. But I believe that other flag days (e.g. DNS) were just co-ordinated between all parties involved.

Kind regards,
Tim

---

From: Tim Bruijnzeels <[email-masked]>
Date: Mon, 31 Aug 2020 20:03:40 +0200

Hi,

> On 29 Aug 2020, at 20:58, Stephen Kent <[email-masked]> wrote:
> 
> Tim,
>> Hi,
> I'm still waiting to see a clear, and hopefully concise, description of how an RP verifies that the pub point data acquired via RRDP is constrained in the hierarchic fashion imposed by the rsync directory model, but that's a separate issue.

Coming up, I just submitted 'draft-ietf-sidrops-deprecate-rsync-00' as requested by the co-chairs on 16 July. Sorry, for the delay. Let's have the discussion there.

> yes, this is a separate discussion.

Possible extension of RFC6492 is a separate discussion.

For 6486bis the WG should be aware that intermittent failure cases can come up, and assume that they will. If this happens high enough in the tree - e.g. RIR shrinks NIR, while NIR still issues affected resources to its members - then this will invalidate a big branch, resulting in not founds. This may be a soft landing, but it is a landing.

In short: Given that I expect that we are never come to a consensus other than the total reject-all-if-one-fails - I can live with this if we must, but don't let it come as a surprise.

> 6486 allows an RP to behave either way, e.g., to accept some but not all inconsistencies to to reject them. So, it's not quite accurate to suggest that RP software that operates in accordance with 6486 will treat individual objects as invalid- it might, or it might not.
> Allowing RPs to ignore object types they don't understand prevents a CA from being able to convey the notion that a new object type is important (to that CA). I don't think this is a good strategy. It means that RP behavior will be ambiguous relative to new object types.

So far none of the objects have seemed to need this flag.

> If we want to have a consistent and flexible approach to accommodating new objects I suggest the strategy I mentioned earlier. Define an additional SIA URI that points to a pub point (and manifest) where we can introduce the next version of the signed object format, one that includes a critical flag, analogous to X.509v3 extensions. This allows each CA to decide which object types have to be processed  by an RP in order for the whole pub point to be accepted vs. rejected. Note that this will require modifying a lot of RFCs, but it is a flexible, extensible approach to this issue.

I agree that it's flexible and extensible. I had not thought of this approach.

But it is a lot of work, not just in RFCs, also in code. It also raises questions about how and when old PPs without the new objects can be deprecated. You can give operators more time to upgrade, but at some point plugs will probably be pulled? Maintaining multiple PPs indefinitely seems rather wasteful.

I would like to hear what others have to say.. I have the feeling that ASPA is getting close, and I would really not like to see it delayed because of this.

If we do go down this road then I think that we should also look at the manifest object itself, and let it convey which object (types) are critical (and while we are at it, we can specify types instead of using filename extensions). That way future object types could introduced more easily perhaps - this obviously needs more discussion but it could even allow for semantics like: 1) new object please test, don't use, 2) new objects, use if you can, 3) new objects, critical - fail if you don't understand.

Tim

> 
> I agree that even if we adopt the current 6486bis , for now, that a flag day is appropriate, and it shoudk be part of the document.
> 
> Steve
> 
>