Subject: [Sidrops] 6486bis: referenced object validation

From: Ben Maddison <[email-masked]>
Date: Fri, 4 Dec 2020 00:42:13 +0200

Hi all,

As a result of the incident last night that caused the number of VRPs
output by some RPs (certainly routinator, possibly others too) to drop
dramatically, a couple of us have been going over the observed behaviour
in comparison with the current text of 6486-bis.

It appears from Job's analysis [1] that the incident was triggered when
several resource certs that were listed on an APNIC issued manifest
became invalid due to their 'notAfter' time expiring, which in turn
caused the affected RPs to consider the manifest invalid and fail the
fetch.

It should be perfectly legitimate for a manifest to list objects which
will become invalid during the lifetime of the manifest, or which are
not yet valid when the manifest is generated (for pre-staging).
The presence of such objects should not invalidate the manifest.

I think there are at least two places that need an update in light of
this:

Section 6.4 states:
    """
    If there are files listed in
    the manifest that cannot be retrieved from the publication point, or
    if they fail the validity tests specified in [RFC6488], the fetch has
    failed and the RP MUST proceed to Section 6.7
    """

The reference should probably be to both RFC6487 and RFC6488, as the
former provides the validation details for .cer and .crl objects, which
are presumably intended to be included in this step.

Additionally, due to the values in the 'notBefore' and 'notAfter' in a
Resource of EE cert, an object that was legitimately included in a
manifest may become invalid when processed later in the manifests
lifetime by an RP.

I would therefore suggest the following revision:
    """
    ... if they fail the validity tests specified in [RFC6487] and [RFC6488],
    with the exception of the checks against certificate Validity From
    and To values described in Section 7.2 of [RFC6487], the fetch has
    failed ...
    """

Although not directly related to the current problem, the list of
objects in section 6.4 for which the ability to process is mandatory 
should perhaps be reduced to those in general circulation today, and not
include BGPsec certs or ghostbuster records, since those add
approximately zero utility to an operator who wishes to run an RP today,
and are therefore unlikely to be priorities for implementation.

Next, section 6.6 states:
    """
    If a current manifest contains entries for objects that are not
    within the scope of the manifest (Section 6.2), the fetch has failed
    and the RP SHOULD proceed to Section 6.7
    """
I believe that the reference to 6.2 is a typo, and should point to
section 2:
    """
    A manifest associated with a CA's repository publication point
    contains a list of:
    o  the set of (non-expired, non-revoked) certificates issued and
       published by this CA,
    o  the most recent CRL issued by this CA, and
    o  all published signed objects that are verifiable using EE
       certificates [RFC6487] issued by this CA.
    """

The term "non-expired" begs the question of whether a certificate should
be non-expired at the time of manifest generation or RP processing to be
in scope.

On the one hand, insisting that a listed certificate be valid throughout the manifest
lifetime is what gave rise to the failure described above, and is
clearly, IMO, the wrong choice.

On the other, insisting that a certificate be valid at the time of
manifest generation precludes certificates which have not yet reached
their notBefore time. Validating as if "at manifest generation" also
appears to contravene the guidance in section 4.6.1 of RFC6487:
    """
    Relying Parties SHOULD NOT attempt to infer from this
    time information that a certificate was valid at a time in the past,
    or that it will be valid at a time in the future, as the scope of an
    RP's test of validity of a certificate refers specifically to
    validity at the current time.
    """

As above, therefore, I would suggest that rather than choosing either
of the above options, the question of temporal expiry should be left 
out of manifest validation altogether.
Thus point one of section 2 becomes:
    """
    A manifest associated with a CA's repository publication point
    contains a list of:
    o  the set of non-revoked certificates issued and published by
       this CA,
    o  the most recent CRL issued by this CA, and
    o  all published signed objects that are verifiable using EE
       certificates [RFC6487] issued by this CA.

    The list MUST include any objects that have a validity period that
    overlaps the period between the thisUpdate and nextUpdate times of
    the manifest.

    ...
    """

There may be other parts of the text that are similarly in need of
update or clarification on this point: I'll update if I find any.

Hopefully the above makes sense. Please let me know if anything is
unclear.

Cheers,

Ben

[1]: https://lists.nlnetlabs.nl/pipermail/rpki/2020-December/000245.html

---

From: George Michaelson <[email-masked]>
Date: Fri, 4 Dec 2020 11:33:37 +1000

I support this kind of change to the wording.

I believe this language addresses my concern with rejection of a
Manifest which was legal before an expiry timer, and becomes illegal
purely because of the expiry of a descendent child CA.

The signing chain over the Manifest itself, its EE, and the CA chain
over that EE still need to be valid, unexpired, unrevoked.

Validation of objects on the Manifest, validity of the manifest given
other errors, are not altered in this change.

cheers

-G

On Fri, Dec 4, 2020 at 8:42 AM Ben Maddison
<benm=[email-masked]> wrote:
> _______________________________________________
> Sidrops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sidrops

---

From: Martin Hoffmann <[email-masked]>
Date: Fri, 4 Dec 2020 11:16:51 +0100

Ben Maddison wrote:
> dramatically, a couple of us have been going over the observed
> behaviour in comparison with the current text of 6486-bis.
> 
> It appears from Job's analysis [1] that the incident was triggered
> when several resource certs that were listed on an APNIC issued
> manifest became invalid due to their 'notAfter' time expiring, which
> in turn caused the affected RPs to consider the manifest invalid and
> fail the fetch.
> 

I think the incident shows a general flaw in the approach that
anything being wrong with any of the objects published by the CA
should lead to all the CA’s objects being invalidated: It prohibits a
CA from publishing invalid objects on purpose.

The validity time is only one such case. Another one that we already
know about is changing resource entitlements in the CA certificate.
This case is actually worse since the CA itself has no influence on
this happening and, as the protocols are now, there is no automated
way of agreeing on a timeline for the process.

It is not entirely unlikely that we will encounter more such issues in
the future.

So perhaps instead of trying to work around each of these issues as
they arise, we should go back and look at what started this revision in
the first place. That was an observation that by manipulating a
publication point’s objects, an attacker can force a route announcement
to become RPKI invalid.

The current approach does solve this problem but it also attempts to
solve a different problem: to protect a CA from accidentally publishing
invalid data. But as we found out, that requires anticipating the
intentions of a CA. Perhaps it is better to trust a CA and accept that
if it fails there will be problems for its resources.

Under this approach, the manifest expresses which objects the CA
intended to publish. If all the objects listed on the manifest are
present with a matching hash, the publication point reflects the intent
of the CA and can be processed. If it contains invalid objects, these
can be discarded individually.

This approach will be simpler and easier to implement. It will avoid
the invalidation of the entire CA because of expired or not yet valid
CA certificates or changing resource entitlements. It may, however,
lead to a partial set of objects applying to a particular resource.
That may be intentional or by accident. Whether it is better to reject
the partial set accepting that an intentionally partial set is rejected
or to accept an accidental partial set to be included depends on the
particular application of RPKI. For ROV, rejecting the set may be
better (although there are consequences of that, too) whereas for
router keys or Ghostbuster records including it would seem to be fine.

In other words, the proposal is to simplify the text for section 6.4
to say:

      The RP MUST acquire all of the files enumerated in the manifest
      (fileList) from the publication point.  If there are files listed
      in the manifest that cannot be retrieved from the publication
      point the fetch has failed and the RP MUST proceed to Section 6.7;
      otherwise, proceed to Section 6.5.

In addition, section 6.6 can be repurposed to specify the behaviour in
case of invalid objects. As a proposal:

   6.6. Invalid Files

      If files listed in on the manifest fail the validity tests
      specified in [RFC6487] and [RFC6488], the fetch has not
      necessarily failed. However, applications of the RPKI may define
      specific consequences if one or more files are invalid.

Alternatively, this section can provide the rules for existing
applications of the RPKI, i.e., ROV, BGPsec, and GBR as well as
delegation to child CAs.

I understand that this proposal is a departure from our current
approach, but I would like to ask the working group to consider it
nonetheless as it keeps validation relatively simple and, by separating
out the individual steps of the overall process, should limit the
number of unforeseen consequences.

Kind regards,
Martin

---

From: Job Snijders <[email-masked]>
Date: Fri, 4 Dec 2020 10:40:06 +0000

On Fri, Dec 04, 2020 at 11:16:51AM +0100, Martin Hoffmann wrote:
> present with a matching hash, the publication point reflects the
> intent of the CA and can be processed. If it contains invalid objects,
> these can be discarded individually.

Indeed, I believe you've now captured the essence of why manifests
exists at all. This is what rpki-client & FORT seem to have implemented.

Other than the hashes matching & files being present, additional
conditions apply: the manifest & EE certificate need to be valid,
current, latest, correctly encoded, part of the cert chain, CRL present,
etc.

Kind regards,

Job

---

From: Tim Bruijnzeels <[email-masked]>
Date: Fri, 4 Dec 2020 11:44:17 +0100

Hi Job,

> On 4 Dec 2020, at 11:40, Job Snijders <[email-masked]> wrote:
> 

I suggested this in August:
https://mailarchive.ietf.org/arch/msg/sidrops/Mldz5a43kPPotslF9bpNz10rysk/

I think a lot of this could have been avoided if we had this discussion then, but I welcome this now..

> current, latest, correctly encoded, part of the cert chain, CRL present,

Indeed

---

From: Ties de Kock <[email-masked]>
Date: Fri, 4 Dec 2020 11:47:17 +0100

> current, latest, correctly encoded, part of the cert chain, CRL present,

However, validating these requirements for just CMS objects (as per -03, instead
of for CAs as well as per -00) leaves open the situation where only part of the
objects for a CA apply. When the ROAs are correct, but the sub-CA is not, this
can cause outages.

I don't have a preference on which way to go, but I would propose that we treat
invalidation (time, c.f. because of semantic errors) for both CA certificates
and other objects similarly.

Ties

---

From: Job Snijders <[email-masked]>
Date: Fri, 4 Dec 2020 10:51:27 +0000

On Fri, Dec 04, 2020 at 11:47:17AM +0100, Ties de Kock wrote:
> However, validating these requirements for just CMS objects (as per
> -03, instead of for CAs as well as per -00) leaves open the situation
> where only part of the objects for a CA apply. When the ROAs are
> correct, but the sub-CA is not, this can cause outages.
> 
> I don't have a preference on which way to go, but I would propose that
> we treat invalidation (time, c.f. because of semantic errors) for both
> CA certificates and other objects similarly.

I am not sure I follow, can you construct an example data set where the
'situation left open' you describe appears?

Kind regards,

Job

---

From: Ties de Kock <[email-masked]>
Date: Fri, 4 Dec 2020 11:57:33 +0100

> On 4 Dec 2020, at 11:51, Job Snijders <[email-masked]> wrote:
> 

Thanks for asking for clarification. I'll try to explain the scenario:

CA certificate with <AS64496, 192.0.2.0/24>
  * Manifest, containing
	  * ROA: AS0, 192.0.2.0/24
	  * sub-CA, resources: <AS64496, 192.0.2.0/24>
	  	* Manifest, containing:
			  * ROA: <AS64496, 192.0.2.0/24>

We have "all or nothing" for ROAs. I argue that a similar situation is present
for sub-CAs if the sub-CA expires.

Kind regards,
Ties

---

From: Tim Bruijnzeels <[email-masked]>
Date: Fri, 4 Dec 2020 12:06:40 +0100

Hi,

> On 4 Dec 2020, at 11:57, Ties de Kock <[email-masked]> wrote:
> 
> 

Indeed there are many such corner cases.

Imho this discussion is complicated because we are conflating object validation and the impact of encountering invalid objects on the resulting VRP set and ultimately routing decisions.

I think it could help to separate these issues. Keep RPKI object tree validation itself simple, but track invalid / rejected objects and discuss how to deal with the VRP set as a separate issue. E.g. we could track all resources that are under some dispute (intersection of what is issued to a CA and issues with objects by the CA) and filter VRPs based on that.

Tim

---

From: Job Snijders <[email-masked]>
Date: Fri, 4 Dec 2020 11:17:01 +0000

> for sub-CAs if the sub-CA expires.

You list two manifests, which means that software should:

round 1: process Manifest 1 containing AS0.roa + sub-CA.cer + crl
    check if as0.roa and sub-CA.cer + crl are present and if the hashes
    match

round 2: process Manifest 2 containing AS64496.roa + crl
    check if AS64496.roa + crl are present and if the hashes match

Now let's image sub-CA.cer is present and the hash match, but turns out
to be expired, it just invalidates sub-CA.cer and thus the software
would never even open manifest 2, because sub-CA.cer is expired.

If sub-CA.cer is expired the ROA for AS64496 indeed will 'disappear' (as
if never existed), leaving just the AS0 ROA. This is as the CA intended.

Kind regards,

Job

---

From: Job Snijders <[email-masked]>
Date: Fri, 4 Dec 2020 12:13:32 +0000

On Fri, Dec 04, 2020 at 11:44:17AM +0100, Tim Bruijnzeels wrote:
> I think a lot of this could have been avoided if we had this
> discussion then, but I welcome this now..

Looking back I'm at a loss how "a lot of this" really could've been
avoided. What additional steps or process should I have followed for
NLNetLabs/RIPE/Cloudflare to sooner understand the problem with their
implementation choices and encourage those organisations to provide a
solution to their users?

As you know, I reported 'this' as a problem in February 2020 to sidrops@

OpenBSD rpki-client implemented the 'this' solution already in March
2020 https://github.com/openbsd/src/commit/8e07ef58f8a3812e87e1b5cf4d956ba264438cd5

FORT implemented also implemented the 'this' solution already back in
March 2020 https://github.com/NICMx/FORT-validator/commit/0ad2c3da34827b68c4f524c0c50f223746af91a5

To protect its business, NTT deployed 'this' in their transit-free
network in April 2020, further proving the viability of the approach on
global scale.

I reported 'this' again as a problem & mentioned the solution in April
2020: https://github.com/NLnetLabs/routinator/issues/319

We had *multiple* virtual SIDROPS meetings where I and others kept
repeating the issue and the solution, in simple english: ignore the
manifest when files are missing or corrupted. Additionally, ignore
objects that are expired or otherwise invalid.

Two RP implementers told me they *first* wanted to see a nearly finished
or even published RFC, before making any changes to their code. I think
RFC-first-code-later is an entirely unworkable model for cryptographic
software actively used in the field. It seems weird to me to set the bar
so high before addressing simple software defects. Conversely, IDR
requires 2 implementations *before* RFC publication (which helps avoid
lots of problems in RFC texts^W^W^W^W^W^W^Wimprove quality of RFCs), a
ritual which I think should be followed by SIDROPS as well.

RFC compliance (or a perception thereof) is very important for
system interopability, but not the end goal in and of itself.

The real goal is to keep the network running, where in this context our
role is to create software for network operators for them to securely
validate the wishes of the Internet number resource holders.

We have to keep the Internet running *while* we discuss things in IETF.
Drafts/RFCs are the byproduct, they come after the coding effort and
deploying the code.

Kind regards,

Job

---

From: Ties de Kock <[email-masked]>
Date: Fri, 4 Dec 2020 14:13:41 +0100

> if never existed), leaving just the AS0 ROA. This is as the CA intended.

Compare that to:

Valid CA certificate with <AS64496, 192.0.2.0/24>
 * Manifest, with valid EE certificate, containing
   * valid CRL
   * ROA: AS0, 192.0.2.0/24, EE cert with 2020-12-01 expiration
   * ROA: AS64496, 192.0.2.0/25, EE cert with 2021-01-01 expiration

This results in a failed fetch when validated today.

Under our interpretation of -00, this manifest would be rejected, since all
objects on a manifest need to be valid (present, hash match, correct, non-expired)
for a fetch to succeed.

I would argue that to be consistent, this should not result in failed fetch, the
expired ROA should not apply, because this is what the CA intended.

What do you think the outcome should be?

Kind regards,
Ties

---

From: Benno Overeinder <[email-masked]>
Date: Fri, 4 Dec 2020 14:13:40 +0100

Hi Job, WG,

> As you know, I reported 'this' as a problem in February 2020 to sidrops@

</snip> timeline for brevity.

Your timeline is correct, as far as I have followed the discussions. 
The remark from Tim is meant to be constructive, not defensive.  Not 
looking back, but looking ahead, I want to focus on how we solve issues 
and discuss them in sidrops.  And indeed, if there is an impasse, 
organise a virtual meeting and discuss the matter to make progress.

> validate the wishes of the Internet number resource holders.

Agree to that!

Let's keep the good spirit of collaboration and learning together.

Best regards,

-- Benno

-- 
Benno J. Overeinder
NLnet Labs
https://www.nlnetlabs.nl/

---

From: Job Snijders <[email-masked]>
Date: Fri, 4 Dec 2020 14:37:21 +0000

On Fri, Dec 04, 2020 at 02:13:40PM +0100, Benno Overeinder wrote:
> Not looking back, but looking ahead, I want to focus on how we solve
> issues and discuss them in sidrops.  And indeed, if there is an
> impasse, organise a virtual meeting and discuss the matter to make
> progress.

Yes, I took it as a sign of convergent thinking. Progress indeed.

However, my question remains, looking forward, what can be done
differently in the future?

Some suggestions from my side:

    - SIDROPS must require multiple implementations before sending an
      Internet-Draft towards the IESG. To meet this requirement,
      Implementers should write an implementation report and share it
      with the group, at least detailing the behavior for each normative
      term.

    - Please listen to operators who report (hard to troubleshoot, hard
      to explain) network outages which resulted from RPKI processing.
      RPKI ROV is supposed to be an incrementally deployable 'fail open'
      mechanism.

    - Implementers should more often share/construct the actual X.509
      files rather than attempting to document the hypothetical states
      of the RPKI in natural language (avoid English or Dutch). We can
      all load X.509 files into our implementations, set currenttime to
      the test case at hand, and discuss results.

    - Implementers should feel empowered to make improvements to their
      software (for the benefit of their users), especially if there is
      perceived ambiguity in RFC texts. An implementer doesn't need the
      blessing of the whole IETF before making a change.

    - Sometimes RFCs are wrong, or worse: the then-WG kicked the ball
      down the road and state 'up to the implementer' (aka 'local
      policy'); in which case the implementer will have to weigh the
      advantages of perceived 'RFC compliance' vs the effects of the
      behavior in the BGP Default-Free Zone. See the previous point.

    - The RPKI is a system that has been deployed on the Internet, this
      sometimes means operators need to quickly deploy fixes to address
      ongoing issues. Time is of the essence when it comes to security
      or reliability issues. So 'code first, RFC later', which goes well
      with point 1: 'implementations first, then publish RFC'.

You mentioned some kind of testbed, indeed it would be wonderful if we
can easily reproduce states of the RPKI and allow implementers to
provide annotations on why software (versions) behaves in a certain
way. The December 1st, 2020 case is definitely one I'd add to the
testbed archive.

> > The real goal is to keep the network running, where in this context
> > our role is to create software for network operators for them to
> > securely validate the wishes of the Internet number resource
> > holders.
> 
> Agree to that!
> 

For sure!

Kind regards,

Job

---

From: Job Snijders <[email-masked]>
Date: Fri, 4 Dec 2020 15:07:12 +0000

On Fri, Dec 04, 2020 at 02:13:41PM +0100, Ties de Kock wrote:
> Under our interpretation of -00, this manifest would be rejected,
> since all objects on a manifest need to be valid (present, hash match,
> correct, non-expired) for a fetch to succeed.

The objects ON the manifest must be present and matching hashes, and the
manifest's EE certificate being correct, non-expired. I think this is
what is meant with the manifest must be 'valid', and 'current'

> I would argue that to be consistent, this should not result in failed
> fetch, the expired ROA should not apply, because this is what the CA
> intended.
> 

Current versions of rpki-client and FORT consider the fetch successful,
but the 'AS0 ROA' in your example is expired, so that ROA will not be
emitted as VRP, as the ROA did not pass validation.

Imagine this scenario: I am an IP prefix broker, and I have a /16. For
this /16 I create a AS0 ROA because I don't want anyone to use the /16
unless I've given explicit permission through the creation of a ROA.
In december customer A leases a /24 for 3 months. In January customer B
leases a different /24 for 5 months, starting at February 2020.

Current time: 4 december 2020, 15:03 UTC
Valid CA <10.0.0.0/16>
  * Manifest, with valid EE certificate, containing
    * valid current CRL
    * ROA: AS0 10.0.0.0/16 (start date 1 july 2020, end date 1 july 2030)
    * ROA: AS1, 10.0.0.0/24 (start date 1 december 2020, end march 2021)
    * ROA: AS2, 10.0.1.0/24 (start date 1 february 2021, end juli 2021)

I believe the above to be real-world examples of how people perceive the
functionality of the RPKI, and this is confirmed by APNIC indicating
that the expiration dates on CA .cer files can match the expiration of
APNIC membership status. 

The beauty here is that a CA (an internet number resource holder) can
pre-provision the routing authorisations ahead of time. Each ROA on this
manifest represents a cryptographic 'product' (think something tangible
you can buy and hold), in this case authorizations to originate IP
space. The CA can also pre-provision the decommissioning of the product,
this is done by aligning the ROA expiration dates to the business
contract. This is an important cryptographic feature to be able to
pre-provision contract terminations.

Different expiration dates (and also start dates) for different objects
on the same manifest is an expected state of the RPKI. I believe the
CA's wishes are honored by checking that all files are present & hashes
matched, and that then in a second step each individual object is
validated (both signatures and validity time)

Kind regards,

Job

---

From: Tim Bruijnzeels <[email-masked]>
Date: Fri, 4 Dec 2020 16:58:02 +0100

Hi Job, all,

> On 4 Dec 2020, at 15:37, Job Snijders <[email-masked]> wrote:
> 

Thank you for taking the time to suggest ways to improve!

We can continue this discussion here, or perhaps even have a separate thread on this. I don't really mind either way. I am just happy with this effort.

>      with the group, at least detailing the behavior for each normative

works for me

>      RPKI ROV is supposed to be an incrementally deployable 'fail open'

Yes, of course.

Also listen to software implementers, and RPKI infrastructure (CA/repo) operators, or anyone who speaks up here. People who do, do so because they all believe strongly in making things better. So, respect each other and trust people's motives. When there is pushback perceived, then this is most likely because the other party is trying to make a point, and not necessarily because they don't acknowledge your issue.

And before I offend anyone with this.. we all suffer from cognitive bias in how we read what the other people wrote, and might have thought, or not said, etc. So, really I mean this as a constructive thought. 1-on-1 emails may also help to get some clarity - now that we do not get to meet anyone in person, before mis communications get out of hand.

>      the test case at hand, and discuss results.

There has been talk of an effort in the dark past, which was abandoned. But in short, yes, I think it would be great if we had a way to achieve this.

>      with point 1: 'implementations first, then publish RFC'.

On the above three points. Generally speaking plausible, and agreed. However, it's not always easy to achieve.

On 6486 in particular there was a strong desire communicated by operators that RPs behave the *same* way. Which is why we were hesitant to implement the changes suggested by you in the GH issue. In fact we even agree with the approach in the issue. But it's different from where the -bis text is headed until now. We felt that the desire for universal behaviour (-bis) outweighed the rush to make a quick change here. You are welcome to disagree with that assessment, but understand that there was well intended reasoning behind it.

> way. The December 1st, 2020 case is definitely one I'd add to the

It would be. I think this is an effort in itself. Do you have a proposal in mind?

For sure!

Tim

---

From: Lukas Tribus <[email-masked]>
Date: Fri, 4 Dec 2020 21:46:00 +0100

Hello!

On Fri, 4 Dec 2020 at 16:58, Tim Bruijnzeels <[email-masked]> wrote:
> On 6486 in particular there was a strong desire communicated by operators
> that RPs behave the *same* way.

If I recall correctly, I expressed this desire, yes.

This was based on my gross underestimation of the complexities of the
RPKI repository system and also a lack of understanding of how routers
will handle multiple RTR servers at the time.

As a network operator, I like the fact that BGP implementations across
vendors, given the same/similar parameters, select the same best path.
This means I can replace vendor X with vendor Y in location Z and keep
the rest of the network as is, without routing loops. However
different VRP outputs from RP's do not cause routing loops in networks
(when deployed correctly). Apart from routing loops in the BGP
analogy, same exact VRP output means that I don't have to think about
what happens if different RP software disagrees, which I can
appreciate. However the RPKI repository system is just too complex and
has too many variables.

In the WebPKI world, although TLS is standardized, TLS clients will
behave differently. Firefox is actively querying OCSP servers, Chrome
will never do that and will wait for Google to push CRL-sets to the
browser. Implementation choices vary widely. If the TLS RFC's would go
into such details like specifying whether Firefox or Chrome is "right"
(should a browser actively query OSCP server and what about walled
gardens [...]), that would probably make RFC ratifications take a
*very long* time (too long to keep up, while the real world moves on)
and defeat it's entire purpose. And how would this impact something
like 1/n-1 record splitting (the workaround for the BEAST
vulnerability, which all browsers implemented after Chrome made the
first step forward [1])?

While the same exact VRP output from different RP's would be *nice to
have* for us network operators, I fail to see how this is actually
necessary. It certainly is more convenient to not have to think about
those mismatches, that's the reason why I advocated for this, but that
is not a luxury we can afford.

This is why I now believe expecting the same VRP output from different
RP's is an unrealistic goal and actually harmful. A software
implementer must feel free to implement fixes for important
operational issues as they see fit.

Best regards,

lukas

[1] https://www.imperialviolet.org/2012/01/15/beastfollowup.html

Which is why we were hesitant to implement the changes suggested by
you in the GH issue. In fact we even agree with the approach in the
issue. But it's different from where the -bis text is headed until
now. We felt that the desire for universal behaviour (-bis) outweighed
the rush to make a quick change here. You are welcome to disagree with
that assessment, but understand that there was well intended reasoning
behind it.

---

From: Job Snijders <[email-masked]>
Date: Fri, 4 Dec 2020 21:23:14 +0000

Dear group,

On Fri, Dec 04, 2020 at 04:58:02PM +0100, Tim Bruijnzeels wrote:
> On 6486 in particular there was a strong desire communicated by
> operators that RPs behave the *same* way.

> you in the GH issue. 

I have a celebratory moment to share with all of you, below is a
terminal transcript and some hashes which will allow you to confirm my
findings.

The below was executed on a modern POSIX.1 implementation with default
settings. I used routinator's '-n' option and a symlink to ensure the
input to both routinator and rpki-client is the same, up and including
to all filesystem metadata.

We have some tals:

    # ls /etc/rpki
    afrinic.tal apnic.tal   arin.tal    disabled    lacnic.tal  ripe.tal

Execute rpki-client, which will fork rsync and gather all data from the
Internet. The system is located in Amsterdam. I've also included the
size of the cache and a hash of all the object filenames that were in
the cache at that moment.

    # rpki-client -jc > /dev/zero 2>&1 && date
    Fri Dec  4 20:33:14 UTC 2020
   
    # cd /var/cache/rpki-client && du -s
    839028  .

    # find . | sha256
    a8c03e2ef66de2532a119f56cdeba0e52acbeb23b8ac3e645d6fa7442c8fed8e

rpki-client VRP set:

    # sort /var/db/rpki-client/csv | sha256
    e0e8de429465def598b18a399d6695968cb7f28db2c2093c95bdc4563240fa4b

routinator 0.8.2-rc1 VRP set:

    # ls -lahtr .rpki-cache/repository/rsync
    lrwxr-xr-x  1 job  job    23B Dec  4 20:09 .rpki-cache/repository/rsync -> /var/cache/rpki-client/

    # ./target/debug/routinator -v -t /etc/rpki vrps -n 2>/dev/zero | sort | sha256
    e0e8de429465def598b18a399d6695968cb7f28db2c2093c95bdc4563240fa4b

>From the above we learn it truly is possible for multiple entirely
different implementations written by different people to arrive at the
**exact** same results, based on data derived from a complex distributed
PKI system with 5 Trust Anchors and 22,084 manifests from tens of
thousands of CAs resulting in the exact same 205,768 unique VRPs.

I *also* went back to December 1st, 2020 at 00:00:13 UTC and fed that
data from the tarball linked at https://lists.nlnetlabs.nl/pipermail/rpki/2020-December/000245.html
into both routinator and rpki-client, and the resulting VRP lists from
both rpki-client and routinator resulted in the same hash:
be9e7fd656c3d143f2eb5cbb0b13d299a3534c6043f49a3330f5ac14854e621f

However rocky the road was to get to this point, this really is a
marvelous achievement at scale for the internet engineering community.
Congratulations NLNetLabs. I hope other implementations will soon arrive
at the be9e7fd656c3d143f2eb5cbb0b13d299a3534c6043f49a3330f5ac14854e621f
checksum when mimicking that december 1st moment with that x509 data
set.

Kind regards,

Job

ps. Please note the above two data points are only single snapshots in
time, but encouraging ones. At present I am aware of implementation
differences between RP implementations related to ASN/BER/DER handling,
and possibly other differences, which will probaably manifest themselves
(pun intended) at one point or another in the future, or already have in
the past.

---

From: Tim Bruijnzeels <[email-masked]>
Date: Fri, 18 Dec 2020 09:28:03 +0100

Dear group,

> On 4 Dec 2020, at 22:23, Job Snijders <[email-masked]> wrote:
> 
> Dear group,
> 
> <snip/>

In summary: rpki-client, fort and routinator now follow the same approach. I believe it's being adopted by octo and the ripe ncc validator as well. I am not sure about rpstir and rcynic.

This begs the question whether we should seek to reword the -bis text to essentially just insist that all current objects on a manifest should be present and accounted for, instead of the more complicated text that it has now which has resulted in at least two cases where all objects under an RIR were rejected - if implemented. It will come as no surprise, that I for one would welcome that change.

Let me quote Martin's motivation and suggestion in Reply to Ben's email earlier in this thread:

> On 4 Dec 2020, at 11:16, Martin Hoffmann <[email-masked]> wrote:
> 
> Ben Maddison wrote:
>>