Subject: [Sidrops] ARIN RPKI Repository issue - Wednesday 24 October 2018

From: John Curran <[email-masked]>
Date: Thu, 25 Oct 2018 07:36:27 +0000

<https://www.arin.net/announcements/2018/20181024_update.html>

ARIN experienced an RPKI Repository issue yesterday afternoon (Wednesday 24 October 2018) which resulted in the number of objects in the repository dropping from its normal hundreds of objects to only 7 objects.  The root cause for the failure was too many ROAs for a resource cert that led to an untested situation whereby we exceeded the MTU for the HSM and an invalid manifest in repository.  It appeared that some validators stop processing after the invalid manifest, thus resulting in a greatly reduced count of objects published.

This condition occurred for much of day, and was resolved via a workaround at approximately 22:00 ET.   The ARIN RPKI Repository appears normal on RPKI monitors at this time, and while we have some work to do to durably address the issue for long-term, we not expect any recurrence of this event.

FYI,
/John

John Curran
President and CEO
American Registry for Internet Numbers

---

From: Christopher Morrow <[email-masked]>
Date: Thu, 25 Oct 2018 11:32:01 -0400

hey!
1) thanks for the info/update/transparency
2) it'd be cool to know: "hsm like FOO don't like getting packets over BAR
kb, be aware if you operate like this"
3) I'm glad we're finding these (and writing test code, and monitoring)
while we are still young in this process

Also, it's nice to occasionally see the sausage making.

On Thu, Oct 25, 2018 at 3:37 AM John Curran <[email-masked]> wrote:

> ARIN experienced an RPKI Repository issue yesterday afternoon (Wednesday
> 24 October 2018) which resulted in the number of objects in the repository
> dropping from its normal hundreds of objects to only 7 objects.  The root
> cause for the failure was too many ROAs for a resource cert that led to an
> untested situation whereby we exceeded the MTU for the HSM and an invalid
> manifest in repository..  It appeared that some validators stop processing
> after the invalid manifest, thus resulting in a greatly reduced count of
> objects published.
>
> This condition occurred for much of day, and was resolved via a workaround
> at approximately 22:00 ET.   The ARIN RPKI Repository appears normal on
> RPKI monitors at this time, and while we have some work to do to durably
> address the issue for long-term, we not expect any recurrence of this
> event.
>
> FYI,
> /John
>
> John Curran
> President and CEO
> _______________________________________________
> Sidrops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sidrops
>

---

From: Randy Bush <[email-masked]>
Date: Thu, 25 Oct 2018 09:22:47 -0700

thanks for post mortem.  i could see this happening.  lesson learned.
thanks.

randy

---

From: Tim Bruijnzeels <[email-masked]>
Date: Fri, 26 Oct 2018 11:37:06 +0200

Hi,

> On 25 Oct 2018, at 09:36, John Curran <[email-masked]> wrote:
> 
> ARIN experienced an RPKI Repository issue yesterday afternoon (Wednesday 24 October 2018) which resulted in the number of objects in the repository dropping from its normal hundreds of objects to only 7 objects.  The root cause for the failure was too many ROAs for a resource cert that led to an untested situation whereby we exceeded the MTU for the HSM and an invalid manifest in repository..  It appeared that some validators stop processing after the invalid manifest, thus resulting in a greatly reduced count of objects published. 

So this caused the existing the MFT to go *stale* after 48 hours, while the embedded EE certificate had not yet expired (it uses 7 days from what I can see).

Validator software treats these cases differently, because the standards are vague on this, I believe too vague:
https://tools.ietf.org/html/rfc6486#section-6.4 <https://tools.ietf.org/html/rfc6486#section-6.4>

There are many words here. And some seem contradictory, but let me quote:
[…]the entity that has published the stale manifest at this publication point, SHOULD be viewed as somewhat suspect, but MAY be used by the RP as per local policy.

There is similar vagueness around discrepancies between repository content and manifests. Also.. local policy.

As a result the RIPE NCC RPKI Validator 2.x series leave this local policy as a choice to the user, but defaults to rejecting all stale MFTs. RIPE NCC RPKI Validator 3.x accepts stale MFTs with warnings, but only rejects when the EE certificate is expired. I do not know what RPSTIR does, but Rcynic does the same I believe. As does NLnet Lab's ‘Routinator 3000’ - coming real soon now :)

I am not 100% sure about RPSTIR but all other validators use the ‘local’ policy that the MFT is the authoritative signed statement of what is current for a CA. In other words anything not listed on a MFT, or objects with a different hash, are rejected.

In my opinion it would be best if the relevant sections in the MFT RFC (6486) were updated to ensure that there are no unexpected differences between RP implementations. Accepting stale, rejecting expired && only processing what is on a valid manifest seems the sane way forward to me, and reflects most of the operational reality.

I would be happy to come up with text for people to shoot at :)

Tim

---

From: John Curran <[email-masked]>
Date: Fri, 26 Oct 2018 14:05:40 +0000

> thanks for post mortem.  i could see this happening.  lesson learned.

Yep – the IBM HSM 4765 can handle for a signing request up to 64K; we’ll need to plan accordingly in requests. 

> thanks.

No problem (we’re aiming for high transparency in operations, given the nature/purpose of the service…)
/John

John Curran
President and CEO

---

From: "Randy Bush" <[email-masked]>
Date: Fri, 26 Oct 2018 09:05:30 -0700

> we’re aiming for high transparency in operations, given the 
> nature/purpose of the service…

if arin really groked the nature/purpose of the service, a number
of things would change

randy

---

From: John Curran <[email-masked]>
Date: Fri, 26 Oct 2018 17:16:54 +0000

On 26 Oct 2018, at 6:05 PM, Randy Bush <[email-masked]> wrote:
> 
>> we’re aiming for high transparency in operations, given the nature/purpose of the service…
> 

Randy - 

That’s quite possible, but note that many things have already been changing:

 - Moved from explicit email logging to access ARIN TAL to click “accept” on TAL page to access
 - Moved from click “accept” on TAL page to simply download TAL from the ARIN TAL page
 - Published explicit directions so software installation tools can download ARIN TAL for user
 - Committed to an easier ROA creation interface as part of the ARIN database-integrated IRR refresh

There’s likely more things that you’d like to see changed with respect to ARIN RPKI service – feel free to send me a list when you have a moment (as we’re likely to revisit a number of things after the UPenn law “ARIN RPKI legal hurdles" study is completed and issues its report in early December.)

Thanks!
/John

John Curran
President and CEO

---

From: Randy Bush <[email-masked]>
Date: Fri, 26 Oct 2018 10:33:46 -0700

Lotta smoke

Tell me she the vast majority of noam address space is allowed to have roas without egregious contract. 

randy, on a phone

> On Oct 26, 2018, at 10:16, John Curran <[email-masked]> wrote:
>

---

From: John Curran <[email-masked]>
Date: Sat, 27 Oct 2018 05:02:50 +0000

To my knowledge, all of the RIRs require a contract to provide services, ARIN included. 

(We’re updated the LRSA four times to make changes as requested by the community, but it still might not meet your expectations.) 

/John 

> On 26 Oct 2018, at 7:33 PM, Randy Bush <[email-masked]> wrote:
> 
> Lotta smoke
>

---

From: Randy Bush <[email-masked]>
Date: Sat, 27 Oct 2018 18:54:31 +0700

> To my knowledge, all of the RIRs require a contract to provide

yes, but only arins may be changed unilaterally by the rir and grabs all
your rights.

but we have been through this a thousand times.  we all know the scoop
and arin's trashing of internet stewardship.

randy

---

From: John Curran <[email-masked]>
Date: Sat, 27 Oct 2018 13:03:31 +0000

> yes, but only arins may be changed unilaterally by the rir and grabs all

Unilateral change by ARIN was removed in 2015; the current RSA/LRSA only allows amendment to conform to an applicable law/caselaw change or after approval of the ARIN membership by ratification vote (ARIN RSA 12/LRSA 4, Sec. 1(e) https://www.arin.net/resources/agreements/rsa.pdf)

> but we have been through this a thousand times.  

Yes, and this isn’t the appropriate forum in any case – those interested in accurate information should probably review the ARIN website and/or ask on an ARIN mailing list.

Thanks!
/John

John Curran
President and CEO

---

From: Randy Bush <[email-masked]>
Date: Sun, 28 Oct 2018 00:26:59 +0700

> Unilateral change by ARIN was removed in 2015;

will look.  if clearly removed without wiggles, that's great.

> Yes, and this isn’t the appropriate forum in any case

shoot the messenger