Subject: [sidr] current state of BGP origin verification

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 11 Jan 2013 08:48:06 +0100 (CET)

Hello.

I work for an ISP. I have historically been present at at least one 
presentation by Randy Bush regarding BGP advertisment verification (which 
I found to be very informative). I thought I'd read up on what the current 
state of affairs are, so I started looking through the archives of this 
WG, plus looking through the past few operator meeting agendas 
(NANOG55-56), (RIPE64-65).

I found "ROVER: BGP Route Origin Verification using Reverse-DNS" at 
NANOG55, "RPKI Propagation" by R. Bush at RIPE65.

Are there others I should read?

Reading the RIPE65 it worries me that this information seems to be created 
by the RIRs. I have some insight in what it takes in the 
operations/administration of gtld and cctld registrar/registry plus global 
anycasted root name server infrastructure. I feel what we're talking about 
here is the same thing, or even more important. If secure BGP verification 
doesn't work, nothing will work. Root name and (cc|g)tld servers get all 
the attention, but root name servers can be offline for a long duration of 
time without seriously affecting the Internet as a whole. The RPKI 
infrastructure needs to be done with same reisiliancy or better it seems.

So trusting SIDR-WG and others to do the protocol standardisation, what 
needs to be done on the operational side to get this running at a level of 
quality needed to be 99.999% available and correct, both from the RIR side 
and documents telling ISPs what they need to do internally to make this 
work properly?

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Fri, 11 Jan 2013 09:18:37 +0100

On Fri, Jan 11, 2013 at 08:48:06AM +0100,
 Mikael Abrahamsson <[email-masked]> wrote 
 a message of 37 lines which said:

> If secure BGP verification doesn't work, nothing will work. [...]
> The RPKI infrastructure needs to be done with same reisiliancy or
> better it seems.

Resiliency of the *servers* (the rsync distributors of certs and ROAs)
is less important than in the DNS case, since the validating caches
can work standalone, without updating the data, for a long
time. (Although a document on timing issues, expiration, duration of
signatures, would be a good idea.)

Resiliency of the data is indeed critical. If a RIR, because of a
sofwtare bug or a hijack of its servers starts to serve bogus data, we
will have a problem.

---

From: SM <[email-masked]>
Date: Fri, 11 Jan 2013 02:27:16 -0800

Hi Mikael,
At 23:48 10-01-2013, Mikael Abrahamsson wrote:
>So trusting SIDR-WG and others to do the protocol standardisation, 
>what needs to be done on the operational side to get this running at 
>a level of quality needed to be 99.999% available and correct, both 
>from the RIR side and documents telling ISPs what they need to do 
>internally to make this work properly?

draft-ietf-sidr-bgpsec-threats-03 discusses about the threat 
model.  It could be used as a starting point to identify the points 
of failures.

Regards,
-sm

---

From: "Roque Gagliano (rogaglia)" <[email-masked]>
Date: Fri, 11 Jan 2013 11:12:36 +0000

Wrong Reference.

Mikael, have you looked at: http://tools.ietf.org/html/draft-ietf-sidr-origin-ops-19

That could be a starting point to look for missing pieces from an OPS perspective.

Roque

On Jan 11, 2013, at 11:27 AM, SM <[email-masked]> wrote:

>> So trusting SIDR-WG and others to do the protocol standardisation, what needs to be done on the operational side to get this running at a level of quality needed to be 99.999% available and correct, both from the RIR side and documents telling ISPs what they need to do internally to make this work properly?
> 
> draft-ietf-sidr-bgpsec-threats-03 discusses about the threat model.  It could be used as a starting point to identify the points of failures.
> 
> Regards,
> -sm 
> _______________________________________________
> sidr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sidr

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 11 Jan 2013 12:12:47 +0100 (CET)

On Fri, 11 Jan 2013, SM wrote:

> draft-ietf-sidr-bgpsec-threats-03 discusses about the threat model.  It 
> could be used as a starting point to identify the points of failures.

Some of the failure scenarios I have heard from the dns TLD world:

Zone file was truncated due to lack of disk space. Insufficient checks was 
in place to notice the truncated zone file before it was replicated out to 
the public-facing DNS infrastructure, thus causing outage.

DNSSEC failure (don't remember the exact details), ie not everything was 
correctly signed, thus causing failure. DNSSEC is an example of a protocol 
which is quite brittle operationally. You make one mistake, and your 
entire domain is toast.

So when implementing RPKI at the RIR level, there should be operational 
advice how everything should be done. Preferrably even at the protocol 
level, there should be a disaster "off" button which the publisher can use 
to just switch off all validation for all records they handle, and this 
should be propagated very quickly.

Frankly, I'm scared when I read the documentation for this (=SIDR) thing. 
I feel it fails the "KISS" test with all the CA, signing, propagation 
infrastructure needed. It's also quite different from anything else that 
the normal network engineer is exposed to historically in their day to day 
work life. It's my opinion that a lot of careful though put into how this 
is actually implemented in real life, making sure there are operational 
tools (I still have not signed my private domain with DNSSEC because the 
operational tools are too hard to use) in place to aid deployment.

The re-signing needed over time to keep DNSSEC (and SSL certificates) up 
to date is something I see people fail routinely still. This needs to be 
automated.

Is it too early in the protocol development process to start looking into 
the operational aspects?

---

From: Arturo Servin <[email-masked]>
Date: Fri, 11 Jan 2013 13:38:22 -0200

Mikael,

	Do you think that
http://tools.ietf.org/html/draft-ietf-sidr-origin-ops-19 covers your
concerns?

	If not, do you recommend to add text to this draft or to focus in a new
document describing the operation of a CA?

Regards,
as

On 11/01/2013 09:12, Mikael Abrahamsson wrote:
>> draft-ietf-sidr-bgpsec-threats-03 discusses about the threat model. 
>> It could be used as a starting point to identify the points of failures.
> 
> Zone file was truncated due to lack of disk space. Insufficient checks
> was in place to notice the truncated zone file before it was replicated
> out to the public-facing DNS infrastructure, thus causing outage.
> 
> correctly signed, thus causing failure. DNSSEC is an example of a
> protocol which is quite brittle operationally. You make one mistake, and
> your entire domain is toast.
> 
> level, there should be a disaster "off" button which the publisher can
> use to just switch off all validation for all records they handle, and
> this should be propagated very quickly.
> 
> Frankly, I'm scared when I read the documentation for this (=SIDR)
> thing. I feel it fails the "KISS" test with all the CA, signing,
> propagation infrastructure needed. It's also quite different from
> anything else that the normal network engineer is exposed to
> historically in their day to day work life. It's my opinion that a lot
> of careful though put into how this is actually implemented in real
> life, making sure there are operational tools (I still have not signed
> my private domain with DNSSEC because the operational tools are too hard
> to use) in place to aid deployment.
> 
> Is it too early in the protocol development process to start looking
> into the operational aspects?
>

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 11 Jan 2013 18:57:53 +0100 (CET)

> http://tools.ietf.org/html/draft-ietf-sidr-origin-ops-19 covers your

I don't understand enough, but this document isn't even close when it 
comes to giving hands-on operational advice.

What I am talking about is:

https://www.ripe.net/lir-services/training/courses/rr

but for people actually maintaining the operational infrastructure. 
Perhaps advice should be to just purchase complete solution from vendor 
instead of trying to run all the components oneself. At least there should 
be overview of the framework, flow diagrams etc, to make people more 
easily understand what's going on.

> 	If not, do you recommend to add text to this draft or to focus in 
> a new document describing the operation of a CA?

I think a new document would be good. I also feel that there needs to be 
advice for a few different deployment scenarios, where one is minimal (you 
give information to RIR, they handle most of that end, the only thing you 
focus on is the route validation for receiving BGP prefixes, how can this 
be achieved with least amount of effort) to those who want to automate 
everything and sign themselves (running a CA).

So basically a HOWTO...

---

From: Arturo Servin <[email-masked]>
Date: Fri, 11 Jan 2013 16:09:46 -0200

Mikael,

	I have thinking about a how-to document, however, would be the IETF a
place to publish something like that?

	*NOGs document seems more appropiate to me.

Regards,
as

On 11/01/2013 15:57, Mikael Abrahamsson wrote:
> instead of trying to run all the components oneself. At least there
> should be overview of the framework, flow diagrams etc, to make people
> more easily understand what's going on.
> 
> advice for a few different deployment scenarios, where one is minimal
> (you give information to RIR, they handle most of that end, the only
> thing you focus on is the route validation for receiving BGP prefixes,
> how can this be achieved with least amount of effort) to those who want
> to automate everything and sign themselves (running a CA).
> 
> So basically a HOWTO...
>

---

From: SM <[email-masked]>
Date: Fri, 11 Jan 2013 08:58:59 -0800

Hi Mikael,
At 03:12 11-01-2013, Mikael Abrahamsson wrote:
>Zone file was truncated due to lack of disk space. Insufficient 
>checks was in place to notice the truncated zone file before it was 
>replicated out to the public-facing DNS infrastructure, thus causing outage.

I heard of that.  It's overlooked as it is a detail that's only 
important when things go wrong.

>DNSSEC failure (don't remember the exact details), ie not everything 
>was correctly signed, thus causing failure. DNSSEC is an example of 
>a protocol which is quite brittle operationally. You make one 
>mistake, and your entire domain is toast.

Yes.

>So when implementing RPKI at the RIR level, there should be 
>operational advice how everything should be done. Preferrably even 
>at the protocol level, there should be a disaster "off" button which 
>the publisher can use to just switch off all validation for all 
>records they handle, and this should be propagated very quickly.

There was a thread on this mailing list about propagation.  There is 
a level of operational advice which a document does not get into as 
it cannot cover everything.  You seem to be asking about failures by 
entities outside your control and an "off" switch to avoid it from 
affecting your organization.  draft-ietf-sidr-bgpsec-threats-03 
doesn't get into that.

>historically in their day to day work life. It's my opinion that a 
>lot of careful though put into how this is actually implemented in 
>real life, making sure there are operational tools (I still have not 
>signed my private domain with DNSSEC because the operational tools 
>are too hard to use) in place to aid deployment.

There's a steep learning curve.  Taken as a whole the documentation 
fails the "KISS" test.  I agree that it is necessary to give a lot of 
careful thought about how this is actually implemented.

>into the operational aspects?

I don't have an opinion about this.

Regards,
-sm

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 11 Jan 2013 21:35:51 +0100 (CET)

On Fri, 11 Jan 2013, SM wrote:

> You seem to be asking about failures by entities outside your control 
> and an "off" switch to avoid it from affecting your organization. 
> draft-ietf-sidr-bgpsec-threats-03 doesn't get into that.

Well, both for me and for the other organisation. If they find out their 
information is corrupt and they don't know exactly how to fix it quickly, 
they should be able to send out something that recursively just turns off 
all filtering based on their entire information set.

What's important to realise is that the current state of affairs is that 
major outages are caused very rarely by fat-fingering or intentional 
disrupting the global BGP system. Getting origin verification is an 
extreme complication of this system that potentially can cause a lot more 
outages than the ones caused by not having it.

I still think origin verification is important and I want to have it, but 
it needs to be robust (as a complete system). A lot can be fixed by 
operational procedures (=money and time), but I really hope extra care is 
put into the design of it as well, to make it operationally more robust 
and easier to use. Also recommendations on what needs to be done over time 
to keep it running would be good.

Then I agree that actual operational procedures are probably best produced 
on the NOG and RIR level, but input from the designers would probably help 
here.

---

From: SM <[email-masked]>
Date: Fri, 11 Jan 2013 14:39:45 -0800

Hi Arturo,
>place to publish something like that?

I don't think that the IETF publishes how-to documents as RFCs.

Regards,
-sm

---

From: Joel jaeggli <[email-masked]>
Date: Fri, 11 Jan 2013 15:15:47 -0800

> I don't think that the IETF publishes how-to documents as RFCs.

if you're thinking something alough the lines of rfc-5635 then certainly
they do.

Origin validation for dummies would be published by IDG however.