Subject: [sidr] some comments and questions regarding rpki-rtr

From: [email-masked]
Date: Sat, 1 Oct 2011 11:42:24 +0200 (CEST)

Hi Randy, Rob, wg,

We have been working on rpki-rtr support in our validator at RIPE NCC over
the past weeks. I found the 6.x sections describing typical scenario
exchanges particularly useful.

I have read the document in the past as well, but as we all know: with
actual implementation come actual questions... so I have a couple:

A = No changes
B = Nonce and cache reset
C = Duplicate announcements / withdrawals
D = Keep alive timeout
E = Cache shutdown

A = No changes
=============
So if I read 5.2 correctly the cache should respond with just 1
end-of-data pdu if there are no updates since the serial included in a
serial request.
But if read 5.4 I can also interpret this that we should respond with 2
pdus: 1 cache response, 0 data records, 1 end-of-data

Can you please tell me which is correct?

Like this?

   Cache                         Router
     ~                             ~
     | <----- Serial Query ------- | R requests data
     |                             |
     | ----- Cache Response -----> | C confirms request
     | ------  End of Data ------> | C sends End of Data
     |                             |   and sends *same* serial
     ~                             ~

This is what we are doing now..

In any case I think it would be useful to have this somewhere in 6.2, or a
separate 6.x section.

B = Nonce and cache reset
=====================

When I read 5.10 the nonce is generated when the cache starts. And reading
6.3 the cache may send a cache reset reply to the client when there are no
incremental updates available.

Does this imply that a new cache nonce should be generated?

I assumed that it did not. The nonce is made when the process starts. So
when a client sends a reset query the same nonce may be kept. The client
just gets a new, full, data set, up to the current serial for that same
nonce.

If not, then we would have to keep track of nonce-s and serials for each
connected child, or reset them all.. I am afraid that would not scale very
well.

Part of the reason I am asking is that we are currently not yet able to
send incremental updates. So our cache always replies as described in 6.3.
We are not resetting the nonce though, and we are seeing duplicate
announcement errors from the routers.

So: is our cache wrong not to reset the nonce?

Can section 6.3 be amended to be explicit about this?

C = Duplicate announcements / withdrawals
===================================

As described in 5.5:
  The cache server MUST ensure that it has told the router client to
   have one and only one IPvX PDU for a unique {prefix, len, max-len,
   asn} at any one point in time.

So this means that cache should exclude duplicates in a full update even
if the same unique {prefix, len, max-len, asn} exists more than once (same
ROA, multiple prefixes, or different ROAs).

I probably missed the discussion on this, but can you explain why this is?
I don't see a conflict. If I get the same announce twice, it's still just
announce?

I am also wondering what this means wrt serial updates. Let me clarify by
example: 10/16 is announced in serial 2, withdrawn in 3, announced again
in 4. The router has serial 1. Should the cache then work out the exact
delta between 1 and 4, or can it send 1-2, followed by 2-3, followed by
3-4.

I can imagine that from the routers perspective it's very useful if the
cache takes care of duplicates and sends just one big delta, and not the
full history since the router last asked.

I am afraid though, that this may cause scaling issues when a potentially
large number of routers use the same cache (cpu), or a large number of
pre-computed deltas need to be kept (memory). I think that if this
responsibility were just handled by the routers we would have much better
scaling on the cache side, and it would be much easier for caches to keep
incremental updates without having to resort to no-incremental-updates
like 6.3 describes.

D = Keep alive timeout
===============

As described in 6.1:
   To limit the length of time a cache must keep the data necessary to
   generate incremental updates, a router MUST send either a Serial
   Query or a Reset Query no less frequently than once an hour.  This
   also acts as a keep alive at the application layer.

So, we have interpreted this to say that it's probably good on our side to
drop the connection after 1 hour. It's must likely dead and we want our
resource back...

When we do this, do you think it would be good if we tried to send an
error pdu just before closure? With a new error code indicating session
timeout?

E = Cache shutdown
==============

When the cache is stopped for whatever reason. Server restart, cache had
irrecoverable internal error, anything else...

Should we send a new type of notify / error (with  new specific code) to
all children so that they can gracefully switch over to another cache --
or wait until we are back?

Or should we just close the connections?

Thanks,
Tim

PS: If you are a rpki-rtr router implementer and you want to do interop
testing with us: please contact me.

---

From: Rob Austein <[email-masked]>
Date: Sat, 01 Oct 2011 12:09:21 -0400

> A = No changes

The intent is to send an empty update, ie, a normal update that
happens to have no data records because nothing has changed.

So it should be cache response followed by end of data.

>      |                             |   and sends *same* serial

Yes.  This is what my implementation does.

> B = Nonce and cache reset

No.  Server should generate a new nonce if and only if the serial
numbers it has been using up until now have become meaningless
(technical term used in the draft is "commensurate" -- nonce changes
when something happens that prevents serial numbers before and after
the event from being "commensurate", ie, measuring the same thing).

The cache nonce is really a light-weight session identifier.
Generating a new nonce is equivalent to terminating any existing
sessions that were using the old nonce.   Serial numbers are not
meaningful across sessions, so serial query across a nonce change is
not possible, router must do a cache reset.

> just gets a new, full, data set, up to the current serial for that same

The server doesn't even have to generate a new nonce when it starts,
so long as the serial numbers are still commensurate.

My server generates a new nonce when it starts up if and only if it
has no record of what nonce and serial numbers it was using when it
was last running.

> announcement errors from the routers.

You should not need to reset the nonce unless something has whacked
the serial numbers.

At least one of the router images currently under test is known to
generate spurious complaints about duplicate announcements, that may
be what you're seeing.  Check with whoever got you that image.

> C = Duplicate announcements / withdrawals

I don't entirely get the urgency behind this one myself, but my
understanding is that Randy and the router guys were concerned that
duplicates meant that the router and cache disagreed on current state,
therefore duplicates were an indication that something is wrong.

> delta between 1 and 4, or can it send 1-2, followed by 2-3, followed by

I work out the exact delta.  It's not that hard, just a bit tedious,
and I pre-compute it once when new updates come in from my validation
code rather than doing it on the fly in the server.

> incremental updates without having to resort to no-incremental-updates

Well, it's not necessarily memory.  I keep the pre-computed deltas on
disk.  How much data that works out to be depends on how far back one
is willing to keep the deltas.

The general assumption, though, is that servers are cheaper than
routers.  Server for this protocol can be a 600 USD 1U BSD or Linux
box; my understanding is that you're not likely to find serious
routers in that price range anytime soon. :)

Also, routers are supposed to spend their time moving packets, not
performing unnecessarily complex internal database manipulations.

So anything we can do on the server to make the router code simpler is
probably a win for the operator who has to pay for all this kit.

> D = Keep alive timeout

Um, the error PDU would almost certainly not get through, pretty much
by definition, because if all is well we would be seeing serial or
reset queries before the keep alive timer expires.

> E = Cache shutdown

I think just closing connections is sufficient.  Router has to cope
with connection just closing, what would it do differently here?

---

From: Randy Bush <[email-masked]>
Date: Sun, 02 Oct 2011 02:02:02 +0900

> pdus: 1 cache response, 0 data records, 1 end-of-data

i added a hammer to 5.2.  it now says

   The cache replies to this query with a Cache Response PDU
   (Section 5.4) if the cache has a, possibly null, record of the
   changes since the serial number specified by the router.  If there
   have been no changes since the router last queried, the cache then
   sends an End Of Data PDU.

> incremental updates available.

while it certainly may send a cache reset, i recommend it not.  the
reset is not a normal condition.  read the text

   The cache may respond to a Serial Query with a Cache Reset, informing
   the router that the cache cannot supply an incremental update from
   the serial number specified by the router.  This might be because the
   cache has lost state, or because the router has waited too long
   between polls and the cache has cleaned up old data that it no longer
   believes it needs, or because the cache has run out of storage space
   and had to expire some old data early.

> Does this imply that a new cache nonce should be generated?

no it does not

> The nonce is made when the process starts. So when a client sends a
> reset query the same nonce may be kept. The client just gets a new,
> full, data set, up to the current serial for that same nonce.

yes

> connected child

if you could keep track of serial and nonces, then we would not need
nonces.

> So this means that cache should exclude duplicates in a full update
> even if the same unique {prefix, len, max-len, asn} exists more than
> once (same ROA, multiple prefixes, or different ROAs).

i was fine until that last parenthetical.  i simply do not understand
it.

> I probably missed the discussion on this, but can you explain why this

relieve the router of baroque checks involving counting of announces and
withdraws.

> I don't see a conflict. If I get the same announce twice, it's still
> just announce?

is it?  or is it two roas?  how many withdraws to remove it?

> I am also wondering what this means wrt serial updates. Let me clarify
> by example: 10/16 is announced in serial 2, withdrawn in 3, announced
> again in 4. The router has serial 1. Should the cache then work out
> the exact delta between 1 and 4, or can it send 1-2, followed by 2-3,
> followed by 3-4.

it can do either.  i recommend the former.

> I can imagine that from the routers perspective it's very useful if
> the cache takes care of duplicates and sends just one big delta, and
> not the full history since the router last asked.

yep

> I am afraid though, that this may cause scaling issues when a
> potentially large number of routers use the same cache (cpu), or a
> large number of pre-computed deltas need to be kept (memory). I think
> that if this responsibility were just handled by the routers we would
> have much better scaling on the cache side, and it would be much
> easier for caches to keep incremental updates without having to resort
> to no-incremental-updates like 6.3 describes.

routers cost many hundreds of thousands of euros, have five year old
cpus in the control plane, and are supposed to be moving packets.  when
it comes to a question of who does the extra work, guess who wins.

> So, we have interpreted this to say that it's probably good on our
> side to drop the connection after 1 hour. It's must likely dead and we
> want our resource back...

otoh you might think that the client might have been otherwise occupied
and be patient for  some amount of time.  your choice.

> error pdu just before closure? With a new error code indicating
> session timeout?

we have avoided asynchronous pdus from the cache.

> Or should we just close the connections?

we have avoided asynchronous pdus from the cache.

a client will see the close or an error when they next send a query.

randy

---

From: Tim Bruijnzeels <[email-masked]>
Date: Mon, 3 Oct 2011 11:18:36 +0200

Randy, Rob,

thank you both for your explanations.

regarding my questions

> B = Nonce and cache reset

I think that reading your responses we were on the right track, but I wasn't 100% sure. I am happy to see your responses confirming this.

> C = Duplicate announcements / withdrawals

Points taken about difficulty on client to know how many withdrawals it would take to negate an announce seen more than once ;and that computational power being lots cheaper on the box running the validator (compared to routers)

> E = Cache shutdown

I wasn't sure about these, but I am not particularly looking for extra complexity. If just closing the channel is okay, that's what we'll do. We'll probably up the time-out threshold a bit (say 90 mins) to avoid cutting the connection on a client router the minute before it would send another serial.

Thanks,
Tim