Subject: Re: [IPsec] Troubleshooting IPsec peer certs (was: Re: IPsec profile feedback wanted (draft autonomic control) plane)

From: Tero Kivinen <[email-masked]>
Date: Thu, 16 Jul 2020 19:47:40 +0300

'Toerless Eckert' writes:
> > If you ever need to talk to devices that are not part of same
> > authorization domain you need that feature (unless you want to
> > provide lots of manual configuration to IKE). 
> > 
> > I.e., if you connect to SGW1, you need to use public/private key pair
> > and certificate from CA1 to authenticate. And if you connect to SGW2
> > you need to use different public/private key pair and different
> > certificate which is from CA2.
> > 
> > You could manually list all possible SGW ip-addresses or names and
> > configure that they must use specific CA, but that gets diffucult when
> > the number of SGWs raise, can cause problems when certificates are
> > renewed, as then you need to update your configuration again.
> > 
> > So what IKEv2 does is that it will connect to the SGW, and SGW will
> > tell that it requires you to use certificate signed by list of CAs and
> > that list include for example CA1. Then initiator can select
> > his own certificate from CA1, and the associated private key and use
> > that when sending the IKE_AUTH to the other end. If he picks wrong CA,
> > meaning wrong private key the responder will fail the authentication
> > and connection is not established.
> > 
> > Note, that the initiator does not really care which of his own private
> > keys and associated certificate it uses, as all of them identify
> > himself, he just need to pick something that is suitable for the
> > remote peer.
> > 
> > What the initiator needs to manually configure is that what kind of
> > certificates are required by remote peer, i.e., which CA must have
> > signed the certificate responder used. This is required for security
> > so SGW1 cannot do IP or DNS hijack and claim to be SGW2 even when
> > using CA1. Good thing for this is that we just configure that SGW1
> > needs to authenticate himself with cert from CA1, and SGW2 needs to
> > use cert from CA2.
> > 
> > Note, that initiator might not have any certificate from CA1 or CA2,
> > it might actually use completely different CAs, i.e., the list of
> > certificate authorities can be asymmetric.
> 
> Good explanation. So, whats the most simple example for this ?
> I have a single VPN but multiple geographic headends and
> there is an obfuscated logic what certs each headends accepts,
> so i do have multiple and certreq helps me to choose ?

I would assume most common case will be when certificates are being
renewed in non-compatible way. I.e., you initially have rsa certs for
both client and server, then someone decides rsa is weak, and we must
move to ecdsa in the future. They will first install new TA certs on
the some servers (lets say they have 1000 of those, and they do not
want to do all of them immediately). Now some of the servers do have
TA1, and some have both TA1, and TA2.

Next clients needs to get new certificates from TA2. This process can
make months, as it is possible that some of the clients are not
connecting to the server for long time.

If you have client which has only TA1, there is no problem, it can
connect to any server which still supports TA1. If you have client
which has both TA1, and TA2, and it connects to old server which only
supports TA1, it needs to recognize this and use TA1 in that case. If
it connects to server using both TA1, and TA2, it most likely wants to
prefer TA2, as that is the direction the transition is going, so TA2
should be preferred if both are supported.

When all servers have been updated to support both TA1, and TA2, then
there is no longer issues, as now all clients which do have TA2, use
it for all servers, and those which have not yet updated to TA1, still
use that.

So the idea is to allow iterative updating of the system, without the
need of flag days or configuration settings to specify things. I.e.,
IKEv2 will automatically detect which TA is supposed to be used, and
use it. 

> I am sure everything is possible, i am just looking for the
> most simple example where that is actually used. I think for
> a specific VPN all my VPN clients always only had one cert.

Current VPNs are quite often very strictly configured to use exactly
the configuration they have, and quite often they use their own IPsec
implementation, i.e., they might not use the IPsec provided by the
operating system.

Things get more complicated if you actually want to have multiple VPNs
in your system each using the same operating system provided IPsec,
but just provide configuration information to that. Of course in that
case you might also have overlapping configurations causing issues,
and thats why I think most of the VPNs are now standalone type of
implementations. 

> > > And IPsec implementations are prone to not
> > > support binding to different UDP ports for different purposes,
> > > so there are always workarounds needed to support multiple
> > > purposes on the same device simultaneously.
> > 
> > In IKEv2 there is IDi, and IDr for specifying purposes mostly. When
> > initiator connects to the host, it can send both IDi (who he is), and
> > IDr (who he wants to talk to). Then responder can see that if he
> > understands IDr (it might be FQDN like example.com or
> > company2.com), and if so, it will return same IDr for its own
> > identity. Initiator tells his own identity in the IDi.
> > 
> > Responder can also return something completely different for IDr, if
> > it does no recognize the IDr, or even if it recognizes, i.e. it might
> > actually return IDr of mail.example.com when initiator send IDr of
> > example.com.
> > 
> > This is similar to what TLS and SNI does, or Host-header in the http.
> > 
> > Again this is only hint, and responder can use that information, or it
> > might ignore it if it feels like so.
> 
> Ok, so i hope i am correct to state that in general we could have
> ignored specifying IDi/IDr in our profile as its not necessary.
> 
> I forgot who proposed we put our nodes IPv6 address into this
> field, but the IPv6 address itself doesn't seem to be very
> useful to identify whether the nodes are in the same network.
> 
> The full name of a node was so far an rfc822Name, now i need to
> fix it to become what i think is going to be a GeneralName,
> and hence it might best be to signal IDi/IDr as ID_DER_ASN1_GN

Note, that the IDi and IDr are what is matched against the IPsec
policy, i.e., nothing in the certificate will cause policy to be
selected, only IDi and IDr does that. After IDi and IDr identifies the
suitable policy, we can go and match the information in the policy
against the certificate. That policy might include information like
how to match the certificate subject alt name or subject name to
identity or information in the policy.

It could even include the list of trusted TAs, but this has the
problem that we need to send list of TAs before we know the ID of the
other end, so in that case we need to send union of all possible TAs
in all of our policies. 

> > Sending full chain including TA is not helpful at all in normal case,
> > and is only marginally helpful for failure cases. The question is that
> > do you want to waste resources 99.99% of time so that in the 0.01%
> > cases there might be slight change that it might be helpful for when
> > someone starts actually doing debugging.
> 
> The use-case are long-lived IPsec connections between routers
> that would stay up until a node reboots, a link fails or somebody
> misplugs a cable. We had network operators that plugged in a UPS
> to a router when they moved routers to a new building after 10 years,
> just because they wanted to keep their router-uptime record.
> 
> For all i care, IKEv2 could be performed by two dancing turtles
> singing the packets. There is no advertisement revenue in this
> use-case in faster IKEv2 handshake. Not even an annoyed toerless
> waiting for his notebook VPN to come up ;-). Sure at some point
> in time after this is sucessfull in enterprise SP backbone networks,
> we will have to revisit optimizing for low-end IoT, but this particular
> issue is by far not the only one for that space.
> 
> We disagree on your surely scientifically analyzed benefit of
> 0.001%. Even a 0.1 % of cases benefit could be a huge benefit.
> In big, badly manaed pops, some link diagnostics is taking weeks
> for folks to bring in better diagnostics gear. Can i promise
> our idea for better IKEv2 diagnostic will help ? No, we can't but
> we only need to establish IMHO that it does not hurt, and i think
> it does not.

Note, that to be able to see the TA sent you need debugging logs. The
certificates are sent inside the encrypted packets, and only way to
know what was there is to have IKEv2 implementation print that
information to some kind of debugging log. So even if you send them
inplace, that does not mean that you can actually use them when
debugging. I would assume most of the mobile handset IPsec
implementations do not provide such debugging logging in general, you
might need to enable some kind of debugging mode or restart the system
in a way where you can see console log etc.

This same of course applies to the end entity certificate, and that
might be issue, i.e., if the IPsec implementation does not even print
that out then you have hard time debugging issues. 

> > I think it is better to save bytes 100% of time...
> 
> You assume that you have other diagnostics channels, and if you
> do, i do agree. In our case we're at the lowest level/first stuff
> that establishes connectivity.

As I said you need other diagnostic channels even if you send the TA,
as it is inside encrypted packet. Of course if you can replace the
device with another device having diagnostic prints and logging using
same IP-address you can make claim to be the responder, and print
debugging information out from there, but it does require you act as a
responder yourself. In that case your special diagnostic tool can
provide much better debugging information anyways than what is
normally provided by the IPsec implementations.

> > Especially when the content you are sending is binary blob
> > which is quite meaningless unless you first decode it and show it to
> > user somehow. What kind of user is going to see these debug printouts
> > printing out the CA?
> 
> I imagine a GUI at the NOC of a network operator:
> 
> Unauthenticated ACP neigbhor on NodeXXX/InterfaceYYY:
>       node:<ipv6addr>@<acp-domain-name>
>    Click <here> to see cert chain diagnostics
> 
> And <here> would show similar decode of the cert chain including TA
> as you would get from e.g.: clicking on the trust icon on a browser.  

So you plan to have IPsec implementations actually saving that
information somewhere, and providing that debugging information to
that gui by some unspecified management protocol, i.e., modify the
ipsec implementations you use to provide the information. Currently I
think most of the implementations do not provide it in easy to use
way...

Luckily in IKEv2 you do need that information much less often than
what happend with IKEv1, where there was lots of parameters which had
to be exactly same in both ends, so only way to verify that was to
have detailed debug logs and check what other end proposed, and what
our policy required.

> Thats about the minimum. I wish we would have not been called back
> using rfc822Name for our solution, because then we could have
> even easily authenticated such a peer from a different domain
> by relying on ACME certificates. Oh well.
-- 
[email-masked]

---

From: Michael Richardson <[email-masked]>
Date: Fri, 17 Jul 2020 18:10:36 -0400

Tero Kivinen <[email-masked]> wrote:
    Tero> Note, that initiator might not have any certificate from CA1 or CA2,
    Tero> it might actually use completely different CAs, i.e., the list of
    Tero> certificate authorities can be asymmetric.

    Toerless> Good explanation. So, whats the most simple example for this ?
    Toerless> I have a single VPN but multiple geographic headends and
    Toerless> there is an obfuscated logic what certs each headends accepts,
    Toerless> so i do have multiple and certreq helps me to choose ?

    > TA1, and some have both TA1, and TA2.

I am not really following this discussion very well.
I'm not sure if this is general advice or ACP specific advince.

In the case that you describe (RSA->ECDSA),  I think that we would sign TA1
with TA2, or vice-versa. (probably one and then the other a month later).
We can feed/update the trust anchors when certificates are renewed, or we can
use draft-ietf-netconf-trust-anchors to change things (the advantages of
having really good security for in-band management)

I prefer relatively short expiries for certificates in the ACP because <STAR reasons>.
I think that all systems have to support both RSA and ECDSA for a big
overlapping period.  I don't think this is a problem.

I think that there is not a problem having a RSA key (TA1) sign an ECDSA EE cert.
I know that I've done it, but I know that some think it uncool.

    > m^Htake months, as it is possible that some of the clients are not

That's true in general, particularly for remote access situations, but ACP
nodes are mostly always reachable.  If they get turned off and put in a box,
then when they return, they have a valid certificate, or they have to
bootstrap again, and that's okay.

    > it for all servers, and those which have not yet updated to TA1, still

You are imagining a period of time where we are installing the trust anchors,
and I don't see it has to be like that.

Have TA1 sign TA2 as a sub-CA, and then pass out TA1->TA2->EE(ecdsa) chain to the
new nodes.  Once this is done for all nodes, you can drop TA1.
In reality, there are probably at least an extra level of CA, possibly two.

Do you think that I should include this situation into draft-richardson-registrar-operational-considerations?

    > IKEv2 will automatically detect which TA is supposed to be used, and

Agreed.

    > identity or information in the policy.

So, we've been forced to use otherName, and this will cause new code in some
of the IKEv2 daemon that we need to use :-(

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Tero Kivinen <[email-masked]>
Date: Tue, 21 Jul 2020 14:00:30 +0300

> I'm not sure if this is general advice or ACP specific advince.

This discussion was not ACP specific, this was just generic reasons
why CERTREQ etc are usuful in general. Anyways I think most ACP cases
do not need CERTREQ, but on the other hand there might be cases where
it is still useful there, and thats why I would hate to see profile
that forbits its use. I.e., it is fine to say in profile, that
CERTREQs might not be needed, but saying that they should/must not be
sent is bad idea. 

> having really good security for in-band management)

If you have proper management, then you can do almost instanteneous
changes, but CERTREQ allows you do do similar things even when you do
not have any kind of management system configuring devices, with much
longer overlaps in configuration. 

> I prefer relatively short expiries for certificates in the ACP
> because <STAR reasons>. I think that all systems have to support
> both RSA and ECDSA for a big overlapping period. I don't think this
> is a problem.
> 
> I think that there is not a problem having a RSA key (TA1) sign an
> ECDSA EE cert. I know that I've done it, but I know that some think
> it uncool.

Sometimes the problem is that there are devices there which do not
support ECDSA at all, which means you are stuck with RSA for both root
and EE for all devices. With CERTREQ some parts of the network can
start use full ECDSA TA and EE and still interoperate with those
devices which support both and get the "coolness" factor of not using
RSA at all :-)

Other similar cases comes when you merge authorization domains
together, i.e., two companies merge and they both had their own CAs
before, and not all devices support both CAs, so you need to pick
correct certificate one when talking to one or another device. 

> Have TA1 sign TA2 as a sub-CA, and then pass out TA1->TA2->EE(ecdsa)
> chain to the new nodes. Once this is done for all nodes, you can
> drop TA1. In reality, there are probably at least an extra level of
> CA, possibly two.
> 

The text I provided earlier was just general IPsec case, I have not
checked out situation in this specific case, so I cannot comment on
that before I read that document... 

> of the IKEv2 daemon that we need to use :-(

No, you are not forced to use anything. Your ID could be KEY_ID with
just random 128-bit binary blob, and you could find your policy entry
based on that. Then in the policy entry there might be information
what should be present in the certificate, or even just hash of the
raw public key (which could be also be sent inside the certificate in
some cases).

IDi and IDr are used to find the policy, policy information is used to
verify that the authorization information provided are correct. That
information usually includes trust anchors, and information what
should be in the certificate signed by those trust anchors.
-- 
[email-masked]

---

From: Michael Richardson <[email-masked]>
Date: Tue, 21 Jul 2020 10:54:25 -0400

> CERTREQs might not be needed, but saying that they should/must not be

I hope that at no point do we forbid the use of anything.
A peer might not expect it/do anything.
The important part is that we say what pieces we *do* require.
There is one more revision of ACP coming out on Monday, and I hope that is
the last one that the IESG will approve on the next telechat.

    > longer overlaps in configuration.

I think that proper management will probably not be in revision 1 of products.
However, a killer-use-case for the ACP is for SDN control channels, so I
could be wrong.

    > devices which support both and get the "coolness" factor of not using

I think that we say that we will tolerate RSA :-)
So you are right: there could be devices which do not do ECDSA on day one,
and in which case, you can't switch the keys at all.

    > correct certificate one when talking to one or another device.

Yes, this is an important case.

    > No, you are not forced to use anything. Your ID could be KEY_ID with

No, the ACP document has been forced to use otherName in it's
certificates. That's nothing to do with IPsec/IKEv2.
But, that that's what IKEv2 will have to deal with.

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 23 Jul 2020 13:53:36 +0200

> and in which case, you can't switch the keys at all.

I don't think this issue applies to ACP because both RSA and ECC / ECDSA
certificates are mandatory. Please check draft-ietf-anima-autonomic-control-plane-27,
section 6.1.1. I hope this should be sufficient, if not, then any text
fixes welcome.

Remember that ACP is not only summetric (no dedicated client/servers), but
also greenfield: aka: no need to think about IN THIS REVISION OF ACP about
legacy compatibility. IF/WHEN in the future we do provide update/extensions
to ACP, THEN we may get into considerations for two generations of nodes
talking to each other.

> Yes, this is an important case.

Right now, draft-ietf-anima-autonomic-control-plane-27 does only specify to still
send a certificate, even if a CERTREQ was received from the peer, and no certificate
would match the TA in the CERTREQ. We discussed how this was for diagnostics.
See section 6.7.3.1.2.

There is nothing in the current ACP text to say whether or not to send CERTREQ
right now. Given how the ACP just provides a refinement profile in section
6.7.3.1.2. on top of RFC8247, and given how RFC8247 does not mention CERTREQ,
that leaves is with a "MAY send CERTREQ" (i paraphrase) from RFC7296 section
3.8.

The domain merge is also of interest in ACP, so we could add a stronger
requirement against CERTREQ, but given how the documents we reference
(RFC8247/RFC7296), i am a bit suspicious about doing so. If CERTREQ is
so useful in non-ACP use caes, why is there no stronger than MAY in RFC7296
or RFC8247 ? Oversight in those IKEv2 RFCs ? Or are there some downsides
coming with it ?

I don't think i will add suc a requirement in this ACP spec because it
may be necessary to make the merge case work, but it is not sufficient,
and the set of components sufficient to support that use case is is
beyond the scope of this ACP base spec.

To give a hint: We only mandate use of EST, and if i wanted to support
a merge, or in general any use case where CERTREQ was useful, then i would
need to have nodes that would have two or more certs and CERTREQ is a
way to pick one of those multiple certs. And EST to the best of my
understandig does not allow to hand out more than one cert to a single
node. At least thats what i remember from having asked the quesrtion a few
times over the last 5 years.

We can still make merge work through extension documents, even without
fixing that EST limitation itself through other extensions of ACP, but
i think we would want a discuss first whether to rater do that or fix EST.

Or someone comes around and tells me this time that EST can be somehow
tricked to provide multiple certs to a single node.

> But, that that's what IKEv2 will have to deal with.

Actually, the ACP text currently says to use ID_IPV6_ADDR, but given how
the name of the ACP node is now an otherName, we _could_ signal it
via ID_DER_ASN1_GN, but IMHO ID_IPV6_ADDR is better.

Cheers
    Toerless

> _______________________________________________
> IPsec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ipsec

-- 
---
[email-masked]