Subject: [TLS] TLS 1.3 / RSA-PSS and unusual key sizes

From: Hanno Böck <[email-masked]>
Date: Fri, 27 Jan 2017 14:30:03 +0100

Hi,

I wanted to warn people about a potential source of bugs with the
deployment of RSA-PSS in TLS 1.3.

Usually the RSA key modulus is a multiple of 8 (2048, 4096 etc.).
However there's no rule that RSA keys can't have other sizes.

Implementing PSS with support for arbitrary key sizes is a bit more
complicated than implementing it for multiples of 8. I wrote the PSS
implementation of NSS as a summer of code project a couple of years ago
and I remember that my first implementation completely failed to
consider this. (The fix for that never got merged afair, I informed NSS
developers about this.)

Back then I also reported a bug in OpenSSL:
https://rt.openssl.org/Ticket/Display.html?id=2315&user=guest&pass=guest

Long story short: It's not unlikely that there are more PSS
implementations having problems with this.
So I strongly recommend that all implementors of TLS 1.3 test their
implementations for key sizes from n*8+1 to N*8+7.

Such keys are rare, but they do exist in the wild. If implementations
failing on that get shipped widely we may see random unexplained errors
when people start migrating to TLS 1.3 in masses.

I had actually considered proposing to change TLS 1.3 in a way that
such keys would be simply forbidden. But I did a check on the censys
data and there were too many of them in the wild, so I thought it
wasn't a feasible idea.

-- 
Hanno Böck
https://hboeck.de/

mail/jabber: [email-masked]
GPG: FE73757FA60E4E21B937579FA5880072BBB51E42

---

From: Tim Taubert <[email-masked]>
Date: Tue, 31 Jan 2017 09:20:26 +0100

> consider this. (The fix for that never got merged afair, I informed NSS

Thanks again Hanno for bringing this to our attention. We patched it
yesterday in NSS 3.29, soon to be merged into Firefox 53.

If anyone else is about to take a look, feel free to use the official
RSA-PSS vectors I converted to SPKI/PKCS8, so you don't have to:

https://hg.mozilla.org/projects/nss/file/c61324648525/gtests/pk11_gtest/pk11_rsapss_vectors.h

- Tim