Subject: Re: [OAUTH-WG] [UNVERIFIED SENDER] Re: client certs and TLS Terminating Reverse Proxies (was Re: I-D Action: draft-ietf-oauth-jwt-introspection-response-08.txt)

From: Neil Madden <[email-masked]>
Date: Fri, 1 Nov 2019 09:06:20 +0000

> 
>> On 31 Oct 2019, at 21:09, Richard Backman, Annabelle <[email-masked]> wrote:
> 
> The comparison the bearer tokens is illustrative of the problems I and others are pointing out:
> Bearer tokens are embedded in the value of the header, not the header itself, which partially alleviates the concern I raised regarding request signing algorithms.
> Bearer tokens are typically relatively short-lived, providing some mitigation against exfiltration through logs.
> Bearer tokens are typically dynamically generated, and are therefore less likely to be embedded in source code or config files in a project’s repository.
> Bearer tokens are called tokens, and are presented as secrets and are always expected to be treated as secrets.
>  
> The random header name is effectively an infinite-lifetime, statically defined bearer token presented in a way that does not at all make clear and obvious that it is a secret that must be protected, and in fact makes it more likely that it will be revealed, rendering it useless. And like all bearer tokens, even under ideal conditions it by definition CANNOT BE USED TO AUTHENTICATE THE SENDER.

Are you saying bearer credentials cannot be used for authentication? Passwords? Session cookies? API keys?

You can change the header name periodically if you really think it might leak.

>  
> There is a certain amount of irony in the idea of the security of a Mutual TLS deployment ultimately coming down to a bearer-token header name passed between the reverse proxy and the protected service. 😂

Well if you don’t do this then in the situations where it would come down to a bearer token, it currently comes down to absolutely nothing at all.

Let’s have a look at some concrete examples, as they are rather easy to find. 

In 2014, Tomcat found and fixed CVE-2014-0099 that was an integer overflow bug that allowed arbitrary request smuggling when running behind a reverse proxy, bypassing all request header sanitisation in the RP. If your app was trusting an X-SSL-Cert header for authentication then this is likely to lead to complete compromise.

Here’s an example of some people who spent hours after a DoS attack before they realised all the IP addresses in their server logs were spoofed because HAProxy hadn’t been configured to strip X-Forwarded-For headers that the backend was trusting. https://serverfault.com/questions/868362/prevent-x-forwarded-for-spoofing-in-haproxy?noredirect=1

Here’s another one with somebody wondering why their logs contain private space IP addresses: https://serverfault.com/questions/908724/private-address-space-ip-found-in-x-forwarded-for

Common theme: apps often blindly trust standard header names by default, while reverse proxies/load balancers commonly forward them by default. 

Here’s a security researcher writing about how to spoof an SSL client certificate header and listing the commonly used header names for several reverse proxies, along with simple ways to bypass header sanitisation: https://medium.com/@d0znpp/https-client-certificate-authentication-security-issues-part-1-3-ae59514357b1

Here’s an old critical vulnerability in Shibboleth that led to arbitrary spoofing of security attributes due to ambiguous normalization of header names after filtering: http://shibboleth.net/community/advisories/secadv_20060615.txt

Here’s an example of people misconfiguring Envoy’s support for client certificate forwarding and then discovering it can be spoofed. https://www.bountysource.com/issues/70466221-correct-behavior-for-x-forwarded-client-cert 

Here’s an example of an app being misconfigured when parsing X-Forwarded-For headers behind an RP leading to a bypass of API key authentication: https://github.com/home-assistant/home-assistant/issues/14345

If you search around of google or stack overflow you can easily find dozens more examples of people being bitten by misconfigured RPs and apps trusting headers. There is a good reason why Security Misconfiguration is in the OWASP top 10. 

Adding a random string to a header name means that in the majority of cases these all-too-common failures don’t immediately lead to a catastrophic compromise of your app. Sure, its not a perfect solution, but it usually stops a common vulnerability being immediately exploitable. Wouldn’t you rather that an attacker has to go to a little more trouble than trying a list of a dozen widely used header names in this situation?

It is not helpful to tell people not to make such a simple and essentially cost-free tweak to their configuration because in some world they might enable trace-level logging of all request headers in a production environment, or their network engineers might post packet captures in powerpoint presentations, or they might simultaneously be using headers to communicate security information to the backend while allowing untrusted clients on the same network to sniff network traffic in the clear.

The alternative at the moment is to do nothing at all, which we know leads to potential vulnerabilities because there is ample evidence in the public record. Stack overflow is a catalogue of people confused about configuring reverse proxy header forwarding. And yet This One Weird Trick Could Save You From Reverse Proxy Trusted Header Misconfiguration Vulnerabilities!

>  
> > if you forget to validate the signature *nothing fails*
>  
> Replace the HMAC with encryption and you solve that problem, as it forces the service to decrypt the value using the correct key in order to access the client certificate data. Whether or not that’s worth doing is something we can debate in the context of an actual proposal.

Sure, and if the final proposal for a standard header comes with optional/recommended support for authenticated encryption of values (and reverse proxy vendors actually implement it) then I’ll be happy enough. And when that finally reaches a good level of adoption I’d advise people to use it. Just don’t make it a JWT with a choice of a dozen algorithms. 

— Neil