Subject: Re: [OAUTH-WG] I-D Action: draft-ietf-oauth-jwt-introspection-response-08.txt

From: Torsten Lodderstedt <[email-masked]>
Date: Sun, 6 Oct 2019 15:31:22 +0200

Hi Travis,

thanks for your feedback! 

> On 26. Sep 2019, at 11:26, Travis Spencer <[email-masked]> wrote:
> 
> Some feedback on this draft compared to the last one that I read:
> 
> * The acronym RS and the term "resource server" are used
> inconsistently. It would read better IMO if the acronym was always
> used after being defined or if resource server was always spelled out.
> Same for AS, but less so. Maybe this is just me, so take it as you
> will.

I will check, especially the plural of RS (RSs?) looks weird to my eyes so I would like to prefer to sometime not to go with the acronym.

Do others have an opinion about this?

> 
> * In section 3, is it saying that JWE is MTI? From the end of section
> 5, it seems not. If so, this statement should be clarified IMO:
> 
>> To support encrypted token introspection response JWTs, the authorization server MUST also be provided with the respective resource server encryption keys and algorithms.
> 
> Maybe just change it to "If the AS supports encrypted token
> introspection response JWTs…"

It wasn’t meant as MTI sind we just provide the conceptual underpinning of the draft in section 3 and I would like to keep that focus. 

I turned the normative into editorial language and softened the wording of the whole paragraph. 

“In order to process the introspection requests in a secure and
    privacy-preserving manner, the authorization server must be able to identify, 
    authenticate and authorize resource servers. If the token introspection 
    responses shall be encrypted, the authorization server must also be provided with 
    the respective resource server encryption keys and algorithms.”   

> 
> * Instead of issuing an expired JWT (or in addition) I think it should
> be valid for the AS to reply with a 201, no content. For this reason,
> I think that section 5 should be updated thusly:
> 
>> If the access token is invalid, expired, has been revoked, or is not intended to be consumed by the calling resource server (audience), the authorization server MUST set the value of the response claim
> "active" to "false" if it chooses to issue a JWT. Otherwise, this
> claim is set to "true". If it does not issue a JWT, the AS MUST reply
> with an HTTP 204, no content, status code. If it does include a JWT
> with an "active" claim of "false", the AS MAY reply with an HTTP 204,
> no content, status code.
> 
> The rational is that the AS may not want to exert effort to make a
> signed/encrypted JWT that is expired. Not having the token is usually
> as good or better than having an expired one.

Inline with RFC 7662, the AS always provides data to the RS even in case the introspected access token is expired. 

> 
> * This statement makes the audience sound too narrow IMO:
> 
>> The value of the "aud" claims MUST identify the resource server receiving the token introspection response.
> 
> Because the AS's policy may stipulate that the audience could be
> others in addition to the RS some wording should be added like "the
> 'aud' claim MUST identify the resource server receiving the token
> introspection response and MAY contain other values based on its
> policy.”

We intentionally specified it that way to make prevent abuse of the token introspection response. See below my thoughts on your use case.

> 
> * The "M" in email in section 5 should not be capitalized IINM.

Good catch, changed it. 

> 
> * In section 5, this is very broad.
> 
>> The AS determines based on the RS-specific policy what claims about the resource owner to return in the token introspection response. The AS MUST ensure that the release of any privacy-sensitive data is legally based.
> 
> So, an AS must follow the law? Is that all that it's saying? Section 9
> seems to restate this point. Is conformance to the law really up to
> this doc to stipulate? If the AS breaks the law, it don't conform to
> the protocol. This seems very meta.

It is a direct response to IESG review comments. People worry about handling of privacy sensitive data what I totally understand. Instead of adding normative language given concrete advice (which does not belong into a technical specification) or prohibiting privacy sensitive data at all (which some might prefer) we added this general advice to remind implementers of their obligation.

I removed the duplication by replacing the sentence you cited with “See Section 9 for further considerations."

> 
> * 1st in section 9 should be spelled out as "first" and IINM it should
> be "first-party" since it's being used as an adjective.

You are the native speaker :-) I changed as you proposed.

> 
> * Last but certainly not least is the restriction that the current
> version places on disallowing of the introspection JWT response as an
> access token. This is done in numerous places (the note in section 5,
> 8.1, etc.). I understand that there are some objection to this usage,
> but we have had this protocol deployed for years, and it's running in
> dozens of customer's facilities. This will break real applications of
> this specification without a clear alternative. As we discussed in
> London last year at the IETF meeting, token exchange isn't a viable
> alternative (even still in its current draft form to the best of my
> knowledge). Without a workable alternative to move to, I emphatically
> but humbly request that this specification remove this restriction.

We clarified the intention of this draft due to various feedback (IESG review and other postings on the list). It is supposed to provide a JWT encoded introspection response and not a transformed access token. The main differences lay in the existence of the “active“ claim and the different „iat“ values. 

In order to prevent JWT confusion, the JWT shall be restricted to the originator of the introspection call. This prevents your use case, but how should we distinguish this from access token replay/abuse?

Also, the JWT produced cannot be sender constrained, which even more makes its use as an access token unadvisable.

In my perception, your proxy transforms the token to make it easier consumable by the target RS. 

I see two ways to approach it:

1) There is a (yet to be specified) function at the AS that transforms an opaque access token into an JWT-based access token while preserving its audience and other properties (like sender constraints). The proxy itself requires authorization to request that functionality but is not an audience. The access token might even be encrypted for the target RS and be unreadable for the proxy. 

Open: How would one implement sender constrained access tokens in that case? I’m asking since the receiving RS obviously has no access to the information from the TLS handshake since TLS termination happens at the proxy (or even in before the request hits the proxy). The RS would need to get provided with the cert fingerprint via a trustworthy header, i.e. the RS must be aware of the fact it sits behind a proxy.

2) If that’s the case, the RS could also process the token introspection response as received by the proxy. In order to check the audience restriction, the introspection response would need to contain another claim containing the original audience claims of the introspected access token. You could build that on top of draft-ietf-oauth-jwt-introspection-response. What would it take?
- the originator of the call would need permission to act on behalf of a RS (implementation detail of your authorization module)
- the JWT would need to contain an additional claim, e.g. “at_aud”, carrying the access token’s audience. 
- use a new header to carry the JWT introspection response to the RS (just beside the header carrying the cert fingerprint)

best regards,
Torsten.

> 
> 
> 
> On Fri, Sep 20, 2019 at 2:28 PM <[email-masked]> wrote:
>> 
>> 
>> A New Internet-Draft is available from the on-line Internet-Drafts directories.
>> This draft is a work item of the Web Authorization Protocol WG of the IETF.
>> 
>>       Title           : JWT Response for OAuth Token Introspection
>>       Authors         : Torsten Lodderstedt
>>                         Vladimir Dzhuvinov
>>       Filename        : draft-ietf-oauth-jwt-introspection-response-08.txt
>>       Pages           : 18
>>       Date            : 2019-09-20
>> 
>> Abstract:
>>  This specification proposes an additional JSON Web Token (JWT)
>>  secured response for OAuth 2.0 Token Introspection.
>> 
>> 
>> The IETF datatracker status page for this draft is:
>> https://datatracker.ietf.org/doc/draft-ietf-oauth-jwt-introspection-response/
>> 
>> There are also htmlized versions available at:
>> https://tools.ietf.org/html/draft-ietf-oauth-jwt-introspection-response-08
>> https://datatracker.ietf.org/doc/html/draft-ietf-oauth-jwt-introspection-response-08
>> 
>> A diff from the previous version is available at:
>> https://www.ietf.org/rfcdiff?url2=draft-ietf-oauth-jwt-introspection-response-08
>> 
>> 
>> Please note that it may take a couple of minutes from the time of submission
>> until the htmlized version and diff are available at tools.ietf.org.
>> 
>> Internet-Drafts are also available by anonymous FTP at:
>> ftp://ftp.ietf.org/internet-drafts/
>> 
>> _______________________________________________
>> OAuth mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/oauth
> 
> _______________________________________________
> OAuth mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/oauth

---

From: Travis Spencer <[email-masked]>
Date: Wed, 16 Oct 2019 16:23:56 +0200

> Hi Travis,

Hey there, Torsten. Sorry for the slow reply. Still jet lagged ;-)

> Do others have an opinion about this?

https://tools.ietf.org/html/rfc7322#section-3.6

> > be valid for the AS to reply with a 201, no content.

Sure, but is that a good line to follow? The client said it accepts
application/jwt, but the server returns application/json. This breaks
HTTP. (I know that that HTTP allows the server to return a different
media type than the one that was requested if there is only one
representation of the resource, but this is not the case here.) So, to
do this in a way that doesn't break (my understanding of) HTTP, the
client would have to accept both application/jwt and application/json.
Because the status code isn't helpful in determining which, the body
has to parsed based on the content-type response header value. I think
this makes it harder on the consumer than it needs to be and
undoubtedly slower.

> We clarified the intention of this draft due to various feedback (IESG review and other postings on the list). It is supposed to provide a JWT encoded introspection response and not a transformed access token.

I continue (perhaps out of will, I admit) not to see how what I'm
asking for a transformation and not a re-encoding of the same.

> The main differences lay in the existence of the “active“ claim and the different „iat“ values.

The addition of the active claim is a change, I admit that. The iat
claim could be the same as the original though.

> In order to prevent JWT confusion, the JWT shall be restricted to the originator of the introspection call.

In section 5, the draft has this: "The value of the 'aud' claims MUST
identify the resource server receiving the token introspection
response." By resource server here, does it mean, literally, the
caller? RFC 6749 seems to pin that down to one "server" so I think so.
If so, this is different from what's commonly used for audience (or at
least "common" among my experiences). As defined in the JWT spec
(https://tools.ietf.org/html/rfc7519#section-4.1.3), "[t]he 'aud'
(audience) claim identifies the recipients that the JWT is intended
for....The interpretation of audience values is generally application
specific." This divergence is confusing. I didn't notice it on first
(or second) read, and highly recommend converging these. I think the
JWT definition makes this spec more useful.

> This prevents your use case, but how should we distinguish this from access token replay/abuse?

As you say in section 8.1: "such an attack can be prevented like any
other token substitution attack by restricting the audience of the
JWT." I just think the audience that is defined in this spec is too
narrow. Audience is more like a domain than a web service client
(e.g., the introspection API client). I see the gateway and the
microservices behind it as the same Resource Server with the same
audience if they are all in the security domain. I see the gateway as
being able to straddle multiple audiences though because it could be
fronting multiple clusters of microservices that are in different
domains (i.e., have different audiences). This straddling of different
domains means the gateway either has a different audience or multiple.

> Also, the JWT produced cannot be sender constrained, which even more makes its use as an access token unadvisable.

Sender constrained tokens do provide additional security properties
that are desirable in some cases, but they are not the only kind that
are advisable. It depends on the situation, right?

> In my perception, your proxy transforms the token to make it easier consumable by the target RS.

This is one of the drivers for what I'm advocating: simple API development.

> 1) There is a (yet to be specified) function at the AS that transforms an opaque access token into an JWT-based access token while preserving its audience and other properties (like sender constrai…

This hypothetical spec would vary from this one in what ways exactly?
That's not a facetious question, but something I think we should
honestly catalog. You listed some ideas below, but I need to think
more about these. That gap will help us make informed decisions and
relates to Justin's question about the gap to Token Exchange as well.
See my concern below about how we fill this gap.

> Open: How would one implement sender constrained access tokens in that case? I’m asking since the receiving RS obviously has no access to the information from the TLS handshake since TLS terminatio…

This is unfortunately the typical case even with the mTLS draft. This
is because SSL is almost never terminated by the AS; in my experience,
TLS termination is instead handled by some very fast proxy.[1] In such
cases, the proxy will pass the cert through to the AS in some
undefined HTTP header with some undefined encoding. The mTLS spec
should have defined this IMO, as it prevents interop for a primary use
case -- sender constrained tokens.

In the case of introspection, I think this verification of the
confirmation method is another aid the proxy could provide to the
microservice. The proxy climbed the chain as it terminated SSL,
fetched CRLs/OCSPs, called LDAP server to get intermediate certs, etc.
Verifying the hash of the cert is not hard for it do as well. If the
proxy doesn't want to parse the body of the introspection HTTP
response though, it will have to forward the cert. The HTTP header and
encoding used is undefined, and that is unfortunate for the sake of
interop. For proxy auth though, we have the HTTP spec, so that could
be leveraged.

> 2) If that’s the case, the RS could also process the token introspection response as received by the proxy. In order to check the audience restriction, the introspection response would need to cont…

This is my concern: we end up with a lot of specs:

1. Introspection RFC
2. JWT introspection RFC
3. Token exchange
4. This hypothetical RFC that profiles #1, #2 and/or #3

Seems overkill. Can't we make more general specs that cover more use
cases? If we don't, the cannon will be so extensive that it'll be
unpredictable for many. This could impede adoption.

[1] I have only come across a situation where the AS terminated SSL
one time, and that was because its mutual TLS impl of the AS didn't
allow for cert-based auth unless it did. (This wasn't Curity which
allows for proxy-based mTLS but some other product.) The customer
reluctantly allowed it and only did TLS-based routing in the proxy.

---

From: Benjamin Kaduk <[email-masked]>
Date: Sun, 20 Oct 2019 19:05:46 -0700

Just on one narrow point:

> case -- sender constrained tokens.

It's not clear to me that mTLS should have defined a protocol from proxy to
backend; that seems like it could be a fairly generic thing and I know of
several people that are working on similar things, to one degree or
another.  draft-schwartz-tls-lb is the example that I can most easily find
in my archives; though it's working with non-TLS-terminating cases, there
is probably some commonality with TLS-terminating cases as well.

-Ben

---

From: Brian Campbell <[email-masked]>
Date: Wed, 23 Oct 2019 06:10:59 -0600

I agree with Ben here that it's not at all clear that the OAuth MTLS
document should have defined a protocol from proxy to backend. I'd go so
far as to say it's well outside the scope of the document and even the
working group. Admittedly It would be nice if such a thing existed but it
would have much wider applicability than one narrow profile of OAuth.

On Sun, Oct 20, 2019 at 8:06 PM Benjamin Kaduk <[email-masked]> wrote:

> > > Open: How would one implement sender constrained access tokens in that
> case? I’m asking since the receiving RS obviously has no access to the
> information from the TLS handshake since TLS termination happens at the
> proxy (or even in before the request hits the proxy). The RS would need to
> get provided with the cert fingerprint via a trustworthy header, i.e. the
> RS must be aware of the fact it sits behind a proxy.
> >

-- 
_CONFIDENTIALITY NOTICE: This email may contain confidential and privileged 
material for the sole use of the intended recipient(s). Any review, use, 
distribution or disclosure by others is strictly prohibited.  If you have 
received this communication in error, please notify the sender immediately 
by e-mail and delete the message and any file attachments from your 
computer. Thank you._

---

From: Justin Richer <[email-masked]>
Date: Wed, 23 Oct 2019 14:13:04 +0000

I also agree. Would it be possible to get this pushed to http or tls? It would be more appropriate there, and very helpful to have a general spec for this.

— Justin

On Oct 23, 2019, at 2:10 PM, Brian Campbell <bcampbell=[email-masked]<mailto:bcampbell=[email-masked]>> wrote:

I agree with Ben here that it's not at all clear that the OAuth MTLS document should have defined a protocol from proxy to backend. I'd go so far as to say it's well outside the scope of the document and even the working group. Admittedly It would be nice if such a thing existed but it would have much wider applicability than one narrow profile of OAuth.

On Sun, Oct 20, 2019 at 8:06 PM Benjamin Kaduk <[email-masked]<mailto:[email-masked]>> wrote:
Just on one narrow point:

On Wed, Oct 16, 2019 at 04:23:56PM +0200, Travis Spencer wrote:
> > Open: How would one implement sender constrained access tokens in that case? I’m asking since the receiving RS obviously has no access to the information from the TLS handshake since TLS termination happens at the proxy (or even in before the request hits the proxy). The RS would need to get provided with the cert fingerprint via a trustworthy header, i.e.. the RS must be aware of the fact it sits behind a proxy.
>

-Ben

_______________________________________________
OAuth mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/oauth

CONFIDENTIALITY NOTICE: This email may contain confidential and privileged material for the sole use of the intended recipient(s). Any review, use, distribution or disclosure by others is strictly prohibited..  If you have received this communication in error, please notify the sender immediately by e-mail and delete the message and any file attachments from your computer. Thank you._______________________________________________
OAuth mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/oauth

---

From: Benjamin Kaduk <[email-masked]>
Date: Thu, 24 Oct 2019 10:03:26 -0700

On Wed, Oct 23, 2019 at 10:13:04AM -0400, Justin Richer wrote:
>    I also agree. Would it be possible to get this pushed to http or tls? It
>    would be more appropriate there, and very helpful to have a general spec
>    for this.

I think it's possible to get such work done in one of those places, but
first someone has to actually write a draft.  Barring that, a HotRFC or
secdispatch slot in Singapore would probably be good for drumming up
interest.  In related news, I am told that draft-duke-quic-load-balancers
had some time in the QUIC interim this month, and may be moving towards
adoption, so time may be short if we want to have a fully general solution.

-Ben

---

From: Rifaat Shekh-Yusef <[email-masked]>
Date: Fri, 25 Oct 2019 10:02:41 -0400

You might want to look at RFC7239, which is trying to address the issue of
the loss of information by proxies.
https://tools.ietf.org/html/rfc7239

The document does not have a parameter to carry the client certificate
information, but it allows for new parameters to be defined.

Would that help in this case?

Regards,
 Rifaat

---

From: Brian Campbell <[email-masked]>
Date: Fri, 25 Oct 2019 13:46:35 -0600

Few thoughts, comments, & questions.

I've always kind of assumed that the ship had already sailed on doing
anything standards related with this because implementations out in the
wild like Apache, NGNIX, IIS, etc. were already providing ad hoc solutions
and had been doing so for a long time. But this thread at least hints that
there may yet be an appetite for such a standardization effort?

I'll be in Singapore but my (long) flight is scheduled to land about the
time when HotRFC starts. I could perhaps do something for secdispatch
though. Ben, is a draft needed for that or can secdispatch be engaged with
some slides amounting to a problem statement? I'm just not confident a
draft could be published in the next week or so before the cut off.

I did put it some work on a conceptually similar issue around token binding
with "HTTPS Token Binding with TLS Terminating Reverse Proxies" - https://
datatracker.ietf.org/doc/draft-ietf-tokbind-ttrp/ not too long ago. It was
a WG item that had gone through WGLC but kinda stalled out waiting on the
chair(s) for the shepard writeup when adoption questions around token
binding came up.

There are lots of ways to approach the problem and solution but I think
perhaps a lot could be taken from that document and applied to the similar
client cert situation.

I did look at RFC7239 when doing that and it could have been made to work
but felt the fit wasn't quite right and would have been more cumbersome to
use than not.

---

From: Benjamin Kaduk <[email-masked]>
Date: Fri, 25 Oct 2019 15:38:05 -0700

> Would that help in this case?

That is interesting and related, though the proposed work I remember
hearing about recently is not quite in the same space.  Specifically, it
wanted to have a secure protocol between proxy and backend, and it's not
clear that reusing https for that is the right thing.

-Ben

---

From: Rifaat Shekh-Yusef <[email-masked]>
Date: Sat, 26 Oct 2019 17:54:58 -0400

On Fri, Oct 25, 2019 at 3:47 PM Brian Campbell <[email-masked]>
wrote:

> wild like Apache, NGNIX, IIS, etc. were already providing ad hoc
> solutions and had been doing so for a long time. But this thread at least
> hints that there may yet be an appetite for such a standardization effort?
>
> I did put it some work on a conceptually similar issue around token
> binding with "HTTPS Token Binding with TLS Terminating Reverse Proxies" -
> https://datatracker.ietf.org/doc/draft-ietf-tokbind-ttrp/ not too long
> ago. It was a WG item that had gone through WGLC but kinda stalled out
> waiting on the chair(s) for the shepard writeup when adoption questions
> around token binding came up.
>
Can you elaborate on this?
These days, with the zero trust model in mind, there are orchestration
tools, e.g. Istio, that easily allows you to establish an MTLS channel
between the reverse proxy/load balancer/API GW and the backend servers.
Why is that not sufficient?
Which part is cumbersome?

Regards,
 Rifaat

>> You might want to look at RFC7239, which is trying to address the issue
>> of the loss of information by proxies.
> *CONFIDENTIALITY NOTICE: This email may contain confidential and
> privileged material for the sole use of the intended recipient(s). Any
> review, use, distribution or disclosure by others is strictly prohibited.
> If you have received this communication in error, please notify the sender
> immediately by e-mail and delete the message and any file attachments from
> your computer. Thank you.*

---

From: Brian Campbell <[email-masked]>
Date: Mon, 28 Oct 2019 06:41:48 -0600

> Which part is cumbersome?

What I meant was only that in the course of writing
https://tools.ietf.org/html/draft-ietf-tokbind-ttrp-09, which aims to
define HTTP header fields that enable a TLS terminating reverse proxy to
convey information to a backend server about the validated Token Binding
Message received from a client, it seemed more straightforward and
sufficient for the use-case to use new HTTP headers to carry the
information rather than to use new fields in the Forwarded header framework
from RFC7239.

---

From: Rifaat Shekh-Yusef <[email-masked]>
Date: Mon, 28 Oct 2019 11:05:10 -0400

Thanks Brian,

I guess my question is: given RFC7239 and the fact that it is
straightforward to secure the channel between the terminating reverse proxy
and the backend service in a cluster, is there anything, from a standard
perspective, that we need to do beyond defining a new parameter to carry
the client certificate information?
You seem suggest that the answer is yes. If so, can you please elaborate on
why is that?

Regards,
 Rifaat

On Mon, Oct 28, 2019 at 8:42 AM Brian Campbell <[email-masked]>
wrote:

>> On Fri, Oct 25, 2019 at 3:47 PM Brian Campbell <
>> [email-masked]> wrote:
>>
>>>
>>> I did look at RFC7239 when doing that and it could have been made to
>>> work but felt the fit wasn't quite right and would have been more
>>> cumbersome to use than not.
>>>
>>>

---

From: Brian Campbell <[email-masked]>
Date: Mon, 28 Oct 2019 09:32:27 -0600

I don't think there's anything beyond defining something to carry the
client certificate information (including the format and encoding). And it
could well be a new RFC7239 parameter. Or it might just be a new HTTP
header on its own.

On Mon, Oct 28, 2019 at 9:05 AM Rifaat Shekh-Yusef <[email-masked]>
wrote:

> You seem suggest that the answer is yes. If so, can you please elaborate
> on why is that?
>
> Regards,
>  Rifaat
>
>
>

---

From: Rifaat Shekh-Yusef <[email-masked]>
Date: Mon, 28 Oct 2019 12:00:02 -0400

Ok, then we are on the same page.
The reason I am asking is that I have a use case where I need such a
mechanism.

What is the advantage you see for defining a new HTTP header over extending
RFC7239?

Regards,
 Rifaat

On Mon, Oct 28, 2019 at 11:32 AM Brian Campbell <[email-masked]>
wrote:

>> On Mon, Oct 28, 2019 at 8:42 AM Brian Campbell <
>>> On Sat, Oct 26, 2019 at 3:55 PM Rifaat Shekh-Yusef <
>>> [email-masked]> wrote:
>>>
>>>>

---

From: Neil Madden <[email-masked]>
Date: Mon, 28 Oct 2019 16:32:37 +0000

While there are some benefits to standardizing headers for this kind of communication, there are some significant downsides - particularly when using headers to communicate critical security information like certs. It is *very* easy to misconfigure a reverse proxy to not strip Forwarded (or whatever) headers from incoming requests, allowing a client to simply supply a certificate as a header without authenticating the TLS connection with the corresponding private key. One good practice to prevent this is to pick a random and unguessable header name (configurable per installation) to be used for communicating the certificate, rather than using something fixed and standard. That way even if you misconfigure the proxy an attacker still has to try and guess the correct header name.

I suppose the same thing could be accomplished by having an extension for including a shared secret (or HMAC tag) in the header to authenticate it.

-- Neil

> On 28 Oct 2019, at 15:32, Brian Campbell <bcampbell=[email-masked]> wrote:
> 
> I don't think there's anything beyond defining something to carry the client certificate information (including the format and encoding). And it could well be a new RFC7239 parameter. Or it might just be a new HTTP header on its own.  
> 
> On Mon, Oct 28, 2019 at 9:05 AM Rifaat Shekh-Yusef <[email-masked] <mailto:[email-masked]>> wrote:
> Thanks Brian,
> 
> I guess my question is: given RFC7239 and the fact that it is straightforward to secure the channel between the terminating reverse proxy and the backend service in a cluster, is there anything, from a standard perspective, that we need to do beyond defining a new parameter to carry the client certificate information?
> You seem suggest that the answer is yes. If so, can you please elaborate on why is that?
> 
> Regards,
>  Rifaat
> 
> 
> 
> On Mon, Oct 28, 2019 at 8:42 AM Brian Campbell <[email-masked] <mailto:[email-masked]>> wrote:
> 
> 
> On Sat, Oct 26, 2019 at 3:55 PM Rifaat Shekh-Yusef <[email-masked] <mailto:[email-masked]>> wrote:
> 
> On Fri, Oct 25, 2019 at 3:47 PM Brian Campbell <[email-masked] <mailto:[email-masked]>> wrote:
> 
> I did look at RFC7239 when doing that and it could have been made to work but felt the fit wasn't quite right and would have been more cumbersome to use than not.  
> 
> 
> These days, with the zero trust model in mind, there are orchestration tools, e.g. Istio, that easily allows you to establish an MTLS channel between the reverse proxy/load balancer/API GW and the backend servers.
> What I meant was only that in the course of writing https://tools.ietf.org/html/draft-ietf-tokbind-ttrp-09 <https://tools.ietf.org/html/draft-ietf-tokbind-ttrp-09>, which aims to define HTTP header fields that enable a TLS terminating reverse proxy to convey information to a backend server about the validated Token Binding Message received from a client, it seemed more straightforward and sufficient for the use-case to use new HTTP headers to carry the information rather than to use new fields in the Forwarded header framework from RFC7239.    
>  
> 
> CONFIDENTIALITY NOTICE: This email may contain confidential and privileged material for the sole use of the intended recipient(s). Any review, use, distribution or disclosure by others is strictly prohibited.  If you have received this communication in error, please notify the sender immediately by e-mail and delete the message and any file attachments from your computer. Thank you.
> 
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/oauth <https://www.ietf.org/mailman/listinfo/oauth>

---

From: "Salz, Rich" <[email-masked]>
Date: Mon, 28 Oct 2019 16:48:34 +0000

Sorry for jumping into this late.

Client <--> proxy <--> backend

The C/P side is protected by TLS.  There must be similar protection on the P/B side, such as client-cert, or a signature over the certificate being forwarded, right?

---

From: Rifaat Shekh-Yusef <[email-masked]>
Date: Mon, 28 Oct 2019 14:51:01 -0400

On Mon, Oct 28, 2019 at 12:48 PM Salz, Rich <[email-masked]> wrote:

> The C/P side is protected by TLS.  There must be similar protection on the
> P/B side, such as client-cert, or a signature over the certificate being
> forwarded, right?
>

To avoid the misconfiguration issue Neil raised, you probably need both: a
client-cert *and* a signature over the certificate being forwarded,
This could still be achieve by extending RFC7239 with new parameter(s).

Regards,
 Rifaat

---

From: "Salz, Rich" <[email-masked]>
Date: Mon, 28 Oct 2019 18:55:19 +0000

*   To avoid the misconfiguration issue Neil raised, you probably need both: a client-cert and a signature over the certificate being forwarded,

I am not so sure.  One can argue that transport-level identity should be secured by transport-level.  But installing a client certificate on a reverse proxy can be difficult.  (Not if the reverse proxy is a CDN, of course :) And I don’t see how having both prevents misconfiguration, but that might be my fault.

  *   This could still be achieve by extending RFC7239 with new parameter(s).

I have no opinion on this part of it.

---

From: Brian Campbell <[email-masked]>
Date: Mon, 28 Oct 2019 13:34:49 -0600

When working on the Token Binding TTRP draft I guess I saw the advantage as
being the simplicity or setting, sanitizing, and reading an independent
header that's only conveying the token binding info about the TLS
connection between the "outside" client/browser and the proxy, (which is
the application/service from that client's perspective). That simplicity
comes at the cost of not being able to represent that information about
multiple hops as would be the case extending RFC7239. The simplicity won
the day in that particular tradeoff. I suspect that would be the case with
client cert info too. But that's admittedly just conjecture at this point.
Although it is, as far as I know, inline with the capabilities of the
popular reverse proxies.

On Mon, Oct 28, 2019 at 10:00 AM Rifaat Shekh-Yusef <[email-masked]>
wrote:

> What is the advantage you see for defining a new HTTP header over
> extending RFC7239?
>
> Regards,
>  Rifaat
>
>
>
> On Mon, Oct 28, 2019 at 11:32 AM Brian Campbell <

---

From: Rifaat Shekh-Yusef <[email-masked]>
Date: Tue, 29 Oct 2019 07:57:25 -0400

Maybe I misunderstood what you meant by "client-cert". If you meant the
proxy client certificate, then that is obviously not enough. You seem to
suggest that you meant the remote client certificate to be installed on the
proxy to be used with the backend system; if this is the case, then this
would work and you would not need the signature, but the issue I see with
this approach is that you need to reconfigure the proxy every time you
change the client certificate, which is not practical if the certificate is
short lived.

Regards,
 Rifaat

On Mon, Oct 28, 2019 at 2:55 PM Salz, Rich <[email-masked]> wrote:

>
>    - To avoid the misconfiguration issue Neil raised, you probably need
>    both: a client-cert *and* a signature over the certificate being
>    forwarded,
>
>
>
> I am not so sure.  One can argue that transport-level identity should be
> secured by transport-level.  But installing a client certificate on a
> reverse proxy can be difficult.  (Not if the reverse proxy is a CDN, of
> course :) And I don’t see how having both prevents misconfiguration, but
> that might be my fault.
>
>
>
>    - This could still be achieve by extending RFC7239 with new
>    parameter(s).
>
>
>

---

From: "Salz, Rich" <[email-masked]>
Date: Tue, 29 Oct 2019 12:05:57 +0000

I mean the cert that the ORIGINAL client presented to the proxy.

From: Rifaat Shekh-Yusef <[email-masked]>
Date: Tuesday, October 29, 2019 at 7:57 AM
To: Rich Salz <[email-masked]>
Cc: Neil Madden <[email-masked]>, Brian Campbell <bcampbell=[email-masked]>, oauth <[email-masked]>
Subject: Re: [OAUTH-WG] client certs and TLS Terminating Reverse Proxies (was Re: I-D Action: draft-ietf-oauth-jwt-introspection-response-08.txt)

Maybe I misunderstood what you meant by "client-cert". If you meant the proxy client certificate, then that is obviously not enough. You seem to suggest that you meant the remote client certificate to be installed on the proxy to be used with the backend system; if this is the case, then this would work and you would not need the signature, but the issue I see with this approach is that you need to reconfigure the proxy every time you change the client certificate, which is not practical if the certificate is short lived.

Regards,
 Rifaat

On Mon, Oct 28, 2019 at 2:55 PM Salz, Rich <[email-masked]<mailto:[email-masked]>> wrote:

---

From: Justin Richer <[email-masked]>
Date: Tue, 29 Oct 2019 15:13:38 -0400

I would argue that making this standard would actually increase the likelihood of developers getting this right, as now instead of following some copy-pasted recipe for NGINX or Apache that they found on the web, they could turn on a standard setting that would take care of both stripping out incoming headers and injecting the appropriate values. And all of that can be covered in the security considerations with a bunch of normative text on top to make sure inbound headers are stripped. What you’re describing below is clever, but ultimately it’s just a small bit of obscurity more than anything real. 

The way things are today, you’ve got to not only pick a header and figure out its format, but also do the injection protection step yourself. Since all of these are disconnected, there are a lot more places that it could fall over. Even a typo where you throw out incoming “CLIENT_CERT” but inject “CLIENT_CERTS” or something like that would be disastrous. 

All in all, I am in favor of this being defined in one standard way, in addition to secure communication between proxies and backends being standardized — but this latter bit really seems like a separate problem.

 — Justin

> On Oct 28, 2019, at 12:32 PM, Neil Madden <[email-masked]> wrote:
> 
>> On 28 Oct 2019, at 15:32, Brian Campbell <bcampbell=[email-masked] <mailto:bcampbell=[email-masked]>> wrote:
>>

---

From: Brian Campbell <[email-masked]>
Date: Tue, 29 Oct 2019 15:55:27 -0600

+1 to pretty much everything Justin said there.

With some facilitating assistance from Ben it looks like there's now an
agenda slot for this in the upcoming secdispatch meeting in Singapore. I'll
attempt to articulate the situation and see if there's interest in finding
a home for the perspective work. Folks that'll be in Singapore and
interested in the topic would be encouraged to attend to the secdispatch
meeting during the Afternoon Session III 1710-1840 on Tuesday.

On Tue, Oct 29, 2019 at 1:13 PM Justin Richer <[email-masked]> wrote:

> I would argue that making this standard would actually increase the
> likelihood of developers getting this right, as now instead of following
> some copy-pasted recipe for NGINX or Apache that they found on the web,
> they could turn on a standard setting that would take care of both
> stripping out incoming headers and injecting the appropriate values. And
> all of that can be covered in the security considerations with a bunch of
> normative text on top to make sure inbound headers are stripped. What
> you’re describing below is clever, but ultimately it’s just a small bit of
> obscurity more than anything real.
>
> The way things are today, you’ve got to not only pick a header and figure
> out its format, but also do the injection protection step yourself. Since
> all of these are disconnected, there are a lot more places that it could
> fall over. Even a typo where you throw out incoming “CLIENT_CERT” but
> inject “CLIENT_CERTS” or something like that would be disastrous.
>
> All in all, I am in favor of this being defined in one standard way, in
> addition to secure communication between proxies and backends being
> standardized — but this latter bit really seems like a separate problem.
>
>  — Justin
>
> While there are some benefits to standardizing headers for this kind of
> communication, there are some significant downsides - particularly when
> using headers to communicate critical security information like certs. It
> is *very* easy to misconfigure a reverse proxy to not strip Forwarded (or
> whatever) headers from incoming requests, allowing a client to simply
> supply a certificate as a header without authenticating the TLS connection
> with the corresponding private key. One good practice to prevent this is to
> pick a random and unguessable header name (configurable per installation)
> to be used for communicating the certificate, rather than using something
> fixed and standard. That way even if you misconfigure the proxy an attacker
> still has to try and guess the correct header name.
>
> I suppose the same thing could be accomplished by having an extension for
> including a shared secret (or HMAC tag) in the header to authenticate it.
>
> -- Neil
>
> On 28 Oct 2019, at 15:32, Brian Campbell <
> bcampbell=[email-masked]> wrote:
>
> review, use, distribution or disclosure by others is strictly prohibited..
> your computer. Thank you.*_______________________________________________
> OAuth mailing list
> [email-masked]

---

From: Hans Zandbelt <[email-masked]>
Date: Wed, 30 Oct 2019 01:52:29 +0100

+1 to Justin's and Brian's comments, I am interested to contribute and I
will try and be there in person as well

Hans.

---

From: Neil Madden <[email-masked]>
Date: Wed, 30 Oct 2019 07:43:50 +0000

Replies below.

> On 29 Oct 2019, at 19:13, Justin Richer <[email-masked]> wrote:
> 

Not really. If the header is cryptographically unguessable (eg includes a 128-bit base32-encoded string) then it provides a real security improvement. The header name becomes a bearer token effectively. There are many ways that a reverse proxy can be misconfigured in a way that compromises the security of trusted headers:

1. There is the simple misconfiguration where you fail to strip trusted headers from incoming requests. For example in HAProxy this is the difference between set-header and add-header directives (and the add-header docs list passing a client cert to the backend as an example use case [1] despite this being insecure). Naive functional testing will still pass, but fairly basic adversarial tests would catch this. A standard header/config option might help with this simple case. 

2. Scoping issues cause rules to not be applied where you expect them to. For example, nginx has multiple levels of scope that you can define header rules (http, server, location). But if you define *any* proxy_set_header directive at the location level (for example) it will ignore *all* such directives at the server or http levels, which can cause header-stripping rules to be silently disabled. 

3. More advanced attacks exploit differences in how individual reverse proxies and application servers parse headers to smuggle headers or even whole requests past the RP [2]. 

Using unguessable header names for transmitting security-critical information between the RP and the app server provides an effective defense in depth against all of these attacks. (They are all forms of confused deputy attack and the unguessable header acts in the same way as an anti-CSRF token to prevent these systematically). 

(I had thought the random header name pattern was widely known, as I’ve heard it mentioned in conversations several times. But now I come to look for a definitive reference to it I can only find it mentioned in our own docs [3] and connect2id’s [4]). 

[1]: http://cbonte.github.io/haproxy-dconv/2.1/configuration.html#4.2-http-request%20add-header
[2]: https://portswigger.net/web-security/request-smuggling
[3]: https://backstage.forgerock.com/docs/am/6.5/oauth2-guide/#provide-mtls-certs
[4]: https://connect2id.com/products/server/docs/guides/tls-proxy 

> The way things are today, you’ve got to not only pick a header and figure out its format, but also do the injection protection step yourself. Since all of these are disconnected, there are a lot mo…

Most load balancers and reverse proxies support some way of setting headers that also strips the incoming header in one directive. Unless the standard gets rid of all the other ways they support to configure this (and removes all the old Stack Overflow recipes) then it won’t necessarily improve things. 

A standard that provides an explicit field for including a secret (heck, we could even call it an access token :-) that SHOULD be used might provide a significant improvement in security. 

> All in all, I am in favor of this being defined in one standard way, in addition to secure communication between proxies and backends being standardized — but this latter bit really seems like a se…

I agree that that is a separate issue. 

— Neil

>>> On Mon, Oct 28, 2019 at 9:05 AM Rifaat Shekh-Yusef <[email-masked]> wrote:
>>>> Thanks Brian,
>>>> 
>>>>> On Mon, Oct 28, 2019 at 8:42 AM Brian Campbell <[email-masked]> wrote:
>>>>> 
>>>>> 
>>>>>> On Sat, Oct 26, 2019 at 3:55 PM Rifaat Shekh-Yusef <[email-masked]> wrote:
>>>>>> 
>>>>>>> On Fri, Oct 25, 2019 at 3:47 PM Brian Campbell <[email-masked]> wrote:
>>>>>>> 
>>>>> What I meant was only that in the course of writing https://tools.ietf.org/html/draft-ietf-tokbind-ttrp-09, which aims to define HTTP header fields that enable a TLS terminating reverse proxy to convey information to a backend server about the validated Token Binding Message received from a client, it seemed more straightforward and sufficient for the use-case to use new HTTP headers to carry the information rather than to use new fields in the Forwarded header framework from RFC7239.    
>>>>>  
>>>>>

---

From: Justin Richer <[email-masked]>
Date: Wed, 30 Oct 2019 09:24:32 -0400

All of these problems can be solved, and I think solved better, by securing the connection between the proxy and the back-end. That way the back end will be able look not only for a specific header, but verify that the header came from the proxy itself. An obscure header name is one way to do that, but it has a lot of issues with it, especially since it’s likely to be selected once during set-up and never changed throughout the lifetime of the deployment. I think there are likely much better solutions here, and they’d address this issue without things getting weird.

And one of the best things about a standard is that you’re still free to completely ignore it if you want to, so people can and will keep following whatever proprietary patterns they want to. But at least a standard mechanism would give us a way to say to newcomers and veterans alike “No really, here’s a way that we all agree works and has these properties”. 

 — Justin

> On Oct 30, 2019, at 3:43 AM, Neil Madden <[email-masked]> wrote:
> 
> Replies below.
> 
> [1]: http://cbonte.github.io/haproxy-dconv/2.1/configuration.html#4.2-http-request%20add-header <http://cbonte.github.io/haproxy-dconv/2.1/configuration.html#4.2-http-request%20add-header>
> [2]: https://portswigger.net/web-security/request-smuggling <https://portswigger.net/web-security/request-smuggling>
> [3]: https://backstage.forgerock.com/docs/am/6.5/oauth2-guide/#provide-mtls-certs <https://backstage.forgerock.com/docs/am/6.5/oauth2-guide/#provide-mtls-certs>[4]: https://connect2id.com/products/server/docs/guides/tls-proxy <https://connect2id.com/products/server/docs/guides/tls-proxy> 
> 
>> 
>>> On Oct 28, 2019, at 12:32 PM, Neil Madden <[email-masked] <mailto:[email-masked]>> wrote:
>>>

---

From: Torsten Lodderstedt <[email-masked]>
Date: Wed, 30 Oct 2019 14:46:05 +0100

+1

> On 30. Oct 2019, at 14:24, Justin Richer <[email-masked]> wrote:
>

---

From: Neil Madden <[email-masked]>
Date: Wed, 30 Oct 2019 13:56:54 +0000

> All of these problems can be solved, and I think solved better, by securing the connection between the proxy and the back-end. That way the back end will be able look not only for a specific header…

The issue has nothing to do with the security of the connection between the proxy and the backend. It is to do with data origin authentication of the headers themselves. All of the headers arrive at the backend over the same connection, but only some of them were created by the proxy. There are undoubtedly better alternatives - e.g. using a shared secret to compute a HMAC over security sensitive headers and include that as an additional header or field. But an unguessable header name is *simple* and effective and works right now with widely implemented functionality. 

There's no need for the header name to ever change - the secret is not exposed to untrusted parties and is a defense-in-depth rather than a primary defense. I also don't know why you consider this a "weird" solution - it's a simple pragmatic solution to a fairly widespread class of security vulnerabilities. It also has the benefit that it forces the backend to "validate" the secret, because it won't find the header if it gets it wrong, whereas it is much easier to forget to validate a HMAC tag. I'll take simple and weird over missing and broken any day.

> And one of the best things about a standard is that you’re still free to completely ignore it if you want to, so people can and will keep following whatever proprietary patterns they want to. But a…

I'm not arguing against a standard, just against a standard that makes it harder to mitigate known security vulnerabilities.

-- Neil

---

From: Torsten Lodderstedt <[email-masked]>
Date: Wed, 30 Oct 2019 15:05:48 +0100

> On 30. Oct 2019, at 14:56, Neil Madden <[email-masked]> wrote:
> 
> There's no need for the header name to ever change - the secret is not exposed to untrusted parties

If the proxy sends certs via an header X to service A and B and someone impersonates A it will find out the secret and use it to inject certs in a connection towards B. We have learned that it is sometime hard to use different secret header names for the same purpose with different request targets. In those cases, a way to authenticate the proxy might by a good solution.  

> and is a defense-in-depth rather than a primary defense. I also don't know why you consider this a "weird" solution - it's a simple pragmatic solution to a fairly widespread class of security vulnerabilities. It also has the benefit that it forces the backend to "validate" the secret, because it won't find the header if it gets it wrong, whereas it is much easier to forget to validate a HMAC tag. I'll take simple and weird over missing and broken any day.
> 
>>

---

From: "Salz, Rich" <[email-masked]>
Date: Wed, 30 Oct 2019 14:07:33 +0000

> But an unguessable header name is *simple* and effective and works right now with widely implemented functionality. 

You mean like admin/admin for administrator access?  There is no such thing as an unguessable name. You claim the name will never be exposed to untrusted parties.  How so?  You are now telling administrators to treat a *name* as securely as they treat a *key* (or password).  If it must be protected like key material, then use it like key material.

The proxy-backend should be TLS, ideally authenticating the proxy.

---

From: Neil Madden <[email-masked]>
Date: Wed, 30 Oct 2019 14:15:34 +0000

On 30 Oct 2019, at 14:05, Torsten Lodderstedt <[email-masked]> wrote:
> If the proxy sends certs via an header X to service A and B and someone impersonates A it will find out the secret and use it to inject certs in a connection towards B.

Being able to impersonate A suggests that the attacker is already on the trusted side of the network and that you are employing zero additional network security controls between the RP and service A and B (e.g., TLS, firewalls, VLAN/switches, etc). Remember, this is intended to mitigate against accidental misconfiguration of the RP and header spoofing tricks coming from the external network. I'm not suggesting it as an alternative to basic network security on the trusted network.

> We have learned that it is sometime hard to use different secret header names for the same purpose with different request targets. In those cases, a way to authenticate the proxy might by a good solution.  

Do you have an example? The RPs and load balancers I'm familiar that support multiple backends all support sending different headers to each of them. Again though, security against attackers inside the trusted network is not part of what the solution is preventing.

Again, authenticating the *connection* from the RP to the backend services is good, but is completely orthogonal to authenticating the headers themselves.

-- Neil

---

From: "Salz, Rich" <[email-masked]>
Date: Wed, 30 Oct 2019 14:18:25 +0000

*   Again, authenticating the *connection* from the RP to the backend services is good, but is completely orthogonal to authenticating the headers themselves.

I strongly disagree.  Authenticating the sender allows the receiver to make a trust decision in the provenance and quality of the data it gets from the sender.  Do you disagree with that?

---

From: Neil Madden <[email-masked]>
Date: Wed, 30 Oct 2019 14:30:45 +0000

Combining responses to related messages (429 error):

On 30 Oct 2019, at 14:07, Salz, Rich <[email-masked]> wrote:
> You mean like admin/admin for administrator access?  There is no such thing as an unguessable name.

I'm thinking of a uniformly random 16 byte name right now. Have at it.

> You claim the name will never be exposed to untrusted parties.  How so?  You are now telling administrators to treat a *name* as securely as they treat a *key* (or password).  If it must be protected like key material, then use it like key material.

Again, this is a defense in depth measure. A config file is fine.

> The proxy-backend should be TLS, ideally authenticating the proxy.

I agree - but completely irrelevant to the current discussion, which is about how the backend distinguishes security-critical headers that the proxy set from security-critical headers that were sneaked past the proxy by a client (through misconfiguration or parsing bug).

> On 30 Oct 2019, at 14:18, Salz, Rich <[email-masked]> wrote:
> 

Yes, see:

 https://en.wikipedia.org/wiki/Confused_deputy_problem <https://en.wikipedia.org/wiki/Confused_deputy_problem>
https://en.wikipedia.org/wiki/Cross-site_request_forgery <https://en.wikipedia.org/wiki/Cross-site_request_forgery>
https://en.wikipedia.org/wiki/Server-side_request_forgery <https://en.wikipedia.org/wiki/Server-side_request_forgery>

and so on and so on. Authenticating the other side of a communication pipe is not sufficient to authenticate the origin of the data contained within those messages. The whole point of a proxy is that it forwards requests from clients. In the face of misconfigurations and parsing bugs the backend cannot distinguish headers that were set by the proxy from headers that were spoofed by the client. *This is the entire problem I have been discussing*. 

-- Neil

---

From: "Salz, Rich" <[email-masked]>
Date: Wed, 30 Oct 2019 16:41:22 +0000

*   I'm thinking of a uniformly random 16 byte name right now. Have at it.
Cute but missing the point.  I don’t have to guess it.  YOU have to securely deploy it across your proxy (however many staff), your backend (however many staff), your application developers (however many), and perhaps your diagnosing or debug teams if they are different.  And then you must make sure that if ANYONE ever takes a packet trace, or makes a slide out of a sample message, that they don’t disclose the header, such as by showing “here’s how we do OAUTH” at a user group meeting.

  *   Again, this is a defense in depth measure. A config file is fine.
No, it’s not.  It’s a multi-party shared secret that isn’t identified as a secret.  It’s not defense in depth, it’s a foundation of sand.

  *   irrelevant to the current discussion, which is about how the backend distinguishes security-critical headers that the proxy set from security-critical headers that were sneaked past the proxy by a client (through misconfiguration or parsing bug).
The backend must trust the proxy.  If you tell the proxy to “use FOOBAR123” as the header of the client certificate, how do you know that the proxy properly provided the client certificate?  Pretending that the backend only has to trust *part* of the proxy, because we use a sekrit header value is just that, pretending.

Your links to confused deputy and CSRF do not address the issues I raised.

  *   Authenticating the other side of a communication pipe is not sufficient to authenticate the origin of the data contained within those messages. The whole point of a proxy is that it forwards requests from clients. In the face of misconfigurations and parsing bugs the backend cannot distinguish headers that were set by the proxy from headers that were spoofed by the client. *This is the entire problem I have been discussing*.
And you believe that configuring a proxy, of which you are skeptical, addresses that concern?

---

From: Neil Madden <[email-masked]>
Date: Wed, 30 Oct 2019 17:03:49 +0000

On 30 Oct 2019, at 16:41, Salz, Rich <[email-masked]> wrote:
> Cute but missing the point.  I don’t have to guess it. 

To quote your previous claim: "There is no such thing as an unguessable name."

> YOU have to securely deploy it across your proxy (however many staff), your backend (however many staff), your application developers (however many), and perhaps your diagnosing or debug teams if they are different.  And then you must make sure that if ANYONE ever takes a packet trace, or makes a slide out of a sample message, that they don’t disclose the header, such as by showing “here’s how we do OAUTH” at a user group meeting.

Even if your deployment team had such staggeringly bad operational security practices as to allow people to take packet captures from an internal network and show them on public slides without any kind of questions being asked, if this actually happens *YOU ARE NO WORSE OFF THAN IN THE SITUATION WHERE YOU USED A WELL-KNOWN HEADER NAME*!

I don't know how many different ways I can say that this is a defense in depth *in addition to* everything else you would normally do to secure traffic between your RP and backend servers. As with all defense in depth, the aim is to be more than 1 configuration mistake away from total compromise.

-- Neil

---

From: "Salz, Rich" <[email-masked]>
Date: Wed, 30 Oct 2019 17:19:34 +0000

*   To quote your previous claim: "There is no such thing as an unguessable name."
Right.  That doesn’t mean *I* have to guess it.

  *   Even if your deployment team had such staggeringly bad operational security practices as to allow people to take packet captures from an internal network and show them on public slides without any kind of questions being asked, if this actually happens *YOU ARE NO WORSE OFF THAN IN THE SITUATION WHERE YOU USED A WELL-KNOWN HEADER NAME*!
Yes you are worse off.  Because that now-exposed header value can be used for spoofing.  As opposed to protection by TLS, and then sending the plaintext message around.

  *   I don't know how many different ways I can say that this is a defense in depth
Because it is not.  It is taking an application-level piece of configuration data and requiring it to be treated as if it were crypto material. Which it cannot be, because multiple parties need to know it (as I said, the proxy, the backend, the app developers, the support team, etc).  It’s defense by “collapsing layers” rather than “in depth.”

  *   As with all defense in depth, the aim is to be more than 1 configuration mistake away from total compromise.
But that is exactly what you are proposing. Exposing the header *is* a total compromise and multiple entities will need to know that header value.

At any rate,  I think we’re done here.

---

From: Neil Madden <[email-masked]>
Date: Wed, 30 Oct 2019 19:18:27 +0000

If you can point out where I recommended disabling TLS or not bothering to strip headers from incoming requests, or anything else along those lines then please let me know. Otherwise, yes we’re done here. 

> On 30 Oct 2019, at 17:19, Salz, Rich <[email-masked]> wrote:
> 
> 
> I don't know how many different ways I can say that this is a defense in depth
> As with all defense in depth, the aim is to be more than 1 configuration mistake away from total compromise.

---

From: "Salz, Rich" <[email-masked]>
Date: Wed, 30 Oct 2019 19:22:05 +0000

If you strip off the TLS, you *still* cannot pass messages around without protecting them as if they had secrets because, well, the header name you are using is secret.  This is not a defense in depth.

---

From: Jim Manico <[email-masked]>
Date: Wed, 30 Oct 2019 16:04:44 -0400

I love you Neil.

--
Jim Manico
@Manicode

> On Oct 30, 2019, at 3:18 PM, Neil Madden <[email-masked]> wrote:
> 
>

---

From: Vladimir Dzhuvinov <[email-masked]>
Date: Thu, 31 Oct 2019 12:36:49 +0200

> All in all, I am in favor of this being defined in one standard way,
> in addition to secure communication between proxies and backends being

-1 for devising a well known header

+1 for a simple way to authenticate a reverse proxy with a web server

With a well known name, in a attack this will get probed first. The well
known header name doesn't guarantee a correct config. And if we have a
new standard for sec headers that must be stripped automatically from
inbound HTTP requests, we cannot expect that will get implemented in all
reverse proxy software overnight.

Because a correct config is not practical as the only line of defense,
when we implemented mTLS we decided to add a length (>= 32 chars) and
randomness check to the header config. I saw some concerns that this may
cause deployment issues. Nobody has complained about that so far.

https://connect2id.com/products/server/docs/config/core#op-tls-clientX509CertHeader

Regarding mistyping, this can be an issue, but has little to no effect
if a long random header gets misstyped (from the inbound strip
directive). With a well known header this will result in a immediate
security hole, which can theoretically go unnoticed.

Here is an example Apache httpd config, to illustrate my point:

RequestHeader set Sec-Client-X509-Cert-liede5vaePeeMiYie0xu2jaudauleing ""
RequestHeader set Sec-Client-X509-Cert-liede5vaePeeMiYie0xu2jaudauleing "%{SSL_CLIENT_CERT}s"

(the first line strips the inbound headers)

Vladimir

-- 
Vladimir Dzhuvinov

---

From: Hans Zandbelt <[email-masked]>
Date: Thu, 31 Oct 2019 11:48:46 +0100

the way I see this is that stripping and setting the headers must be part
of the implementation of the protocol itself, it should not be something
left to a non-atomic piece of configuration by an administrator; I
implemented it like this in the Apache implementation of Brian's TTRP spec
[1] and have been doing this for the OIDC and OAuth Apache modules that set
headers for backends to consume [2]; and yes, I have made mistakes [3], but
IMHO it should not be a problem to use a well known header and make the
implementer (not the admin) get it right by pointing this out in the spec,
as is done for many other pitfalls

Hans.

[1]
https://github.com/zmartzone/mod_token_binding/blob/master/src/mod_token_binding.c#L481-L483
[2]
https://github.com/zmartzone/mod_auth_openidc/blob/master/src/mod_auth_openidc.c#L164-L175
[3] https://www.cvedetails.com/cve/CVE-2017-6413/

On Thu, Oct 31, 2019 at 11:37 AM Vladimir Dzhuvinov <[email-masked]>
wrote:

> known header name doesn't guarantee a correct config. And if we have a new
> standard for sec headers that must be stripped automatically from inbound
> HTTP requests, we cannot expect that will get implemented in all reverse
> proxy software overnight.
>
> Regarding mistyping, this can be an issue, but has little to no effect if
> a long random header gets misstyped (from the inbound strip directive).
> With a well known header this will result in a immediate security hole,
> which can theoretically go unnoticed.
>

-- 
[email-masked]
ZmartZone IAM - www.zmartzone.eu

---

From: Torsten Lodderstedt <[email-masked]>
Date: Thu, 31 Oct 2019 12:00:50 +0100

> On 30. Oct 2019, at 15:15, Neil Madden <[email-masked]> wrote:
> 

Is there such a thing as a trusted network? I’m not sure what you mean but micro services call each other so network segregation would not help to prevent replay of a leaked security header. But (see below) different secret headers can be used to mitigate the threat I raised (at least limit the impact to an individual service).

> Do you have an example?

I checked back and it seems I had misunderstood. It’s possible to use different secret header names, one per proxy-to-service connection. 

> The RPs and load balancers I'm familiar that support multiple backends all support sending different headers to each of them. Again though, security against attackers inside the trusted network is not part of what the solution is preventing.
>

---

From: Vladimir Dzhuvinov <[email-masked]>
Date: Thu, 31 Oct 2019 13:25:58 +0200

This is a good story.

How about requiring reverse proxies to automatically scrub all inbound
HTTP headers that start with "Sec-"?

If a sec header has the format "Sec-[NAME]-[random-chars]" we get:

* The "Sec-" prefix will  cause new compliant reserve proxies to
automatically scrub the inbound HTTP header.

* The NAME part still makes the header easily identifiable (I think Rich
Salz had this concern).

* The random chars part, potentially optional, will provide a line of
defence against config errors in old legacy reverse proxies.

All concerns should then get covered.

Vladimir

On 31/10/2019 12:48, Hans Zandbelt wrote:
> the way I see this is that stripping and setting the headers must be
> part of the implementation of the protocol itself, it should not be
> something left to a non-atomic piece of configuration by an
> administrator; I implemented it like this in the Apache implementation
> of Brian's TTRP spec [1] and have been doing this for the OIDC and
> OAuth Apache modules that set headers for backends to consume [2]; and
> yes, I have made mistakes [3], but IMHO it should not be a problem to
> use a well known header and make the implementer (not the admin) get
> it right by pointing this out in the spec, as is done for many other
> pitfalls
>
> Hans.
>
> [1] https://github.com/zmartzone/mod_token_binding/blob/master/src/mod_token_binding.c#L481-L483
> [2] https://github.com/zmartzone/mod_auth_openidc/blob/master/src/mod_auth_openidc.c#L164-L175
> On Thu, Oct 31, 2019 at 11:37 AM Vladimir Dzhuvinov
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>>     All in all, I am in favor of this being defined in one standard
>>     way, in addition to secure communication between proxies and
>>     backends being standardized — but this latter bit really seems
>>     like a separate problem.
>>
>>      — Justin
>
>     With a well known name, in a attack this will get probed first.
>     The well known header name doesn't guarantee a correct config. And
>     if we have a new standard for sec headers that must be stripped
>     automatically from inbound HTTP requests, we cannot expect that
>     will get implemented in all reverse proxy software overnight.
>
>     Because a correct config is not practical as the only line of
>     defense, when we implemented mTLS we decided to add a length (>=
>     32 chars) and randomness check to the header config. I saw some
>     concerns that this may cause deployment issues. Nobody has
>     complained about that so far.
>
>     Regarding mistyping, this can be an issue, but has little to no
>     effect if a long random header gets misstyped (from the inbound
>     strip directive). With a well known header this will result in a
>     immediate security hole, which can theoretically go unnoticed.
>
> [email-masked] <mailto:[email-masked]>
> ZmartZone IAM - www.zmartzone.eu <http://www.zmartzone.eu>

-- 
Vladimir Dzhuvinov

---

From: Hans Zandbelt <[email-masked]>
Date: Thu, 31 Oct 2019 12:38:15 +0100

+10

On Thu, Oct 31, 2019 at 12:26 PM Vladimir Dzhuvinov <[email-masked]>
wrote:

> On Thu, Oct 31, 2019 at 11:37 AM Vladimir Dzhuvinov <
> [email-masked]> wrote:
>

---

From: "Salz, Rich" <[email-masked]>
Date: Thu, 31 Oct 2019 13:13:06 +0000

How about requiring reverse proxies to automatically scrub all inbound HTTP headers that start with "Sec-"?
https://tools.ietf.org/html/draft-ietf-httpbis-client-hints-07

Making a header value “secret” will not protect anything.

---

From: Vladimir Dzhuvinov <[email-masked]>
Date: Thu, 31 Oct 2019 18:45:22 +0200

Thanks for this note. According to the abstract the advertising is
intended for "request headers for proactive content negotiation"
(Accept-*), which should exclude all other types of header. I looked at
the Security Considerations and wrote to the author with the suggestion
to note that implementations must be careful not to advertise any other
names, e.g. the names of headers intended to be set by the proxy for use
by a web server.

Vladimir

---

From: "Richard Backman, Annabelle" <[email-masked]>
Date: Thu, 31 Oct 2019 18:55:28 +0000

Relying on a fixed, random header name for security, even as a “defense in depth” measure, is dangerous. In order for this mechanism to be effective, the header name must be random (in the cryptographic sense) and must be kept secret. It needs to be withheld from request logs or error logs, either on the reverse proxy or on the service. It cannot be committed to code repositories. Including it as a signed header in request signing algorithms that require an explicit list of signed headers (such as AWS Signature Version 4<https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html>, draft-cavage-http-signatures<https://tools.ietf.org/html/draft-cavage-http-signatures-12>, draft-ietf-oauth-signed-http-request<https://tools.ietf.org/html/draft-ietf-oauth-signed-http-request>) turns that signature metadata into a secret, meaning that it cannot be logged, etc. If signatures are sent to a central authority for processing, that authority must also know not to log the list of signed headers. I could go on, but I hope this is enough to express that there are SO MANY ways that header names can and will be revealed, and they aren’t always obvious.

What we are talking about is a message authenticity problem. The best practices for providing message authenticity involve applied crypto, e.g., an HMAC or digital signature over the header contents. If an implementation does that, then the random header name is unnecessary. This approach is immune to the sort of misconfiguration scenarios that have been discussed in this thread, as they would result in the reverse proxy failing to provide a properly signed header to the service.

–
Annabelle Richard Backman
AWS Identity

From: OAuth <[email-masked]> on behalf of Vladimir Dzhuvinov <[email-masked]>
Organization: Connect2id Ltd.
Date: Thursday, October 31, 2019 at 4:27 AM
To: "[email-masked]" <[email-masked]>
Subject: Re: [OAUTH-WG] client certs and TLS Terminating Reverse Proxies (was Re: I-D Action: draft-ietf-oauth-jwt-introspection-response-08.txt)

This is a good story.

* The "Sec-" prefix will  cause new compliant reserve proxies to automatically scrub the inbound HTTP header.

* The NAME part still makes the header easily identifiable (I think Rich Salz had this concern).

* The random chars part, potentially optional, will provide a line of defence against config errors in old legacy reverse proxies.

Vladimir
On 31/10/2019 12:48, Hans Zandbelt wrote:
the way I see this is that stripping and setting the headers must be part of the implementation of the protocol itself, it should not be something left to a non-atomic piece of configuration by an administrator; I implemented it like this in the Apache implementation of Brian's TTRP spec [1] and have been doing this for the OIDC and OAuth Apache modules that set headers for backends to consume [2]; and yes, I have made mistakes [3], but IMHO it should not be a problem to use a well known header and make the implementer (not the admin) get it right by pointing this out in the spec, as is done for many other pitfalls

Hans.

On Thu, Oct 31, 2019 at 11:37 AM Vladimir Dzhuvinov <[email-masked]<mailto:[email-masked]>> wrote:
All in all, I am in favor of this being defined in one standard way, in addition to secure communication between proxies and backends being standardized — but this latter bit really seems like a separate problem.

 — Justin

With a well known name, in a attack this will get probed first. The well known header name doesn't guarantee a correct config. And if we have a new standard for sec headers that must be stripped automatically from inbound HTTP requests, we cannot expect that will get implemented in all reverse proxy software overnight.

Because a correct config is not practical as the only line of defense, when we implemented mTLS we decided to add a length (>= 32 chars) and randomness check to the header config. I saw some concerns that this may cause deployment issues. Nobody has complained about that so far.

Regarding mistyping, this can be an issue, but has little to no effect if a long random header gets misstyped (from the inbound strip directive). With a well known header this will result in a immediate security hole, which can theoretically go unnoticed.

Vladimir

--

Vladimir Dzhuvinov

--
[email-masked]<mailto:[email-masked]>
ZmartZone IAM - www.zmartzone.eu<http://www.zmartzone.eu>

--

Vladimir Dzhuvinov

---

From: "Salz, Rich" <[email-masked]>
Date: Thu, 31 Oct 2019 19:00:33 +0000

*   Relying on a fixed, random header name for security, even as a “defense in depth” measure, is dangerous.

Thank you.  +1

---

From: Neil Madden <[email-masked]>
Date: Thu, 31 Oct 2019 20:16:39 +0000

On 31 Oct 2019, at 18:55, Richard Backman, Annabelle <richanna=[email-masked]> wrote:
> 
> 
> Relying on a fixed, random header name for security, even as a “defense in depth” measure, is dangerous. In order for this mechanism to be effective, the header name must be random (in the cryptographic sense) and must be kept secret. It needs to be withheld from request logs or error logs, either on the reverse proxy or on the service. It cannot be committed to code repositories.

Just like any other bearer token. I mean this is the *OAuth* WG, right? Where we regularly recommend people send bearer tokens in headers? I don’t really understand how that can be considered secure to send over the internet to a cloud service, but suddenly becomes insecure when done inside the firewall within a datacenter between a reverse proxy and an app server. 

I mean, are people honestly suggesting that randomizing header names introduces *new* vulnerabilities that aren’t present when you use a well-known name? “Dangerous” even! 

> Including it as a signed header in request signing algorithms that require an explicit list of signed headers (such as AWS Signature Version 4, draft-cavage-http-signatures, draft-ietf-oauth-signed-http-request) turns that signature metadata into a secret, meaning that it cannot be logged, etc. If signatures are sent to a central authority for processing, that authority must also know not to log the list of signed headers. I could go on, but I hope this is enough to express that there are SO MANY ways that header names can and will be revealed, and they aren’t always obvious.
>  

I agree that HMAC signatures are generally better, but which reverse proxies support signing headers?

Actually HMAC signatures do have one significant downside compared to using a random header name - if you forget to validate the signature *nothing fails*. If you mess up the name of a random header then your app doesn’t see any credentials and fails noisily. 

— Neil

---

From: Vladimir Dzhuvinov <[email-masked]>
Date: Thu, 31 Oct 2019 22:48:31 +0200

I agree wholeheartedly that an HMAC or even signature is preferable.
With HMAC vs signature having their relative pros / cons, in terms of
computation and key distribution. If a standard for that is created, and
I am a web application developer, I'll be able to handle the header
checks in the application layer code relatively easily. But I have much
less control over how / where the application is going to be deployed,
and if the reverse proxy has this header security implemented. Software
and cloud vendors can sometimes be significantly behind in such
developments. Even basic handling of self-signed client certificates has
been an issue for us in popular cloud environments.

With that in mind, the "secret" headers can be configured with most
currently available reverse proxy software. I see it as a useful
intermediate step, until a standard for authenticated headers gets
agreed on and eventually implemented in reverse proxies. Can you comment
on that from this perspective?

Over the years I have come to the conclusion that standards which can be
implemented in the application layer or as a simple config stand a
better chance. Standards which rely on other layers or on broad vendor
or developer support to become useful run the risk of not getting
adopted in practice. Token binding suffered from that.

Vladimir

On 31/10/2019 20:55, Richard Backman, Annabelle wrote:
>
> Relying on a fixed, random header name for security, even as a
> “defense in depth” measure, is dangerous. In order for this mechanism
> to be effective, the header name must be random (in the cryptographic
> sense) and must be kept secret. It needs to be withheld from request
> logs or error logs, either on the reverse proxy or on the service. It
> cannot be committed to code repositories. Including it as a signed
> header in request signing algorithms that require an explicit list of
> signed headers (such as AWS Signature Version 4
> <https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html>,
> draft-cavage-http-signatures
> <https://tools.ietf.org/html/draft-cavage-http-signatures-12>,
> draft-ietf-oauth-signed-http-request
> <https://tools.ietf.org/html/draft-ietf-oauth-signed-http-request>)
> turns that signature metadata into a secret, meaning that /it/ cannot
> be logged, etc. If signatures are sent to a central authority for
> processing, that authority must also know not to log the list of
> signed headers. I could go on, but I hope this is enough to express
> that there are SO MANY ways that header names can and will be
> revealed, and they aren’t always obvious.
>
>  
>
> What we are talking about is a message authenticity problem. The best
> practices for providing message authenticity involve applied crypto,
> e.g., an HMAC or digital signature over the header contents. If an
> implementation does that, then the random header name is unnecessary.
> This approach is immune to the sort of misconfiguration scenarios that
> have been discussed in this thread, as they would result in the
> reverse proxy failing to provide a properly signed header to the service.
>
>  
>
> – 
>
> *From: *OAuth <[email-masked]> on behalf of Vladimir Dzhuvinov
> <[email-masked]>
> *Organization: *Connect2id Ltd.
> *Date: *Thursday, October 31, 2019 at 4:27 AM
> *To: *"[email-masked]" <[email-masked]>
> *Subject: *Re: [OAUTH-WG] client certs and TLS Terminating Reverse
> Proxies (was Re: I-D Action:
> draft-ietf-oauth-jwt-introspection-response-08.txt)
>
>  
>
> This is a good story.
>
>     the way I see this is that stripping and setting the headers must
>     be part of the implementation of the protocol itself, it should
>     not be something left to a non-atomic piece of configuration by an
>     administrator; I implemented it like this in the Apache
>     implementation of Brian's TTRP spec [1] and have been doing this
>     for the OIDC and OAuth Apache modules that set headers for
>     backends to consume [2]; and yes, I have made mistakes [3], but
>     IMHO it should not be a problem to use a well known header and
>     make the implementer (not the admin) get it right by pointing this
>     out in the spec, as is done for many other pitfalls
>
>      
>
>     Hans.
>
>      
>
>             All in all, I am in favor of this being defined in one
>             standard way, in addition to secure communication between
>             proxies and backends being standardized — but this latter
>             bit really seems like a separate problem.
>
>              
>
>              — Justin
>
>         With a well known name, in a attack this will get probed
>         first. The well known header name doesn't guarantee a correct
>         config. And if we have a new standard for sec headers that
>         must be stripped automatically from inbound HTTP requests, we
>         cannot expect that will get implemented in all reverse proxy
>         software overnight.
>
>         defense, when we implemented mTLS we decided to add a length
>         (>= 32 chars) and randomness check to the header config. I saw
>         some concerns that this may cause deployment issues. Nobody
>         has complained about that so far.
>
>         Regarding mistyping, this can be an issue, but has little to
>         no effect if a long random header gets misstyped (from the
>         inbound strip directive). With a well known header this will
>         result in a immediate security hole, which can theoretically
>         go unnoticed.
>
>         RequestHeader set
>         Sec-Client-X509-Cert-liede5vaePeeMiYie0xu2jaudauleing ""
>
>         RequestHeader set
>         Sec-Client-X509-Cert-liede5vaePeeMiYie0xu2jaudauleing
>         "%{SSL_CLIENT_CERT}s"
>

---

From: "Richard Backman, Annabelle" <[email-masked]>
Date: Thu, 31 Oct 2019 21:08:52 +0000

The comparison the bearer tokens is illustrative of the problems I and others are pointing out:

  *   Bearer tokens are embedded in the value of the header, not the header itself, which partially alleviates the concern I raised regarding request signing algorithms.
  *   Bearer tokens are typically relatively short-lived, providing some mitigation against exfiltration through logs.
  *   Bearer tokens are typically dynamically generated, and are therefore less likely to be embedded in source code or config files in a project’s repository.
  *   Bearer tokens are called tokens, and are presented as secrets and are always expected to be treated as secrets.

The random header name is effectively an infinite-lifetime, statically defined bearer token presented in a way that does not at all make clear and obvious that it is a secret that must be protected, and in fact makes it more likely that it will be revealed, rendering it useless. And like all bearer tokens, even under ideal conditions it by definition CANNOT BE USED TO AUTHENTICATE THE SENDER.

There is a certain amount of irony in the idea of the security of a Mutual TLS deployment ultimately coming down to a bearer-token header name passed between the reverse proxy and the protected service. 😂

> if you forget to validate the signature *nothing fails*

Replace the HMAC with encryption and you solve that problem, as it forces the service to decrypt the value using the correct key in order to access the client certificate data. Whether or not that’s worth doing is something we can debate in the context of an actual proposal.

–

From: Neil Madden <[email-masked]>
Date: Thursday, October 31, 2019 at 1:17 PM
To: "Richard Backman, Annabelle" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Subject: [UNVERIFIED SENDER] Re: [OAUTH-WG] client certs and TLS Terminating Reverse Proxies (was Re: I-D Action: draft-ietf-oauth-jwt-introspection-response-08.txt)

Just like any other bearer token.. I mean this is the *OAuth* WG, right? Where we regularly recommend people send bearer tokens in headers? I don’t really understand how that can be considered secure to send over the internet to a cloud service, but suddenly becomes insecure when done inside the firewall within a datacenter between a reverse proxy and an app server.

Including it as a signed header in request signing algorithms that require an explicit list of signed headers (such as AWS Signature Version 4<https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html>, draft-cavage-http-signatures<https://tools.ietf.org/html/draft-cavage-http-signatures-12>, draft-ietf-oauth-signed-http-request<https://tools.ietf.org/html/draft-ietf-oauth-signed-http-request>) turns that signature metadata into a secret, meaning that it cannot be logged, etc. If signatures are sent to a central authority for processing, that authority must also know not to log the list of signed headers. I could go on, but I hope this is enough to express that there are SO MANY ways that header names can and will be revealed, and they aren’t always obvious.

— Neil

---

From: "Richard Backman, Annabelle" <[email-masked]>
Date: Thu, 31 Oct 2019 21:34:20 +0000

I think the HMAC/signature approach is actually easier for cloud vendors to implement. The threat model is much simpler to understand, and risk mitigation mostly boils down to well-understood cryptographic hygiene patterns. And again, it’s obvious what values are secrets and what values aren’t.

–

Date: Thursday, October 31, 2019 at 1:50 PM

I agree wholeheartedly that an HMAC or even signature is preferable. With HMAC vs signature having their relative pros / cons, in terms of computation and key distribution. If a standard for that is created, and I am a web application developer, I'll be able to handle the header checks in the application layer code relatively easily. But I have much less control over how / where the application is going to be deployed, and if the reverse proxy has this header security implemented. Software and cloud vendors can sometimes be significantly behind in such developments. Even basic handling of self-signed client certificates has been an issue for us in popular cloud environments.

With that in mind, the "secret" headers can be configured with most currently available reverse proxy software. I see it as a useful intermediate step, until a standard for authenticated headers gets agreed on and eventually implemented in reverse proxies. Can you comment on that from this perspective?

Over the years I have come to the conclusion that standards which can be implemented in the application layer or as a simple config stand a better chance. Standards which rely on other layers or on broad vendor or developer support to become useful run the risk of not getting adopted in practice. Token binding suffered from that.

Vladimir

–

From: OAuth <[email-masked]><mailto:[email-masked]> on behalf of Vladimir Dzhuvinov <[email-masked]><mailto:[email-masked]>
Organization: Connect2id Ltd.
Date: Thursday, October 31, 2019 at 4:27 AM
To: "[email-masked]"<mailto:[email-masked]> <[email-masked]><mailto:[email-masked]>
Subject: Re: [OAUTH-WG] client certs and TLS Terminating Reverse Proxies (was Re: I-D Action: draft-ietf-oauth-jwt-introspection-response-08.txt)

This is a good story.

Vladimir

Hans.

 — Justin

Vladimir

--

Vladimir Dzhuvinov

--

Vladimir Dzhuvinov

OAuth mailing list

---

From: Hans Zandbelt <[email-masked]>
Date: Thu, 31 Oct 2019 22:49:49 +0100

for me this thread is running a bit wild: if we don't believe that
implementers get incoming header removal right, why do we believe the same
implementers get randomizing and/or HMAC signing right
setting headers on a trusted leg between proxies and origin servers has
been a best practice for decades, regardless of implementation bugs (which
similarly exist in crypto implementations)
catering for non-trusted legs is fine, but don't let it stand in the way of
the obvious

Hans.

On Thu, Oct 31, 2019 at 10:35 PM Richard Backman, Annabelle <richanna=
[email-masked]> wrote:

> I think the HMAC/signature approach is actually easier for cloud vendors
> to implement. The threat model is much simpler to understand, and risk
> mitigation mostly boils down to well-understood cryptographic hygiene
> patterns. And again, it’s obvious what values are secrets and what values
> aren’t.
>
>
>
> –
>
> *From: *OAuth <[email-masked]> on behalf of Vladimir Dzhuvinov <
> [email-masked]>
> *Date: *Thursday, October 31, 2019 at 1:50 PM
> I agree wholeheartedly that an HMAC or even signature is preferable. With
> HMAC vs signature having their relative pros / cons, in terms of
> computation and key distribution. If a standard for that is created, and I
> am a web application developer, I'll be able to handle the header checks in
> the application layer code relatively easily. But I have much less control
> over how / where the application is going to be deployed, and if the
> reverse proxy has this header security implemented. Software and cloud
> vendors can sometimes be significantly behind in such developments. Even
> basic handling of self-signed client certificates has been an issue for us
> in popular cloud environments.
>
> intermediate step, until a standard for authenticated headers gets agreed
> on and eventually implemented in reverse proxies. Can you comment on that
> from this perspective?
>
> implemented in the application layer or as a simple config stand a better
> chance. Standards which rely on other layers or on broad vendor or
> developer support to become useful run the risk of not getting adopted in
> practice. Token binding suffered from that.
>
> Vladimir
>
> Relying on a fixed, random header name for security, even as a “defense in
> depth” measure, is dangerous. In order for this mechanism to be effective,
> the header name must be random (in the cryptographic sense) and must be
> kept secret. It needs to be withheld from request logs or error logs,
> either on the reverse proxy or on the service. It cannot be committed to
> code repositories. Including it as a signed header in request signing
> algorithms that require an explicit list of signed headers (such as AWS
> Signature Version 4
> <https://tools.ietf.org/html/draft-ietf-oauth-signed-http-request>) turns
> that signature metadata into a secret, meaning that *it* cannot be
> logged, etc. If signatures are sent to a central authority for processing,
> that authority must also know not to log the list of signed headers. I
> could go on, but I hope this is enough to express that there are SO MANY
> ways that header names can and will be revealed, and they aren’t always
> obvious.
>
>
>
> practices for providing message authenticity involve applied crypto, e.g.,
> an HMAC or digital signature over the header contents. If an implementation
> does that, then the random header name is unnecessary. This approach is
> immune to the sort of misconfiguration scenarios that have been discussed
> in this thread, as they would result in the reverse proxy failing to
> provide a properly signed header to the service.
>
>
>
> –
>
> *From: *OAuth <[email-masked]> <[email-masked]> on behalf
> of Vladimir Dzhuvinov <[email-masked]> <[email-masked]>
> *To: *"[email-masked]" <[email-masked]> <[email-masked]> <[email-masked]>