Subject: [OAUTH-WG] Access to raw HTTP request body

From: Eran Hammer-Lahav <[email-masked]>
Date: Thu, 3 Dec 2009 01:01:32 -0700

While access to the raw HTTP request body is often hard to obtain (due to use of streams, protocol layers, etc.) for the purpose of generating a body hash, is it reasonable to expect the client and server to have access to the raw body when sending a form-encoded body request?

In other words, in the case of form-encoded bodies, is it reasonable to expect the client and server to be able to simply hash the body instead of normalizing its key/value pairs content?

We already established it is reasonable to expect the client and server to have access to the raw HTTP request URI. If we can ask clients and servers to include form-encoded bodies as-transmitted in the signature base string, we will remove the need to decode, sort, encode, and concatenate the list of parameters. Instead, each component can be given a fixed spot in the base string. This and changing the separator characters to new line should make producing the base string significantly easier (which has been a big concern).

Comments? (please don't dismiss access to the raw body without thinking closely about the specific circumstances of a form-encoded body).

EHL

---

From: Brian Eaton <[email-masked]>
Date: Fri, 4 Dec 2009 13:51:38 -0800

On Thu, Dec 3, 2009 at 12:01 AM, Eran Hammer-Lahav <[email-masked]> wrote:
> While access to the raw HTTP request body is often hard to obtain (due to use of streams, protocol layers, etc.) for the purpose of generating a body hash,
> is it reasonable to expect the client and server to have access to the raw body when sending a form-encoded body request?

No.  There are common platforms where getting access to the raw HTTP
request body requires major surgery.

For example: HTTP proxies typically stream bodies, asking them to hold
it all in memory doesn't scale.

Another example: application frameworks typically handle things like
request normalization and validation early on in request processing.
By the time you get to the point of doing authentication and
authorization on requests, the raw bytes of the body are gone.

The OAuth body signing extension was arranged in such a way that
server-side validation of the request body was optional, you can
validate the rest of the request even if you don't have access to the
body.  That's pretty important.

Cheers,
Brian

---

From: Eran Hammer-Lahav <[email-masked]>
Date: Fri, 4 Dec 2009 15:01:13 -0700

You might be missing my question. I am asking specifically about form-encoded bodies.

If it is not clear what I am getting at: I would like to have a single way to sign the request body based on the body hash proposal. I want to apply this to form-encoded bodies as well and drop the need to parse and normalize form-encoded bodies.

So my question is, while I know it is often hard to get access to the raw body, in cases when it is *form-encoded* is it still hard or is the client/server able to access it given it is a simple (single) string. Even if you can't get access to the *actual* body, you should be able to figure it out from other data. Is this a reasonable expectation?

Why is this important? Because if we can do this, we can drop all the parameter decoding/sorting/encoding/etc. dance and just take the raw HTTP request URI, the request method, host, and port, and a hash of the body. Put them together CRLF separated and sign it.

So please tell me we can do that... :-)

EHL

> -----Original Message-----
> From: Brian Eaton [mailto:[email-masked]]
> Sent: Friday, December 04, 2009 1:52 PM
> To: Eran Hammer-Lahav
> Cc: OAuth WG ([email-masked])
> Subject: Re: [OAUTH-WG] Access to raw HTTP request body
> 
> > While access to the raw HTTP request body is often hard to obtain (due
> > to use of streams, protocol layers, etc.) for the purpose of generating a
> body hash, is it reasonable to expect the client and server to have access to
> the raw body when sending a form-encoded body request?
> 
> For example: HTTP proxies typically stream bodies, asking them to hold it all
> in memory doesn't scale.
> 
> Another example: application frameworks typically handle things like request
> normalization and validation early on in request processing.
> By the time you get to the point of doing authentication and authorization on
> requests, the raw bytes of the body are gone.
> 
> The OAuth body signing extension was arranged in such a way that server-
> side validation of the request body was optional, you can validate the rest of
> the request even if you don't have access to the body.  That's pretty
> important.
> 
> Cheers,
> Brian

---

From: Brian Eaton <[email-masked]>
Date: Fri, 4 Dec 2009 14:07:15 -0800

> You might be missing my question. I am asking specifically about form-encoded bodies.

Yep, I follow.

There are some differences between getting access to form-encoded vs
non-form-encoded request bodies, but I think in practice it's
difficult no matter what.

There are a significant number of organizations that don't offer any
OAuth protected form-encoded APIs, because their authentication code
doesn't have access to any portion of the request body.

> If it is not clear what I am getting at: I would like to have a single way to sign
> the request body based on the body hash proposal. I want to apply this to
> form-encoded bodies as well and drop the need to parse and normalize
> form-encoded bodies.

I love this idea, but we should understand that most organizations
won't end up validating the body authentication.  I suspect that's
true for both form-encoded and non-form-encoded requests.  The issue
isn't really the encoding.  It's where authentication plugs in to
request processing.

Cheers,
Brian

---

From: Eran Hammer-Lahav <[email-masked]>
Date: Fri, 4 Dec 2009 15:38:18 -0700

You seem to validate my approach. If it is equally hard to peek into the body for form-encoded or otherwise, and the approach implementers are taking is to simply not support any form-encoded parameters in their API, we might as well remove this special case.

Are there cases where form-encoded bodies change after set by the client (in a lower layer)? Or where the server might have access to the processed form-encoded parameters but not the original encoded body string?

EHL

> Sent: Friday, December 04, 2009 2:07 PM
> To: Eran Hammer-Lahav
> non-form-encoded request bodies, but I think in practice it's difficult no
> matter what.
> 
> There are a significant number of organizations that don't offer any OAuth
> protected form-encoded APIs, because their authentication code doesn't
> have access to any portion of the request body.
> 
> > If it is not clear what I am getting at: I would like to have a single
> > way to sign the request body based on the body hash proposal. I want
> > to apply this to form-encoded bodies as well and drop the need to
> > parse and normalize form-encoded bodies.
> 
> I love this idea, but we should understand that most organizations won't end
> up validating the body authentication.  I suspect that's true for both form-
> encoded and non-form-encoded requests.  The issue isn't really the
> encoding.  It's where authentication plugs in to request processing.
> 
> Cheers,
> Brian

---

From: Brian Eaton <[email-masked]>
Date: Fri, 4 Dec 2009 14:48:32 -0800

On Fri, Dec 4, 2009 at 2:38 PM, Eran Hammer-Lahav <[email-masked]> wrote:
> Are there cases where form-encoded bodies change after set by the client (in a lower layer)?

I think it depends on the client.  Most of the client code I've looked
at assumes that authentication happens with (surprise surprise) bearer
tokens.  It's a real pain for client authors to move that code around.

> Or where the server might have access to the processed form-encoded parameters but not the original encoded body string?

This definitely happens, it's how the vanilla java servlet API works.
I think the same is true of django.

Cheers,
Brian

---

From: Eran Hammer-Lahav <[email-masked]>
Date: Fri, 4 Dec 2009 15:55:56 -0700

Let me try this another way (this is not aimed at Brian specifically):

Is my proposal of signing the form-encoded body as an opaque string instead of parsing its content going to exclude *existing* use case supported by OAuth 1.0? I am not asking for theories or guesses but actual deployment experience to show this is not a good idea.

EHL

> Sent: Friday, December 04, 2009 2:49 PM
> To: Eran Hammer-Lahav
> I think it depends on the client.  Most of the client code I've looked at
> assumes that authentication happens with (surprise surprise) bearer tokens.
> It's a real pain for client authors to move that code around.
> 
> > Or where the server might have access to the processed form-encoded
> parameters but not the original encoded body string?
>

---

From: Eran Hammer-Lahav <[email-masked]>
Date: Fri, 4 Dec 2009 22:29:32 -0700

I am proposing to make anything to do with the body optional and at the discretion of the server whether they want to require or validate it. And either way, stop treating form-encoded bodies differently than any other.

EHL

From: Greg Brail [mailto:[email-masked]]
Sent: Friday, December 04, 2009 6:07 PM
To: Brian Eaton; Eran Hammer-Lahav
Cc: OAuth WG ([email-masked])
Subject: RE: [OAUTH-WG] Access to raw HTTP request body

In Java Servlet 2.5, if I'm reading section SRV3.1.1 of the spec correctly the servlet has the opportunity to get the individual parameters (and thus direct the servlet engine to parse the parameters automatically), or to read the raw request body. There is no way to get both.

So for servlets, at least, it seems to me that the answer is that in any practical use case, access to the original POST body can't be counted upon.

On a related note, I'll reiterate that accessing the raw request body can add a layer of inefficiency to an HTTP proxy, such as our own, especially when the body is large.

Is there a possibility that we can require signing the POST body when the OAuth protocol is used to ask for request and access tokens, but NOT require such signing when authenticated requests are made later on? That way an implementer could choose to take the performance hit for the two methods in the OAuth protocol flow (at least in 1.0) that absolutely require that the POST body be signed, but not for everything else.

I suppose that is the point of the body-signing extension, so forgive me if I haven't read that one yet.

                                                            greg

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Brian Eaton
Sent: Friday, December 04, 2009 5:49 PM
To: Eran Hammer-Lahav
Cc: OAuth WG ([email-masked])
Subject: Re: [OAUTH-WG] Access to raw HTTP request body

Cheers,

Brian

_______________________________________________

OAuth mailing list

[email-masked]<mailto:[email-masked]>

https://www.ietf.org/mailman/listinfo/oauth

---

From: Richard Barnes <[email-masked]>
Date: Mon, 7 Dec 2009 15:31:09 -0500

Eran,

It seems like a bad idea to treat query parameters in form-encoded  
bodies differently from query parameters in the request URI.  To the  
developer, there's not a whole lot of difference between submitting a  
form via GET or POST; if anything, there seems to be a preference for  
POST because it doesn't expose the parameters in the request URI.  Not  
protecting query parameters carried in a form-encoded body would mean  
that parameters in GET submissions would be authenticated but those in  
POSTs wouldn't be.  That's a pretty big difference for two things that  
are currently considered to be pretty similar.

On the other hand, my impression from reading OAuth 1.0 is that it  
doesn't really require access to the raw request body, just to the  
names and values of the parameters.  The only issue in that case is  
that the signer and verifier know whether the values they're handling  
are raw or decoded, to avoid double-encoding.  And the only case where  
that gets you into trouble is when the HTTP platform you're using  
doesn't always give you one or the other.  Are there cases where  
that's true?  Are there implementations that switch?  All the ones  
I've developed with have done one or the other (mostly decoded).    
(For that matter, they're in the same form as the GET parameters, so  
there's no difference in how they're handled.)

--Richard

On Dec 5, 2009, at 12:29 AM, Eran Hammer-Lahav wrote:

> I am proposing to make anything to do with the body optional and at  
> the discretion of the server whether they want to require or  
> validate it. And either way, stop treating form-encoded bodies  
> differently than any other.
>
> EHL
>
> In Java Servlet 2.5, if I’m reading section SRV3.1.1 of the spec  
> correctly the servlet has the opportunity to get the individual  
> parameters (and thus direct the servlet engine to parse the  
> parameters automatically), or to read the raw request body. There is  
> no way to get both.
>
> So for servlets, at least, it seems to me that the answer is that in  
> any practical use case, access to the original POST body can’t be  
> counted upon.
>
> On a related note, I’ll reiterate that accessing the raw request  
> body can add a layer of inefficiency to an HTTP proxy, such as our  
> own, especially when the body is large.
>
> Is there a possibility that we can require signing the POST body  
> when the OAuth protocol is used to ask for request and access  
> tokens, but NOT require such signing when authenticated requests are  
> made later on? That way an implementer could choose to take the  
> performance hit for the two methods in the OAuth protocol flow (at  
> least in 1.0) that absolutely require that the POST body be signed,  
> but not for everything else.
>
> I suppose that is the point of the body-signing extension, so  
> forgive me if I haven’t read that one yet.
>
>                                                             greg
>
> > Or where the server might have access to the processed form- 
> encoded parameters but not the original encoded body string?
>
> [email-masked]
> [email-masked]

---

From: Eran Hammer-Lahav <[email-masked]>
Date: Mon, 7 Dec 2009 13:50:24 -0700

My impressions are somewhat different. Twitter API uses POST with query parameters and this seems to be the growing trend. POST with form-encoded parameters is mostly common in browsers submitting forms. After all, form-encoded parameters are part of the HTML specification whether in the query or body.

I think OAuth 1.0 made a mistake getting involved in the application layer semantics when authenticating a request. It belongs at the HTTP layer which doesn't understand anything about query structure or content type.

Parameter encoding has been the most problematic part of 1.0, and so far this use case has not been defended on this list. These issues are now driving companies like Y!, MS, and Google to dump crypto altogether and use secret-less bearer tokens. Without preventing bearer tokens, I would like to make they use less appealing. Offering a coverage method that helps with that is important.

However, there is an easy solution. We can include another coverage method that includes form-encoded bodies like 'base+form-encoded'. It will use a similar normalization scheme as 1.0 but simpler, building on top of the 'base' method.

I am happy to give it a try if others express support for it.

EHL

From: Richard Barnes [mailto:[email-masked]]
Sent: Monday, December 07, 2009 12:31 PM
To: Eran Hammer-Lahav
Cc: Greg Brail; Brian Eaton; OAuth WG ([email-masked])
Subject: Re: [OAUTH-WG] Access to raw HTTP request body

Eran,

It seems like a bad idea to treat query parameters in form-encoded bodies differently from query parameters in the request URI.  To the developer, there's not a whole lot of difference between submitting a form via GET or POST; if anything, there seems to be a preference for POST because it doesn't expose the parameters in the request URI.  Not protecting query parameters carried in a form-encoded body would mean that parameters in GET submissions would be authenticated but those in POSTs wouldn't be.  That's a pretty big difference for two things that are currently considered to be pretty similar.

On the other hand, my impression from reading OAuth 1.0 is that it doesn't really require access to the raw request body, just to the names and values of the parameters.  The only issue in that case is that the signer and verifier know whether the values they're handling are raw or decoded, to avoid double-encoding.  And the only case where that gets you into trouble is when the HTTP platform you're using doesn't always give you one or the other.  Are there cases where that's true?  Are there implementations that switch?  All the ones I've developed with have done one or the other (mostly decoded).   (For that matter, they're in the same form as the GET parameters, so there's no difference in how they're handled.)

--Richard

EHL

Cc: OAuth WG ([email-masked]<mailto:[email-masked]>)

                                                            greg

-----Original Message-----
From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Brian Eaton
Sent: Friday, December 04, 2009 5:49 PM
To: Eran Hammer-Lahav
Cc: OAuth WG ([email-masked]<mailto:[email-masked]>)
Subject: Re: [OAUTH-WG] Access to raw HTTP request body

Cheers,
Brian