Subject: [OAUTH-WG] Proposed resolution for issue 26

From: Mike Jones <[email-masked]>
Date: Fri, 23 Sep 2011 14:00:08 +0000

Issue #26 http://trac.tools.ietf.org/wg/oauth/trac/ticket/26 asks whether the semantics of scope strings should be changed to require that the % character be interpreted as introducing a percent-encoded character that follows.  My proposed resolution is that %-encoding not be required in the specification; therefore no textual change would be made to the specification in response to this issue.  The reasoning behind this resolution is as follows:

1.  Interpretation of scope strings already requires semantic agreement on the meaning of the scope strings between the parties participating the OAuth flow.  Should an encoding be used for scope strings in a particularly deployment context, it is reasonable for participants to have agreed upon that encoding, just as they agree on other OAuth configuration parameters.

2.  More than one encoding methodology could reasonably be employed in scope strings.  For instance, base64url encoding of scope values could be used in some contexts.  Quoting characters with '\' is another possibility.  I see no compelling reason to mandate %-encoding over other potential encoding methods.

3.  Mandating %-encoding unnecessarily complicates implementations without providing a clear compensating benefit sufficient warrant the additional complexity.  For example, it seems unnecessary to mandate that the scope strings "email" and "%65mail" MUST compare as being equal in all implementations.

4.  If an encoding methodology for scope strings is mandated, this should be done in the OAuth Core specification - not the OAuth Bearer Token specification.

5.  I am aware of no existing practice that utilizes %-encoding of scope values.

                                                                -- Mike

---

From: Barry Leiba <[email-masked]>
Date: Sat, 24 Sep 2011 09:33:06 -0400

> My proposed resolution is that %-encoding not be required in the
> specification

I agree with your analysis, now that I see it laid out clearly.  I
would feel better, though, if there were text in the document that
explained that to others, who read it later.  Perhaps, using your
words, we could make this change to section 2.4:

OLD
   The "scope" attribute is a space-delimited list of scope values
   indicating the required scope of the access token for accessing the
   requested resource.  The "scope" attribute MUST NOT appear more than
   once.

NEW
   The "scope" attribute is a space-delimited list of scope values
   indicating the required scope of the access token for accessing the
   requested resource.  The "scope" attribute MUST NOT appear more than
   once.

   Interpretation of scope strings requires semantic agreement on the
   meaning of the scope strings between the parties participating the
   OAuth flow.  Should an encoding be used for scope strings in a
   particular deployment context, participants have to have agreed
   upon that encoding, just as they agree on other OAuth configuration
   parameters.

Does that work?

Barry

---

From: "Manger, James H" <[email-masked]>
Date: Sun, 25 Sep 2011 00:40:44 +1000

> From: Mike Jones

I disagree. This does not resolve the ambiguity. I think there is some confusion between 1) the internal structure (eg encoding) of a scope string, and 2) escaping done when transporting the scope string in a protocol element.

> 1.  Interpretation of scope strings already requires semantic agreement on the meaning of the scope strings between the parties participating the OAuth flow.  Should an encoding be used for scope s…

A client app does NOT always need semantic agreement on the meaning of scope strings.
Consider a client app that makes a request to a server and gets a 401 response with 'WWW-Authenticate: Bearer ... scope="XXXXX"'. The client app now knows it can run an OAuth exchange to get the required permission to retry the request, and it should ask for the given scope during that OAuth exchange.
The client app needs to copy XXXXXX from the response header into an OAuth exchange. It does not need to know anything more "semantics" about this scope value. XXXXXXX can be opaque to the client app. The client app does need to know any decode/encode steps involved in transferring this HTTP Bearer response value to an OAuth exchange value.

If "XXXXXXX" is "1%2A2", should the client app redirect the user to
  https://example.org/authz?scope=1%2A2&...
or
  https://example.org/authz?scope=1%252A2&...
(or can the answer vary between different servers!!)

> 2.  More than one encoding methodology could reasonably be employed in scope strings.  For instance, base64url encoding of scope values could be used in some contexts.  Quoting characters with '\'…

Servers can choose any encoding they want (including none) when defining their own scope values (as logical Unicode strings).
That is a separate choice to the encoding that is used in the HTTP Bearer response header when transporting a scope value.

> 3.  Mandating %-encoding unnecessarily complicates implementations without providing a clear compensating benefit sufficient warrant the additional complexity.  For example, it seems unnecessary to…

When the scope string "email" is transported in JSON the receiver MUST compare "\u0065mail" as equal. When it is transported as a URI query parameter the receiver MUST compare "%65mail" as equal. This is normal JSON and query parameter processing.

> 4.  If an encoding methodology for scope strings is mandated, this should be done in the OAuth Core specification - not the OAuth Bearer Token specification.

The core spec implies that a scope value can consist of any Unicode characters except space.
The core spec transports scope values in JSON or url-encoded fields that have well-defined escaping mechanisms that are sufficient to handle all Unicode string, hence all possible scope values.

The Bearer spec transports scope values in an HTTP response header parameter. It defines no escaping mechanism so it cannot handle all Unicode strings (ie it cannot handle all scope values that are possible in core).

It is strange for the Bearer spec to restrict scope values. It should leave that to the core. Hence the Bearer spec should be able to transport any scope value allowed in the core (ie any Unicode string). Hence an escaping mechanism needs to be specified for the HTTP response header scope field. 

> 5.  I am aware of no existing practice that utilizes %-encoding of scope values.

Then a solution that avoids defining an escaping mechanism might work. It requires the Bearer spec to explicitly say "The Bearer mechanism only works with scope values that are limited to these ~90 ASCII characters (that can be transported in an HTTP response header field without an escaping mechanism), which is more restrictive than OAuth core". Or better still if you like this approach, apply this restriction in core to all scope values.

--
James Manger

---

From: Mike Jones <[email-masked]>
Date: Mon, 26 Sep 2011 19:03:19 +0000

Sounds good to me.  Are others good with this wording?

				-- Mike

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Barry Leiba
Sent: Saturday, September 24, 2011 6:33 AM
To: Mike Jones
Cc: [email-masked]
Subject: Re: [OAUTH-WG] Proposed resolution for issue 26

> My proposed resolution is that %-encoding not be required in the 

I agree with your analysis, now that I see it laid out clearly.  I would feel better, though, if there were text in the document that explained that to others, who read it later.  Perhaps, using your words, we could make this change to section 2.4:

OLD

NEW

Does that work?

Barry

---

From: Mike Jones <[email-masked]>
Date: Mon, 26 Sep 2011 19:05:41 +0000

While you take the viewpoint that the bearer spec is restricting scope values, in fact, the spec intentionally allows all characters that can be safely communicated in an HTTP response header parameter to be used.  About whether those characters employ an encoding methodology to sometimes represent other characters or abstractions, I believe that Barry's proposed wording hits the nail on the head:

				Best wishes,
				-- Mike

-----Original Message-----
From: Manger, James H [mailto:[email-masked]] 
Sent: Saturday, September 24, 2011 7:41 AM
To: Mike Jones; [email-masked]
Subject: RE: Proposed resolution for issue 26

--
James Manger

---

From: "Manger, James H" <[email-masked]>
Date: Tue, 27 Sep 2011 10:20:31 +1000

> While you take the viewpoint that the bearer spec is restricting scope values, in fact,
> the spec intentionally allows all characters that can be safely communicated in an HTTP
> response header parameter to be used.

But "all characters that can be safely communicated in an HTTP response header parameter" is only a subset of the characters that OAuth Core allows in a scope value (any Unicode string excluding space). I don't understand how this isn't the Bearer spec restricting scope values.

P.S. You recognize here that non-ASCII chars cannot be safely communicated in an HTTP response header parameter. This is why Julian was concerned about the spec saying the error_description holds raw UTF-8.
[Actually the ABNF for error_description restricts it to 93 ASCII chars so when the text says it is UTF-8 encoded it is raising the potential problem of arbitrary UTF-8 in HTTP headers unnecessarily.]

--
James Manger

---

From: "Manger, James H" <[email-masked]>
Date: Wed, 28 Sep 2011 13:50:52 +1000

I'll have another go trying to explain the problem I see with the scope parameter in the Bearer spec.

Consider a French social network that decides to offer an API using OAuth2. It chooses 3 scope values for parts of the API related to family, friends, and business colleagues:
* "famille"
* "ami"
* "collègues"
Let's focus on the last scope.

The site describes the scope and its semantics in HTML developer docs. That works.
  <dt>coll&#xE8;gues</dt><dd>...</dd>

Client apps construct authorization URIs to which users are sent. That works.
  https://example.fr/authz?scope=coll%C3%A8gues...

The authorization server issues credentials in a JSON token response. That works.
  { "access_token":"SlAV32hkKG", "scope":"coll\u00E8gues", ...}

The authorization server also supports implicit grants. That works.
  Location: https://app.client.org/callback#scope=coll%C3%A8gues...

Client apps request protected resources (without needing to mention scope). That works.
  Authorization: Bearer vF9dft4qmT

A protected resource server responds with a 401 error when a bearer token is wrong. They don't know what to put in the HTTP response header scope parameter: "collègues" does not fit.

One server knows HTTP headers have historically used ISO-8859-1.
  WWW-Authenticate: Bearer scope="coll<E8>gues", error_description="opps"...

Another server sees that error_description is specified as UTF-8 so uses that.
  WWW-Authenticate: Bearer scope="coll<C3><A8>gues", error_description="opps"...

A third server reasons that the value will be copied to an authz URI so uses URI-escaping.
  WWW-Authenticate: Bearer scope="coll%C3%A8gues", error_description="opps"...

A fourth server thinks JSON-escaping looks most like HTTP's quoted-string quoting (both use '\').
  WWW-Authenticate: Bearer scope="coll\u00E8gues", error_description="opps"...

A fifth uses RFC 5987 "Character Set and Language Encoding for HTTP Header Field Parameters":
  WWW-Authenticate: Bearer scope*=UTF-8''coll%C3%A8gues, error_description="opps"...

It is a total interoperability failure for the client apps. The paragraph in the Bearer spec saying the encoding of the scope values is up to each "particular deployment context" looks like a cruel joke to the app and library developers.

--
James Manger

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Mike Jones
Sent: Tuesday, 27 September 2011 5:03 AM
To: Barry Leiba
Cc: [email-masked]
Subject: Re: [OAUTH-WG] Proposed resolution for issue 26

				-- Mike

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Barry Leiba
Sent: Saturday, September 24, 2011 6:33 AM
To: Mike Jones
Cc: [email-masked]
Subject: Re: [OAUTH-WG] Proposed resolution for issue 26

OLD

NEW

Does that work?

Barry

---

From: Barry Leiba <[email-masked]>
Date: Wed, 28 Sep 2011 09:40:00 -0400

I'd like to see more participation in this thread, besides just from
Mike and James.  What do others think?

Barry, as chair

On Mon, Sep 26, 2011 at 3:05 PM, Mike Jones <[email-masked]> wrote:
> While you take the viewpoint that the bearer spec is restricting scope
> values, in fact, the spec intentionally allows all characters that can be
> safely communicated in an HTTP response header parameter to be
> used.  About whether those characters employ an encoding
> methodology to sometimes represent other characters or abstractions,
> I believe that Barry's proposed wording hits the nail on the head:
>

On Mon, Sep 26, 2011 at 8:20 PM, Manger, James H
<[email-masked]> wrote:
> But "all characters that can be safely communicated in an HTTP response
> header parameter" is only a subset of the characters that OAuth Core
> allows in a scope value (any Unicode string excluding space). I don't
> understand how this isn't the Bearer spec restricting scope values.
>
>
> P.S. You recognize here that non-ASCII chars cannot be safely
> communicated in an HTTP response header parameter. This is why
> Julian was concerned about the spec saying the error_description holds
> raw UTF-8.
> [Actually the ABNF for error_description restricts it to 93 ASCII chars so
> when the text says it is UTF-8 encoded it is raising the potential problem
> of arbitrary UTF-8 in HTTP headers unnecessarily.]
>
> --
> James Manger

On Tue, Sep 27, 2011 at 11:50 PM, Manger, James H
<[email-masked]> wrote:
> Consider a French social network that decides to offer an API using OAuth2.
> It chooses 3 scope values for parts of the API related to family, friends, and
> business colleagues:
> * "famille"
> * "ami"
> * "collègues"
> A protected resource server responds with a 401 error when a bearer token
> is wrong. They don't know what to put in the HTTP response header scope
> parameter: "collègues" does not fit.
>
> It is a total interoperability failure for the client apps. The paragraph in the Bearer
> spec saying the encoding of the scope values is up to each "particular
> deployment context" looks like a cruel joke to the app and library developers.
>
> --
> James Manger

---

From: Julian Reschke <[email-masked]>
Date: Wed, 28 Sep 2011 15:46:23 +0200

On 2011-09-28 15:40, Barry Leiba wrote:
> ...

I agree that if a protocol element needs to support non-ASCII 
characters, the spec should be *crystal clear* how it's going to be 
transported over the wire.

Note that before James' mail I wasn't even beware that the same the 
problem I noticed for error_description also affects much more relevant 
protocol elements.

With my HTTPbis-editor hat on:

- use a consistent way of encoding in all parts of the header fields,

- and try to avoid raw UTF-8. There are alternatives, such as applying 
another round of percent-encoding, or using RFC 5987 encoding in the 
first place.

Finally, what's best in other wire formats may not be the best in header 
fields; insisting to use the same encoding will cause pain.

Best regards, Julian

---

From: Mike Jones <[email-masked]>
Date: Wed, 28 Sep 2011 15:58:36 +0000

I'd also like to see more working group participation in this discussion.

				Thanks,
				-- Mike

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Barry Leiba
Sent: Wednesday, September 28, 2011 6:40 AM
To: [email-masked]
Subject: Re: [OAUTH-WG] Proposed resolution for issue 26

I'd like to see more participation in this thread, besides just from Mike and James.  What do others think?

Barry, as chair

On Mon, Sep 26, 2011 at 3:05 PM, Mike Jones <[email-masked]> wrote:
> values, in fact, the spec intentionally allows all characters that can 
> be safely communicated in an HTTP response header parameter to be 
> used.  About whether those characters employ an encoding methodology 
> to sometimes represent other characters or abstractions, I believe 
> that Barry's proposed wording hits the nail on the head:
>

On Mon, Sep 26, 2011 at 8:20 PM, Manger, James H <[email-masked]> wrote:
>> While you take the viewpoint that the bearer spec is restricting 
>> scope values, in fact, the spec intentionally allows all characters 
>> that can be safely communicated in an HTTP response header parameter to be used.
>
> But "all characters that can be safely communicated in an HTTP 
> response header parameter" is only a subset of the characters that 
> OAuth Core allows in a scope value (any Unicode string excluding 
> space). I don't understand how this isn't the Bearer spec restricting scope values.
>
>
> communicated in an HTTP response header parameter. This is why Julian 
> was concerned about the spec saying the error_description holds raw 
> UTF-8.
> [Actually the ABNF for error_description restricts it to 93 ASCII 
> chars so when the text says it is UTF-8 encoded it is raising the 
> potential problem of arbitrary UTF-8 in HTTP headers unnecessarily.]
>
> --
> James Manger

On Tue, Sep 27, 2011 at 11:50 PM, Manger, James H <[email-masked]> wrote:
> It chooses 3 scope values for parts of the API related to family, 
> friends, and business colleagues:
> * "famille"
> * "ami"
> * "collègues"
> A protected resource server responds with a 401 error when a bearer 
> token is wrong. They don't know what to put in the HTTP response 
> header scope
> It is a total interoperability failure for the client apps. The 
> paragraph in the Bearer spec saying the encoding of the scope values 
> is up to each "particular deployment context" looks like a cruel joke to the app and library developers.
>
> --
> James Manger
_______________________________________________
OAuth mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/oauth

---

From: Marius Scurtescu <[email-masked]>
Date: Wed, 28 Sep 2011 09:36:20 -0700

> Mike and James.  What do others think?

I think James made it pretty clear that we have a problem and that we have
to solve it.

One solution would be for the core spec to limit the characters that can be
used in a scope, such that scopes are HTTP header safe. I think this would
be pretty limiting and fragile.

Another solution would be for the bearer spec to specify what escaping
should be used. RFC 5987 seems like the only good choice.

Marius

> values, in fact,
> > Client apps request protected resources (without needing to mention
> scope).
> > That works.
> > A third server reasons that the value will be copied to an authz URI so
> uses
> > URI-escaping.
> > A fifth uses RFC 5987 "Character Set and Language Encoding for HTTP
> Header
> > Field Parameters":

---

From: Brian Campbell <[email-masked]>
Date: Thu, 29 Sep 2011 07:56:33 -0600

+1 to "I think James made it pretty clear that we have a problem and
that we have to solve it."

On Wed, Sep 28, 2011 at 10:36 AM, Marius Scurtescu
<[email-masked]> wrote:
>> > be
>> > token
>> > A fourth server thinks JSON-escaping looks most like HTTP's
>> > quoted-string
>> > quoting (both use '\').
>> > paragraph in the Bearer