Subject: [OAUTH-WG] simplifying escaping

From: "Manger, James H" <[email-masked]>
Date: Thu, 1 Oct 2009 17:01:00 +1000

Escaping (or encoding) has seemed to be a big headache for OAuth implementations.

I am a little reluctant to bring it up again as the text and examples in the spec have got clearer.

However, there is room to make it considerably simpler.

Various request fields are combined into one string that is signed: method, uri, parameters.

Currently ‘&’ characters are used to separate these fields. However, escaping is required since ‘&’ is allowed in the content of some of the fields (and is used as a separator within some fields).

By choosing a different separator -- such as newline (U+000A) -- that is not allowed in the individual fields, a whole layer of escaping can be eliminated. As a bonus, the string to be signed is easier to read during debugging and in spec examples.

Amazon Web Services (AWS) support HTTP request signing with HMAC in a very similar way to OAuth. It uses newline (U+000A) to separate fields. It does not need to escape the HTTP method, URI, or query string as OAuth does.

AWS S3: http://docs.amazonwebservices.com/AmazonS3/latest/dev/RESTAuthentication.html

AWS EC2: http://docs.amazonwebservices.com/AWSEC2/latest/DeveloperGuide/using-query-api.html

Perhaps OAuth could switch from:

 esc(method) & esc(uri) & esc(esc(name1)=esc(value1) & esc(name2)=esc(value2)…)

to

 method \n

 uri \n

 esc(name1)=esc(value1) & esc(name2)=esc(value2)…

Sorting of parameters is done at a slightly awkward stage: after escaping names and values, but before combining them into a name=value string. This means you cannot sort the map of name/value pairs that the higher layers of the client will be using (as no escaping has been applied). You can’t simply sort a list of strings (as there are two parts: the name is the primary key, the value is a secondary key). Some OAuth libraries use tricks like building “esc(name) esc(value)” for each parameter (with a space between the escaped parts) that can be sorted as a string (space U+0020 sorts before any possible escaped character) -- then the library needs to rebuild the strings with “=” instead of space to do the rest of the calculation.

I would swap the order of steps 2 & 3 in 6.1.2. “Normalize Request Parameters”.

OAuth could require servers to only issue tokens and secrets that use unreserved characters: A-Z a-z 0-9 - . _ ~.

That eliminates a few more places where escaping can cause interop hassles.

A server could use “base64 with URL and filename safe alphabet” (and omitting the unnecessary trailing “=”) if it needs to represent arbitrary bytes in a token or secret. This would be an internal matter for the service. It is specified in RFC 3548 “The base16, Base32, and Base64 Data Encodings” (which is a better reference for even normal base64 than RFC 2045). The safe alphabet just substitutes “-“ and “_” for “+” and “/”.

OAuth could use this modified base64 for oauth_signature -- though I am not sure if that would be

Escaping values before putting them in the Authorization HTTP header can be eliminated. It is a bit strange at the moment that a URI-escaping mechanism (%-escaping) is used in an HTTP header quoted-string, which has a separate escaping mechanism.

These changes break backward-compatibility with OAuth 1.0 -- but I am fairly certain that was broken anyway (eg if we eliminate SHA-1).

James Manger
[email-masked]<mailto:[email-masked]>
Identity and security team — Chief Technology Office — Telstra