Subject: host-meta: template syntax hassles

From: "Manger, James H" <[email-masked]>
Date: Tue, 27 Oct 2009 16:01:42 +1100

draft-hammer-hostmeta-02<http://tools.ietf.org/rfcmarkup?doc=draft-hammer-hostmeta#section-3.2.1> “Host-Meta: Web Host Metadata” defines a URI template syntax and variables based on parts of a URI [§3.2.1.].

I don’t think the current arrangement can work as easily as the examples in the draft suggest.

The example of a link to each resource’s author presumably wants “;by” to be appended to the resource URI’s path — but that is not what the “{+uri};by” template does.

  <Link> <Rel>author</Rel> <URITemplate>{+uri};by</URITemplate> </Link>

For URIs with query strings, “;by” is appended to the last query parameter value. This is almost certainly NOT desired. The order of query parameters is normally irrelevant but this template changes the value of whichever parameter happens to be last.

A more correct template is probably: “{+scheme}://{+authority}{+path};by?{+query}”

I am not sure that even this is correct if the path is empty (eg you may get the invalid URI “http://example.net;by?”)

There will also be a trailing “?” when there are no query parameters, which may be mostly harmless, but is strictly different from a URI without the “?” so caching and comparisons are adversely affected.

Another example from the draft is “{+uri}&test”. This either makes “&test” part of the path, or a new query parameter (depending on whether or not the URI already has a query string). This is probably never desirable. If a template wants to add a query parameter it probably needs to specify

  “{+scheme}://{+authority}{+path}?{+query}&test”.

Even this is not ideal as it produces unusual URIs such as “http://example.net/article?&test”, which again may be mostly harmless but is strictly different from a URI with just “?test” so caching and comparisons are adversely affected.

Another example from the spec is “http://meta.{host}:8080{+path}?{+query}”.

If some URIs might have a userinfo component then the template needs to be “http://{+userinfo}@meta.{host}:8080{+path}?{+query}”. However, when there is no userinfo the resulting URI starts http://@… that (in some clients at least, eg curl) causes a “Authorization: Basic Og==” header to be included when requesting the URI. That could be harmful.

P.S. It should probably also use {+host}, instead of {host}.

The minimal (partial) solution is to change the examples in the draft to be realistic (eg work for all URIs in the scope, with or without query strings).

Another solution is to define a more sophisticated template syntax. A syntax that lists a prefix that is only substituted when the variable is defined (non-null) may be almost sufficient. That address the problem that the defined variables for URI parts (scheme, authority, query, fragment, userinfo, host, port) don’t include the separator characters used when combining the parts.

In the host-meta case a better solution would be to define the translation from resource URI to link URI with a regular expression and replacement string. The replacement string can reference “capturing groups” in the regular expression. I am sure all modern programming languages offer broadly similar regex-based replacement functionality. For instance, for Java see String#replaceFirst(regex, replacement)<http://java.sun.com/javase/6/docs/api/java/lang/String.html#replaceAll(java.lang.String,%20java.lang.String)> and Matcher.appendReplacement<http://java.sun.com/javase/6/docs/api/java/util/regex/Matcher.html#appendReplacement%28java.lang.StringBuffer,%20java.lang.String%29>.

Then the question is where is the regex specified: in the <Link> or as the <Subject>?

Using a regular expression to indicate all the URIs that the metadata (XRD) applies to (ie the XRD’s subject) feels flexible and appropriate. Host-meta would not need to be a special case with its own <host-meta:Scope>. Metadata that applied to all the URIs in a subdirectory (eg all under /blog/*) would be easy to define.

<XRD>

 <SubjectRegex>https?://(?:www\.)?example\.com/([^?#]*)(\?[^#]*)?(#\.*)?</SubjectRegex>

 <Link><Rel>author</Rel><URIPattern>https://www.example.com/\1;by\2</URLPattern></Link>

</XRD>

It is a pity regexs can be awkward to write (& harder to read). Including an example (or default value?) in the spec that captures the scheme, authority, path, query & fragment could be almost equivalent to the current template proposal (with much more flexibility, but less friendly variable names).

James Manger
[email-masked]<mailto:[email-masked]>
Identity and security team — Chief Technology Office — Telstra

---

From: Eran Hammer-Lahav <[email-masked]>
Date: Mon, 26 Oct 2009 22:32:47 -0700

Thanks James.

The examples are obviously limited to URI with specific characteristics. If the presence of these examples is misleading, I will be happy to change them to put the {uri} at the end as a query parameter.

Defining a more complex syntax is absolutely out of scope. The template syntax is kept intentionally simple and a subset of Roy’s proposal for a URI template standard in hopes that when that work is completed, it will be forward compatible (code written for the new proposal being able to handle old files).

EHL

From: [email-masked] [mailto:[email-masked]] On Behalf Of Manger, James H
Sent: Monday, October 26, 2009 10:02 PM
To: [email-masked]
Subject: host-meta: template syntax hassles

<XRD>

James Manger

---

From: "Manger, James H" <[email-masked]>
Date: Tue, 27 Oct 2009 18:04:28 +1100

Eran,

> The examples are obviously limited to URI with specific characteristics.

> If the presence of these examples is misleading,

host-meta is explicitly not limited to URIs with specific characteristics (as it covers one or more entire hosts) so I think they are misleading.

[It may even be potentially dangerous: attacker sends “http://example.com/xxx?” to victim; victim (& server) treat it as equivalent to “http://example.com/xxx”; but meta-data is retrieved from a different source given {+uri};meta template: from /xxx (server ignores unexpected “?;meta” query string in this request) vs from /xxx;meta (that server recognizes).]

> I will be happy to change them to put the {uri} at the end as a query parameter.

We can’t have the only realistic example in the spec being {uri} as a query parameter. It is no use having templates in that case.

Changing the extension (*.html to *.xrd), appending “;meta” to the path, adding an “_about” query parameter, using a metadata.example.com sub-domain etc all sound like realistic choices hosts might make. The spec should include examples for these sorts of choices. If the examples are too complex then the only answer is to simplify the syntax, not just omit the examples.

If examples like {+scheme}://{+authority}{+path};by?{+query} and {+scheme}://{+authority}{+path}?{+query}&test  are too complex we need a simpler syntax.

James Manger

<XRD>

</XRD>

James Manger

---

From: Cyrus Daboo <[email-masked]>
Date: Tue, 27 Oct 2009 09:15:21 -0400

Hi Eran,

--On October 26, 2009 10:32:47 PM -0700 Eran Hammer-Lahav 
<[email-masked]> wrote:

> Defining a more complex syntax is absolutely out of scope. The template
> syntax is kept intentionally simple and a subset of Roy’s proposal for
> a URI template standard in hopes that when that work is completed, it
> will be forward compatible (code written for the new proposal being able
> to handle old files).
>
>

There was an old URI template draft: 
<http://tools.ietf.org/html/draft-gregorio-uritemplate-03>? Has any thought 
been given to reviving that, or is Roy working on something else altogether?

-- 
Cyrus Daboo

---

From: Julian Reschke <[email-masked]>
Date: Tue, 27 Oct 2009 14:21:46 +0100

Cyrus Daboo wrote:
> <http://tools.ietf.org/html/draft-gregorio-uritemplate-03>? Has any 
> thought been given to reviving that, or is Roy working on something else 
> altogether?

As far as I recall, Roy has been working on it in July, but I'm not sure 
what the current status is...

See: <http://code.google.com/p/uri-templates/>

BR, Julian

---

From: Eran Hammer-Lahav <[email-masked]>
Date: Tue, 27 Oct 2009 08:55:45 -0700

The most recent proposal (not yet submitted as an I-D but available from the link Julian mentioned below) is very different from the current draft. Ideally, host-meta would simply reference it and be done with it (which is the main goal of the community working on this proposal). However, it is unlikely that the template draft will be completed and published as an RFC in the near future. What I have attempted to do is take the bare minimum needed to have some kind of basic template functionality while at the same time allowing upgrade in the future. While there is no guarantee that the template proposal is not going to change yet again, I feel more comfortable with this approach than making up a whole new, competing, template syntax.

I am open to other suggestions, however, I am completely opposed to the creation of any new template syntax unrelated to the draft-gregorio-uritemplate effort.

Hopefully Roy will find time soon to publish his new syntax as an I-D.

EHL

> -----Original Message-----
> From: Julian Reschke [mailto:[email-masked]]
> Sent: Tuesday, October 27, 2009 6:22 AM
> To: Cyrus Daboo
> Cc: Eran Hammer-Lahav; Manger, James H; [email-masked]
> Subject: Re: host-meta: template syntax hassles
> 
> Cyrus Daboo wrote:
> > Hi Eran,
> >

---

From: Eran Hammer-Lahav <[email-masked]>
Date: Tue, 27 Oct 2009 09:11:45 -0700

> From: [email-masked] [mailto:apps-discuss-
> [email-masked]] On Behalf Of Manger, James H
> Sent: Tuesday, October 27, 2009 12:04 AM
> Subject: RE: host-meta: template syntax hassles
> 
> Eran,
> 
> host-meta is explicitly not limited to URIs with specific
> characteristics (as it covers one or more entire hosts) so I think they
> are misleading.

I will change them to be more consistent with the scope. However, it is ultimately the job of the document author to make sure the templates are compatible with the URI namespace it covers. Suggestions to attribute specific schemes or resource subsets to links have been rejected in the past as too complex and without actual use cases, so I am not inclined to complicate the overall schema.

> [It may even be potentially dangerous: attacker sends
> “http://example.com/xxx?” to victim; victim (& server) treat it as
> equivalent to “http://example.com/xxx”; but meta-data is retrieved from
> a different source given {+uri};meta template: from /xxx (server
> ignores unexpected “?;meta” query string in this request) vs from
> /xxx;meta (that server recognizes).]

I think this is more about poor URI and template design than a security threat. This will always be the case unless the only variable allowed is the full URI string. Protocols making security-sensitive decisions based on URI templates should watch out for such manipulations. I will add this to the security section of the spec.
 
> We can’t have the only realistic example in the spec being {uri} as a
> query parameter. It is no use having templates in that case.
> 
> Changing the extension (*.html to *.xrd), appending “;meta” to the
> path, adding an “_about” query parameter, using a metadata.example.com
> sub-domain etc all sound like realistic choices hosts might make. The
> spec should include examples for these sorts of choices. If the
> examples are too complex then the only answer is to simplify the
> syntax, not just omit the examples.
> 
> If examples like {+scheme}://{+authority}{+path};by?{+query} and
> {+scheme}://{+authority}{+path}?{+query}&test  are too complex we need
> a simpler syntax.

I don’t think they are too complex. I will add a more extensive section with such examples. You might be right that the syntax as currently proposed might not be able to accomplish all these use cases (due to empty parameters and inability to condition the inclusion of a separator - &, ?, etc. - for non-empty parameters). Roy's proposal handles these cases but at some point it becomes obscene to copy/paste larger chunks of that work. The simplest solution would be to define a few more variables which will include the separator when not empty.

I'll give it a try in the next draft.

Thanks,

EHL

---

From: "Manger, James H" <[email-masked]>
Date: Wed, 28 Oct 2009 09:51:56 +1100

One solution (for simple, but correct, URI templates for common situations in host-meta) is to define more variables.

For instance, in addition to ‘scheme’, ‘authority’, ‘path’, ‘query’… offer the following variables:

‘toDir’ — the URI up to, but excluding, the last ‘/’ in the path

‘afterDir’ — the URI from the last segment of the path to the end, excluding the fragment

‘toExt’ — the URI up to, but excluding, the last ‘.’ in the last segment of the path (or to the end of the path if there is no such dot)

 ‘toPath’ — the URI up to the end of the path

‘afterPath’ — the URI from after the path to the end, excluding the fragment

‘toQueryPlus’ — the URI excluding any fragment, followed by ‘&’ if there is a query string or ‘?’ otherwise (ie ready for a new query parameter to be appended)

‘beforeHost’ — the URI up to, but excluding, the host

‘fromHost’ — the URI from the host to the end, excluding the fragment

Template examples:

{+toExt}.xrd{+afterPath} — replace, say, .html extension with .xrd to get metadata

{+toQueryPlus}_=meta — add a query parameter named “_” with a value “meta” to get metadata

{+toDir}/.meta/{+afterDir} — metadata is in a special .meta/ subdirectory of each directory

{+toPath};meta{+afterPath} — append “;meta” to the path to get the metadata

{+beforeHost}meta.{+fromHost} —metadata is hosted on a sub-domain

The to/after (and before/from) pairs basically offer easy access to different points in the URI to insert a new element. I think these examples are fairly simple ways to implement common URI manipulations safely.

James Manger