Subject: Re: On the proliferation of well-known URLs; draft-nottingham-site-meta-01

From: Sam Johnston <[email-masked]>
Date: Sat, 2 May 2009 11:07:37 +0200

Brad,

Sounds like a good idea, though I'd avoid using characters which are likely
to break scripts/filters/etc.

Perhaps something like "/__meta__" would be a better bet?

Sam

On Wed, Apr 29, 2009 at 7:50 PM, Brad Fitzpatrick <[email-masked]> wrote:

> I'd like to discuss the proliferation of well-known URLs, notably the new
> one proposed in draft-nottingham-site-meta-01.
>
> I talked to Eran Hammer-Lahav about this, but he suggested I bring it up
> here:
>
> I object to the use of /host-meta for two reasons:
>
> 1) I feel that /host-meta is too casual of a name and prone to collisions.
> It matches /^[\w\-]+$/, which I think is a subset of a fair number of sites'
> usernames.
>
> 2) There are already too many well-known URLs cluttering up the namespace:
>
> /robots.txt
> /favicon.ico
> /crossdomain.xml
>
> We can't fix those, but rather than make another one (and don't kid
> yourself: /host-meta won't be the last one) and make the situation worse, I
> propose we do the respectful thing and make a well-known directory to put
> this and all future well-known files in.  e.g.:
>
> /;well_known/host-meta
>
> i.e. put something ugly and weird in there, like a semicolon, to minimize
> the chance that it interferes with people's existing URL structure.
>
> Hopefully when the next spec decides to add a new well-known URL, they put
> it under /;well_known/.  "But host-meta is the final one, forever!", you
> say.  I doubt it.  XRD will become passé, or people will object to doing two
> HTTP requests when they really want to do one, so yet another well-known URL
> will be born.  Let's give it a future home now.
>
> Thoughts?
>
> - Brad
>
>
> _______________________________________________
> Apps-Discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss
>
>

---

From: Mark Baker <[email-masked]>
Date: Sat, 2 May 2009 05:30:00 -0700

> It matches /^[\w\-]+$/, which I think is a subset of a fair number of sites'

Perhaps, but I don't see any cause for concern here;

http://www.google.com/search?q=inurl:host-meta

> yourself: /host-meta won't be the last one)

I agree, but because there will be people who don't know about the
protocol, and that will be the case no matter which protocol is used.

> and make the situation worse, I

The Web doesn't have directories, it has URIs, and both what you
describe, and what's in the draft, define a well known URI per domain.
 So I really don't see any practical difference here.

Mark.

---

From: Brad Fitzpatrick <[email-masked]>
Date: Sat, 2 May 2009 09:11:14 -0700

> http://www.google.com/search?q=inurl:host-meta

Yet.  But once we pick something easily-creatable, you'd see a lot more
bogus ones, or you'd see resistance to people supporting it, because
host-meta is already in the namespace that they use for other things.  I'm
not making up this concern:  I'm already hearing that from people when I
talk about having them implement this.

> protocol, and that will be the case no matter which protocol is used.

I don't follow.  I'm proposing a common prefix, not protocol, for well-known
URIs cluttering up the namespace.  Unrelated to host-meta, I want a company
in the future that _needs_ to have their little file at the top-level of a
domain to be able to follow in our footsteps and use
/;wellknown/new-file.txt

> The Web doesn't have directories, it has URIs,

Yes, thanks.  :)  I'm proposing that our well-known URI that we're creating
here has a prefix which other well-known URIs in the future can share, to
stop the build-up of well-known URIs in the namespace that sites have
effectively allocated for other things, like users:

http://twitter.com/host-meta
http://profiles.google.com/host-meta
http://identi.ca/host-meta

Those don't exist at present, but it's not crazy to think there are sites
out there where URLs like those could exist.

> and both what you

There's no functional difference.  The difference is only not cluttering up
people's namespaces.

The practical difference is not annoying sites.

---

From: Brad Fitzpatrick <[email-masked]>
Date: Sat, 2 May 2009 09:13:05 -0700

> to break scripts/filters/etc.

That's kinda the point.

> Perhaps something like "/__meta__" would be a better bet?

That's even worse than host-meta in the sense that it's more common of a
username pattern (more sites allow "_" than "-" in a username).  And one of
my main concerns is the number of sites that give users a URL like
site.com/USERNAME

---

From: Eran Hammer-Lahav <[email-masked]>
Date: Mon, 4 May 2009 08:18:17 -0700

This makes sense but I don’t have strong views here. My only concern is to get this adopted for host-meta so whatever proves to be the path of least resistance works for me.

On the business of using odd characters, isn’t choosing any ‘*/’ prefix going to solve your concerns? No site I know allows using ‘/’ for usernames, short URLs, etc. I’m afraid making the prefix “ugly” will cause people not to use it and will make documenting it confusing to people who will not be expecting something like that.

EHL

From: [email-masked] [mailto:[email-masked]] On Behalf Of Brad Fitzpatrick
Sent: Saturday, May 02, 2009 9:13 AM
To: Sam Johnston
Cc: [email-masked]
Subject: Re: On the proliferation of well-known URLs; draft-nottingham-site-meta-01

On Sat, May 2, 2009 at 2:07 AM, Sam Johnston <[email-masked]<mailto:[email-masked]>> wrote:
Brad,

Sounds like a good idea, though I'd avoid using characters which are likely to break scripts/filters/etc.

That's kinda the point.

That's even worse than host-meta in the sense that it's more common of a username pattern (more sites allow "_" than "-" in a username).  And one of my main concerns is the number of sites that give users a URL like site.com/USERNAME<http://site.com/USERNAME>

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 4 May 2009 11:01:05 -0500

> domain to be able to follow in our footsteps and use

The well-known prefix should be used for new things like robots.txt,
host-meta, ...  That relies on developers to get that right, but I think
eventually enough will get it right for this to make a difference, thus
this idea seems worthwhile.

It would be nice to have a registry of well-known URI components.

> out there where URLs like those could exist.

Exactly.  Why should some bright eyed developer come up with the Next
Big Thing that steps all over some other peoples' URIs when said
developer could use a reserved part of the namespace at no extra cost to
themselves?

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 4 May 2009 11:06:30 -0500

On Mon, May 04, 2009 at 08:18:17AM -0700, Eran Hammer-Lahav wrote:
> This makes sense but I don’t have strong views here. My only concern
> is to get this adopted for host-meta so whatever proves to be the path
> of least resistance works for me.
> 
> On the business of using odd characters, isn’t choosing any ‘*/’
> prefix going to solve your concerns? No site I know allows using ‘/’
> for usernames, short URLs, etc. I’m afraid making the prefix “ugly”
> will cause people not to use it and will make documenting it confusing
> to people who will not be expecting something like that.

But sites do often allow users to have files, so "*/" falls down.

---

From: Eran Hammer-Lahav <[email-masked]>
Date: Mon, 4 May 2009 09:45:00 -0700

> -----Original Message-----
> From: Nicolas Williams [mailto:[email-masked]]
> Sent: Monday, May 04, 2009 9:07 AM
> To: Eran Hammer-Lahav
> Cc: Brad Fitzpatrick; Sam Johnston; [email-masked]

That goes beyond what we can reasonable protect against. If you are going to let users do whatever they want on your root namespace, you are going to have other problems. Sites allowing users to have files under custom sub-directories will need more protections than just some *potentially* illegal characters.

EHL

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 6 May 2009 20:15:51 +1000

I don't buy this argument, because twitter et al have clearly worked  
around robots.txt, favicon.ico and many others without any apparent  
problem. If you run a Web site that allows users to create arbitrary  
URLs, it's your responsibility to anticipate this, and I'd note that  
this seems to be working just fine with the non-standard locations  
above; why wouldn't they work with a standard one?

That said, if people think it's necessary to address this, I can see  
two possibilities (adding semicolons and other such line noise to the  
path is ugly and brings its own problems, as pointed out):

1) using ".host-meta" -- dot-files are a widely-recognise convention,  
and are relatively friendly to implementations, or

2) using /host-meta/* or for that matter, /well-known/* -- i.e.,  
setting aside a path component whose children are registered path  
components, one per application, instead of having a single  
"directory" file. This increases the "footprint" of the solution in  
terms of URI namespace impingment, but reduces the number of requests  
(often).

Thoughts?

Cheers,

On 30/04/2009, at 3:50 AM, Brad Fitzpatrick wrote:

> I'd like to discuss the proliferation of well-known URLs, notably  
> the new one proposed in draft-nottingham-site-meta-01.
>
> I talked to Eran Hammer-Lahav about this, but he suggested I bring  
> it up here:
>
> 1) I feel that /host-meta is too casual of a name and prone to  
> collisions.  It matches /^[\w\-]+$/, which I think is a subset of a  
> fair number of sites' usernames.
>
> yourself: /host-meta won't be the last one) and make the situation  
> worse, I propose we do the respectful thing and make a well-known  
> directory to put this and all future well-known files in.  e.g.:
>
> /;well_known/host-meta
>
> i.e. put something ugly and weird in there, like a semicolon, to  
> minimize the chance that it interferes with people's existing URL  
> structure.
>
> Hopefully when the next spec decides to add a new well-known URL,  
> they put it under /;well_known/.  "But host-meta is the final one,  
> forever!", you say.  I doubt it.  XRD will become passé, or people  
> will object to doing two HTTP requests when they really want to do  
> one, so yet another well-known URL will be born.  Let's give it a  
> future home now.
>
> Thoughts?
>
> - Brad
>

--
Mark Nottingham     http://www.mnot.net/

---

From: Sam Johnston <[email-masked]>
Date: Wed, 6 May 2009 12:34:42 +0200

On Wed, May 6, 2009 at 12:15 PM, Mark Nottingham <[email-masked]> wrote:

> 1) using ".host-meta" -- dot-files are a widely-recognise convention, and
> are relatively friendly to implementations, or
>
> 2) using /host-meta/* or for that matter, /well-known/* -- i.e., setting
> aside a path component whose children are registered path components, one
> per application, instead of having a single "directory" file. This increases
> the "footprint" of the solution in terms of URI namespace impingment, but
> reduces the number of requests (often).
>
> Thoughts?
>

The second option sounds better than the first which sounds better in turn
than the ugly URLs (which are sure to trip application firewalls, proxies,
filters, etc. all over the shop).

For host-meta <http://www.google.com/search?q=inurl%3Ahost-meta> (and
meta-data <http://www.google.com/search?q=inurl:meta-data> for that matter,
which could be an even better option) we're not standing on any toes -
according to Google at least.

I would suggest mentioning that users should not be able to create content
under this hierarchy as a security consideration. It may also be worth
looking at how existing root pollution might be migrated over time.

Sam

---

From: Eran Hammer-Lahav <[email-masked]>
Date: Wed, 6 May 2009 09:53:09 -0700

I would like to keep the single-file we have proposed. The main use cases for host-meta today are mostly focused on using Link-Pattern records more than Link records. I also want to preserve the ability to place these files elsewhere (on a different path or host) which is easier to do with a Link record than a 3xx in most deployments.

So the question is where to put it and what to name it.

I like the idea of a directory.

So we either find a new name for the directory (I don't like /well-known/*), or use /host-meta/* and find a new name for the file (such as directory, index, global, etc.).

EHL

> From: [email-masked] [mailto:apps-discuss-
> [email-masked]] On Behalf Of Mark Nottingham
> Sent: Wednesday, May 06, 2009 3:16 AM
> To: Brad Fitzpatrick; [email-masked]

---

From: Brad Fitzpatrick <[email-masked]>
Date: Wed, 6 May 2009 10:54:38 -0700

On Wed, May 6, 2009 at 3:15 AM, Mark Nottingham <[email-masked]> wrote:

> I don't buy this argument, because twitter et al have clearly worked around
> robots.txt, favicon.ico and many others without any apparent problem.

Because those two have been around for ages.  We can't just keep adding
more, though.  You can't anticipate all the future crap people are going to
add.

> If you run a Web site that allows users to create arbitrary URLs, it's your
> responsibility to anticipate this, and I'd note that this seems to be
> working just fine with the non-standard locations above; why wouldn't they
> work with a standard one?

With future standards they don't know about yet?

> That said, if people think it's necessary to address this, I can see two
> possibilities (adding semicolons and other such line noise to the path is
> ugly and brings its own problems, as pointed out):
>

I'd be happy with either:

/.wellknown/host-meta
/,wellknown/host-meta

A comma is weird enough that it's not likely to be in potential use, but
nice enough to be acceptable in filesystem components (for those actually
storing this stuff on filesystem) and isn't a shell special character for
people fetching these URLs with curl, etc.

But the dotfile directory is more 'normal' and I could see people being more
accepting of it.  I like it because no site should be insane enough to let
people's usernames start with periods.  On the other hand,

http://profiles.google.com/.wellknown/

... could become valid, if the existing user [email-masked] creates a
public profile.  (periods anywhere in gmail usernames are ignored)

But I'm not worried about the google profile case.  /.wellknown/ is still
good.

- Brad

---

From: Brad Fitzpatrick <[email-masked]>
Date: Wed, 6 May 2009 16:13:41 -0700

> robots.txt, favicon.ico and many others without any apparent problem.

Counter-point:

http://profiles.google.com/robots.txt
http://profiles.google.com/favicon.ico

I'll file a bug, but that just goes to show that people DO have problems
with this.

(For the record:  it redirects to the user's canonical profile URL....)

Apparently robotstxt at gmail.com exists (I tried to create the account and
it said taken), which Google properties treat identically to "robots.txt"
(periods aren't signifigant) so that URL above is for a Google user that
doesn't have a public profile or has a public profile with a obfuscated
numeric URL.

Please, no more top-level URLs.  Let's minimize special-casing in
future/current web apps.