Subject: Re: A new RFC for Web Addresses/Hypertext References: Background wrt LEIRIs

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 01 May 2009 16:19:21 +0900

Hello Henry,

Many thanks for this very good overview. I'm cross-posting this to the 
IRI list ([email-masked]) because Lisa at one point proposed to have 
this kind of discussion there, as well as to the Apps Discuss list 
([email-masked]) to reach out to the relevant people in the IETF. 
I have also copied Lisa and Alex directly. I guess this is overall a bit 
too agressive of a cross-posting (but please tell me if you think I have 
missed somebody important). However, I hope we can converge quickly on 
where to move forward with what bits of the discussion/work.

On 2009/04/29 0:31, Henry S. Thompson wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> There are currently five documents in this space (that I am aware of):
>
>   [URI] The current RFC governing URIs:
>     http://tools.ietf.org/html/rfc3986
>
>   [IRI] The current RFC governing IRIs:
>     http://tools.ietf.org/html/rfc3987
>
>   [IRI-BIS] The most recent draft of a planned update for the RFC
>             governing IRIs:
>     http://tools.ietf.org/html/draft-duerst-iri-bis-04
>
>   [LEIRI] A W3C Note defining Legacy Extended IRIs (extracted from [IRI-BIS]):
>     http://www.w3.org/TR/leiri/
>
>   [WEBADDR] A preliminary draft of a possible RFC for Web Addresses
>             (extracted from HTML5 [1]):
>     http://www.w3.org/html/wg/href/draft.html [not yet in RFC format,
>                                                converted version expected
>                                                RSN]
>
> On the TAG telcon of 2009-04-17, there was some sense that this is too
> many specs in the same space. . .

Agreed.

> In order to contextualize and perhaps stimulate a possible effort to
> seek a rationalization here, here's _my_ understanding of how we got
> here.
>
> [URI] is the mature stage of a spec. which has been revised a number
>    of times.  It carries a certain amount of historical baggage with
>    it, particularly its restriction to 7-bit characters, but that also
>    ensures wide interoperability and preserves access to legacy
>    applications.
>
> [IRI] was intended to address the needs of the expanding Internet
>    and Web community, allowing most of Unicode into most parts of IRIs.
>    Rather than require upgrades in a wide range of applications and
>    uses, it did not set up IRIs as a _replacement_ for URIs across the
>    board, but as a _complement_ to URIs.  It therefore included an
>    explicit trancoding algorithm, for converting IRIs to URIs.
>
> [IRI-BIS] was initiated by the editors of [IRI] to correct several
>    errata to [IRI] and to address the exclusion from [IRI] of certain
>    characters and character ranges.

Yes. In that sense, it is not a separate document from [IRI], just an 
update. That's very usual for IETF work, a bit less for W3C work if one 
looks only at Recommendations.

So the effective number of documents is down from five to four.

> [LEIRI] had its origins in the XML family of W3C specifications.
>    The XML specification itself [2], as well as a number of other
>    XML-related specifications (including XML Base, XML Schema, XPointer
>    Framework, XML Signature) all involve appeal to a process for
>    converting arbitrary strings which are intended to identify web
>    resources into URIs.  They all incorporate more-or-less identical
>    prose excerpted from the XLink specification [3] which specifies how
>    this is to be done.
>
>    The XML Core WG has long been unhappy with this state of affairs,
>    and the impending release of new editions of several of these specs
>    encouraged the WG to try to establish a single normative reference
>    for the concept of a string for identifying web resources in XML
>    documents and a process for converting them to URIs, which
>    acknowledged and built on the IRI specification.
>
>    After drafting a document to serve this purpose, discussion with the
>    editors of [IRI-BIS] convinced all concerned that since a new
>    version of the IRI spec was already in progress, the best thing to
>    do, to respect precedent and to avoid unnecessary proliferation, was
>    to include the relevant definitions in [IRI-BIS], and in fact that
>    has been done [4].  Once it became apparent, however, that the
>    progress of [IRI-BIS] to Draft Standard status was likely to be
>    considerably delayed for reasons outside its editors' control, the
>    Core WG, with the agreement and co-operation of the editors of
>    [IRI-BIS], published [LEIRI] as a Working Group Note, so that the
>    re-issue of new editions of the relevant XML-familty specs could go
>    ahead.  The intention is to issue a revision of [LEIRI] replacing
>    its contents with a reference to [IRI-BIS] as soon as [IRI-BIS]
>    becomes a Draft Standard.

Yes. If that works out as described above (which I very much hope it 
will), then [LEIRI] will silently disappear.

This would reduce the number of documents from four down to three.

For the IETF side, I'd like to give a bit more background on LEIRIs.
(see http://tools.ietf.org/html/draft-duerst-iri-bis-05#section-7)

The main thing it does is to allow ASCII characters not allowed in URIs 
(and therefore not allowed in IRIs) back into what's otherwise 
essentially IRIs.

The reason for why a number of XML specs (as listed above) differ from 
the IRI spec is that these specs adopted a very early and simple 
definition of IRIs (before the name IRI every existed). Later, when the 
IRI spec got tightened, these specs didn't want to follow this 
tightening because, while XML is very strict in what it accepts and what 
not, it doesn't want to retract promises once given. Another reason is 
that in XML, context and escaping conventions allow to include 
essentially any character, whereas for URIs and IRIs in general, this is 
not the case.

> [WEBADDR] had in some ways a similar origin to [LEIRI], starting out
>    as a section of the HTML5 spec which addressed the process by which
>    existing browsers process strings to produce URIs which can be
>    dereferenced.

Yes indeed. It changes a space to %20, the same as for LEIRIs.

>    It differs from [LEIRI] in the exact set of
>    characters which it escapes,

Has anybody done an analysis?

It seems to provide more detail about '[' and ']', escaping them 
depending on context. It could be that that's also necessary for LEIRIs.

But "any occurrences of percent-encoding in the Web address will be 
double-encoded at this step." looks extremely scary.

>    and in the special handling it mandates
>    for the encoding of characters in the 'query' part of a URI.

See more about that in my reply to Dan.

> I am sure that the above summaries can be improved.  In particular it
> would be helpful have clear statements from their respective
> authors/owners as to what the _requirements_ for the three new
> documents ([IRI-BIS], [LEIRI] and [WEBADDR]) are.  Only after we have
> those would it make sense to turn to the question of whether we can
> merge some or all of them.

Okay, I'm usually not good at requirements, but for [IRI-BIS], they 
might look about as follows:

- Be usable in general, not just in a specific context
  (such as XML, HTML,...)
- Move to Draft Standard (or, if that turns out to not be possible,
   make sure we can do so on the next round.
- Try to avoid fragmentation (terms such as "Human Readable Resource
   Identifiers" or "Web Addresses" or so can lead to quite a bit of
   confusion when the main goal is to deal with occasional legacy data
   that nobody should have produced anyway)
- Include the (currently ongoing) update of IDNA (in particular affects
   section 4, Bidi, and references); that's what's currently holding
   back progress.

Are these the things that you looked for when you said 'Requirements'?
Or something else?

Regards,    Martin.

> ht
>
> [1] http://dev.w3.org/html5/spec/Overview.html#urls
> [2] http://www.w3.org/TR/xml/#dt-sysid
> [3] http://www.w3.org/TR/2001/REC-xlink-20010627/#link-locators
> [4] http://tools.ietf.org/html/draft-duerst-iri-bis-04#section-7
> - --
>         Henry S. Thompson, School of Informatics, University of Edinburgh
>                           Half-time member of W3C Team
>        10 Crichton Street, Edinburgh EH8 9AB, SCOTLAND -- (44) 131 650-4440
>                  Fax: (44) 131 651-1426, e-mail: [email-masked]
>                         URL: http://www.ltg.ed.ac.uk/~ht/
> [mail really from me _always_ has this .sig -- mail without it is forged spam]
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.2.6 (GNU/Linux)
>
> iD8DBQFJ9yFQkjnJixAXWBoRAq5tAJwMb/0jpU6XwLbYNqyt2s4uNwTcQACdHx4B
> F/J04oFFOeDHZLTT9Y0qkT0=
> =f6+L
> -----END PGP SIGNATURE-----
>
>

-- 
#-# Martin J. Dürst, Professor, Aoyama Gakuin University
#-# http://www.sw.it.aoyama.ac.jp   mailto:[email-masked]

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 01 May 2009 18:18:49 +0900

Hello Dan,

[same added cross-postings as for previous mail.]

On 2009/05/01 1:33, Dan Connolly wrote:
> On Tue, 2009-04-28 at 16:31 +0100, Henry S. Thompson wrote:
> [...]
> [...]
> Good question. I had hoped to document this in the draft by
> now.
>
> One of the main requirements the design in [WEBADDR] takes
> on is the non-western search engine problem.
>
> My understanding is that MS IE implemented a pre-IRI,
> pre-unicode convention that form submission data should
> go in the encoding that the form page was encoded in,
> and the servers bought into this.

I'm not sure this was IE, or Netscape. It was definitely very early in 
the game. It may already have been in some of the very early localized 
browsers. Or we may simply see it as a consequence of the fact that 
early browsers didn't do a conversion to Unicode, but kept data in the 
incoming encoding internally.

For servers, it was also easy to adapt. On the assumption that most 
forms are sent from pages downloaded from the same server, it's easy to 
guess that a server working e.g. with EUC-JP would send its documents 
out in EUC-JP and would prefer getting its form data back in EUC-JP.
So everything worked out relatively easily.

HTML4 actually sanctions this behavior, as follows (see 
http://www.w3.org/TR/html401/interact/forms.html#adef-accept-charset):
The default value for this attribute is the reserved string "UNKNOWN". 
User agents may interpret this value as the character encoding that was 
used to transmit the document containing this FORM element.

[btw, I have no idea what HTML5 does (or doesn't) do with 
accept-charset. For a long time, I thought it was dead, but then a few 
years ago, somebody told me that Mozilla had implemented it as part of 
their drive to become more standards-compliant.]

> Mozilla tried to follow the IRI spec, but
> "This breaks most sub-category links on a big greek on-line pc&  gadget
> shop
> (http://www.e-shop.gr) rendering the site unusable."
> https://bugzilla.mozilla.org/show_bug.cgi?id=284474

Unfortunately, the test page mentioned in that bug report is no longer 
available. It seems the domain expired.

The issue of what to do with form submission for IRIs is actually one of 
the currently open issues for the IRI spec. Please see 
http://www.w3.org/International/iri-edit/#IRI-and-forms-107.

At http://lists.w3.org/Archives/Public/public-iri/2007Jul/0000.html,
I write "Where exactly is the boundary between these two
behaviours?".

What I mean is the following. Query parts exist:
a) in "freestanding" IRIs, e.g. on a napkin, or when typed into
    the address field of a browser.
b) in plain links, e.g. a@href, or img@src,...
c) when created as part of a form submission

It seems clear to me that for a), the only choice we have is to use 
UTF-8. Things such as "the page currently being displayed below" or "the 
encoding corresponding to the language version of the browser used" just 
don't make sense.

It also seems clear to me that for c), using the encoding of the 
enclosing page is as reasonable a choice as it always was. This can be 
handled completely outside any IRI-related spec, and as far as I can 
see, seems to have been done already
(see http://www.w3.org/TR/html5/forms.html#url-encoded-form-data,
step 6.2.1; somebody needs to check the other submission methods, or 
confirm that they are irrelevant in this context.)
[Oh, and accept-charset on forms is alive in HTML5, which means that 
there is always a way to indicated that you want the query part in UTF-8.]
(see http://www.w3.org/TR/html5/forms.html#attr-form-accept-charset)
I think it would be good to point out in a guideline to authors that a 
single value, preferrably UTF-8, is best.]

So, b) is really the hard case. I definitely would have preferred if 
this went with UTF-8. The test page mentioned at
https://bugzilla.mozilla.org/show_bug.cgi?id=284474 talks about pressing 
the submit button, not about pressing a link, so this may have been c) 
rather than b), but there is no way to check anymore. But it would 
probably make sense to recheck to make sure that it also applies to b) 
(or provide a pointer to a test that's still available).

> I mostly made my peace with this in a June 2008
> discussion:
>
> Re: expected results for URI encoding tests?
> http://lists.w3.org/Archives/Public/public-html/2008Jun/0369.html
>
>
> p.s. I wish I could tell the story of the non-western
> search engine problem/requirement more straightforwardly,
> but I seem to have some sort of writer's block.

Well, the problem is that when a server gets data (in a path component, 
and even more in a query part), they need to know what encoding it is to 
make sense of it and provide a reasonable answer. The reason why you are 
calling this the "search engine" problem may be that this problem is 
most prominent with search engines, because everybody agrees that people 
want to search in their language, not limited to US-ASCII.
But it applies to any kind of query parts.

Some search engines have their own way to passing encoding information, 
as an example, in 
http://www.google.com/search?q=%E9%9D%92%E5%B1%B1&ie=utf-8&oe=utf-8, 
"ie" indicates input encoding (i.e. what's sent to the server, and "oe" 
indicates output encoding (what should be sent back).

Currently, I really don't understand why we more or less managed to move 
to "path part is UTF-8" from a time where path parts also where encoded 
in the encoding of the page that contained the link, whereas we haven't 
been able to make this transition for query parts. Is the reason that 
path parts are much more used independent of any electronic substrate 
that would allow to know the encoding (e.g. sides of buses)? Is the 
reason that "the encoding is the encoding of the containing page" was 
considered as being a defacto standard worth following for query parts, 
but just an implementation accident for path parts? Was it that RFC 3987 
(and the drafts leading up to it) were not clear enough on how to handle 
query parts, and how this related to form submission?
Is it that cgi scripts and the like (which usually process query parts) 
are much more difficult to change than the servers themselves (which 
usually process path parts)? Was it that IE made the decision, at a time 
where it could have gone both ways, but that now, there is much more 
i18n content, and it's harder to change?

Regards,    Martin.

> Somebody else care to give it a try? TimBL? Ian?
> Henri?
>

---

From: Giovanni Campagna <[email-masked]>
Date: Fri, 1 May 2009 11:37:19 +0200

2009/5/1 "Martin J. Dürst" <[email-masked]>:
> It seems to provide more detail about '[' and ']', escaping them depending
> on context. It could be that that's also necessary for LEIRIs.
>

It did such an analysis. You find it at
<http://lists.w3.org/Archives/Public/www-archive/2009Apr/0064.html>
(originally sent to the WHATWG mailing list, forwarded by Ian Hickson
to [email-masked])

Giovanni

---

From: [email-masked] (Henry S. Thompson)
Date: Mon, 04 May 2009 17:52:09 +0100

Henry S. Thompson wrote:

> Once it became apparent, however, that the progress of [IRI-BIS] to
> Draft Standard status was likely to be considerably delayed for
> reasons outside its editors' control

Two questions have arisen wrt this:

 1) Why "Draft Standard"?

    My mistake -- I misunderstood the IETF nomenclature -- all we need
    is for [IRI-BIS] to be published as an RFC with an RFC number.

 2) What "reasons outside its editors' control"?

    I don't trust my memory of the relevant rumours sufficiently to
    commit it to print: Martin, would you care to hazard a guess as to
    when [IRI-BIS] might be published as an RFC, and/or tell us what
    the hold-up was or is?

ht
- --

---

From: Larry Masinter <[email-masked]>
Date: Mon, 4 May 2009 11:54:02 -0700

Even if there are different requirements for different protocol elements,
I think it is important to have a single document which describes all
of the URI-like protocol elements that are used on the web, even if
it means that ONE document defines THREE different protocol elements.

What is unacceptable is to have four different documents with no clear
delineation between them.

It's quite possible that some of the discussion currently 
targeted at generic URIs and IRIs and LEIRIs and Web Addresses 
should instead be moved into the documents for http:, ftp:, 
and file: URIs, and some into a separate document which describes
the processing model by which a form, form-data corresponding
to the form, and a submit URI and method can be turned into
a URI (if the method is GET), etc.

Larry

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of "Martin J. Dürst"
Sent: Friday, May 01, 2009 2:19 AM
To: Dan Connolly
Cc: Henry S. Thompson; [email-masked]; [email-masked]; Apps Discuss; Lisa Dusseault; Alexey Melnikov
Subject: Re: A new RFC for Web Addresses/Hypertext References: Background wrt LEIRIs

Hello Dan,

HTML4 actually sanctions this behavior, as follows (see 
http://www.w3.org/TR/html401/interact/forms.html#adef-accept-charset):

Regards,    Martin.