Subject: Re: [apps-discuss] Working Group Last Call:draft-ietf-appsawg-xml-mediatypes

From: "Murray S. Kucherawy" <[email-masked]>
Date: Thu, 7 Nov 2013 13:44:03 -0800

Colleagues, but specifically Tom, Erik, Dave, Julian and Bjoern:

Can you have a look at the latest version of this draft (or the diffs,
available via the tracker) and let us know if you think it's ready to go in
its latest form?

Thanks,
-MSK, APPSAWG co-chair

On Thu, Oct 17, 2013 at 9:23 AM, t.petch <[email-masked]> wrote:

> ----- Original Message -----
> From: "Julian Reschke" <[email-masked]>
> To: "Henry S. Thompson" <[email-masked]>
> Cc: "Bjoern Hoehrmann" <[email-masked]>; <[email-masked]>
> Sent: Tuesday, October 15, 2013 6:16 PM
> > On 2013-10-15 19:08, Henry S. Thompson wrote:
> > > Julian Reschke writes:
> > >
> > >> The point I was trying to make is that we now have a registry. The
> > >> registry is authoritative; if we move the definition of "+xml",
> it's
> > >> sufficient to simply update the registry.
> > >
> > > I don't agree.  If we don't include 'Updates: 6839', then 6839 will
> > > not get an 'Updated by [3023bis]', and anyone reading 6839 will be
> > > free to conclude that it defines +xml, which would be false.
> >
> > Anyone reading it that way would be reading it incorrectly :-)
> >
> > But anyway, that's something the IESG can worry about.
>
> Having just read about the difficulty in finding potential members of
> the IESG, I think we should do what we can as a WG to make the task of
> the IESG simpler, in this case by putting forward a rough consensus from
> the WG.
>
> I am with Henry on this one.  It is true that if your world revolves
> around web pages, then the IANA web page will take you to the current
> definition whereever that is.  But if you start with RFC, for example
> with the rfc-index, and that does not mention an update to an RFC, then
> you will go astray.
>
> Comparing the two definitions, in RFC6839 and this I-D, they seem to me
> like chalk and cheese, as for example in who has change control and who
> the contact is, so that the RFC6839 one will be wiped from the face of
> the earth; specifying 'Updates' seems the least we can do.
>
> Tom Petch
>
> > Best regards, Julian
> >
>
>
> _______________________________________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss
>

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Thu, 07 Nov 2013 23:28:55 +0100

>available via the tracker) and let us know if you think it's ready to go in

It seems that in the current draft Unicode signatures take precedence
over media type `charset` parameters. This is a very recent change that
renders all implementations that conform to RFC 3023 non-compliant and
it makes the XML types inconsistent with other types like text/plain.

It would not be possible to apply the same change to text/plain without
making it practically impossible to encode text/plain using ISO-8859-1.
Pretty much all text formats that allow you to put non-ASCII bytes at
the beginning of a document have this problem if their media types uses
the `charset` parameter mechanism.

The XML types have the same problem as the draft currently proposes it;
a resource like data:application/xml;charset=utf-32,%FF%FE%00%00... will
probably be interpreted as malformed UTF-16 encoded document even though
it clearly declares UTF-32 and is well-formed when interpreted as such.
Similar for `data:application/xml;charset=utf-16le,%FF%FE...`.

I think this is a serious and problematic change that needs wide review
prior to a two-week Last Call period.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: [email-masked] (Henry S. Thompson)
Date: Thu, 07 Nov 2013 22:40:34 +0000

> it makes the XML types inconsistent with other types like text/plain.

3023 did not specify behaviour at all in this case, leaving
implementors free to choose.  Implementors have all chosen to take the
BOM as definitive.  The proposed text merely ratifies this.

ht
-- 
       Henry S. Thompson, School of Informatics, University of Edinburgh
      10 Crichton Street, Edinburgh EH8 9AB, SCOTLAND -- (44) 131 650-4440
                Fax: (44) 131 650-4587, e-mail: [email-masked]
                       URL: http://www.ltg.ed.ac.uk/~ht/
 [mail from me _always_ has a .sig like this -- mail without it is forged spam]

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Fri, 08 Nov 2013 00:16:44 +0100

>BOM as definitive.  The proposed text merely ratifies this.

That internal byte sequences take precedence over external labels is not
a reasonable interpretation of RFC 3023 and the XML 1.0 specification.
Treating data:application/xml;charset=utf-32,%FF%FE%00%00... as UTF-16
encoded document for instance directly contradicts section F.1 of the
XML 1.0 Recommendation. And no, giving precendence to the signature is a
recent trend among some browser developers. It is not what I implemented
e.g. for W3C's Markup Validator. If the XML community wishes to make a
change such as this, that is fine by me, but I have seen little to no
evidence that people are aware of this proposal, fully understand it,
and agree with it, and claims such as yours above do not help to change
that. More helpful would be test results from a diverse set of implemen-
tations; that should be easy if implementers all have chosen as you are
suggesting. My http://search.cpan.org/dist/HTML-Encoding/ is a "fail".
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Julian Reschke <[email-masked]>
Date: Thu, 07 Nov 2013 16:10:01 -0800

On 2013-11-07 13:44, Murray S. Kucherawy wrote:
> available via the tracker) and let us know if you think it's ready to go
> in its latest form?
>
> Thanks,
> -MSK, APPSAWG co-chair

Actually, I looked at the diffs wrt to my comments, and they seem to be 
addressed.

I want to do another pass at the document; at least the 2616/HTTPbis 
references need to be cleaned up (see the mail I sent in October).

Best regards, Julian

---

From: Erik Wilde <[email-masked]>
Date: Thu, 07 Nov 2013 17:06:59 -0800

> available via the tracker) and let us know if you think it's ready to go

i looked at 
http://tools.ietf.org/rfcdiff?url2=draft-ietf-appsawg-xml-mediatypes-04.txt&url1=draft-ietf-appsawg-xml-mediatypes-02.txt 
and i am happy with how my comments have been addressed in the latest draft.

thanks and cheers,

dret.

-- 
erik wilde | mailto:[email-masked]  -  tel:+1-510-2061079 |
            | UC Berkeley  -  School of Information (ISchool) |
            | http://dret.net/netdret http://twitter.com/dret |

---

From: [email-masked] (Henry S. Thompson)
Date: Fri, 08 Nov 2013 15:17:51 +0000

Bjoern Hoehrmann writes:

> a reasonable interpretation of RFC 3023

This draft replaces 3023.

> and the XML 1.0 specification.  Treating
> data:application/xml;charset=utf-32,%FF%FE%00%00... as UTF-16
> encoded document for instance directly contradicts section F.1 of
> the XML 1.0 Recommendation.

Appendix F is non-normative, but in any case your statement is
incorrect.  F.1 applies only in the _absence_ of a charset parameter.
F.2 applies when there is one, and it says:

  When multiple sources of information are available, their relative
  priority and the preferred method of handling conflict should be
  specified as part of the higher-level protocol used to deliver XML.

So it's entirely in-scope for 3023bis to specify that a BOM is
definitive in the presence of a charset parameter.

Furthermore, your example is broken in another way -- XML _requires_
any document not in UTF-8 or UTF-16 to begin with an XML encoding
declaration.  The above data:... document does not do so.  So it's not
an XML document at all, and so not a legitimate use of the
application/xml media type.

> And no, giving precendence to the signature is a recent trend among
> some browser developers.

I don't know about 'recent', but it's widespread.  IE9, Chrome
32.0.1700.4, Opera 12.16, Safari 5.1.7 and Firefox 25 all treat a BOM
as definitive despite the presence of a conflicting charset parameter
value, as do expat and rxp.  libxml2 is the only counter-example I'm
aware of.

> More helpful would be test results from a diverse set of
> implementations; that should be easy if implementers all have chosen
> as you are suggesting. 

See the above list, which includes about as diverse a set of browsers
as you can get. . .

ht

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Fri, 08 Nov 2013 21:23:03 +0100

>value, as do expat and rxp.  libxml2 is the only counter-example I'm

I think I have made clear why I think this change needs much broader
review, but I guess it does not hurt to offer the test case a third
time, `data:application/xml;charset=utf-32,%FF%FE%00%00...`. I've put
it up at http://www.websitedev.de/temp/application-xml-utf32-le.xml
so people can easily check that IE9 treats that as UTF-32 even though
it starts with a UTF-16 Unicode Signature.

I also mentioned the W3C Markup Validator as an implementation that
gives precedence to the `charset` parameter in the `Content-Type`
header and I just checked, it, too, treats it as a UTF-32 document.

http://www.websitedev.de/temp/application-xml-utf32-le-decl.xml is
the same document with an encoding declaration. Whether an encoding
is required can easily determined by grepping the XML specification
for "In the absence". Accordingly the W3C Markup Validator finds no
error with either document. For external entities for example

  In the absence of external character encoding information (such as 
  MIME headers), parsed entities which are stored in an encoding other
  than UTF-8 or UTF-16 MUST begin with a text declaration (see 4.3.1
  The Text Declaration) containing an encoding declaration

You also list Expat. I was under the impression that Expat does not
actually support reading HTTP responses. I downloaded the sources for
expat-2.1.0 and the string `charset` only appears under `xmlwf/` which
would be an expat-based utility, but even there the code seems dead.
Similar for "Host", "Content-Type", "HTTP/", "winsock", "socket". So,
I think Expat also does not do as you describe.

And rxp-1.5.0 does seem to have some HTTP support, but a grep for the
string `Content-Type` returns unsuccessfully. That usually means the
Content-Type header is ignored entirely.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Tony Hansen <[email-masked]>
Date: Sun, 10 Nov 2013 00:16:18 -0500

On 10/15/2013 3:45 AM, Erik Wilde wrote:
> On 2013-09-17 10:22 , Julian Reschke wrote:
>
>> Network Working Group                                          C. Lilley
>> Internet-Draft                                                       W3C
>> Obsoletes: 3023 (if approved)                                  M. Murata
>> Updates: 4289, 6839 (if approved)      International University of Japan
>
> i guess i have the same question here as julian: why would it update
> RFC 6839, instead of just referencing it? if this is just the
> technicality of RFC 6839 referencing this draft, then that would
> probably answer my question.

There's a very specific reason that I feel that this draft needs to
update 6839 and not just reference it: it's changing the Change
controller for +xml from the IETF/IESG to another entity. I don't think
that is something that can/should be done just through a registry change.

    Tony

---

From: Julian Reschke <[email-masked]>
Date: Tue, 12 Nov 2013 14:22:34 +0100

On 2013-11-07 22:44, Murray S. Kucherawy wrote:
> ...

I'm generally happy with the changes, and defer charset/BOM discussions 
to Björn.

With respect to HTTP references, I'd like the spec to:

- normatively reference httpbis
- only informatively reference RFC2616 (if needed at all)

In particular:

>       NOTE: Section 5.3.2HTTPbis [HTTPbis] does not support Accept
>       headers of the form "Accept: */*+xml" and so this header MUST NOT
>       be used in this way.

s/Section 5.3.2HTTPbis [HTTPbis]/Section 5.3.2 of [RFCxxxx]/

>    [HTTPbis]  Fielding, R., "Hypertext Transfer Protocol (HTTP/1.1)
>               [revised]", ietf-httpbis-p2-semantics (work in progress),
>               September 2013.

Substitute with:

    [RFCxxxx]     Fielding, R., Ed. and J. Reschke, Ed., "Hypertext
                  Transfer Protocol (HTTP/1.1): Semantics and Content",
                  draft-ietf-httpbis-p2-semantics-24 (work in progress),
                  September 2013.

also:

>       encoded in base64.  For binary clean transports (e.g.  HTTP
>       [RFC2616]), no content-transfer-encoding is necessary (or even
>       possible, in the case of HTTP) for 7bit, 8bit or binary data.

s/HTTP [RFC2616]/HTTP [RFCxxxx]/

>    As described in [RFC2781], the UTF-16 family must not be used with
>    media types under the top-level type "text" except over HTTP or HTTPS
>    (see section 19.4.2 of [RFC2616] for details).  Hence this example is

s/see section 19.4.2 of [RFC2616]/see Appendix A.2 of [RFCxxxx]/

>    default character sets for the text/xml... types has been resolved by
>    [HTTPbis] changing [RFC2616] by removing the ISO-8859-1 default and
>    not defining any default at all, as well as [RFC6657] updating

s/[HTTPbis]/[RFCxxxx]/

Best regards, Julian

---

From: [email-masked] (Henry S. Thompson)
Date: Tue, 12 Nov 2013 14:18:17 +0000

Julian Reschke writes:

> I'm generally happy with the changes, and defer charset/BOM
> discussions to Björn.
>

I'm happy to do this, but would like confirmation from Murray that
this is acceptable IETF practice, given that HTTPbis is still in Last
Call.

ht

---

From: Julian Reschke <[email-masked]>
Date: Tue, 12 Nov 2013 15:25:02 +0100

> this is acceptable IETF practice, given that HTTPbis is still in Last

HTTPbis is past IETF LC, so in theory it should be ahead of this document.

Best regards, Julian

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 12 Nov 2013 16:11:02 -0800

> this is acceptable IETF practice, given that HTTPbis is still in Last
I think you can just refer to HTTPbis as a work-in-progress, and the RFC
Editor will hold this document until that one is ready.  It doesn't have to
delay our part of the process.

-MSK

---

From: t.petch <[email-masked]>
Date: Thu, 14 Nov 2013 18:32:31 +0000

Sigh

The point I raised has been addressed, which is good.

But, doing the homework I should have done earlier, how does this I-D
interact with RFC6657 which says
"registrations for "text/*" media
   types that can transport charset information inside the corresponding
   payloads (such as "text/html" and "text/xml") SHOULD NOT specify the
   use of a "charset" parameter, nor any default value, in order to
   avoid conflicting interpretations should the "charset" parameter
   value and the value specified in the payload disagree"

Seems to me that this RFC text is being updated and that that should
appear both in rfc-index (and on the IANA website).

And, what about all the application/*+xml registrations in existence,
along with a few message/*+xml and model/*+xml?  How will anyone looking
up these have any idea that this I-D provides guidance for them (which
is what this I-D is claiming)?

I think that you are giving IANA too big a challenge and that more
specific guidance is needed in section 10.

Tom Petch

----- Original Message -----
From: "Murray S. Kucherawy" <[email-masked]>
To: "t.petch" <[email-masked]>; "Erik Wilde" <[email-masked]>;
"Dave Cridland" <[email-masked]>
Cc: "Julian Reschke" <[email-masked]>; "Henry S. Thompson"
<[email-masked]>; "Bjoern Hoehrmann" <[email-masked]>; "IETF Apps
Discuss" <[email-masked]>
Sent: Thursday, November 07, 2013 9:44 PM

---

From: Julian Reschke <[email-masked]>
Date: Thu, 14 Nov 2013 19:50:52 +0100

> appear both in rfc-index (and on the IANA website).

Um, no. That RFC is not updated-

The intent of the sentence you quote was of course to apply to *new* 
registrations. It was not a goal to force a change on existing ones.

> is what this I-D is claiming)?

Is it? What specifically? (Or is this again about new registrations?)

Best regards, Julian

---

From: [email-masked] (Henry S. Thompson)
Date: Fri, 15 Nov 2013 11:04:56 +0000

> appear both in rfc-index (and on the IANA website).

I don't think the spec updates that RFC.  It is consistent with it.
It only recommends (with a 'SHOULD') the use of a charset param. when
there is _known_ to be _no_ encoding decl. in the payload.

> is what this I-D is claiming)?

Such registrations should have referred to 3023 or 6839.  Anyone
following such references will find that those specs have been
replaced/updated by the new spec.

Note that this is precisely what 6839 did, and that's why it is listed
as updated 3023.

> specific guidance is needed in section 10.

I don't think any challenge at all arises.

ht

---

From: [email-masked] (Henry S. Thompson)
Date: Mon, 18 Nov 2013 15:49:07 +0000

> it starts with a UTF-16 Unicode Signature.

It starts with a UTF-32 signature, per both the UNICODE 6.2
documentation (see e.g. Table 16-6 in Chapter 16 [1]) and the XML
specification (see the first table in Appendix F.1 [2]), so the
question of precedence isn't actually addressed by this one.

> . . .

> I think Expat also does not do as you describe.

You're right -- I should have been clearer -- Python's
xml.parser.expat embedding was what I tested, which doesn't access the
HTTP response directly (as is also the case with lxml, even though it
uses libxml2).

> Content-Type header is ignored entirely.

Correct.

So for the two main Python embeddings, and rxp, a BOM does, in
practice, take precedence over the charset param, because they ignore
the response header altogether.  The 'requests' Python package, on the
other hand, does give precedence to a charset param if one is present.

Let me try to summarise, with the help of a test page I've put
together [3], drawing on your examples and some of mine.

That page contains tests of 4 related byte-sequences, each provided as
a separate resource and as a data: URI.  Their text content is
identical (<p>CU+2019U+00E9st domage</p>), but they differ in encoding
(UTF-16LE vs. UTF-32LE) and in the presence or absence of an
appropriate BOM.

There are 6 tests, 3 for each encoding (read 'xx' below as 16 or 32),
as follows:
 label filename  description
 ---------------------------
 b     bxxle     with BOM, served with correct charset param
 bX    bxxleX    with BOM, served with conflicting charset param
 nbX   nbxxleX   without BOM, served with conflicting charset param

In each test the byte-sequence is made available in three forms:
 a data: URI as the src of an iframe;
 an HTTP resource via the src of an iframe;
 an HTTP resource via the href of an a.
 
Here's a tabulation of the results for eight contemporary browsers
which I've tested

  C    Chrome 33.0.1707, Windows 7 (same results from Chrome 30.0, Debian)
  FF   Firefox 25.0, Windows 7, (same results from Firefox 25, Ubuntu)
  IE   Internet Explorer 9.0.22, Windows 7
  O    Opera 12.6, Windows 7 (in auto-detect mode)
  S    Safari 5.1.7, Windows 7 (same results from Safari, MacOs 10)

  Y means renders correctly (as HTML, that is, w/o the <p></p>),
  YT means renders correctly (as plain text, that is, with the <p></p>),
  YX means renders correctly (as 'tree view' XML, including <p></p>),
  E means gives an error,
  N means a blank

The result codes appear twice, the first time for the data: URI (in
its iframe), labelled 'd:', the second for the HTTP resource in its
iframe, labelled 'h:'.

  encoding      16LE        32LE
              d:  h:       d:  h:
label          
          C:   Y  Y    |    Y  Y
   b      FF:  YX YX   |    E  E
          IE:  E  Y    |    E  Y   
          O:   YX YX   |    E  YT   
          S:   Y  Y    |    Y  Y
 ---------------------------------------
          C:   Y  Y    |    Y  Y
  bX      FF:  YX YX   |    E  E
          IE:  E  N    |    E  Y   
          O:   YX YX   |    E  YT   
          S:   Y  Y    |    Y  Y
 ---------------------------------------
          C:   E  E    |    E  E
 nbX      FF:  E  E    |    E  E
          IE:  E  N    |    E  N   
          O:   YX YT   |    E  YT   
          S:   E  E    |    E  E
 ------------------------------------------

There's a lot of data there, but we can simplify the analysis by
reducing it along some dimensions.

First of all, the third condition, nbX, is the uncontroversially
broken case: there is no BOM in the byte-sequence and the charset
parameter conflicts with the actually encoding used.  And five out of
the six browsers tested in fact detect the error and (except for IE in
the resource cases) signal it.  The exception is Opera, which
apparently operates some kind of recovery strategy.  This condition
was included in the test set so that we know that the browsers _can_
detect the conflict, and it appears that they can.

Second of all, the 32-bit encoding column.   Clearly the Firefoxs, IE
and Opera simply don't support 32-bit data: URI decoding, and the
Firefoxes don't handle 32-bit at all.  And, more to the point, the b32
and b32X conditions show identical results, so they give us no insight
into the question of fundamental interest, that is, the relative
priority of BOM and charset parameter.

So we can focus our attention on the b16 and b16X conditions.  Here we
can see that a) Chrome*2 and Safari show identical results (no
surprise here), but also FF*2 and Opera show identical results.
Furthermore, once we notice the IE _never_ accepts data: URIs, then
collapse the result codes into two broader categories: decoding
success vs. failure (S vs. F), the difference between the three result
columns goes away:

  encoding    16LE
label             
          C*2/S:    S  
   b      FF*2/O:   S
          E:        S
 -----------------
          C*2/S:    S  
  bX      FF*2/O:   S 
          E:        F

So, seven out of eight of the browsers that we tested treat a UTF-16
BOM as authoritative when a conflicting charset param is present.

Moving over to APIs and libraries, it's harder to know what's a
more-or-less comprehensive survey.  Here's what I know:

Two out of four C/C++ libraries (rxp, xerces-c) treat a BOM as
authoritative, and this is _not_ because they don't have access to the
charset parameter, but rather because their designers have
specifically chosen not to consider it.  xmllint however makes the
opposite choice, and treats a charset, if present, as authoritative.

Expat is a surprising case -- it provides a parameter which can be
used to pass in a character encoding name, but it will ignore this if
it detects a different encoding in its input byte-stream (tested via
both the Perl and Python embeddings, and confirmed by examining the
source).  So in fact expat does explicitly to treat a BOM as
authoritative.

Similarly xerces-j treats a BOM as authoritative, again based on a
specific choice, even documented in the source code, to ignore the
charset parameter.

Summary: For five 'native' XML processors:

  1 Ignores the charset param altogether: rxp
  3 Explicitly uses in-band information (BOM or simply sniffing) to override
     a charset or other external information (expat, xerces-c, xerces-j)
  1 Treats the charset as authoritative if present (libxml2)
  
So we have 7 out of 8 web browsers (or 4 out of browser 5 code-bases)
which treat a UTF-16LE BOM as authoritative, and 4 out of 5 native
XML APIs which do the same.

(I'm not ignoring your reminder that you have built an add-on to
perl's HTML::Parser module which treats the charset as authoritative,
but since that module does not qualify as a conformant XML parser in
any case, it's not really relevant to 3023bis).

(The php built-in XMLReader extension, which is documented as relying
on 'libxml', behaves similarly to rxp, not libxml2, as far as I can
see).

So, net-net: I think the above tabulation counts as rough consensus.
Certainly it's not a reason _not_ to adopt the approach specified in
section 3.6 of the most recent 3023bis draft [4].

ht

[1] http://www.unicode.org/versions/Unicode6.2.0/ch16.pdf
[2] http://www.w3.org/TR/REC-xml/#sec-guessing-no-ext-info
[3] http://www.ltg.ed.ac.uk/~ht/ov-test/wrapper.html
-- 
       Henry S. Thompson, School of Informatics, University of Edinburgh
      10 Crichton Street, Edinburgh EH8 9AB, SCOTLAND -- (44) 131 650-4440
                Fax: (44) 131 650-4587, e-mail: [email-masked]
                       URL: http://www.ltg.ed.ac.uk/~ht/
 [mail from me _always_ has a .sig like this -- mail without it is forged spam]