Subject: [apps-discuss] I-D Action: draft-ietf-appsawg-xml-mediatypes-05.txt

From: [email-masked]
Date: Tue, 19 Nov 2013 04:09:19 -0800

A New Internet-Draft is available from the on-line Internet-Drafts directories.
 This draft is a work item of the Applications Area Working Group Working Group of the IETF.

	Title           : XML Media Types
	Author(s)       : Henry S. Thompson
                          Chris Lilley
	Filename        : draft-ietf-appsawg-xml-mediatypes-05.txt
	Pages           : 27
	Date            : 2013-11-19

Abstract:
   This specification standardizes three media types -- application/xml,
   application/xml-external-parsed-entity, and application/xml-dtd --
   for use in exchanging network entities that are related to the
   Extensible Markup Language (XML) while defining text/xml and text/
   xml-external-parsed-entity as aliases for the respective application/
   types.  This specification also standardizes the '+xml' suffix for
   naming media types outside of these five types when those media types
   represent XML MIME entities.

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-appsawg-xml-mediatypes

There's also a htmlized version available at:
http://tools.ietf.org/html/draft-ietf-appsawg-xml-mediatypes-05

A diff from the previous version is available at:
http://www.ietf.org/rfcdiff?url2=draft-ietf-appsawg-xml-mediatypes-05

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

---

From: [email-masked] (Henry S. Thompson)
Date: Tue, 19 Nov 2013 12:23:14 +0000

internet-drafts writes:

> A New Internet-Draft is available from the on-line Internet-Drafts
> directories.  This draft is a work item of the Applications Area
> Working Group Working Group of the IETF.
>
> 	Title           : XML Media Types

As for previous drafts, an editors' diff is available, at

  http://www.w3.org/XML/2012/10/3023bis/draft-ietf-appsawg-xml-mediatypes-05_diff.html

This draft contains only very modest changes, which address all but
one outstanding comment.

My reasoning for not changing 3.6 in response to Bjoern Hoehrmann's
objection to the treatment of a BOM as authoritative even in the
presence of a charset parameter [1] is set out in [2].  Martin Duerst
(in another thread) summarises my reasoning, as follows:

  What's most important now is to know what receivers actually
  accept. We are not in a design phase, we are just updating the
  definition ... and making sure we fix problems if there are
  problems, but we have to use the installed base for the main
  guidance

Given the wide review these drafts have received (thanks in particular
to Julian Reschke and Erik Wilde), and the recent endorsement by the
W3C XML Core WG [3], I hope we can move this back to WG Last Call, get
an official review, and take it forward.

ht

[1] http://www.ietf.org/mail-archive/web/apps-discuss/current/msg10810.html
[2] http://www.ietf.org/mail-archive/web/apps-discuss/current/msg10883.html
[3] http://www.ietf.org/mail-archive/web/apps-discuss/current/msg10849.html
-- 
       Henry S. Thompson, School of Informatics, University of Edinburgh
      10 Crichton Street, Edinburgh EH8 9AB, SCOTLAND -- (44) 131 650-4440
                Fax: (44) 131 650-4587, e-mail: [email-masked]
                       URL: http://www.ltg.ed.ac.uk/~ht/
 [mail from me _always_ has a .sig like this -- mail without it is forged spam]

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 19 Nov 2013 16:03:00 +0100

* Henry S. Thompson wrote:
>presence of a charset parameter [1] is set out in [2].

I said that I think this is a serious and problematic change that needs
wide review prior to a two-week Last Call period and that the change is
fine by me provided there is evidence of wide review. It would be easy,
for instance, to ask on a set of mailing lists, say

  * [email-masked]
  * [email-masked]
  * [email-masked]

for endorsements for one part of the proposal which would require XML
implementations to treat

  data:application/xml;charset=utf-32,%FF%FE%00%00...

as malformed UTF-16 encoded document, and if the question is not mis-
leading and there are endorsements by qualified individuals then I'd be
satisfied that this part of the proposal has received adequate review.
You could also point me to messages in archive of this list where such
individuals have provided rationale for their support of this change.

I note that the "Changes from RFC 3023" appendix of the document does
not mention the changes in question and you need to have a good grasp
of the issues to notice them when carefully reading the document. If
the document had carefully explained the changes where appropriate, for
instance, for the issue above, which is just one among several, e.g.

  NOTE: While appendix F.1 of the XML 1.0 Recommendation suggests that
  an initial byte sequence of FF FE 00 00 is indicative of the UTF-32
  character encoding, the rules of this specification require such a
  sequence to be interpreted as a UTF-16 Byte Order Mark followed by a 
  U+0000 character rendering the document malformed. It is therefore
  impossible to use UTF-32 in application/xml, ... entities; a charset
  parameter cannot be used to prevent misinterpretation of documents
  encoded in UTF-32 when using the media types defined in this memo.

then there might be a plausible claim people knew of this change and
some of its implications and signed off on it. The document does not
have that, so I find it reasonable to believe that people are unaware
of these changes, have not considered their implications, and possibly
would object to them if they did. I can't even tell myself whether a
note such as the one above accurately reflects what you mean to propose.

I do not think it would be useful to continue the discussion between
only the two of us, and I find your misrepresentation of my position
most unhelpful. I do appreciate that you have now started to gather
some information on what running code actually does, but let's have a
look at your claims:

  Expat is a surprising case -- it provides a parameter which can be
  used to pass in a character encoding name, but it will ignore this if
  it detects a different encoding in its input byte-stream (tested via
  both the Perl and Python embeddings, and confirmed by examining the
  source).  So in fact expat does explicitly to treat a BOM as
  authoritative.

Well then let us examine the source:

  /* This is what detects the encoding.  ...
  */
  
  static int
  initScan(const ENCODING * const *encodingTable, 
  ...
      case 0xEFBB:
        /* Maybe a UTF-8 BOM (EF BB BF) */
        /* If there's an explicitly specified (external) encoding
           of ISO-8859-1 or some flavour of UTF-16
           and this is an external text entity,
           don't look for the BOM,
           because it might be a legal data.
        */

What does that suggest about expat always looking for a BOM that
overrides anything else? Or how about this claim:

  (I'm not ignoring your reminder that you have built an add-on to
  perl's HTML::Parser module which treats the charset as authoritative,
  but since that module does not qualify as a conformant XML parser in
  any case, it's not really relevant to 3023bis).

The W3C Markup Validator uses my HTML::Encoding module to detect the
character encoding of HTML and XHTML documents. XHTML documents use
whatever rules there are for XML documents to detect the encoding. It
is an implementation of "Given a HTTP response, what is the encoding
of the XML document in it", which is what most of the draft is about.

Inconsistent, ad-hoc, and changing character encoding detection rules
have been a long-standing concern of mine and I have tried to reach out
e.g. in http://www.unicode.org/mail-arch/unicode-ml/y2010-m10/0003.html
to others to improve the situation. It is not too much to ask of the
Applications Area Working Group to do the same.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: "Rushforth, Peter" <[email-masked]>
Date: Tue, 19 Nov 2013 15:31:06 +0000

Henry,

I am not very familiar with BOM technology, so please excuse the beginner nature of this comment.

In section 3.6, your use of the term "in-band" is not correct, I believe.  The band referred to is the communication channel.  When that channel is HTTP, the Content-Type and Accept header values *are* in-band.  

I think the question is "which metadata is _more_ authoritative".  I have read on another (w3c) list the idea that the authoritativeness of metadata is inversely proportional to the square of the distance from the payload, with the idea being that the payload is most authoritative.  I disagree, to the extent that the payload is *not* _metadata_, but data.  If a server transmits an image/png as image/jpeg, it is an error, in my opinion.  So it should be for an incorrectly charset-characterized XML file.  Otherwise, how would a server operator/author ever know if he had a misconfigured server?

HTTP stands for "HyperText" Transfer Protocol.  In contrast to Martin's assertion on another thread, the IETF HTTP protocol is based on hypertext, in that it provides *textual* slots for substitution by clients.  To the extent that a BOM is binary information, it is not a suitable slot for hypertext substitution.  

Do browsers provide the opportunity to *create* a BOM when transmitting XML content?  Browsers are less and less important conveyors of XML information these days.  I have read that (for example) Chrome plans to remove the xml parser from its codebase. Browsers should therefore not be the implementations chosen to prioritize how XML is registered for use on the web.  

Sincerely,
Peter Rushforth

> -----Original Message-----
> From: apps-discuss [mailto:[email-masked]] On 
> Behalf Of Henry S. Thompson
> Sent: November 19, 2013 07:23
> To: [email-masked]; [email-masked]
> Subject: Re: [apps-discuss] I-D Action: 
> draft-ietf-appsawg-xml-mediatypes-05.txt
> 
> internet-drafts writes:
> 
> This draft contains only very modest changes, which address 
> all but one outstanding comment.
> 
> My reasoning for not changing 3.6 in response to Bjoern 
> Hoehrmann's objection to the treatment of a BOM as 
> authoritative even in the presence of a charset parameter [1] 
> is set out in [2].  Martin Duerst (in another thread) 
> summarises my reasoning, as follows:
> 
> Given the wide review these drafts have received (thanks in 
> particular to Julian Reschke and Erik Wilde), and the recent 
> endorsement by the W3C XML Core WG [3], I hope we can move 
> this back to WG Last Call, get an official review, and take 
> it forward.
> 
> ht
> 
> [1] 
> http://www.ietf.org/mail-archive/web/apps-discuss/current/msg1
0810.html
> [2] 
> http://www.ietf.org/mail-archive/web/apps-discuss/current/msg1
0883.html
> [3] 
> http://www.ietf.org/mail-archive/web/apps-discuss/current/msg1
0849.html
> [mail from me _always_ has a .sig like this -- mail without 
> it is forged spam] _______________________________________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss
>

---

From: [email-masked] (Henry S. Thompson)
Date: Tue, 19 Nov 2013 17:42:24 +0000

>   * [email-masked]

Happy to email them, thanks for the suggestion.

> as malformed UTF-16 encoded document

Not so.  Did you see my earlier message [2]?  Both XML and UNICODE
documentation identify FF FE 00 00 as a UTF-32 BOM.

>From F.1 of XML:

    With a Byte Order Mark:

   00 00 FE FF  UCS-4, big-endian machine (1234 order)
   FF FE 00 00  UCS-4, little-endian machine (4321 order)

> and if the question is not mis- leading and there are endorsements
> by qualified individuals then I'd be satisfied that this part of the
> proposal has received adequate review.  You could also point me to
> messages in archive of this list where such individuals have
> provided rationale for their support of this change.
>
> of the issues to notice them when carefully reading the document.

Thanks, good suggestion.

> If the document had carefully explained the changes where
> appropriate, for instance, for the issue above, which is just one
> among several, e.g.
>
>   U+0000 character rendering the document malformed.

I still don't understand where you're seeing this in the spec.
Certainly not my intention.  Would it help to add a para. about UCS-4
to section 4, referring to Unicode 6.2 and/or XML appendix F.1 for
signature detection?

> most unhelpful.

No intentional misrepresentation, I assure you.  See subsequent
message for low-level discussion of our disagreement about expat,
which those who don't care about such things are welcome to skip.

> of the XML document in it", which is what most of the draft is about.

Ah, I only looked at the documentation associated with HTML::Encoding
when I installed it.  So, yes, along with xmllint, this would have to
change if the spec is published as it now stands.

> Applications Area Working Group to do the same.

I absolutely agree -- increasing interoperability in this space is
obviously of great importance.  Given the wide disparity among
available implementations, what I've proposed not only endorses the
existing majority among implementations so far as I can see but also
is simple to understand, explain and implement.

I'll draft a proposed message to the above lists tomorrow and
send it to you for review.

ht

---

From: [email-masked] (Henry S. Thompson)
Date: Tue, 19 Nov 2013 17:44:36 +0000

Bjoern Hoehrmann writes:

> I do appreciate that you have now started to gather some information
> on what running code actually does, but let's have a look at your
> claims:
>

Look further down, and you will see that the actual code which follows
is

      if (state == XML_CONTENT_STATE) {
        int e = INIT_ENC_INDEX(enc);
        if (e == ISO_8859_1_ENC || e == UTF_16BE_ENC
            || e == UTF_16LE_ENC || e == UTF_16_ENC)
          break;
      }

At the beginning of documents, state != XML_CONTENT_STATE, so the
UTF-8 BOM is only ignored for external entities, not for documents as
such.  It seems to me that treating documents and external entities
differently is explicitly contrary to the XML specification, which
clearly states:

  "If the replacement text of an external entity is to begin with the
  character U+FEFF, and no text declaration is present, then a Byte
  Order Mark MUST be present, whether the entity is encoded in UTF-8
  or UTF-16."

Looking further up, we see

    case 0xFEFF:
      if (INIT_ENC_INDEX(enc) == ISO_8859_1_ENC
          && state == XML_CONTENT_STATE)
        break;

i.e. that UTF-16 BOMs are only ignored if a) you're looking at an
external entity and b) [some encoding] is ISO_8859_1_ENC

[some encoding] -- I spent an hour trying to figure out what encoding
the parser for external entities is passed and how to affect it so as
to cause the 'break' above to happen in a way that would cause an
error if the external entity has a UTF-8 or UTF-16 BOM, but I
couldn't.  I invite people to retrieve [3]--[6] and see if you can
provoke an error by any form of invocation of expat 2.1.0.  I have not
been able to -- for example

  > xmlwf -m -d /tmp -e ISO-8859-1 -x wrap-local-b8.xml

perfectly happily produces

  <div><p>C’ést domage</p></div>  

despite the external entity beginning with a UTF-8 BOM.

> overrides anything else?:

As just noted, in practice, it seems that it does -- for reasons I
understand for top-level documents, and reasons I don't understand for
external entities.

Of course expat will never manage to process a UTF-32-encoded document
with or without a BOM or correct charset, because expat doesn't
implement UTF-32 _at all_, but that's a separate issue, surely.

ht

[3] http://www.ltg.ed.ac.uk/~ht/ov-test/wrap-local-b16le.xml
[4] http://www.ltg.ed.ac.uk/~ht/ov-test/wrap-local-b8.xml
[5] http://www.ltg.ed.ac.uk/~ht/ov-test/b16le.xml
[6] http://www.ltg.ed.ac.uk/~ht/ov-test/b8.xml
-- 
       Henry S. Thompson, School of Informatics, University of Edinburgh
      10 Crichton Street, Edinburgh EH8 9AB, SCOTLAND -- (44) 131 650-4440
                Fax: (44) 131 650-4587, e-mail: [email-masked]
                       URL: http://www.ltg.ed.ac.uk/~ht/
 [mail from me _always_ has a .sig like this -- mail without it is forged spam]

---

From: "Martin J. Dürst" <[email-masked]>
Date: Wed, 20 Nov 2013 20:07:57 +0900

Hello Henry, others,

I'm sorry this is very late, but I managed to review most of the -04 
draft. I checked -05, so the comments below apply to -05.

First, let me say that my comments in the first review (months ago, 
mostly against closely interleaving spec parts and historical notes) 
have been addressed very well.

This time, the comments are much more on details.

Copyright notice: Given the long history of this draft, I'd guess that 
this document needs the following addition in the copyright:

    This document may contain material from IETF Documents or IETF
    Contributions published or made publicly available before November
    10, 2008.  The person(s) controlling the copyright in some of this
    material may not have granted the IETF Trust the right to allow
    modifications of such material outside the IETF Standards Process.
    Without obtaining an adequate license from the person(s) controlling
    the copyright in such materials, this document may not be modified
    outside the IETF Standards Process, and derivative works of it may
    not be created outside the IETF Standards Process, except to format
    it for publication as an RFC or to translate it into languages other
    than English.

(This can easily be produced with a setting on some attribute in the XML 
source.)

Section 3 and Section 8: These sections have more than a page of text 
before the first subsection. I suggest to add one or more additional 
subsection titles at the start or very close to the start of the section 
for better structuring.

Section 3 says:

   document entities  The media types application/xml or text/xml MAY be
       used.

First, it would be good to have some syntactic delimiter (colon maybe) 
between "document entities" and the rest. Same for the other items.

Second, RFC 2119 defines MAY as follows: "This word, or the adjective 
"OPTIONAL", mean that an item is truly optional." This is quite a bit 
misleading in the sentence above. Using application/xml or text/xml for 
XML document entities is the default case, not just an optional option. 
I suggest something like "The media types application/xml or text/xml, 
or a more specific media type, SHOULD be used." (A should without 
additional qualification is probably too strong.)

    external parsed entities  The media types application/xml-external-
       parsed-entity or text/xml-external-parsed-entity SHOULD be used.
       The media types application/xml and text/xml MUST NOT be used
       unless the parsed entities are also well-formed "document
       entities" and are referenced as such.

The last clause ("and are referenced as such") is confusing. Stuff is 
just served or sent; on the server side, it's unclear how it's being 
referenced, and so such a condition does not make sense operationally.

Note starting with:
       Note that [RFC3023] (which this specification obsoletes)
       recommended the use of text/xml and text/xml-external-parsed-
       entity for document entities and external parsed entities,

Because of the indenting, it looks as if this note only applies to the 
immediately preceding item ("external parameter entities"), but 
content-wise, it seems to apply more generally. The note should be 
outdented (if that's possible) or should be moved to another place where 
it's less confusing to the reader as to what it applies to.

Para starting with:
    Compared to [RFC2376] or [RFC3023], this specification alters the
    charset handling of text/xml and text/xml-external-parsed-entity,

This is very long, in particular the first sentence. A very easy first 
step towards improvement would be to use a period before the "however", 
and change "however" to "However". Any additional untangling would be 
appreciated, too. Also, "for the text/xml... types" should be changed to 
"for types with a top-level media type text". (several instances)

Last paragraph before Section 3.1: It was unclear what exactly the spec 
tried to say here. I suggest to add a sentence at the end, e.g. "Such 
processing is not specified in this document."

Section 3.1, Encoding considerations (and elsewhere): The term "charset 
encoding" shows up. This is an unfortunate mixture of terminology. MIME 
has the "charset" parameter, and XML has the "encoding" 
pseudo-attribute, but this doesn't mean that these two words should be 
combined just like this. Also, this isn't used uniformly through the 
spec, e.g. there are things like "ASCII-compatible character sets" (see 
also http://www.w3.org/MarkUp/html-spec/charset-harmful.html).

I suggest, in Section 2, to shortly talk about the fact that MIME has 
the "charset" parameter, and XML has the "encoding" pseudo-attribute, 
and then use a single term. I'd personally suggest "character encoding" 
(see e.g. RFC 3986), but I'd be happy with any term that has been used 
widely already.

Also, we have "7bit or 8bit data, for example data with charset encoding 
UTF-8 or US-ASCII". In general speach, a chiasmus is something nice, but 
it's generally only confusing in specs, so I'd change this to "7bit or 
8bit data, for example data with charset encoding US-ASCII or UTF-8".

Section 3.1, Applications that use this media type: There is a missing 
"and" but a superfluous comma : "is supported by a wide range of generic 
XML tools (editors, parsers, Web agents, ...)*,* *and* generic and 
task-specific applications." Probably, reordering makes this easier to 
read: "is supported by generic and task-specific applications and a wide 
range of generic XML tools (editors, parsers, Web agents, ...)."

Section 3.2: Text/xml Registration
This is defined as an "alias", but the Media Type registry (e.g. in 
contrast to the charset registry) doesn't know the concept of an alias. 
So this should be reworded, e.g. saying that the registration 
information is the same. This also applies to Section 3.4.

Section 3.3, Encoding considerations (and other items): There are two 
"as" prepositions in short succession. What about "Same as 
application/xml, see Section 3.1." or some such?

Section 3.3, Interoperability considerations:
                                                  Identifying XML
       external parsed entities with their own content type should
       enhance interoperability of both XML documents and XML external
       parsed entities.
Lowercase "should" SHOULD be avoided! (there are other cases, too) I 
suggest to change to "will", or just say "enhances".

Section 3.6:
    XML MIME producers are RECOMMENDED to provide means for XML MIME
    entity authors to control the supply of charset parameters for their
    entities, for example by enabling user-level configuration of
    filename-to-Content-Type-header mappings on a file-by-file or suffix
    basis.
"control the supply" reads as if these charset parameters were in ample 
or short supply. I suggest to replace "supply" with "presence" or 
"presence or absence".

Section 3.6:
It may be helpful to create (sub)subsections for producers and 
consumers, because that's what many readers of the spec will look for.

Section 3.6, para starting with (and the following citation and para):
    When a charset parameter is specified for an XML MIME entity, then
This is way too lengthy and complicated. The first sentence is almost 
six lines long. This is another case of mixing history and justification 
with the hard facts, and should be untangled.

"Section 4.3.3 of the [XML] specification": Please fix this to "Section 
4.3.3 of [XML]" (as in other locations) or "Section 4.3.3 of the XML 
specification [XML]".

"When MIME producers conform to the requirements on them stated above,"
"on them" is redundant and should be removed.

Section 4: I'm not sure why this is a separate section, as the content 
is tightly related to Section 3.6. At the minimum, I suggest moving the 
stuff about BOMs from 3.6 to 4. A better solution would be to promote 
3.6 to a section, and include the current section 4 in there (with 
appropriate additional subsections as suggested above).

                                       byte order mark (BOM), which is a
    hexadecimal octet sequence 0xFE 0xFF (or 0xFF 0xFE, depending on
    endianness)
A byte order mark is a character, not an octet sequence. Also, better 
say which endianness is which. This would result in
                                       byte order mark (BOM), which
    appears as the hexadecimal octet sequence 0xFE 0xFF (big-endian)
    or 0xFF 0xFE (little-endian)
The change from "is" to "appears as" is also needed for UTF-8.

    Applications which convert XML into "utf-8" SHOULD add a BOM after
    conversion is complete.
There are two problems there:
1) "after conversion is complete", if taken literally, would lead to 
very efficient implementations (adding three bytes at the start of a 
long file). This clause should therefore be removed.
2) There is absolutely no need for a SHOULD. SHOULDs are only used when 
otherwise, there are interoperability problems, but XML in UTF-8 without 
a BOM 'should' not have any such problems. MAY seems much more 
appropriate here.

Section 5: "IRI" is mentioned without a reference. The reference was 
dropped between -04 and -05 because it looked as if it wasn't needed, 
but it should be put back in (with "Dueerst" fixed to "Duerst").

    A registry of XPointer schemes [XPtrReg] is maintained at the W3C.
    Document authors SHOULD NOT use unregistered schemes.  Scheme authors
    SHOULD register their schemes ([XPtrRegPolicy] describes requirements
    and procedures for doing so).
I fully agree with the SHOULDs here, but they don't belong in this spec.

    When a URI has a fragment identifier, it is encoded by a limited
    subset of the repertoire of US-ASCII [ASCII] characters, as defined
    in [RFC3986].
I'm not sure what this helps here. A pointer to the relevant parts of 
the XPointer spec(s) would be better, because some issues with respect 
to XPointer encoding in URIs (and IRIs) can be rather tricky.

Section 6:
Again very complicated language. I'd shorten the background information 
drastically, e.g. as follows (replacing the first *two* paragraphs of 
Section 6):
    An XML MIME entity of type application/xml, text/xml,
    application/xml-external-parsed-entity or
    text/xml-external-parsed-entity MAY use the xml:base attribute, as
    described in [XMLBase], to establish a base URI for that entity
    (see Section 5.1 of [RFC3986]).

Section 8: A Naming Convention...
I suggest removing the "a". In RFC 3023, this was in many ways just a 
trial, and so "a" was appropriate. Today, this doesn't have to be 
stressed anymore, and there are no other naming conventions for 
XML-Based Media Types.

"pattern '*/*+xml'": This is shell notation applied to something else 
than file names. It's close to the syntax allowed in an HTTP Accept: 
header, but (as correctly noted in the draft) not the same. It should be 
obvious to many readers, but it would be better if it were clearly 
explained.

       When an XML-based media type is restricted to UTF-8, it is not
       necessary to introduce the charset parameter.  "UTF-8 only" is a
       generic principle and UTF-8 is the default of XML.
I'm not sure what ""UTF-8 only" is a generic principle" is referring to. 
I guess it refers to the idea that using UTF-8 only for certain use 
cases on the Internet simplifies things a lot and is therefore a good 
idea. I fully agree. But a) this should be clearer, and b) it should be 
separated from "UTF-8 is the default of XML", because the former is a 
justification for the antecedent of the previous sentence (which I don't 
think is actually necessary), whereas the later is a justification for 
the conclusion made in that sentence. So in order of decreasing preference:
1) Remove ""UTF-8 only" is a generic principle and"
2) Split the second sentence into two, explaining the "generic 
principle" in slightly more detail.

"Similarly, media subtypes that do not represent XML MIME...": I don't 
see any similarity to what comes before. If a connective is really 
needed, I'd use "Conversely", but the best solution would be to do 
without connectives altogether.

8.1: "Referencing": Please use a slightly longer subsection title to 
make it easier for readers to understand what this subsection is talking 
about. Maybe "Registration Template Details"?

"Registrations for new XML-based media types under top-level types"
Please remove "under top-level types". It doesn't add any information.

8.2, Reference:  Replaces "This specification" with "RFC XXXX".
This makes the template more portable. There are more occasions that 
would benefit from the same change.

8.2, Fragment identifier considerations:
The two provisions "they MAY restrict the syntax to a specified subset 
of schemes" and "They MAY further require support for other registered 
schemes" look okay, but they leave open the question of what's the 
default. As far as I understand, barenames and element scheme pointers 
are the (bare :-) minimum, and so the first provision seems unnecessary. 
Removing that provision (and removing the "further" in the second 
provision) should make it clear that the minimum is the default.

             For fragment identifiers matching the syntax defined in
             [XPointerFramework], where the fragment identifier does
             _not_ resolve per the rules specified there, then process as
             specified in "xxx/yyy+xml";
Is this the case of an unregistered XPointer scheme? If yes, it would be 
good to mention here (not with MUSTard) that this is a bad idea. If not, 
I don't understand what case this addresses.

Section 9:
The last para before subsection 9.1 should be moved to the start of this 
section.

                            the charset portion, if any, of the value of
    the MIME Content-type header
I'd prefer to keep the full Content-type header, as in the examples in 
RFC 3023. Why? I think something like
Content-type: application/xml; charset="UTF-8"
is easier to read than
Content-type charset: charset="UTF-8"
Either this can use different types in the different examples, or use 
application/xml throughout. I'd personally prefer the later.

                            and the XML MIME entity may contain other
    data in addition to the XML declaration;
The 'may' here is misleading a) because lower-case 'may' SHOULD be 
avoided and b) because (except for an XML declaration) there 'may' 
indeed be no data, but that would be exceedingly rare. So I would change 
this to something like "and the XML MIME entity will contain other data 
in addition to the XML declaration (or might be empty);",
where the parenthetical in my opinion isn't even necessary.

Section 9.1:

Why is there no <?xml version="1.0">, similar to 9.2? Why are there no 
cases without any XML declaration (difficult to represent in the current 
way but very realistic)?

    If sent using a 7-bit transport (e.g., SMTP[RFC0821]), the XML MIME
    entity MUST use a content-transfer-encoding of either quoted-
    printable or base64.
I may be wrong here, but in my understanding, it would be possible to 
send pure US-ASCII labeled as UTF-8 through 7-bit transport (i.e. what 
defines the transport is the actual data and not the potential bit 
patterns allowed by the charset).

Section 9.1 and 9.2: The 7-bit/8-bit/binary considerations repeat what's 
already said in 3.1, so they should be replaced with a pointer to that 
section.

9.3: The title includes "and 8-bit MIME entity", but the considerations 
apply equally well e.g. to encoding="iso-2022-jp", which is 7-bit.

9.5: This is lengthy. Remove "and UTF-8 Entity" from the title, and 
simply say that this is interpreted as UTF-8 because that's the default 
for XML.

9.6 "Observe that the BOM does not exist." -> "Observe that the BOM 
isn't present." or "Observe that there is no BOM." (the BOM exists as 
well as any other character :-)

That's as far as I got. More maybe over the weekend or next week, sorry.

Regards,   Martin.

---

From: [email-masked] (Henry S. Thompson)
Date: Tue, 26 Nov 2013 18:13:35 +0000

Martin J. Dürst writes:

> . . .
> Section 3.1, Encoding considerations (and elsewhere): The term
> "charset encoding" shows up. This is an unfortunate mixture of
> terminology. MIME has the "charset" parameter, and XML has the
> "encoding" pseudo-attribute, but this doesn't mean that these two
> words should be combined just like this. Also, this isn't used
> uniformly through the spec, e.g. there are things like
> "ASCII-compatible character sets" (see also
> http://www.w3.org/MarkUp/html-spec/charset-harmful.html).

Yes, alas, this area of the spec is . . . inconsistent at best.  Some
of this was inherited from 3023, some added more recently.

The problem begins as far back as *2. Notational Conventions* [1]:

  As defined in [RFC2781] (informative), the three character sets
  "utf-16", "utf-16le", and "utf-16be" are used to label UTF-16
  text. In this specification, "the UTF-16 family" refers to those
  three character sets. By contrast, the phrases "utf-16" or UTF-16 in
  this specification refer specifically to the single charset
  "utf-16".

I'm thinking of changing this to read more along the following lines:

  XML (in an XML or Text declaration) and MIME (in a charset
  parameter) use a common vocabulary to identify the coded character
  set (CCS) and character encoding scheme (CES) of a particular byte
  stream.  The CCS of all XML documents is [UNICODE], so in practice
  all that is at issue is the CES.  A CES determines encoding form
  and, where relevant, serialization. XML requires support for the
  UTF-8 and UTF-16 encoding form, and allows others.  In this
  specification, UTF-8, UTF-16 and UTF-32 refer to the encoding forms,
  independent of serialisation.

  As documented in [RFC3629], [RFC2781] and [UNICODE] section 3.9, the
  MIME charset "utf-8" is used to label byte streams which encode
  UNICODE text using the UTF-8 encoding form, the three MIME charsets
  "utf-16", "utf-16le", and "utf-16be" are used to label byte streams
  which encode UNICODE text using the UTF-16 encoding form, and the
  three MIME charsets "utf-32", "utf-32le" and "utf32-be" likewise for
  UTF-32.

  In this specification we will use the phrases "charset parameter"
  and "encoding declaration" to refer to whatever MIME charset is
  specified by a MIME charset parameter or XML encoding declaration
  respectively, reserving the phrase "character encoding" for the
  encoding form and, where relevant, serialization, actually used in a
  particular XML MIME entity.

and then make further changes throughout along those lines.  What do
you think?

ht

[1] http://tools.ietf.org/html/draft-ietf-appsawg-xml-mediatypes-05#section-2

---

From: "Martin J. Dürst" <[email-masked]>
Date: Wed, 27 Nov 2013 19:07:16 +0900

Hello Henry,

> I'm thinking of changing this to read more along the following lines:

This is quite a good start, but has some inaccuracies (and loses the 
distinction between "the UTF-16 family" and "utf-16" in the text above, 
which is quite relevant (because of the special treatment that that XML 
gives "utf-16").

Also, both CCS and CES are used for a single character encoding. XML 
does not have a single CCS, but a single 'document character set'. So I 
don't think any text about CCS or CES is needed.

So I have tried a re-write, but I'm not totally sure this will work for 
all of the document. Please check.

    Both XML (in an XML or Text declaration using the encoding pseudo-
    attribute) and MIME (in a Content-Type header field using the
    charset parameter) use a common set of labels [IANA-charsets] to
    identify the MIME charset (mapping from byte stream to character
    sequence [RFC2978]).

    In this specification we will use the phrases "charset parameter"
    and "encoding declaration" to refer to whatever MIME charset is
    specified by a MIME charset parameter or XML encoding declaration
    respectively, reserving the phrase "character encoding" for MIME
    charset actually used in a particular XML MIME entity.

    XML requires support for the MIME charsets UTF-8 and UTF-16, and
    allows others. In this specification, the term "UTF-16" is restricted
    to the MIME charset of the same name as defined in [RFC2781], whereas
    the term "the UTF-16 family" refers to "utf-16", "utf-16le", and
    "utf-16be".

    [IANA-charsets]
    http://www.iana.org/assignments/character-sets/character-sets.xhtml

That should do it. Regards,   Martin.

---

From: [email-masked] (Henry S. Thompson)
Date: Thu, 28 Nov 2013 16:30:09 +0000

Martin J. Dürst writes:

> So I have tried a re-write, but I'm not totally sure this will work
> for all of the document. Please check.

Thanks!

>    identify the MIME charset (mapping from byte stream to character

Perfect.

>    charset actually used in a particular XML MIME entity.

OK.

>    the term "the UTF-16 family" refers to "utf-16", "utf-16le", and

So this is, I think, not how I want to go.  Rather, more like

     UNICODE defines three "encoding forms", which are independent of
     serialization, namely UTF-8, UTF-16 and UTF-32.  This
     specification follows this precedent.  In particular, note that
     UTF-16-encoded XML documents may be serialised into MIME entities
     in one of two ways: either big-endian, labelled (optionally)
     "utf-16" or "utf-16be", or little-endian, labelled (optionally)
     "utf-16" or "utf-16le".  As UTF-8 can only be serialized in one
     way, the only possible label for UTF-8-encoded documents when
     serialised into MIME entities is "utf-8".

ht