Subject: Re: [apps-discuss] Priority of in-band vs. out-of-band encoding information

From: [email-masked] (Henry S. Thompson)
Date: Fri, 04 Oct 2013 18:59:41 +0100

Trying again with a simpler question:

Is anyone aware of any media-type registration or other spec. which
discusses the status of a Mime entity which
 a) begins with a BOM;
 b) has an associated charset param which is incompatible with that
    BOM
?

ht
-- 
       Henry S. Thompson, School of Informatics, University of Edinburgh
      10 Crichton Street, Edinburgh EH8 9AB, SCOTLAND -- (44) 131 650-4440
                Fax: (44) 131 650-4587, e-mail: [email-masked]
                       URL: http://www.ltg.ed.ac.uk/~ht/
 [mail from me _always_ has a .sig like this -- mail without it is forged spam]

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Fri, 04 Oct 2013 20:25:58 +0200

> b) has an associated charset param which is incompatible with that

A few individuals employed by browser vendors have decided some time ago
that Unicode signatures should override higher level meta data; that is
the opposite of what had been unanimity before then. This should already
be implemented in some of their products for formats like XML, CSS, and
more. What the XML specification or RFC3023bis or the IETF might have to
say on the matter is of no concern to them. I find the reasoning behind
their decision unpersuasive and dangerous, but I have not seen anyone
else having a problem with it, neither technically nor procedurally, not
as far as I can remember anyway. The CSS specifications are an example
of the kind you seek. My RFC 4329 would be another (with caveats). An
example for a format that ostensibly and explicitly disallows BOMs is
JSON as defined in RFC 4627. Many old formats let the charset override
inline markers by virtue of the inline markers being added later, HTML
4.01 for instance does discuss BOMs, but limits their effect to content
labeled as "UTF-16" without indication of the byte order. Historically,
you probably encounter different approaches after the XML specification
got modified to clarify the status of the UTF-8 Unicode signature.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Ned Freed <[email-masked]>
Date: Sat, 05 Oct 2013 16:01:37 -0700 (PDT)

> In reviewing the relationship between
> draft-ietf-appsawg-xml-mediatypes and the XML specification itself, I
> was reminded that the XML spec. defers to "RFC3023 or successor" with
> respect to the relative priority of the charset parameter
> and any available document-internal information about character
> encoding (BOM and/or the XML encoding declaration).

> Empirical evidence suggests that wrt both XML and HTML media types,
> web browsers give priority to a BOM. That is, if the body of an HTTP
> response begins with a BOM, it is treated as authoritative and any
> charset parameter in a Content-Type header is ignored.  The current
> draft of the HTML5 spec. makes this explicit.

> Is anyone aware of any existing media type registration or other RFC
> which addresses this issue explicitly?

No, not really. There may have been some talk about having what became RFC 6657
go there, but it never happened.

I will note that when it comes to sniffing and UTF-8/UTF-16, I regard this
entire issue as "mostly harmless". A document that begins with a 16-bit BOM
cannot possibly be valid utf-8, and the chances that a UTF-16 document without
a BOM will parse as vaid UTF-8 are very low. So this sort of guessing tends to
work, although whether or not it's actually useful is a different question.

In my experience the place where making guesses about charsets is highly
problematic isn't with XML and isn't with UTF-8 or UTF-16. It used to be the
attempts to tell the difference between the various iso-8859-* variants and
friends, these days the problems have moved to the the CJK space, where there
are now multiple variant charsets that have been built on top iso-2022-jp,
euc-jp, and so on that add various Unicode characters to these charsets in
incompatible ways. (I used to think that GB18030 (basically GB2312 extended to
cover all of Unicode) was a dumb thing to do, but given what's happened on the
Japanese side of things I've decided I was wrong.)

				Ned