Subject: [apps-discuss] Thoughts on text/* encoding defaults

From: Julian Reschke <[email-masked]>
Date: Mon, 06 Jun 2011 14:42:15 +0200

Hi there.

In Prague, we had a few hallway conversations with respect to the 
default encoding of text/* media types.

Below are my notes (references to the relevant spec sections, 
information about a recent change in HTTPbis, and a rough proposal about 
how to proceed).

Best regards, Julian

-- snip --

1) RFC 2046 says that the default is US-ASCII

"Note that the character set used, if anything other than US- ASCII, 
must always be explicitly specified in the Content-Type field." -- 
<http://greenbytes.de/tech/webdav/rfc2046.html#rfc.section.4.1.2.p.18>

2) RFC 2616 says it's ISO-8859-1

"The "charset" parameter is used with some media types to define the 
character set (Section 3.4) of the data. When no explicit charset 
parameter is provided by the sender, media subtypes of the "text" type 
are defined to have a default charset value of "ISO-8859-1" when 
received via HTTP. Data in character sets other than "ISO-8859-1" or its 
subsets MUST be labeled with an appropriate charset value. See Section 
3.4.1 for compatibility problems." -- 
<http://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.3.7.1.p.4>

3) For text/xml, RFC 3023 says it's US-ASCII, no matter what 2616 says :-)

"Conformant with [RFC2046], if a text/xml entity is received with the 
charset parameter omitted, MIME processors and XML processors MUST use 
the default charset value of "us-ascii"[ASCII].  In cases where the XML 
MIME entity is transmitted via HTTP, the default charset value is still 
"us-ascii".  (Note: There is an inconsistency between this specification 
and HTTP/1.1, which uses ISO-8859-1[ISO8859] as the default for a 
historical reason.  Since XML is a new format, a new default should be 
chosen for better I18N.  US-ASCII was chosen, since it is the 
intersection of UTF-8 and ISO-8859-1 and since it is already used by 
MIME.)" -- <http://tools.ietf.org/html/rfc3023#section-3.1>

The problem

Recipients do not implement this; they take the absence of encoding 
information as indicator to inspect the payload; this is at least true 
for text/xml and text/html (see 
<http://www.w3.org/TR/REC-xml/#sec-guessing> and 
<http://www.w3.org/TR/2011/WD-html5-20110405/parsing.html#determining-the-character-encoding>)

Current development: HTTPbis, P3 has dropped drop the default and 
delegate to the relevant media type definitions (see 
<http://trac.tools.ietf.org/wg/httpbis/trac/ticket/20>, 
<http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p3-payload-14.html>).

Left to do:

a) Revise RFC 2046; allow text/* types that carry encoding information 
inline to do the expected thing (overriding the US-ASCII default); warn 
against doing so in new registrations (recommend to only support UTF-8, 
and require to always explicitly include the charset parameter, such as 
text/vcard is going to do it?)

b) Revise RFC 3023 to delegate text/xml charset defaults to revision of 
2046?

Best regards, Julian

---

From: Dzonatas Sol <[email-masked]>
Date: Mon, 06 Jun 2011 11:56:49 -0700

On 06/06/2011 05:42 AM, Julian Reschke wrote:
> inline to do the expected thing (overriding the US-ASCII default); 
> warn against doing so in new registrations (recommend to only support 
> UTF-8, and require to always explicitly include the charset parameter, 
> such as text/vcard is going to do it?)
>
> b) Revise RFC 3023 to delegate text/xml charset defaults to revision 
> of 2046?
>

This has probably been said before by someone else, yet there is some 
reason about the above action which makes me concerned when we don't 
first assume use of "us-ascii" that there may be loss of compatibility 
with 7bit communications (and compression algorithms), especially in 
hardware that doesn't require updates based on known standards. I know 
UTF-8 includes the literals, yet if we strip the 8th bit on 
unicoded-to-ascii escape sequences then is the media content still 
compatible (even size wise before and after compression)? Or would this 
cause unknown error correction where UTF-8 is an explicit replacement of 
"us-ascii". I think that there is the significant point to 7bit being 
"must" by that 7bit mode being specifically implied by "ascii" because 
the 8th bit in UTF-8 is not "human readable" in regards to plain XML. To 
only support UTF-8 means one more bit is being added to the 7bit standard.

I do recognize you want to resolve the conflict for further support with 
ISO & I18N. I think what you want why some tried to progress "code 
pages" in context.

-- 
--- https://twitter.com/Dzonatas_Sol ---
Web Development, Software Engineering, Virtual Reality, Consultant