Subject: [apps-discuss] draft-ietf-appsawg-xml-mediatypes vs. JSON and BOM and UTF-8

From: Larry Masinter <[email-masked]>
Date: Tue, 7 Jan 2014 19:01:31 +0000

I have some discussion topics for draft-ietf-appsawg-xml-mediatypes which I will send out one-by-one.
This first one was discussed but here I'm making some specific suggestions.

It cannot be right that everyone specifying a text-based media type should have to go through the process of deciding, for themselves, independently, how to decide between conflicting sources of information about charset, coming from an initial BOM, from external metadata, from any supplied "charset" parameter of the media type, and from internal embedded metadata such as found in XML.

If the future is UTF-8, UTF-8, UTF-8, then the two documents should say so, right at the beginning.

http://tools.ietf.org/html/draft-ietf-appsawg-xml-mediatypes-06#section-3
      Going forward, XML producers SHOULD use UTF-8 exclusively, without any BOM. For compatibility with existing implementations, processing rules are given.

Then section 3.1 XML MIME producers
    "XML MIME producers" generating a MIME body (who SHOULD encode the XML as UTF-8 without a BOM, and SHOULD also include a UTF-8 encoding declaration)
And
   'XML MIME wrappers" those that are not re-encoding an XML body and just want to deliver it via MIME
   who can follow the guidelines in 3.1. 
   I don't really understand what a "XML-unaware MIME producer" is. If they're "unaware", why are they reading this spec?

Larry
--
http://larry.masinter.net

-----Original Message-----
From: "Martin J. Dürst" [mailto:[email-masked]] 
Sent: Saturday, December 14, 2013 4:26 AM
To: Tim Bray
Cc: Larry Masinter; IETF Apps Discuss
Subject: Re: [apps-discuss] Unicode BOM detection in JSON vs. draft-ietf-appsawg-xml-mediatypes

On 2013/12/14 7:20, Tim Bray wrote:
> I actually think 4627bis is perfectly clear. You can use UTF-{8/16/32} in
> theory; in practice only UTF-8 works over the wire; you MUST NOT emit a BOM
> but if some dork does, it’s OK to forgive. There’s no charset parameter on
> the media type.
>
> If we were doing XML in 2013, I think we’d probably end up at the same
> place. Note that the original XML spec was written in ISO-8859-1 :)

Yes. On the surface, some of the issues in JSON and XML may look 
similar, but in actual practice, they are quite different. So a common 
model or a common spec doesn't make sense.

We have had some kind of common model from around 1996 or so, which was 
that the outside (charset parameter) has precedence. This was based on 
the assumption that there were transcoding proxies than didn't look 
inside the document, and on statements from Netscape (when it was still 
the dominant browser) that this was what they did.

It turned out that there were few if any transcoding proxies, and 
external information didn't get preserved well in a file system, and so 
the practice moved more and more to give priority of internal 
information over external.

In ietf-appsawg-xml-mediatypes, we can see one specific example of what 
this history has resulted in: carefully defined specific steps, but not 
too pretty an overall picture.

The model of the future is much simpler: UTF-8, UTF-8, UTF-8. Once 
everything is UTF-8, the BOM will die a quiet death, too.

Regards,   Martin.

> On Fri, Dec 13, 2013 at 2:10 PM, Larry Masinter<[email-masked]>  wrote:
>
>>   Compare
>> http://tools.ietf.org/html/draft-ietf-json-rfc4627bis-09#section-8.1
>>
>> and
>> http://tools.ietf.org/html/draft-ietf-appsawg-xml-mediatypes-06#section-3
>>
>>
>>
>> it’s hard to tell exactly what either is saying, much less whether they’re
>> saying
>>
>> the same thing.   I don’t want to slow down either of these, but some work
>> on
>>
>> a model for how MIME types should deal with the relationship between
>>
>>
>>
>> -          Charset parameter in content-type
>>
>> -          Possible BOM maker
>>
>> -          Other in-content character set indication (like in XML)
>>
>> -          Other protocol sources of charset information
>>
>>
>>
>> JSON, XML, HTML, IRIs, elements based on those, and likely many other
>>
>> protocol elements are defined using BNF or other syntactic description
>>
>> techniques over the space of sequence-of-Unicode-characters
>>
>> (sequence-of-Unicode-code-point).
>>
>>
>>
>> Has there been any attempts to create a separate BCP that JSON and XML
>>
>> Could (in the future) reference? The JSON working group spent only
>>
>> a few thousand emails on this.
>>
>>
>>
>> Larry
>>
>> --
>>
>> http://larry.masinter.net
>>
>>
>>
>>
>>
>
>
>
> _______________________________________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss

---

From: Patrik Fältström <[email-masked]>
Date: Tue, 7 Jan 2014 20:22:55 +0100

> If the future is UTF-8, UTF-8, UTF-8, then the two documents should say so, right at the beginning.

+1000

   Patrik

---

From: [email-masked] (Henry S. Thompson)
Date: Wed, 08 Jan 2014 10:00:10 +0000

> which I will send out one-by-one.

Great, thanks.

> This first one was discussed but here I'm making some specific suggestions.

I'll respond later to the larger issue here, but not for a few days,
TAG meeting this week.

> ...

>     "XML MIME producers" generating a MIME body (who SHOULD encode
>     the XML as UTF-8 without a BOM, and SHOULD also include a UTF-8
>     encoding declaration)
> And
>    'XML MIME wrappers" those that are not re-encoding an XML body
>    and just want to deliver it via MIME who can follow the
>    guidelines in 3.1.
> I don't really understand what a "XML-unaware MIME producer" is. If
> they're "unaware", why are they reading this spec?

I think this makes sense -- it means any MIME producer who respects
media type registrations as far as possible, including this one, but
doesn't have XML processing capabiities, so in particular can't detect
or make use of XML encoding declarations.

I can try to clarify this in the draft.

ht
-- 
       Henry S. Thompson, School of Informatics, University of Edinburgh
      10 Crichton Street, Edinburgh EH8 9AB, SCOTLAND -- (44) 131 650-4440
                Fax: (44) 131 650-4587, e-mail: [email-masked]
                       URL: http://www.ltg.ed.ac.uk/~ht/
 [mail from me _always_ has a .sig like this -- mail without it is forged spam]

---

From: Larry Masinter <[email-masked]>
Date: Thu, 9 Jan 2014 01:44:08 +0000

I said
> they're "unaware", why are they reading this spec?

But I think I'd missed this the first pass, I think I just don't like the -aware terminology here.
Please don't clarify.

There are three ways content can advertise its encoding:
  Charset parameter to media type
   Internal charset declaration
   Initial BOM (indicates utf16be, utf16le, utf-32, utf-8)

I think things get simpler if you treat the non-aware case just one of the exceptions to the "SHOULD" for transmitters.
That is, rather than dividing the world into "aware" and "not-aware", just make two categories:

* Those than can send UTF-8 without a BOM and with an internal charset declaration of UTF-8.
* Those that can't do so.

"producers of XML  SHOULD produce UTF-8, with no BOM, with no charset parameter, with an internal UTF-8 charset declaration", we make this the default; it is a new requirement. Those that can't are in some situation of deprecated legacy implementation or limited capabilities. They can't transcode into UTF-8, can't add or remove a BOM, or can't add or modify an internal charset declaration. Do the best you can, but try to make sure there's a charset parameter and that it matches the actual encoding or an internal charset declaration (because then processors will fail. 

Of course, there are more constraints on processors, because they need to deal with legacy, but this will give clear advice to senders.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 09 Jan 2014 10:55:25 +0900

> "producers of XML  SHOULD produce UTF-8, with no BOM, with no charset parameter, with an internal UTF-8 charset declaration", we make this the default; it is a new requirement. Those that can't are…

Why would the default here be *with* an internal UTF-8 declaration? 
Documents without such a declaration are also well-formed/valid UTF-8 
XML documents (assuming that they are well-formed/valid otherwise).

Moving to an UTF-8 only world, not having to declare UTF-8 anywhere 
should be part of that world. It is not for HTML, unfortunately, but it 
is perfectly possible for XML.

Regards,   Martin.

---

From: Ned Freed <[email-masked]>
Date: Mon, 13 Jan 2014 08:19:28 -0800 (PST)

> It cannot be right that everyone specifying a text-based media type should
> have to go through the process of deciding, for themselves, independently, how
> to decide between conflicting sources of information about charset, coming from
> an initial BOM, from external metadata, from any supplied "charset" parameter
> of the media type, and from internal embedded metadata such as found in XML.

It may not be "right" but it is the reality of the situation.

> If the future is UTF-8, UTF-8, UTF-8, then the two documents should say so,

I believe the future is UTF-8, including but not limited to its use in XML, and
we should do what we can to promote it. But beliefs about the future don't
necessarily belong in an RFC.

Moreover, is this the right place and the right organization to make such a
statement about XML? The IETF doesn't own the XML specification, the W3C does.
And this is a document about how to register XML media types, not about how to
use XML.

Mind you, given my own beliefs I don't personally object to such a statement if
there is consensus to include it. I just wonder if it is appropriate.

				Ned

---

From: Larry Masinter <[email-masked]>
Date: Wed, 22 Jan 2014 20:31:30 +0000

> > have to go through the process of deciding, for themselves, independently,. ...

> It may not be "right" but it is the reality of the situation.

Agree it's the reality.

> necessarily belong in an RFC.

We have many documents that give guidelines. 

> And this is a document about how to register XML media types, not about how
> to  use XML.

Organization: yes. We're giving guidelines for communicating text using MIME,
and the interaction of the 'charset' parameter in the metadata with other
sources of encoding information.

Place: Partly.  Guidelines for future text media types belong in a MIME BCP.

> there is consensus to include it. I just wonder if it is appropriate.

Although the general policy doesn't belong in appsawg-xml-mediatypes, but
It would be appropriate to advise senders of text/xml and application/xml
to send UTF-8, to not include a BOM, to recommend whether or not to use
 a charset="UTF-8" parameter, to recommend whether or not to include
an internal charset declaration, even when receivers recognize and interpret
other encodings.

This is an interoperability consideration for a revised specification of 
a widely deployed protocol, based on the belief that UTF-8 is becoming
generally even more the default in text-based MIME types.

Those kinds of forward-interoperability requirements seem to be
common in protocols... a SHOULD introduces a new policy to
correct past interoperability difficulties.

Larry