Subject: [apps-discuss] Unicode BOM detection in JSON vs. draft-ietf-appsawg-xml-mediatypes

From: Larry Masinter <[email-masked]>
Date: Fri, 13 Dec 2013 22:10:25 +0000

Compare   http://tools.ietf.org/html/draft-ietf-json-rfc4627bis-09#section-8.1
and  http://tools.ietf.org/html/draft-ietf-appsawg-xml-mediatypes-06#section-3

it's hard to tell exactly what either is saying, much less whether they're saying
the same thing.   I don't want to slow down either of these, but some work on
a model for how MIME types should deal with the relationship between

-          Charset parameter in content-type

-          Possible BOM maker

-          Other in-content character set indication (like in XML)

-          Other protocol sources of charset information

JSON, XML, HTML, IRIs, elements based on those, and likely many other
protocol elements are defined using BNF or other syntactic description
techniques over the space of sequence-of-Unicode-characters
(sequence-of-Unicode-code-point).

Has there been any attempts to create a separate BCP that JSON and XML
Could (in the future) reference? The JSON working group spent only
a few thousand emails on this.

Larry
--
http://larry.masinter.net

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Fri, 13 Dec 2013 14:15:07 -0800

> a few thousand emails on this.
Not in APPSAWG, but it sounds like the JSON WG has already spent a lot of
time on it; perhaps they're in a better position to do so?

If not, the usual points apply:

1) Is this work APPSAWG should take on?  Who will provide reviews?  Who
will contribute text?

2) We need to get some other stuff moved onward before we can take on new
work.

-MSK

---

From: Tim Bray <[email-masked]>
Date: Fri, 13 Dec 2013 14:20:33 -0800

I actually think 4627bis is perfectly clear. You can use UTF-{8/16/32} in
theory; in practice only UTF-8 works over the wire; you MUST NOT emit a BOM
but if some dork does, it’s OK to forgive. There’s no charset parameter on
the media type.

If we were doing XML in 2013, I think we’d probably end up at the same
place. Note that the original XML spec was written in ISO-8859-1 :)

On Fri, Dec 13, 2013 at 2:10 PM, Larry Masinter <[email-masked]> wrote:

>  Compare
> http://tools.ietf.org/html/draft-ietf-json-rfc4627bis-09#section-8.1
>
> and
> http://tools.ietf.org/html/draft-ietf-appsawg-xml-mediatypes-06#section-3
>
>
>
> it’s hard to tell exactly what either is saying, much less whether they’re
> saying
>
> the same thing.   I don’t want to slow down either of these, but some work
> on
>

---

From: "Martin J. Dürst" <[email-masked]>
Date: Sat, 14 Dec 2013 21:25:36 +0900

> place. Note that the original XML spec was written in ISO-8859-1 :)

Yes. On the surface, some of the issues in JSON and XML may look 
similar, but in actual practice, they are quite different. So a common 
model or a common spec doesn't make sense.

We have had some kind of common model from around 1996 or so, which was 
that the outside (charset parameter) has precedence. This was based on 
the assumption that there were transcoding proxies than didn't look 
inside the document, and on statements from Netscape (when it was still 
the dominant browser) that this was what they did.

It turned out that there were few if any transcoding proxies, and 
external information didn't get preserved well in a file system, and so 
the practice moved more and more to give priority of internal 
information over external.

In ietf-appsawg-xml-mediatypes, we can see one specific example of what 
this history has resulted in: carefully defined specific steps, but not 
too pretty an overall picture.

The model of the future is much simpler: UTF-8, UTF-8, UTF-8. Once 
everything is UTF-8, the BOM will die a quiet death, too.

Regards,   Martin.

> On Fri, Dec 13, 2013 at 2:10 PM, Larry Masinter<[email-masked]>  wrote:
>
>>   Compare
> _______________________________________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 14 Dec 2013 14:25:37 +0100

On 14 Dec 2013, at 13:25, Martin J. Dürst <[email-masked]> wrote:

> The model of the future is much simpler: UTF-8, UTF-8, UTF-8.

Absolutely.  Meta-comment:

"The future is already here — it's just not very evenly distributed."*)

We (as a standardization organization) need to prioritize enabling the way forward over bracing up the past.
(I think we are in reasonable shape now with JSON, but there was more kicking and screaming than I would have thought.)

Grüße, Carsten

*) William Gibson, of course.