Subject: [apps-discuss] [Technical Errata Reported] RFC6839 (4367)

From: RFC Errata System <[email-masked]>
Date: Fri, 15 May 2015 06:10:52 -0700 (PDT)

The following errata report has been submitted for RFC6839,
"Additional Media Type Structured Syntax Suffixes".

--------------------------------------
You may review the report below and at:
http://www.rfc-editor.org/errata_search.php?rfc=6839&eid=4367

--------------------------------------
Type: Technical
Reported by: Yakov Shafranovich <[email-masked]>

Section: 3.1

Original Text
-------------
Encoding considerations:

      Per [RFC4627], JSON is allowed to be represented using UTF-8,
      UTF-16, or UTF-32.  When JSON is written in UTF-8, JSON is 8bit
      compatible ([RFC2045]).  When JSON is written in UTF-16 or UTF-32,
      JSON is binary ([RFC2045]).

Corrected Text
--------------
Encoding considerations:  binary as per section 11 of RFC 7159

Notes
-----
RFC 7159, section 11 specifies that encoding for JSON is binary.

Instructions:
-------------
This erratum is currently posted as "Reported". If necessary, please
use "Reply All" to discuss whether it should be verified or
rejected. When a decision is reached, the verifying party (IESG)
can log in to change the status and edit the report, if necessary. 

--------------------------------------
RFC6839 (draft-ietf-appsawg-media-type-suffix-regs-08)
--------------------------------------
Title               : Additional Media Type Structured Syntax Suffixes
Publication Date    : January 2013
Author(s)           : T. Hansen, A. Melnikov
Category            : INFORMATIONAL
Source              : Applications Area Working Group
Area                : Applications
Stream              : IETF
Verifying Party     : IESG

---

From: Barry Leiba <[email-masked]>
Date: Fri, 15 May 2015 14:18:37 +0100

And yet this RFC predates 7159, so how can 7159 be taken to support errata
for this RFC?

Barry

On Friday, May 15, 2015, RFC Errata System <[email-masked]>
wrote:

> Reported by: Yakov Shafranovich <[email-masked] <javascript:;>>
>
> Section: 3.1
>
> Original Text
> -------------
> Encoding considerations:
>

---

From: Ned Freed <[email-masked]>
Date: Fri, 15 May 2015 08:11:10 -0700 (PDT)

This is bogus and should be rejected. 

				Ned

> _______________________________________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss

---

From: Ned Freed <[email-masked]>
Date: Fri, 15 May 2015 08:12:59 -0700 (PDT)

> And yet this RFC predates 7159, so how can 7159 be taken to support errata

I suppose it could if it corrected something in the earlier RFC. But it
doesn't. The earlier RFC is simply providing a more complete description of the
encoding considerations that exist for JSON. Neither is incorrect though, so it
doesn't make any sort of case for being an error.

				Ned

---

From: Yakov Shafranovich <[email-masked]>
Date: Fri, 15 May 2015 11:22:14 -0400

Thank you. If both are correct, can either one be specified for a new
+json media type registration. I am a member of the W3C CSVW group
where this is originating from.

This is in regards to the following registration:

https://www.ietf.org/mail-archive/web/media-types/current/msg00686.html

At the W3C, the following issue was raised:

https://github.com/w3c/csvw/issues/546

" "When JSON is written in UTF-8, JSON is 8bit compatible ([RFC2045])
so, 8bit not binary, since utf-8 is the default encoding your media
type registration says binary rfc6839 (which you reference) says the
above"

Yakov

---

From: Tony Hansen <[email-masked]>
Date: Fri, 15 May 2015 12:36:17 -0400

On 5/15/15 9:31 AM, Yakov Shafranovich wrote:
> [For context, this is originating from the work at the W3C regarding CSV files]
>
> There appears to be an issue about how to specify encoding
> considerations for media types that can be encoded in UTF-8, UTF-16
> and UTF-32. For media types, the valid choices are 7-bit, 8-bit and
> binary, which would mean that UTF-16 and UTF-32 are binary. For JSON
> specifically, since both RFCs define JSON, there is a conflict.
>
> There are two ways to write this then:
>
> 1. As in RFC 6839:
>
> "When JSON is written in UTF-8, JSON is 8bit compatible ([RFC2045]).
> When JSON is written in UTF-16 or UTF-32, JSON is binary ([RFC2045])."
>
> 2. As per RFC 7159:
>
> "binary"
>
> What I am arguing is that the second approach would make more sense.
> Just like RFC 7159 choose to use "binary" in case of multiple UTF
> encodings, we should follow the same approach in RFC 6839. If not,
> then RFC 7159 should have errata pointing back to RFC 6839.

Hmmm, it's too bad we didn't catch this before 7159 was published. 7159
is wrong, or at least incomplete.

When JSON is written in UTF-8, you MAY use an encoding of 8-bit, or you
MAY use an encoding of binary. When JSON is written in UTF-16 or -32,
you MUST use an encoding of binary.

This is because of the definition of the encoding system definitions of
7-bit, 8-bit and binary, which is totally orthogonal to ANY media type.
The definition of UTF-8 is, in and of itself, compatible with the
definition of 8-bit encoding.

    Tony

>
> Thanks,
> Yakov
>
> On Fri, May 15, 2015 at 9:18 AM, Barry Leiba <[email-masked]> wrote:

---

From: Yakov Shafranovich <[email-masked]>
Date: Fri, 15 May 2015 12:43:43 -0400

Thank you! I will relay this information to my WG at the W3C.

I guess this means the errata should really be reported on RFC 7159 instead?

---

From: Tony Hansen <[email-masked]>
Date: Fri, 15 May 2015 12:53:58 -0400

There is one more constraint for a file written with utf-8 to use the
encoding of 8bit: the lines are limited to 998 bytes in length (not
counting the CRLF line terminators). See RFC 2045 for details.

It's up to you whether you want to write an errata against RFC 7159.

    Tony Hansen

---

From: Yakov Shafranovich <[email-masked]>
Date: Fri, 15 May 2015 09:31:03 -0400

1. As in RFC 6839:

2. As per RFC 7159:

"binary"

Thanks,
Yakov

---

From: Ned Freed <[email-masked]>
Date: Sat, 16 May 2015 10:11:19 -0700 (PDT)

> What I am arguing is that the second approach would make more sense.

And that's a fine opinion to have, but others disagree. The fact remains that
both statements are correct; we have always allowed a simple statement of the
most general encoding that applies to the type or a more detailed  statement of
what encodings apply under certain circumstances. The registrations rules allow
either approach.

I also note that you have yet to identify any sort of conflict here.

> then RFC 7159 should have errata pointing back to RFC 6839.

You are free to use whichever approach you wish. But in no case is an erratum
appropriate.

				Ned

---

From: Graham Klyne <[email-masked]>
Date: Sat, 16 May 2015 19:19:36 +0100

> counting the CRLF line terminators). See RFC 2045 for details.

I think that would be a problem for JSON - there's no way (I know of) to break 
long text strings over multiple lines, so that restriction would rule out very 
long string values.  So maybe "binary" is the right choice.

#g
--

---

From: Dave Cridland <[email-masked]>
Date: Sat, 16 May 2015 19:56:58 +0100

On 16 May 2015 19:19, "Graham Klyne" <[email-masked]> wrote:
> I think that would be a problem for JSON - there's no way (I know of) to
break long text strings over multiple lines, so that restriction would rule
out very long string values.  So maybe "binary" is the right choice.

Plenty of languages don't use line termination in the same way, and in
principle therefore plenty of things might not work as 8-bit.

All that means is that once serialized, the application needs to test for
long lines and use something different if needs be, like binary or base 64.

But that's no different to HTML, or plain text.

---

From: Ned Freed <[email-masked]>
Date: Sat, 16 May 2015 18:21:16 -0700 (PDT)

> long string values.  So maybe "binary" is the right choice.

The same can be said for XML - multimegabyte XML objects containing no line
breaks at all are pretty common. And while you can in theory encode text()
nodes as CDATA, introducing line breaks into XML isn't something you can just
do and leave the semantics unchanged.

It's also the case that the line terminators in an XML or JSON object may not
always be CRLF, and thus an object may not be compatible with 7bit or 8bit on
that basis.

But the fact remains that a lot of XML and JSON objects are 8bit or even 7bit
compatible. In fact a lot of formats based on XML or JSON are by their nature
_always_ 8bit or 7bit compatible. And I see some value in pointing that out
where it's appropriate to do so.

I suppose we could insist on a full rundown of the issues in every
registration. But do you really think this is a useful thing to do, rather
than leaving it a judgement call on the part of the person constructing
the registration?

				Ned

---

From: Graham Klyne <[email-masked]>
Date: Sun, 17 May 2015 15:25:04 +0100

> than leaving it a judgement call on the part of the person constructing

No, I don't.

I raised the point because I've found that I can construct XML in a way that 
avoids very long lines (YMMV).  But I've noticed in my work with JSON that there 
is no way to encode long string values over multiple lines (even when the 
strings themselves are multiline values).

Checking back... the encoding considerations for +xml in 
https://tools.ietf.org/html/rfc6839#section-4.1 call for 7bit or 8bit for UTF-8, 
and binary for UTF-16/32.  Your comment suggests some UTF-8 encoded XML would 
not satisfy the 8bit constraint on line length, and binary should at least be an 
option there.

And, as noted, https://tools.ietf.org/html/rfc7159#section-11 just says "binary" 
for JSON, which seems OK to me, but as you say 7bit or 8bit might be OK for some 
JSON data.

(I suspect that in practice these concerns don't arise as 7bit and 8bit are 
primarily email concerns, and my experience is that JSON is mostly used in HTTP 
exchanges.)

#g
--

---

From: Nico Williams <[email-masked]>
Date: Wed, 20 May 2015 15:46:42 -0500

On Sun, May 17, 2015 at 03:25:04PM +0100, Graham Klyne wrote:
> I raised the point because I've found that I can construct XML in a
> way that avoids very long lines (YMMV).  But I've noticed in my work
> with JSON that there is no way to encode long string values over
> multiple lines (even when the strings themselves are multiline
> values).

+1.  Binary is the correct thing to say for any JSON-using protocol
unless the schema used allows one to ensure that UTF-8-encoded JSON
texts have "lines" no longer than 1000 bytes -- but why bother?

> And, as noted, https://tools.ietf.org/html/rfc7159#section-11 just
> says "binary" for JSON, which seems OK to me, but as you say 7bit or
> 8bit might be OK for some JSON data.

Right, and that's true, but not often possible to ensure.

> (I suspect that in practice these concerns don't arise as 7bit and
> 8bit are primarily email concerns, and my experience is that JSON is
> mostly used in HTTP exchanges.)

Agreed.  This is one of those how many angels can dance on a pin head
kind of debates.  The erratum could be accepted or rejected, and it
wouldn't make a difference.  Since the MIME registration exists outside
the RFC, I think it's better to correct it than to not, and note that
it's not necessary to correct the RFC to correct the registration.  Add
some text discussing the interop considerations if you like.

Nico
--

---

From: Larry Masinter <[email-masked]>
Date: Sat, 23 May 2015 22:54:10 +0000

The error reported should be accepted as an erratum on RFC6939 which
says:

which is false. RFC 2045 defines 8bit data

   "8bit data" refers to data that is all represented as relatively
    short lines with 998 octets or less between CRLF line separation
    sequences [RFC-821]), but octets with decimal values greater than 127
    may be used.  As with "7bit data" CR and LF octets only occur as part
    of CRLF line separation sequences and no NULs are allowed.

but UTF-8 JSON may well contain sequences that include CR or LF
octets not part of CRLF sequences, and may also contain more than
998 octets between CRLF line separation sequences.

Following http://www.ietf.org/iesg/statement/errata-processing.html:

It’s a matter of judgment whether this error COULD cause implementation
or deployment problems (someone assuming they could use 8bit for arbitrary
utf8 JSON?) but it seems possible. 

The change proposed makes a reference to RFC 7159 which is unnecessary,
it could just say “Encoding Considerations: binary”.

It seems like a tautology to have a media type which in general
requires “binary” but to note “If an instance of the media type
meets the requirements for 7bit or 8bit transport then 7bit or
8bit encoding is compatible”, since you could say that for any
binary media type.

 RFC 6838 “Media Type Registration” section 4.8 says
"Possible values of this [Encoding Considerations] field are:”

and lists 7bit, 8bit, binary, framed as the possible values.

Larry
—
http://larry.masinter.net

---

From: "Martin J. Dürst" <[email-masked]>
Date: Sun, 24 May 2015 16:03:01 +0900

> 8bit encoding is compatible”, since you could say that for any

Well, there are media types where although only binary is good enough to 
cover every case, many or even most cases are covered by 8bit or even 
7bit. The JSON-related types are definitely in this category, and some 
of the types with a +json suffix may even be 7bit in all cases.

On the other hand, there are types where the vast majority of instances 
won't work with 7bit, and most also won't work with 8bit. Image formats 
and video formats will be in this category.

Using the text that you cite above makes sense for the former category, 
but not for the later.

Regards,   Martin.