Subject: [apps-discuss] JSON Content Rules and the need for schema languages for JSON

From: Andrew Newton <[email-masked]>
Date: Thu, 20 Sep 2012 14:58:38 -0400

All,

Since the topics of JSON Schema and schema languages in general are
being debated here, I thought I'd go ahead and publish a draft idea I
had for formalizing expected JSON structures. The draft is here:

http://www.ietf.org/id/draft-newton-json-content-rules-00.txt

While the notion that this is better (or worse) than JSON Schema is
subjective, I do think it turns out to be more concise.

Also, I think it worth mentioning what the ALTO working group has
done. They have a protocol to pass data in JSON, and to describe their
JSON structures they have repurposed C struct syntax to describe it.
You can find that draft here:

http://www.ietf.org/id/draft-ietf-alto-protocol-13.txt

While some see no value in schema languages for JSON (or schema
languages in general), there does appear to be an interest in methods
more formal than prose to describe expected content of JSON data. At
the very least, they can provide clarity for protocol specifications.
I'll note that while I was working up one of the examples for my
draft, I found an error in the prose of another JSON protocol on which
I had been working.

-andy

---

From: Nico Williams <[email-masked]>
Date: Thu, 20 Sep 2012 17:08:13 -0500

So, I really like JSON, but I do have some complaints, really, only
one severe complaint:

 - There's no standard way to encode binary data.

   This is a big deal.  If I wanted to design a protocol that uses
   crypto and JSON for encoding then I'd need binary data.

   Yeah, yeah, I can base64-encode binary data, but at the very
   least any schema language ought to be able to indicate that
   some field is expected to bear base64-encoded data.  I'd rather
   have a length-and-raw-octet-string encoding available though...

 - Lack of arbitrary types for associative array keys.  (This is only
   really a problem for me because of Core Foundation like APIs
   that have associative arrays that do allow arbitrary types as
   keys.  So, this is not a serious complaint.)

 - ECMAScript Unicode issues...   This isn't really about JSON
   so maybe I shouldn't bore you with these.

Also, I do have a need to re-encode ASN.1 DER-encoded data in JSON for
debugging/tracing/logging purposes.  I'd like a JER for ASN.1.  Yeah,
I know, y'all are not gonna care, and given the purpose there's hardly
a need to standardize this, but others may need it too.  Besides,
wouldn't it be nice to be able to re-encode certificates in JSON?
CSRs in particular would be convenient to be able to convert to/from
JSON...

Nico
--

---

From: "Paul C. Bryan" <[email-masked]>
Date: Thu, 20 Sep 2012 15:09:53 -0700

>  - There's no standard way to encode binary data.

+1! One of my biggest pet peeves.

> CSRs in particular would be convenient to be able to convert to/from

Paul

---

From: Nico Williams <[email-masked]>
Date: Thu, 20 Sep 2012 17:18:50 -0500

> So, I really like JSON, but I do have some complaints, really, only

Crap, I forgot another important complaint:

 - There's no canonical ordering and encoding.  This is very
   important for protocols using crypto!

BTW, I get it that a lot of you have a visceral dislike for ASN.1 and
its encoding rules, but at least it has had canonical ones for a very
long time.  XMLdsig and XMLenc have mostly left me wondering (e.g.,
the fact that XMLdsig doesn't (didn't?) have AEAD modes, by generic
composition or otherwise, seems like asking for trouble).  And there's
no such thing for JSON anyways.

Now, we don't need any fancy JSON crypto spec: just give us octet
strings and a canonical ordering/encoding, and we can lift crypto from
elsewhere.

Nico
--

---

From: Andrew Newton <[email-masked]>
Date: Thu, 20 Sep 2012 18:21:11 -0400

> +1! One of my biggest pet peeves.

Is there a text format that does this well?

-andy

---

From: Nico Williams <[email-masked]>
Date: Thu, 20 Sep 2012 17:23:28 -0500

> Is there a text format that does this well?

Dunno.  But here's what I'm looking for:

 - strings: '...'
 - octet strings: x'<hex-encoded>' or b'<base64-encoded>'

I'd love an option for not-really text JSON where octet strings are
encoded as <length><actual octet string>, but this is probably a
stretch.

Nico
--

---

From: Andrew Newton <[email-masked]>
Date: Thu, 20 Sep 2012 18:29:38 -0400

> no such thing for JSON anyways.

Being that there is XER, perhaps JER would find some interest.

Having worked on RPKI, I can tell you that ASN.1 DER is no fun because
it needs specialized tools to even view a payload. Just emitting DER
is an exercise. That's why text formats are more popular, I would
guess. Also finding software developers who want ASN.1 as part of
their career path is impossible.

> strings and a canonical ordering/encoding, and we can lift crypto from

Have you looked at the JOSE working group?

-andy

---

From: Tim Bray <[email-masked]>
Date: Thu, 20 Sep 2012 15:31:09 -0700

The lesson from XML is that canonicalization is hard and usually not worth
the effort.  For your binary, use base64, keys & certs aren’t that big, and
vanish in the rounding error of the video & audio that’s filling network
pipes.  The number of successful modern platform-agnostic data formats that
allow raw binary is zero; there’s a reason for that.  JSON is not quite
right for your purposes, but the huge amount of deployed software makes
doing the extra work to use it a good trade-off. -T

On Thu, Sep 20, 2012 at 3:18 PM, Nico Williams <[email-masked]>wrote:

> _______________________________________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss
>

---

From: Nico Williams <[email-masked]>
Date: Thu, 20 Sep 2012 17:31:59 -0500

> their career path is impossible.

I won't dispute that.  I don't mind having to use more tools.  I don't
necessarily care for every bit of on-the-wire payload to be
human-readable (this may just be because I work in the security area,
and once you bring in crypto you always need tools to view the
messages anyways).

> Have you looked at the JOSE working group?

No.  I will.  Thanks for the tip!

---

From: "Paul C. Bryan" <[email-masked]>
Date: Thu, 20 Sep 2012 15:35:01 -0700

> Is there a text format that does this well?

I haven't seen one. :-|

Paul

---

From: Nico Williams <[email-masked]>
Date: Thu, 20 Sep 2012 17:40:37 -0500

> allow raw binary is zero; there’s a reason for that.  JSON is not quite

Is that so?  I don't think so, not for media data anyways.  Also,
real-time media protocol designers tend to want to minimize extraneous
bits (so payloads fit in small packets), like authentication tags.
Also, I've seen complaints about TLS not using alignments that are
friendly to implementations in hardware.  Text is nice, I know, but
it's not the right tool for all cases.

> doing the extra work to use it a good trade-off. -T

I'm not convinced.  I do think JSON is right for at least some purposes:

 - to re-encode things like certificates and CSRs for human
   consumption (and in the case of CSRs possibly for UIs
   that allow editing, in which case we need re-encoding back
   into DER)

 - to programmatically manipulate complex data (e.g., in JS,
   or via CF<->JSON bridging or equivalent)
 - to log things like Kerberos KDC requests, CSRs, ...

Is JSON right for protocols that need crypto?  I'd say "not if you
knew that on day zero", but if you later realize that your protocol is
the correct layer to retrofit crypto into, then you'll be happy to
find canonical encoding rules.  But I'll agree to defer that as much
as possible.

Nico
--

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 20 Sep 2012 19:07:45 -0400

+1 on the Binary type, I think it is also useful to add in a DateTime type
so that protocols do not attempt to roll their own.

>    important for protocols using crypto!

I disagree. I have never seen the need for DER encoding.

VeriSign only switched to generating DER encoding about 5 or so years after
they started business. That probably means there are still some non-DER
roots.

Now the bigger problem with DER is the particular approach they took with
the definite encodings that use a length indicator of indefinite size. Its
yuk city. But I still don't see any need or use for canonical encoding and
I don't think that there is anything out there that will break if presented
with non-canonical

The idea that people are going to shred a certificate, store it in a
directory and then recombine is just potty.

> no such thing for JSON anyways.

It had canonical ones but they were ignored without detrimental effect for
a very long time.

-- 
Website: http://hallambaker.com/

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 20 Sep 2012 19:18:25 -0400

I am actually sensing a pretty wide sense of agreement here on quite a
number of issues.

A JER might well be useful as a way to migrate specs from ASN.1 to JSON. We
have done this in reverse as well. KEYPROV had to define an ASN.1 binding
as well as an XML because people who had a PKI that was 100% ASN.1 did not
want to use XML just for key provisioning.

But I had to respond here as the last time I wrote an ASN.1 decoder it was
written in JavaScript...

---

From: Nico Williams <[email-masked]>
Date: Thu, 20 Sep 2012 18:37:33 -0500

> I am actually sensing a pretty wide sense of agreement here on quite a

That had me doing a double take, but I think that's right.

> have done this in reverse as well. KEYPROV had to define an ASN.1 binding as
> well as an XML because people who had a PKI that was 100% ASN.1 did not want
> to use XML just for key provisioning.

There are a number of decent open source ASN.1 tools nowadays (yeah,
yeah, not nearly as many as XML tools), and it might not be hard to
add JER support to some of them (I'm thinking of asn1c and Heimdal's
ASN.1 compiler).

> But I had to respond here as the last time I wrote an ASN.1 decoder it was

That's another thing: JS needs better binary data handling.

Nico
--

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 21 Sep 2012 01:46:29 +0200

> - octet strings: x'<hex-encoded>' or b'<base64-encoded>'

Indeed.  The problem is not that the binary data has to be carried around in some encoded text form, but that it is not identified as binary data.
(Lose the hex idea, though.  Unneeded choice.)
Which raises the question: What is the status on binary data (really: byte strings) in ECMAscript?

Grüße, Carsten

---

From: Andrew Newton <[email-masked]>
Date: Thu, 20 Sep 2012 20:08:41 -0400

> so that protocols do not attempt to roll their own.

Given that both JSON Content Rules and JSON Schema have both base64
and dateTime, what's the difference between the base text format
saying a string is base64 or dateTime vs a schema for JSON saying a
string is base64 or dateTime?

-andy

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 20 Sep 2012 21:32:26 -0400

My problem with "Proposed JSON Schema" is not a lack of features but an
overabundance of pointless and unnecessary ones.

JSON Content Rules looks like a better approach. I am not sure how useful
the URI, phone number or email types are but they are probably harmless.
Domain names and IP address formats are probably useful.

I would probably lose the range constraints on integers, I have not found
them remotely useful in a protocol except for things like IPv4 port numbers
and even there the constraint probably doesn't help very much.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 21 Sep 2012 18:17:12 +0900

On 2012/09/21 8:46, Carsten Bormann wrote:
> On Sep 21, 2012, at 00:23, Nico Williams<[email-masked]>  wrote:
>

Last I heard (that was in a WebSocket discussion, so it must be a year 
or two ago), it was being worked on. WebSocket has both text (UTF-8) and 
binary messages, and in order to handle binary messages in JavaScript, a 
binary datatype is needed.

Just checked a bit now, found e.g. 
https://developer.mozilla.org/en-US/docs/JavaScript_typed_arrays/ArrayBuffer. 
It looks like rather than using a very simple datatype with a literal 
syntax, they went for a rather complex architecture. They must have had 
their reasons, but please don't ask me why.

If the above is true, then introducing something like
b'<base64-encoded>' would mean JSON is no longer a subset of JS.

Regards,    Martin.

---

From: Nico Williams <[email-masked]>
Date: Fri, 21 Sep 2012 10:06:55 -0500

On Fri, Sep 21, 2012 at 4:17 AM, "Martin J. Dürst"
<[email-masked]> wrote:
>> Indeed.  The problem is not that the binary data has to be carried around
>> in some encoded text form, but that it is not identified as binary data.
> Last I heard (that was in a WebSocket discussion, so it must be a year or
> two ago), it was being worked on. WebSocket has both text (UTF-8) and binary
> messages, and in order to handle binary messages in JavaScript, a binary
> datatype is needed.
>

That would probably be their reason: an octet string literal can't be
had without changing ECMAScript, or, in our case, without making JSON
no longer a subset of ECMAScript.

For some protocols that's something I'm willing to live with.  I doubt
we'd reach consensus on that, so we may have to settle for "the schema
tells you if a string is actually intended to be a base64 encoding of
an octet string", in which case this will continue to be my biggest
pet peeve about JSON.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Fri, 21 Sep 2012 10:08:15 -0500

> string is base64 or dateTime?

I don't always want to use a schema, and when I don't, I like the data
types to be self-describing.  And in JSON types are self-describing,
except for the lack of an octet string literal.

---

From: Nico Williams <[email-masked]>
Date: Fri, 21 Sep 2012 10:10:17 -0500

I should note that one JSON encoding/decoding (to/from CF-like
objects) library I've used/hacked on has flags for specifying how
strictly to adhere to JSON and can encode things that JSON cannot by
extending JSON.

Nico
--

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 21 Sep 2012 17:35:07 +0200

> no longer a subset of ECMAScript.

It might be possible to develop a way to express the difference between a UTF-8 string and a base64-encoded byte string in a way that is backwards compatible with RFC 4627 JSON.  If JSON had comments, you might do

"foob" /*utf-8*/
and
"foob" /*b64*/

Now, JSON does not have comments, and the only variability we have for expressing this kind of out-of-band information appears to be whitespace.
E.g., interpret a RFC 4627 JSON string as a base64-encoded byte string if/only if it is preceded by, say, two newlines.

So a JSON reader not aware of this encoding would need to rely on a schema to know when a string is text or base-64-encoded bytes.
A JSON writer not aware of this would be quite unlikely to emit the two newlines.
(And we'd need a similar covert channel to indicate a JSON text is actually using this encoding.)

(I'm well aware of the security considerations of this kind of backwards compatibility.  But it could be made to work in practice.)

> an octet string", in which case this will continue to be my biggest

Exactly.

Grüße, Carsten

---

From: James M Snell <[email-masked]>
Date: Fri, 21 Sep 2012 08:39:05 -0700

Introducing a binary data type within JSON would just be a mistake. Full
stop. If you do need to represent binary data within JSON, Base64 works
quite well and is all that is really needed. (Refer to [1] for a real world
example).

As far as dates are concerned, RFC3339 has generally become the defacto
standard for the majority of JSON-based applications I've seen (and have
developed). JSON itself, however, does not need an explicit date type. In
the specs I've written, I've found the following to be more than sufficient
to get the point across: "Unless otherwise specified, all properties
specifying date and time values MUST conform to the "date-time" production
in [RFC3339]."

[1] http://activitystrea.ms/specs/json/schema/activity-schema.html#binary

- James

On Fri, Sep 21, 2012 at 2:17 AM, "Martin J. Dürst"
<[email-masked]>wrote:

>> On Sep 21, 2012, at 00:23, Nico Williams<[email-masked]**>  wrote:
>>
> two ago), it was being worked on. WebSocket has both text (UTF-8) and
> Just checked a bit now, found e.g. https://developer.mozilla.org/**
> en-US/docs/JavaScript_typed_**arrays/ArrayBuffer<https://developer.mozilla.org/en-US/docs/JavaScript_typed_arrays/ArrayBuffer>.
> ______________________________**_________________
> https://www.ietf.org/mailman/**listinfo/apps-discuss<https://www.ietf.org/mailman/listinfo/apps-discuss>
>

---

From: Nico Williams <[email-masked]>
Date: Fri, 21 Sep 2012 10:47:05 -0500

> E.g., interpret a RFC 4627 JSON string as a base64-encoded byte string if/only if it is preceded by, say, two newlines.

Interesting idea.  It may well be good enough for me (considering I
can't have the way I'd like it).

> (I'm well aware of the security considerations of this kind of backwards compatibility.  But it could be made to work in practice.)

Well, no further security considerations than we'd have today when
just encoding with base64, except, of course, that we would need to
ensure that any canonicalization does not strip out these marker
newlines.

Nico
--

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 21 Sep 2012 10:28:00 -0700

You folks are getting way off track, and I can see Crockford holding his head in his hands and weeping. Or maybe reaching for a gun.

Go off and invent something new, fine, but please don't call it "JSON" anything. And, specify it as something broadly applicable at your peril (lest myself and others watching this in horror have something to aim at).

Cheers,

--
Mark Nottingham
http://www.mnot.net/