Subject: [apps-discuss] Concise Binary Object Representation (CBOR)

From: Paul Hoffman <[email-masked]>
Date: Wed, 22 May 2013 07:49:36 -0700

Greetings again. Carsten Bormann and I have proposed a new compact binary format, CBOR: <http://tools.ietf.org/html/draft-bormann-cbor>. The design goals for the format are listed in the introduction; most notably, code compactness is considered as important as message compactness, and we think we got a lot of the latter with no sacrifice of the former.

A few appsy folks have already contributed ideas, but we would like to hear more. Given its applicability to protocols across the Applications area (and actually across the IETF), we think this would be an appropriate to be an appsawg work item if others here think so as well. Please let us know what you think.

--Paul Hoffman

---

From: James M Snell <[email-masked]>
Date: Wed, 22 May 2013 08:47:10 -0700

At first read it looks interesting, and can definitely see that a lot
of thought was put into the design. There have, however, been a number
of previous attempts at alternative compact binary representations (or
at least discussions) that did not seem to really go anywhere. What is
the current implementation status of this? Are there implementations
available or any existing plans to use this new format in a specific
app or spec? I'm largely just curious about the context and motivation
behind this...

On Wed, May 22, 2013 at 7:49 AM, Paul Hoffman <[email-masked]> wrote:
> _______________________________________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 22 May 2013 09:00:55 -0700

On May 22, 2013, at 8:47 AM, James M Snell <[email-masked]> wrote:

> at least discussions) that did not seem to really go anywhere.

We did ours differently than others: we started with an ordered set of design goals and stuck to them. That means that our early arguments about changes / bikeshedcolors had something we could compare against, and it make coming up with a solution much more tractable.

> What is
> app or spec?

Both Carsten and I did versions (in Ruby and Python) to make sure the spec made sense and the examples worked. At least one other member of this list has apparently done a version in JavaScript. 

> I'm largely just curious about the context and motivation
> behind this...

There are many contexts. It might be useful in CORE and other constrained-environment work. It might be useful for the next protocols that need a compact binary representation and the WG doesn't want to fight over the specific features. If the design goals fit your needs (and we actually met them...), then it's useful to have it be widely available.

--Paul Hoffman

---

From: Dave Crocker <[email-masked]>
Date: Wed, 22 May 2013 09:11:03 -0700

On 5/22/2013 8:47 AM, James M Snell wrote:
> There have, however, been a number

Indeed, Jack Haverty, then of MIT, lobbied hard for an especially 
concise, variable-length encoding form for email, back when RFC 733 was 
being developed, in 1977 for email.

We thought his idea clever but deciding to stay with the existing 
simple-ASCII practice, for messages that looked pretty much as they do 
today.

Jack's scheme certainly wasn't identical to the CBOR proposal, but it 
looks like the design approach is related.

I've always thought the approach a good one... assuming that it makes 
the right choices for what data to represent the most concisely.

For that, there should be some empirical data cited and possibly some 
discussion of significant earlier, related encoding schemes.  I suspect 
BER is the major one to consider.

d/

-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 22 May 2013 12:14:24 -0400

Meh...

I think we can all agree that *A* binary format would be useful and that
two dozen are not useful. The whole problem is how to convince people to
use one particular scheme rather than the alternatives.

ASN.1 has a decent binary encoding. The problem is that it also has some
terrible ones and the ASN.1 schema language was designed by a committee.

Text encodings do have problems. Floating Point values do not round trip
from IEEE floating point representations without special handling so the
format causes a loss in precision that can cause real trouble in scientific
work. And nobody likes base64 encoding binary values.

I would like to have available a BER like encoding for JSON. But this
proposal seems to be something rather different.

There is also the XML binary encoding effort that took place. They had a
competition and everything. And at the end the only real use for the work
was that it shut up the constant carping from people complaining that Web
Services in XML were verbose.

On Wed, May 22, 2013 at 12:00 PM, Paul Hoffman <[email-masked]>wrote:

> We did ours differently than others: we started with an ordered set of
> design goals and stuck to them. That means that our early arguments about
> changes / bikeshedcolors had something we could compare against, and it
> make coming up with a solution much more tractable.
>
> > What is
> Both Carsten and I did versions (in Ruby and Python) to make sure the spec
> made sense and the examples worked. At least one other member of this list
> has apparently done a version in JavaScript.
>
> There are many contexts. It might be useful in CORE and other
> constrained-environment work. It might be useful for the next protocols
> that need a compact binary representation and the WG doesn't want to fight
> over the specific features. If the design goals fit your needs (and we
> actually met them...), then it's useful to have it be widely available.
>
> --Paul Hoffman

-- 
Website: http://hallambaker.com/

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 22 May 2013 09:16:33 -0700

On May 22, 2013, at 9:14 AM, Phillip Hallam-Baker <[email-masked]> wrote:

> I think we can all agree that *A* binary format would be useful and that two dozen are not useful.

Nope, that is patently absurd. That would only be true if everyone agreed on all the design goals and decisions for the one binary format, and that clearly cannot happen.

--Paul Hoffman

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 22 May 2013 12:26:03 -0400

On Wed, May 22, 2013 at 12:16 PM, Paul Hoffman <[email-masked]>wrote:

> two dozen are not useful.
>
> Nope, that is patently absurd. That would only be true if everyone agreed
> on all the design goals and decisions for the one binary format, and that
> clearly cannot happen.
>

We all use RFC822 header style despite the rather obvious fact it does not
meet every need. Most of us used XML when that was fashionable despite the
baroque nature. Pretty much everyone is piling into JSON right now.

For a binary format to be useful it has to be something that is shared
across multiple projects so that we can get code reuse.

It might be that there is utility in three formats with different features
but I rather doubt that is the case as I have never heard of a project fail
because XML or JSON did not have enough expressive capability. The problems
with XML come from the complexity and the extensibility being rather badly
thought out.

We already have a few projects that have defined such schemes already. TLS
for example.

---

From: James M Snell <[email-masked]>
Date: Wed, 22 May 2013 09:27:57 -0700

On Wed, May 22, 2013 at 9:00 AM, Paul Hoffman <[email-masked]> wrote:
[snip]
> Both Carsten and I did versions (in Ruby and Python) to make sure the spec made sense and the examples worked. At least one other member of this list has apparently done a version in JavaScript.

Good to know, thank you.

> There are many contexts. It might be useful in CORE and other constrained-environment work. It might be useful for the next protocols that need a compact binary representation and the WG doesn't wa…

The main point of my question was to find out if this was specifically
attached to some other piece of work or if it's a standalone "it's
here if you need it and want it" type of effort. There is certainly
nothing wrong with either approach, it's just better to have that
context. I can see how this would be useful in the general sense, just
not sure I can imagine any cases where it's something I would
absolutely use.

- James

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 22 May 2013 18:29:06 +0200

> I would like to have available a BER like encoding for JSON.

What is it that you like about BER?
(And, as you mentioned there are a couple, which BER do you like :-)?

> But this proposal seems to be something rather different.

We try to learn.

CBOR can sure do everything a "BER for JSON" could do.  And a little more.

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 22 May 2013 18:38:31 +0200

On May 22, 2013, at 18:27, James M Snell <[email-masked]> wrote:

> attached to some other piece of work

One motivation is that we need some concise binary form for CoRE work.
But we didn't think it was a good idea to define something that is limited to that, as long as we can keep the simplicity.
And it turned out that it wasn't too hard to make it useful for many applications where JSON comes to mind first and then you start longing for something binary.

Grüße, Carsten

---

From: Dave Cridland <[email-masked]>
Date: Wed, 22 May 2013 18:40:03 +0100

> (And, as you mentioned there are a couple, which BER do you like :-)?
There's only one BER, but there are also DER, CER, PER, XER, and probably
others I can't instantly recall.

> CBOR can sure do everything a "BER for JSON" could do.  And a little more.

I think that's the right approach.

My only concerns - not hills to die on, but worth discussion, I think - are
that:

 - I'd be keen to see a clean subset defined which always can be translated
to JSON. My gut feeling is that this would be useful.

 - I'm instantly put on guard by the notion that there are multiple
representations for the number 7. I would in general much prefer to see
only one. I admit this may not be possible, or simple - maps, for instance,
have no intrinsic ordering, enforcing an integer use the most compact
representation is awkward, and so on. But having a canonical representation
might well be missed later.

Dave.

---

From: Tony Finch <[email-masked]>
Date: Wed, 22 May 2013 18:42:01 +0100

Why not BSON or BJSON or UBJSON or MsgPack or ... ?

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Forties, Cromarty: East, veering southeast, 4 or 5, occasionally 6 at first.
Rough, becoming slight or moderate. Showers, rain at first. Moderate or good,
occasionally poor at first.

---

From: Dave Cridland <[email-masked]>
Date: Wed, 22 May 2013 18:43:12 +0100

> Why not BSON or BJSON or UBJSON or MsgPack or ... ?

I was wondering whether Paul was going to do a compatible variant called
PHOFF, actually.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 22 May 2013 13:45:58 -0400

> (And, as you mentioned there are a couple, which BER do you like :-)?

There is only one BER as far as I know. The problem is they also have DER,
XER etc. etc.

The worst of these is DER which was a terrible implementation of an idea
that was totally unnecessary in the first place. Canonicalization is an
unnecessary and broken idea. The idea that people would pull X.509v3 certs
apart and reconstitute them to check the signatures was silly. Nobody ever
did, nobody ever will.

The rationale for XML Dig sig canonicalization is a little better but
should have been taken as evidence that SOAP was broken and we needed to do
something different.

The main advantage of BER is that they are a known quantity and anyone
proposing to make changes needs to give reason.

But I would still subset to remove features not needed by JSON.

> CBOR can sure do everything a "BER for JSON" could do.  And a little more.

I would like no more, no less.

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 22 May 2013 10:55:27 -0700

On May 22, 2013, at 10:40 AM, Dave Cridland <[email-masked]> wrote:

> My only concerns - not hills to die on, but worth discussion, I think - are that:
> 
>  - I'd be keen to see a clean subset defined which always can be translated to JSON. My gut feeling is that this would be useful.

We tried for that in Section 4. If we can make that clearer or more precise, we'd be happy to.

>  - I'm instantly put on guard by the notion that there are multiple representations for the number 7.

That was my first reaction as well.

> I would in general much prefer to see only one. I admit this may not be possible, or simple - maps, for instance, have no intrinsic ordering, enforcing an integer use the most compact representation is awkward, and so on. But having a canonical representation might well be missed later.

We could add that as an option for the protocols that care about it. The set of rules is small, so it's not a big deal.

--Paul Hoffman

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 22 May 2013 10:57:33 -0700

> Why not BSON or BJSON or UBJSON or MsgPack or ... ?

Because they didn't meet the design objectives that we listed in Section 1. MessagePack had a lot of what we wanted, which is why we say "CBOR was inspired by MessagePack" in the acknowledgements.

--Paul Hoffman

---

From: Dave Crocker <[email-masked]>
Date: Wed, 22 May 2013 11:04:03 -0700

> Because they didn't meet the design objectives that we listed in Section 1. MessagePack had a lot of what we wanted, which is why we say "CBOR was inspired by MessagePack" in the acknowledgements.

A response with the semantics of "because we liked the one we chose 
better" seems less responsive than what I took the question to be asking.

I assume the nature of 'why' was for some discussion of the technical 
differences that justify the choice.

That's certainly what I'd be interested in seeing and am suggesting a 
summary of it be added to the document.

d/

---

From: Tony Finch <[email-masked]>
Date: Wed, 22 May 2013 19:16:24 +0100

On 22 May 2013, at 19:04, Dave Crocker <[email-masked]> wrote:
> 
> I assume the nature of 'why' was for some discussion of the technical differences that justify the choice.

Right. This is an over-populated niche, and there need to be compelling reasons to disregard existing specs with running code.

Tony.

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 22 May 2013 11:25:11 -0700

> Right. This is an over-populated niche, and there need to be compelling reasons to disregard existing specs with running code.

Sure, we can add an analysis of our design goals against existing formats as an appendix. 

--Paul Hoffman

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 22 May 2013 14:40:43 -0400

On Wed, May 22, 2013 at 2:16 PM, Tony Finch <[email-masked]> wrote:

> Right. This is an over-populated niche, and there need to be compelling
> reasons to disregard existing specs with running code.
>
> Tony.
>

Emphasis here being 'compelling'...

The whole problem with this space is that it is so easy for someone to make
a proposal. It doesn't really take much knowledge of the problem either.

I have a code generator for protocols. Give it an API signature and it
dumps out code to serialize and deserialise JSON. If people can agree on
one binary alternative to JSON I will add it to the generator. But I don't
plan to offer more than one.

https://sourceforge.net/projects/jsonschema/

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 22 May 2013 20:42:03 +0200

On May 22, 2013, at 20:04, Dave Crocker <[email-masked]> wrote:

> That's certainly what I'd be interested in seeing and am suggesting a summary of it be added to the document.

The summary of our objectives is in section 1.

A summary of all the formats that we looked at would be great for an encyclopedia, but not for the protocol specification.
(And, as I mentioned in prt

In 1980, when the specification for a new programming language came out that most people by now have forgotten, one major innovation was that it was accompanied by a "rationale" document.
This was much bigger than the specification.  And, somehow, when I read it, I had the impression that much of it was used to justify design decisions that were untenable.

Yes, we could document the rationale behind each of the about five design decisions that went into CBOR.
But doesn't the specification of something as simple as this speak for itself?

Grüße, Carsten

---

From: James M Snell <[email-masked]>
Date: Wed, 22 May 2013 11:43:54 -0700

Best bet: define the syntax, provide a couple of great open source
implementations, let people know about them.. then time will tell
whether it'll be worthwhile to standardize. There shouldn't be any
rush on getting an RFC done.

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 22 May 2013 20:55:37 +0200

>  - I'm instantly put on guard by the notion that there are multiple representations for the number 7.

Indeed, the lack of the "unique encoding" property creates heartburn for at least one author of the specification.
(We have gone to considerable length in getting unique encodings in CoAP options.  Even there, we didn't quite succeed.)

One good way to ensure unique encoding for unsigned integers is the o256 encoding documented in Appendix A.4 of http://tools.ietf.org/html/draft-bormann-coap-misc-25

This is just not the right way for CBOR.

> I would in general much prefer to see only one. I admit this may not be possible, or simple - maps, for instance, have no intrinsic ordering, enforcing an integer use the most compact representatio…

There is a tradeoff between the complexity of the serializer and that of the deserializer.
Complete canonicalization puts the onus on the serializer; e.g., you have to sort the keys of a map before encoding it.
On the other hand, the deserializer knows exactly what to expect when -- if you have seen "b", you know there won't be an "a".

(There is also the use of c14n for signature.  I don't want to go there...)

I think we have the right balance between serializer and deserializer here -- if you have a very constrained serializer, it may be expedient to leave out code for always choosing the shortest representation.  The deserializer already has to have the ability to handle longer representations, so the net value of this design decision is positive.

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 22 May 2013 21:18:25 +0200

> rush on getting an RFC done.

Of course, the degree of success of a technical specification is dominated by external effects, such as the highly random process in which it is picked up by implementers.

What I am trying to do here is just provide a very good specification.

Good marketing, open-source reference implementations, TED speaking engagements, articles in Network World, pickup by CPU vendors that put in useful instructions for generating/parsing it, cloud services, funding for related research etc. (in other words, mindshare) comes after that.  

However, this is a specification that could (and should) be used by other specifications that the IETF and other organizations will be creating.  So saying "you can have the RFC after the first TED talk" is not helping those other specifications along.

(This is not an argument against running code, just against "let the market decide on whatever and then standardize".)

Grüße, Carsten

---

From: Dave Crocker <[email-masked]>
Date: Wed, 22 May 2013 12:36:42 -0700

> Good marketing, open-source reference implementations, TED speaking engagements, articles in Network World, pickup by CPU vendors that put in useful instructions for generating/parsing it, cloud se…

don't forget to give away CBOR buttons in Berlin, although you might 
have to explain to folks that it's not your login.

d/

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 22 May 2013 13:20:34 -0700

On May 22, 2013, at 11:43 AM, James M Snell <[email-masked]> wrote:

> whether it'll be worthwhile to standardize.

Fully disagree. As a great recent counter-example, see what is happening in the MessagePack world which is following exactly what you suggest, and there is a lot of bad feelings.

> There shouldn't be any

There is no rush to get it done, but there is no reason to not publish it once it is stable and has gotten a lot of good input.

--Paul Hoffman

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 22 May 2013 13:23:38 -0700

> (There is also the use of c14n for signature.  I don't want to go there...)

And here y'all get to see the two authors disagree. (Don't worry, there has been plenty of that in the past few months.) The -01 will have a section on canonicalization that will say the same thing as much of Section 3: if a protocol wants to take on canonicalization it can, but it is not required by the format spec.

--Paul Hoffman

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 22 May 2013 22:39:01 +0200

> Why not BSON or BJSON or UBJSON or MsgPack or ... ?

Good question, and one that deserves to be answered (maybe not in a formal document).  
But, as Dave has alluded to, any such discussion MUST start with RFC 713.
I urge you to read it if you are interested in this space.

What have we learned in the 37 years since RFC 713?
Less than one might think.

I think the most important observation was that it is not the character that is the fundamental data type, but the text string.
We also have a pretty good and well accepted encoding for it: UTF-8.  So that is one centerpiece of CBOR.

The other sea change that has happened is that a number of S-expression based formats have come and gone, but JSON seems to be here to stay.
So JSON has been a strong inspiration, including the map (dictionary, hash, table, ...) type called "object" in JSON.

In contrast to RFC 713's well thought-out variable-size scheme, we did a fixed partitioning of the type byte (3 + 5 bits) because that is easier to handle both in high-speed and in very constrained implementations (of course, what is very constrained today might have been quite comfortable in 1976, except that we have to cram in so much more functionality today).

We are also using an item count and not a byte count for structured (non-atomic in RFC 713 terms) items.

There is no equivalent in CBOR to RFC 713's b-REPEAT -- we entirely left out compression mechanisms.

Grüße, Carsten

---

From: "Manger, James H" <[email-masked]>
Date: Thu, 23 May 2013 11:49:19 +1000

CBOR [draft-bormann-cbor] looks great.

The only big problem I see is that sizes have to be known upfront. You need to know how many bytes, array elements, or map pairs you have BEFORE you start encoding them. This really hinders streaming content. JSON does not have this limitation so CBOR should not introduce it. Removing this limitation would better meet objective 5: "be applicable to both constrained nodes and high-volume applications".

This should not be hard to solve, nor to implement.
For instance define initial bytes for "Array", "Map", "Byte array in parts", and "End". An array is the items between "Array" and "End". A map is an even number of items between "Map" and "End". There would be no need for the 64 initial bytes assigned to known-size versions of arrays and maps.

The 32 initial bytes for known-length byte arrays would remain. The "Byte array in parts" initial byte would be an additional option. Between "Byte array in parts" and "End" can be any number of known-length byte array items.

--
James Manger

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 22 May 2013 22:04:41 -0400

On Wed, May 22, 2013 at 9:49 PM, Manger, James H <
[email-masked]> wrote:

> The only big problem I see is that sizes have to be known upfront. You
> need to know how many bytes, array elements, or map pairs you have BEFORE
> you start encoding them. This really hinders streaming content. JSON does
> not have this limitation so CBOR should not introduce it. Removing this
> limitation would better meet objective 5: "be applicable to both
> constrained nodes and high-volume applications".
>
> For instance define initial bytes for "Array", "Map", "Byte array in
> parts", and "End". An array is the items between "Array" and "End". A map
> is an even number of items between "Map" and "End". There would be no need
> for the 64 initial bytes assigned to known-size versions of arrays and maps.
>
> The 32 initial bytes for known-length byte arrays would remain. The "Byte
> array in parts" initial byte would be an additional option. Between "Byte
> array in parts" and "End" can be any number of known-length byte array
> items.
>

That is the reason that DER encoding is such a major pain. Only DER makes
it even harder as the internal sizes are indeterminate...

It has to be possible to serialize and deserialize without any additional
state or pre-computation other than maintaining the position in the tree
walk.

Having to know the number of objects in a list would make transcoding from
JSON to binary form a real pig. It would be necessary to buffer objects
before you can push out the first byte.

This is the type of issue that is critical for me for an on the wire
format. Being able to index into specific content is the sort of thing I
might care about in a storage format. I might care about things like byte
alignment as well. Or maybe not.

---

From: Nico Williams <[email-masked]>
Date: Thu, 23 May 2013 00:27:27 -0500

Thoughts:

 - if we're going to have an RFC on JSON, might as well have one on
some binary encoding of JSON

 - yet another encoding.  yay.

 - this encoding does suffice for encoding JSON

 - this encoding can go beyond JSON -- this is good, but a strict
JSON-equivalent subset should be specified

 - yeah, multiple ways of encoding a given value suck; in particular,
for the JSON-equivalent subset, there should be only doubles, no
floats, no integers

 - might as well have a type for "octet string" -- no base64 encoding please

Nico
--

---

From: "Paul E. Jones" <[email-masked]>
Date: Thu, 23 May 2013 02:47:18 -0400

There are actually several XML binary encodings defined today.  But I assume
the one to which you refer is the one in the W3C called Efficient XML
Interchange (EXI).  I looked into that and compared the various compression
results against a vanilla compression algorithm (namely, zlib).  I recorded
the results here:

http://www.packetizer.com/xml/papers/xml_compression_results.pdf

Basically, using EXI would reduce the size of messages and the more the
encoder knew about the schema, the better it would do... for small messages.
What was interesting is that the results I got rather consistently showed
that zlib would outperform most other binary encodings for messages of about
1K or larger.  However, l don't really want an encoding that requires the
encoder to be aware of a specific schema.  It might change tomorrow or next
year.

Zlib worked well since much of the data being compressed was, of course,
text.  And text compresses pretty well.  It just has a fairly large
footprint that is visible with smaller messages.

The "no schema" variant of EXI did pretty well on small messages, so I think
if COBR can achieve compression at a comparable rate or better, that would
be great.  Now, it was mentioned elsewhere in the thread (and I'll admit
I've not read the draft) that streaming might be an issue.  Steaming is not
an issue with EXI (at least not with the "no schema" variant), as new tags
are encoded into the binary output as they are first seen.

ASN.1 PER did an really good job at compressing data.  Some of the ASN.1
syntax can be a bit hairy, but the encoding was small.  And it worked well
since the data it was encoding was binary data, too.  With most of the newer
protocols, binary optimization is a challenge because the protocols use so
much text (media types, URLs, etc.)  So, if COBR will exist mainly to
compress text data, it will not compress too well.  I assume it is being
introduced to produce a tight wire encoding of (largely) binary data.

Paul

From: [email-masked] [mailto:[email-masked]]
On Behalf Of Phillip Hallam-Baker
Sent: Wednesday, May 22, 2013 12:14 PM
To: Paul Hoffman
Cc: IETF Apps Discuss
Subject: Re: [apps-discuss] Concise Binary Object Representation (CBOR)

Meh...

I think we can all agree that *A* binary format would be useful and that two
dozen are not useful. The whole problem is how to convince people to use one
particular scheme rather than the alternatives.

 

 

On Wed, May 22, 2013 at 12:00 PM, Paul Hoffman <[email-masked]>
wrote:

We did ours differently than others: we started with an ordered set of
design goals and stuck to them. That means that our early arguments about
changes / bikeshedcolors had something we could compare against, and it make
coming up with a solution much more tractable.

There are many contexts. It might be useful in CORE and other
constrained-environment work. It might be useful for the next protocols that
need a compact binary representation and the WG doesn't want to fight over
the specific features. If the design goals fit your needs (and we actually
met them...), then it's useful to have it be widely available.

--Paul Hoffman

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 23 May 2013 09:15:08 +0200

> for the JSON-equivalent subset, there should be only doubles, no

This is about like saying there should only ever be 64-bit integers (or 128-bit, while we are at it).
The serializer can choose one of multiple representations (or, if it is expedient, it can stick with one).
Like with different size of integers, there is little effort in a deserializer to support different sizes of floats.

> - might as well have a type for "octet string" -- no base64 encoding please

Yes, we are just not using the liturgical name "octet string" :-)

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 23 May 2013 09:50:37 +0200

On May 23, 2013, at 03:49, "Manger, James H" <[email-masked]> wrote:

> sizes have to be known upfront

Indeed.  The old "counted vs. delimited" debate.

JSON is delimited because is HAS TO BE -- counted is a non-starter for text formats (as amply demonstrated by FORTAN Hollerith data).

CBOR is counted because that is the best fit for a binary format.
Fast deserializers really benefit from knowing counts upfront (and that also helps with constrained deserializers).
(The number of things they will have to create, e.g., the number of data items; not the byte size of their representation -- counting bytes here was a mistake RFC 731 and ASN.1 BER shared and that creates much pain in BER.)
With a byte-oriented encoding, counting also happens to require fewer bytes than delimiting for small counts.
So if the serializer is in a position to provide count information in time, we should use it.

CBOR supports streaming by having self-delimited data items -- you can send many of these in sequence for a stream (as in, say, an XMPP-like protocol).

So streaming *of* data items is not a problem with CBOR.
Now let's look at "streaming" *within* a data item.

The actual need for this is a fringe case, but one that is worth looking at.
The only significant use case I know is a state-limited (on-the-fly) converter from a delimited format (such as JSON) to CBOR.
(Unfortunately, that is not just a use case in gateways; using CBOR as a plug-in replacement for JSON may saddle one with an API that makes the CBOR serializer such a converter.)
Putting together an aquarium when being handed fish sticks.

Anything that addresses this specific use case needs to be a hybrid.
(Purely delimited is possible for binary, but tends to be more taxing on the deserializer; it also often requires additional work at the serializer to enable data transparency.)
I tried to avoid designing a hybrid.
Counted is the way things have been moving to in much of CS for the last 10, 15 years.

As you say, delimiters (push/pop, or maybe some form of continuation scheme) can be easily hacked into CBOR, the only victim being its architecture.
Should we?

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 23 May 2013 10:11:36 +0200

On May 23, 2013, at 08:47, "Paul E. Jones" <[email-masked]> wrote:

> compression

CBOR is not about compression.
It is about compact encoding, achieving reasonable message sizes with reasonable effort.
In many cases, compression is premature optimization.
CBOR only avoids premature pessimization.

Compressed (e.g., via deflate/gzip) JSON works very well for a large number of applications.
CBOR is about generating and consuming data streams with much less complexity.
Complexity
-- eats CPU and thus energy,
-- creates interoperability problems and other bugs,
-- may not fit into constrained systems.
But these considerations may simply not be the overriding ones for many applications, and LZ77 can achieve efficiencies that a simple serializer can't, so absolutely go ahead and deflate JSON for these.

EXI is not a bad serializer.  
There are a number of weird design decisions in there that I would prefer to avoid.
More importantly, EXI is based on an XML data model; the JSON model is what I'm interested in.
EXI also has to be servant to two masters: schema-informed and schema-less.
Finally, it has a medium (but not severe) case of design by committee.
All this creates a bit more complexity than I like, but, again, it is not nearly as bad as it could be.

Grüße, Carsten

---

From: Tony Finch <[email-masked]>
Date: Thu, 23 May 2013 10:17:06 +0100

Paul Hoffman <[email-masked]> wrote:
> Fully disagree. As a great recent counter-example, see what is happening
> in the MessagePack world which is following exactly what you suggest,
> and there is a lot of bad feelings.

Do you have a link we can look at to find out more?

Tony.

---

From: "Manger, James H" <[email-masked]>
Date: Thu, 23 May 2013 23:15:21 +1000

> JSON is delimited because is HAS TO BE -- counted is a non-starter for
> text formats (as amply demonstrated by FORTAN Hollerith data).
> 
> Fast deserializers really benefit from knowing counts upfront (and that
> also helps with constrained deserializers).

Is this because the deserializer can allocate exactly the right memory for an array (of pointers to elements), instead of having to guess then resize as necessary?

> (The number of things they will have to create, e.g., the number of
> data items; not the byte size of their representation -- counting bytes
> here was a mistake RFC 731 and ASN.1 BER shared and that creates much
> pain in BER.)

Ok. I can see than an item count can avoid *some* of the difficulties with a count of encoded bytes.

> bytes than delimiting for small counts.

Save 1 byte per map and array that has fewer than 28 items. Sounds nice, but not that compelling.

> CBOR supports streaming by having self-delimited data items -- you can
> send many of these in sequence for a stream (as in, say, an XMPP-like
> protocol).
> 

Supporting streaming -- but only at the top layer of a protocol -- doesn't sound good. If it is useful at a "top-layer" surely it is useful at other layers as well. Protocols are often embedded in other protocols. Once you have something that relies on a stream *of* data items, and you want to embed it in another layer you have to stop using CBOR. That's bad.

> The only significant use case I know is a state-limited (on-the-fly)
> converter from a delimited format (such as JSON) to CBOR.
> (Unfortunately, that is not just a use case in gateways; using CBOR as
> a plug-in replacement for JSON may saddle one with an API that makes
> the CBOR serializer such a converter.)

This doesn't feel like a "fringe case". Converting from JSON is one of the listed objectives (#6). The draft has a section on how to do it.

It is generally only for potentially large things that you need to bother with something like CBOR for efficiency, instead of staying with JSON. It is these large things that are going to have unknown sizes at the start.

Plenty of languages pass around iterators, instead of arrays or maps, because they are a useful abstraction that covers more situations -- but they don't give you a size up front.

> (Purely delimited is possible for binary, but tends to be more taxing
> on the deserializer; it also often requires additional work at the
> serializer to enable data transparency.)

What does "enable data transparency" mean?

> As you say, delimiters (push/pop, or maybe some form of continuation
> scheme) can be easily hacked into CBOR, the only victim being its
> architecture.
> Should we?

Yes.

I'm not convinced that situations where you don't know the size of a byte array, text, array, or map up front are so rare that CBOR can ignore them.

I am not familiar enough with squeezing performance from a deserializer to judge whether a hybrid is worth the complexity cost over just delimiters for maps & arrays.

--
James Manger

---

From: Dave Crocker <[email-masked]>
Date: Thu, 23 May 2013 06:52:47 -0700

> More importantly, EXI is based on an XML data model; the JSON model is what I'm interested in.

Beginner's question:

      How does/should the differences in data model affect the choices 
in encoding scheme?  I'm looking for the (relatively) short answer.

Tnx.

d/

---

From: Dave Cridland <[email-masked]>
Date: Thu, 23 May 2013 15:21:34 +0100

On Thu, May 23, 2013 at 2:52 PM, Dave Crocker <[email-masked]> wrote:

>      How does/should the differences in data model affect the choices in
> encoding scheme?  I'm looking for the (relatively) short answer.
>

EXI makes certain assumptions about the data model in order to achieve the
most efficient encoding. EXI specifically can take into account not only
the generalized DOM, but also the schema of the XML encoded. Any encoding
scheme with domain knowledge embedded in it, as it were, can get more
efficient encoding (since it's effectively encoding that domain information
implicitly in its mere use rather than transmitting the model explicitly).
Of course, it doesn't mean that any domain-specific encoding will
automatically be better than a generalized one...

Another example, and one that you may be familiar with, is the way that
IMAP set syntax uses its knowledge of the data domain to make very compact
representations, which can be seen clearly in the difference between a
standard RFC 3501 SEARCH response and the RFC 4731 ESEARCH response, which
(can) carry the same information in different representations - the latter
using considerably fewer octets on the wire in all but degenerate cases.

In any case, this means that given a particular data model, selecting a
suitably apt encoding scheme will yield better results.

(This all said, you could pick an XML representation of the JSON model, and
then EXI encode that one, since the DOM is a superset of the JSON model - I
think. Not saying that's even remotely sensible...)

Dave.

---

From: Dave Crocker <[email-masked]>
Date: Thu, 23 May 2013 07:49:42 -0700

On 5/23/2013 7:21 AM, Dave Cridland wrote:
> EXI makes certain assumptions about the data model in order to achieve
> the most efficient encoding. EXI specifically can take into account not
> only the generalized DOM, but also the schema of the XML encoded.

What does that mean in terms of specifics?  For example, what kind of 
assumptions?  The above is such a general statement, there's no way to 
know what it translates into, in terms of encoding style choices.

d/

---

From: Dave Cridland <[email-masked]>
Date: Thu, 23 May 2013 15:59:56 +0100

On Thu, May 23, 2013 at 3:49 PM, Dave Crocker <[email-masked]> wrote:

> assumptions?  The above is such a general statement, there's no way to know
> what it translates into, in terms of encoding style choices.
>
>
To switch example, JSON has no encoding for sets, or binary data, because
neither exist in the model it's encoding. Because they don't exist, there's
no need to encode them - if you wanted to encode them (because your source
model differs from that assumed), you'd need to do something else (like
base64 encoding). This naturally reduces efficiency.

CBOR clearly has a superset of this model - so it does have binary data,
but still does not have sets, for instance.

Dave.

---

From: Dave Crocker <[email-masked]>
Date: Thu, 23 May 2013 08:01:00 -0700

On 5/23/2013 7:59 AM, Dave Cridland wrote:
> To switch example, JSON has no encoding for sets, or binary data,
> because neither exist in the model it's encoding. Because they don't
> exist, there's no need to encode them - if you wanted to encode them
> (because your source model differs from that assumed), you'd need to do
> something else (like base64 encoding). This naturally reduces efficiency.

that's pretty specific.  thanks.

d/

---

From: Nico Williams <[email-masked]>
Date: Thu, 23 May 2013 10:35:33 -0500

> This is about like saying there should only ever be 64-bit integers (or 128-bit, while we are at it).

ECMAScript specifically only allows that.  That's where the
restriction comes from.

Many a ECMAScript and/or JSON implementation use NaN coding, which is
probably why this restriction is there in the first place (though
that's just a guess; I'm sure there's a list archive somewhere with a
lengthy discussion I could check; I don't care).

> Like with different size of integers, there is little effort in a deserializer to support different sizes of floats.

I wouldn't insist on having a canonical representation.  I would
prefer to have a canonical representation if possible.  The biggest
issue would be the need to sort object keys, which I'd not insist on
(but it's nice to have the option to, and many a JSON encoder provides
this).

> Yes, we are just not using the liturgical name "octet string" :-)

Yes it's there and I missed it?

Nico
--

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 23 May 2013 11:41:26 -0400

>  - might as well have a type for "octet string" -- no base64 encoding

+1 to Nico.

The only types I have found a need for in my JSON schema are:

Integer
Float
String (UTF8)
DateTime   (As RFC 3339 format string)
Binary (as base64 encoded string)
List (X)  - JSON List

I do think that it is worth calling out DateTime and Binary for special
treatment because they both appear very frequently in code. I have not
found a need for any other special format.

For Integers, I only use double for representation but that is a design
choice that does not affect bits on the wire. For a binary format I would
prefer that like JSON, the wire format be neutral with respect to the code
representation.

Since most of the proposals are of the form <type> <data> where <type> and
we only have 7 fundamental types it is quite easy to make the type octet
combine the type and length information. So if the integer fits in 2 bytes
then just use 2 bytes and mark the length in the type octet, the number 256
could be represented as:

X2 01 00

I would use twos compliment for all values and so a 64 bit unsigned integer
could potentially have 9 octets but that would be a rare event.

A side benefit to this approach is that big numbers are easily supported.

Representing floats is a lot harder than the specs I have written seem to
understand. In particular the main reason I would want a binary format for
JSON is that it is NOT possible to round trip floats from decimal format to
binary without special care.

So the main reason for using a binary JSON is going to be not introducing
the errors, possibly cumulative errors that round tripping from decimal to
binary introduces.

So we do need to have 32 bit and 64 it floats.

---

From: Nico Williams <[email-masked]>
Date: Thu, 23 May 2013 11:13:46 -0500

Also, I believe this is the fifth binary encoding of JSON proposed
thus far.  Can we list the ones that have been proposed thus far?

Here's a [no-doubt partial] list:

 - BSON
 - MsgPack
 - Smile
 - UBJSON
 - CBOR

That's just the ones I recall seeing before or which I found in 30
seconds of searching.

Some of these are used in actual protocols right now.

An analysis of these might be nice.

(While you're at that, and considering variable length encodings of
things... SQLite4 has some really neat variable length encodings of
all its data types, specifically designed to make sorting via memcmp()
possible.  I'm not sure what relevance that might have here, but it
should be of interest to anyone designing variable length encodings of
anything.)

I do agree that we need a schema-less encoding (which means we can't
get as good as PER and friends).  We probably don't need a
schema-capable encoding for JSON (though that might be nice).  We do
need schema-aware encodings for some protocols, but we have plenty
already, and none are nor need to be JSONish.

(I've wanted to define a JSON Encoding Rules for ASN.1, primarily to
make debugging apps that use ASN.1-based protocols easier (dump PDUs
as ASN.1).)

Nico
--

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 23 May 2013 18:23:12 +0200

> ECMAScript specifically only allows that.  That's where the

JSON ≠ JavaScript.  
JavaScript's numbers are IEEE 754 double precision.
That is not at all a property of JSON.  
(It *is* something you care about if one or two of the peers speaking JSON actually are written in JavaScript and you are using JavaScript primitives to operate on the JSON.)

> lengthy discussion I could check; I don't care).

Now you lost me -- JSON doesn't have NaNs.
(Another thing that CBOR has that you can't use in a JSON subset.)

> (but it's nice to have the option to, and many a JSON encoder provides

I think Paul is working on c14n for CBOR.
As I said, I don't want to go there...
(If you think you need c14n, you probably have a bug in your architecture.)

> Yes it's there and I missed it?

Yes, we call them "byte strings" (major type 2).
(They are actually the main reason I started work on CBOR.)

Grüße, Carsten

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 23 May 2013 09:28:54 -0700

On May 23, 2013, at 9:13 AM, Nico Williams <[email-masked]> wrote:

> thus far.  

A note on terminology: CBOR is not a "binary encoding of JSON". It has a design goal of supporting all JSON data types for conversion to and from JSON. That design goal (which we think we have met) is far down the list of design goals.

> Can we list the ones that have been proposed thus far?
> 

As we said yesterday, we will have an appendix of other binary formats, saying how they do and don't meet the design goals for CBOR. That should help people decide whether reusing one of the existing formats is more worthwhile than CBOR.

--Paul Hoffman

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 23 May 2013 18:38:09 +0200

> Also, I believe this is the fifth binary encoding of JSON proposed

1) CBOR is not a "binary encoding of JSON".
(It can be used as one, though.)

2) There are many more proposals in this space, and it is hard to draw a line.
E.g. Google protocol buffers is a prominent example.
Bencode is a great one, too.

> An analysis of these might be nice.

Yes.  Find a grad student :-)

Of course, I did my own analysis, but I didn't see a need to write it up.
(More specifically, I didn't see a funding agency interested in this part of reality.
But I'm willing to be surprised :-)

> should be of interest to anyone designing variable length encodings of

If you are looking for a neat variable length encoding of integers designed to sort well, look no further than Appendix A.4 of draft-bormann-coap-misc.
But CBOR wasn't trying to be neat, it is trying to be practical.

> already, and none are nor need to be JSONish.

I think we are in good agreement here.

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 23 May 2013 18:53:56 +0200

> Binary (as base64 encoded string)

CBOR is a pretty good match then (we also have a map/dictionary/table/JSON object).
(There is never a need to base-64 encode anything in CBOR.)

> Since most of the proposals are of the form <type> <data> where <type> and we only have 7 fundamental types it is quite easy to make the type octet combine the type and length information. So if the integer fits in 2 bytes then just use 2 bytes and mark the length in the type octet, the number 256 could be represented as:
> 
> X2 01 00

Yep.
(CBOR has a slightly different encoding of the first byte to be able to encode the number right there if it fits).

> I would use twos compliment for all values and so a 64 bit unsigned integer could potentially have 9 octets but that would be a rare event.

CBOR uses two different encoding types for unsigned and negative numbers.  See Figure 2 for an easy way to convert signed integers to that.

> A side benefit to this approach is that big numbers are easily supported.

CBOR also has a representation for big (> 64-bit) numbers (based on the byte string).

> Representing floats is a lot harder than the specs I have written seem to understand. In particular the main reason I would want a binary format for JSON is that it is NOT possible to round trip floats from decimal format to binary without special care.

Representing floats is rather easy -- IEEE 754 is well-documented.  So CBOR uses the three main binary floating point formats from that (Half, Single, Double).
If you do need (negative) base-10 exponents, we have a special Decimal fraction (inspired by YANG and EXI), trying not to use decimal mantissae.

> So the main reason for using a binary JSON is going to be not introducing the errors, possibly cumulative errors that round tripping from decimal to binary introduces.

That is one good reason (not the main one I'm interested in).

> So we do need to have 32 bit and 64 it floats.

Strictly, that is not needed, as every 32-bit float can be easily converted into a 64-bit float.
But the inverse (going down from 64 to 32 to 16, and checking whether that is lossless) is also easy.

Grüße, Carsten

---

From: Nico Williams <[email-masked]>
Date: Thu, 23 May 2013 12:02:23 -0500

>> ECMAScript specifically only allows that.  That's where the

I'm aware.  But defining protocols that can't be implemented in JS
because they use numbers that JS can't represent seems problematic.

> (Another thing that CBOR has that you can't use in a JSON subset.)

NaN coding is an implementation technique where all values are
internally represented as doubles, with signal bits of NaNs used to
store pointers (for strings, arrays, objects) and representations of
null, true, and false.

> (They are actually the main reason I started work on CBOR.)

OK, thanks.

---

From: Tony Finch <[email-masked]>
Date: Thu, 23 May 2013 18:16:33 +0100

> (Another thing that CBOR has that you can't use in a JSON subset.)

It is probably worth putting something in the security considerations
about NaNs. See for instance
http://article.gmane.org/gmane.comp.lang.lua.general/70705
and http://trac.webkit.org/changeset/64706

Also, the multiple forms of encoding for integers mean that applications
have to be careful how they handle range and equality checks - there's an
analogous problem to the loose UTF-8 parser security bugs.

Tony.

---

From: Nico Williams <[email-masked]>
Date: Thu, 23 May 2013 12:19:54 -0500

> (It can be used as one, though.)

The superset of JSON in CBOR is not in itself appealing enough, I
think.  We already have a *lot* of binary encodings.

It's only the connection to JSON that makes this appealing.  You might
disagree, but you're probably biased :)  You're asking that we
standardize this though, so you should do something to justify one
more encoding in a sea of encodings.

> E.g. Google protocol buffers is a prominent example.

But it requires a schema.  (And what the heck was so wrong with PER?
But whatever.)

> Yes.  Find a grad student :-)

I think it falls on you to provide some justification for this.  It
wouldn't be hard, but you might find that an existing, *deployed*
encoding is good enough (or maybe you wouldn't but we would).  I think
I'd rather standardize a good-enough encoding that's been deployed
than create a new one, but maybe the improvements CBOR brings are
compelling enough.  Either way you should definitely provide some
analysis.

> Of course, I did my own analysis, but I didn't see a need to write it up.

That's... odd.  You need consensus.  Why wouldn't your analysis help you get it?

> But I'm willing to be surprised :-)

Do you intend to use CBOR for any protocols you want standardized at
the IETF?  Or are you just hoping others will?

Nico
--

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 23 May 2013 11:06:38 -0700

> because they use numbers that JS can't represent seems problematic.

As I said earlier on this thread, there is already an implementation in JavaScript.

--Paul Hoffman

---

From: James M Snell <[email-masked]>
Date: Thu, 23 May 2013 11:35:40 -0700

That's well and good, from everything I've seen so far in this thread,
the collective majority opinion can be summarized as "Ugh... Groan..
Another one? Really?"  ... That being said, there's really nothing
*technically* wrong with CBOR that I can spot and there are a number
of very nice features included. Posting the I-D is good, and having
implementations available is good, but having a running debate of the
technical merits of CBOR vs. Everything Else is pointless. It would be
great to see a comparison of CBOR to other previous attempts in this
space added to the next iteration of the draft.

As far as the I-D being an apps area wg work item... I'm not sure I
personally see the value in that yet. There's nothing wrong with
handling it as an individual, independent submission for a few
iterations and just seeing how support shapes up around it (if it
does). If the value of this particular binary encoding becomes readily
apparent to everyone later on, then we can go on from there. (just my
opinion of course).

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 23 May 2013 12:24:30 -0700

On May 23, 2013, at 11:35 AM, James M Snell <[email-masked]> wrote:

> Another one? Really?"  

Just a note that this is one of the only ones that people are groaning about that has an Internet Draft and might go through the IETF consensus process. Carsten and I (maybe naively) thought that doing this in this environment, instead of say posting ephemeral specs on a web page and not having it be clear where the community fit it, was a good thing.

> ... That being said, there's really nothing
> of very nice features included.

So far, the one technical change that falls within our design goals that we have heard in the past two days is the possible addition of, or changing to, streaming for arrays and maps. Like Carsten, I'm still unsure of the tradeoff of "can also do streaming" versus "having two ways of doing the same thing has lots of downsides". FWIW, I disagree about needing it for converting from JSON to CBOR, since holding a counter of items from the JSON object that is prepended to the eventual CBOR item is trivial.

> Posting the I-D is good, and having

Am doing so. And of course, everyone will agree with the criteria for the comparisons! :-)

> does).

That would work too. Some ADs prefer things like this to come from WGs, some are fine with individual submission; thus our question in the first message in the thread.

--Paul Hoffman

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 23 May 2013 15:31:48 -0400

Completely off topic here but raised by the discussion. JSON needs to
address NaN encoding.

NaN is essential for exchange of numeric data in scientific applications.
JSON is going to be used in those applications. It would be better to have
one way to use JSON in scientific applications rather than have different
work arounds emerge.

There is an ongoing effort to document JSON, this should be included in
those efforts.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 23 May 2013 15:42:17 -0400

> Some of these are used in actual protocols right now.

+1

I don't see any particular reason to favor this new proposal.

> already, and none are nor need to be JSONish.

What I think you intend here is that the encoding should not be
schema-aware. Whether or not you have a formal schema is not the issue
here, the issue is whether it is necessary to know the schema to make sense
of the wire format.

One of the main reasons that JSON and XML encodings are attractive is that
you can look at the wire format and make sense of them. ASN.1 BER encoding
can also be read with an appropriate dump tool. It has to be possible to do
the same with Binary JSON.

Some of the compact XML representations used knowledge of the schema to
improve packing density and some could be read using a dump tool but
required knowledge of the mapping from the text representation to the
binary to be able to convert from one to the other.

I think it is clear that any JSON binary coding has to be readable without
knowledge of a schema and that a non-schema aware tool must be able to
convert between the two formats.

Insisting on a single representation for a number seems much less important
to me. Whether 7 is represented in one, two, four or eight bytes seems to
me to be something that can be left to the implementations.

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 23 May 2013 21:55:55 +0200

On May 23, 2013, at 21:31, Phillip Hallam-Baker <[email-masked]> wrote:

> Completely off topic here but raised by the discussion. JSON needs to address NaN encoding.
> 
> NaN is essential for exchange of numeric data in scientific applications. JSON is going to be used in those applications. It would be better to have one way to use JSON in scientific applications rather than have different work arounds emerge.

The applications I have seen trying to address the problem have simply used null to encode NaN.
There is rarely a case when you need to be able to express both null and NaN in one place (use CBOR, then :-).

A bigger problem is -Infinity/Infinity, but similar hacks can be used there, too (say, use false/true, or stringify).

> There is an ongoing effort to document JSON, this should be included in those efforts.

JSON MUST NOT change, so I don't think I agree with that (neither does the proposed charter of the WG).

Grüße, Carsten

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 23 May 2013 16:08:31 -0400

On Thu, May 23, 2013 at 3:55 PM, Carsten Bormann <[email-masked]> wrote:

> > NaN is essential for exchange of numeric data in scientific
> applications. JSON is going to be used in those applications. It would be
> better to have one way to use JSON in scientific applications rather than
> have different work arounds emerge.
>

There are many work-arounds possible. A standard should only have one.

> A bigger problem is -Infinity/Infinity, but similar hacks can be used
> there, too (say, use false/true, or stringify).
>

I don't think we have ever accepted a standard that declares it can't be
changed. We have certainly changed all the ones we started with. Even ASCII
was changed.

There are several ways that NaN and +/- infinity can be introduced without
changing JSON syntax. The point is there should only be one way that code
does so.

---

From: Nico Williams <[email-masked]>
Date: Thu, 23 May 2013 15:07:40 -0500

On Thu, May 23, 2013 at 2:42 PM, Phillip Hallam-Baker <[email-masked]> wrote:
> schema-aware. Whether or not you have a formal schema is not the issue here,
> the issue is whether it is necessary to know the schema to make sense of the
> wire format.

Indeed, I meant that the encoding doesn't require a schema.  A schema
may still be used by the app, of course, but the codec has no
knowledge of if, and in particular the decoder cannot be required to
have knowledge of the schema.

> the same with Binary JSON.

Well, they make some sense.  Documentation really helps though :)

(And BER... can be dumped, but when implicit tagging is used you lose
plenty of type metadata and have to resort to heuristics.)

> required knowledge of the mapping from the text representation to the binary
> to be able to convert from one to the other.

Right, like FastInfoSet (which uses PER).

> convert between the two formats.

Yes.  If we need further compression then use zlib or similar.

> to me. Whether 7 is represented in one, two, four or eight bytes seems to me
> to be something that can be left to the implementations.

I agree that it's less important, and I won't insist on it.  It's just
a mild irritant to not have c14n, but maybe it's just me and I should
get over it for good.

Nico
--

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 23 May 2013 16:14:18 -0400

On Thu, May 23, 2013 at 3:24 PM, Paul Hoffman <[email-masked]> wrote:

> Just a note that this is one of the only ones that people are groaning
> about that has an Internet Draft and might go through the IETF consensus
> process. Carsten and I (maybe naively) thought that doing this in this
> environment, instead of say posting ephemeral specs on a web page and not
> having it be clear where the community fit it, was a good thing.

I don't remember you being appointed to address the issue.

Since almost all the response to your proposal has been that people don't
like your design choices, and since you make it abundantly clear that you
are not interested in our input, I can't quite see where a consensus is
going to come. Unless that is the consensus is that your proposal sucks.

I would certainly object to a format that was counted being granted any
degree of IETF recognition lest someone try to force me to use it in the
future.

My experience of coding ASN.1 DER (in C and javascript) leads me to reject
any counted scheme as a non starter.

---

From: Nico Williams <[email-masked]>
Date: Thu, 23 May 2013 15:13:33 -0500

> Just a note that this is one of the only ones that people are groaning about that has an Internet Draft and might go through the IETF consensus process. Carsten and I (maybe naively) thought that d…

There's a huge world outside the IETF.  The folks working on Simple
and BSON, to take two examples, may not care about bringing their work
to us, but then, they may also be annoyed by our blindsiding them with
a standards-track binary JSON (I know, CBOR's a superset) encoding
that's not interoperable with theirs.

In particular, if there's wide deployment of one of them, why
shouldn't we pick one of them?

Should we not invite the wider JSON community to this discussion?

Nico
--

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 23 May 2013 16:45:41 -0400

> plenty of type metadata and have to resort to heuristics.)

I forgot about implicit tagging but that is of course the reason that
implicit tagging in ASN.1 is evil.

> a mild irritant to not have c14n, but maybe it's just me and I should

I think we really have to get over the C14n fetish. It forces us to do
things like sort tags into order before serialization. I have been doing
digital signatures and network security for 20 years now and I have never
once come across a situation where canonicalization would add value except
for situations like DKIM where (1) it is necessary to work round legacy
infrastructure with obnoxious behavior and (2) the ability to verify a
signature is not absolutely critical.

There are only really two cases of interest for authenticating data. The
first is when data is in motion and is passing through gateways and other
intermediaries that might change it. While it is in theory possible that a
gateway might change the data in a way that modifies the bits on the wire
without making semantic changes I have yet to see such a gateway in
practice and what would be the point? Email gateways that perform line
wrapping of text should have been strangled at birth and so should their
XML and JSON equivalents.

The second case is where data is at rest. That is not really the IETF
mandate but the same arguments apply. If you want to authenticate data at
rest then you had better keep the original serialization of the data.

Chunked encoding

One requirement that I have not seen addressed yet is the need to support
chunked encodings so that data can be streamed with a finite buffer space.

If I had a binary coding of JSON available, the number one use case for me
right now would be to package a JSON Signature with the signed data. Or to
use it to create PKCS#7 like structures round an arbitrary MIME content
type.

So I would like to have the ability to specify the length of a text string
or a binary object as a series of chunks, each chunk being preceded by a
length field. It would also be convenient to be able to change the chunk
type from Binary to String in mid stream so as to allow a transcoder that
guessed that the data type was 'base64' based on the first 1000 bytes to
revise that guess when byte 1001 contains a non base64 character.

Null type

Allowing data nulls in a binary format is also very useful. This allows for
padding to align on byte boundaries and can be used to reserve space in a
binary file for pointers and indexes to be added later on.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 23 May 2013 17:28:52 -0400

> Should we not invite the wider JSON community to this discussion?

BSON is the binary format used by MongoDB which is widely used in network
projects and especially Web-ish, JSON projects.

I don't particularly like their design choices but I don't find any of them
terrible.

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 23 May 2013 15:12:52 -0700

> that's not interoperable with theirs.

Sure. The many communities don't seem to be talking to each other.

> shouldn't we pick one of them?

If the design goals of any of them don't align with the others, why should we pick just one? As a specific example, some of the many formats are optimized for message size at the expense of extensibility, while others are optimized for extensibility at the expense of message size. Why should only one of those be forced on everyone else?

> Should we not invite the wider JSON community to this discussion?

We should indeed. That's why Carsten and I did this as an Internet Draft and brought it here instead of grabbing some domain name and getting just our friends involved.

--Paul Hoffman

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Fri, 24 May 2013 17:02:16 +0000

Paul said:

>environment, instead of say posting ephemeral
> specs on a web page and not having it be clear where the community fit
>it, was a good thing.

>I don't remember you being appointed to address the issue.

Who appoints people to do anything at the IETF?  MsgPack is one of the
best of the formats in this space.  When that group was approached with an
opportunity to have help standardizing their format, they reacted
completely rationally:

- What value would having a standard add?  We already interop in practice.
- Doing anything at the IETF is incredibly painful and slow.  See this
conversation for an example.
- The IETF doesn't have any interesting expertise in this space.
- We want to maintain change control in our community.
- Why would we want our spec to be ugly like an RFC? (I added that on
their behalf, but they would have come to it eventually)

I would expect anyone doing work above layer 4 to have similar issues.

>going to come. Unless that
> is the consensus is that your proposal sucks.

I haven't seen that much reaction to the proposal that says it sucks.
I've seen a couple of people say "that looks like it would work".  I've
seen people wonder if it's needed.  I've seen proposed changes that were
greeted with thoughtful responses.  Your assertion that the authors aren't
interested in our input seems hyperbolic at best.  At worst, it's somewhat
hurtful for no apparent reason.

There are some things I don't like about it, but it solves one of my
biggest problem with JSON, which is sending unformatted octet strings.

I'm the mystery person that wrote a JavaScript implementation.  Granted,
it's in node.js, not for browsers, but I didn't see anything that was
un-implementable in a browser that has typed arrays.  The biggest PITA was
learning enough about half-floats to get them to work.  Appendix D now has
the information needed so that a later implementer won't have as much of a
problem.

>My experience of coding ASN.1 DER (in C and javascript) leads me to
>reject any counted scheme as a non starter.

I've read the whole thread, and didn't see adequate justification for that
worldview.  Could you reiterate it, please?

-- 
Joe Hildebrand

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 24 May 2013 13:14:28 -0400

> worldview.  Could you reiterate it, please?

If you have a counter scheme it is necessary to have the whole data
structure in memory to serialize it. Many network applications require the
use of an intermediary with a fixed buffer length.

Now ASN.1 BER definite length encoding makes matters even worse by
specifying lengths in octets rather than the number of entries in an object
but the same objection applies. And DER encoding is just plain insane as
the lengths of the lengths also varies.

A very frequent use case for an encoding format is to take a stream of data
of unknown size and package it in real time. For example I would like to be
able to take a video stream and encrypt it using JSON encryption and then
put a digital signature at the end.

BSON does not do that quite like I would like but I can fake the same
effect by using an array of Binary rather than one Binary chunk and the
same for strings.

With a counted representation I can't even fake it. So I won't be able to
use CBOR for my applications which all involve cryptography. I can do what
I need to do in JSON though.

I see no evidence that counts help in the slightest for efficiency. Even
DER encoding does not really help.

Trying to use a packed encoding as an in-memory encoding is just a bad idea.

---

From: Nico Williams <[email-masked]>
Date: Fri, 24 May 2013 12:53:56 -0500

On Fri, May 24, 2013 at 12:14 PM, Phillip Hallam-Baker <[email-masked]> wrote:
> On Fri, May 24, 2013 at 1:02 PM, Joe Hildebrand (jhildebr)
> <[email-masked]> wrote:
> but the same objection applies. And DER encoding is just plain insane as the
> lengths of the lengths also varies.

Yes, but note that CBOR does much better than DER: it counts elements
(of arrays, of objects), not bytes (except for strings, which are
byte-counted).

It might be nice to have an array/object of indefinite length encoding
where the end is indicated by a special type whose only purpose is to
mark the end of the array/object (CER has this).  But don't try to use
this for strings: you end up having to escape the end-of-string marker
-- ugh!

> put a digital signature at the end.

Note BTW that JSON *is* amenable to online encoding (and decoding).
Binary encodings should keep this feature.

> BSON does not do that quite like I would like but I can fake the same effect
> by using an array of Binary rather than one Binary chunk and the same for
> strings.

I think that's as well as you can expect to do without having to
resort to escaping binary/string characters.

Anyways, +1.

Nico
--

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 24 May 2013 14:08:23 -0400

On Fri, May 24, 2013 at 1:53 PM, Nico Williams <[email-masked]>wrote:

> effect

The alternative would be to declare new data types for chunked strings and
chunked binary which would be closed by an end marker.

The advantage to that approach is that it can be used by a transcoder that
has a fixed buffer size converting from JSON to the binary encoding.

It would be fairly easy to add to BSON as they have plenty of unused code
points. The existing scheme is not very satisfactory as they represent
lengths as fixed 32 bit integers!! Kind of disappointing when a single
video file can easily be over 4Gb. Or at least would be if most equipment
didn't have to dance around brain dead file systems with the same
limitation.

Sure counts of items are better than ASN.1's counts of bytes. But I can't
see any utility in the counts of objects either. If you are trying to
navigate to a particular point in the structure you are going to have to
parse the whole stream regardless. The only way you could skip over a whole
object to go onto the next would be with something like the ASN.1 definite
length scheme.

---

From: Nico Williams <[email-masked]>
Date: Fri, 24 May 2013 13:43:10 -0500

> chunked binary which would be closed by an end marker.

Ah, sure, turn strings (bytes or Unicode) into arrays of chunks.

> object to go onto the next would be with something like the ASN.1 definite

Sure, that's nice, but it sucks for the encoder.

---

From: "Cullen Jennings (fluffy)" <[email-masked]>
Date: Fri, 24 May 2013 23:11:33 +0000

Phil, 

I think your email is out of line and that behavior like this is the largest threat to the IETF begin relevant in the future. If we treat people like this, they are going to take their work elsewhere. I've talked to people about why they don't bring standards work to the IETF and the list of reasons why is surpassingly short and pretty well illustrated by this thread. 

Cullen
 
More inline …

> I don't remember you being appointed to address the issue. 

Actually, I think you, and the rest of the IETF did appoint them. We encourage people to bring good technical work and discuss it. Clearly binary encodings is an important topic for IETF protocols - This seems like a perfectly reasonable list to bring the email discussion to. 

> 
> Since almost all the response to your proposal has been that people don't like your design choices, and since you make it abundantly clear that you are not interested in our input, I can't quite see where a consensus is going to come. Unless that is the consensus is that your proposal sucks.

That was not my read of the thread so far. 

> 
> I would certainly object to a format that was counted being granted any degree of IETF recognition lest someone try to force me to use it in the future.

This is not BCP that must be used by all future IETF protocols. It's an option for work that gets consensus to use it.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 24 May 2013 22:23:12 -0400

On Fri, May 24, 2013 at 7:11 PM, Cullen Jennings (fluffy)
<[email-masked]>wrote:

>
> Phil,
>
> I think your email is out of line and that behavior like this is the
> largest threat to the IETF begin relevant in the future. If we treat people
> like this, they are going to take their work elsewhere. I've talked to
> people about why they don't bring standards work to the IETF and the list
> of reasons why is surpassingly short and pretty well illustrated by this
> thread.
>
> Cullen
>

I was responding to Paul Hoffman. I hardly think he is quite the shrinking
violet you suggest or a novice likely to be deterred from IETF involvement.
He certainly isn't going to be deterred by anything I say.

In particular I was responding to his rather premature claim that the
difference between his proposal and the others was IETF 'consensus'. We
have had all of two days discussion of this proposal while the MongoDB folk
have a whole community built around theirs.

More generally, one of the reasons I see people not wanting to bring work
to IETF is the not invented here attitude. I was more than a little annoyed
when the IETF response to SOAP was to quick march BEEP to proposed standard
in an attempt to trump W3C and OASIS and so were many others.

On May 23, 2013, at 2:14 PM, Phillip Hallam-Baker <[email-masked]> wrote:
> Actually, I think you, and the rest of the IETF did appoint them. We
> encourage people to bring good technical work and discuss it. Clearly
> binary encodings is an important topic for IETF protocols - This seems like
> a perfectly reasonable list to bring the email discussion to.

I did not object to it being brought to the list. What I objected to was
the claim of IETF consensus as the distinctive advantage for the protocol.

The IETF does have a body that is officially tasked with such things
although it doesn't actually do that work. It is called the IAB. Paul's
argument would make sense if it was being made by a member of the IAB
tasked with driving us towards consensus on a binary encoding. Absent an
explicit appointment to address the issue Paul's proposal is just another
Internet Draft.

What I interpreted the majority of the comments preceding it to be saying
was 'why not document one of the existing specs' which is of course what
was done with JSON itself. Instead we have a completely new proposal with
no deployed base being proposed and the only advantage to this proposal is
that it has an Internet Draft.

If the only problem with BSON etc is that they are ephemeral then making
them concrete might be a good start. If there are other problems then a
description would be useful.

> > Since almost all the response to your proposal has been that people
> don't like your design choices, and since you make it abundantly clear that
> you are not interested in our input, I can't quite see where a consensus is

I saw several people raise requirements, all I saw was pushback of the form
'we have decided to do it our way'. I have not seen any of the design
choices being justified against requirements. I have not seen a technical
argument being made for any of the choices other than for the counted vs
delimited issue whare the answer was 'because'.

"Use an existing scheme" would be a perfectly reasonable proposal.

"Develop a new scheme based on requirements proposed by IETF community and
those in the BSON etc community" would also be a reasonable proposal.

"Use our scheme, it has an Internet Draft" does not seem to be a reasonable
proposal.

> This is not BCP that must be used by all future IETF protocols. It's an
> option for work that gets consensus to use it.
>

I don't believe that for two reasons.

First I have heard that exact statement used before about many similar
projects and then I have sat in rooms where the people who said that use
would not be mandated are telling another group that they are expected to
use it. Remember when people were being told that they had to use BEEP for
their Web service? I do.

Second, this particular spec is like JSON in that it only has value if
everyone uses the same thing. There really is not room for more than one
binary encoding of JSON in IETF.

JSON was originally an informational RFC. If we are going to have a binary
encoding then we should probably follow the same approach and that
candidates should be published as informational and we should let the
community make a consensus choice from amongst them.

I see now that my real objection is to CBOR being initially standards
track. What we should do is to begin with Informational descriptions for
existing standards that are widely used (MongoDB's BSON for example) and
Experimental proposals for anything new. Then we should see if there is the
possibility of convergence on one of the proposals.

I dropped a note into the BSON mailing list to see what their response
might be but with no reply so far.

---

From: Nico Williams <[email-masked]>
Date: Fri, 24 May 2013 22:28:56 -0500

Thinking about what *I* want in a binary JSON encoding, and taking
into account PHB's points about online encoding and decoding, all I
really want, the one thing I badly want, is counted-bytes and chunked
Unicode and octet strings.  That's it.  So picture JSON, complete with
square and curly brackets, but no commas nor colons, just strings
(byte-counted or chunked), some encoding for numbers, booleans, and
null.  It's the handing of scalars that sucks about JSON: string
escaping, number printing and parsing.

Something like this:

{<Unicode string of length 3>foo<Unicode string of length 3>bar<join
to preceding string, length 3>baz<Unicode string of length
3>num<integer value 5>}

as an encoding of { "foo": "barbaz", "num": 5 }, with "barbaz" chunked.

One of the nice things about such an encoding is that it should be
possible to implement as a fairly small variation on existing code:
it's almost only a different way of encoding scalar types -- the only
other difference being that commas and colons are not needed.

With a variable-length encoding of integers and IEEE 754 64-bit
doubles for reals... that's compact enough.  Not nearly as compact as
we could get with schemas and PER-like encodings, but good enough for
a schema-less encoding.

Nico
--

---

From: "Cullen Jennings (fluffy)" <[email-masked]>
Date: Sat, 25 May 2013 13:42:17 +0000

On May 24, 2013, at 8:23 PM, Phillip Hallam-Baker <[email-masked]> wrote:

> On Fri, May 24, 2013 at 7:11 PM, Cullen Jennings (fluffy) <[email-masked]> wrote:
> 
> Phil,
> 
> I was responding to Paul Hoffman. I hardly think he is quite the shrinking violet you suggest or a novice likely to be deterred from IETF involvement. He certainly isn't going to be deterred by anything I say. 

Agree and Carsten also is certainly capable of knocking me into line when I am wrong so I realize they are both experienced and not likely to be scared away from  the IETF by anything one person says. But it can still be frustrating, demotivating, and suck all the energy away from getting the technical work done. 

> 
> In particular I was responding to his rather premature claim that the difference between his proposal and the others was IETF 'consensus'. We have had all of two days discussion of this proposal while the MongoDB folk have a whole community built around theirs. 
> 

Ok - I don't think I saw them saying this had IETF consensus - clearly they don't. I do think that it makes sense for them to try and develop consensus around a solution that meets a broader sets of needs than what Mongo does.  That does not make what mongo is doing wrong for mongo but it is one of the reasons why there are a variety of ways of doing binary encoding of data that have evolved over time as the requirements, constraints, and target deployment environments have changed. 

> 
> More generally, one of the reasons I see people not wanting to bring work to IETF is the not invented here attitude. I was more than a little annoyed when the IETF response to SOAP was to quick march BEEP to proposed standard in an attempt to trump W3C and OASIS and so were many others. 

I generally agree that the IETF does better when it takes more than one community of interest and helps bridge them to create a more general and widespread solution that works for both. And sure I see lots of people proposing, lets have the IETF reinvent X - we are going to have some of both of these behaviors and at times it is a good thing. 

Some times I am happy when the IETF effectively rubber stamps a Cisco or Microsoft or some open source projects technology. Other times I am very happy when they do not rubber stamp it but instead look at the underlying needs and take a more multiparty collaborative approach. Both have their time and place to achieve the real goal of making the internet work.  

> I did not object to it being brought to the list. What I objected to was the claim of IETF consensus as the distinctive advantage for the protocol.
> 
> The IETF does have a body that is officially tasked with such things although it doesn't actually do that work. It is called the IAB. Paul's argument would make sense if it was being made by a member of the IAB tasked with driving us towards consensus on a binary encoding. Absent an explicit appointment to address the issue Paul's proposal is just another Internet Draft.
> 
> 
> What I interpreted the majority of the comments preceding it to be saying was 'why not document one of the existing specs' which is of course what was done with JSON itself. Instead we have a completely new proposal with no deployed base being proposed and the only advantage to this proposal is that it has an Internet Draft.
> 
> If the only problem with BSON etc is that they are ephemeral then making them concrete might be a good start. If there are other problems then a description would be useful.
> 
>  
> I saw several people raise requirements, all I saw was pushback of the form 'we have decided to do it our way'. I have not seen any of the design choices being justified against requirements. I have not seen a technical argument being made for any of the choices other than for the counted vs delimited issue whare the answer was 'because'.
> 
> 
> "Develop a new scheme based on requirements proposed by IETF community and those in the BSON etc community" would also be a reasonable proposal.
> 
> "Use our scheme, it has an Internet Draft" does not seem to be a reasonable proposal. 
> 
>  
> First I have heard that exact statement used before about many similar projects and then I have sat in rooms where the people who said that use would not be mandated are telling another group that they are expected to use it. Remember when people were being told that they had to use BEEP for their Web service? I do.
> 
> Second, this particular spec is like JSON in that it only has value if everyone uses the same thing. There really is not room for more than one binary encoding of JSON in IETF. 
> 
> 
> JSON was originally an informational RFC. If we are going to have a binary encoding then we should probably follow the same approach and that candidates should be published as informational and we should let the community make a consensus choice from amongst them.
> 
> I see now that my real objection is to CBOR being initially standards track. What we should do is to begin with Informational descriptions for existing standards that are widely used (MongoDB's BSON for example) and Experimental proposals for anything new. Then we should see if there is the possibility of convergence on one of the proposals.
> 
> 
> I dropped a note into the BSON mailing list to see what their response might be but with no reply so far.
> 
> 
> --

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Sat, 25 May 2013 12:13:14 -0400

> we could get with schemas and PER-like encodings, but good enough for

I like this proposal, more a sort of JSON-B as in JSON Encoding B

The only Con I can think of is that it will still be backwards incompatible
so why not be more compact? And people might worry about JSON-B getting
confused with JSON.

For my requirements, I think I would still like to be able to binary encode
floating point numbers. The issue there is the precision loss from round
tripping and the ability to encode NaN and +/- infinity

But we do have the entire ASCII code set above 128 to play with (actually
we have more but above 128 is plenty)

The result would be slightly less efficient than a true binary JSON but not
by very much. The tags will still be there.

JSON taged items have an overhead of 4 bytes per entry:  "<tag>":<data>,

(You can add in spaces but they aren't necessary.)

It would not be difficult to reduce those 4 bytes to one. But it isn't a
big win either. The win comes from not having to Base64 the binary chunks.

For purposes of planning tags and making sure there is enough space it is
probably best to start off expansively and considering all the possible
types we might recognize

x80    Terminal String 8 bit length
x81    Terminal String 16 bit length
x82    Terminal String 32 bit length
x83    Terminal String 64 bit length

x84    Non Terminal String Chunk 8 bit length
x85    Non Terminal String Chunk 16 bit length
x86    Non Terminal String Chunk 32 bit length
x87    Non Terminal String Chunk 64 bit length

x88    Terminal Binary 8 bit length
x87    Terminal Binary 16 bit length
x8A    Terminal Binary 32 bit length
x8B    Terminal Binary 64 bit length

x8C    Non Terminal Binary Chunk 8 bit length
x8D    Non Terminal Binary Chunk 16 bit length
x8E    Non Terminal Binary Chunk 32 bit length
x8F    Non Terminal Binary Chunk 64 bit length

x90    IEEE 754 Floating Point binary16  (1)
x91    IEEE 754 Floating Point binary32
x92    IEEE 754 Floating Point binary64
x94    IEEE 754 Floating Point binary128  (1)

x96    IEEE 754 Floating Point decimal32 (1)
x97    IEEE 754 Floating Point decimal64 (1)
x98    IEEE 754 Floating Point decimal128  (1)

xA0    Unsigned Integer 8 (1)
xA1    Unsigned Integer 16 (1)
xA2    Unsigned Integer 32 (1)
xA3    Unsigned Integer 64 (1)
xA4    Unsigned Integer 128 (1)

xA5    Signed Integer 8 (1)
xA6    Signed Integer 16 (1)
xA7    Signed Integer 32 (1)
xA8    Signed Integer 64 (1)
xA9    Signed Integer 128 (1)

xAA    True
xAB    False
xAC    Null

(1) The need to implement these codes is debatable. But the tag asignment
scheme should not foreclose the possibility.

That still leaves a block of 72 codes unused. So if people wanted to go
back and do binary tagging at a later date (JSON-C) that would be possible.

I did try to work out some sort of clever bit mask trick so that the lower
bits of the code would specify the number of additional bytes to follow but
that does not work so well as there are 128 bit values to consider. And at
the end of the day there are only going to be 60 odd code maximum so an
array works fine.

I considered Nico's proposal of a 'continuation block' but that would
require a reader to read in the start of the next block before it can know
what to do with the previous data. The writer should know when it is
starting to write out a chunk that more chunks might follow or not. If no
more data follows the writer just puts out x80 x00 or x88 x00 to close the
stream.

{To follow the rest it might help to look at the diagrams in
http://www.json.org/]

There are a few tricks that could be used here to further reduce space.
consider the production "<tag>":<data>,

The initial " is not really needed since the only valid productions
following an object open brace are the close brace or an element entry. So
" is only needed if you desperately want to have the possibility of a tag
'}drop tables'. OK now I get it, leave the initial " in for Randall Munroe
http://xkcd.com/327/

If the reader sees a code above 7F it can only be binary data so the ":
separator between the tag and the data are superfluous and could be elided

The binary data descriptions are all defined length and so the terminal ,
is not needed.

So if people wanted to, we could adjust the FSR to allow these
abbreviations in JSON-B and save the four bytes per object entry overhead.
But that would still leave the tags there and those can't be eliminated
without some sort of dictionary, either being passed on the wire (which is
what compressors are really doing) or out of band (which is what schema
aware is really about).

So for JSON-C we would need ways to define a binding of a tag to a code and
ways of specifying those codes in object productions.

Keeping the initial " helps us here because it means that the only
currently valid characters after the initial { are ", } and whitespace.

We might just conceivably need more than 64K codes, so the code value is
potentially a 32 bit space.

So the codes I would define are:

C0   8 bit tag code follows
C1   16 bit tag code follows
C2   32 bit tag code follows

C4    8 bit definition follows
C5    16 bit definition follows
C6    32 bit definition follows

C7    8 bit tag with definition follows
C8    16 bit tag with definition follows
C9    32 bit tag with definition follows

the codes c4-c9 would be followed with a string definition. So the first
occurrence of { "foo" : "data" } would become:

x7B               {
xC7 x01 x80 x03 x66 x6f x6f    "foo":     [Code 1]
x80 x04 x64 x61 x74 x61          "data"
x7D               }

On the second occurrence the definition is already there:

x7B               {
xC0 x01  [Code 1]
x80 x04 x64 x61 x74 x61          "data"
x7D               }

An implementation could simply dump the dictionary out at the start of the
message using the C4-C6 codes.

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 25 May 2013 22:07:44 +0200

On May 23, 2013, at 19:19, Nico Williams <[email-masked]> wrote:

> what the heck was so wrong with PER?

That is actually an interesting story waiting to be analyzed.

PER was enough of a problem that it became one of the major forces killing H.323.

Of course, that was at a time when we didn't yet all understand that relying on closed-source implementations was creating tons of technical debt.
Also, this was before sourceforge, stackexchange, github, and all the other things that help us manage complex collaborations much better today.
So I'm not sure the same level of complexity (or even PER itself) would die such a horrible death today.
(Actually, EXI might qualify as a nice demo that it can work quite well today.)

But I'm trying to work on the other end of the complexity scale, so somebody else has to write that story down...

Grüße, Carsten

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Sat, 25 May 2013 20:40:31 -0400

Just a thought.

JSON-C would meet most if not all the requirements for HTTP/2.0 encoding.

It is easy to parse/emit, it is easy to adapt an existing JSON stack to
emit JSON-C. It is efficient in coding and in space and a Web service can
use the same stack for HTTP framing and for the payload.

On Sat, May 25, 2013 at 12:13 PM, Phillip Hallam-Baker <[email-masked]>wrote:

> The only Con I can think of is that it will still be backwards
> incompatible so why not be more compact? And people might worry about
> JSON-B getting confused with JSON.
>
>
> For my requirements, I think I would still like to be able to binary
> encode floating point numbers. The issue there is the precision loss from
> round tripping and the ability to encode NaN and +/- infinity
>
> The result would be slightly less efficient than a true binary JSON but
> not by very much. The tags will still be there.
>
> So for JSON-C we would need ways to define a binding of a tag to a code
> and ways of specifying those codes in object productions.
>