Subject: [apps-discuss] Reviving JSON Schema

From: Francis Galiegue <[email-masked]>
Date: Wed, 5 Sep 2012 14:32:00 +0200

Hello everyone,

An organization now exists on GitHub which tries and revives JSON
Schema. The latest accepted draft has expired for more than a year
now, but this doesn't prevent JSON Schema from being used in the wild
today -- with varying levels of support and conformance.

The main repository of this organization is located here:

https://github.com/json-schema/json-schema

The goal is to have, in a reasonable time frame, another draft, or set
of drafts, suitable for submission.

Right now, the organization is still a loosely knit team of people
with interests in the subject (my personal interest is validation, I
have a Java implementation which already has some users), and this
mail is a humble request addressed at seasoned IETF members to ask for
guidance as to what should be done to:

* write a sane enough draft, or set of drafts,
* having it, or them, reviewed and commented.

Humbly yours,
-- 
Francis Galiegue, [email-masked]
JSON Schema: https://github.com/json-schema
"It seems obvious [...] that at least some 'business intelligence'
tools invest so much intelligence on the business side that they have
nothing left for generating SQL queries" (Stéphane Faroult, in "The
Art of SQL", ISBN 0-596-00894-5)

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 05 Sep 2012 15:32:12 +0200

Hi,

FWIW, my I-D draft-lhotka-yang-json-01 allows for using YANG as a schema (or rather data modelling) language for JSON.

Lada

Francis Galiegue <[email-masked]> writes:

> _______________________________________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss

-- 
Ladislav Lhotka, CZ.NIC Labs
PGP Key ID: E74E8C0C

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 5 Sep 2012 11:37:42 -0400

I think what is needed here is a mapping between the data structures
used in today's mainstream programming languages and JSON.

Encoding that mapping in JSON does not interest me at all as it is not
a programming language. As far as I am concerned there is C, C#, Java
and the rest. The only distinction between those as far as the data
model is concerned is whether there is inheritance (C#, Java) or not.

There are big differences between Java and C# as far as features are
concerned, but they both have essentially the same data model: C types
and structures plus single inheritance.

The rest divide up into three groups, the good, the bad and the ugly.
The 'good' are strongly typed functional languages like F# that are
grounded in a data model close enough to C that they can be used for
real work.

The bad are the legacy languages like Cobol, Fortran, Pascal and a
host of toys developed by academics for 'teaching'. People still use
them but they don't use them to write JSON Web Services and if they
did they would not want a schema so they can be safely ignored.

The ugly are the weakly/untyped 'scripting'languages like Perl. These
don't 'need' a schema because they don't 'need types. Instead people
either learn how to debug their scripts without the benefits of strong
typing or they learn to live with the bugs the machine could have
caught for them.

So rather than develop yet another schema language I would suggest a
better approach would be to look at the scheme used in SSL/TLS which
has a C-like schema used to describe the protocol exchanges.

That is what I am doing at any rate. The only difference being that I
don't like the clutter introduced by the semicolons and braces which
the C people thought they needed at the time. I prefer an occam/python
style with indentation denoting block structure.

-- 
Website: http://hallambaker.com/

---

From: Francis Galiegue <[email-masked]>
Date: Wed, 5 Sep 2012 22:43:15 +0200

> used in today's mainstream programming languages and JSON.

Disagree: JSON is a data format defined by an offical RFC, and this
RFC is language neutral. Tying JSON, and JSON Schema, to the
structures available in language X, Y or Z would, to my eyes, defeat
the purpose not only of JSON Schema, but of JSON itself.

---

From: Tim Bray <[email-masked]>
Date: Wed, 5 Sep 2012 13:53:34 -0700

inline

> used in today's mainstream programming languages and JSON.

That varies between unnecessary and impossible.  In modern
dynamically-typed languages (ruby/python/etc) there’s no necessity;
JSON encodes hashes, lists, and a few primitive types, all of which
are built-ins.

And then Java-like languages, there’s just really no good way to model
the following, which might be a reasonable protocol element

[  { "foo": 1, "bar" : false },   23,  "humpty-dumpty" ]

at any level richer than List<Object>; so why bother?

> don't 'need' a schema because they don't 'need types.

This is humor, I imagine?  There are no other languages like Perl,
thank goodness.   And as noted above, the types in modern
dynamically-typed language map with zero friction onto those expressed
by JSON.  -Tim

---

From: Francis Galiegue <[email-masked]>
Date: Wed, 5 Sep 2012 23:11:02 +0200

On Wed, Sep 5, 2012 at 10:53 PM, Tim Bray <[email-masked]> wrote:
> That varies between unnecessary and impossible.

This may sound lame, but... +1.

Especially since you mentioned Java. I have a complete JSON Schema
validating implementation written in Java, and _it is more complete
with regards to validation purposes than all other implementations out
there, including the ones written in JavaScript_. I also know for sure
that it is actually being used by other people than myself for
validating JSON data as per the currently available, even though dead,
JSON Schema draft
(http://tools.ietf.org/html/draft-zyp-json-schema-03)

And there is WJElement, which is written in... Yes... C!

Anyway. JSON Schema will get input, what I was asking in the first
place is what was required for a draft, or series thereof, to be
submitted. I personally am but one member of the Github organization,
and while I am no stranger to writing documentation, I have been
aiming this documentation at users, not the IETF... And IETF, as far
as I can see, means a process has to be followed. And I'd like
insights into the process, along with comments on the existing
material!

For the recall: https://github.com/json-schema/json-schema. This is a
plea from a first-time IETF mailing list poster at this point: I do
believe I have the technical grounds right, I am just completely
ignorant about the process at hand. And I do believe JSON Schema is
the way forward with regards to JSON data processing, and not only
validation (which is why I came to JSON Schema in the first place).

Thank you for reading,

---

From: SM <[email-masked]>
Date: Wed, 05 Sep 2012 15:30:08 -0700

Hi Francis,
>validation (which is why I came to JSON Schema in the first place).

If you think that something is a good idea, write an I-D.  The 
submission guidelines are at 
http://www.ietf.org/ietf-ftp/1id-guidelines.html  The submission form 
is at https://datatracker.ietf.org/submit/  You could discuss the I-D 
on the apps-discuss mailing list.  The process is simple; the author 
puts in the effort to get the I-D through the publication 
process.  You can ask if you need help with the process details.

Regards,
-sm

---

From: Francis Galiegue <[email-masked]>
Date: Thu, 6 Sep 2012 01:11:40 +0200

[...]
> at any level richer than List<Object>; so why bother?

Err...

http://fasterxml.github.com/jackson-databind/javadoc/2.0.5/com/fasterxml/jackson/databind/JsonNode.html

---

From: Tim Bray <[email-masked]>
Date: Wed, 5 Sep 2012 16:18:12 -0700

> http://fasterxml.github.com/jackson-databind/javadoc/2.0.5/com/fasterxml/jackson/databind/JsonNode.html

Sure, but I think Phillip was looking for a mapping into native data
structures, not JSON objects. My argument is that this is trivial in
Ruby/Python/Javascript, but impractical in Java & friends. -T

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 5 Sep 2012 19:26:36 -0400

> JSON encodes hashes, lists, and a few primitive types, all of which

I think you are taking your favored approach and gifting it the
adjective 'modern'.

The advantage of strong static typing is that you can catch errors at
compile time rather than run time. Thats not possible with dynamic
typing by definition.

There are certainly applications where static typing can get in the
way of the programmer. But just as a programming language that
requires a context sensitive grammar is probably doing something
wrong, a protocol that relies on dynamic typing is probably doing
something wrong as well.

> [  { "foo": 1, "bar" : false },   23,  "humpty-dumpty" ]

It might be a protocol element but I can't see it being a 'reasonable' one.

> at any level richer than List<Object>; so why bother?

The reason to bother is that standards making is the process of making
choices that don't matter. A protocol exchange that can't be described
more precisely than List<Object> is most likely a confused mess.

> dynamically-typed language map with zero friction onto those expressed

I guess which category you put Ruby/Python in depends on how you want
to use them.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 5 Sep 2012 19:33:47 -0400

> Ruby/Python/Javascript, but impractical in Java & friends. -T

I have a generator for exactly that mapping, albeit for C# rather than Java.

Creating that mapping is precisely the reason I need a schema. That
and being able to document the protocol:

http://www.ietf.org/id/draft-hallambaker-omnibroker-02.txt

The code examples and the text are both generated from the schema.
That generator is set up to produce C# but C is only a little bit
harder.

---

From: Francis Galiegue <[email-masked]>
Date: Thu, 6 Sep 2012 01:58:43 +0200

On Thu, Sep 6, 2012 at 1:18 AM, Tim Bray <[email-masked]> wrote:
[...]

> structures

Yes, and this is territory which JSON Schema must not venture into,
imho: JSON is, by virtue, a language-neutral standard, And we should
thank it for that ;)

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 5 Sep 2012 20:37:08 -0400

> imho: JSON is, by virtue, a language-neutral standard, And we should

That is a non-sequetor, if there is a schema that makes any sense it
should define a mapping that can be applied to any well specified
language.

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 8 Sep 2012 14:58:04 +1000

This conversation seems to be getting very quickly off-track.

The original question was whether there's interest in progressing JSON Schema, as currently defined, not whether we want to pile on new requirements.

Personally, I struggle with this. XML Schema caused *so* much pain and had *so* much opportunity cost that my automatic reaction is to scream *no*. 

However, JSON is not the same as XML. The metamodel is much simpler, and much more well-aligned with most programming languages, so the interop problems we had with XML schema may not eventuate. 

Furthermore, I think there are interesting use cases for JSON Schema; e.g., helping automate documentation and QA. 

However, I know people have other uses for them, and some of them have potential to cause a lot of harm (read: interoperability problems, lock-in to specific stacks, etc.) that we've seen with XML.

(in that sense, btw, I strongly disagree with PHB, but I suspect that won't surprise many)

As such, if (and that's a big "if") we embark on giving JSON Schema some status, I'd propose several requirements:

 - adherence to the spirit of simplicity in JSON (i.e., bias towards *not* adding any particular feature / capability)
 - focus only on documentation and qa as use cases; NOT "language binding"
 - target should be to describe simple protocol data structures, not any arbitrary JSON
 - no addition of semantics to the document by the schema; it's only describing structure
 - evolution should be very carefully defined, not left to chance

I'd have a very hard time with anything that went against these. 

Cheers,

--
Mark Nottingham   http://www.mnot.net/

---

From: John Bradley <[email-masked]>
Date: Sat, 8 Sep 2012 09:31:33 -0300

I agree with Mark.

In doing openID Connect having a JSON schema would have helped with describing and constraining the messages.   It might have helped with OAuth itself in some cases.

I have also had a recent inquiry from some OASIS people wanting to know what the state of JSON Schema is.

If we focus on documentation and qa use-cases there is value in working on it.

John B.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Sat, 8 Sep 2012 11:38:27 -0400

-1 on the XML Schema thing +1 on the rest

XML Schema is FUBAR for reasons that go beyond XML syntax. The fundamental
problem is that XML Schema tries to be a substitute for DTDs and ends up
importing all their peculiarities.

Trying to use XML Schema as a data modeling language is very painful even
if you don't use the peculiar features of XML Schema like model groups.
There are actually two type systems going on in parallel. XML element tags
are in fact types for the data and XML Schema types are meta-types. Oh and
both can do inheritance.

When I was writing the SAML 1.0 spec I got so fed up of XML Schema that I
wrote my own schema tool to dump out XML Schema. I am going to be using my
schema to dump out both. Sometimes people want XML and sometimes they want
JSON and sometimes they want ASN.1

The encoding syntax should not affect the schema structure at all.

What we need here is a simple schema language that can map cleanly and
simply to a rational subset of any sane programming language (or other
schema) people might want to use.

I don't see a great value to using JSON syntax for the schema language but
that can be a choice. JSON syntax and XML are both cluttered up with
unnecessary junk that I don't want to be typing.

What I think we want for a JSON schema language is something that is as
simple as JSON itself. So there would be a small set of intrinsic types:

No-brainer: Integer, Float, String
Useful to have: DateTime, Binary

Add in inheritance, lists and types defined in the schema and that is all
you need to define a protocol.

If you want to define an extensible protocol then you need to consider what
is going to happen when an unknown tag is encountered, accept, reject or
ignore. That leads to wanting to have an 'ANY' type so that protocol
designers can specify the exact point where the protocol is extensible.

And since we want to generate documentation from the schema we have to have
a slot for that.

So in my protogen tool (which should be up on github soon) I specify a type
thus:

Type Message
   Description
       |Abstract message class from which all messages inherit
   Abstract
   DateTime Sent

Type ConnectRequest
   Inherits Message
   String Username
   String Domain

That is all that is needed to specify the wireline format. But to generate
a client server implementation you need to know how messages are grouped
into requests and responses.

Transaction Connect
    Request ConnectRequest
    Response ConnectResponse

This then allows the compiler to generate code for the client and/or server
in your favorite language. I use C# because it is designed to support this
type of code generation. Java is less friendly.

The client side is easy, a synchronous method in C# would have the signature

class Client {
    public ConnectResponse Connect (ConnectRequest Request) ;

The compiler will also generate asynchronous callbacks using delegates just
as soon as I work out how they should look. I think the cleanest way would
be to simply pass a delegate that is going to be called when it completes.

If the communication pattern is more complicated than request/response then
it is probably best to just code it rather than trying to define the
communication pattern in the compiler.

Implementing the server side means overriding the dispatch methods for
server initialization, termination and the dispatch methods for each
transaction. C# makes this very easy as it has partial classes. But Java is
not that much more complicated.

Now my protocol compiler does go beyond just being a schema tool but not
all that far.

WSDL and XML Schema are terrible implementations of good ideas. They are
over complicated with bells whistles and curlicues that are never ever
necessary and the designers themselves don't seem to really understand.

But the basic idea of having a machine readable definition of the protocol
is a good one.

On Sat, Sep 8, 2012 at 12:58 AM, Mark Nottingham <[email-masked]> wrote:

> The original question was whether there's interest in progressing JSON
> Schema, as currently defined, not whether we want to pile on new
> requirements.
>
> Personally, I struggle with this. XML Schema caused *so* much pain and had
> *so* much opportunity cost that my automatic reaction is to scream *no*.
>
> However, JSON is not the same as XML. The metamodel is much simpler, and
> much more well-aligned with most programming languages, so the interop
> problems we had with XML schema may not eventuate.
>
> Furthermore, I think there are interesting use cases for JSON Schema;
> e.g., helping automate documentation and QA.
>
> However, I know people have other uses for them, and some of them have
> potential to cause a lot of harm (read: interoperability problems, lock-in
> to specific stacks, etc.) that we've seen with XML.
>
> As such, if (and that's a big "if") we embark on giving JSON Schema some
> status, I'd propose several requirements:
>
>  - adherence to the spirit of simplicity in JSON (i.e., bias towards *not*
> adding any particular feature / capability)

---

From: Andrew Newton <[email-masked]>
Date: Mon, 10 Sep 2012 19:22:12 -0400

> I'd have a very hard time with anything that went against these.

I think these are good goals. As for "documentation and qa", I'm not
sure what you mean by QA. Are you suggesting that schemas could be
used to programmatically determine if a JSON file/stream is adherent
to the schema? If so, I don't think that's a good goal as a schema can
only carry things so far before custom application-specific logic is
needed.

Documentation is a good goal, but I don't think what is given in
draft-zyp-json-schema-03 is well suited for documentation. It is too
verbose. Perhaps if there was something like compact RelaxNG notation
for JSON.

-andy