Subject: [apps-discuss] Guidance on RFC 4627 as reference

From: Andrew Newton <[email-masked]>
Date: Tue, 18 Sep 2012 07:08:08 -0400

All,

Excuse me for asking some basic questions. What is the current
guidance for using RFC 4627 as a normative reference? And as that is a
registration of the media type, is 4627 also an acceptable normative
reference for the JSON syntax?

-andy

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 18 Sep 2012 07:35:39 -0400

Didn't we do away with the 'down-reference' prohibition? So using an
informational draft as a normative reference in a standards draft should be
OK.

As a practical matter it is easier to follow RFC 4627 than to fish the JSON
spec out of the ECMA spec (at least the versions I have read). So I would
cite RFC 4627 as normative..

But the ECMA specification is the original standard and they did the
original work so they should probably get a non-normative reference.

Which is of course backwards but that is pretty much a consequence of the
original RFC being issued.

It is pretty clear that change control for using JSON in application
protocols is going to have to rest with IETF as the IETF criteria are going
to be stricter. RFC4267 is not a sufficient description in my view, you
have to state things such as whether the order of objects matters, what to
do if an unexpected tag is encountered and several other issues.

Those are issues for a protocol but not an application programming
language.

It probably makes sense if the IETF came to a consistent approach on such
issues at some point in the future.

On Tue, Sep 18, 2012 at 7:08 AM, Andrew Newton <[email-masked]> wrote:

> _______________________________________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss
>

-- 
Website: http://hallambaker.com/

---

From: Julian Reschke <[email-masked]>
Date: Tue, 18 Sep 2012 13:51:45 +0200

On 2012-09-18 13:08, Andrew Newton wrote:
> ...

As far as I can tell, JSON is fully defined by in RFC 4627 (both the 
media type and the syntax).

Best regards, Julian

---

From: Barry Leiba <[email-masked]>
Date: Tue, 18 Sep 2012 09:11:26 -0400

> informational draft as a normative reference in a standards draft should be

Indeed, what Phill says.  All we need to do is call it out in the Last
Call announcement.  What's more, 4627 is in the downref registry:
   http://trac.tools.ietf.org/group/iesg/trac/wiki/DownrefRegistry
...so it's already got an exemption from even needing to be called out
like that.

Apart from that, Your Friendly ADs have been discussing the
possibility of reclassifying it as Proposed Standard.  Comments about
such a move are welcome on this thread.

> cite RFC 4627 as normative.

As Julian points out, RFC 4627 is actually the only reference for JSON
now (which is why we're thinking of changing its status).  Far from
just registering the media type, it fully defines the grammar, and the
current version of the ECMA spec actually normatively references RFC
4627.

Barry, Applications AD

---

From: Andrew Newton <[email-masked]>
Date: Tue, 18 Sep 2012 09:20:59 -0400

> ...so it's already got an exemption from even needing to be called out

Excellent! Exactly what I wanted to know.

> such a move are welcome on this thread.

I support moving 4627 to Proposed Standard. PHB raises some good
points regarding object member uniqueness and ordering, but I do not
think they need to be addressed in the PS. Those issues are more
appropriate for a separate BCP.

> current version of the ECMA spec actually normatively references RFC

Good to know. Thanks.

-andy

---

From: Julian Reschke <[email-masked]>
Date: Tue, 18 Sep 2012 15:27:27 +0200

> I support moving 4627 to Proposed Standard. PHB raises some good

+1

---

From: Julian Reschke <[email-masked]>
Date: Tue, 18 Sep 2012 15:39:58 +0200

On 2012-09-18 13:35, Phillip Hallam-Baker wrote:
> protocols is going to have to rest with IETF as the IETF criteria are
> going to be stricter. RFC4267 is not a sufficient description in my
> view, you have to state things such as whether the order of objects
> matters, what to do if an unexpected tag is encountered and several
> other issues.
> ...

- The ordering of objects does not matter. There's nothing to define here.

- Handling unexpected tags is an application-level issue; the answer is 
likely to depend on the use case.

What else are you looking for?

Best regards, Julian

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 18 Sep 2012 06:49:29 -0700

> such a move are welcome on this thread.

+1. It's more of a standard than many of our standards.

--Paul Hoffman

---

From: John C Klensin <[email-masked]>
Date: Tue, 18 Sep 2012 12:13:27 -0400

--On Tuesday, September 18, 2012 06:49 -0700 Paul Hoffman
<[email-masked]> wrote:

>> possibility of reclassifying it as Proposed Standard.
>> Comments about such a move are welcome on this thread.
> 

To say on this list, as requested, what has been said in other
places, the Security Considerations section of 4627 is, to be
much more polite than it deserves, not up to the standard for
IETF Standards Track documents.  I think moving it to PS without
some fix to that (either a new document that could be identical
to 4627 other than having that section fixed or an updating
document that would replace that section) would be a disservice
to the community and reduction in the acceptable minimum quality
of standards track protocol specifications.

   john

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 18 Sep 2012 12:32:56 -0400

> - The ordering of objects does not matter. There's nothing to define here.

You have to decide whether it matters or not. JSON does not commit to one
approach or the other.

It makes a very big difference when you try to implement a protocol. As far
as the bits on the wire go there is no difference between List<Object> and
Bag<Object> but there is a very big difference at the application layer.

In my protocol I am sending a JSON structure that contains an ordered list
of potential choices for a connection to a service endpoint. In other words
the order darn well matters in that case and anyone who simply throws the
data into a hash tree is going to end up with the wrong result.

> likely to depend on the use case.

Precisely my point. Application protocols have to decide whether they care
about these things.

This is actually the sort of thing that becomes apparent when you use a
schema. So if you are writing something like SAML you might have:

Type Assertion
    Statements List Any
    Conditions List Any
    Advice List Any

So the following would be valid JSON (according to a particular mapping,
give or take some punctuation):

{ "Assertion" [ {"Authenticate" {...}} ],
  "Conditions" [],
  "Advice" [] }

There are three lists and any tagged object can be inserted into any of the
lists, so the structure can be extended in certain places but the following
would be a parse error:

{ "Assertion" [ {"Authenticate" {...}} ],
  "Conditions" [],
  "Advice" []
  "NewList" [] }

Deciding whether the order in which members of a tagged structure like the
above occur matters is another choice. In SAML the order does matter but
there is no particular reason that it needs to matter.

Just citing RFC 4627 is not sufficient to describe how to implement a
protocol.

If the draft is going to be revised and become a Proposed Standard, I would
like to see it mention these issue as choices that a protocol using JSON
syntax must address clearly.

Simply deciding to use JSON does not remove the obligation to be precise
when stating what is acceptable and what is not. If you are going to have a
specification that says order does not matter then you are going to have to
test for that when you do interop testing. Contrawise if you decide that
order does matter then you are going to have to test for that.

---

From: Julian Reschke <[email-masked]>
Date: Tue, 18 Sep 2012 18:45:31 +0200

On 2012-09-18 18:32, Phillip Hallam-Baker wrote:
>
>
> On Tue, Sep 18, 2012 at 9:39 AM, Julian Reschke <[email-masked]
> <mailto:[email-masked]>> wrote:
>
> You have to decide whether it matters or not. JSON does not commit to
> one approach or the other.

It doesn't matter in Javascript, so trying to make it matter in JSON 
will break with many existing implementations.

> It makes a very big difference when you try to implement a protocol. As
> far as the bits on the wire go there is no difference between
> List<Object> and Bag<Object> but there is a very big difference at the
> application layer.

Indeed. If you need ordering you can't rely on object order.

> In my protocol I am sending a JSON structure that contains an ordered
> list of potential choices for a connection to a service endpoint. In
> other words the order darn well matters in that case and anyone who
> simply throws the data into a hash tree is going to end up with the
> wrong result.

...which is what some browsers do (and rightly so), so your protocol has 
a design problem :-).

>     - Handling unexpected tags is an application-level issue; the answer
>     is likely to depend on the use case.
>
>
> Precisely my point. Application protocols have to decide whether they
> care about these things.

Yes.

> There are three lists and any tagged object can be inserted into any of
> the lists, so the structure can be extended in certain places but the
> following would be a parse error:
>
> Deciding whether the order in which members of a tagged structure like
> the above occur matters is another choice. In SAML the order does matter
> but there is no particular reason that it needs to matter.
>
>
>

That is true, but that's not a defect of RFC 4627.

> If the draft is going to be revised and become a Proposed Standard, I
> would like to see it mention these issue as choices that a protocol
> using JSON syntax must address clearly.

I agree for the bot about handling unexpected content (== extension 
elements).

> when stating what is acceptable and what is not. If you are going to
> have a specification that says order does not matter then you are going
> to have to test for that when you do interop testing. Contrawise if you
> decide that order does matter then you are going to have to test for that.

Again, if you make ordering significant you are in trouble. And yes, as 
this question comes up from time to time, a revision of 4627 could 
mention that.

Best regards, Julian

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 18 Sep 2012 12:47:49 -0400

How can you use tools 'correctly' without having a definition of what
'correct' is?

I can't see a place in the JSON specification where it says that the
following are equivalent:

[ {"name": "foo", "value": "one"}, {"name": "bar", "value": "two"}]
[ {"name": "foo", "value": "two"}, {"name": "bar", "value": "one"}]

If you want them to be equivalent then you have to say so and this has to
be something the WG discusses and comes to consensus on.

This is what we do with RFC822 headers, there is an explicit statement in
the spec that says that the order in which the headers are presented does
not change the semantics.

My opinion is that considering the elements to be lists of items rather
than bags is better because it is less work using appropriate tools and the
resulting representation is more compact.

On Tue, Sep 18, 2012 at 12:39 PM, Joe Hildebrand (jhildebr) <
[email-masked]> wrote:

> On 9/18/12 9:32 AM, "Phillip Hallam-Baker" <[email-masked]> wrote:
>
> >simply throws the data into a hash tree is going
> > to end up with the wrong result.
>
> If it's possible, I'd suggest exploring other design patterns.  Instead of
> the order mattering in:
>
> { "foo": "one", "bar": "two" }
>
> You could use:
>
> [ {"name": "foo", "value": "one"}, {"name": "bar", "value": "two"}]
>
> Or some other non-ambiguous approach.
>
> >would like to see it mention these issue as choices that a protocol using
> >JSON syntax must address clearly.
>
> It seems reasonable to give guidelines to protocol implementors.  Whether
> that belongs in this document or not is a separate question.  If we can
> come to consensus on what those guidelines are quickly, it wouldn't hurt
> to include them -- but that seems unlikely.
>
> >when stating what is acceptable and what is not. If you are going to have
> >a specification that says order does not matter then you are going to
> >have to test for that when you do interop
> > testing. Contrawise if you decide that order does matter then you are
> >going to have to test for that.
>
> Or: use the tools correctly according to the guidelines that may exist one
> day, and you won't have this interop issue.
>
> --
> Joe Hildebrand
>
>
>
>

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 18 Sep 2012 12:51:49 -0400

RFC 4627 states:

2.3.  Arrays

   An array structure is represented as square brackets surrounding zero
   or more values (or elements).  Elements are separated by commas.

      array = begin-array [ value *( value-separator value ) ] end-array

So has JavaScript redefined the meaning of 'Array' then?

how vewy peculiar.

I suggest that you fix your broken language or change the term 'Array' to
'Bag' in the specification because where I come from 'Array' and 'List' can
be used interchangeably in some contexts but 'Array' and 'Bag' cannot.

On Tue, Sep 18, 2012 at 12:45 PM, Julian Reschke <[email-masked]>wrote:

>> <mailto:[email-masked]>**> wrote:
>>
> It doesn't matter in Javascript, so trying to make it matter in JSON will
> break with many existing implementations.
>
>
> ...which is what some browsers do (and rightly so), so your protocol has a
> design problem :-).
>
>
> this question comes up from time to time, a revision of 4627 could mention
> that.
>
> Best regards, Julian
>
>

---

From: Andrew Newton <[email-masked]>
Date: Tue, 18 Sep 2012 12:53:20 -0400

> How can you use tools 'correctly' without having a definition of what

I agree with Phillip. What may seem obvious to those who use an API
that uses hashes is not so obvious to those who don't (and in the
IETF, you can count on the majority of reviewers not being programmers
and so not realizing the issue).

-andy

---

From: Julian Reschke <[email-masked]>
Date: Tue, 18 Sep 2012 18:55:00 +0200

> So has JavaScript redefined the meaning of 'Array' then?

No. Arrays are indeed ordered. But ordering inside an *object* is not 
significant.

> I suggest that you fix your broken language or change the term 'Array'
> to 'Bag' in the specification because where I come from 'Array' and
> 'List' can be used interchangeably in some contexts but 'Array' and
> 'Bag' cannot.

Philip, you asked:

> you have to state things such as whether the order of objects matters

It does not matter in objects, but it does matter in arrays.

Best regards, Julian

---

From: James M Snell <[email-masked]>
Date: Tue, 18 Sep 2012 09:56:31 -0700

A couple points of clarification... the ordering of objects in a JSON Array
*does* matter. The ordering of key:value pairs within an object does not.
[1,2] is not equivalent to [2,1] while {"a":"b","c":"d"} is equivalent to
{"c":"d","a":"b"}. The spec could be clearer on this point, definitely.

- James

---

From: Julian Reschke <[email-masked]>
Date: Tue, 18 Sep 2012 18:57:44 +0200

> [ {"name": "foo", "value": "two"}, {"name": "bar", "value": "one"}]

They are not equivalent.

> If you want them to be equivalent then you have to say so and this has
> to be something the WG discusses and comes to consensus on.
>
> This is what we do with RFC822 headers, there is an explicit statement
> in the spec that says that the order in which the headers are presented
> does not change the semantics.
>
> than bags is better because it is less work using appropriate tools and
> the resulting representation is more compact.

Again, you can't change application/json to have semantics that 
Javascript does not have.

Best regards, Julian

---

From: James M Snell <[email-masked]>
Date: Tue, 18 Sep 2012 10:10:13 -0700

In fact... the current introduction of RFC4627 makes this all quite clear...

[snip]

   An object is an unordered collection of zero or more name/value
   pairs, where a name is a string and a value is a string, number,
   boolean, null, object, or array.

   An array is an ordered sequence of zero or more values.

   The terms "object" and "array" come from the conventions of
   JavaScript.

   JSON's design goals were for it to be minimal, portable, textual, and
   a subset of JavaScript.

[snip]

- James

On Tue, Sep 18, 2012 at 9:57 AM, Julian Reschke <[email-masked]>wrote:

> Again, you can't change application/json to have semantics that Javascript
> does not have.
>
> Best regards, Julian
>
> ______________________________**_________________
> https://www.ietf.org/mailman/**listinfo/apps-discuss<https://www.ietf.org/mailman/listinfo/apps-discuss>
>

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 18 Sep 2012 13:16:32 -0400

When I use language, I try to be precise. The order of objects and the
order of entries within objects are not the same.

Even so, when writing a spec I usually call out such distinctions clearly
if I notice them and telling people that this is what they need to do is
something that I would like RFC 4627-BIS to address specifically.

In another part of the schema compiler I allow objects with repeated
elements:

{ "A": 1, "A": 2, "B": 3 }

Now my code does not see a difference between that and

{ "B": 3, "A": 1, "A": 2 } or { "A": 1, "B": 3, "A": 2 }

The serializer will emit the data in a canonical order but accepts them in
any order. So an application could consider the following to be different:

{ "A": 2, "A": 1, "B": 3 }

The same is almost certainly true of the hash bag approach, at least for
some hash baggers but not necessarily of others. Whether the order of a
repeated element is significant or not is going to have quite an impact on
your design.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 18 Sep 2012 13:23:02 -0400

Which would seem to be a statement that should occur in the section headed
'grammar'...

Note that pretty much all the preceeding examples given by folk also
violate the following:

   An object structure is represented as a pair of curly brackets
   surrounding zero or more name/value pairs (or members).  A name is a
   string.  A single colon comes after each name, separating the name
   from the value.  A single comma separates a value from a following
   name.  The names within an object SHOULD be unique.

I don't think you can have a SHOULD there in a standard of this type.
Either the spec allows for non-unique names or it does not.

It would seem reasonable to require names to be unique since the case of
repeated names can be addressed using arrays.

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 18 Sep 2012 17:26:49 +0000

>Either the spec allows for non-unique names or it does not.

Agree with that.  It does not, so MUST is appropriate.

>repeated names can be addressed using arrays.

No, they aren't.  Please don't complicate something simple because you
don't want to change your syntax to match the simplicity.

-- 
Joe Hildebrand

---

From: James M Snell <[email-masked]>
Date: Tue, 18 Sep 2012 10:38:23 -0700

Again, the spec is very clear that the intent is to adopt the JavaScript
semantics. JavaScript does not require that names within an object be
unique, but it does specify that Property Attributes have a singular Value,
meaning that if a given property is repeated within a structure, it's value
is overwritten by each subsequent instance..

That is, given your example,

  The value of "A" is 2.

The fact that there are two values for "A" given is not, by itself, an
error. However, unexpected results could occur depending on how a developer
processes the object structure and multiple instances of a property are
almost always indicative of a bug. The SHOULD here is appropriate, then, in
that developers really shouldn't be specifying a given property more than
once but it's technically not an error if they do.

- James

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 18 Sep 2012 13:39:16 -0400

> don't want to change your syntax to match the simplicity.

?? what I am saying is that someone who feels the need to write

Should instead write

{ "A" : [1, 2], "B": 3 }

Since every use case for a repeated tag can be met by a tag with a list of
items, there is no need for the repeated tag.

Thus it is reasonable to change the SHOULD to a MUST in a BIS to be used as
a normative reference in protocol specs.

So the changes to section 2.2 are:

1) State that the order of the taged entries does not matter (no, putting
that in the introduction does not count).

2) Change the SHOULD only occur once to MUST NOT occur more than once.

That still leaves open what an implementation may do when it encounters an
unexpected, unknown or illegal tag. Which is the sort of thing that the
security section should address.

At any rate, I think this means that the idea of trying to use JSON syntax
to describe JSON schemas is not going to work. I for one can't imagine
wanting to use a schema notation where blocks are sometimes denoted using
{} and sometimes by [] and the difference is essential to the semantics.

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 18 Sep 2012 17:50:57 +0000

On 9/18/12 10:39 AM, "Phillip Hallam-Baker" <[email-masked]> wrote:

>Since every use case for a repeated tag can be met by a tag with a list
>of items, there is no need for the repeated tag.

+1.

>Thus it is reasonable to change the SHOULD to a MUST in a BIS to be used
>as a normative reference in protocol specs.

James Snell gave a good chapter/verse on this in his last message.  From a
JSON-only point of view, I understand the SHOULD now (although I like
SHOULD's to explain the cases when you wouldn't), but for many *protocol*
uses of JSON, we should specify what a receiver should do with repeated
keys.  By default, the answer is likely "undefined/choose-one".

>that in the introduction does not count).

Doesn't hurt anything.

>2) Change the SHOULD only occur once to MUST NOT occur more than once.

That's the easiest change; we could discuss whether the slight difference
to the nuance provided by James is important.

>That still leaves open what an implementation may do when it encounters
>an unexpected, unknown or illegal tag. Which is the sort of thing that
>the security section should address.

+1

>At any rate, I think this means that the idea of trying to use JSON
>syntax to describe JSON schemas is not going to work. I for one can't
>imagine wanting to use a schema notation where blocks are sometimes
>denoted using {} and sometimes by [] and the difference is essential to
>the semantics.

I do not agree with your conclusion in any way.  draft-zyp-json-schema has
it's nits, but seems like a reasonable starting point.

-- 
Joe Hildebrand

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 18 Sep 2012 11:18:52 -0700

On Sep 18, 2012, at 9:13 AM, John C Klensin <[email-masked]> wrote:

> IETF Standards Track documents.  

I would be interested to hear where those "other places" are. The security considerations seem fine to me, but you obviously have thought of additions you want (or at least you think others have). Listing them here would be useful for the discussion.

--Paul Hoffman

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 18 Sep 2012 14:50:40 -0400

On Tue, Sep 18, 2012 at 2:18 PM, Paul Hoffman <[email-masked]> wrote:

> I would be interested to hear where those "other places" are. The security
> considerations seem fine to me, but you obviously have thought of additions
> you want (or at least you think others have). Listing them here would be
> useful for the discussion.
>

I would like to see John's list as well. My treatment would be something
along the following:

1) Code injection attacks

[This is what the rather opaque language in the existing text seems to be
attempting to address]

2) Data overflow attacks

[Parsers need to consider cases such as buffer overrun, integer overflow,
etc]

3) Message size limitations

[Separate from the question of buffer overrun there is the issue of sending
messages that are so long that they cause resource exhaustion and a DoS
attack]

4) Error handling

[What should a parser do if it encounters malformed JSON,

5) Resynchronization

[How does a receiver that has received one malformed JSON message identify
the start of the next item]

6) Integrity

[JSON does not provide for integrity checking and this needs to be
considered in the framing protocol]

---

From: James M Snell <[email-masked]>
Date: Tue, 18 Sep 2012 12:30:42 -0700

I can readily agree that the security considerations a quite lacking and
you're right that there are areas where the language could be tightened up
editorially (though I am -1 on making normative syntax and 2119 changes).
Overall, I would be +1 on a rfc4627bis effort and could volunteer my time
as an editor for such an effort.

- James

On Tue, Sep 18, 2012 at 11:50 AM, Phillip Hallam-Baker <[email-masked]>wrote:

>
>
> On Tue, Sep 18, 2012 at 2:18 PM, Paul Hoffman <[email-masked]>wrote:
>
>> I would be interested to hear where those "other places" are. The
>> security considerations seem fine to me, but you obviously have thought of
>> additions you want (or at least you think others have). Listing them here
>> would be useful for the discussion.
>>
>
>
> [Separate from the question of buffer overrun there is the issue of
> sending messages that are so long that they cause resource exhaustion and a
> DoS attack]
>
> 4) Error handling
>

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 18 Sep 2012 13:22:08 -0700

On Sep 18, 2012, at 11:50 AM, Phillip Hallam-Baker <[email-masked]> wrote:

> I would like to see John's list as well. My treatment would be something along the following:
> 
> [This is what the rather opaque language in the existing text seems to be attempting to address]
> 
> 2) Data overflow attacks
> 
> [Parsers need to consider cases such as buffer overrun, integer overflow, etc]
> 
> [Separate from the question of buffer overrun there is the issue of sending messages that are so long that they cause resource exhaustion and a DoS attack]

Are any of these specific to JSON? They apply to any format, yes? I'm fine with adding them, but in doing so, we aren't talking about JSON.

> [What should a parser do if it encounters malformed JSON, 

That's not a security consideration, that's a protocol implementation issue. I don't think we should specify it, give the wildly different environments where JSON parsing happens.

> 5) Resynchronization
> 
> [How does a receiver that has received one malformed JSON message identify the start of the next item]

Same as 4.

> 6) Integrity
> 
> [JSON does not provide for integrity checking and this needs to be considered in the framing protocol]

Sure, I guess that is a "security consideration" for any format.

--Paul Hoffman

---

From: "Martin J. Dürst" <[email-masked]>
Date: Wed, 19 Sep 2012 11:51:14 +0900

> [ {"name": "foo", "value": "two"}, {"name": "bar", "value": "one"}]

Maybe that was obvious to most, but these two are of course not 
equivalent, the first one has a value of "one" for a name of "foo", the 
second one has a value of "two".

To fix the example, a better question would have been whether the 
following two are equivalent:

[ {"name": "foo", "value": "one"}, {"name": "bar", "value": "two"}]
[ {"name": "bar", "value": "two"}, {"name": "foo", "value": "one"}]

Now we have two arrays, both with the same two element, and only the 
order is different. The answer to whether they are equivalent or not has 
already been given: They are not, because order is significant in arrays.

Whether it's the intro or some later section (or both) that say that 
arrays are ordered and objects/hashes are unordered is definitely a 
question of spec quality, but claiming that there's no definition is 
overblown.

Regards,   Martin.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Wed, 19 Sep 2012 11:59:31 +0900

On 2012/09/19 2:50, Joe Hildebrand (jhildebr) wrote:
> On 9/18/12 10:39 AM, "Phillip Hallam-Baker"<[email-masked]>  wrote:

> keys.  By default, the answer is likely "undefined/choose-one".

This answer should not be defined by protocol. It would mean that it's 
no longer possible to use generic JSON parsers.

> to the nuance provided by James is important.

I'd definitely suggest a very strong MUST NOT send. We may need some 
checks on existing implementations to decide whether it's better to 
define this as an error on the receiver side (but the current SHOULD 
seems to indicate it's not), or to leave it undefined.

Regards,   Martin.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 18 Sep 2012 23:52:18 -0400

On Tue, Sep 18, 2012 at 4:22 PM, Paul Hoffman <[email-masked]> wrote:

> Are any of these specific to JSON? They apply to any format, yes? I'm fine
> with adding them, but in doing so, we aren't talking about JSON.

If the syntax is sufficiently general, the statements in the SC section
should be applicable to anything else. But that does not mean that they can
or should be omitted.

If someone is writing a parser/emitter module, they need to think about how
the developer using the module can address those concerns and whether that
code belongs inside or outside their module.

> That's not a security consideration, that's a protocol implementation
> issue. I don't think we should specify it, give the wildly different
> environments where JSON parsing happens

Security considerations are just that, considerations.

The SC set should tell the implementer what issues they need to be thinking
about when they write code.

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 18 Sep 2012 16:39:38 +0000

You could use:

-- 
Joe Hildebrand

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 18 Sep 2012 17:22:48 +0000

>order of entries within objects are not the same.

Please *read* 4627 before asking for changes?

>{ "A": 1, "A": 2, "B": 3 }

Then it's not JSON, at least according to the spirit.  From section 2.2:

"The names within an object SHOULD be unique."

I'd argue that ought to be MUST, but you don't have a good reason to
ignore the SHOULD. 

I'd also be ok with adding language that says that describes what you you
do if you receive an invalid object like that; one possibility would be to
return an error, another would be to pick a random instance, there are
likely others.

-- 
Joe Hildebrand

---

From: SM <[email-masked]>
Date: Wed, 19 Sep 2012 14:20:53 -0700

>such a move are welcome on this thread.

There is an erratum.  A reclassification would require ignoring 
it.  I'd suggest a quick open and shut path.  My guess is that it 
might not be quick*.  The Security Considerations section of RFC 4627 
seems to be on the economical side.

Regards,
-sm

* JSON seems to be the IETF secret sauce these days.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 19 Sep 2012 17:56:38 -0400

On Wed, Sep 19, 2012 at 5:20 PM, SM <[email-masked]> wrote:

> There is an erratum.  A reclassification would require ignoring it.  I'd
> suggest a quick open and shut path.  My guess is that it might not be
> quick*.  The Security Considerations section of RFC 4627 seems to be on the
> economical side.
>
> Regards,
> -sm
>

The security considerations is definitely light to the point of being non
existent. But as with many specs, there really isn't a set of security
considerations that it specific to JSON data format.

If the spec has to be rewritten then I would like to see the security
considerations address the general security issues so that we can do it
once and not have to repeat it in other documents. I would also like some
clarifications. But I don't see that as a reason for blocking uprating the
spec as is if that is an option.

We still get a second bite at the revisions cherry in the Proposed to
Standard move. I think a new RFC would be necessary for that.

---

From: Ned Freed <[email-masked]>
Date: Wed, 19 Sep 2012 18:15:55 -0700 (PDT)

> considerations that it specific to JSON data format.

Quite correct, and so that's what the security considerations need to say:
They depend on what's being represented, and here are some of the potential
issues that could arise.

This is NOT hard to do; we've done it many times before.

> spec as is if that is an option.

Simple reclassification really isn't an option at this point, because as it is
the media type registration in the current document violates the rules for such
things. (And this has nothing to do with document status.) It should never have
been published with this text in the first place, and given the chance to fix
it that a change in standards grade presents, we are obliged to do so.

> Standard move. I think a new RFC would be necessary for that.

It's necessary now. Whether or not it's necessary later will depend on how
good a job we do.

				Ned

---

From: Ned Freed <[email-masked]>
Date: Thu, 20 Sep 2012 08:01:43 -0700 (PDT)

> with adding them, but in doing so, we aren't talking about JSON.

If I understand you correctly, it's unnecessary to call out a security 
consideration if it's something that is common to the class of things something
belongs in?

If that's the case, then words really fail me. Just as one example, there is a
broad class of scripting langauges which, if executed, can completely 
compromise and even destroy the machine they run on, and which therefore
require the use of external authentication, access control and integrity
checking. But because these concerns apply to such langauges as a class we
don't need to bother to warn people about them?

I'm sorry, but that just doesn't fly. And BTW, the answer to your question is
no, there are abundant examples of completely passive formats that are not
vulnerable to, say, code injection attacks.

> That's not a security consideration, that's a protocol implementation issue.
> I don't think we should specify it, give the wildly different environments
> where JSON parsing happens.

Whether or not it's a security consideration depends on what the JSON is being
used for. The fact that JSON is a container format that can be used for almost
any purpose broadens the security considerations considerably, and that
absolutely needs to be discussed.

> Same as 4.

Same response.

> Sure, I guess that is a "security consideration" for any format.

Wrong again. The obvious counterexample is a format that provides all the
integrity checking it needs internally and thus doesn't require any sort of
external integrity service. But more generally, while you can of course invent
uses for pretty much any format imagineable that require integrity (or
confidentiality) protection, there are formats which are designed for
destributing public information and which do not involve active or executable
content or pointers to other stuff. In such cases there really are no integrity
concerns worth talking about.

				Ned

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 20 Sep 2012 08:52:56 -0700

> consideration if it's something that is common to the class of things something

You do not understand me correctly. Note that I said that I was fine with adding them. If we add them, we should point out that they are not about JSON, but any format.

> vulnerable to, say, code injection attacks.

Can you give your best example? I'm completely missing it. If you can put "code" in JSON, I expect you can put "code" in any format.

> absolutely needs to be discussed.

Fully agree that it should be discussed. Note that Phill said "What should a parser do", not "discuss the topic". I support the second (as you do), but I object to the first.

> external integrity service.

Which formats do that?

> But more generally, while you can of course invent

Errr, right. Are you saying JSON involves active or executable content? Or pointers to other stuff? If so, I agree that those are security considerations. I'm not seeing it in the JSON definition, however. What am I missing?

--Paul Hoffman

---

From: Ned Freed <[email-masked]>
Date: Thu, 20 Sep 2012 10:18:09 -0700 (PDT)

> You do not understand me correctly. Note that I said that I was fine with
> adding them. If we add them, we should point out that they are not about JSON,
> but any format.

OK, but that's still incorrect, just in a different way. See below.

> Can you give your best example? I'm completely missing it. If you can put
> "code" in JSON, I expect you can put "code" in any format.

Given that "code" and "data" are effectively equivalent (thank you von
Neumann), and given the existence of steaganography, sure, "code", for some
value of the word, can appear in any format. But being able to put code into a
format doesn't necessarily translate into an injection attack. An injection
attack occurs during some sort of subsitution operation where the substituted
material is inadequately checked or quoted. For that to occur there necessarily
has to be some sort of substitution operation that is a regular part of
processing the material.

Given these constraints, it's probably easier to list formats that are
vulnerable to code injection attacks, but if you insist on specific
examples, most image formats or even text/plain qualify.

And yes, you can describe highly unlikely scenarios where some moron inserts
raw image data directly into SQL or HTML or whatever. But there has to be some
amount of common sense applied to writing security considerations, because if
there isn't it ends up that since any concievable usage is possible everything
is vulnerable to everything. So if you're describing a format where
substitution operations are a regular part of its use, then you need to talk
about code injection attacks. And if you aren't, you don't.

> Fully agree that it should be discussed. Note that Phill said "What should a
> parser do", not "discuss the topic". I support the second (as you do), but I
> object to the first.

But that's sort of my point. What a parser should do depends on the usage
semantics. So no, getting into specifics of what a parser should do isn't
appropriate, but it's important to note that different usages require different
sorts of error handling and maybe list some of the possibilities and generally
when some approach is appropriate.

> Which formats do that?

An obvious example would be multipart/signed. The entire point of
multipart/signed to sign the content, which of course involves an integrity
check. More specific examples would be any XML format that uses XML's signature
capabilities. And while I can't name a specific JSON-based format that has
built in integrity checks, it's entirely possible to define one.

As it happens I'm working on the registration of an ASN.1-based format right
now that incorporates an incredibly elaborate multi-level signature mechanism. 

And for that matter, integrity check doesn't necessarily mean signed. There are
lots of formats that incorporate hashes and CRC checks.

> Errr, right. Are you saying JSON involves active or executable content?

It most definitely can and in practice is used that way. And that possibility
also need to be mentioned.

> Or pointers to other stuff? If so, I agree that those are security
> considerations. I'm not seeing it in the JSON definition, however. What am I
> missing?

You're missing the fact that JSON is a container format that admits a
wide range of specific uses, and the security considerations need to
encompass that range of potential uses. In effect the security considerations
for container formats become a guide for what to look at when evaluating
security for specific applications of the format.

It is necessarily the case that writing security considerations for container
formats is more difficult than for a format intended for a specific 
application. But it's entirely doable.

				Ned

---

From: James M Snell <[email-masked]>
Date: Thu, 20 Sep 2012 10:51:29 -0700

I believe the specific concern for JSON is that, in many environments --
especially browsers -- JSON is often parsed using the JavaScript eval
function, which allows for the execution of arbitrary JavaScript code.
While the JSON specification itself restricts the syntax to non-executable
data, a malicious party could intentionally construct an invalid-JSON
object such that arbitrary code is executed within the browser. This
particular concern is unique for JavaScript relative to other formats (XML,
HTML) and really ought be addressed within the security considerations of
rfc4627. Specifically, implementors need to be warned against passing
unvalidated JSON to the eval function.

On Thu, Sep 20, 2012 at 8:52 AM, Paul Hoffman <[email-masked]> wrote:

> Errr, right. Are you saying JSON involves active or executable content? Or
> pointers to other stuff? If so, I agree that those are security
> considerations. I'm not seeing it in the JSON definition, however. What am
> I missing?
>
> --Paul Hoffman

---

From: Julian Reschke <[email-masked]>
Date: Thu, 20 Sep 2012 19:57:54 +0200

> also need to be mentioned.

But that's not JSON; it's just something that remotely looks like it.

I have nothing against warning people to *always* use strict JSON 
parsers (not JS eval) to process JSON, but we should be clear about 
what's JSON and what is not.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Thu, 20 Sep 2012 20:01:20 +0200

On 2012-09-20 19:51, James M Snell wrote:
> While the JSON specification itself restricts the syntax to
> non-executable data, a malicious party could intentionally construct an
> invalid-JSON object such that arbitrary code is executed within the
> browser. This particular concern is unique for JavaScript relative to
> other formats (XML, HTML) and really ought be addressed within the
> security considerations of rfc4627. Specifically, implementors need to
> be warned against passing unvalidated JSON to the eval function.
> ...

...need to be warned not to ever pass JSON to eval.

If you can validate it, you have already parsed it, and there's no point 
anymore to use eval().

Best regards, Julian

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 20 Sep 2012 14:05:07 -0400

I think the real problem with the vague references to 'code' in the JSON
RFC is that they mean absolutely nothing. I am not sure that they even bear
mention.

The real problem with interpreted languages is that they make it easy to
mix the control and data channel which is of course what makes code
injection attacks possible. SQL mixes the two in convoluted ways.

What the security statement needs to say than is that encoders need to
ensure that strings are appropriately escaped in all cases and that code
does not simply do things like

printf ("{ \"tag\" : \"%0\" }", foo);

It has to be

printf ("{ \"tag\" : \"%0\" }", StringPrep (foo));

I think that is the problem the language is hinting at but failing to make
clear.

If people use JSON to carry data that is SQL commands, shell commands or
similar then these issues are going to be security considerations.

That is why the types of encoding we prefer in a security sensitive
application tend to use either an explicit length encoding (ASN.1 approach)
or randomized guard values or both.

The real objective is to make sure that nobody can try to just wing it like
you can in a scripting language.

If you write code to be used in very dangerous environments then you have a
different approach.

On Thu, Sep 20, 2012 at 1:57 PM, Julian Reschke <[email-masked]>wrote:

> I have nothing against warning people to *always* use strict JSON parsers
> (not JS eval) to process JSON, but we should be clear about what's JSON and
> what is not.
>
> Best regards, Julian
>

---

From: Julian Reschke <[email-masked]>
Date: Thu, 20 Sep 2012 20:12:11 +0200

On 2012-09-20 20:05, Phillip Hallam-Baker wrote:
> RFC is that they mean absolutely nothing. I am not sure that they even
> bear mention.
> ...

RFC 4627 doesn't mention "code" in the sense of programming code at all.

> I think that is the problem the language is hinting at but failing to
> make clear.
> ...

I think that security considerations for *recipients* are much more 
important; and guess what, RFC 4627 has some.

It's nice if senders are not vulnerable to code injection, but if 
recipients parse JSON as JSON, they are by definition not vulnerable.

> application tend to use either an explicit length encoding (ASN.1
> approach) or randomized guard values or both.
> ...

That's all very interesting, but how does that affect JSON?

I would recommend that anybody who wants something specifically changed 
in RFC 4627 first checks whether it might there already (me guilty of 
that as well (*)), and then makes a *concrete* proposal.

Best regards, Julian

(*) RFC 4627 explains how to use a regexp to validate that a given octet 
sequence is safe to be passed to JS eval().

---

From: Barry Leiba <[email-masked]>
Date: Thu, 20 Sep 2012 14:13:23 -0400

> such a move are welcome on this thread.

Someone mentioned to me that my use of the term "reclassify" might
have been unclear.  To be completely clear: we aren't considering
changing the classification of RFC 4627 as it stands.  It would only
be done by updating the document and publishing a new RFC at Proposed
Standard.

Barry, Applications AD

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 20 Sep 2012 11:15:28 -0700

On Sep 20, 2012, at 10:18 AM, Ned Freed <[email-masked]> wrote:

> value of the word, can appear in any format.

I'm glad we agree on that.

> But being able to put code into a

That is a very good definition.

> examples, most image formats or even text/plain qualify.

We fully disagree about text/plain, then. Text/plain is often used for configuration files that are slurped in with inadequate checking. 

> about code injection attacks. And if you aren't, you don't.

Fully agree. But it seems we disagree about which formats those are.

> built in integrity checks, it's entirely possible to define one.

Of course. But I thought we were talking about formats like JSON, not profiles of that format. 

> now that incorporates an incredibly elaborate multi-level signature mechanism. 

Been there, done that. Have fun.

> lots of formats that incorporate hashes and CRC checks.

If they are not protected with a cryptographic key, then they don't have integrity checking. The attacker simply replaces both the text and the hash/CRC.

> also need to be mentioned.

OK, I think I see what I missed. You are conflating active and executable content. JSON does not have direct support for active content. JavaScript and Python (and probably others) can treat a JSON object as an executable assignment. It makes good sense to have a security consideration that says "don't do that".

> encompass that range of potential uses.

And you're missing the point that formats like text/plain and text/html have the same property.

> In effect the security considerations

Correct.

> application. But it's entirely doable.

Fully agree.

--Paul Hoffman

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 20 Sep 2012 11:15:56 -0700

On Sep 20, 2012, at 10:51 AM, James M Snell <[email-masked]> wrote:

> I believe the specific concern for JSON is that, in many environments -- especially browsers -- JSON is often parsed using the JavaScript eval function, which allows for the execution of arbitrary JavaScript code. While the JSON specification itself restricts the syntax to non-executable data, a malicious party could intentionally construct an invalid-JSON object such that arbitrary code is executed within the browser. This particular concern is unique for JavaScript relative to other formats (XML, HTML) and really ought be addressed within the security considerations of rfc4627. Specifically, implementors need to be warned against passing unvalidated JSON to the eval function.

Yes, definitely.

--Paul Hoffman

---

From: James M Snell <[email-masked]>
Date: Thu, 20 Sep 2012 11:25:51 -0700

Agreed. The one specific change that I can see right off hand is an
explicit recommendation to utilize JavaScript's JSON.parse function instead
of eval in all cases, despite the fact that JSON can be checked by regex
first. The latter is, by far, the more reliable and secure approach.

- James

On Thu, Sep 20, 2012 at 11:12 AM, Julian Reschke <[email-masked]>wrote:

> [snip]
>
> I would recommend that anybody who wants something specifically changed in
> RFC 4627 first checks whether it might there already (me guilty of that as
> well (*)), and then makes a *concrete* proposal.
>
> Best regards, Julian
>

---

From: Julian Reschke <[email-masked]>
Date: Thu, 20 Sep 2012 20:35:38 +0200

On 2012-09-20 20:25, James M Snell wrote:
> explicit recommendation to utilize JavaScript's JSON.parse function
> instead of eval in all cases, despite the fact that JSON can be checked
> by regex first. The latter is, by far, the more reliable and secure
> approach.
> ...

Indeed.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 20 Sep 2012 14:44:14 -0400

The issue here that the RFC seems to hint at (the language is next to
meaningless) is that JSON does not provide the types of integrity checking
that should be used if you are using it to wrap something like SQL commands
or other data to be passed to a scripting language.

There are encodings that are explicitly designed to make that type of use
safe and there are ways of applying JSON that can make it safe but JSON
does not provide an automatic check.

If that is not the issue that the SC section is trying to explain then lets
have someone else try to explain what the SC in the media type registration
means.

---

From: Julian Reschke <[email-masked]>
Date: Thu, 20 Sep 2012 22:09:54 +0200

On 2012-09-20 20:44, Phillip Hallam-Baker wrote:
> meaningless) is that JSON does not provide the types of integrity
> checking that should be used if you are using it to wrap something like
> SQL commands or other data to be passed to a scripting language.
> ...

WHICH language?

As far as I can tell, the spec describes how to pass names strings, 
numbers and boolean around. It doesn't care at all what you put into 
strings (and it shouldn't).

> There are encodings that are explicitly designed to make that type of
> use safe and there are ways of applying JSON that can make it safe but
> JSON does not provide an automatic check.

Automatic checks of what? Strings are strings are strings. What do you 
want to check?

> If that is not the issue that the SC section is trying to explain then
> lets have someone else try to explain what the SC in the media type
> registration means.

It says:

>    Generally there are security issues with scripting languages.  JSON
>    is a subset of JavaScript, but it is a safe subset that excludes
>    assignment and invocation.

That sounds pretty clear to me.

>    A JSON text can be safely passed into JavaScript's eval() function
>    (which compiles and executes a string) if all the characters not
>    enclosed in strings are in the set of characters that form JSON
>    tokens.  This can be quickly determined in JavaScript with two
>    regular expressions and calls to the test and replace methods.
>
>       var my_JSON_object = !(/[^,:{}\[\]0-9.\-+Eaeflnr-u \n\r\t]/.test(
>              text.replace(/"(\\.|[^"\\])*"/g, ''))) &&
>          eval('(' + text + ')');

These are instructions how to prevent a a Javascript code injection in 
case you choose to use the JS eval() function to parse the JSON.

Best regards, Julian

---

From: Ned Freed <[email-masked]>
Date: Thu, 20 Sep 2012 13:45:18 -0700 (PDT)

> But that's not JSON; it's just something that remotely looks like it.

A while back we wrote an RFC explaining how to represent the Sieve mail
filtering langauge in XML. At that time someone told me they had done the same
thing using JSON. Syntactically this absolutely was JSON, but semantically it
was Sieve, an executable language. 

Now, maybe to you that's not JSON but rather "something that remotely looks
like it". But to me it most definitely *is* JSON, because abstract JSON without
some attached semantics is both meaningless and useless.

This is what I mean when I call JSON a container format. And there is long
established precedent for describing both the security considerations inherent
in the abstract format as well as those that are likely to arise when semantics
are attached. 

> what's JSON and what is not.

Well, if you're talking about having to violate syntax rules in order for
executable semantics to attach, then I'm afraid you're talking nonsense.

				Ned

---

From: Julian Reschke <[email-masked]>
Date: Thu, 20 Sep 2012 23:08:58 +0200

> executable semantics to attach, then I'm afraid you're talking nonsense.

Depends on the layer we look at.

To make JSON code executable as JavaScript *code*, you need to violate 
the JSON syntax.

Of course you can define uses of JSON (or text/plain, or text/xml, or 
...) that have executable semantics on a different layer, but that's not 
what I was referring to.

Best regards, Julian

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 20 Sep 2012 17:22:57 -0400

>> SQL commands or other data to be passed to a scripting language.

Well the spec seems to have an assumption that the JSON data is being
consumed by a JavaScript implementation (and I say seems because it is
clear as mud).

I would have to actually want to understand JavaScript to understand the
point it tries to make. That is not exactly appropriate for an SC section
(to put it mildly).

This is one instance of code/data conflation which is the principal
mechanism for attacking Web Services. Come to that it is the main method of
attack on any system since the attackers usual objective is to execute code.

The security of JSON depends on the encoder properly escaping strings. If
this is not done an attacker can do an injection attack so code that thinks
it is producing messages like the following:

{ "user" : "alice" }

Can be tricked into producing a message like the following:

{ "user" : "alice", "privs" : "admin" }

What the user did to create that was to set their username to the string
<alice", "privs" : "admin>

There are encodings in use that are specifically designed to avoid that
particular issue. ASN.1 does not have that problem as the data fields are
length delimited rather than relying on escaping which is rather safer.
Another way to avoid them as issues is to use a schema and a compiler to
manage the mapping from one to another.

The main reason I might want to use something other than JSON as a data
encoding would probably be to achieve better efficiency or to ensure that
issues like this are excluded in a systematic fashion.

---

From: Julian Reschke <[email-masked]>
Date: Thu, 20 Sep 2012 23:33:11 +0200

On 2012-09-20 23:22, Phillip Hallam-Baker wrote:
>
>
> On Thu, Sep 20, 2012 at 4:09 PM, Julian Reschke <[email-masked]
> point it tries to make. That is not exactly appropriate for an SC
> section (to put it mildly).
>
> mechanism for attacking Web Services. Come to that it is the main method
> of attack on any system since the attackers usual objective is to
> execute code.
>
>
> The security of JSON depends on the encoder properly escaping strings.
> If this is not done an attacker can do an injection attack so code that
> thinks it is producing messages like the following:
>
> { "user" : "alice" }
>
> particular issue. ASN.1 does not have that problem as the data fields
> are length delimited rather than relying on escaping which is rather
> safer. Another way to avoid them as issues is to use a schema and a
> compiler to manage the mapping from one to another.
>
> encoding would probably be to achieve better efficiency or to ensure
> that issues like this are excluded in a systematic fashion.
> ...

I still have trouble following.

If your have a broken sender, you can get broken messages. That's true 
for all formats, but I agree that some formats might be easier to exploit.

Anyway: if you want the JSONbis RFC to be better, please make a concrete 
proposal.

Best regards, Julian

---

From: Ned Freed <[email-masked]>
Date: Thu, 20 Sep 2012 13:55:22 -0700 (PDT)

> We fully disagree about text/plain, then. Text/plain is often used for
> configuration files that are slurped in with inadequate checking.

Doesn't change the fact that text/plain is defined as *unformatted* material.
So the correct term is "misused", not "used". The problem with label misuse is
that the minute you start treating that as not only a serious security concern
in and of itself (which of course it is), but also one whose ramifications must
be fully elaborated at each and every turn you're back again to the problem of
every conceivable security consideration appearing in every possible context.

Not too long ago I did OCR on a PNG file to extract some configuration
information. I could easily envision a system that automated this. That doesn't
mean it's appropriate to talk about code injection attacks for image/png.

What we do to address this is say "labels may be misleading" and ideally give
examples of some of the more pernicious problems, and of course the problem of
sensitive formatted material being shuffled around as plain text is a big one.

> Of course. But I thought we were talking about formats like JSON, not

But the point is profiles (to use your term) are going to exist where integrity
protections is provided, just as profiles are going to exist where it isn't
needed. And there will be some where it essentially it be applied externally.

And the point is to describe these issues and tradeoffs in the security
considerations.

> If they are not protected with a cryptographic key, then they don't have
> integrity checking. The attacker simply replaces both the text and the
> hash/CRC.

I'm well aware of the issues, thanks. Maybe you and perhaps others wish it were
otherwise, but the term "integrity protection" doesn't necessarily mean
cryptographic checks are involved.

> OK, I think I see what I missed. You are conflating active and executable
> content. JSON does not have direct support for active content. JavaScript and
> Python (and probably others) can treat a JSON object as an executable
> assignment. It makes good sense to have a security consideration that says
> "don't do that".

I never said anything about the direct executable case, so I don't see how I
could be conflating them. All of this has been about the indirect case.

> And you're missing the point that formats like text/plain and text/html have

Please don't put words in my mouth. I never said anything about text/html, and
since text/html as defined supports a wide variety of both active and
executable content any claim to the contrary would be absurd.

But as for text/plain, you're the one conferring properities on the format *as*
*defined* that it simply does not have. Again, misuse of this and almost
everything else abounds, but the minute you start trying to account for every
sort of misuse everywhere you're sunk.

				Ned

---

From: James M Snell <[email-masked]>
Date: Thu, 20 Sep 2012 14:43:36 -0700

Perhaps it's not clear because there is no such assumption being made at
all in the spec. Perhaps the one point that can and should be made clearer
is that despite json being defined as a subset of the JavaScript syntax,
json must not be processed as JavaScript.

I'm sorry, but I must have missed something. Are we playing a game of State
the Obvious now? The existing json spec very clearly details escaping. Any
implementation that fails to conform to the basic requirements of the spec
should be expected to function incorrectly. Such issues are not the fault
of the spec, however.

- james

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 20 Sep 2012 19:26:55 -0400

What makes it a security consideration is the possibility that an ATTACKER
might intentionally create a malicious encoding.

It is like the fact that C uses a braindead array scheme makes it easy to
create buffer overrun errors that are impossible in BASIC, FORTRAN, Pascal,
C#, Java and pretty much every other language. Relying on programmers to
write correct code without security holes is a bad security practice.

Using a schema tool to generate encoders is usually a sufficient control.

It is not a big security concern but it is a concern.

On Thu, Sep 20, 2012 at 5:33 PM, Julian Reschke <[email-masked]>wrote:

> If your have a broken sender, you can get broken messages. That's true for
> all formats, but I agree that some formats might be easier to exploit.
>

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 20 Sep 2012 19:35:19 -0400

On Thu, Sep 20, 2012 at 5:43 PM, James M Snell <[email-masked]> wrote:

> I'm sorry, but I must have missed something. Are we playing a game of
> State the Obvious now?
>
Bingo!

Yes, we are playing a game of state the obvious because that is what most
security issues tend to end up being. If you have a million lines of code
then the chance of at least one of them having an obvious error is pretty
much 100%. It is rather obvious that a buffer overrun is likely to be a bad
thing but it took over a decade for people to realize that it allowed
people to smash the stack and pown the machine.

And given that it has taken us the best part of a day to get to the point
where the explanation is obvious I think that the problem itself is far
from obvious.

This attack vector (in different incarnations) is currently the most
commonly used attack against Web Services.

---

From: Ned Freed <[email-masked]>
Date: Thu, 20 Sep 2012 16:28:55 -0700 (PDT)

> the Obvious now?

Yes, in point of fact that's exactly what security considerations sections
often end up doing. At least from the perspective of some. But not everyone has
the same experience or is able to draw conclusions that others see as obvious.

The implications of substituting strings into SQL are obvious to anyone who
is familiar with the syntax. But that has not exactly eliminated SQL injections
attacks.

And no, detailing this stuff in the security considerations for
application/json won't prevent it from happening for JSON. But it might help a
little.

As seems to always be the case in matters of security, there are tradeoffs and
a balance must be struck. Detailing every conceivable thing no matter how
unlikely is not the answer, but neither is some rather cryptic directions about
what functions to use, which is what is there now.

> The existing json spec very clearly details escaping. Any

This is about producing the best outcome, not finding fault.

				Ned

---

From: Ned Freed <[email-masked]>
Date: Thu, 20 Sep 2012 16:36:40 -0700 (PDT)

> ...) that have executable semantics on a different layer, but that's not

But it isn't what was being discussed in the message you were responding to.
If you want to change the subject, you need to say as much.

				Ned

---

From: "Paul C. Bryan" <[email-masked]>
Date: Thu, 20 Sep 2012 16:49:44 -0700

On Thu, 2012-09-20 at 19:26 -0400, Phillip Hallam-Baker wrote:

> What makes it a security consideration is the possibility that an
> ATTACKER might intentionally create a malicious encoding.

What malicious encoding? Your example wouldn't have been parsed by any
JSON parser I'm familiar with, accidentally or otherwise.

> It is like the fact that C uses a braindead array scheme makes it easy
> to create buffer overrun errors that are impossible in BASIC, FORTRAN,
> Pascal, C#, Java and pretty much every other language. Relying on
> programmers to write correct code without security holes is a bad
> security practice.

So, you're proposing we indicate under security considerations that
relying on programmers to write correct code without security holes is a
bad security practice?

> Using a schema tool to generate encoders is usually a sufficient

You don't need a schema tool to encode well-formed JSON. 

> It is not a big security concern but it is a concern.

So you raised it why then?

Paul

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 20 Sep 2012 21:45:40 -0400

On Thu, Sep 20, 2012 at 7:49 PM, Paul C. Bryan <[email-masked]> wrote:

> **

If you have an encoding that depends on correct escaping for safety then
the attack is possible. The attack is on the encoder, not the parser.

> relying on programmers to write correct code without security holes is a

It is certainly a security concern and in many environments it is not an
acceptable practice.

The point of having Security Concerns is that they should list the issues
that should be verified in a code review.

The rules for writing a Security Consideration are that it has to specify
all the Security Considerations regardless of whether they are raised
elsewhere. That is the point of having it as a mandatory section.

> You don't need a schema tool to encode well-formed JSON.

It is probably a less costly control than code review which is the only
alternative I am familiar with. Telling people to just hire competent
coders is not an answer, programming is like sex, everybody thinks that
they are good at it.

> So you raised it why then?

Because people asked what should go in the security considerations section
and I am on the Security Directorate so I was telling people the type of
comments I would make if I was reviewing.

---

From: "Paul C. Bryan" <[email-masked]>
Date: Thu, 20 Sep 2012 19:18:43 -0700

On Thu, 2012-09-20 at 21:45 -0400, Phillip Hallam-Baker wrote:

>         > What makes it a security consideration is the possibility
>         > that an ATTACKER might intentionally create a malicious
>         > encoding.
>         
>         What malicious encoding? Your example wouldn't have been
>         parsed by any JSON parser I'm familiar with, accidentally or
>         otherwise.
>         
>         
> 
> 
> If you have an encoding that depends on correct escaping for safety
> then the attack is possible. The attack is on the encoder, not the
> parser. 

Your example wouldn't have been improperly encoded by any JSON encoder
I'm familiar with, accidentally or otherwise. Can you provide a concrete
example of this attack?

>         It is like the fact that C uses a braindead array scheme makes
>         it easy to create buffer overrun errors that are impossible in
>         BASIC, FORTRAN, Pascal, C#, Java and pretty much every other
>         language. Relying on programmers to write correct code without
>         security holes is a bad security practice.
>         
>         So, you're proposing we indicate under security considerations
>         that relying on programmers to write correct code without
>         security holes is a bad security practice?
> 
> 
> It is certainly a security concern and in many environments it is not
> an acceptable practice.

So your answer is, yes, you're recommending such an addition? Is there
precedence in an RFC where such text exists under security
considerations?  

> The point of having Security Concerns is that they should list the
> issues that should be verified in a code review.

Is this purpose documented somewhere normative, or your professional
opinion? I don't see anything about code reviews in RFC 2223 or 3552.

> The rules for writing a Security Consideration are that it has to
> specify all the Security Considerations regardless of whether they are
> raised elsewhere. That is the point of having it as a mandatory
> section.

Are such rules documented somewhere, or your opinion? I don't see
anything about needing to specify ALL security considerations, no matter
where they are raised, or how pertinent they may (or may not!) be.    

> It is probably a less costly control than code review which is the
> only alternative I am familiar with. Telling people to just hire
> competent coders is not an answer, programming is like sex, everybody
> thinks that they are good at it.

It's unclear to me how this anecdote may be relevant to the topic at
hand. Are you suggesting others here are advocating telling people to
just hire competent coders? I've not heard anyone take such a position
on this list.

> Because people asked what should go in the security considerations
> section and I am on the Security Directorate so I was telling people
> the type of comments I would make if I was reviewing. 

Is there a security consideration that you would consider unsuitable for
inclusion in the security considerations section? If, for example, I
were to hypothesize of a vector of attack against JSON that only works
should its text is encoded on punched cards in a character encoding of
my own invention, would you consider this worthy of inclusion?

Paul

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 20 Sep 2012 22:32:08 -0400

Do your encoding with a call to printf as I originally suggested and fail
to wrap the data value with a call to a routine to escape the string and
you will get the vulnerability I cite.

This is really not an obscure attack. It is the JSON equivalent of attacks
that account for a huge proportion of attacks right now. In fact with the
deployment of DEP and similar and the easy ways to do a buffer overrun bug
being closed [1] they are probably the most common form of vulnerability
right now.

[1] Yeah there is still re-entrant code but most hackers haven't got the
chops for that, code injection is a much easier game.

On Thu, Sep 20, 2012 at 10:18 PM, Paul C. Bryan <[email-masked]> wrote:

> Your example wouldn't have been improperly encoded by any JSON encoder I'm
> familiar with, accidentally or otherwise. Can you provide a concrete
> example of this attack?
>
>
> programmers to write correct code without security holes is a bad security
> practice.
>
> precedence in an RFC where such text exists under security considerations?
>
> Are such rules documented somewhere, or your opinion? I don't see anything
> about needing to specify ALL security considerations, no matter where they
> are raised, or how pertinent they may (or may not!) be.
>
>
> It's unclear to me how this anecdote may be relevant to the topic at hand.
> Are you suggesting others here are advocating telling people to just hire
> competent coders? I've not heard anyone take such a position on this list.
>
>
> section and I am on the Security Directorate so I was telling people the
> type of comments I would make if I was reviewing.
>
>
> inclusion in the security considerations section? If, for example, I were
> to hypothesize of a vector of attack against JSON that only works should
> its text is encoded on punched cards in a character encoding of my own
> invention, would you consider this worthy of inclusion?
> Paul
>

---

From: "Paul C. Bryan" <[email-masked]>
Date: Thu, 20 Sep 2012 20:08:47 -0700

On Thu, 2012-09-20 at 22:32 -0400, Phillip Hallam-Baker wrote:

> Do your encoding with a call to printf as I originally suggested and
> fail to wrap the data value with a call to a routine to escape the
> string and you will get the vulnerability I cite.

So if I understand your position correctly, a vulnerability in the use
of C's printf function is worthy of publication under JSON's security
considerations. By this logic, this same vulnerability should be cited
in the security considerations section of every conceivable protocol
where one may ever possibly (even unreasonably) use C's printf function,
which means virtually any text-encoding protocol. Is this correct? 

> This is really not an obscure attack. It is the JSON equivalent of
> attacks that account for a huge proportion of attacks right now. In
> fact with the deployment of DEP and similar and the easy ways to do a
> buffer overrun bug being closed [1] they are probably the most common
> form of vulnerability right now.

I don't quarrel with your assertion that bad formatting functions in
languages can expose security vulnerabilities. However, I don't think
your voluminous and strongly worded suggestions that we should cite
every conceivable vulnerability—despite the nature or relevance to the
protocol specifically—is either realistic or helpful.

> [1] Yeah there is still re-entrant code but most hackers haven't got
> the chops for that, code injection is a much easier game.

Uh huh.

Paul

>         >         > What makes it a security consideration is the
>         >         > possibility that an ATTACKER might intentionally
>         >         > create a malicious encoding. 
>         >         
>         >         What malicious encoding? Your example wouldn't have
>         >         been parsed by any JSON parser I'm familiar with,
>         >         accidentally or otherwise. 
>         >         
>         > 
>         > 
>         > 
>         > If you have an encoding that depends on correct escaping for
>         > safety then the attack is possible. The attack is on the
>         > encoder, not the parser. 
>         
>         
>         
>         
>         Your example wouldn't have been improperly encoded by any JSON
>         encoder I'm familiar with, accidentally or otherwise. Can you
>         provide a concrete example of this attack?
>         
>         
>         
>         
>         >         It is like the fact that C uses a braindead array
>         >         scheme makes it easy to create buffer overrun errors
>         >         that are impossible in BASIC, FORTRAN, Pascal, C#,
>         >         Java and pretty much every other language. Relying
>         >         on programmers to write correct code without
>         >         So, you're proposing we indicate under security
>         >         considerations that relying on programmers to write
>         >         correct code without security holes is a bad
>         >         security practice? 
>         > 
>         > 
>         > 
>         > It is certainly a security concern and in many environments
>         > it is not an acceptable practice.
>         
>         
>         
>         
>         So your answer is, yes, you're recommending such an addition?
>         Is there precedence in an RFC where such text exists under
>         security considerations?  
>         
>         
>         
>         > The point of having Security Concerns is that they should
>         > list the issues that should be verified in a code review.
>         
>         
>         
>         
>         Is this purpose documented somewhere normative, or your
>         professional opinion? I don't see anything about code reviews
>         in RFC 2223 or 3552.
>         
>         
>         
>         
>         > The rules for writing a Security Consideration are that it
>         > has to specify all the Security Considerations regardless of
>         > whether they are raised elsewhere. That is the point of
>         > having it as a mandatory section.
>         
>         
>         
>         
>         Are such rules documented somewhere, or your opinion? I don't
>         see anything about needing to specify ALL security
>         considerations, no matter where they are raised, or how
>         pertinent they may (or may not!) be.    
>         
>         
>         
>         
>         > It is probably a less costly control than code review which
>         > is the only alternative I am familiar with. Telling people
>         > to just hire competent coders is not an answer, programming
>         > is like sex, everybody thinks that they are good at it.
>         
>         
>         
>         
>         It's unclear to me how this anecdote may be relevant to the
>         topic at hand. Are you suggesting others here are advocating
>         telling people to just hire competent coders? I've not heard
>         anyone take such a position on this list.
>         
>         
>         
>         
>         > Because people asked what should go in the security
>         > considerations section and I am on the Security Directorate
>         > so I was telling people the type of comments I would make if
>         > I was reviewing. 
>         
>         
>         
>         
>         Is there a security consideration that you would consider
>         unsuitable for inclusion in the security considerations
>         section? If, for example, I were to hypothesize of a vector of
>         attack against JSON that only works should its text is encoded
>         on punched cards in a character encoding of my own invention,
>         would you consider this worthy of inclusion? 
>         Paul
> 
> 
> 
> 
> 
> 
> 
> --

---

From: "Manger, James H" <[email-masked]>
Date: Fri, 21 Sep 2012 14:43:34 +1000

>> I'm familiar with, accidentally or otherwise.

Hence the Security Considerations section should tell people to use a dedicated JSON encoder instead of writing JSON by hand with printf calls, since it is too easy to make escaping error with the latter approach.

--
James Manger

---

From: Julian Reschke <[email-masked]>
Date: Fri, 21 Sep 2012 08:57:14 +0200

> If you want to change the subject, you need to say as much.

For the record: I didn't want to change the topic (nor did I, I think).

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Fri, 21 Sep 2012 09:01:47 +0200

On 2012-09-21 01:35, Phillip Hallam-Baker wrote:
> ...
>     I'm sorry, but I must have missed something. Are we playing a game
>     of State the Obvious now?
>
> Bingo!
>
> Yes, we are playing a game of state the obvious because that is what
> most security issues tend to end up being. If you have a million lines
> of code then the chance of at least one of them having an obvious error
> is pretty much 100%. It is rather obvious that a buffer overrun is
> likely to be a bad thing but it took over a decade for people to realize
> that it allowed people to smash the stack and pown the machine.
> And given that it has taken us the best part of a day to get to the
> point where the explanation is obvious I think that the problem itself
> is far from obvious.
>
>

I personally don't believe that "stating the obvious" is something we 
should do in Security Considerations.

First, because those implementers who even get the obvious things wrong 
are unlikely to even look at the spec.

Second, because adding tons of "obvious" things to the SC totally 
distracts from the *non-obvious* things.

Now what's obvious and what's not of course depends on the background. 
Anybody who has dealt with HTML and XML professionally should know that 
constructing messages with simple string operations is dangerous. Maybe 
that's something that's not was obvious in the JSON world yet.

Best regards, Julian

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 21 Sep 2012 11:41:52 -0400

I think it rather obvious that a random number generator used to generate
session keys has to be random. But it took me over a week to beat the idea
into the Netscape security guy who thought that the current date and time
and the IP address was enough randomness for a 128 bit seed.

After he finally got the point I sent him the design note for the random
number scheme I developed for Shen which he then put in the file and handed
over to Taher and co when they took over security. First thing they did was
read the file, find my design note and conclude that the design must be
solid. They never looked at the code. A year later they were hacked when
someone decompiled the code (who would expect an attacker would do that?).

Due to a similar IETF history with Kerberos there is no way that you would
get an RFC issued today without an SC note that the random number generator
has to be random and reference the RFC on the topic.

JSON-BIS is not going to issue without a similar note stating the blatantly
obvious with respect to an escaped encoding.

The point of doing this is

1) To inform the implementer
2) To tell the tester what to look for
3) To assure people reading the spec that the designers are competent.

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 21 Sep 2012 10:21:31 -0700

On 21/09/2012, at 12:01 AM, Julian Reschke <[email-masked]> wrote:
> 
> I personally don't believe that "stating the obvious" is something we should do in Security Considerations.
> 
> First, because those implementers who even get the obvious things wrong are unlikely to even look at the spec.
> 
> Second, because adding tons of "obvious" things to the SC totally distracts from the *non-obvious* things.
> 
> Now what's obvious and what's not of course depends on the background. Anybody who has dealt with HTML and XML professionally should know that constructing messages with simple string operations is dangerous. Maybe that's something that's not was obvious in the JSON world yet.

+1 and well said. 

We agonise over Security Considerations, failing to consider that the people who need to read them won't, so all we do is make ourselves feel better that we've "improved security."

--
Mark Nottingham
http://www.mnot.net/

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 21 Sep 2012 13:32:31 -0400

The primary point of the SC sections is to demonstrate that we at least
have considered these issues.

But the people who read them are often managers and people writing RFPs. If
you have exposure to the policy level you will know that there are many
people who make decisions about technology that have never written a single
line of code and never intend to.

This is not a major issue. If it was a major issue then the response would
be 'heel no, don't endorse this format'.

The process we have here is that when drafts are submitted, someone on
SECDIR is going to do a review and submit comments in last call. I thought
people wanted to know what to put in the SC section to avoid a DISCUSS.
Sorry if that was not clear.

I don't see the need to be so defensive here. I suggest the following as a
path forward, I will write up an SC section for the draft that I think will
cover likely objections. The other outstanding issues with the current
draft seem to be moving the statement about object ordering into the syntax
section and removing the MIME type registration and instead putting in an
IANA section requesting the existing entry be updated.

Anything else missing? Has anyone talked to the original author on this?

On Fri, Sep 21, 2012 at 1:21 PM, Mark Nottingham <[email-masked]> wrote:

> We agonise over Security Considerations, failing to consider that the
> people who need to read them won't, so all we do is make ourselves feel
> better that we've "improved security."
>
> --
> Mark Nottingham
> http://www.mnot.net/
>
>
>
>
>

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 21 Sep 2012 10:40:46 -0700

On 21/09/2012, at 10:32 AM, Phillip Hallam-Baker <[email-masked]> wrote:

> Has anyone talked to the original author on this?

Yes, I have, at the request of the ADs.

He's fine with it going standards-track, and has a few ambiguities he'd like to fix. He is concerned about people seeing an opportunity to "improve it", thereby breaking compatibility, but it seems we have this pretty well in hand.

Regards,

--
Mark Nottingham
http://www.mnot.net/

---

From: Paul Hoffman <[email-masked]>
Date: Fri, 21 Sep 2012 10:49:13 -0700

On Sep 21, 2012, at 10:32 AM, Phillip Hallam-Baker <[email-masked]> wrote:

> The primary point of the SC sections is to demonstrate that we at least have considered these issues.

A giant -1 to that.

--Paul Hoffman