Subject: Re: [apps-discuss] Last Call: <draft-ietf-appsawg-json-pointer-07.txt> (JSON Pointer) to Proposed Standard

From: Robert Sayre <[email-masked]>
Date: Sat, 5 Jan 2013 16:19:03 -0800

Mark,

The WG's reasoning, as stated in your message below, seems flawed.
Messages since your last communication on this matter have shown:

1) The ambiguity around arrays makes the patch format unsuitable for
common concurrent editing algorithms.
2) The ambiguity is likely to occur in the real world, for a couple of
different reasons.
3) It's not possible to tell whether a JSON Pointer document is
syntactically correct in isolation.

Additionally, you raised this point in your message below:
>
> the patch author already has to understand the semantics of the document they're patching

That claim does not seem to be well-justified, and it could be
meaningless to the person implementing patch software (for example:
https://github.com/sayrer/json-sync).

This issue is a problem in practice, and it's a problem in theory as
well. JSON-Patch messages aren't sufficiently self-descriptive, so
they aren't appropriate for use in a RESTful system.

A response containing technical reasoning seems in order, since the
points raised by myself and others on this issue are unrelated to the
WG's previous thinking.

- Rob

On Sun, Dec 16, 2012 at 9:41 PM, Mark Nottingham <[email-masked]> wrote:
> Robert,
>
> This was discussed extensively in the Working Group.
>
> The root of the issue was that some people reflexively felt that this was necessary, but upon reflection, we decided it wasn't; although it seems "natural" to some, especially those coming from a static language background, it didn't provide any utility.
>
> You might argue that someone who (for example) adds to "/foo/1" in the mistaken belief that it's an array, when in fact it's an object, will get surprising results. That's true, but if we were to solve this problem, that person would still need to understand the underlying semantics of "foo" to do anything useful to it -- and I'm not hearing anyone complain about that (I hope).
>
> Put another way -- do you really think that people PATCHing something as if it's an array (when in fact it's an object) is a significant, real-world problem, given that the patch author already has to understand the semantics of the document they're patching? I don't, and the WG didn't either.
>
> Regards,
>
>
> On 17/12/2012, at 3:36 PM, Robert Sayre <[email-masked]> wrote:
>
>> The cost of fixing it seems low, either by changing the path syntax of
>> JSON pointer or changing the names of operations applied to arrays.
>> Array-like objects are common enough in JavaScript to make this a
>> worry. The other suggestions either assume a particular policy for
>> concurrent edits or require more machinery (test operation etc).
>> Wouldn't it be simpler to make the patch format more precise?
>>
>> - Rob
>>
>> On Sun, Dec 16, 2012 at 4:33 PM, Matthew Morley <[email-masked]> wrote:
>>> I am usually lurking and struggling to keep up with these posts. But, I
>>> concur with James, this really is a non-issue in practice.
>>>
>>> The JSON Pointer expresses a path down a JSON object to a specific context.
>>> The Patch expresses a change within or to that context.
>>> Everything about the both standards is about that end context.
>>>
>>> If you want to confirm the type of the context before applying a patch, this
>>> should probably be part of a test operation. I'm not sure if this is
>>> possible at this point (?), but that is where the logic should exist.
>>>
>>>
>>>
>>> On Sun, Dec 16, 2012 at 12:22 AM, James M Snell <[email-masked]> wrote:
>>>>
>>>>
>>>>
>>>>
>>>> On Sat, Dec 15, 2012 at 8:36 PM, Robert Sayre <[email-masked]> wrote:
>>>>>
>>>>> On Fri, Dec 14, 2012 at 9:17 AM, Markus Lanthaler
>>>>> <[email-masked]> wrote:
>>>>>>
>>>>>> Hmm.. I think that’s quite problematic. Especially considering how JSON
>>>>>> Pointer is used in JSON Patch.
>>>>>
>>>>> I agree--I provided the same feedback privately. It seems
>>>>> straightforwardly unsound.
>>>>>
>>>>
>>>> In practice it doesn't seem to be much of an issue.
>>>>
>>>> Specifically, if I GET an existing document and get an etag with the JSON,
>>>> then make some changes and send a PATCH with If-Match, the fact that any
>>>> given pointer could point to an array or object member doesn't really matter
>>>> much.
>>>>
>>>> For example:
>>>>
>>>>> GET /the/doc HTTP/1.1
>>>>
>>>>  <  HTTP/1.1 200 OK
>>>>     ETag: "my-document-tag"
>>>>     Content-Type: application/json
>>>>
>>>>     {"1":"foo"}
>>>>
>>>>> PATCH /the/doc HTTP/1.1
>>>>     If-Match: "my-document-etag"
>>>>     Content-Type: application/json-patch
>>>>
>>>>     [{"op":"add","path":"/2","value":"bar"}]
>>>>
>>>> Generally speaking, someone should not be using PATCH to perform a partial
>>>> modification if they don't already have some knowledge in advance what they
>>>> are modifying. The only time the apparent ambiguity becomes an issue is when
>>>> a client is blindly sending a patch to an unknown endpoint... in which case,
>>>> you get whatever you end up with.
>>>>
>>>> - James
>>>>
>>>>
>>>>>
>>>>> - Rob
>>>>>
>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Markus Lanthaler
>>>>>>
>>>>>> @markuslanthaler
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> From: James M Snell [mailto:[email-masked]]
>>>>>> Sent: Friday, December 14, 2012 5:41 PM
>>>>>> To: Markus Lanthaler
>>>>>> Cc: IETF Discussion; IETF Apps Discuss
>>>>>> Subject: Re: [apps-discuss] Last Call:
>>>>>> <draft-ietf-appsawg-json-pointer-07.txt> (JSON Pointer) to Proposed Standard
>>>>>>
>>>>>>
>>>>>>
>>>>>> JSON Pointer does not distinguish between objects and arrays. That is
>>>>>> not determined until the pointer is applied to an actual object instance...
>>>>>> the pointer "/1" is valid against {"1":"a"} or ["a","b"]
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Dec 14, 2012 at 2:51 AM, Markus Lanthaler
>>>>>> <[email-masked]> wrote:
>>>>>>
>>>>>> I've asked that before but didn't get an answer. So let me ask again
>>>>>> (even
>>>>>> though I'm quite sure it has already been asked by somebody else).
>>>>>>
>>>>>> How does JSON Pointer distinguish between objects and arrays? E.g.
>>>>>> consider
>>>>>> the following JSON document:
>>>>>>
>>>>>> {
>>>>>>  "foo": "bar",
>>>>>>  "1": "baz"
>>>>>> }
>>>>>>
>>>>>> As I read the draft, the JSON Pointer "/1" would evaluate to "baz" even
>>>>>> though that's probably not what the author intended. Is there a way to
>>>>>> avoid
>>>>>> that?
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Markus
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Markus Lanthaler
>>>>>> @markuslanthaler
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: [email-masked] [mailto:apps-discuss-
>>>>>>> [email-masked]] On Behalf Of The IESG
>>>>>>> Sent: Tuesday, December 11, 2012 4:01 PM
>>>>>>> To: IETF-Announce
>>>>>>> Cc: [email-masked]
>>>>>>> Subject: [apps-discuss] Last Call: <draft-ietf-appsawg-json-pointer-
>>>>>>> 07.txt> (JSON Pointer) to Proposed Standard
>>>>>>>
>>>>>>>
>>>>>>> The IESG has received a request from the Applications Area Working
>>>>>>> Group
>>>>>>> WG (appsawg) to consider the following document:
>>>>>>> - 'JSON Pointer'
>>>>>>>  <draft-ietf-appsawg-json-pointer-07.txt> as Proposed Standard
>>>>>>>
>>>>>>> The IESG plans to make a decision in the next few weeks, and solicits
>>>>>>> final comments on this action. Please send substantive comments to
>>>>>>> the
>>>>>>> [email-masked] mailing lists by 2012-12-25. Exceptionally, comments
>>>>>>> may
>>>>>>> be
>>>>>>> sent to [email-masked] instead. In either case, please retain the
>>>>>>> beginning of the Subject line to allow automated sorting.
>>>>>>>
>>>>>>> Abstract
>>>>>>>
>>>>>>>
>>>>>>>   JSON Pointer defines a string syntax for identifying a specific
>>>>>>> value
>>>>>>>   within a JSON document.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> The file can be obtained via
>>>>>>> http://datatracker.ietf.org/doc/draft-ietf-appsawg-json-pointer/
>>>>>>>
>>>>>>> IESG discussion can be tracked via
>>>>>>>
>>>>>>> http://datatracker.ietf.org/doc/draft-ietf-appsawg-json-pointer/ballot/
>>>>>>>
>>>>>>>
>>>>>>> No IPR declarations have been submitted directly on this I-D.
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> apps-discuss mailing list
>>>>>>> [email-masked]
>>>>>>> https://www.ietf.org/mailman/listinfo/apps-discuss
>>>>>>
>>>>>> _______________________________________________
>>>>>> apps-discuss mailing list
>>>>>> [email-masked]
>>>>>> https://www.ietf.org/mailman/listinfo/apps-discuss
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> apps-discuss mailing list
>>>>>> [email-masked]
>>>>>> https://www.ietf.org/mailman/listinfo/apps-discuss
>>>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> apps-discuss mailing list
>>>> [email-masked]
>>>> https://www.ietf.org/mailman/listinfo/apps-discuss
>>>>
>>>
>>>
>>>
>>> --
>>> Matthew P. C. Morley
>> _______________________________________________
>> apps-discuss mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/apps-discuss
>
> --
> Mark Nottingham   http://www.mnot.net/
>
>
>

---

From: James M Snell <[email-masked]>
Date: Sat, 5 Jan 2013 17:31:04 -0800

Robert,

I may have missed it, but can you provide a non-theoretical example of this
problem that you're suggesting exists in practice?

> common concurrent editing algorithms.

Why? I'm still not seeing how it's unsuitable. Again, a non-theoretical
example would be helpful.

> 2) The ambiguity is likely to occur in the real world, for a couple of

Such as? What are the reasons?

> syntactically correct in isolation.

There is no such thing as a "JSON Pointer document" and I have absolutely
no idea what "syntactically correct in isolation" means with regards to
this problem you're suggesting. If I see "/a/b/1", that is a syntactically
correct JSON Pointer... whether or not it points to anything specific
depends entirely on the specific JSON structure it is applied to. If I had
to guess, you're saying that it's not possible to tell if "/a/b/01" is a
valid JSON Pointer or not given nothing but the pointer? If so, who cares
really? The JSON Pointer is not useful unless it's applied to an actual
JSON structure, it's only at that point that we really ought to care about
validity.

Still not seeing the problem.

> they aren't appropriate for use in a RESTful system.
What would be "sufficiently self-descriptive" ? Again, a non-theoretical
example and suggested alternative that we can compare would be helpful for
context.

It is possible that I missed a couple of posts on this over the holiday so
if you already provided an example, please do let me know and I'll go
hunting through the archives.

- James

> > The root of the issue was that some people reflexively felt that this
> was necessary, but upon reflection, we decided it wasn't; although it seems
> "natural" to some, especially those coming from a static language
> background, it didn't provide any utility.
> >
> > You might argue that someone who (for example) adds to "/foo/1" in the
> mistaken belief that it's an array, when in fact it's an object, will get
> surprising results. That's true, but if we were to solve this problem, that
> person would still need to understand the underlying semantics of "foo" to
> do anything useful to it -- and I'm not hearing anyone complain about that
> (I hope).
> >
> > Put another way -- do you really think that people PATCHing something as
> if it's an array (when in fact it's an object) is a significant, real-world
> problem, given that the patch author already has to understand the
> semantics of the document they're patching? I don't, and the WG didn't
> either.
> >
> > Regards,
> >
> >

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 6 Jan 2013 12:48:10 +1100

Robert,

I neither represent the WG (except in as far as I attempt to do so as document editor), nor do I judge consensus in the WG (the Chairs do, although at this late date, the IESG are making the decisions).

That said, if we were starting this from scratch, I *personally* could see adding some syntax to distinguish intent, as I don't see it causing any huge amount of harm (besides leading some to believe that JSON Pointer is a proto-schema language). 

However, at this point, doing so really a judgement call; we have multiple implementations, and we shouldn't force them to change arbitrarily. As far as I can see, you haven't convinced anyone that this is a serious enough problem to do so (and I don't appear to be the only one to hold that opinion, by any means). Furthermore, it's not clear that the use cases you have in mind (since you have brought up JSON Sync) are in-scope for these specifications.

Soon, the IESG will make its determination, so you'd likely be much more productive by laying out your argument cogently to them, rather than focusing on me.

If we were to open up to further changes, I'd like to see us discuss things like allowing header modifications in the format, and specifying the target's intended media type (which was already discussed and rejected in the WG). 

However, I'm even more interested in getting this format published, in the knowledge that HTTP PATCH has been defined for some time, but is effectively useless (at least against JSON) without any defined, stable patch format. If we find serious deficiencies, nothing stops us, or you, or anyone else from defining a new patch format (as James is already doing).

Cheers,

---

From: Robert Sayre <[email-masked]>
Date: Sat, 5 Jan 2013 18:29:48 -0800

On Sat, Jan 5, 2013 at 5:48 PM, Mark Nottingham <[email-masked]> wrote:
>
> However, at this point, doing so really a judgement call; we have multiple implementations, and we shouldn't
> force them to change arbitrarily.

The word "arbitrarily" seems inappropriate here. I raised at least
four technical issues and your message addresses none of them.

> As far as I can see, you haven't convinced anyone that this is a serious enough problem to do so (and I don't
> appear to be the only one to hold that opinion, by any means).

Did you read this thread? Markus Lanthaler and Conal Tuohy raised
similar points.

> Furthermore, it's not clear that the use cases you have in mind (since you have brought up JSON Sync)
> are in-scope for these specifications.

That assertion is both unsubstantiated and incorrect. json-sync has
identical primitive operations to JSON Patch (create/edit/remove vs
add/replace/remove). The JSON Patch document defines Copy and Move in
terms of the add/replace, so those are mostly syntactic sugar. The
only meaningful delta is the "test" operation, and I do plan to add
that to json-sync, since it's a good way to make application-specific
assertions.

> However, I'm even more interested in getting this format published,

Well, I guess someone has something they want to ship...

- Rob

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 6 Jan 2013 13:59:46 +1100

> four technical issues and your message addresses none of them.

... and I explained why. 

> Did you read this thread? Markus Lanthaler and Conal Tuohy raised

Yes. 

> that to json-sync, since it's a good way to make application-specific

Yes, you've brought that to our attention several times. If you wanted this spec to align with your software, it would have been much easier if you'd got involved before Last Call.

> Well, I guess someone has something they want to ship...

Right. I'll let that statement stand on its own; I think anyone who's been participating or watching the WG can assess how justified it is.

Always a pleasure, Rob.

---

From: Robert Sayre <[email-masked]>
Date: Sat, 5 Jan 2013 20:20:19 -0800

On Sat, Jan 5, 2013 at 6:59 PM, Mark Nottingham <[email-masked]> wrote:
>
> Yes, you've brought that to our attention several times. If you wanted
> this spec to align with your software, it would have been much easier
> if you'd got involved before Last Call.

Well, there shouldn't be any big adjustments to my software at all,
and the document generally looks good. This is just a bug: two parties
can apply the same patch and get different results, without
encountering an error.

> Right. I'll let that statement stand on its own; I think anyone who's been
> participating or watching the WG can assess how justified it is.

Ah. I meant that the WG seems to be favoring "running code" a little
too heavily in the presence of a bug. It's an old argument, and it's
boring: "We can't change it now, there are already twelve users!"

> Always a pleasure, Rob.

That tone leaves something to be desired, but no matter. This is a bug
and the WG should fix it. I don't think more process emails are
necessary.

- Rob

---

From: James M Snell <[email-masked]>
Date: Sat, 5 Jan 2013 20:55:52 -0800

> can apply the same patch and get different results, without

Not seeing the bug... applying the same patch to different resources that
have different states ought to have different results. #worksasintended
#wontfix #moveonthereisnothingmoretoseehere

- james

> >>> However, at this point, doing so really a judgement call; we have
multiple implementations, and we shouldn't
> >>> As far as I can see, you haven't convinced anyone that this is a
serious enough problem to do so (and I don't
> > Yes, you've brought that to our attention several times. If you wanted
this spec to align with your software, it would have been much easier if
you'd got involved before Last Call.
> > Right. I'll let that statement stand on its own; I think anyone who's
been participating or watching the WG can assess how justified it is.
> >>>>> The root of the issue was that some people reflexively felt that
this was necessary, but upon reflection, we decided it wasn't; although it
seems "natural" to some, especially those coming from a static language
background, it didn't provide any utility.
> >>>>>
> >>>>> You might argue that someone who (for example) adds to "/foo/1" in
the mistaken belief that it's an array, when in fact it's an object, will
get surprising results. That's true, but if we were to solve this problem,
that person would still need to understand the underlying semantics of
"foo" to do anything useful to it -- and I'm not hearing anyone complain
about that (I hope).
> >>>>>
> >>>>> Put another way -- do you really think that people PATCHing
something as if it's an array (when in fact it's an object) is a
significant, real-world problem, given that the patch author already has to
understand the semantics of the document they're patching? I don't, and the
WG didn't either.

---

From: Robert Sayre <[email-masked]>
Date: Sat, 5 Jan 2013 21:25:40 -0800

On Sat, Jan 5, 2013 at 8:55 PM, James M Snell <[email-masked]> wrote:
> have different states ought to have different results.

This argument is fallacious. Consider this JSON patch:

{ "op": "remove", "path": "/1" }

This patch can be generated by removing a key from a hashtable by the
sender, and then applied to an array by the recipient (which may
result in array shifts etc). A good quality patch format would not
permit such an obvious ambiguity, because applying that patch can fail
all parties. The resulting document does not reflect the intent of any
author.

I have obviously said my piece. And, fwiw, I don't think the IESG
should contradict the WG.

- Rob

---

From: James M Snell <[email-masked]>
Date: Sat, 5 Jan 2013 21:58:55 -0800

[{"op":"type","path":"","value":"array"},{"op":"remove","path":"/1"}]

Problem solved. Still no bug, and still nothing I can see that needs
fixing.

I've said my piece on it to. Afaic, this spec is done and ready to go.

- James

---

From: "Paul C. Bryan" <[email-masked]>
Date: Sun, 06 Jan 2013 15:35:10 -0800

> On Sat, 2013-01-05 at 16:19 -0800, Robert Sayre wrote:

[snip]

> common concurrent editing algorithms.

Common concurrent editing algorithms should, in my opinion, use
techniques to ensure the state of the resource (relative to the edits)
is known. In HTTP, we have ETag and If-Match/If-None-Match
preconditions. In JSON Patch, we have (a rudimentary) test operation.

[snip]

> syntactically correct in isolation.

There is no such thing as a JSON Pointer document.

> they aren't appropriate for use in a RESTful system.

99% of RESTful systems I'm familiar with are based on HTTP. Where
optimistic concurrency is acceptable, HTTP preconditions seems to
provide acceptable coverage. Where more granularity or more pessimistic
concurrency is required, implementors are free to use their own
mechanisms, including more expressive predicates (as has been proposed
here, with my endorsement) and/or resource locking. These are
intentionally out of scope for JSON Patch.

> boring: "We can't change it now, there are already twelve users!"

I don't agree that this is a bug; it lacks a feature that you and some
others have requested. Our reasoning for resisting such change is
legitimate.

The reason I value implementations is because they endorse the
specification through tangible action. Some of their authors have
participated in this forum to help improve the specification and create
consensus around it. Unfortunately, you've raised objections quite late
in the process, and I'm personally not persuaded that the issues you've
raised warrants (likely significant) changes.

Paul

---

From: Robert Sayre <[email-masked]>
Date: Sun, 6 Jan 2013 16:01:34 -0800

On Sun, Jan 6, 2013 at 3:35 PM, Paul C. Bryan <[email-masked]> wrote:

Thank you for the technical response.

>
> 1) The ambiguity around arrays makes the patch format unsuitable for common
> concurrent editing algorithms.
>
>
> Common concurrent editing algorithms should, in my opinion, use techniques
> to ensure the state of the resource (relative to the edits) is known. In
> HTTP, we have ETag and If-Match/If-None-Match preconditions. In JSON Patch,
> we have (a rudimentary) test operation.

That is sequential editing, not concurrent editing. Here are a few
links to make sure we're not talking past each other:

http://en.wikipedia.org/wiki/Operational_transformation#Basics

The above overview section contains a two-step insert/delete process
bearing an uncanny resemblance to a JSON Patch document.

Here's some software that uses patch documents in this fashion:

http://en.wikipedia.org/wiki/Operational_transformation#OT_software

> There is no such thing as a JSON Pointer document.

Yes, I meant JSON Pointer /reference/. This error seems tangential to
the thread in general, though, which is why I didn't bother to correct
myself when others raised this point.

>
> This issue is a problem in practice, and it's a problem in theory as well.
> JSON-Patch messages aren't sufficiently self-descriptive, so they aren't
> appropriate for use in a RESTful system.
>
>
> 99% of RESTful systems I'm familiar with are based on HTTP. Where optimistic
> concurrency is acceptable, HTTP preconditions seems to provide acceptable
> coverage. Where more granularity or more pessimistic concurrency is
> required, implementors are free to use their own mechanisms, including more
> expressive predicates (as has been proposed here, with my endorsement)
> and/or resource locking. These are intentionally out of scope for JSON
> Patch.

Fully disagree. My point is that the array ambiguity makes it hard to
implement more optimistic concurrency.

> Ah. I meant that the WG seems to be favoring "running code" a little too
> heavily in the presence of a bug. It's an old argument, and it's boring: "We
> can't change it now, there are already twelve users!"
>
>
> others have requested.

The only rationale I've seen prior to my message was something about
"feeling natural" and "static languages" that I admit I don't fully
understand. Are there other technical arguments I've missed? I'm happy
to go read up.

> Our reasoning for resisting such change is
> legitimate.

This last assertion really isn't qualified very well. Do you have a
more extensive rationale, or is a simple contradiction the extent of
your argument?

- Rob

---

From: Robert Sayre <[email-masked]>
Date: Sun, 6 Jan 2013 17:15:05 -0800

>> we have (a rudimentary) test operation.
...
> links to make sure we're not talking past each other:

Actually, let me restate my point in terms of RFC5789 (HTTP PATCH).
That will make it easier to communicate.

RFC 5789 Section 2.2 (Error Handling) defines error conditions which
correspond directly to the point at hand: 'Conflicting state' and
'Conflicting modification'. Section 5 of the JSON Patch document
directly references RFC5789, Section 2.2.

With that in mind, let's note that there are several normative
requirements in the JSON Patch document directed at conflicting state.
One such example is from Section 4.2 'remove'. It reads: "The target
location MUST exist for the operation to be successful.". If a server
received an HTTP JSON Patch request attempting to delete a
non-existent location, this text from RFC5789 would seem to apply:

"Conflicting state:  Can be specified with a 409 (Conflict) status
      code when the request cannot be applied given the state of the
      resource.  For example, if the client attempted to apply a
      structural modification and the structures assumed to exist did
      not exist ..."

The text above wouldn't be necessary in JSON Patch or RFC5789 if
RFC5789 required checking ETags and preconditions for all use cases
(it doesn't). The larger point is that RFC5789, and patch formats in
general, make all sorts of allowances for *non-conflicting* concurrent
edits to a common ancestor. The problem with leaving this JSON Pointer
array ambiguity in the draft is that patch messages which should
trigger '409 Conflict' errors can be mistakenly and 'successfully' (in
the HTTP sense) applied to a different structure than intended.

In summary, the JSON Patch draft allows patch documents to be
formulated that make it impossible to correctly implement RFC5789, a
normative reference.

Here are the questions the IESG focuses on during review:
"Reviews should focus on these questions: 'Is this document a
reasonable basis on which to build the salient part of the Internet
infrastructure? If not, what changes would make it so?'"

For JSON Patch, the answer to the first question is 'no', because of a
deficiency in JSON Pointer. The change needed to make these documents
acceptable as part of the Internet infrastructure is to make Arrays
explicit in JSON Pointer syntax.

- Rob

---

From: Matthew Morley <[email-masked]>
Date: Sun, 6 Jan 2013 22:23:47 -0500

On Sun, Jan 6, 2013 at 8:15 PM, Robert Sayre <[email-masked]> wrote:

> techniques

For me the deficiency is not in the pointer, but patch format being
generated.

One approach is to push that *one* test, structure conformity, into the
pointer syntax. Another is via the type operation.

If a vague patch is generated, vague results are to be expected.

Testing for *just* the structure does not really create a verbose patch
either. Which is why I am not overly in favor of a syntax specific to
arrays, with this argument.

For example, if you are replacing a key in a object, the json-pointer to
get there, *and* the value would be required to ensure it is not
vague. Setting /a/b/c to 6 when it was 4, and applying that patch without
such tests to a document where /a/b/c is 13, is also vague.

Without tests, the patch format is optimistic at best, there is no escaping
this fact. Changing the spec for the json-pointer syntax to address
vagueness in json-patch specification seems wrong to me.

If json-pointer is not well suited, because of the desire for a descriptive
path which includes structure and value, perhaps a different specification
is need. One that provides both path, structure, and value confirmation in
the pointer string. Though at that point is more of a query path, so
something like JsonPath (http://goessner.net/articles/JsonPath/)?

I prefer the test/type operations and the json-pointer specification, with
optimistic patches.

-- 
Matthew P. C. Morley

---

From: Conal Tuohy <[email-masked]>
Date: Tue, 08 Jan 2013 08:39:07 +1000

On 07/01/13 13:23, Matthew Morley wrote:
> One approach is to push that *one* test, structure conformity, into 
> the pointer syntax. Another is via the type operation.
>
It seems to me, on the contrary, that the deficiency is in the pointer 
syntax, and I think it would be a mistake to try to work around that 
deficiency in JSON Patch. Because aren't there other things which one 
might do with JSON Pointer than use it with JSON Patch? There's been 
mention of having it registered as a URI fragment identifier syntax for 
JSON for example. JSON Pointers could then end up all over the place, 
outside of patches. IMHO JSON Pointer needs to be taken seriously as a 
technology in its own right.

-- 
Conal Tuohy
HuNI <http://huni.net.au/> Technical Coordinator 
<http://apidictor.huni.net.au/trac/wiki/ConalSpace>
Victorian eResearch Strategic Initiative 
<http://versi.edu.au/about-us/versi-team#Con>
Skype: conal.tuohy
Twitter: @conal_tuohy <https://twitter.com/conal_tuohy>

---

From: "Paul C. Bryan" <[email-masked]>
Date: Mon, 07 Jan 2013 17:33:10 -0800

On Sun, 2013-01-06 at 16:01 -0800, Robert Sayre wrote:

> This last assertion really isn't qualified very well.

It would have been better for me to state this is my opinion, based on
discussions that were animated from similar objections raised in the
past. 

Paul

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 7 Jan 2013 20:40:19 -0800

I apologize for being absent for this thread until now.  Vacation and
medical matters interfered with me keeping current.

First, with my participant hat on:

I've been occasionally comparing this work to conventional UNIX "patch" to
try to maintain a point of reference as these works developed.  As such,
I'm swayed by the argument (which, as I recall, was part of working group
deliberations prior to WGLC) that we have the "test" operations, so people
generating patch documents should use them to ensure proper context before
applying any of the operations that alter the target.  UNIX "patch"
accomplishes this by default by surrounding the lines to be changed in the
target with context lines that aren't changed, and so must exist precisely
as-is before the change can be made or the change is rejected.  Consider a
target file comprising 26 lines, each containing the next character of the
upper-case English alphabet and a newline, but the M and the N lines are
swapped.  A typical patch to fix this would look like so:

--- x   Mon Jan  7 20:27:36 2013
+++ y   Mon Jan  7 20:27:40 2013
@@ -10,8 +10,8 @@
 J
 K
 L
-N
 M
+N
 O
 P
 Q

The default for UNIX "diff" is to produce three lines of context above and
below the change to be made to ensure the change is being made in the right
place.  One could also request no lines of context, yielding:

--- x   Mon Jan  7 20:27:36 2013
+++ y   Mon Jan  7 20:27:40 2013
@@ -13 +12,0 @@
-N
@@ -14,0 +14 @@
+N

But this doesn't bother to check any context, except of course to ensure
that the target file is at least 14 lines long.  Although the top one is
clearly safer, both are actually legal patches.

In my view, these two JSON documents present a language for referencing and
object and changing it, and also for querying for context, just like the
conventional UNIX diff/patch format does.  But in neither the UNIX case nor
the proposed JSON case is the context part mandatory to use (though one
could certainly argue it's foolish to skip doing so).  That seems fine to
me.

Then, with my co-chair hat on:

Although I hear and understand Robert's position that this is an important
thing that needs to be addressed, it is not my view after reviewing this
thread that there's rough consensus to reopen the question.  Please note
that this is not an "it's too late in the process to change this" position,
but rather one that notes that the burden of supporting a change to
something that already has rough consensus is on the person proposing the
change, and I don't believe Robert has succeeded here.

That said, I would ask the document editors to consider adding a paragraph
or an appendix indicating this issue was considered during development of
the work and the current format was deliberately selected, preferably with
some supporting text.  This will ensure future readers will not interpret
the chosen design as a bug, but rather an intentional design choice.

-MSK, APPSAWG co-chair

On Mon, Jan 7, 2013 at 5:33 PM, Paul C. Bryan <[email-masked]> wrote:

> **
> discussions that were animated from similar objections raised in the past.
>
> Paul
>

---

From: James M Snell <[email-masked]>
Date: Mon, 7 Jan 2013 20:47:17 -0800

If I may, I would follow this up with a suggestion that a separate I-D that
provides a more complete treatment of fully-concurrent patch operations
would be helpful. JSON-Patch is largely designed around atomic and
sequential modification operations and is not, necessarily a great match
for the kind of OT-style mechanisms Robert was referencing. I don't
personally have any use cases that would require the level of concurrency
Robert is suggesting but it would be an interesting pursuit nonetheless.

On Mon, Jan 7, 2013 at 8:40 PM, Murray S. Kucherawy <[email-masked]>wrote:

> In my view, these two JSON documents present a language for referencing
> and object and changing it, and also for querying for context, just like
> the conventional UNIX diff/patch format does.  But in neither the UNIX case
> nor the proposed JSON case is the context part mandatory to use (though one

---

From: Matthew Morley <[email-masked]>
Date: Tue, 8 Jan 2013 01:26:29 -0500

On Mon, Jan 7, 2013 at 5:39 PM, Conal Tuohy <[email-masked]>wrote:

> deficiency in JSON Patch. Because aren't there other things which one might
> do with JSON Pointer than use it with JSON Patch? There's been mention of
> having it registered as a URI fragment identifier syntax for JSON for
> example. JSON Pointers could then end up all over the place, outside of
> patches. IMHO JSON Pointer needs to be taken seriously as a technology in
> its own right.
>

Couldn't agree more about it being taken seriously in its own right. :)

JSON Pointer for me exists outside of JSON Patch, always has and will do
the way we think about structures. As it represents both a resolution path
and an identity string (both ends of the path concept). I see value from
the identity view, in describing a location that is aware of being inside
an array.

But JSON Pointer should not be changed just because of issues with JSON
Patch, especially when JSON Patch is attempting to address those issues
with other mechanisms within the specification. That is all I was trying to
express. The syntax change should be for other reasons, if it is going to
be made.

My personal experience (for what its worth): In the past I've tried a
number of syntaxes like JSON Pointer. Mostly a.b.c.0 and even a.b.c:0 at
times to address the same issues suggested here. Though my experiences
pushed me towards a single syntax using a.b.c.0, and thus my support for
/a/b/c/0 over /a/b/c:0.

The system at first used the . or : syntax, combined with dynamic tokens,
being pointers themselves, to resolve other pointers. So it was not
reasonable to know ahead of time if an end point was an array or an object.
 "a.b.c.{d.e.f}" could end up in an array or in an object, depending on the
value at d.e.f at the time of resolution. Especially with many layers of
tokens to resolve, and changing data structures.

I found in practice, it didn't really matter, so the choice of . or : was
phased out. At the end of the day the two syntaxes point to mutually
exclusive points within the data, so that `meta data` about the structure
was removed from the syntax we used. It didn't add value, even if it added
clarity at times. We also had functions at the end of paths, but that goes
beyond the JSON focus of the JSON Pointer goals, so those points are not
relevant here.

This discussion thread seems to be getting overly complicated, but JSON
Pointer changes should come from the JSON Pointer view point and that
specifications goals, not from short comings in JSON Patch.

-- 
Matthew P. C. Morley

---

From: Robert Sayre <[email-masked]>
Date: Mon, 7 Jan 2013 22:33:02 -0800

On Mon, Jan 7, 2013 at 8:40 PM, Murray S. Kucherawy <[email-masked]> wrote:
>
> we have the "test" operations, so people
> applying any of the operations that alter the target.

I don't see how can the "test" operation be used to determine the
intended target structure of an operation with a JSON Pointer of "/1".
Maybe there should be an example in the draft.

- Rob

---

From: Robert Sayre <[email-masked]>
Date: Mon, 7 Jan 2013 23:57:55 -0800

On Mon, Jan 7, 2013 at 8:40 PM, Murray S. Kucherawy <[email-masked]> wrote:
> it is not my view after reviewing this
> thread that there's rough consensus to reopen the question.

I agree with this assessment.

- Rob

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 8 Jan 2013 09:19:05 +0100

For those who wonder what all the fuzz in this thread is about, let me try to explain this with some different terminology, exposing what kind of intuition these widely diverging value judgements might derive from.

In programming, there are two camps: static typers and dynamic typers.  Static typers want to annotate their programs with information about the objects being manipulated so the compiler can help them catch careless mistakes even before the program runs.  Dynamic typers don't care about that as much, because they know they have to write tests anyway (not all programming errors are exposable as type errors) and these will uncover the same careless mistakes.  [The runtime system will actually check types once the specific context of execution is available, that's why it's called "dynamic typing".]

The programming debate is beside the point here for a number of reasons*), but it does shape the thinking of programmers, and it seems in particular it shaped the thinking of some of the commenters here.

/a/1/b/2 is something that a dynamic typer would come up with.  The specific semantics are well-defined, but only in the specific execution context (what JSON document this is being applied to).
/a:1/b:2 (to use James' strawman syntax that distinguishes JSON object keys from array indices) exposes more typing info, so it is more statically typed, and it will catch mismatches between the types that the JSON pointer assumed to be present and those that actually are.

For a static-typing fundamentalist, it is viscerally unacceptable to forego the opportunity of catching this kind of "type error".

For a non-fundamentalist, this is more of an issue of probabilities, and the interlocking of mechanisms.  
Which percentage of usage errors will be caught by this?
Many, many mismatches between JSON pointers and JSON instances to will just not happen to trigger this particular type error.
So it can be argued that the ability to catch it doesn't really help much: If you care about mismatch errors, you already need to have something else in place to catch them -- relying only on the likelihood of a mismatch to trigger an array/object mismatch would be imprudent. So in reality, the ability to catch the type error buys you nothing.
(Or, actually, very little, as improved diagnostics is always worth something in a debugging situation.)

TL;DR: there is no "ambiguity" at all.  Please stop referring to an "ambiguity".  There is none.
Just a missed opportunity to catch an error, caused by not sending along (redundant) type information.

(You will gather that my 2 cents for this change are "not worth it", but I was more interested in explaining the mere existence of this discussion, first.  Applying the wrong intuitions to an engineering decision is one of the major causes of suboptimal design...)

Grüße, Carsten

*) Well, for a start, we are not here to catch errors in programming.  Programming languages are difficult because they are the human interface to the computer's programmability.  JSON pointers are a pure machine-to-machine mechanism, so most of the issues in programming just don't arise.  Porting your intuitions from programming to this is not going to work very well.

---

From: Jared Rosoff <[email-masked]>
Date: Tue, 8 Jan 2013 00:26:34 -0800

hi team, i'm new to the discussion here, but wanted to jump in. i work on
mongodb, a json database, and i wanted to share how we deal with these
issues.

mongodb uses almost the same notation for pointers ("a.b.c" instead of
"/a/b/c"). We also index arrays in the same way as json pointer ( "a.0"
refers to the 0th element of the array called "a"). and this works fine in
practice. (ref: http://docs.mongodb.org/manual/core/document/#dot-notation)

our update syntax is different tho. the verbs in mongodb updates for json
documents are more specific:

set / unset / rename (operations on fields)
inc (increment integer values)
push / pop / pull (operations on arrays)
addToSet / removeFromSet (operations on arrays)

(ref http://docs.mongodb.org/manual/reference/operators/#update)

since update operations are more specific and type dependent, it's easy to
throw an error if an unexpected type is encountered (e.g. try to push onto
a field that has a non-array value) and to act smartly on empty fields ( if
path to push is empty, we assume it should be an array, create it, and then
push the value onto it).

i concur the the pointer syntax is fine and ambiguity comes from the
definition of operators in json patch.

-j

---

From: "Markus Lanthaler" <[email-masked]>
Date: Tue, 8 Jan 2013 11:13:27 +0100

On Monday, January 07, 2013 11:39 PM, Conal Tuohy wrote:

> > For me the deficiency is not in the pointer, but patch
> > format being generated.
> >
> > One approach is to push that *one* test, structure conformity,
> >  into the pointer syntax. Another is via the type operation.
> >
> It seems to me, on the contrary, that the deficiency is in the
> pointer syntax, and I think it would be a mistake to try to work
> around that deficiency in JSON Patch. Because aren't there other
> things which one might do with JSON Pointer than use it with JSON
> Patch? There's been mention of having it registered as a URI
> fragment identifier syntax for JSON for example. JSON Pointers
> could then end up all over the place, outside of patches. IMHO
> JSON Pointer needs to be taken seriously as a technology in its
> own right.

I would like to second that. Since JSON Pointer and JSON Patch will be two
independent standards I expect (hope) that JSON Pointer will be used for
many other things than patching and that's exactly the reason why I raised
this in the first place.

I still believe that the current ambiguity might hinder many valuable use
cases in the future. I do understand that we are already quite late in the
process but since the fix is rather trivial I don't see a compelling reason
to not resolve this now.

Cheers,
Markus

--
Markus Lanthaler
@markuslanthaler

---

From: Robert Sayre <[email-masked]>
Date: Tue, 8 Jan 2013 08:19:07 -0800

Hi Jared,

Earlier in the thread, I actually directly asked whether software that
operates on arbitrary JSON was in scope for this WG (my example was
CouchDB), after having suggested either changing the path syntax or
renaming the array operations.[0]

The editors didn't respond, except with process points or simple
contradictions without rationale. My conclusion is that software such
as MongoDB must be out of scope here. To their credit, the editors did
point out that anyone is free to try again, and I plan to do just
that. There's no reason to hold up this WG, since they seem to be
burnt out.

- Rob

[0] http://www.ietf.org/mail-archive/web/apps-discuss/current/msg08552.html

On Tue, Jan 8, 2013 at 12:26 AM, Jared Rosoff <[email-masked]> wrote:
> throw an error if an unexpected type is encountered (e.g. try to push onto a
> field that has a non-array value) and to act smartly on empty fields ( if
>> and an identity string (both ends of the path concept). I see value from the
>> identity view, in describing a location that is aware of being inside an
>> array.
>>
>> Patch, especially when JSON Patch is attempting to address those issues with
>> other mechanisms within the specification. That is all I was trying to
>> express. The syntax change should be for other reasons, if it is going to be
>> made.
>>

---

From: James M Snell <[email-masked]>
Date: Tue, 8 Jan 2013 09:06:50 -0800

Fortunately the JSON-Patch syntax is designed such that it is possible to
extend the range of defined operations without breaking basic structure...
much as I have done with the json-predicates draft. You would need to
define a new media type to associate with the extensions but that's not
difficult.

Consider something along the lines of.. application/json-patch-ext...

[
  {"op": "unset", "path": "/a/b"},
  {"op": "inc", "path": "/a/c"},
  {"op": "push", "path": "/a/d", "value": 1}
]

- James

---

From: Jared Rosoff <[email-masked]>
Date: Tue, 8 Jan 2013 09:52:46 -0800

I'm definitely not trying to expand the scope beyond the current draft.
More trying to share some experience with an existing implementation of
something similar.

A few comments on path syntax based on this experience.

1) It seems like some of the examples raised on this thread conflate JSON
documents with Javascript objects. According to the json spec, an object
always starts with an "{" and ends with a "}". Therefore, while the json
path "/1" could refer to elements in { "1": "a" } and [1,2], the latter is
not a correct JSON object.

2) Should a JSON pointer be able to refer to the "whole document"? In the
spec, it says that the path "" would refer to the whole JSON document. I
don't think this should be possible. If you want to replace the entire
document, you should be using an HTTP PUT to replace the content, rather
than a patch. Eliminating the ability to refer to the whole document would
also release some ambiguity in certain cases.

For example, earlier it was pointed out that if i have this document

{ "a": 1 }

And then apply this patch

{ op: "add", path: "", value: [1,2,3] }

then it would result in the document after patch being

[1,2,3]

however, this is not a valid json document as it is not enclosed by { }.

Instead, I think that the empty path should refer to the element of the
document named with an empty string.

So with { a: 1 }, and the op { op: "add", path: "", value: [1,2,3] }
would result in this document:

{ a: 1, "": [1,2,3] }

This also means that the leading slash is somewhat superfluous and could be
omitted from the syntax (since the paths "", and "/" would both resolve to
the "" named element of the document. This makes parsing paths easier as
well, since you can just split the path on the "/" and take the count of
parts to figure out how deep you're going.

Given { a: 1, "": [1,2,3] }, i would refer to the 2nd array element with
the path "/2". When parsing this i'd split on "/", which gives me two
tokens, the empty string (on the left of hte /) and the number 2 (on the
right of the /).

-j

On Tue, Jan 8, 2013 at 8:19 AM, Robert Sayre <[email-masked]> wrote:

> [0]
> http://www.ietf.org/mail-archive/web/apps-discuss/current/msg08552.html
>
>
> > practice. (ref:
> http://docs.mongodb.org/manual/core/document/#dot-notation)
> >
> might
> with

---

From: James M Snell <[email-masked]>
Date: Tue, 8 Jan 2013 09:58:24 -0800

> however, this is not a valid json document as it is not enclosed by { }.

The root of a JSON Document can be either an object or an array.

- James

---

From: Jared Rosoff <[email-masked]>
Date: Tue, 8 Jan 2013 10:13:59 -0800

my bad.

regardless, if you want to replace the whole document, is that a patch? or
should that better be done with PUT?

---

From: James M Snell <[email-masked]>
Date: Tue, 8 Jan 2013 10:14:59 -0800

+1

---

From: Robert Sayre <[email-masked]>
Date: Tue, 8 Jan 2013 10:57:00 -0800

>From a JS shell session:

% JSON.stringify([1,2])
"[1,2]"

- Rob

---

From: Barry Leiba <[email-masked]>
Date: Tue, 8 Jan 2013 16:41:06 -0500

Let me suggest that y'all remove the IESG and the main IETF discussion list
(and me) from the CC list if you continue this discussion (which you're
welcome to do).  Actually, I suggest that if you do continue it, you start
a new thread with a new subject, on apps-discuss only.

Thanks.

Barry

---

From: Robert Sayre <[email-masked]>
Date: Wed, 9 Jan 2013 00:17:24 -0800

Awesome news: JSON Patch has enough IESG votes to become a standards
track document, even though JSON Pointer still has a discuss. This way
of doing things is very efficient--lots of documents can be approved
this way.

- Rob

On Tue, Jan 8, 2013 at 1:41 PM, Barry Leiba <[email-masked]> wrote:
> welcome to do).  Actually, I suggest that if you do continue it, you start a
> new thread with a new subject, on apps-discuss only.
>
> Thanks.
>
> Barry
>

---

From: Hector Santos <[email-masked]>
Date: Tue, 08 Jan 2013 18:59:09 -0500

Thanks Carsten for your explanations.  As having experience with both 
styles of programming as you describe and also interpretive vs p-code 
vs static compiler writing for our servers and clients, it would seem 
to me that if the both syntaxes are possible, then the solution is 
more implementator specific. In other words, implementators can deal 
with both if robustness is desired.  It sounds "kludgy" but it sounds 
like there exist both styles in practice already, so unless breaking 
existing implementators is not a concern in the name of going with the 
"correct and prefer" syntax, then both need to be supported.

The first thing that came to mind when this interesting thread started 
is whether a solution is desired for a public open-ended query 
client/server system versus this being more proprietary (presuming the 
idea of "patching" is an high authentication required concept).  It 
would seem to me that they might be various clients used against a 
server and it would be up to the server to support all possibilities 
to remain robust.   The client itself is probably having its "JIT" 
(Just-In-Time) parsing/compiling/error trapping, etc or it might be 
more static to catch canned solutions.  The client can perhaps have 
all its data I/O resolved by the "ide" new JSON support for data C/S 
(including "AJAX") transactions.  Come to think to it, the server 
implementation itself COULD have its syntax checking before release 
the page for production and/or if dynamically generated then it would 
probably use a preferred syntax.

It seems the answer is to support both syntaxes IFF there is compiler 
or interpretation logic capability to trapping both and if there are 
already existing clients and servers, note (highlight) that servers 
|need to|SHOULD| be ready for both.

I will say one thing, if we ever needed a HTTP based patching need or 
requirement to add support to our product web server component, it 
doesn't seem I have any choice - it would be prudent to support both 
especially on the client side.  What method I would use for our own 
data exchanges may be preferred method, but the server would need to 
do both.

--
HLS

Carsten Bormann wrote:
> Gr��e, Carsten

-- 
HLS

---

From: Barry Leiba <[email-masked]>
Date: Wed, 9 Jan 2013 13:26:09 -0500

> of doing things is very efficient--lots of documents can be approved

Sarcasm aside, you raise a good question, so let me explain:
We are, or at least we mostly try to be, more than just automata
blindly following process.  And so:

1. If there are documents that go together as a group, the responsible
AD (me, in this case) will usually not give the final approval for any
before they are all ready.  We did this recently with four EAI
documents (there were two with DISCUSS positions, and we held all four
back until everything was resolved), and we'll doing it again with the
six httpbis documents (I can't imagine giving final approval to, say,
four parts while two others sit in continued discussion).  We have a
status for this: "Approved, Point Raised".

2. Even if there are no DISCUSS positions held (as is now the case,
since Adrian cleared his this morning) we can and often do (and it's
certain that I will tomorrow) use the "Approved, Point Raised" status
to hold off the final approval until the responsible AD is satisfied
that everything's ready, to the extent that "everything" is ever ready
here.

3. And even if I should be inattentive or misguided enough to give the
final approval for Patch while still holding Pointer back for
discussion, because Patch has a normative reference to Pointer it will
not be published, and will wait in the RFC Editor queue in a "missing
reference" hold.

So just because a document meets the formal process criteria for
approval, we do use our judgment and try to do the right thing.
People will disagree about whether, in the end, the right thing has
been done, but nothing is automatic.

Barry

---

From: Robert Sayre <[email-masked]>
Date: Wed, 9 Jan 2013 10:32:03 -0800

> to hold off the final approval until the responsible AD is satisfied

Since you say it's certain, what is the reasoning? The WG seems
satisfied with their work.

- Rob

---

From: Barry Leiba <[email-masked]>
Date: Wed, 9 Jan 2013 13:54:08 -0500

> satisfied with their work.

I want to double-check a few things, first with the chairs and then
perhaps with the WG.  You'll probably see mail about it next week.

Barry