Subject: Re: [apps-discuss] JSON Patch

From: Michael Dürig <[email-masked]>
Date: Wed, 2 Nov 2011 13:39:58 +0000

Hi,

This looks very interesting. There seem to be a lot of similarities to 
an earlier proposal [1, 2]. What is missing (wrt. to [2]) is a reorder 
operation.

Furthermore I think the notion of the JSON path used in the operations 
should be defined somewhere.

Michael

[1] http://lists.w3.org/Archives/Public/w3c-dist-auth/2010JulSep/0008.html
[2] http://www.slideshare.net/uncled/jsop

---

From: Julian Reschke <[email-masked]>
Date: Wed, 02 Nov 2011 14:57:02 +0100

On 2011-11-02 14:39, Michael Dürig wrote:
> ...

That's defined in 
<https://tools.ietf.org/html/draft-pbryan-zyp-json-pointer-02>...

As an XML person, seeing an XPath-like syntax make me happy. But 
wouldn't be "dot" notation be much simpler, given where JSO comes from?

Best regards, Julian

---

From: "Paul C. Bryan" <[email-masked]>
Date: Wed, 02 Nov 2011 10:22:44 -0700

Thanks everyone for the feedback so far. Some replies:

On Wed, 2011-11-02 at 13:39 +0000, Michael Dürig wrote: 

> What is missing (wrt. to [2]) is a reorder operation.

The ability to move items in an array has come up and seems
straightforward. A need (and semantics) of moving a value between two
arbitrary locations in a JSON document is not well understood. 

> wouldn't be "dot" notation be much simpler, given where JSO comes

A few of reasons we went with slashes:

1. Less confusing where array elements are involved.
2. Slashes are less likely to appear in object member names, hence less
percent-encoding.
3. Pointers are intended to be specified in URI fragment identifiers,
and would appear to some (unsophisticated humans or machines) like a
filename extension.  

Paul

---

From: Julian Reschke <[email-masked]>
Date: Thu, 10 Nov 2011 11:01:01 +0100

> arbitrary locations in a JSON document is not well understood.

+1. So are you planning to add this? Would it make sense to make a 
concrete proposal?

> 1. Less confusing where array elements are involved.

...I'll have to ask :-). What about changing the notation for array 
elements to use [ and ]? More complexity in the expression language, but 
also more readability, I believe...

> and would appear to some (unsophisticated humans or machines) like a

Thanks for the explanation.

Best regards, Julian

---

From: Martin Thomson <[email-masked]>
Date: Thu, 10 Nov 2011 21:45:06 +1100

On 10 November 2011 21:01, Julian Reschke <[email-masked]> wrote:
> ...I'll have to ask :-). What about changing the notation for array elements
> to use [ and ]? More complexity in the expression language, but also more
> readability, I believe...

That sounds like a good idea.  Use JSON string representation and you
can deal with any sort of identifier without much new code...
["foo"][3]["bar"]  The approach worked for JSON itself, why not here
too?

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 10 Nov 2011 17:05:51 -0600

On 10/11/2011, at 4:01 AM, Julian Reschke wrote:

> ...I'll have to ask :-). What about changing the notation for array elements to use [ and ]? More complexity in the expression language, but also more readability, I believe...

+1 -- I gave similar feedback.

--
Mark Nottingham
http://www.mnot.net/

---

From: Julian Reschke <[email-masked]>
Date: Tue, 22 Nov 2011 17:05:23 +0100

> +1. So are you planning to add this? Would it make sense to make a

OK, here's draft proposal:

4.4. move

    The "move" operation moves an existing array element. The "to" 
member indicates the array position as integer value. This operation is 
equivalent to removing the element identified by "move", an inserting it 
again at the position "to".

Example:

    An example target JSON document:

    {
        "foo": [ "bar", "qux", "baz" ]
    }

    A JSON Patch document:

    [
        { "move": "/foo/1", "to": 2}
    ]

    The resulting JSON document:

    {
        "foo": ["qux", "bar", "baz"]
    }

Q: is a special case like "end" needed?

Best regards, Julian

---

From: "Paul C. Bryan" <[email-masked]>
Date: Tue, 22 Nov 2011 10:24:57 -0800

Looks good to me. I don't think "end" is needed, as the end index can be
explicitly specified.

Paul

---

From: Julian Reschke <[email-masked]>
Date: Tue, 22 Nov 2011 19:27:29 +0100

> Looks good to me. I don't think "end" is needed, as the end index can be

It might make it easier to move something to the end when you don't know 
the whole array (for instance, because there may be race conditions)... 
Not sure how important that is for this use case, though...

Best regards, Julian

---

From: Martin Thomson <[email-masked]>
Date: Wed, 23 Nov 2011 10:09:06 +1100

>> Looks good to me. I don't think "end" is needed, as the end index can be

I don't see there being much value either.

> It might make it easier to move something to the end when you don't know the
> whole array (for instance, because there may be race conditions)... Not sure
> how important that is for this use case, though...

I don't like the idea that you could be designing for race conditions
like that. Use conditional requests (thinking HTTP) or some form of
locking if it really must be at the end.

--Martin

---

From: "Manger, James H" <[email-masked]>
Date: Wed, 23 Nov 2011 12:21:17 +1100

JSON pointer array indices start at zero so the example patch document below actually moves "qux" to the end of the array, instead of "bar" to the middle.

On escaping: how about replacing every '/' in an object member's name with the Unicode REPLACEMENT CHARACTER U+FFFD when creating a JSON pointer.

It means a JSON pointer cannot reference an object member that actually has a U+FFFD char in its name, but that seems so unlikely that it could be an acceptable restriction.
Escaping and unescaping is easy: eg in Java s.replace('/', '\uFFFD') to escape; s.replace('\uFFFD', '/') to unescape.
It should cause less confusion when another layer (eg URI, fragment...) adds escaping, such as %xx.

--
James Manger

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Julian Reschke
Sent: Wednesday, 23 November 2011 3:05 AM
To: Paul C. Bryan
Cc: [email-masked]
Subject: Re: [apps-discuss] JSON Patch

4.4. move

Example:

    {

    A JSON Patch document:

    [

    {

Best regards, Julian
_______________________________________________
apps-discuss mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/apps-discuss

---

From: Graham Klyne <[email-masked]>
Date: Tue, 22 Nov 2011 20:10:00 +0000

On 22/11/2011 16:05, Julian Reschke wrote:
> The "move" operation moves an existing array element. The "to" member indicates
> the array position as integer value. This operation is equivalent to removing
> the element identified by "move", an inserting it again at the position "to".
>
>
> Example:
>

Er, aren't Javascript arrays 0-based?  I'd expect the same to apply to JSON.

I'd also suggest instead of
   "an inserting it again at the position "to"."
to say
   "and inserting it again so that it occupies position "to" in the resulting 
array."

This just avoids any ambiguity about whether or not "to" is specified with 
respect to the original array before removing the element to be moved.

#g

---

From: Julian Reschke <[email-masked]>
Date: Wed, 23 Nov 2011 13:38:20 +0100

> Looks good to me. I don't think "end" is needed, as the end index can be

Ack.

Also, we may want to say that to="" currently only allows an array 
index; and that anything else is reserved for future use (such as moving 
things around in the tree).

Best regards, Julian

---

From: "Paul C. Bryan" <[email-masked]>
Date: Wed, 23 Nov 2011 09:19:04 -0800

I'm now inclined to simply make "to" a JSON pointer as well.
Semantically it's clear, It's easy to apply, and JSON diff
implementations can be smart if they want and notice something moved
from one node in the graph to another.

Paul

---

From: Julian Reschke <[email-masked]>
Date: Wed, 23 Nov 2011 18:37:36 +0100

> from one node in the graph to another.

+1, except that I'm not convinced that leaving this optional is a good 
idea...

---

From: "Paul C. Bryan" <[email-masked]>
Date: Wed, 23 Nov 2011 10:02:22 -0800

> +1, except that I'm not convinced that leaving this optional is a good 

I'd say if JSON pointer is adopted for "to", it should only be JSON
pointer.

Paul

---

From: Julian Reschke <[email-masked]>
Date: Wed, 23 Nov 2011 19:06:31 +0100

On 2011-11-23 19:02, Paul C. Bryan wrote:
>> >  I'm now inclined to simply make"to"  a JSON pointer as well.
> I'd say if JSON pointer is adopted for "to", it should only be JSON pointer.
>
> Paul

Works for me.

(What I had in mind was to consider something not starting with / as 
relative pointer...)

Best regards, Julian

---

From: Martin Thomson <[email-masked]>
Date: Thu, 24 Nov 2011 13:03:06 +1100

On 23 November 2011 03:05, Julian Reschke <[email-masked]> wrote:
>   The "move" operation moves an existing array element. The "to" member
> indicates the array position as integer value. This operation is equivalent
> to removing the element identified by "move", an inserting it again at the
> position "to".
>   A JSON Patch document:
>
>   [

A thought occurred.  What about this?

[
  { "move" : "/foo/1", "to": "/foo/2" }
]

And allow for movement anywhere in the document.

---

From: Martin Thomson <[email-masked]>
Date: Thu, 24 Nov 2011 13:08:30 +1100

> On escaping: how about replacing every '/' in an object member's name with the Unicode REPLACEMENT CHARACTER U+FFFD when creating a JSON pointer.

Interesting that you choose U+FFFD in the same way that backslash was
chosen as an escape character in the first place.  I'm not a big fan
of that approach.  It also results in a nine character escape sequence
for a URI if you UTF-8 and %-encode.

> It means a JSON pointer cannot reference an object member that actually has a U+FFFD char in its name [...]

I might just start using that if you decide to prohibit it now.

--Martin

---

From: Tony Hansen <[email-masked]>
Date: Wed, 23 Nov 2011 21:19:14 -0500

> (What I had in mind was to consider something not starting with / as 

If you introduce relative pointers, you also need the concept of .. to 
move up the path, and have to define semantics for moving past the 
beginning of the path, and the concept of normalizing a path, and 
probably a dozen other things that are normally part of relative pathing.

In thinking about the example, I find "/foo/2" much clearer than just 
"2" as the target.

Given that Javascript uses 0-based arrays, should we also be using 
0-based paths here? Or has that horse already left the barn?

     Tony Hansen

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 24 Nov 2011 16:14:12 +0900

On 2011/11/24 11:08, Martin Thomson wrote:
> of that approach.

Yes, please don't. The semantics of U+FFFD is mostly a character that 
wasn't successfully converted from some other encoding. Overloading that 
with "escaping a slash" is a bad idea.

Regards,    Martin.

> It also results in a nine character escape sequence

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 24 Nov 2011 16:16:01 +0900

On 2011/11/24 11:03, Martin Thomson wrote:
> On 23 November 2011 03:05, Julian Reschke<[email-masked]>  wrote:

The syntax looks good. But what is /foo/2 in the "to" part exactly 
referring to? Is is array position 2 after /foo/1 has been removed? Or 
before?

Regards,    Martin.

---

From: "Paul C. Bryan" <[email-masked]>
Date: Wed, 23 Nov 2011 23:54:42 -0800

After.

---

From: Julian Reschke <[email-masked]>
Date: Thu, 24 Nov 2011 09:44:53 +0100

> referring to? Is is array position 2 after /foo/1 has been removed? Or

*After*. See the definition.

Best regards, Julian

---

From: "Manger, James H" <[email-masked]>
Date: Fri, 25 Nov 2011 12:58:37 +1100

> with "escaping a slash" is a bad idea.

JSON pointer theoretically needs a proper escaping mechanism since it reserves 1 char as a delimiter. However, choosing 1 char not to support avoids a fair amount of inevitable confusion and complexity. I chose U+FFFD REPLACEMENT CHARACTER. If you don’t like that choice, how about U+001F INFOMATION SEPARATOR ONE.

Not supporting pointers for names with, say, a U+FFFD char is not ideal but hinders almost no practical uses.
Escaping '/' with '\/' could be worse. It seems certain to cause significant confusion. You can no longer simply split a pointer on '/' characters, but need to use, say, a regular expression to parse a pointer. A / can be legitimately escaped in a JSON string as "\/", and apparently often is [http://stackoverflow.com/questions/1580647/json-why-are-forward-slashes-escaped] so developers will read and write 0, 1, 2, or 3 backslashes (but only 1 %2F), when they are dealing with a name with a '/', or a '\'. Will "\x" be undefined when x isn’t \ or /?
 
Object member name:
Logically:  Bjørn/Carsten and bar
In JSON  : "Bjørn/Carsten" and "bar"
      Or : "Bjørn\/Carsten" and "bar"

JSON pointer:
Logically:  Bjørn\/Carsten/bar
In JSON  : "Bjørn\\/Carsten/bar"
      Or : "Bjørn\\\/Carsten\/bar"
In IRI   :  Bjørn%2F/Carsten/bar

--
James Manger

---

From: Julian Reschke <[email-masked]>
Date: Fri, 25 Nov 2011 14:05:01 +0100

> JSON pointer theoretically needs a proper escaping mechanism since it reserves 1 char as a delimiter. However, choosing 1 char not to support avoids a fair amount of inevitable confusion and comple…

I like that; it avoids adding a whole new escaping sequence, and I think 
it's ok to sacrifice compatibility with identifiers that contain control 
characters.

> In IRI   :  Bjørn%2F/Carsten/bar

Not sure I get the example.

So, the member name is

   Bjørn/Carsten and bar

in JSON, that's

   "Bjørn/Carsten and bar"

no?

In JSON pointer with 0x1f-substution we get (serialized as JSON):

   "/Bjørn\u001fCarsten and bar"

In an IRI, we'd have:

   /Bjørn%1FCarsten%20and%20bar

and, in a URI:

   /Bj%C3%B8rn%1FCarsten%20and%20bar

Best regards, Julian

---

From: "Manger, James H" <[email-masked]>
Date: Mon, 28 Nov 2011 09:50:25 +1100

> I like that; it avoids adding a whole new escaping sequence, and I think it's ok to sacrifice compatibility with identifiers that contain control characters.

> Not sure I get the example.

Sorry for the confusing example, Julian.
I meant to show a JSON pointer with 2 references.
Eg a pointer to the value 3 in this JSON example:
{"Bjørn/Carsten":{"bar":3}}
I didn't get it quite right as I omitted the leading "/" that all pointers start with.

--
James Manger