Subject: [apps-discuss] Json Patch Extended Operations - *Experimental*

From: James M Snell <[email-masked]>
Date: Wed, 5 Sep 2012 10:03:34 -0700

I've recently been experimenting with the JSON Patch format in a variety of
scenarios in order to gain some additional implementation experience. I've
found that in a number of cases it can be useful to apply more additional
types of test operations. I am considering writing these up as a separate
I-D as an extension to Json-Patch but I'm not yet fully convinced that it's
something that should be done; so I wanted to solicit some feedback from
the group at large.

In my experiments, I have toyed around with the following additional test
operations:

"not"       -- aggregate test... all of the contained operations must
evaluate to false.. must not contain change operations.. tests only
"and"       -- aggregate test... all of the contained operations must
evaluate to true.. must not contain change operations.. tests only
"or"        -- aggregate test... any of the contained operations must
evaluate to true.. must not contain change operations.. tests only
"typeOf"    -- tests to ensure that the referenced object is the specified
type (using javascript typeof)
"contains"  -- does the toString value of the referenced object contain the
specified value
"startsWith"-- does the toString value of the referenced object start with
the specified value
"endsWith"  -- does the toString value of the referenced object end with
the specified value
"matches"   -- does the toString value of the referenced object match the
given regular expression
"lessThan"  -- if the value is numeric or a date, returns true if node
value is more than specified value
"moreThan"  -- if the value is numeric or a date, returns true if node
value is more than specified value

For example, consider the following hypothetical, extended json-patch

  [
    { "and": [
        { "not": [
            {"moreThan", "/a/bar", "value": 123},
            {"matches", "/a/baz", "/sample/ig"}
          ]
        },
        {"contains": "/a/foo", "value": "123"},
        {"typeOf": "/a/b/version", "value": "number"}
      ]
     },
    {"add": "/a/b/c", "value": "foo"},
    {"replace": "/a/b/version", value: 1}
  ]

In this example, the change operations ("add" and "increment") are only
applied if and only if:

  * The /a/b/version property exists and is a numeric value,
  * The /a/foo property contains the characters "123" in it's string
representation,
  * The /a/baz property does not match the regular expression /sample/ig,
and
  * The /a/bar property is numeric and is less than or equal to 123

Yes, this adds quite a bit more complexity and yes, it's not yet clear if
this is something of value. I'm not arguing that we SHOULD do this, but in
a variety of experiments such tests have demonstrated usefulness.

Also, I was asked recently whether JSON-Patch could be made to apply
changes over a set of properties within a document that match a given
criteria. For example, consider the following source JSON document:

  {
    "items": [
      { "actor": {"displayName": "James" },
        "verb": "post",
        "object": {"objectType": "note", "content": "test 1" }},
      { "actor": {"displayName": "Joe" },
        "verb": "post",
        "object": {"objectType": "note", "content": "test 2" }},
      { "actor": {"displayName": "Sally" },
        "verb": "share",
        "updated": "2012-12-12T12:12:12Z",
        "object": {"objectType": "book", "id": "urn:foo:1" }},
    ]
  }

Suppose I wish to add a "published" property to all items that currently do
not have one... I could accomplish this using a new extended "each"
operation...

  [
    { "each": "/items",
      "apply": [
        {"not": [{"test":"/updated"}]},
        {"add": "/updated", "value": "2012-12-12T12:12:12Z"}
      ]
    }
  ]

This evaluates as: For each member of /items, test to see if the member has
an updated property. If not, add it with the value "2012-12-12T12:12:12Z".
If /items is a primitive, the "each" operation would fail as an error
condition. If /items is an array, then each iterates over each member of
the array. If /items is an object, each iterates over each of that's
objects member properties.

Note, however, that this differs quite a bit from the existing semantics of
json-patch. For one, a negative test within the "apply" would not signal an
error condition that would halt the entire patch operation. The negative
test would simply mean that the modification for that particular element
would not be applied. All of the changes within.

Again, I'm not yet convinced this is a good idea but experimentation has
demonstrated that it's at least a workable approach. I'd very much like to
hear from others on whether or not this is something we should or should
not attempt to pursue.

Thoughts?

- James

---

From: "Paul C. Bryan" <[email-masked]>
Date: Wed, 05 Sep 2012 17:06:45 -0700

I agree that these additional constructs could come in handy under the
right circumstances. That said, I see no way to bound the scope, because
any addition could be deemed handy under the right circumstances.

I wrestled initially with the consequences of adding the test operation
when it was first suggested. My notable concern was that it could creep
into more than just a simple equality test, perhaps even be pushed to
qualify particular operations (as you're suggesting), and ultimately
lead toward the need to sport a Turing-complete query syntax! Okay,
that's going too far, but hopefully this illustrates the extent of my
concern: slippery slope, in for a penny in for a pound, etc.

The use case that drove the equality test for me was (quasi-) RESTful:
there's a clear need to test equality of the partial state of a JSON
document as a precondition of its modification. In the case of HTTP
PATCH in particular, this presumes that the patch document is not being
applied blindly. Your suggested extended operations definitely go beyond
that scope. 

Paul

On Wed, 2012-09-05 at 10:03 -0700, James M Snell wrote:

> I've recently been experimenting with the JSON Patch format in a
> variety of scenarios in order to gain some additional implementation
> experience. I've found that in a number of cases it can be useful to
> apply more additional types of test operations. I am considering
> writing these up as a separate I-D as an extension to Json-Patch but
> I'm not yet fully convinced that it's something that should be done;
> so I wanted to solicit some feedback from the group at large.
> 
> 
> 
> In my experiments, I have toyed around with the following additional
> test operations:
> 
> 
> "typeOf"    -- tests to ensure that the referenced object is the
> specified type (using javascript typeof)
> "contains"  -- does the toString value of the referenced object
> contain the specified value
> "startsWith"-- does the toString value of the referenced object start
> with the specified value
> "endsWith"  -- does the toString value of the referenced object end
> with the specified value
> "matches"   -- does the toString value of the referenced object match
> the given regular expression
> In this example, the change operations ("add" and "increment") are
> only applied if and only if:
> 
> 
>   * The /a/baz property does not match the regular
> expression /sample/ig, and
> Yes, this adds quite a bit more complexity and yes, it's not yet clear
> if this is something of value. I'm not arguing that we SHOULD do this,
> but in a variety of experiments such tests have demonstrated
> usefulness.
> 
> 
> Suppose I wish to add a "published" property to all items that
> currently do not have one... I could accomplish this using a new
> extended "each" operation...
>  
>   [
>     { "each": "/items", 
>       "apply": [
> This evaluates as: For each member of /items, test to see if the
> member has an updated property. If not, add it with the value
> "2012-12-12T12:12:12Z". If /items is a primitive, the "each" operation
> would fail as an error condition. If /items is an array, then each
> iterates over each member of the array. If /items is an object, each
> iterates over each of that's objects member properties. 
> 
> 
> Note, however, that this differs quite a bit from the existing
> semantics of json-patch. For one, a negative test within the "apply"
> would not signal an error condition that would halt the entire patch
> operation. The negative test would simply mean that the modification
> for that particular element would not be applied. All of the changes
> within.
> 
> 
> Again, I'm not yet convinced this is a good idea but experimentation
> has demonstrated that it's at least a workable approach. I'd very much
> like to hear from others on whether or not this is something we should
> or should not attempt to pursue.
> 
> 
> Thoughts?
> 
> 
> - James
> 
> 
> 
> 
> 
> _______________________________________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss

---

From: James M Snell <[email-masked]>
Date: Wed, 5 Sep 2012 17:11:31 -0700

Indeed. It is definitely a slippery slope. At the very least, I think this
can demonstrate that *IF* (and it's a big if) an application needs
additional support for a broader range of test options, it can be
implemented in a way that is generally consistent with the existing
json-patch format -- and defined as an extension of json patch albeit with
a different media type. For now, then, I'll likely table this and chalk it
up as an interesting experiment until and unless there is more of a
demonstrated need.

On Wed, Sep 5, 2012 at 5:06 PM, Paul C. Bryan <[email-masked]> wrote:

> **
> right circumstances. That said, I see no way to bound the scope, because *
> any* addition could be deemed handy under the right circumstances.
>
> qualify particular operations (as you're suggesting), and ultimately lead
> toward the need to sport a Turing-complete query syntax! Okay, that's going
> too far, but hopefully this illustrates the extent of my concern: slippery
> slope, in for a penny in for a pound, etc.
>
> document as a precondition of its modification. In the case of HTTP PATCH
> in particular, this presumes that the patch document is not being applied
> blindly. Your suggested extended operations definitely go beyond that
> scope.
>
> Paul
>
>
> I've recently been experimenting with the JSON Patch format in a variety
> of scenarios in order to gain some additional implementation experience.
> I've found that in a number of cases it can be useful to apply more
> additional types of test operations. I am considering writing these up as a
> separate I-D as an extension to Json-Patch but I'm not yet fully convinced
> that it's something that should be done; so I wanted to solicit some
> feedback from the group at large.
>
>
>
>  "contains"  -- does the toString value of the referenced object contain
> the specified value
>
> if this is something of value. I'm not arguing that we SHOULD do this, but
> in a variety of experiments such tests have demonstrated usefulness.
>
>
>
>  Suppose I wish to add a "published" property to all items that currently
> do not have one... I could accomplish this using a new extended "each"
> operation...
>
>
>
>    [
>
>      { "each": "/items",
>
>        "apply": [
>
>  This evaluates as: For each member of /items, test to see if the member
> has an updated property. If not, add it with the value
> would fail as an error condition. If /items is an array, then each iterates
> over each member of the array. If /items is an object, each iterates over
> each of that's objects member properties.
>
>
>
>  Note, however, that this differs quite a bit from the existing semantics
> of json-patch. For one, a negative test within the "apply" would not signal
> an error condition that would halt the entire patch operation. The negative
> apps-discuss mailing [email-masked]://www.ietf.org/mailman/listinfo/apps-discuss
>
>
>

---

From: Matthew Morley <[email-masked]>
Date: Wed, 5 Sep 2012 21:12:16 -0400

I think that ultimately you will see something like what you posted
appearing. But it will be more of a conditional wrapper applying json
patches, not something that should live within the json patch structure (or
specification).

JSON Schema to define matches, JSON Patch to describe the changes, perhaps
a conditional reasoning structure would help bind them all into a language
agnostic data reasoning set?

Joining the crowd late on the patch discussions, looking forward to reading
through json patch tonight as I have some questions I need formulate and
pose. I'm really partial to the dot notation, but slashes work. :)

I developed a system in the past that heavily depended on self contained
logic structures in json. It used a quirk rule engine to be very reactive
to selective changes inside the object tree. Seeing json patch I now am
wondering if I should have been generating that as the both the log and the
tool of the change.

The biggest value, I feel, in json patch is the ability to reference a deep
path from a known point via a string. This provides a method of reasoning
not just about the value at the end of a path, but to also use the path
itself as an identity token.

A system that uses "/a/b/c" to change a value, can also share the
path/identity to other processes that want to then react to that change,
possibly applying their own changes and triggering further changes.
Relationships can be described using the shortest path to related fields,
then the path strings appended until you can reason about huge sets of
changes from a root point well above the original path itself.

Not sure if others are doing work like this, it came about for me from the
need to represent complex reactive data relationships in a language
agnostic way, and in a very practical sense to implement uniform
browser/server interactions.

On Wed, Sep 5, 2012 at 8:11 PM, James M Snell <[email-masked]> wrote:

>> *any* addition could be deemed handy under the right circumstances.
>>
>>    * The /a/baz property does not match the regular expression
>> /sample/ig, and
>>
>> currently do not have one... I could accomplish this using a new extended
>> "each" operation...
>>
>>
>>
>>    [
>>
>>      { "each": "/items",
>>
>>        "apply": [
>>
>> semantics of json-patch. For one, a negative test within the "apply" would
>> not signal an error condition that would halt the entire patch operation.
>> The negative test would simply mean that the modification for that
>> particular element would not be applied. All of the changes within.
>>
>>
>>
>> has demonstrated that it's at least a workable approach. I'd very much like
>> to hear from others on whether or not this is something we should or should
>> not attempt to pursue.
>>
>>
>>
>>  Thoughts?
>>
>>
>>
>>  - James
>>
>>
>>
>>
>>

-- 
Matthew P. C. Morley

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 8 Sep 2012 14:12:38 +1000

Effectively, we don't have extensibility in JSON Patch right now -- and I think that's a Feature[tm].

However, nothing's stopping you from defining a superset of JSON Patch as another media type -- and I agree that there are cases where this'd be very interesting. 

Cheers,

On 06/09/2012, at 10:11 AM, James M Snell <[email-masked]> wrote:

> Indeed. It is definitely a slippery slope. At the very least, I think this can demonstrate that *IF* (and it's a big if) an application needs additional support for a broader range of test options, it can be implemented in a way that is generally consistent with the existing json-patch format -- and defined as an extension of json patch albeit with a different media type. For now, then, I'll likely table this and chalk it up as an interesting experiment until and unless there is more of a demonstrated need.
> 
> I agree that these additional constructs could come in handy under the right circumstances. That said, I see no way to bound the scope, because any addition could be deemed handy under the right circumstances.
> 
> I wrestled initially with the consequences of adding the test operation when it was first suggested. My notable concern was that it could creep into more than just a simple equality test, perhaps even be pushed to qualify particular operations (as you're suggesting), and ultimately lead toward the need to sport a Turing-complete query syntax! Okay, that's going too far, but hopefully this illustrates the extent of my concern: slippery slope, in for a penny in for a pound, etc.
> 
> The use case that drove the equality test for me was (quasi-) RESTful: there's a clear need to test equality of the partial state of a JSON document as a precondition of its modification. In the case of HTTP PATCH in particular, this presumes that the patch document is not being applied blindly. Your suggested extended operations definitely go beyond that scope. 
> 
> Paul
> 
> 
>> I've recently been experimenting with the JSON Patch format in a variety of scenarios in order to gain some additional implementation experience. I've found that in a number of cases it can be useful to apply more additional types of test operations. I am considering writing these up as a separate I-D as an extension to Json-Patch but I'm not yet fully convinced that it's something that should be done; so I wanted to solicit some feedback from the group at large.
>> 
>> 
>> In my experiments, I have toyed around with the following additional test operations:
>> 
>> 
>> "not"       -- aggregate test... all of the contained operations must evaluate to false.. must not contain change operations.. tests only
>> "and"       -- aggregate test... all of the contained operations must evaluate to true.. must not contain change operations.. tests only
>> "or"        -- aggregate test... any of the contained operations must evaluate to true.. must not contain change operations.. tests only
>> "typeOf"    -- tests to ensure that the referenced object is the specified type (using javascript typeof)
>> "contains"  -- does the toString value of the referenced object contain the specified value
>> "startsWith"-- does the toString value of the referenced object start with the specified value
>> "endsWith"  -- does the toString value of the referenced object end with the specified value
>> "matches"   -- does the toString value of the referenced object match the given regular expression
>> "lessThan"  -- if the value is numeric or a date, returns true if node value is more than specified value
>> "moreThan"  -- if the value is numeric or a date, returns true if node value is more than specified value
>> 
>> 
>> In this example, the change operations ("add" and "increment") are only applied if and only if:
>> 
>> 
>>   * The /a/foo property contains the characters "123" in it's string representation,
>>   * The /a/baz property does not match the regular expression /sample/ig, and
>> Yes, this adds quite a bit more complexity and yes, it's not yet clear if this is something of value. I'm not arguing that we SHOULD do this, but in a variety of experiments such tests have demonstrated usefulness.
>> 
>> 
>> Also, I was asked recently whether JSON-Patch could be made to apply changes over a set of properties within a document that match a given criteria. For example, consider the following source JSON document:
>> 
>> 
>>   {
>>     "items": [
>> Suppose I wish to add a "published" property to all items that currently do not have one... I could accomplish this using a new extended "each" operation...
>>  
>>   [
>>     { "each": "/items", 
>>       "apply": [
>> This evaluates as: For each member of /items, test to see if the member has an updated property. If not, add it with the value "2012-12-12T12:12:12Z". If /items is a primitive, the "each" operation would fail as an error condition. If /items is an array, then each iterates over each member of the array. If /items is an object, each iterates over each of that's objects member properties. 
>> 
>> 
>> Note, however, that this differs quite a bit from the existing semantics of json-patch. For one, a negative test within the "apply" would not signal an error condition that would halt the entire patch operation. The negative test would simply mean that the modification for that particular element would not be applied. All of the changes within.
>> 
>> 
>> Again, I'm not yet convinced this is a good idea but experimentation has demonstrated that it's at least a workable approach. I'd very much like to hear from others on whether or not this is something we should or should not attempt to pursue.
>> 
>> 
>> Thoughts?
>> 
>> 
>> - James
>> 
>> 
>> 
>> 

--
Mark Nottingham   http://www.mnot.net/