Subject: [Simple] XCAP issue: batching

From: Jonathan Rosenberg <[email-masked]>
Date: Tue, 03 Jun 2003 01:01:34 -0400

Folks,

The main open issue in xcap at this time is batching. We discussed it 
during the interim, and I have done some additional investigating 
since then.

The issue is this. In many cases, a client will want to modify a 
document from X to Y. This modification may require a series of 
changes to the document. However, the intermediate versions of the 
document which result from those changes do not represent a state that 
the client finds acceptable. This could be because they are not 
compliant to the schema, or because they are wrong or inappropriate. 
An example is moving a buddy from a white list to a black list. That 
would require two xcap operations - one to remove from the white list, 
one to add to the black list. After completing the first step, the 
buddy is on NEITHER list, and if a subscription should arrive at 
exactly that moment, the wrong thing would happen. This is basically 
the consistency property of ACID.

There are several ways this feature can be provided:

Soln 1: Least Common Parent

The "least common parent" is the XML element whose descendants include 
all of the nodes to be modified in the transaction, and for whom no 
descendant also has this property. So, to provide the transaction, the 
client would read the value of the least common parent, modify all of 
the elements as needed, and write it back to the server.

The main drawback of this approach is overhead. The client may be 
forced to read and write a lot of extra elements which are not 
actually modified, but are within the subtree of the least common 
parent. For buddy lists, this might mean reading the entire list, 
modifying some of the entries, and writing it back. Not pretty.

This approach works with xcap as written now.

Approach II: Body Commands

In this approach, we introduce a new operation that can be done on a 
document. The body of the request (POST) would contain XML that 
contains the sequence of operations to perform. So:

POST 
http://xcap.example.com/services/presence-lists/users/joe/list1.xml?batch

<batch>
   <delete>/presence-lists/list[@name=friends]/entry[@name=bill]</delete>
<append to="/presence-lists/list[@name=friends]/entry[@name=bob]">
  <entry name="jack" uri="sip:[email-masked]/>
</append>

</batch>

this would first delete bill, and then add jack, in one atomic 
operation. Its kind of soapish. A variation is to actually use soap.

Approach III: WebDav

During the interim, we talked about using webdav. The idea there was 
to first have the client LOCK the document, then perform a series of 
changes (PUT, POST, DELETE), and then UNLOCK. AFter the LOCK, all of 
the modification requests could be pipelined. The overhead here is 
probably pretty small, since HTTP has few mandatory headers.

I looked into webdav, and ran into the following issues:

1. Webdav doesnt specify a read lock. We would need to standardize 
that in order to use it.

2. The document would need to support a bunch of other webdav 
operations, such as PROPFIND. You cant JUST do locking. You dont need 
all of webdav, though.

3. There is no rollback or versioning. So, if there is an intermediate 
failure, there is no easy way to go back. We could fix that by 
introducing versioning into webdav.

Seems like a long road.

Option IV: COPY/MOVE

We can enable batched operations by defining a piece of the URI 
namespace thats a "scratchpad". Documents in this area can never be 
read directly by anyone but the document owner. To make a bunch of 
changes, the client would COPY (using webdav copy) the document into 
the area, using a random filename as the target. It then makes the 
changes using a series of PUT/POST/DELETE, with pipelining. When done, 
it performs a Webdav MOVE to place it back into the right place. We 
can specify some rules for the server on only allowing a move of a 
dcument back to the place where it was copied from.

I *think* we can support MOVE and COPY (and the additional webdav 
headers) without all of the rest of webdav, but I would need to 
confirm that. I didnt find any words that said MOVE or COPY can be 
applied to a non-webdav resource.

If any intermediate operation should fail, the client just DELETEs the 
document from the scratch area, and the original remains as it was. We 
would also probably want to have documents in the scratch area expire 
automatically after some time, so as to cleanup from incomplete 
transactions.

Assuming it is possible to MOVE or COPY non-webdav resources, my 
current preference is for option IV.

Comments, questions, or other proposals?

Thanks,
Jonathan R.
-- 
Jonathan D. Rosenberg, Ph.D.                600 Lanidex Plaza
Chief Scientist                             Parsippany, NJ 07054-2711
dynamicsoft
[email-masked]                     FAX:   (973) 952-5050
http://www.jdrosen.net                      PHONE: (973) 952-5000
http://www.dynamicsoft.com

_______________________________________________
Simple mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/simple

---

From: [email-masked]
Date: Tue, 3 Jun 2003 17:14:51 +0300

I don't yet have a preference, I have to think about the options a little more. I do have a question about option II though:

Does going with option II potentially mean eliminating the need of the DELETE method usage? and perhaps PUT as well? So we only have POST and GET.

Regards,
Hisham

> -----Original Message-----
> From: ext Jonathan Rosenberg [mailto:[email-masked]]
> Sent: Tuesday, June 03, 2003 8:02 AM
> To: [email-masked]
> Subject: [Simple] XCAP issue: batching
> 
> 
> Folks,
>

---

From: [email-masked]
Date: Tue, 3 Jun 2003 18:54:33 +0300

I meant something like this:

POST http://xcap.example.com/services/presence-lists/users/joe/list 1.xml

/Hisham

> Sent: Tuesday, June 03, 2003 6:47 PM
> To: Khartabil Hisham (NMP/Helsinki)
> Cc: [email-masked]
> Subject: Re: [Simple] XCAP issue: batching
> 
> 
> 
> 
> [email-masked] wrote:
> > I don't yet have a preference, I have to think about the options a
> > little more. I do have a question about option II though:
> > 
> > Does going with option II potentially mean eliminating the need of
> > the DELETE method usage? and perhaps PUT as well? So we only have
> > POST and GET.
> 
> I suppose so, but at the cost of a lot of overhead. Basically, in the 
> case of buddy lists, to delete a buddy, you'd have to re-POST the 
> entire list, just without that buddy. Not what we want to do, I think.
> 
> Also, that would not work for deleting documents, where an explicit 
> DELETE is still needed.
> 
> -Jonathan R.
> 
> --

---

From: [email-masked]
Date: Sun, 8 Jun 2003 23:36:39 +0300

Hi,

My prefence would be to go for option II, if we could do also conference control by making XCAP a bit more general. That would still be much lighter to implement than SOAP. 

The problem with option IV is that it seems to require extensions on HTTP level. That might make it difficult to implement on top of a common HTTP stack, which is an issue in certain environments. Otherwise option IV would be the best one. Have to see how big job incorporating the new methods/headers actually is.

Markus  

> Sent: 03 June, 2003 08:02
> To: [email-masked]

---

From: [email-masked]
Date: Sun, 8 Jun 2003 23:36:39 +0300

Hi,

Markus

---

From: Adam Roach <[email-masked]>
Date: Mon, 9 Jun 2003 11:26:28 -0500

Jonathan Rosenberg [mailto:[email-masked]] writes:

> I looked into webdav, and ran into the following issues: [...]

What I liked most about this proposal wasn't that it was based on
WEBDAV (although that was a nice bonus) as much as the model of
locking the record, performing operations on it as appropriate
(with connection-reuse and pipelining if desired), and then
committing the record. The strongest appeal lies in the fact that
the operations for adding a record to a document are identical
for locked and non-locked writes.

Since we're already having to define a mechanism for different
operations associated with a POST, I would contend that the same
effect could be achieved rather trivially by adding operations
"lock", "commit", and "rollback". For example, an atomic
add/delete/replace would be the following operations (with the
middle three being pipelined):

POST
http://xcap.example.com/services/presence-lists/users/joe/list1.xml?lock=/pr
esence-lists/list[@name=friends]

POST
http://xcap.example.com/services/presence-lists/users/joe/list1.xml?append=/
presence-lists/list[@name=friends]/entry[@name=bob]

DELETE
http://xcap.example.com/services/presence-lists/users/joe/list1.xml?/presenc
e-lists/list[@name=friends]/entry[@name=dave]

POST
http://xcap.example.com/services/presence-lists/users/joe/list1.xml?replace=
/presence-lists/list[@name=friends]/entry[@name=bill]

POST
http://xcap.example.com/services/presence-lists/users/joe/list1.xml?commit=/
presence-lists/list[@name=friends]

(Or, if something goes wrong)

POST
http://xcap.example.com/services/presence-lists/users/joe/list1.xml?rollback
=/presence-lists/list[@name=friends]/

And, of course, clients wouldn't be *required* to use the lock mechanism
at all; it's only necessary when multiple updates need to be atomic.

You raise the issue of read-locking; just to be sure I understand: do you
see the need to lock a record, perform multiple reading operations on it
that must yield a consistent view of the data, and then unlock it? I'm
having a hard time constructing use cases for this ability. If we need
it, though, we could define additional operations of "rlock" and
"runlock" (and possibly "upgrade" and "downgrade").
 
/a

---

From: Adam Roach <[email-masked]>
Date: Mon, 9 Jun 2003 11:26:28 -0500

POST

POST

DELETE

POST

POST

POST

 
/a