Subject: RE: Proposal: WebDAV and transactions

From: "Pill, Juergen" <[email-masked]>
Date: Mon, 9 Sep 2002 16:06:16 +0200

Hello Alan,

You put it to the point.

From the protocol point of view, there may be only minor needs to have TA
functions. A user issues a request, gets an answer and the server is clean
again. This is a (server wise) very lean approach.

From an API layer, there are many scenarios an application wants to block
some commands within a TA. Unfortunately an API can not simulate TA behavior
without help (TA support) from the server. This was the intention of the
mail. Batch operations would not help an API, if the API wants to perform
different operations depending on the succors response code or body.

Having TA spreading several servers is far more complicated (here a 2 phase
commit protocol would be needed), this was not intended by the current TA
approach. I think this is something, as you pointed out, for  "someday".

The TA approach would be an extension, some servers would support it (they
have to deal with TA state) some would not. If a client does not request
Tas, or a server does not support TA, the behavior and performance would be
identical to today's situation. If a client/API would require and request
TAs, the server has to deal with more (space and logic) overheads.

Best regards,

Juergen

 -----Original Message-----
From: 	Babich, Alan [mailto:[email-masked]] 
Sent:	Monday, September 02, 2002 22.59 PM
To:	'B. Shadgar'; Pill, Juergen; [email-masked]
Subject:	RE: Proposal: WebDAV and transactions

One thing you do NOT want under any circumstances are 
begin-transaction, end-transaction, and abort-transaction 
methods. That approach simply does not scale beyond a single
user, and is a performance disaster.

You absolutely need to stick to single method calls that can 
Perform Multiple operations on the server in a single 
Transaction. Batching up operations is a valid approach.

Transactions aren't that easy to deal with. That becomes
Even more true when multiple servers are involved. Someday,
WebDAV might get that far.

Transactions aren't new. They're decades old. The effort would
Profit from the decades of thought that has gone before.
For example, see the DMA standard for one approach. (But
remember: DMA specifies an API, not a protocol, and WebDAV is a 
protocol.)

Dr. Alan F. Babich

---

From: "Pill, Juergen" <[email-masked]>
Date: Wed, 11 Sep 2002 15:21:26 +0200

Hello,

I would like to suggest to open a new Thread for the "Batch method"
discussion or use the already started one. 

A batch method would solve the API problem situation only partly. If the API
could send all TA containing commands (without having to take decision based
on response codes or response body content), the batch method would work.
For a general purpose TA aware API I would not like to assume this.

The TA methods and the Batch methods could be seen as orthogonal features.
If the BATCH method is executed outside a TA_BEGIN, TA_COMMIT bracket, all
nested methods within the BATCH operation are executed in auto-commit mode
(each command is executed complete or not at all, some may executed with
success, some may return response error codes). If the BATCH method is
executed within a TA bracket, the BATCH method (and ALL nested contained
methods)  is due to the TA boundaries and all send methods (batch, including
the nested methods and the methods send parallel to BATCH) are either
committed or aborted.
If all methods within the batch method should be executed within a single
TA, following coding within the batch method would be required:
- open a TA (e.g. TA_BEGIN or MKTransaction)
- specify the all nested commands, e.g. PUT, DELETE, etc.
- close the TA (e.g. TA_END or Delete <ta_resource_uri>)

Both functionalities (BATCH and TA) have their distinct advantages and
computing power. A BATCH method would not solve all of the TA requirements.
TA methods would not completely solve the BATCH requirements.

Do you agree or doesn't this make sense at all?

Best regards,

Juergen

 -----Original Message-----
From: 	B. Shadgar [mailto:[email-masked]] 
Sent:	Wednesday, September 11, 2002 14.40 PM
To:	Julian Reschke; [email-masked]
Subject:	Re: Proposal: WebDAV and transactions

Julian Reschke wrote:

> > From: [email-masked]
> > [mailto:[email-masked]]On Behalf Of B. Shadgar
> > Sent: Wednesday, September 11, 2002 12:02 PM
> > To: Eric Sedlar; [email-masked]
> > Subject: Re: Proposal: WebDAV and transactions
> >
> > ...
> >
> > I think Batch method can help a lot in favor of transactions, But
> > I don't know
> > why still it has not been accepted as a standard method in WebDAV
> > protocol ?
>
> Well, make a proposal that works and let the discussion start.
>
> --
> <green/>bytes GmbH -- http://www.greenbytes.de -- tel:+492512807760

A proposal for Batch method is following:

Batch Method

The Batch method is introduce to processes a bunch of methods as one
single transaction and to improve the overload of internet connection.

The request message body of a BATCH method MUST contain one-transaction
and  request-set XML elements. It is included by one or more request XML
elements. processing of request elements MUST occur in the order request
elements are received (i.e., from top to bottom). Request elements can
either all be executed or some executed which will be specified by value
of one-transaction element. If the  value is true, all of request
elements MUST be executed, thus if any error occurs during processing
all executed  requests MUST be undone and resopnse MUST return the
result of failed request. Otherwise, when the one-transaction element's
value is false, some of requests can be executed and a proper response
with 207 statuse code  MUST be prepared to shows which requested has
been done and why the others refused.

The request XML element is included by header, body and
expected-status-code elements. Header and body element contains the
header and body of given request and expected-status-code is
representing the result is expected. It can be expressed by one value or
a range of values e.g  200 <= X < 300 by xml elements.

I consider Pill's example in his proposal and try to make my mean clear:

>> Request

BATCH  /bar/  HTTP/1.1
Host: www.foo.bar
Content-Type: text/xml; charset="utf-8"
Content-Length: xxxx

<?xml version="1.0" encoding="utf-8" ?>
<D:request-set xmlns:D="DAV:">
   <D:one-transaction>true</D:one-transaction>
   <D:request>
      <D:header> The header of PUTmethod  </D:header>
      <D:body> the Body of PUT method </D:body>
      <D:expected-status-code> <or>
                                                    <li>204</li>
                                                    <li>201</li>
                                                 </or>
      </D:expected-status-code>
   </D:request>
   <D:request>
      <D:header> The header of PROPPATCH method  </D:header>
      <D:body> the Body of PROPPATCH method </D:body>
      <D:expected-status-code> 200 </D:expected-status-code>
   </D:request>
   <D:request>
      <D:header> The header of UNLOCK method  </D:header>
      <D:body> the Body of UNLOCK method </D:body>
      <D:expected-status-code>204</D:expected-status-code>
   </D:request>
   <D:request>
      <D:header> The header of Delete method  </D:header>
      <D:body> the Body of Delete method </D:body>
      <D:expected-status-code> 204</D:expected-status-code>
   </D:request>
 </D:request-set>

>> Respons

HTTP/1.1  200 Ok

or

>> Response

HTTP/1.1 403 Forbidden
Content-Type: text/xml; charset="utf-8"
Content-Length: xxxx

    <?xml version="1.0" encoding="utf-8" ?>
    <D:response>
       <D:method>
               <D:status>HTTP/1.1 204 No Content </D:status>
       </D:method>
       <D:method>
               <D:status>HTTP/1.1 200 Ok</D:status>
       </D:method>
       <D:method>
                 <D:status>HTTP/1.1 204 No Content </D:status>
        </D:method>
       <D:method>
                  <D:href>http://www.foo.bar/bar/resource3>
                  <D:status>HTTP/1.1 403 Forbidden</D:status>
       </D:method>
     </D:response>

We can consider a bunch of delete methods when the one-transaction
element was false. In this case the format of the request and response
is the same, but the one-transaction element is set with false value,
and therefore failing some of Delete method dosn't cause to undone the
batch method. The response can inform the client which delete methods
are failed.

Does it make sence?

Regards,
Bita.

---

From: "Pill, Juergen" <[email-masked]>
Date: Wed, 11 Sep 2002 15:41:52 +0200

Hello Bita,

Although I think that Batch method can help regarding the transactions, but
it
is not enough. Because, it can only handle a transaction which made by a
sequence of WebDAV methods. However it is not the whole problem of
transactions.
Consider an application which is using WebDAV protocol as a part of it work
and
it needs to do some other functionality along with the WebDAV methods as
one
transaction. In this case my proposal of batch method doesn't help that
application. It would necessary need to TA-methods or change the server code
to
provide its own function. The latter one is an anathema.

Do I understand you right, that you think of a scenario, in which the
application wants to access/modify data from a (or multiple) WebDAV server
and from one (or many) TA aware different data sources (.e.g. an SQL
database) AND have that access/modification wrapped in a single TA.

In this case a two-phase commit protocol (and a Transaction Monitor) would
be required, assuming WebDAV supports TA (in the 2 phase commit flavor) and
all other associated DBs and datastores would support the 2PC protocol too.

We do not have this 2PC requirement, and I believe this could be an
extension to the proposed TA requirement. (at a minimum the TA_PREPARE
method needs to be supplied). Possible we could go two steps to reach this
2PC target.

Best regards,

Juergen

 -----Original Message-----
From: 	B. Shadgar [mailto:[email-masked]] 
Sent:	Wednesday, September 11, 2002 15.21 PM
To:	Roy T. Fielding; [email-masked]
Subject:	Re: Proposal: WebDAV and transactions

"Roy T. Fielding" wrote:

> > I think it is clear that the best way to address this requirement is
> > through
> > batching up WebDAV METHODS into a single request.  We also have
> > requirements
> > for doing batch methods for other reasons, and I know Microsoft has also
> > run
> > into this requirement and extended the standard this way.
>
> I don't think that is clear at all.  Batching methods does nothing to
> help with transactions that cannot be accomplished just as easily
> (and without breaking HTTP security) by including a transaction identifier
> in each request.  Changing the packaging of a sequence of interrelated
> requests does not change the fact that they are a sequence of interrelated
> requests.  Eventually, something has to process those requests in
sequence,
> as a transaction.  The server simply needs a way to accumulate requests
> and a method to commit them, and therefore it makes sense to have methods
> for begin-, abort-, and commit-transaction.

If you consider the proposal of batch method I have already sent to mailing
list, and if you consider that server would have the following schema of
handling a given method:

try {
     TA.begin();
     XXX Method();
    TA.commit();
} catch ( Exception e) {
    TA.rollback()
}

The whole of batch method would be considered as a one transaction without
having TA-methods like begin, abort and commit. For each request in the
Batch
method we just simply call the corresponding method which is not a separate
transaction.

>
>
> >> If the gun refuses to shoot you in the
> >> foot, that is a better design for the gun.
>
> In that case, a gun that is incapable of shooting at all would be
> a "better" design.  Considering requirements in isolation is a waste
> of energy.

I agree with this. Every body can make its choice to shoot or not.

>
>
> >> In the spirit of that analogy, I
> >> am proposing that the entire transaction, conditional processing and
all,
> >> should be invoked by one method call. That would do a lot to help
protect
> >> the performance of the system.
>
> It would do nothing to help protect the performance of the system.
> There is no effective difference between a sequence of requests with
> time-outs per request and a batch request with time-outs per read.
> The server still has to accumulate requests in storage and is
> subject to the same service problems with stateful behavior.
>
> The Web already has stored procedures.  There is nothing preventing an
> application developer from building complex transactions within a
> single method handler.  The problem is when you allow multiple methods
> with differing semantics to take place under the rubric of another
> method, which removes HTTP's ability to govern access rights per method.
>
> ....Roy

Although I think that Batch method can help regarding the transactions, but
it
is not enough. Because, it can only handle a transaction which made by a
sequence of WebDAV methods. However it is not the whole problem of
transactions.
Consider an application which is using WebDAV protocol as a part of it work
and
it needs to do some other functionality along with the WebDAV methods as
one
transaction. In this case my proposal of batch method doesn't help that
application. It would necessary need to TA-methods or change the server code
to
provide its own function. The latter one is an anathema.

If I am wrong please make me correct.

Regards
Bita.

---

From: "Pill, Juergen" <[email-masked]>
Date: Thu, 12 Sep 2002 11:19:13 +0200

Hello,

Yesterday I was made aware of the TIP protocol
(http://www.ietf.org/rfc/rfc2371.txt). It defines a 2PC TA protocol between
TMs (transaction managers) and requires the application to pass the TID
token to the resource manager (in our scenario the WebDAV server). [The
communication between the application and the resource manager (RM) is not
covered in the standard, except for the fact, that the TID token is passed).

I can see now 3 scenarios:

a) auto commit
The application is not TA aware and sends the WebDAV commands without any TA
commands or TA token. In this case the WebDAV server acts as today (every
request is executed in a single atomic TA). 
b) global TA
In this case multiple resource managers (e.g. multiple webdav server and/or
multiple other datasources) are involved on a single node or on multiple
nodes (with a single or multiple TM). In this case the application sends the
TA_BEGIN to the local TIP aware TM, receives a TID token, which is passed to
all participating RM at every request call. In case of WebDAV the place
could be a header (in case of e.g. PUT, the body is already reserved for
content). The WebDAV server (if the implementation is TIP aware) would
recognize the TID token and enlist at the local/global TM via the TIP
protocol. At the end of the TA the application would send the TA_Commit TIP
command to the local TM and the TM would initiate the 2PC execution to all
participating TM via the TIP protocol. 
c) local TA
In this scenario the application will talk to exactly one RM (the webdav
server) and possibly a full blown TM is not available. If the server would
implement both WebDAV and (at least the 1PC part of) the TIP protocol, the
TA_BEGIN could be send to the same server as all following WebDAV commands
followed by a TA_END command.

I can see following advantages going the TIP path:

The TIP protocol is already standardized, no need for WebDAV to introduced
something new (especially new methods). If WebDAV is used outside the TIP
protocol, performance and scalability stay identical to today's situation.
The decision if a WebDAV server is TIP aware will be driven by the specific
implementation. If a server supports scenario "c" only, would be
server-implementor defined too.
If we could agree on a specific header name to carry the TID token, all TIP
enabled WebDAV servers would be interoperable and could participate in a 2PC
environment. This header agreement would be the only WebDAV/HTTP extension
necessary.

The same would apply for http too. I agree with Roy, this feature is valid
in an http context too and specific to WebDAV. Would there be interest to
agree at this header for HTTP protocol?

Does this make sense?

Best regards,

Juergen

 -----Original Message-----
From: 	Roy T. Fielding [mailto:[email-masked]] 
Sent:	Wednesday, September 11, 2002 18.22 PM
To:	B. Shadgar
Cc:	[email-masked]
Subject:	Re: Proposal: WebDAV and transactions

I'll try to explain.  It is fundamental to HTTP that intermediaries
are able to inspect the method of an HTTP request in order to determine
the semantics of a request, and thereby its associated access control
and/or authentication requirements.  What you are suggesting is that
we bypass that feature in HTTP via a new syntax that tunnels arbitrary
methods within XML via HTTP, effectively bypassing all of the benefits
of using HTTP in the first place.

Sound familiar? It should, since this is the same objection that I have
frequently given to the use of SOAP for tunneling arbitrary semantics
though an HTTP POST method.

Transactions can be accomplished by using a request to ask the server
for a transaction context, possibly multi-level, and then sending
the sequence of requests with a calculated transaction-request-id
in the header fields, to each the server responds with 202 Accepted
or an error, and finalized with a commit or abort method, to which
the server responds with the appropriate message for the entire
commit.  Such a transaction mechanism accomplishes the same thing
without violating the requirements of HTTP.

In any case, this is OUT OF SCOPE for WebDAV.  It is an HTTP issue
that is not specific to authoring and will not be solved by an
authoring-specific solution.

....Roy

---

From: "Babich, Alan" <[email-masked]>
Date: Thu, 12 Sep 2002 12:11:18 -0700

-----Original Message-----
From: Roy T. Fielding [mailto:[email-masked]] 
Sent: Thursday, September 12, 2002 7:01 AM
To: Babich, Alan
Cc: [email-masked]
Subject: Re: Proposal: WebDAV and transactions

...snip...

<Alan>
> Between the method calls, the resources changed have to be locked on the
> server.
</Alan>

<Roy>
That's not how I described transactions.  I said that the requests are
collected on the server and only applied when the commit message is
received.  If the state has changed such that the methods are no longer
applicable, then the method transaction will fail on commit. 
</Roy>

That directly addresses the central issue, and I managed to miss that
somehow. >>My whole point is that when the server actually executes the
transaction, the server should have everything and do the entire transaction
all at once -- retrievals, updates, and conditional processing included.<<
Sorry if I didn't make that clear.

If you want to dribble the instructions for the transaction over piece by
piece by making separate method calls until it all gets to the server,
without actually doing any updates or retrievals, that's not something that
I am concerned with much. It would be more efficient to just send over all
the stuff at once, but I don't think it's that big of a deal to send it over
in pieces. So, doing that is OK with me.

If you want the server to do some security checking and other processing on
the individual pieces as they arrive, before actually executing the entire
transaction in the commit step, that's fine. That would help the client
pinpoint a certain subset of the possible problems in each step of the
transaction.

                                 ---

I think it would be interesting to see how the conditional processing of the
methods is specified. How much generality is achievable without getting into
syntax that doesn't resemble the current methods at all? To make arbitrary
transactions convenient, you want "program" variables, conditional
"statements", iteration "statements", and "subroutines" with parameters. If
we don't go that far, perhaps a useful set of transactions could still be
specified.

I also think it would be interesting to see how the results of the
transaction are specified. Any part of it could fail.

Alan

---

From: "Clemm, Geoff" <[email-masked]>
Date: Thu, 12 Sep 2002 18:27:52 -0400

DeltaV supports two transaction visibility models.

If you just have one client involved in a transaction, then you can use
the "working-resource" model.  The server assigns your client a new URL for
each
resource you are changing in a transaction, and those changes do not become
visible to other clients until you CHECKIN those changes.

If you have multiple clients involved in a transaction, then you can use
the "workspace" model.  A client allocates a workspace for the transaction,
and any other client that wants to see (or add to) the transaction can
accesses that workspace.  In this model, you commit the transaction by a
MERGE
request, at which point your changes are seen by all clients.

But in either case, a client by default only sees changes after they are
committed.  

Cheers,
Geoff

-----Original Message-----
From: Sridhar Erukulla [mailto:[email-masked]]
Sent: Thursday, September 12, 2002 5:27 PM
To: Clemm, Geoff; [email-masked]
Subject: RE: Proposal: WebDAV and transactions

Naive questions:

What if the old values can't be assigned to some/all of the filed
transaction?

What would an another client see if they query for these properties while
performing the alternative method for a transaction? the old or the new
values? Here the new values should be visible only after all the requests in
the batch are completed, is this something easy to implement?

Sridhar
-----Original Message-----
From: [email-masked]
[mailto:[email-masked]]On Behalf Of Clemm, Geoff
Sent: September 12, 2002 4:20 PM
To: [email-masked]
Subject: RE: Proposal: WebDAV and transactions

The Lock/Unlock doesn't give you the "rollback" functionality
(i.e. the client has to remember the old values, and live long
enough to restore them if the later updates fail).

But it would be interesting to investigate whether the
DeltaV CHECKOUT/CHECKIN/UNCHECKOUT protocol could give clients most
of the benefits of transactions (a non-versioning server would
only keep the most recently checked in "version" of a resource).
You would create an "activity" for a given transaction,
make your changes, and then CHECKIN the activity to commit
all the changes, or UNCHECKOUT to roll them back.

Cheers,
Geoff

-----Original Message-----
From: Gary Cowan [mailto:[email-masked]]
Sent: Thursday, September 12, 2002 3:43 PM
To: 'Jim Whitehead'; [email-masked]
Subject: RE: Proposal: WebDAV and transactions

Well I suppose, but isn't this what wrapping updates with a Lock/Unlock pair
is intended to accomplish. I have nothing against transactions, I just don't
have any need for them spanning multiple WebDAV methods. Yet:)

-----Original Message-----
From: Jim Whitehead [mailto:[email-masked]]
Sent: Thursday, September 12, 2002 3:24 PM
To: [email-masked]
Subject: RE: Proposal: WebDAV and transactions

One common example is to be able to write a resource and its properties in a
single operation. You want this to be a transaction, so that the resource
doesn't get updates before the properties are written, and so that if the
properties cannot be written, the resource is reverted back to it's
pre-write state.

- Jim

> [mailto:[email-masked]]On Behalf Of Gary Cowan
> Sent: Thursday, September 12, 2002 12:21 PM
> To: 'Babich, Alan'; 'Roy T. Fielding'; [email-masked]
> At the risk of sounding naive, I cannot understand why anyone
> would want or
> need transactions in a WebDAV session.
>
> Sent: Thursday, September 12, 2002 3:11 PM
> To: 'Roy T. Fielding'; [email-masked]

---

From: "Clemm, Geoff" <[email-masked]>
Date: Mon, 16 Sep 2002 09:41:13 -0400

Just to be clear here, I wasn't suggesting that we actually use the
DeltaV protocol as a WebDAV transaction protocol.  You need different
versioning and transactioning methods for a server that supports both
versioning and transactions, so that a client can say whether it wants
versioning behavior or transactioning behavior.

I was just making the point that a repository that supported
transactions (but not versioning) could sensibly interact with a
DeltaV client by defining transaction boundaries on certain DeltaV
constructs.  Conversely, depending on how the transactioning
protocol is defined, a repository that supported versioning
(but not transactions) might be able to sensibly interoperate
with a transactioning client by mapping the transactioning
requests into server-side workspace functionality.

Cheers,
Geoff

-----Original Message-----
From: Zaretzke, Peter [mailto:[email-masked]]
Sent: Monday, September 16, 2002 7:54 AM
To: [email-masked]'
Subject: RE: Proposal: WebDAV and transactions

Some thoughts regarding the question "Who needs transactions with WebDAV?" 
For me the major benefit of being able to control a transaction is
CONSISTENCY (and convenience). Keeping track of multiple dependent
modifications on WebDAV resources can become hard for a client and is
usually not what a application programmer wants to do. He wants to define
his "transaction" spanning multiple (WebDAV) calls and depending on the
return codes or his user decision he wants to commit or rollback
(all-or-nothing). Millions of SQL applications are working this way and I
can't see what is wrong with it.  
Batch processing does not really help here since some applications rely on
that intermediate modifications are available at the server, e.g. for query
purposes. Collecting modifications on the client would make query processing
far more complex since the server result has to be combined with the client
side modified resources. Additionally I would like to have the ability to
react on the return codes for single calls. BTW: processing a batch request
requires transactions on the server anyway (all-or-nothing), otherwise it
would be possible to create inconsistencies if the 3rd of 5 dependent PUTs
fails.  
Another aspect is working ISOLATION. Workspaces and activities are helpful
here. But creating something like a transaction with it, means programming
it on the client. Why should somebody re-invent the "transaction logging
wheel" in his application? Will somebody accept to go back to file-based
application programming style (without transactions) if he is familiar with
SQL database programming? 
For me it does not matter if the COMMIT, ROLLBACK methods are part of the
WebDAV protocol. If there is already a protocol that covers this - great,
let's take it and make it a (optional?) WebDAV extension like DASL or ACL.
But I definitely need server transactions to write complex WebDAV
applications without too complex coding. 
I see a WebDAV server as a Resource Manager like any database with nice
extensions like versioning, configuration management, etc. In my opinion a
successful general-purpose WebDAV server has to cover the common RM
features. I'm currently working on a Content Repository API on top of a
WebDAV server, which is intended to become compliant to JSR170
(http://www.jcp.org/jsr/detail/170.jsp) where transaction control is planned
to be part of Level 2. 
Any comments?