Subject: RE: Proposal: WebDAV and transactions

From: "Babich, Alan" <[email-masked]>
Date: Wed, 11 Sep 2002 14:04:06 -0700

-----Original Message-----
From: Roy T. Fielding [mailto:[email-masked]] 
Sent: Wednesday, September 11, 2002 5:14 AM
To: Eric Sedlar
Cc: Babich, Alan; 'Julian Reschke'; Pill, Juergen; [email-masked]
Subject: Re: Proposal: WebDAV and transactions

...snip...

>> In the spirit of that analogy, I
>> am proposing that the entire transaction, conditional processing and all,
>> should be invoked by one method call. That would do a lot to help protect
>> the performance of the system.

It would do nothing to help protect the performance of the system.
There is no effective difference between a sequence of requests with
time-outs per request and a batch request with time-outs per read.
The server still has to accumulate requests in storage and is
subject to the same service problems with stateful behavior.

...snip...

....Roy

That's completely false. 

Last first. The only way to eliminate state on the server that must be
maintained between method calls ("stateful behavior") is to do the entire
transaction during one method call. Period.

Going over the network multiple times for multiple method calls clearly
consumes more computing resources and elapsed time than making a single
method call. There is more overhead on the client (which doesn't matter) and
the server (which matters a lot). That's really obvious.

But where client application programmers can really kill the performance of
the entire system is by introducing the possibility of user think time into
the middle of a transaction. That is only possible when the transaction can
consist of multiple method calls: 

Between the method calls, the resources changed have to be locked on the
server. (Otherwise, you wouldn't have a transaction, because it would have
the ACID properties.) If a resource that is needed is locked by another
concurrent transaction, your transaction has to wait. If you wait, you have
to worry about waiting forever. DBMS's typically solve that problem by doing
deadlock analysis and picking a transaction to abort and rollback, if
necessary. The method call can not return until the waiting, if any, is
finished. Some remote systems don't do deadlock analysis -- they time out.
Timing out has a terrible impact on performance if it happens more than
rarely.

Let me give you a real life example that actually happened to me. I was
checking in at the gate on a flight and requesting a seat assignment. The
attendant pulled up a screen showing the availability of the kind of seat I
wanted. In the few seconds during which he said "Yes, we can satisfy your
request" and I said "OK, do it", the seat disappeared, and he had to say,
"Sorry, the assignment failed, try again.". The reason that happened is that
there were two complete transactions involved, not one. The first
transaction was a retrieval of the seat availability of the entire plane.
The second was the update transaction. In between the transactions, another
transaction got in there. 

If you provide the ability to do the retrieval as the first part of the
transaction (i.e., as a method call), all the seats on the airplane will
have to stay locked while I make up my mind. Then throughput would be so
terrible that it would be unacceptable. No one else in the country could get
a seat assignment on that plane while I was deciding what to do. 

Unfortunately, I know from plenty of personal experience that that is
exactly the way most client application programmers will initially try to
program it. Programming it that way will allow them to "shoot themselves in
the foot". Not allowing that (by requiring that transactions be a single
method call) will prevent the performance disaster (as well as eliminate
stateful behavior on the server).

In any case, it is completely false that there is no performance difference.

Alan Babich

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 12 Sep 2002 16:01:23 +0200

> subject to the same service problems with stateful behavior.

I suggest you learn how to write an HTTP server before making statements
like that.

> transaction during one method call. Period.

That's specious.  The only difference between one method containing many
requests and one connection containing many methods is syntax and
the number of server responses on that connection.  In one case you
are building state because of HTTP methods and the other because of
XML methods.  The server responses are not in the critical path and
therefore do not impact performance other than giving the client
important information that might be useful to determining whether
it should abort the transaction.

> the server (which matters a lot). That's really obvious.

Pipelined requests make the difference irrelevant.  The overhead of
parsing XML far exceeds the overhead imposed by parsing multiple HTTP
requests.  And, yes, non-idempotent methods can be pipelined if they
are sent as part of the context of a transaction, since building a
transaction context is inherently safe.

> consist of multiple method calls.

Unless you have time-outs on the transaction, which is true of any
performance limited system.  The protocol represents a wire syntax.
If you think that merely packaging multiple requests into a single
request method will force clients to send all requests at once, then
you seriously underestimate the creativity of HTTP client developers.
At least with multiple methods the server has a chance to respond
appropriately in the middle of the stream of requests, rather than
having to special-case the in-body request timeout handler.

> server.

That's not how I described transactions.  I said that the requests are
collected on the server and only applied when the commit message is
received.  If the state has changed such that the methods are no longer
applicable, then the method transaction will fail on commit.

....Roy

---

From: "Julian Reschke" <[email-masked]>
Date: Thu, 12 Sep 2002 21:03:20 +0200

Hi,

(hopefully catching Roy's attention...-)

RFC2616 says about the 302 response status [1]:

"The requested resource resides temporarily under a different URI. Since the
redirection might be altered on occasion, the client SHOULD continue to use
the Request-URI for future requests. This response is only cacheable if
indicated by a Cache-Control or Expires header field.

The temporary URI SHOULD be given by the Location field in the response.
Unless the request method was HEAD, the entity of the response SHOULD
contain a short hypertext note with a hyperlink to the new URI(s).

If the 302 status code is received in response to a request other than GET
or HEAD, the user agent MUST NOT automatically redirect the request unless
it can be confirmed by the user, since this might change the conditions
under which the request was issued. "

...and about the Location header [2]:

"The Location response-header field is used to redirect the recipient to a
location other than the Request-URI for completion of the request or
identification of a new resource. For 201 (Created) responses, the Location
is that of the new resource which was created by the request. For 3xx
responses, the location SHOULD indicate the server's preferred URI for
automatic redirection to the resource. The field value consists of a single
absolute URI.

       Location       = "Location" ":" absoluteURI"

The question: are there any restrictions on what type or absoluteURI(s) are
allowed? For instance, is it legal to redirect to

a) mailto: URLs?

b) file: URLs?

Regards,

Julian

[1] <http://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.10.3.3>
[2] <http://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.14.30>

--
<green/>bytes GmbH -- http://www.greenbytes.de -- tel:+492512807760

---

From: "Jim Whitehead" <[email-masked]>
Date: Thu, 12 Sep 2002 12:19:03 -0700

> applicable, then the method transaction will fail on commit.

IMO, this model would work.

- Jim

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 13 Sep 2002 11:35:39 +0200

> allowed?

There are no such restrictions.

....Roy

---

From: "Dennis E. Hamilton" <[email-masked]>
Date: Fri, 13 Sep 2002 08:31:00 -0700

1.	WHAT ARE THE REQUIREMENTS?

It seems to me that the conversation on transactions is looking at different
approaches to something that might be transactions and comparing those
possible implementations without being sure that we are talking about the
same concept for transactions and whatever the requirements might be in the
context of WebDAV.

I think it is very valuable at this point to stand back and investigate
exactly what is required for transactions and what the constraints on their
support would be for WebDAV.

Here are some things that I noticed in thinking about this.  Subject-matter
experts would have something more useful to say:

2.	WHAT ARE TRANSACTIONS?

I think there needs to be agreement at this level.  This is a key point of
understanding before being able to proceed.

Here is what occurs to me:

2.1	Transaction protection is a way to assure that a persistent store
(database, etc... ) never presents an unrecognized inconsistent state, and
the failure modes always revert to a consistent state.  My understanding is
that in the general case the notion of consistency is beyond the consistency
of the store abstraction.  It is there to support the consistency required
by the application in which the store is used instrumentally.  The
consistency at the granularity of the store abstractions (e.g., WebDAV with
locking) is assumed to be handled already.  Now we are talking about
consistency in coordinated use of a store for satisfying requirements that
are operable at a different level of abstraction.

2.2	Put simply and quite generally, a transaction looks atomic to everyone
but the originating process, and it looks atomic to the originating process
outside of the critical section of the transactioned process.  And the
effect is to achieve a consistent state to all observers outside of critical
sections (the one place where rules are broken in order to be implemented).
And it is requirements of the implemented application that govern what those
consistency/atomicity conditions are and what their satisfaction is.

2.3	Transaction Processing models are highly abstracted, because we are
constrained from knowing all the particular ways that transactions may be
required in securing the consistent operation of an application.  So support
is utilitarian.  There will be limitations on the scope of support, and
understanding what those are and what the cost of going beyond it (as well
as the economy of staying within it) needs to be understood.

3. 	REQUIREMENTS FOR TRANSACTION-PROCESSING SUPPORT

Key things that I think have to be agreed around requirements for
transactions are.   I am not that great in thinking at the requirements
level, so if these don't read like requirements bear with me.  Consider
instead what might be the requirement behind quasi-requirements like these:

3.1.	What is the required atomic appearance of consistency outside of
critical (transaction-protected processing) sections?  (Express as a
principle and what people are to get out of it, not so much as a given
solution.)

3.2.	What is the required appearance of consistency within critical
sections?  (E.g., do changes to an underlying store appear to have taken
place just as they would if transaction-protection were not requested?)

3.3.	What is the ability to integrate in a transaction regime that extends
beyond the operation of a single WebDAV service?  Related to scope of
support (3.5) and

3.4.	The unobtrusiveness of transaction support in the expression of
transaction-protected processes (e.g., that there be little or no difference
beyond what it takes to signify that an operation is protected under an
identified transaction) - only expressed at the level that doesn't foretell
an implementation or protocol approach

3.5.  What is the scope of required consistency support (the place where
use-cases might help the most).

4.	CONSTRAINTS

There are constraints that will apply to the acceptable introduction of
transaction-processing support in the WebDAV protocol.  These limit the
solution space available for responding to requirements.  (Again, I am not
that practiced at differentiating between constraints and requirements.  I
recognize value in doing so.)  Here are some candidates:

4.1.	Actually using an (existing) transaction model at all.  I notice in (3)
I am presuming that transaction-protection techniques are required.  I do
think that is a given, because it is an integrative requirement (integrating
with available rather-standard mechanisms, integrating from the standpoint
of the knowledge and experience required to develop and validate
applications).  But the requirement for that or the constraint of that needs
to be surfaced.

4.1.	Supporting provisions must honor the STEP model and the spirit of HTTP
to no lesser degree than WebDAV already does (that is, simply tunneling
another protocol wouldn't count)

4.2.	If there is a way to provide a scaled-down, high-value case, it must be
convincing and not merely intended that it does scale.  (I don't know how
you do that without demonstrating a full solution, but who knows ... .)

4.3.	And scaling between non-support and (degrees of) support.  Like,
reusing already-defined methods and supplementing their request bodies or
something like that, wherever possible.  Something about failure cases in
this area, along with conditions on use of HTTP and WebDAV error status,
etc.

-- Dennis

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]]On Behalf Of Jim Whitehead
Sent: Thursday, September 12, 2002 12:58
To: Gary Cowan; [email-masked]
Subject: RE: Proposal: WebDAV and transactions

> Well I suppose, but isn't this what wrapping updates with a
> Lock/Unlock pair is intended to accomplish. I have nothing
> against transactions, I just don't have any need for them
> spanning multiple WebDAV methods. Yet:)

Lock/unlock doesn't give you rollback capability. Even though you're the
only person who can write to a resource during a lock, it still may be the
case that you're unable to complete the PROPPATCH part of a PUT/PROPPATCH
pair. The TCP connection may go down, the server may go down, you may be
exceeding your quota, etc.

Now, if the total number of use cases is relatively small, it probably makes
more sense to create new methods explicitly for these use cases, instead of
a general purpose transaction mechanism.

- Jim

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]]On Behalf Of Roy T. Fielding
Sent: Thursday, September 12, 2002 07:01
To: Babich, Alan
Cc: [email-masked]
Subject: Re: Proposal: WebDAV and transactions

[ ... ]

....Roy

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]]On Behalf Of Babich, Alan
Sent: Monday, September 02, 2002 13:59
To: 'B. Shadgar'; Pill, Juergen; [email-masked]
Subject: RE: Proposal: WebDAV and transactions

One thing you do NOT want under any circumstances are
begin-transaction, end-transaction, and abort-transaction
methods.

[ ... ]