Subject: WebDAV and Dynamic Content

From: [email-masked]
Date: Tue, 16 Nov 1999 16:14:46 -0500

Although WebDAV is silent about dynamic content, it is clear that it will
be used to author the source of such content in web application
development. In the spirit of incremental development, content authors will
want to test and debug their dynamic resource in a tight edit, execute,
debug cycle. This implies that extra publish steps required to make
authored resources available to the server runtime, and running multiple
servers for authoring and testing are less than optimal.

Even though specific support for dynamic content is currently out of scope
for WebDAV, we should explore the issues and options to be sure there are
no protocol consequences or opportunities that should be addressed now.
This note is the beginning of a thread to do that.

Dynamic content presents two fundamental problems for a WebDAV server:
1. How does the server distinguish between access to source and access to
processed results
2. How are resources made available to server-side extensions that access
them through the file system
We'll explore each of these issues from the point of view of the client,
and both non-versioning and versioning servers.

Accessing the Source of Dynamic Resources

For many dynamic resources, the dynamic content and the source resources
are different MIME types and have different URLs. For example Java source
resources end in .java while binary executable resources end in .class.
These resources present no problem because the client can do a GET on the
source URL to get the source, and a GET on the separate executable URL to
get the processed results. The DAV:src property on the executable can be
used to get the source resource URL.

For other resources, the dynamic content and the source have the SAME URL.
For example, Java Server Pages, Active Server Pages, server-side includes,
server-side scripting, etc. This presents a problem because there's only
one method to get both, GET. The DAV:src property doesn't help because it
has the same URL as the executable. And, there's no header on GET that can
be used to distinguish source from dynamic results. We could use the Accept
header to specify what MIME type is desired, and use that to distinguish
source from dynamic results, but that might now work either as they could
have the same MIME type, and the dynamic resource might support multiple
MIME types.

The FrontPage extensions resolve this conflict by either using a method
extension to get the source, or a new header. Could someone supply the
details? They might be a useful model for how we should extend WebDAV.

The current WebDAV doesn't provide any interoperable way to resolve this
issue. However, advanced collections and binding does. Here is a list of
possible solutions (assuming advanced collections), and their pros and
cons:

1. Configure the WebDAV server to not process any dynamic content. Require
the document author to publish the modified source to another server that
is configured to process dynamic content, and not return source (might be
another WebDAV server). The publish operation could be done with a cross
server COPY. The web application developer uses two different URLs to two
different servers to distinguish the source resource from the dynamic
resource. A variant of this approach would be to use virtual hosts in a
single server, but this is more server dependent.
+ works with existing WebDAV servers, doesn't even need advanced
collections
+ also solves the second issue, how the runtime gets access to WebDAV
resources through the file system
- creates redundant resources consuming both developer and CPU time and
disk space
- results in the potential for stale copies of runtime resources
introducing difficult to find bugs
- requires an extra publish step after every update that's easy to forget
- requires the administration of an extra Web server
- user has to remember two URLs, one for the source and one for the dynamic
results. However, they could differ only in the server name or port.

2. For every dynamic resource, create a binding from the dynamic resource
to its source, and configure the server so that the URL for the source
binding does not invoke dynamic content. For example, bind HelloWorld.jsps
to HelloWorld.jsp and configure the server so that resources ending in
.jsps are text/plain while resources ending in .jsp are processed Java
Server Pages.
+ only need one server running
+ no duplicate resources
+ no need to copy, and therefore no possibility of forgetting and getting
stale dynamic resources
+ no publish step, the BIND only has to be done once.
- have to do the BIND for each dynamic resource, but only once
- have to administer the server to not process some URLs (the ones bound to
the dynamic resources for the purpose of accessing their source)
- user has to remember two URLs

3. Extend WebDAV to include a Process-Source Boolean header. GET on a URL
with Process-Source set to 'F' would return the dynamic content while 'T'
would return the source. If the content is not dynamic, the header is
effectively ignored as both 'T' and 'F' would have the same effect.
+ only need one server
+ no duplicate resources
+ no copy and no stale dynamic resources
+ no BIND step or publish
+ no administration required to distinguish source and dynamic content URLs
+ user only has to remember one URL
- requires an extension/change to WebDAV
- introduces another header

4. Add a GETSOURCE method.
+ no new header required
+ user only has to remember one URL
- another method that usually does the same thing as GET

Allowing dynamic resource to access resources using the file system

Most dynamic resources, i.e., servlets, Java Server Pages, Active Server
Pages, server-side scripting in HTML, CGI-BIN applications, etc. that are
invoked by a Web server assume Web resources managed by the server are
stored in the file system, often at specific locations relative to their
own file system path. This is very unfortunate coupling between the
server-side extensions, and the Web server implementation, but it is
standard practice. For many WebDAV servers, and almost certainly versioning
WebDAV servers, the resources are not stored in the file system, but rather
are stored in some private repository. For the authoring environment, this
is no problem because the authoring client just does GETs and PUTs and
isn't aware of the server's implementation. However, any dynamic content
processed by the Web server is free to open files on the server as it
wishes, and if those files are Web resources authored through WebDAV, they
might not be visible to the dynamic resource and its runtime environment.
Ideally, the dynamic resource would reference other resources through a URL
so it could access them through WebDAV with versioning support and proper
access controls. But in cases where they just open files, we have to have
some other solution.

1. Require the user to publish from the WebDAV store to the server runtime.
+ it works
- introduces redundancy with associated system resource consumption and
update anomalies.
- requires a separate publish step for every update before it can be
tested. Likely to cause lots of bugs

2. Mirror the resources in the file system.
+ makes them accessible to dynamic resources
- introduces redundancy with associated system resource consumption and
update anomalies.
- what version of a versioned resource would be mirrored? Some published
workspace?
- requires users to do a publish step to control when the server cache is
updated or requires servers to keep them up to date.

3. Build a file system device driver on top of WebDAV so that applications
that access the file system will transparently access WebDAV resources.
+ eliminates the redundancy
+ eliminates the publish step
- has performance implications
- introduces OS dependencies, there's no standard file system access API.
- file system APIs may not be capable of specifying enough information or
parameters to support versioning, properties, locking, etc.

---

From: Brett McCormick <[email-masked]>
Date: Fri, 19 Nov 1999 11:08:30 -0800 (PST)

> used to get the source resource URL.

These would not be examples of dynamic resources.  Java source
resources are manually compiled into class files.  These aren't being
generated dynamically.

> have the same MIME type, and the dynamic resource might support multiple

This is normally how dynamic content works.  Input file->code->output.
Sometimes the input file is the code, sometimes its the output,
sometimes its a little bit of both.  Or there could be many input
files, which is quite likely, or even multiple outputs.  What does DAV
do with multipart content?

> details? They might be a useful model for how we should extend WebDAV.

It sounds shortsighted, as there can be multiple sources.  Does the
new method get all sources?  What about PUTing the source back to
server?

--brett

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 19 Nov 1999 22:56:40 -0800

>1. How does the server distinguish between access to source and access to

Different URI.

>them through the file system

By authoring the server-side extension configuration (a different URI,
or maybe a property of the URI or the immediate collection, but most
definitely not part of the WebDAV standard).  In most cases this is
just automatic based on some characteristic (known to the human author)
of the source URI on that particular server's namespace.

>...

That is never true.  Do not equate filenames to resources.  Every filename
in Apache can correspond to many different resources, and many resources
have no corresponding filename.  The server just maps some imaginary
namespace onto the filename space.

>1. Configure the WebDAV server to not process any dynamic content. Require

Not an option.

>.jsps are text/plain while resources ending in .jsp are processed Java

The answer.

>- have to do the BIND for each dynamic resource, but only once

The binding may be implicit based on the server's configuration.  The
server can tell you what the source URL is after the resource is created
(or, more likely, what the normal GET URL is after the source is PUT to
its own URL).

>the dynamic resources for the purpose of accessing their source)

The server already does that.

>- user has to remember two URLs

Or do a PROPFIND for the source URLs (there can be many more than one).

>- introduces another header

Absolutely not.  It doesn't work for multi-source resources and completely
bolloxes caching.

>- another method that usually does the same thing as GET

A derived resource (the resource that you are calling dynamic) has by its
nature a set of URI that make up its source.  It is far more natural,
and much easier for the server, to treat a natural property of a resource
as a WebDAV property on a WebDAV resource.

>Allowing dynamic resource to access resources using the file system

This has no relevance to WebDAV, aside from the need for access control
to specify whether the user has the right to create resources that include
other resources on the server.

....Roy

---

From: Serge Knystautas <[email-masked]>
Date: Sat, 20 Nov 1999 02:20:09 -0500

I'm very interested in better understanding the pros and cons of using
the same URI for viewing dynamic content, and editing it's source.  For
Perl scripts, TCL scripts, JSPs, ASPs, and eventually XML documents
using XSLT and more advanced stuff, I would think WebDAV could really
benefit from consideration of this usage.

Comments below...

> of the source URI on that particular server's namespace.

I would prefer to not have to educate each human author (probably
frequently different for each web server) to be able to use WebDAV in
this context.  Surely the software and protocol could realize and
indicate when it wants processed content and when it wants source
content.

> namespace onto the filename space.

Again, <gorilla voice>Me see /folder/foo.bar.  Me want edit
/folder/foo.bar.  Why me must edit /folder/foo.bars?</gorilla voice>

> >1. Configure the WebDAV server to not process any dynamic content. Require

Um, could you complete this thought and explain why you do not consider
this an adequate option?

> Or do a PROPFIND for the source URLs (there can be many more than one).

I think of a user as a human (who must remember).  Then there's the
client software that does a PROPFIND.  I'm not sure how your comment
relates to his point.  Could you expand on it?

> Absolutely not.  It doesn't work for multi-source resources and completely

Well, I need to read up on WebDAV-related caching and multi-source
resources, but could you expand on how this doesn't work and how caching
is bolloxed up?

Serge Knystautas
Loki Technologies
http://www.lokitech.com

---

From: Brett McCormick <[email-masked]>
Date: Sat, 20 Nov 1999 00:32:30 -0800 (PST)

I'm not exactly sure why we're having this thread, there is an
established mechanism for this in the webdav spec.  But, in my view:

A URI refers to a distinct resource.  A URI cannot simultaneously
refer to the source and output of a dynamic resource.  That's what the
source element is for.

To say that you should be able to access the source *and* output of a
dynamic resource is to say that one uri does not equal another, when
that is clearly not true.

The source(s) and output are different resources.  Therefore, access
to them should be via distinct URIs.

More comments below...

> indicate when it wants processed content and when it wants source

The education of each author is already required.  You have to tell
them that what they need to edit is at /foo/bar/baz/, and you can also
tell them at the same time that to access the source they should use
/source/foo/bar/baz.

The client doesn't get to choose if they get processed content or not,
they get what they requested: the content of the given URI.

> /folder/foo.bar.  Why me must edit /folder/foo.bars?</gorilla voice>

That's why webdav compliant editors/publishers should pay attention to
the established mechanism for resolving this problem (?).

> Um, could you complete this thought and explain why you do not consider

It's a little hard to sum up.  In has to do with the fact that:

1) DAV functionality is often an add-on to existing, functional http servers.
2) the dav implementation more than likely has no control over whether other
   parts of the http server perform their requested function
3) the dav implementation will also not likely know all the actual
   source(s) of the dynamic content

For instance:

I have an apache module which dynamically generates GIF files based on
the request URI.  How do you suggest adding functionality to DAV
(mod_dav, in this case) that would prevent me from returning the
dynamic content and instead return the "source" of the dynamic content.

I do not know how mod_dav would behave if I tried to PUT to one of the
urls, I would hope that it would complain appropriately.  I may try
this.

> relates to his point.  Could you expand on it?

His point is that there is already a way of doing all this, why not
use it?

> resources, but could you expand on how this doesn't work and how caching

Ok.  You request http://foo.com/bar/.  Your browser is configured to
use a proxy, so it requests it.  You then attempt to publish to this
url.  Your DAV client also makes a GET request to the same URI, except
this time attaching the appropriate proprietary header.  The proxy has
no idea what that header means, and of course returns the cached
dynamic content of the URI, which you edit and accidentally publish to
the source.

It all comes down to the fact that you can't use one URI to refer to
two separate resources.

--brett

---

From: Jim Davis <[email-masked]>
Date: Sun, 21 Nov 1999 16:15:46 +0100

At 02:20 AM 11/20/99 -0500, Serge Knystautas wrote:

Serge, I take the point of your remarks to be that, all things being equal,
you would prefer a definition of the protocol such that it is very easy to
use, so that even "gorillas" (as in your example) can use it.  I guess we
all would, but it seems to not be possible.

I regret that I do not see any way to make WebDAV that simple.  As Einstein
said "Everything should be made as simple as possible, but not more so".

There is an understandable temptation to hope (or at least wish) that
somehow the software agent or server can be make smart enough to "guess"
whether a URL is being used to refer to the "source" of dynamic resource or
the result of executing it.  But I do not think this is possible.

Others have proposed adding a proprietary header that controls the
processing.  But as Roy as explained, this is a bad approach.

Really the simplest thing is to use two distinct URLs for two objects.

If you don't find Roy's explaination (and Brett's elaboration) sufficient,
I will try myself to make the case.  But perhaps you (and others) are now
convinced?

By the way have you been tracking the design discussions around the BIND
method?

As for Exchange, it's really too bad that you guys made the choice you did
without raising it in this forum first.  you could have gotten a lot of
good advice for a very low price.  Well, I suppose it's too late to change
Exchange's behavior.  I just hope no one decided to follow their example.

---

From: Serge Knystautas <[email-masked]>
Date: Sun, 21 Nov 1999 14:29:24 -0500

Jim,

You're right that I would like a gorilla to be able to edit these
resources.  :)  I think fundamentally that if the WebDAV group decides
to only support editing static resources and not dynamic resources, it
will be a limited protocol.  Roy and Brett raised some of the technical
challenges, but I believe the challenges are just that... challenges,
and not unconquerable obstacles.

Comments below...

> the result of executing it.  But I do not think this is possible.

Ok, I'm making the assumption that a client using WebDAV to retrieve a
resource could be knowledgeable enough to request either the source or
processed version of the code (because of added functionality in the
WebDAV protocol).  For static content, this would be the same.  The
client could therefore know not to let a user edit the processed results
and store those back over the source code.  I'm suggesting we add
something to WebDAV to make it aware of the difference between processed
and source, and NOT force clients guess.

> processing.  But as Roy as explained, this is a bad approach.

This is probably using your words other than how you meant them, so I
apologize in advance.  To me proprietary header means an individual or
organization owns/invents/only they use that header.  To which I would
respond that if the WebDAV group added a specific header to control
processing, it would no longer be proprietary.

> I will try myself to make the case.  But perhaps you (and others) are now

Well, you can probably guess I'm not convinced.  I'm just hoping to
point out to the group is that this is very, very useful functionality,
and I consider it short-sighted to limit WebDAV to only support static
resources, especially with the proliferation of server scripting
languages where code is embedded within content (typically HTML).  The
WebDAV group can decide they do not wish to support dynamic content (as
it seems to have already been decided).  To that I'll add that I think
vendors implementing WebDAV (such as Microsoft and others as WebDAV
grows in acceptance) will end up supplying this functionality outside of
the WebDAV standard if the standard will not grow to support it.  This
will lead to proprietary headers, or other short-sighted and limited
ways of doing this (that Roy and Brett and you and many others will be
able to point out the flaws to).  I think it's in WebDAV's best interest
to put it's bright minds towards this problem rather than deciding "not
my problem" or "we don't support that."

I hope I haven't ruffled too many feathers.  I'm normally just a lurker
here and will shortly return to that.  Again, I think this is needed
functionality, and if we don't figure out how to support it, others
will, and you'll get splintered efforts with poorly implemented
approaches.

Serge Knystautas
Loki Technologies
http://www.lokitech.com

---

From: "Roy T. Fielding" <[email-masked]>
Date: Sun, 21 Nov 1999 19:15:24 -0800

>and not unconquerable obstacles.

I think you are missing the point.  There are no challenges in what I
described, just choices -- the ability to edit dynamic resources is
already present in the protocol.  The challenge you are describing is
how to represent the protocol actions to the user, which I just don't
think is relevant.  A "smart" client can represent the protocol actions
however it wants, including as if the client were "editing" the dynamic
resource, but this does not change the protocol one bit.

The protocol must perform operations on static content because that
is the only way it can be done.  You don't edit the output of a C program;
you edit the static content of the code that becomes that program.
One way or another, the client must be informed that the content via GET
is from a derived source and thus must be edited at the source(s).  But
what the client sees and the user sees are two entirely different concepts.
A smart client will understand the meaning of a "you can't edit me, go over
there to my source" response and present it to the user in such a way that
an author will understand.  If they don't understand it, then they really
aren't the author of that resource and the right thing to do is to tell
them to piss off rather than mistakenly replace a derived resource with
the wrong content.

>and source, and NOT force clients guess.

That is already present in the protocol, as the source property.
The reason that the source isn't advertized as a external link on every
GET response is because some people don't like to publish how their
resources are generated, and it doesn't make an efficient tradeoff given
that a typical resource is going to be GET'd millions of more times
than it will be PUT.

>my problem" or "we don't support that."

You can rest assured that WebDAV would not have been published as an RFC
if I had even the slightest belief that it didn't support the authoring
of all types of resources present on the Web today, either within the
main draft or as part of collections.  You just have to remember that a
resource does not always map to a singular file, but all resources that
are not static are derived from some other resources, and by following the
derivation tree you will eventually find all of the static things that
must be edited in order to modify the content of a resource.

A server cannot "abstract away" those actions because the server doesn't
know what the user is trying to do.  Assuming that the user is just a
dumb browser-editor would eliminate the ability of the Web to support
more sophisticated clients, thus artificially restricting the variety
of Web applications.  A user agent, on the other hand, knows its own
intended purpose (e.g., web editor, file editor, folder manager,
maintenance spider, etc.) and is capable of presenting the raw WebDAV
protocol actions in a way that can be understood by its own user, at
whatever level of sophistication you deem appropriate.

The same principles that apply to dynamic content also apply equally to
content negotiation, scripts, servlets, configurations, versioning, etc.
This is why the Web works.

....Roy

---

From: Serge Knystautas <[email-masked]>
Date: Sun, 21 Nov 1999 23:26:02 -0500

> that a typical resource is going to be GET'd millions of more times

Thanks for the tip (and for your patience).  I will look into the source
property.  Do most client/servers implement this, or is this relatively
unsupported right now (is this class 1 or class 2 feature, if that
question makes sense)?

Serge Knystautas
Loki Technologies
http://www.lokitech.com

---

From: Greg Stein <[email-masked]>
Date: Sun, 21 Nov 1999 21:40:17 -0800 (PST)

> unsupported right now (is this class 1 or class 2 feature, if that

mod_dav supports it as a "dead" property and depends on the server admin
to set up the appropriate structures on the server.

Some future version of mod_dav (sooner or later... I dunno) will support a
directive to automatically set up DAV:source properites, aliases, and
content-type settings.

I posted a note a couple days ago about how mod_dav might handle the
property.

Cheers,
-g

-- 
Greg Stein, http://www.lyra.org/