Subject: Proposal for work on an efficient, browser-friendly, HTTP-based communication protocol for fine-grained information exchange

From: Julian Reschke <[email-masked]>
Date: Thu, 12 Aug 2010 10:36:59 +0200

Proposal for work on an efficient, browser-friendly, HTTP-based 
communication protocol for fine-grained information exchange

HTTP/1.1 (RFC 2616) already contains a set of tools for modifying 
resources , namely the methods PUT, POST, and DELETE.

Many systems have been built on top of this, most of them in an ad-hoc 
manner (which is ok when client and server are controlled by the same 
developers).

We would like to cover some of the following use cases extending the 
resource oriented model.

(1) An simple javascript based browser application should be able to 
read fine-grained information (comparable to WebDAV properties) in a 
simple manner using a defined JSON format to be consumed in an intuitive 
fashion.

(2) A simple HTML Form should be able to write information in a patch 
oriented manner containing both binary (file) data and fine-grained, 
typed information using a multipart POST.

(3) A simple javascript application should be able to write information 
in a patch oriented fashion using a defined JSON-diff PATCH content-type 
to update fine-grained information.

There are also several extensions/applications of HTTP in this space, 
such as:

- WebDAV (RFC 4918), which defines (a) a collection model and methods to 
manipulate collections/namespaces, (b) a metadata (=property) model, and 
(c) locking. Other RFCs add extensions on top of this, such as 
Versioning (RFC 3253) and ACLs (RFC 3744).

- The Atom feed format (RFC 4287) and AtomPub (RFC 5023) use a simpler, 
not necessarily hierarchic collection model (which, depending on the use 
case, may be a plus), but does not provide many features WebDAV + 
friends define. Notably, namespace operations are absent.

WebDAV and AtomPub have been very successful so far. WebDAV gets used 
both as a plain remote filesystem protocol (as observed by clients being 
shipped with all operating systems, and both Apache httpd and IIS 
supporting it), and for specific applications, such as Versioning 
(subversion), Calendaring (CalDAV), etc. The same is true for AtomPub, 
which actually may not be used a lot in practice for the original use 
case (feed authoring), but for many other things instead.

Both of those protocol specifications are not easily consumed by 
websites and applications running current browsers and require a lot of 
client-sided scripting to cover simple read and write use cases.

There's a proposal for a protocol called "JSOP", which addresses these 
use cases, which we may want to consider as input for this work: 
<http://www.slideshare.net/uncled/jsop>

So what's wrong with WebDAV?

Since the time WebDAV was designed, we have learned a lot how to use the 
Web and HTTP. Such as:

- if you want to expose data for read operations, make it available to 
GET, and assign URIs,

- consider cacheability, atomicity, and performance of sync operations 
(for instance, syncing large collections),

- be careful with new HTTP methods -- avoid them for things that are not 
of generic use (good: MKCOL, bad: MKCALENDAR) and keep in mind that 
certain platforms (HTML forms, Flash...) can't use them,

- when defining formats, also define internet media types.

Also, in the last few months, new (and not so new) techniques have 
finally been published as RFCs, such as:

- HTTP PATCH method (RFC 5789)

- HTTP Link Header and Link Relations Registry 
(http://tools.ietf.org/html/draft-nottingham-http-link-header-10, in the 
RFC Editor queue)

- Service Discovery through well-known URIs (RFC 5785)

Another potential building block are URI templates (work in progress: 
http://tools.ietf.org/html/draft-gregorio-uritemplate-04)

Considering all of these pieces, it's quite obvious that there's a 
number of specs that would be useful on their own, but could also, 
combined together, form the basis of an interesting authoring protocol:

# Data Model

1) Define a collection model (hierarchy, naming), and a representation 
format.

Can we re-use the WebDAV collection model here? Web application authors 
probably would prefer a JSON representation, so can we simply define 
this as an alternate representation of a DAV:multistatus description of 
a collection?

2) Define namespace operations in terms of manipulating collection 
representations (also consider a mapping to COPY/MOVE).

3) Define a media type to use with PATCH for modifying these 
representations.

4) Define a property model (something like the intersection between 
WebDAV properties and Java Content Repository (JSR-283) properties?)

# Authoring through HTML forms and POST

Define how POST with multipart/form-data (RFC 2388) can be used for 
authoring both content and properties.

# URIs for collection browsing

Assign either hardwired or discoverable URIs for inspecting collections 
(URI templates?). Or maybe link relations for collection navigation 
(similar work for versioning: RFC 5829).

# Improvements to WebDAV

1) Clarify how MOVE and COPY can operate on non-WebDAV resources (this 
question comes up quite frequently).

2) Define how to use POST on WebDAV collections to add members (done: 
see http://greenbytes.de/tech/webdav/#draft-reschke-webdav-post, in RFC 
Editor queue).

3) Define media types (multiple?) for DAV:multistatus.

4) Define a discovery mechanism for GETtable representations of 
PROPFIND/REPORT results (old proposal: 
http://greenbytes.de/tech/webdav/draft-reschke-http-get-location-latest.html).

5) Define a mapping between link-typed WebDAV properties and generic 
Link relations (see proposal: 
http://lists.w3.org/Archives/Public/w3c-dist-auth/2008OctDec/0026.html).

Although some of this will only be partially related to WebDAV, we think 
that this mailing list might be a good venue for discussion.

Expected deliverables from this activity would be:

1) Definition of a very simply data model and a representation format 
for it (required JSON, optionally XML).

2) A format suitable for manipulating the data format above using PATCH 
(potentially tunneled through POST).

3) A binding from multipart/form-data/POST to this model.

4) A separate (?) document explaining how these ingredients would be 
combined in practice.

Extensions to WebDAV and mappings from/to WebDAV could be useful, but 
would not be a core part of this activity. (That is, we can do without 
if no volunteers speak up).

Note  that not all of these specs necessarily need to be on the 
standards  track; for instance, there might be candidates for 
Informational RFCs as  well (see 
<http://tools.ietf.org/html/rfc2026#section-4> for details).

Feedback appreciated.

Julian Reschke
David Nüscheler

PS: people not familiar with the IETF may want to have a look at 
<http://www.ietf.org/tao.html>

---

From: Julian Reschke <[email-masked]>
Date: Fri, 13 Aug 2010 10:35:25 +0200

On 12.08.2010 10:36, Julian Reschke wrote:
> ...

6) A media type for PATCH that allows setting WebDAV properties and the 
content in a single request (PUT/PROPPATCH combined).

Best regards, Julian

---

From: Geoffrey M Clemm <[email-masked]>
Date: Fri, 13 Aug 2010 21:58:00 -0400

One of the things I'd like to see with this work is to settle the various
syntax debates, by just defining mappings to a variety of syntaxes.

One key part of this is already identified below, i.e. define a GET URI
scheme for any method that involves reading data.
In addition, define a POST body syntax for any method that is not in
HTTP/1.1.
Finally, define an AtomPub and JSON variant, in addition to the XML that
WebDAV would normally use.

This would then allow us to focus on the semantics, and avoid all the
syntax debates.
And if a particular syntax has trouble expressing some of the semantics,
that would highlight why one syntax might be preferable to another..

Cheers,
Geoff

---

From: Jukka Zitting <[email-masked]>
Date: Thu, 12 Aug 2010 19:54:41 +0200

Hi,

> communication protocol for fine-grained information exchange

Sounds useful! See [1] for a few basic use cases and requirements I
outlined for such a protocol last year. It seems like the proposed
protocol could match these needs well, and I'd be eager to participate
in ironing out the details. See below for some initial comments.

[1] http://jukkaz.wordpress.com/2009/11/18/content-repository-over-http/

> [... WebDAV / Atom(Pub) ...]
> Both of those protocol specifications are not easily consumed by websites
> and applications running current browsers and require a lot of client-sided
> scripting to cover simple read and write use cases.

I'd add that also on server-side processing JSON or HTML forms is
usually easier than handling WebDAV or AtomPub.

> probably would prefer a JSON representation, so can we simply define this as
> an alternate representation of a DAV:multistatus description of a
> collection?

It would be good if anything that can be expressed by this protocol
could be straightforwardly mapped to WebDAV and/or Atom. The reverse
does not need to be true, I'd rather go for simplicity than strive for
a full one-to-one mapping with another protocol.

> 4) Define a property model (something like the intersection between WebDAV
> properties and Java Content Repository (JSR-283) properties?)

As above, I'd focus on core property types shared by existing standards.

> Assign either hardwired or discoverable URIs for inspecting collections (URI
> templates?). Or maybe link relations for collection navigation (similar work
> for versioning: RFC 5829).

I'd rather avoid hardwiring URIs.

> 1) Definition of a very simply data model and a representation format for it
> (required JSON, optionally XML).
>

5) A test suite for validating compliance of implementations. (Not
sure if this fits the scope of deliverables from an IETF WG, but
should at least be pursued as a parallel effort.)

> <http://www.ietf.org/tao.html>

Thanks! This was my first post here, so let me briefly introduce
myself: I've been working with open source content management since
-97 and for the past few years I've been focusing on Apache Jackrabbit
and other related Apache projects. I work for Day Software and have
participated in the JCR standardization effort in the JCP. I'm from
Finland, but currently based in Basel, Switzerland.

BR,

Jukka Zitting

---

From: Julian Reschke <[email-masked]>
Date: Sun, 15 Aug 2010 12:33:33 +0200

On 14.08.2010 03:58, Geoffrey M Clemm wrote:
> One of the things I'd like to see with this work is to settle the
> various syntax debates, by just defining mappings to a variety of syntaxes.
>

Right. I think if we do this right it might address the biggest 
shortcoming of WebDAV (the non-adressability of PROPFIND results, and 
its implications on every other part of HTTP, such as cacheing or range 
requests).

> In addition, define a POST body syntax for any method that is not in

Do you mean for non-read methods? From a consistency point of few that 
sounds logical.

However, I'm not convinced it's needed; the main issue with extension 
methods nowadays seem to be with broken intermediaries, and that can be 
addressed by using https (which is often ok for authoring operations).

If we did this, we probably would need to spec X-Method-Override (shudder).

> that would highlight why one syntax might be preferable to another.

+1

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Sun, 15 Aug 2010 13:05:23 +0200

On 12.08.2010 19:54, Jukka Zitting wrote:
> Hi,
>
> On Thu, Aug 12, 2010 at 10:36 AM, Julian Reschke<[email-masked]>  wrote:

Yes, that's a good read. When we wrote the proposal I had forgotten 
about it already, otherwise I would have stolen more from it :-)

> usually easier than handling WebDAV or AtomPub.

I think that depends entirely on libraries. For WebDAV, there are good 
libraries out there (Jackrabbit and WebDAV for JAX-RS come to mind).

Actually, JAX-RS + the WebDAV extensions described in 
<http://weblogs.java.net/blog/mkarg/archive/2009/02/release_10_of_w_1.html> 
might be a nice prototyping environment for what we're doing.

> a full one-to-one mapping with another protocol.

Yes.

> As above, I'd focus on core property types shared by existing standards.

Indeed.

We'll need to think about types (none, some, many?), cardinality, and 
naming.

> I'd rather avoid hardwiring URIs.

+1; but it will make things slightly more complicated.

> Finland, but currently based in Basel, Switzerland.

Welcome to the IETF (in case we decide to make this an IETF activity)! 
Otherwise, just welcome to this mailing list... :-)

Best regards, Julian

---

From: Yves Lafon <[email-masked]>
Date: Thu, 19 Aug 2010 08:51:33 -0400 (EDT)

On Thu, 12 Aug 2010, Julian Reschke wrote:

> Proposal for work on an efficient, browser-friendly, HTTP-based communication 
> protocol for fine-grained information exchange
>
> HTTP/1.1 (RFC 2616) already contains a set of tools for modifying resources , 
> namely the methods PUT, POST, and DELETE.
>
> Many systems have been built on top of this, most of them in an ad-hoc manner 
> (which is ok when client and server are controlled by the same developers).
>
> We would like to cover some of the following use cases extending the resource 
> oriented model.
>
> (1) An simple javascript based browser application should be able to read 
> fine-grained information (comparable to WebDAV properties) in a simple manner 
> using a defined JSON format to be consumed in an intuitive fashion.
>
> oriented manner containing both binary (file) data and fine-grained, typed 
> information using a multipart POST.
>
> (3) A simple javascript application should be able to write information in a 
> patch oriented fashion using a defined JSON-diff PATCH content-type to update 
> fine-grained information.
>
> There are also several extensions/applications of HTTP in this space, such 
> as:
>
> manipulate collections/namespaces, (b) a metadata (=property) model, and (c) 
> locking. Other RFCs add extensions on top of this, such as Versioning (RFC 
> 3253) and ACLs (RFC 3744).
>
> - The Atom feed format (RFC 4287) and AtomPub (RFC 5023) use a simpler, not 
> necessarily hierarchic collection model (which, depending on the use case, 
> may be a plus), but does not provide many features WebDAV + friends define. 
> Notably, namespace operations are absent.

It seems that there are two main uses cases here, the first one is to
replace WebDAV's way of handling metadata on a resource, and the second
one goes into the service/data discovery space. Would it be better to
divide clearly the tasks along those lines ?

Also, at that point, the format (JSON) is irrelevant, the data format
should come after defining the goals, and the protocol bits. XFORMS has 
been mentionned, but once again it is premature, and this discussion 
should take place after figuring out the right interactions needed.

For service discovery, well-known URIs is a possibility, especially for
services "on the server", but it might also be useful to to this via Link:
(like a Printer Resource claiming that it can be contacted via IPP, 
without having to contact a wkURI first), so extending the scope would 
be good.

>
> Both of those protocol specifications are not easily consumed by websites and 
> applications running current browsers and require a lot of client-sided 
> There's a proposal for a protocol called "JSOP", which addresses these use 
> cases, which we may want to consider as input for this work: 
> Since the time WebDAV was designed, we have learned a lot how to use the Web 
> and HTTP. Such as:
>
> - if you want to expose data for read operations, make it available to GET, 
> and assign URIs,
>
> - consider cacheability, atomicity, and performance of sync operations (for 
> instance, syncing large collections),
>
> - be careful with new HTTP methods -- avoid them for things that are not of 
> generic use (good: MKCOL, bad: MKCALENDAR) and keep in mind that certain 
> platforms (HTML forms, Flash...) can't use them,
>
> Also, in the last few months, new (and not so new) techniques have finally 
> been published as RFCs, such as:
>
> (http://tools.ietf.org/html/draft-nottingham-http-link-header-10, in the RFC 
> Editor queue)
>
> Considering all of these pieces, it's quite obvious that there's a number of 
> specs that would be useful on their own, but could also, combined together, 
> form the basis of an interesting authoring protocol:
>
>
> # Data Model
>
> an alternate representation of a DAV:multistatus description of a collection?
>
> 3) Define a media type to use with PATCH for modifying these representations.
>
> Define how POST with multipart/form-data (RFC 2388) can be used for authoring 
> both content and properties.
>
>
> 2) Define how to use POST on WebDAV collections to add members (done: see 
> http://greenbytes.de/tech/webdav/#draft-reschke-webdav-post, in RFC Editor 
> queue).
>
> 5) Define a mapping between link-typed WebDAV properties and generic Link 
> relations (see proposal: 
> Although some of this will only be partially related to WebDAV, we think that 
> this mailing list might be a good venue for discussion.
>
>
> 4) A separate (?) document explaining how these ingredients would be combined 
> in practice.
>
> Extensions to WebDAV and mappings from/to WebDAV could be useful, but would 
> not be a core part of this activity. (That is, we can do without if no 
> volunteers speak up).
>
> Note  that not all of these specs necessarily need to be on the standards 
> track; for instance, there might be candidates for Informational RFCs as 
> well (see <http://tools.ietf.org/html/rfc2026#section-4> for details).
>
>
> Feedback appreciated.
>
> Julian Reschke
> David Nüscheler
>
>
>

-- 
Baroula que barouleras, au tiéu toujou t'entourneras.

         ~~Yves