Subject: Re: Proposal for work on an efficient, browser-friendly, HTTP-based communication protocol for fine-grained information exchange

From: Wenbo Zhu <[email-masked]>
Date: Fri, 13 Aug 2010 01:02:56 -0700

Very plausible .. and comments inline. - Wenbo

> -------- Original Message --------
> Subject: Proposal for work on an efficient, browser-friendly, HTTP-based
> communication protocol for fine-grained information exchange
> Date: Thu, 12 Aug 2010 10:36:59 +0200
> From: Julian Reschke <[email-masked]>
> To: WebDAV <[email-masked]>
>
> Proposal for work on an efficient, browser-friendly, HTTP-based
> communication protocol for fine-grained information exchange
>
> HTTP/1.1 (RFC 2616) already contains a set of tools for modifying
> resources , namely the methods PUT, POST, and DELETE.
>
> Many systems have been built on top of this, most of them in an ad-hoc
> manner (which is ok when client and server are controlled by the same
> developers).
>
> We would like to cover some of the following use cases extending the
> resource oriented model.
>
> (1) An simple javascript based browser application should be able to
> read fine-grained information (comparable to WebDAV properties) in a
> simple manner using a defined JSON format to be consumed in an intuitive
> fashion.
>
> (2) A simple HTML Form should be able to write information in a patch
> oriented manner containing both binary (file) data and fine-grained,
> typed information using a multipart POST.
>
> (3) A simple javascript application should be able to write information
> in a patch oriented fashion using a defined JSON-diff PATCH content-type
> to update fine-grained information.
>
> There are also several extensions/applications of HTTP in this space,
> such as:
>
> - WebDAV (RFC 4918), which defines (a) a collection model and methods to
> manipulate collections/namespaces, (b) a metadata (=property) model, and
> (c) locking. Other RFCs add extensions on top of this, such as
> Versioning (RFC 3253) and ACLs (RFC 3744).
>
> - The Atom feed format (RFC 4287) and AtomPub (RFC 5023) use a simpler,
> not necessarily hierarchic collection model (which, depending on the use
> case, may be a plus), but does not provide many features WebDAV +
> friends define. Notably, namespace operations are absent.
>
> WebDAV and AtomPub have been very successful so far. WebDAV gets used
> both as a plain remote filesystem protocol (as observed by clients being
> shipped with all operating systems, and both Apache httpd and IIS
> supporting it), and for specific applications, such as Versioning
> (subversion), Calendaring (CalDAV), etc. The same is true for AtomPub,
> which actually may not be used a lot in practice for the original use
> case (feed authoring), but for many other things instead.
>
> Both of those protocol specifications are not easily consumed by
> websites and applications running current browsers and require a lot of
> client-sided scripting to cover simple read and write use cases.
>
> There's a proposal for a protocol called "JSOP", which addresses these
> use cases, which we may want to consider as input for this work:
> <http://www.slideshare.net/uncled/jsop>
>
> So what's wrong with WebDAV?
>
> Since the time WebDAV was designed, we have learned a lot how to use the
> Web and HTTP. Such as:
>
> - if you want to expose data for read operations, make it available to
> GET, and assign URIs,
>
> - consider cacheability, atomicity, and performance of sync operations
> (for instance, syncing large collections),
>
> - be careful with new HTTP methods -- avoid them for things that are not
> of generic use (good: MKCOL, bad: MKCALENDAR) and keep in mind that
> certain platforms (HTML forms, Flash...) can't use them,
>
> - when defining formats, also define internet media types.
>
> Also, in the last few months, new (and not so new) techniques have
> finally been published as RFCs, such as:
>
> - HTTP PATCH method (RFC 5789)
>
> - HTTP Link Header and Link Relations Registry
> (http://tools.ietf.org/html/draft-nottingham-http-link-header-10, in the
> RFC Editor queue)
>
> - Service Discovery through well-known URIs (RFC 5785)
>
> Another potential building block are URI templates (work in progress:
> http://tools.ietf.org/html/draft-gregorio-uritemplate-04)
>
> Considering all of these pieces, it's quite obvious that there's a
> number of specs that would be useful on their own, but could also,
> combined together, form the basis of an interesting authoring protocol:
>
>
> # Data Model
>
> 1) Define a collection model (hierarchy, naming), and a representation
> format.
>
> I have seen many debates around representation formats when the underlying
meta-model is often ignored ... and the meta-model should cover, in addition
to hierarchy, relations. And naming should allow for different
representations too, e.g. with the URI template[] being one of them.

> Can we re-use the WebDAV collection model here? Web application authors
> probably would prefer a JSON representation, so can we simply define
> this as an alternate representation of a DAV:multistatus description of
> a collection?
>
> 2) Define namespace operations in terms of manipulating collection
> representations (also consider a mapping to COPY/MOVE).
>
> 3) Define a media type to use with PATCH for modifying these
> representations.
>
> 4) Define a property model (something like the intersection between
> WebDAV properties and Java Content Repository (JSR-283) properties?)
>
>
> # Authoring through HTML forms and POST
>
> Define how POST with multipart/form-data (RFC 2388) can be used for
> authoring both content and properties.
>
>
> # URIs for collection browsing
>
> Assign either hardwired or discoverable URIs for inspecting collections
> (URI templates?). Or maybe link relations for collection navigation
> (similar work for versioning: RFC 5829).
>
>
> # Improvements to WebDAV
>
> 1) Clarify how MOVE and COPY can operate on non-WebDAV resources (this
> question comes up quite frequently).
>
> 2) Define how to use POST on WebDAV collections to add members (done:
> see http://greenbytes.de/tech/webdav/#draft-reschke-webdav-post, in RFC
> Editor queue).
>
> 3) Define media types (multiple?) for DAV:multistatus.
>
> 4) Define a discovery mechanism for GETtable representations of
> PROPFIND/REPORT results (old proposal:
>
> http://greenbytes.de/tech/webdav/draft-reschke-http-get-location-latest.html
> ).
>
> 5) Define a mapping between link-typed WebDAV properties and generic
> Link relations (see proposal:
> http://lists.w3.org/Archives/Public/w3c-dist-auth/2008OctDec/0026.html).
>
> Although some of this will only be partially related to WebDAV, we think
> that this mailing list might be a good venue for discussion.
>
>
> Expected deliverables from this activity would be:
>
> 1) Definition of a very simply data model and a representation format
> for it (required JSON, optionally XML).
>
> 2) A format suitable for manipulating the data format above using PATCH
> (potentially tunneled through POST).
>
> 3) A binding from multipart/form-data/POST to this model.
>
> 4) A separate (?) document explaining how these ingredients would be
> combined in practice.
>
> Extensions to WebDAV and mappings from/to WebDAV could be useful, but
> would not be a core part of this activity. (That is, we can do without
> if no volunteers speak up).
>

Resource-based concurrency-control and sync (revision logs) specs may be
developed on top of these deliverables as well.

>
> Note  that not all of these specs necessarily need to be on the
> standards  track; for instance, there might be candidates for
> Informational RFCs as  well (see
> <http://tools.ietf.org/html/rfc2026#section-4> for details).
>
>
> Feedback appreciated.
>
> Julian Reschke
> David Nüscheler
>
>
>
> PS: people not familiar with the IETF may want to have a look at
> <http://www.ietf.org/tao.html>
>
>
>

---

From: Wenbo Zhu <[email-masked]>
Date: Fri, 13 Aug 2010 14:41:30 -0700

[re-repost]

---

From: Julian Reschke <[email-masked]>
Date: Sun, 15 Aug 2010 14:10:18 +0200

On 13.08.2010 10:02, Wenbo Zhu wrote:
> ...
> I have seen many debates around representation formats when the
> underlying meta-model is often ignored ... and the meta-model should
> cover, in addition to hierarchy, relations. And naming should allow for
> different representations too, e.g. with the URI template[] being one of
> them.

I think we need to be careful; if we over-engineer the data model we may 
scare away parts of the audience we want to include. It should be 
possible to define something that has an easy-to-use JSON representation 
but still have a solid model in the background.

Including relations into the model makes sense to me. Links and link 
relations are important.

I'm not sure I understand the point about naming; could you elaborate?

> ...

Concurrency control as in locking? (be it optimistic or pessimistic)

Best regards, Julian

---

From: Wenbo Zhu <[email-masked]>
Date: Thu, 19 Aug 2010 11:00:57 -0700

On Sun, Aug 15, 2010 at 5:10 AM, Julian Reschke <[email-masked]>wrote:

> scare away parts of the audience we want to include. It should be possible
> to define something that has an easy-to-use JSON representation but still
> have a solid model in the background.
>
Separate the logic naming scheme from its syntactic representation ...  and
the former should be derived from the "background model". When relationships
are included, entities may be addressed via links too.

> Concurrency control as in locking? (be it optimistic or pessimistic)
Yes.

---

From: Julian Reschke <[email-masked]>
Date: Sun, 22 Aug 2010 17:08:14 +0200

On 19.08.2010 20:00, Wenbo Zhu wrote:
>     I think we need to be careful; if we over-engineer the data model we
>     may scare away parts of the audience we want to include. It should
>     be possible to define something that has an easy-to-use JSON
>     representation but still have a solid model in the background.
>
> Separate the logic naming scheme from its syntactic representation ...
>   and the former should be derived from the "background model". When

Yes. We just need to make sure that by doing it this way, the wire 
format doesn't get more complicated than necessary.

> relationships are included, entities may be addressed via links too.

I think if we do things right, optimistic locking based on HTTP/1.1 
conditional requests and etags will work.

I'm not convinced there's big interest for pessimistic locking (à la 
WebDAV locking) at this point of time.

Best regards, Julian

---

From: David Nuescheler <[email-masked]>
Date: Mon, 23 Aug 2010 15:58:58 +0200

Hi All,

Julian and I (and occasionally many others) have been discussing a
development effort like this for quite while.
I was involved in mapping the scope of WebDAV (and friends) into a
Java API called JCR[1][2][3] (on which for example Geoff, Julian and
Roy participated) and due to my Day[4] job (pun intended) I find
myself very often in a situation where I need have a standard
web-browser or client-sided web-application interact with a server to
exchange fine-grained information.

Thanks a lot to everybody for all the comments and input.

I guess my main take from this is that I completely agree that we need
to separate the "model"-conversation from the
"format/binding"-conversation.

I would like to mention though that in my mind the goal of making this
effort relevant, efficient and simple is extremely important as well.
While I appreciate that the separation of the discussions, I would
like to volunteer to work on bindings to a JSON + PATCH (multipart
POST) very early on in the process and keep it in sync with the model
as a living set of examples for the interaction with the more abstract
model, to ensure that we keep things practical.

I think there are of a large number of very similar, JSON/POST-based
"protocols" that are defined in an ad-hoc manner by developers. So
there should be quite a bit of experience out there, that can help us
gauge the importance some of requirements fairly quickly. I think if
we manage to take the combined the experience from WebDAV, AtomPub and
JCR (and possibly more domain specific efforts like CMIS) in terms of
the overall scope and the initial relevance of certain features we
should be in great shape.

regards,
david

[1] http://jcp.org/en/jsr/detail?id=170
[2] http://jcp.org/en/jsr/detail?id=283
[3] http://jcp.org/en/jsr/detail?id=333
[4] http://www.day.com

-- 
David Nuescheler
Chief Technology Officer
mailto: [email-masked]

web:  http://www.day.com/ http://dev.day.com
twitter: @davidnuescheler

---

From: Michael Wechner <[email-masked]>
Date: Thu, 26 Aug 2010 09:52:23 +0100

> exchange fine-grained information.

can you give a "real-world" example such that I can understand better :-) ?

And also where the existing "protocols" are lacking?

Thanks

Michael

---

From: Julian Reschke <[email-masked]>
Date: Fri, 27 Aug 2010 16:20:01 +0200

> "format/binding"-conversation.

Ok,

here's a list of things we need to consider (probably incomplete, feel 
free to add)

1. Collections

1.1 Hierarchy: WebDAV collections are hierarchical. A "direct" member of 
a collection has the parent collections URI + one additional path 
segment. JCR same. This also means that relative paths do the obvious 
thing. CMIS/AtomPub: no constraints on the paths (AFAIU).

1.2 Multiple containment: allowed in WebDAV through multiple bindings, 
in JCR through shared nodes. CMIS/AtomPub: not constrained anyway.

1.3 Identification: WebDAV: the same of a resource within a collection 
is a unique identifier. JCR: same (except that for same-name-siblings, 
the array index may change when siblings are removed).

1.4 Ordering: optional features in WebDAV and JCR.

1.5 Member naming: in WebDAV by path segment (URI syntax), in JCR per 
optional namespace name + path segment.

1.6 Name encoding: in WebDAV per spec ASCII (+ URI percent encoding), in 
JCR Unicode.

2. Properties

2.1 Cardinality: properties can only occur once on a resource, but there 
are ways to express lists (WebDAV, JCR)

2.2 Typing: optional in WebDAV (see RFC 4316). Set of predefined types 
in JCR (int, float, string, date, URI, ...)

2.3 Naming: namespace name + local name (WebDAV, JCR)

3. Content

3.1 A single binary stream per HTTP in WebDAV (ignoring content 
negotiation for now which is tricky in authoring); modeled as binary 
property in JCR (with only conventions on naming)

Best regards, Julian

---

From: David Nuescheler <[email-masked]>
Date: Fri, 27 Aug 2010 21:38:25 +0200

Hi Michael,

Thanks a lot for your question.

> And also where the existing "protocols" are lacking?

I am more than happy to give you a couple of real-life use cases that
may give you a little bit of insight on what i have in mind.

Let's assume as a starting point that I have a standard HTML form in a
browser that I want a user to enter information. While it doesn't
really matter what the application is, to make it really tangible
let's say it is a form that I want to let a user update their profile
information, such as their first name, last name and birthdate.

If we assume that this is fine-grained information that is persisted
on the server, one would assume that the user profile could be
considered a resource (in the WebDAV sense) and the fine-grained
information (first name, last name and birthdate) would lend itself to
be properties on that resource (in the webdav sense).

Assuming that I wanted to transport this information to a WebDAV
server using WebDAV as the communication protocol between my browser
and my WebDAV server I would find myself in a situation where I would
have to write a very healthy chunk of javascript code working with XHR
to come up with the PROPPATCH method and the respective XML for the
body of the request to be able to persist my information to WebDAV
server.
If we extend the use case to a user uploading their profile picture,
then the the task to interact with my WebDAV server goes from
"difficult" to "impossible" using my standard browser.

The goal of this effort is to define an interaction pattern that
allows a standard browser to interact with the server in a
fine-grained and efficient manner, yet honoring a lot of the semantics
and model that have been established around WebDAV.

So the above example, in my mind could result in a very simple form
that could look like this. In straight up (somewhat simplified) html
...

---
<form method="POST" action="/profiles/david">
	<input name="firstname" />
	<input name="lastname" />
	<input name="birthdate" />
	<input name="picture" type="file" />
	<input type="submit" />
</form>
---

...and would not require the hundreds of lines of code javascript and
a very cooperative browser to allow me to update my profile.

Further more if we assume that if the client-sided javascript
application becomes more complex and would like to interact with the
fine-grained information from a JavaScript/XHR standpoint there would
be the ability to easily access with a GET the above information as
something like:

---
{
firstname: "David",
lastname: "Nuescheler",
birthdate: "Oct 31 1974"
}
---

... or update the above with something along the lines of a PATCH or a
POST with a body similar to...

---
-birthdate
^lastModifiedBy : "me"
---

... which would remove the birthdate property and update the
lastModifiedBy, directly from a Javascript application.

As suggested in the thread before, I agree that the conversations
around the model that we are operating on and the respective bindings
to specific formats need to be separate conversations, but I think to
make it more tangible and real-life as you requested I thought that it
may make sense to wrap it into the example.

Please let me know if that addresses your question or if it is
something else that you were looking for.

regards,
david

---

From: David Nuescheler <[email-masked]>
Date: Sun, 29 Aug 2010 21:05:05 +0200

Hi Julian,

thanks for kickstarting the conversation...

Here my initial take on the topics below.

> 1. Collections
> 1.1 Hierarchy: WebDAV collections are hierarchical. A "direct" member of a
> collection has the parent collections URI + one additional path segment. JCR
> same. This also means that relative paths do the obvious thing.
> CMIS/AtomPub: no constraints on the paths (AFAIU).
I think the WebDAV hierarchy approach is great and very important
since referencing by (relative) URL is core to basic web things.

> 1.2 Multiple containment: allowed in WebDAV through multiple bindings, in
> JCR through shared nodes. CMIS/AtomPub: not constrained anyway.
While I think that multiple containment is interesting, it is
definitely not at the top of my priority list.

> 1.3 Identification: WebDAV: the same of a resource within a collection is a
> unique identifier. JCR: same (except that for same-name-siblings, the array
> index may change when siblings are removed).
I think the WebDAV way works perfectly.

> 1.4 Ordering: optional features in WebDAV and JCR.
I think this is more important for fine-grained content since when we
talk about DOM-like structures that are persisted on the server of
course the sort order makes a big difference. (As a side comment:
mapping things to JSON in the back of my mind creates a bit of tension
here, since sort-order of JSON object is undefined, while all
implementations in browsers keep the sort-order, technically a JSON
parser is not obligated to do that...)

> optional namespace name + path segment.
I am not sure if we need a special (xml-like) namespace management, it
seems like a lot of overhead.

> 1.6 Name encoding: in WebDAV per spec ASCII (+ URI percent encoding), in JCR
> Unicode.
ASCII + URI encoding sounds great

> 2. Properties
> 2.1 Cardinality: properties can only occur once on a resource, but there are
> ways to express lists (WebDAV, JCR)
I think there is no need to change that.

> 2.2 Typing: optional in WebDAV (see RFC 4316). Set of predefined types in
> JCR (int, float, string, date, URI, ...)
If we look at the types in a browser javascript runtime it seems that
number, strings, boolean and Date seem like obvious candiates.

> 2.3 Naming: namespace name + local name (WebDAV, JCR)
Great.

> 3. Content
> 3.1 A single binary stream per HTTP in WebDAV (ignoring content negotiation
> for now which is tricky in authoring); modeled as binary property in JCR
> (with only conventions on naming)
I think in JCR we went all out and in my mind went too far with
binaries. I think I would be happy with having a single optional
binary stream.
More importantly though since this is about fine-grained information
the typical case will be having "no" binary at all, but just a tree of
properties (and "nodes?").

regards,
david

---

From: Julian Reschke <[email-masked]>
Date: Sun, 29 Aug 2010 21:28:14 +0200

> thanks for kickstarting the conversation...

:-)

I was planning to actually insert my opinion as well, but was waiting 
for other's views first.

Yes.

> definitely not at the top of my priority list.

Indeed. As we have seen in WebDAV and JCR, it can be added later on 
without affecting existing code (just be a bit careful when phrasing 
things, not confusing the identifier with the thing being identified).

> I think the WebDAV way works perfectly.

+1

> ASCII + URI encoding sounds great

Yes, but I think we need to require that percent encoding actually is 
used to encode UTF-8 sequences. In WebDAV, the interop problems between 
*nix boxes (octet sequences as filenames) and Windows boxes (Unicode) 
have been painful.

> number, strings, boolean and Date seem like obvious candiates.

Yes. I also believe that special casing links would be good.

> properties (and "nodes?").

Would a zero length content work as well?

Best regards, Julian

---

From: David Nuescheler <[email-masked]>
Date: Mon, 30 Aug 2010 20:08:19 +0200

Hi Julian,

thanks for the additional color.

Let me chime in on one aspect...

> Would a zero length content work as well?

Well, personally, I would rather avoid that route.

I thought about this from various different aspects and while it of
course works from an implementation and usage standpoint I would argue
that it sets the wrong expectation and targets the wrong use cases.

In my mind the general case is that the "nodes" (or the lack of a
better term) do not have a "binary stream" associated, and in
exceptional cases they do. I see the fine-grained nature more similar
to rows of a table in relational database.
So in my mind it is important to identify the "binary content" as the
special case and make sure that the "binary content-less" concept is
treated as the general case, and not the other way around.

I realize that this is just a matter of setting the perception
correctly but that's precisely why would like to be careful ;)

regards,
david