Subject: [apps-discuss] remote storage: per-user cloud storage for the web

From: "Michiel B. de Jong" <[email-masked]>
Date: Mon, 25 Nov 2013 13:23:35 +0100

At http://remotestorage.io/ we are nearing completion of version -02 of 
the 'remotestorage' internet draft:

https://github.com/remotestorage/spec/blob/master/draft-dejong-remotestorage-02.txt

After input from stakeholders (current storage providers 5apps.com as 
well as application developers Nick Jennings and Jan-C. Borchardt), we 
added the following new features, compared to the -01 version:

* all servers should now support HEAD requests in addition to GET, PUT, 
DELETE and OPTIONS.

* all servers should now send Content-Length headers back on HEAD and 
GET requests.

* the folder description ("directory listing", when querying a URL that 
ends in a slash "/") now gives information about Content-Type and 
Content-Length as well as ETag, and it is now a JSON-LD document.

* optional extension #1: putting the bearer token into the URI query 
parameter instead of in the Authorization header.

* optional extension #2: support for Content-Range headers on GET requests.

The document also grew from 10 to 20 pages, mainly because we added wire 
transcripts, and a computer-science-style discussion of how to do 
distributed versioning with a remoteStorage server.

Comments very very welcome! We are hoping to concentrate more interest 
in this topic.

If you send your comments before 10 December then we can still edit the 
I-D before we publish it. You can reply to this thread, or open an issue 
on https://github.com/remotestorage/spec/issues?state=open

cheers!
Michiel

---

From: Julian Reschke <[email-masked]>
Date: Mon, 25 Nov 2013 13:44:22 +0100

> * all servers should now support HEAD requests in addition to GET, PUT,

HEAD support is already required by HTTP.

> * all servers should now send Content-Length headers back on HEAD and

So you don't need support for content that is streamed (length unknown 
in advance?).

> ...

This seems to overlap with WebDAV and AtomPub. Did you consider using 
one of those?

Best regards, Julian

---

From: "Michiel B. de Jong" <[email-masked]>
Date: Mon, 25 Nov 2013 15:19:14 +0100

hi Julian,

thanks for your comments!

> HEAD support is already required by HTTP.

true, but we never said servers should implement all of HTTP. For 
instance, chunked transfer-coding is also required by HTTP/1.1 and we 
also don't require servers to support that. is that not clear from the 
text? maybe we should mention that explicitly, then.

> So you don't need support for content that is streamed (length unknown 

correct. it's just a simple file-system-like storage, no streaming.

> This seems to overlap with WebDAV and AtomPub. Did you consider using 

yes, we actually started with full WebDAV, and then looked at what we 
could remove.

we consider  the actual REST interface to the simplest part actually, 
it's just pretty standard read-write web. very similar to the API that 
GoogleDrive and Dropbox expose. there are a few choices in the format, 
but that's pretty trivial to abstract from.

the innovation, we believe, is in the combination with:

* CORS headers,
* client-side OAuth (not by default leaking credentials to any server),
* WebFinger for per-user service discovery,

making it usable as a per-user data storage. See 
http://nobackend.org/2013/11/cross-origin-backend.html for discussion.

Cheers,
Michiel

---

From: Julian Reschke <[email-masked]>
Date: Mon, 25 Nov 2013 15:22:58 +0100

> text? maybe we should mention that explicitly, then.

So you are sub-setting HTTP? Not a good idea.

Best regards, Julian

---

From: Jan Algermissen <[email-masked]>
Date: Mon, 25 Nov 2013 15:25:15 +0100

> So you are sub-setting HTTP? Not a good idea.

In addition: Why would you need to subset HTTP? The off-the shelf connectors implement it already anyhow. Where is the point?

Jan

> 
>> ...
> 
> Best regards, Julian
> _______________________________________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss

---

From: Dave Cridland <[email-masked]>
Date: Mon, 25 Nov 2013 14:47:14 +0000

On Mon, Nov 25, 2013 at 2:19 PM, Michiel B. de Jong <
[email-masked]> wrote:

> instance, chunked transfer-coding is also required by HTTP/1.1 and we also
> don't require servers to support that. is that not clear from the text?
> maybe we should mention that explicitly, then.
>
>
Oh, dear lord, that is horrible. Your document actually says it's based on
HTTPS, and the only hint that you're intending to subset is that both HTTP
and HTTPS are informative references.

If you're expecting to chop up HTTP in a sane way without respecifying it,
I think you're onto a loser.

If I implementing this according to your specification, I would start with
an HTTP implementation and move on from there - if your protocol differs
from HTTP, then that's a non-starter - at the very least I'll need to
ensure I can turn off whichever features is it you're implicitly casting
aside. The latter certainly involves being very careful about the library I
use.

In the case of chunked encoding, I find it actually quite useful for
anything but static files. FWIW, I recently worked with a remote-storage
style API, and found the lack of chunked encoding support in this
particular implementation to be a real pest. For one thing, I was dealing
with CTE encoded email parts, and an exact length is hard to determine
before decoding (and often impossible without having the data to hand).

>> So you don't need support for content that is streamed (length unknown in
>> advance?).
>>
>
This is a bad idea.

It'd be reasonable if the underlying building blocks didn't support this,
but they do already, so you're not restricting your goals, but restricting
your use-cases. I also have to carefully ensure my HTTP library doesn't
switch to using Chunked, making it a not-HTTP library. I have no idea what
else I need to turn off.

>
>  This seems to overlap with WebDAV and AtomPub. Did you consider using one
>> of those?
>>
>
Have you had any reviews or opinions from WebDAV experts? I appreciate that
WebDAV is a complex protocol, and as the highest point on the stack it's
the logistically easiest thing to reinvent, but I'd be interested in
whether the likes of Lisa Dusseault have made any comments here.

It could be that they'll be able to give you useful directions to explore
that'd give you the overall requirements you're seeking to fill without a
lot of the excess you're hoping to avoid - and not preclude use of things
like WebDAV in the future.

Dave.

---

From: "Michiel B. de Jong" <[email-masked]>
Date: Tue, 26 Nov 2013 19:11:15 +0100

hi!

thanks for your comments, we had a bit of discussion about the 
"sub-setting http" issue with server implementers, and the basic 
conclusion that you have a valid point.

maybe we should say that we are defining an API (a storage protocol, 
basically) and not a network protocol?

in practice, we want to ban "exotic" behavior unless it can be dealt 
with at the browser-level. So for instance, a server offering SPDY is 
ok, because the browser executes its XMLHttpRequests in the same way, 
whether over SPDY or not.

On the other hand, we don't want servers responding with any status 
codes that are not short-listed in section 5 
https://github.com/remotestorage/spec/blob/2b2091104696a0105d0176e9547a3be84facb336/draft-dejong-remotestorage-02.txt#L278-L317 
), because an exotic status code in an http response would not be 
handled transparently by the browser; it would reach the JavaScript 
code, which would not know what to do with it.

in practice, i think the only two points where our I-D subsets http are 
not mentioning chunked transfer codings, and using only 13 out of the 
40+ response codes that http allows.

it also doesn't use all the verbs, for instance it leaves POST unused, 
but the http spec says all verbs apart from GET and HEAD are optional, 
so we are ok there.

what is your opinion about allowing only a short-list of status codes? 
don't all server APIs do this in practice?

Many thanks for your comments so far,
Michiel

On 25-Nov-13 15:47, Dave Cridland wrote:
> On Mon, Nov 25, 2013 at 2:19 PM, Michiel B. de Jong 
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>     hi Julian,
>
>     instance, chunked transfer-coding is also required by HTTP/1.1 and
>     we also don't require servers to support that. is that not clear
>     from the text? maybe we should mention that explicitly, then.
>
>
> Oh, dear lord, that is horrible. Your document actually says it's 
> based on HTTPS, and the only hint that you're intending to subset is 
> that both HTTP and HTTPS are informative references.
>
> If you're expecting to chop up HTTP in a sane way without respecifying 
> it, I think you're onto a loser.
>
> If I implementing this according to your specification, I would start 
> with an HTTP implementation and move on from there - if your protocol 
> differs from HTTP, then that's a non-starter - at the very least I'll 
> need to ensure I can turn off whichever features is it you're 
> implicitly casting aside. The latter certainly involves being very 
> careful about the library I use.
>
> anything but static files. FWIW, I recently worked with a 
> remote-storage style API, and found the lack of chunked encoding 
> support in this particular implementation to be a real pest. For one 
> thing, I was dealing with CTE encoded email parts, and an exact length 
> is hard to determine before decoding (and often impossible without 
> having the data to hand).
>
>
>         So you don't need support for content that is streamed (length
>         unknown in advance?).
>
>
> It'd be reasonable if the underlying building blocks didn't support 
> this, but they do already, so you're not restricting your goals, but 
> restricting your use-cases. I also have to carefully ensure my HTTP 
> library doesn't switch to using Chunked, making it a not-HTTP library. 
> I have no idea what else I need to turn off.
>
>
>         This seems to overlap with WebDAV and AtomPub. Did you
>         consider using one of those?
>
>
>     yes, we actually started with full WebDAV, and then looked at what
>     we could remove.
>
>
> Have you had any reviews or opinions from WebDAV experts? I appreciate 
> that WebDAV is a complex protocol, and as the highest point on the 
> stack it's the logistically easiest thing to reinvent, but I'd be 
> interested in whether the likes of Lisa Dusseault have made any 
> comments here.
>
> It could be that they'll be able to give you useful directions to 
> explore that'd give you the overall requirements you're seeking to 
> fill without a lot of the excess you're hoping to avoid - and not 
> preclude use of things like WebDAV in the future.
>
> Dave.

---

From: Julian Reschke <[email-masked]>
Date: Tue, 26 Nov 2013 19:25:25 +0100

> code, which would not know what to do with it.

That's exactly the kind of subsetting that is bad.

Maybe you could explain what *exactly* the problem with "exotic" status 
codes is? The HTTP spec is pretty clear about how to process them.

BTW: while looking at 
<https://github.com/remotestorage/spec/blob/2b2091104696a0105d0176e9547a3be84facb336/draft-dejong-remotestorage-02.txt#L278-L317>:

>        * 400 for all malformed requests (e.g. foreign characters in the
>              path or unrecognized http verb, etcetera), as well as for
>              all PUT and DELETE requests to folders,

There's already status code 501 for unrecognized verbs.

> don't all server APIs do this in practice?

Not all; for instance WebDAV gives *examples* but always reminds 
implementers that other status codes can occur too.

Best regards, Julian

---

From: "Michiel B. de Jong" <[email-masked]>
Date: Wed, 27 Nov 2013 13:08:54 +0100

hm, it just seemed better to me to keep it simple. we are not defining 
general-purpose webservers here, quite specifically just storage 
servers. this is a much smaller problem space than the web as a whole. 
we don't need all of http's features, and it's more valuable to us that 
their behavior is predictable. see for instance

http://atnan.com/blog/2008/08/08/transfer-encoding-chunked-or-chunky-http/

also, since all data on a remoteStorage server is put there by a 
remoteStorage client, and we don't support something like 
http://www.webdav.org/specs/rfc4437.html#METHOD_MKREDIRECTREF to create 
redirects, it will never occur that a server responds with a redirect 
header. the list of status codes used to implement remoteStorage is 
simply shorter than the list of status codes that browsers support in 
order to display content from the variety of webservers out there.

the web is exciting in its variety. remoteStorage servers are boring in 
their predictability. that's a good thing. :)

so to clarify the situation, i added this paragraph:

https://github.com/remotestorage/spec/blob/1baccdddfd8606596583f19561d0eed3003d2251/draft-dejong-remotestorage-02.txt#L167-L177

as far as i know we are not deviating from http in any place other than 
with the chunked encoding and with the shorter list of allowed status codes.

i did add chunked encoding as a 'MAY' now, though, and maybe we'll end 
up making it a 'MUST' (in line with HTTP/1.1) in version -03, after we 
experiment a bit with it in practice.

many thanks for your contributions!

Cheers,
Michiel

On 26-Nov-13 19:25, Julian Reschke wrote:
> Maybe you could explain what *exactly* the problem with "exotic" 
> status codes is? The HTTP spec is pretty clear about how to process them.
>
> BTW: while looking at

---

From: Julian Reschke <[email-masked]>
Date: Wed, 27 Nov 2013 13:22:19 +0100

Quoting:

>         * Whereas [HTTP, section 10] allows many different status codes,
>           remoteStorage servers SHOULD send the status codes mentioned
>           in section 5 of this Internet Draft, and clients MAY treat any
>           deviation from this as a server malfunction.

I still don't understand how that helps. What problem are you solving here?

Best regards, Julian

---

From: "Michiel B. de Jong" <[email-masked]>
Date: Wed, 27 Nov 2013 14:18:51 +0100

> I still don't understand how that helps. What problem are you solving 

if we allow servers a choice of all response codes then the client 
implement code to deal with each one of them. most of that code would 
never be used in practice, and thus be easily end up being buggy and 
lead to incompatibility.

we are solving those three problems (unnecessary work for client 
implementers, risk of bugs, and risk of incompatibility) by saying 
exactly which 13 out of 41 (if i counted correctly) status codes we 
need, to get the job done. the other 28 status codes are useful to some 
part of the web, but they are useless to our specific problem space, so 
it's safer to disable them.

cheers,
Michiel

---

From: Julian Reschke <[email-masked]>
Date: Wed, 27 Nov 2013 14:30:57 +0100

> never be used in practice, and thus be easily end up being buggy and

The code to implement unknown codes is trivial. Just treat 2xx as 200, 
3xx as 300, 4xx as 400, and 5xx as 500.

> it's safer to disable them.

I disagree. By unnecessarily creating a profile:

a) you teach people to handle status codes incorrectly,

b) make it impossible to introduce new codes,

c) might make it impossible to use existing libraries,

d) ignore that intermediaries might change messages,

e) make it unnecessarily hard to have a server that supports "your" 
protocol and somebody else's protocol on the same URI.

Best regards, Julian

---

From: "Markus Lanthaler" <[email-masked]>
Date: Wed, 27 Nov 2013 15:02:10 +0100

Michiel,

> never be used in practice, and thus be easily end up being buggy and

I think you should try to look at it from a different perspective.
draft-dejong-remotestorage-02 does not specify clients, it specifies a
protocol (or Web API if you prefer) on top HTTP. As such, it's enough to
define the protocol operations you need and leave the rest undefined. You
can't prevent server (or intermediaries!) from sending different status
codes anyway. Clients *will* have to deal with them. But you don't have to
specify how because the HTTP specification already does.

> it's safer to disable them.

No, it's not. Clients will have to deal with them anyway in practice. The
only thing you achieve by doing so IMO is that some off-the-shelf HTTP
libraries can't be used to implement conforming remoteStorage servers.

Hope this helps,
Markus

--
Markus Lanthaler
@markuslanthaler

---

From: "Michiel B. de Jong" <[email-masked]>
Date: Wed, 27 Nov 2013 16:01:04 +0100

OK, i changed it so that the checklist of status codes to take into 
account is only a summary of what follows from section 10 of rfc2616 and 
section 3.1 of rfc6750:

https://github.com/remotestorage/spec/blob/30f8b53430fa5fce765e94bb8ff991468875356e/draft-dejong-remotestorage-02.txt#L284-L286

so we are no longer sub-setting http. :) thanks a lot for this improvement!

what do people think in general of the endeavor? is anybody interested 
in starting a broader discussion about per-user storage?

in particular, i think:

* the current big industry stakeholders in this space are probably: 
Dropbox, GoogleDrive, iCloud, and SkyDrive.
* apart from these four, many parties probably have an interest in 
disrupting the per-user storage market, either to neutralize it as a 
business risk, or to create a level basis for entering.
* remoteStorage is a grassroots project, we are about 1 1/2 people. :) 
it is obvious that we cannot do this without the help from you who are 
reading this.

we hope some of the following parties may be interested in joining this 
effort:
* ISPs and mobile operators who want to offer more than just bandwidth 
to their users
* handset manufacturers who want to disrupt the iOS+iCloud / 
Android+GoogleDrive / WindowsPhone+SkyDrive power blocks
* people who care about open and free technology, also on this new layer.

again, comments very welcome!

cheers,
Michiel

---

From: "Markus Lanthaler" <[email-masked]>
Date: Wed, 27 Nov 2013 16:10:43 +0100

On Wednesday, November 27, 2013 4:01 PM, Michiel B. de Jong wrote:
> https://github.com/remotestorage/spec/blob/30f8b53430fa5fce765e94bb8ff9
> 91468875356e/draft-dejong-remotestorage-02.txt#L284-L286
> 

+1

> in starting a broader discussion about per-user storage?

It's very interesting work but I think the real problem is not the client
but the applications integrating remoteStorage into phones, laptops etc.
Dropbox does a fantastic job there.. What we need I think is an open source
alternative of their client for all major OS.

Just my 2 cents

--
Markus Lanthaler
@markuslanthaler