Subject: Re: [apps-discuss] Non-proprietary sync (was Re: Is e-mail going to die?

From: Mark Nottingham <[email-masked]>
Date: Thu, 4 Jul 2013 15:34:54 +1000

Hi Michiel,

I'm very interested in this area; I think we need open standards here desperately.

I potentially have a lot of feedback for you (both editorial and more substantial), but wanted to first know: what is the remoteStorage community's intent in publishing this draft? 

I.e., do you think you're nearly done, and just need to publish a specification? Are you willing to consider substantial changes to the specification, even if it breaks your current implementations if there's good reason?

Also, what kind of timeline are you on -- i.e., do you have particular milestones you want to hit?

Finally, what's the preferred venue for feedback and discussion? Will you or anyone else from remoteStorage be coming to the Berlin meeting?

Cheers,

On 17/06/2013, at 9:02 PM, Michiel de Jong <[email-masked]> wrote:

> On 2013-06-17 10:09, Dave Raggett wrote:
> > Non-proprietary sync is the next step
> 
> That is exactly what "remote storage" tries to propose: a non-proprietary protocol that does something like Dropbox/GoogleDrive/SkyDrive over cross-origin HTTP.
> 
> We uploaded the -01 version of the I-D last week:
> 
>     http://www.ietf.org/id/draft-dejong-remotestorage-01.txt
> _______________________________________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss

--
Mark Nottingham   http://www.mnot.net/

---

From: "Michiel B. de Jong" <[email-masked]>
Date: Fri, 05 Jul 2013 18:49:55 +0200

Hi Mark,

> I'm very interested in this area; I think we need open standards here

Great!

> I potentially have a lot of feedback for you (both editorial and more
> substantial), but wanted to first know: what is the remoteStorage
> community's intent in publishing this draft?

we hadn't really formulated that until now, so we had a little 
discussion about it; we didn't entirely finish discussing it yet, but 
overall i think most of us agree that the ietf would probably be the 
right platform for developing the idea of remoteStorage further.

we started working on this spec as an open source project because we 
identified a need for one - its development so far was partially 
financed by donations, partially by people donating their time.

It's of course scary to give up control over something we care so much 
about, but we also trust the ietf would do a better job at guarding the 
public's best interest than our small community of quite inexperienced 
volunteers. :)

> I.e., do you think you're nearly done, and just need to publish a
> specification?

it's becoming quite crystallized within its very limited defined set of 
features. you can get an idea of what we consider outstanding issues by 
browsing https://github.com/remotestorage/spec/issues our current 
process is to go through that list twice a year, and see which of those 
issues merit a text change. we are trying to change as little as 
possible each time, to avoid an entropy walk into feature bloat. :)

> Are you willing to consider substantial changes to the
> specification, even if it breaks your current implementations if
> there's good reason?

yes, we do this every 6 months anyway. what type of changes did you 
have in mind?

so far, even though these are themes that continuously come up as 
requests, we have resisted adding for instance:
- byte ranges on GET (and PUT?) requests,
- query parameters like sorting/filtering/paging,
- access control lists,
- long polling and/or other sorts of changes-feeds
- retrieving the historic revision history of a certain document (as 
well as undo/rollback)
- atomic transactions
- etc.

this way, we try to keep the job of the server as simple as possible.

> particular milestones you want to hit?

we try to go as fast as we can, developing the remotestorage.js client 
library for it, data formats for the things we want to store on there, 
try to work out how to structure data into folders so that it's easy to 
fetch later, write apps that use it, and server implementations that are 
compliant with it, with (if you add up the volunteers) about 3 full-time 
people. that's our main constraint. :)

we have been trying to make breaking changes only twice a year, 
although we're probably going to have to disobey that self-imposed rule 
soon anyway because of a critical bug we found this week.

>
> Finally, what's the preferred venue for feedback and discussion?

currently it's a combination of 
https://github.com/remotestorage/spec/issues and 
http://remotestorage.io/community/ but we're open to changing that to 
the apps-discuss mailinglist for instance? do you think that would be 
appropriate?

> Will
> you or anyone else from remoteStorage be coming to the Berlin 
> meeting?

i live in Berlin, and given our project's tight budget, i hadn't bought 
a ticket yet. but if there is a possibility to chat about remoteStorage 
with you and/or other people there, then i'll definitely get one for 
both the meeting and the newcomer event! :)

can say roughly in which areas you would propose changing the draft?

Cheers,
Michiel

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 10 Jul 2013 15:30:39 +1000

On 06/07/2013, at 2:49 AM, Michiel B. de Jong <[email-masked]> wrote:

> we hadn't really formulated that until now, so we had a little discussion about it; we didn't entirely finish discussing it yet, but overall i think most of us agree that the ietf would probably be the right platform for developing the idea of remoteStorage further.
> 
> we started working on this spec as an open source project because we identified a need for one - its development so far was partially financed by donations, partially by people donating their time.
> 
> It's of course scary to give up control over something we care so much about, but we also trust the ietf would do a better job at guarding the public's best interest than our small community of quite inexperienced volunteers. :)

Great.

> it's becoming quite crystallized within its very limited defined set of features. you can get an idea of what we consider outstanding issues by browsing https://github.com/remotestorage/spec/issues our current process is to go through that list twice a year, and see which of those issues merit a text change. we are trying to change as little as possible each time, to avoid an entropy walk into feature bloat. :)

Sounds very reasonable.

> yes, we do this every 6 months anyway. what type of changes did you have in mind?
> 
> so far, even though these are themes that continuously come up as requests, we have resisted adding for instance:
> - retrieving the historic revision history of a certain document (as well as undo/rollback)
> - atomic transactions
> - etc.
> 
> we try to go as fast as we can, developing the remotestorage.js client library for it, data formats for the things we want to store on there, try to work out how to structure data into folders so that it's easy to fetch later, write apps that use it, and server implementations that are compliant with it, with (if you add up the volunteers) about 3 full-time people. that's our main constraint. :)
> 
> we have been trying to make breaking changes only twice a year, although we're probably going to have to disobey that self-imposed rule soon anyway because of a critical bug we found this week.

OK.

> currently it's a combination of https://github.com/remotestorage/spec/issues and http://remotestorage.io/community/ but we're open to changing that to the apps-discuss mailinglist for instance? do you think that would be appropriate?

I think on the community site is fine for now; if it becomes a bigger effort, or traffic is too great, you could either create a separate mailing list, or move to an IETF list. YMMV, of course.

>> Will
>> you or anyone else from remoteStorage be coming to the Berlin meeting?
> 
> i live in Berlin, and given our project's tight budget, i hadn't bought a ticket yet. but if there is a possibility to chat about remoteStorage with you and/or other people there, then i'll definitely get one for both the meeting and the newcomer event! :)

That would be great. However, it might be premature for you to register to attend, since there won't be any time scheduled to discuss the draft at a meeting (AFAIK - APPSAWG chairs?). 

So, it might be better to arrange a "Bar BoF" (i.e., informal meeting) for folks who are interested to discuss this; registration isn't necessary for that (as it often occurs in a bar or similar space). Even if that doesn't happen, I'd love to talk over a coffee.

This might also be helpful, FWIW:
  http://www.ietf.org/tao.html

> can say roughly in which areas you would propose changing the draft?

Let me get my thoughts together and write something down; sorry, it might take a while (both HTTP and moving house are keeping me busy right now).

Cheers,

---

From: "Michiel B. de Jong" <[email-masked]>
Date: Wed, 10 Jul 2013 11:57:46 +0200

On 2013-07-10 07:30, Mark Nottingham wrote:
> it might be better to arrange a "Bar BoF" (i.e., informal
> meeting) for folks who are interested to discuss this; registration
> isn't necessary for that (as it often occurs in a bar or similar
> space). Even if that doesn't happen, I'd love to talk over a coffee.

ok, great! yes, let's do that then.

> Let me get my thoughts together and write something down; sorry, it
> might take a while (both HTTP and moving house are keeping me busy
> right now).

awesome, thanks! :)

Cheers,
Michiel

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 10 Jul 2013 07:03:19 -0700

> ok, great! yes, let's do that then.

When you decide a time and place for the informal meeting, by all means send a message to this list; others here would be interested as well.

--Paul Hoffman

---

From: "Michiel B. de Jong" <[email-masked]>
Date: Tue, 16 Jul 2013 16:54:48 +0200

On 2013-07-10 16:03, Paul Hoffman wrote:
> When you decide a time and place for the informal meeting, by all
> means send a message to this list; others here would be interested as
> well.
>
> --Paul Hoffman

yes! I propose the following, let me know if you think this will work:

On the Monday evening, directly after the technical plenary finishes 
(which should be around 19:40 
https://datatracker.ietf.org/meeting/87/agenda.txt ), I will hold up a 
sign reading "non-proprietary sync Bar BoF", so we can gather. Then 
among the people who show up, we decide to either do it there in the 
foyer, or go for some food, or whatever the group decides. If someone 
has trouble locating us, they can phone me on +49-1577.6858.499.

The proposed topic of this Bar BoF will be "non-prioprietary cloud 
sync", so something like Dropbox, GoogleDrive, SkyDrive, etcetera, but 
as an open standard. One work-in-progress for this would be 
remoteStorage, by François Kooman and myself, in this internet draft: 
http://www.ietf.org/id/draft-dejong-remotestorage-01.txt.

cheers,
Michiel

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 16 Jul 2013 11:36:30 -0400

I can't be there and I can't participate right now due to a court case I am
involved in. There is relevant IP that is being asserted but the priority
date is 11th April 1995. So I think this is a good time to start looking at
an open standard.

It seems to me that the question everyone will ask is 'how is this
different from a network file store'.

The answer in my view is that a network file store supports file locking
and incremental updates which are very complicated to implement. This is
not what most people want or need in a backup or file replication system.

People might be interested in the tool I am currently working on, the
Protocol Designer's Workbench which is a set of tools that allow a
developer to specify a protocol in an abstract schema language that is not
tied to any given encoding which can then be used to generate
documentation, examples and production code for a low volume client and
server.

At present the tool generates code in C# that uses JSON for encoding of
protocol messages and HTTP for transport. The next version will support C
as an option and at least one binary encoding of JSON. Support for
conformance testing and high volume production services will be added over
time.

The big advantage to using a tool is that the documentation is always
locked to the specification and the code. So instead of someone making a
change to the document but not the spec and people discovering the issue 6
months down the line, everything is always in sync.

On Tue, Jul 16, 2013 at 10:54 AM, Michiel B. de Jong <
[email-masked]> wrote:

> (which should be around 19:40 https://datatracker.ietf.org/**
> meeting/87/agenda.txt <https://datatracker.ietf.org/meeting/87/agenda.txt>), I will hold up a sign reading "non-proprietary sync Bar BoF", so we can
> gather. Then among the people who show up, we decide to either do it there
> in the foyer, or go for some food, or whatever the group decides. If
> someone has trouble locating us, they can phone me on +49-1577.6858.499.
>
> The proposed topic of this Bar BoF will be "non-prioprietary cloud sync",
> so something like Dropbox, GoogleDrive, SkyDrive, etcetera, but as an open
> standard. One work-in-progress for this would be remoteStorage, by François
> Kooman and myself, in this internet draft: http://www.ietf.org/id/draft-**
> dejong-remotestorage-01.txt<http://www.ietf.org/id/draft-dejong-remotestorage-01.txt>
> .
>
>
> cheers,
> Michiel
>
> ______________________________**_________________
> https://www.ietf.org/mailman/**listinfo/apps-discuss<https://www.ietf.org/mailman/listinfo/apps-discuss>
>

-- 
Website: http://hallambaker.com/

---

From: Melvin Carvalho <[email-masked]>
Date: Thu, 25 Jul 2013 19:01:09 +0200

On 10 July 2013 07:30, Mark Nottingham <[email-masked]> wrote:

> overall i think most of us agree that the ietf would probably be the right
> platform for developing the idea of remoteStorage further.
> >
> identified a need for one - its development so far was partially financed
> by donations, partially by people donating their time.
> >
> browsing https://github.com/remotestorage/spec/issues our current process
> is to go through that list twice a year, and see which of those issues
> merit a text change. we are trying to change as little as possible each
> time, to avoid an entropy walk into feature bloat. :)
>
> Sounds very reasonable.
>
>
> library for it, data formats for the things we want to store on there, try
> to work out how to structure data into folders so that it's easy to fetch
> later, write apps that use it, and server implementations that are
> > we have been trying to make breaking changes only twice a year, although
> we're probably going to have to disobey that self-imposed rule soon anyway
> because of a critical bug we found this week.
>
> OK.
>
> http://remotestorage.io/community/ but we're open to changing that to the
> apps-discuss mailinglist for instance? do you think that would be
> appropriate?
>
> I think on the community site is fine for now; if it becomes a bigger
> effort, or traffic is too great, you could either create a separate mailing
> list, or move to an IETF list. YMMV, of course.
>
>
> >> Will
> with you and/or other people there, then i'll definitely get one for both
> the meeting and the newcomer event! :)
>
> That would be great. However, it might be premature for you to register to
> attend, since there won't be any time scheduled to discuss the draft at a
> meeting (AFAIK - APPSAWG chairs?).
>
> So, it might be better to arrange a "Bar BoF" (i.e., informal meeting) for
> folks who are interested to discuss this; registration isn't necessary for
> that (as it often occurs in a bar or similar space). Even if that doesn't
> happen, I'd love to talk over a coffee.
>
> Let me get my thoughts together and write something down; sorry, it might
> take a while (both HTTP and moving house are keeping me busy right now).
>

FYI: here's a comparison of two of the cloud storage read/write
technologies developed at the W3C.  Linked Data Platform is a working draft
at last call, Graph Store Platform is a W3C REC.

https://docs.google.com/document/d/1lofZv855H5D3JDSefpEiE3NBM30YBh2W4pkmAGvlLtM/edit#heading=h.utn64k9xpepq

Perhaps an idea to clone this document and add a third column for remote
storage so that people can compare.  Perhaps the main difference here is
the versioning strategy ...

---

From: "Michiel B. de Jong" <[email-masked]>
Date: Fri, 26 Jul 2013 16:44:56 +0200

Just to re-confirm and as a reminder, this will happen Monday after the 
technical plenary in Berlin:

> http://www.ietf.org/id/draft-dejong-remotestorage-01.txt.

Hope to see you there!

Cheers,
Michiel

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 26 Jul 2013 11:23:41 -0400

This type of work will require two sets of code.

The easy part is the code implementing the actual protocol that the client
uses to talk to the server.

The harder part will be code to make the remote source visible as a local
drive in e.g. Windows and Unix etc. and package it in a way that makes it
robust.

It occurs to me that some of the necessary functionality might already be
present in GIT etc. A source code manager is simply a remote filestore that
supports file versioning and keeps all the old version numbers and allows
different users to manage different views of the same tree.

Another use case that would enable quite a few IETF types to spend time on
this would be to support gathering digital evidence. It would be very
useful for me to have a drive where I could copy files and cause them to be
automatically signed and digitally notarized.

I think the first requirement is actually a core requirement.

The second is probably not core requirements but could be supported if the
core protocol had some sort of capability that allowed file attributes to
be added so as to support the 'directory views' that the first implies.

On Fri, Jul 26, 2013 at 10:44 AM, Michiel B. de Jong <
[email-masked]> wrote:

>> https://datatracker.ietf.org/**meeting/87/agenda.txt<https://datatracker.ietf.org/meeting/87/agenda.txt>), I will hold up a
>> http://www.ietf.org/id/draft-**dejong-remotestorage-01.txt<http://www.ietf.org/id/draft-dejong-remotestorage-01.txt>
>> .
>>
>
> Hope to see you there!
>
>
>
> Cheers,
> Michiel

---

From: "Michiel B. de Jong" <[email-masked]>
Date: Mon, 29 Jul 2013 18:56:47 +0200

To be more specific, I will be standing in the Potsdam Foyer (the 
snacks area outside the big Potsdam 1/3 room) at 19:40, holding up a 
hand-written banner that should be pretty hard to miss.

I will be wearing the same "unhosted.org" hoodie as in this picture:

https://michielbdejong.com/me.jpg (although I will not ride the same 
camel) ;)

On 2013-07-26 16:44, Michiel B. de Jong wrote:
> Just to re-confirm and as a reminder, this will happen Monday after
> the technical plenary in Berlin:
>