Subject: Re: [Storagesync] Storagesync Digest, Vol 5, Issue 1

From: Hugo González Labrador <[email-masked]>
Date: Tue, 01 Dec 2015 22:14:05 +0100

Hi,

from my point of view the remoteStorage project addresses a subset of
the use cases of the  WebDAV specification.

The main difference I can observe is that the specification is built on
REST instead of XML-based communication.

I personally like much more REST than WebDAV because it is more
developer friendly and it is faster to develop.
 Maybe the remoteStorage API becomes the next WebDAV :)

However, the remoteStorage API does not provide a way of synchronising
data, this task is delegated to the developers.
Is there a plan to provide such feature based on the outcome of this
draft?

BTW, I want to introduce ClawIO (http://clawio.github.io), a research
project to benchmark different synchronisation protocols against
different data backends with special attention to provide a common sync
API.

A common API for different sync protocols is being created based on the
architecture specified in this draft (control and data servers) and the
one from the CERN EOS project (management, disk and queue servers).

The Phase I has implemented the ownCloud Sync Protocol and Phase II will
implement the SeaFile Sync Protocol.
The choice of these protocols among others is because they are really
opposed to each other in terms of syncing (delta vs non-delta,
state-based vs log/event/git-based sync …), so finding a common approach
is more challenging.

Providing a base specification/architecture to measure the feasibility
of this draft is one of the objectives of the project.

I believe that the work being done here and in ClawIO are supplementary
to each other and I think mutual collaboration could be beneficial for
both sides.

Also, if there is interest, the remoteStorage API can be added to
ClawIO.

Best regards,

Hugo Gonzalez Labrador

On Tue, Dec 1, 2015, at 09:00 PM, [email-masked] wrote:
> Send Storagesync mailing list submissions to
> 	[email-masked]
> 
> To subscribe or unsubscribe via the World Wide Web, visit
> 	https://www.ietf.org/mailman/listinfo/storagesync
> or, via email, send a message with subject or body 'help' to
> 	[email-masked]
> 
> You can reach the person managing the list at
> 	[email-masked]
> 
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Storagesync digest..."
> Today's Topics:
> 
>    1. New version of draft-dejong-remotestorage    Internet-Draft
>       available (Michiel de Jong)
>    2. Re: New version of draft-dejong-remotestorage Internet-Draft
>       available (Gihan Dias)
>    3. Re: New version of draft-dejong-remotestorage Internet-Draft
>       available (Fei Song)
> _______________________________________________
> Storagesync mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/storagesync
> Email had 3 attachments:
> + [Storagesync] New version of draft-dejong-remotestorage      
> Internet-Draft available
>   2k (message/rfc822)
> + Re: [Storagesync] New version of draft-dejong-remotestorage
> Internet-Draft available
>   1k (message/rfc822)
> + Re: [Storagesync] New version of draft-dejong-remotestorage
> Internet-Draft available
>   2k (message/rfc822)

---

From: Linhui Sun <[email-masked]>
Date: Wed, 2 Dec 2015 15:20:30 +0800

Hi

> Is there a plan to provide such feature based on the outcome of this
I'm a little bit confused why you say the remoteStorage does not provide
that. Is that because remoteStorage does not perform like a typical sync
services (e.g. dropbox...) or you are saying something else?

> architecture specified in this draft (control and data servers) and the
I cannot find a distributed architecture in this draft. It seems that they
handle metadata and content data together, just like a normal web service.

Regards,
Linhui

---

From: "Fei Song" <[email-masked]>
Date: Wed, 2 Dec 2015 15:59:43 +0800

Hi all

--------------
Fei Song
>handle metadata and content data together, just like a normal web service.

Is it another draft related with ClawIO? Not remoteStorage?

---

From: Hugo González Labrador <[email-masked]>
Date: Wed, 02 Dec 2015 10:56:14 +0100

draft?
> I'm a little bit confused why you say the remoteStorage does not
> provide that. Is that because remoteStorage does not perform like
> a typical sync services (e.g. dropbox...) or you are saying
> something else?
Yes, because it does not offer synchronisation capabilities.
API.
> I cannot find a distributed architecture in this draft. It seems that
> they handle metadata and content data together, just like a normal web
> service.

ClawIO is fully distributed. Every logical unit is a different server
than be scaled out. Data and Metadata channels are independent from each
other and reside on different servers.

> Regards, Linhui
is more challenging.
both sides.
ClawIO.
Best regards,
Hugo Gonzalez Labrador
Storagesync mailing list

---

From: Linhui Sun <[email-masked]>
Date: Wed, 02 Dec 2015 18:18:49 +0800

Hi

On 周三, 12月 2, 2015 at 17:56, Hugo González Labrador <[email-masked]>
wrote:
On Wed, Dec 2, 2015, at 08:20 AM, Linhui Sun wrote:
Hi
2015-12-02 5:14 GMT+08:00 Hugo González Labrador < [email-masked] [[email-masked]] > :
Hi,
from my point of view the remoteStorage project addresses a subset of
the use cases of the WebDAV specification.
The main difference I can observe is that the specification is built on
REST instead of XML-based communication.
I personally like much more REST than WebDAV because it is more
developer friendly and it is faster to develop.
Maybe the remoteStorage API becomes the next WebDAV :)
However, the remoteStorage API does not provide a way of synchronising
data, this task is delegated to the developers.
Is there a plan to provide such feature based on the outcome of this
draft?
I'm a little bit confused why you say the remoteStorage does not provide that.
Is that because remoteStorage does not perform like a typical sync services
(e.g. dropbox...) or you are saying something else?
Yes, because it does not offer synchronisation capabilities. Got it. And What I am wondering is that do we need to include those
capabilities in a base specifications. Since it is hard to standardize the
capabilities like dedup or delta. Maybe a better way is to define those in a
separate specification, something like extensions. For a base document, we just
need to define how to perform sync operations. BTW, I want to introduce ClawIO ( [http://clawio.github.io] http://clawio.github.io [http://clawio.github.io] ), a research
project to benchmark different synchronisation protocols against
different data backends with special attention to provide a common sync
API.
A common API for different sync protocols is being created based on the
architecture specified in this draft (control and data servers) and the
I cannot find a distributed architecture in this draft. It seems that they
handle metadata and content data together, just like a normal web service.
ClawIO is fully distributed. Every logical unit is a different server than be
scaled out. Data and Metadata channels are independent from each other and
reside on different servers. That is widely employed in popular sync services. And that is also beneficial
to privacy to some extent. But in the context of sync in browser (which is
mainly considered in the remoteStorage), I don't know whether this is
reasonable. But I think we should deploy distributed architecture although it
will make things complicated.
Regards, Linhui
Regards,
Linhui
one from the CERN EOS project (management, disk and queue servers).
The Phase I has implemented the ownCloud Sync Protocol and Phase II will
implement the SeaFile Sync Protocol.
The choice of these protocols among others is because they are really
opposed to each other in terms of syncing (delta vs non-delta,
state-based vs log/event/git-based sync …), so finding a common approach
is more challenging.
Providing a base specification/architecture to measure the feasibility
of this draft is one of the objectives of the project.
I believe that the work being done here and in ClawIO are supplementary
to each other and I think mutual collaboration could be beneficial for
both sides.
Also, if there is interest, the remoteStorage API can be added to
ClawIO.
Best regards,
Hugo Gonzalez Labrador
On Tue, Dec 1, 2015, at 09:00 PM, [email-masked] [[email-masked]] wrote:
> [email-masked] [[email-masked]]
>
> [https://www.ietf.org/mailman/listinfo/storagesync] https://www.ietf.org/mailman/listinfo/storagesync
[https://www.ietf.org/mailman/listinfo/storagesync]
> [email-masked] [[email-masked]]
>
> [email-masked] [[email-masked]]
>
> [email-masked] [[email-masked]]
> [https://www.ietf.org/mailman/listinfo/storagesync] https://www.ietf.org/mailman/listinfo/storagesync
[https://www.ietf.org/mailman/listinfo/storagesync]
_______________________________________________
Storagesync mailing list
[email-masked] [[email-masked]]
[https://www.ietf.org/mailman/listinfo/storagesync] https://www.ietf.org/mailman/listinfo/storagesync
[https://www.ietf.org/mailman/listinfo/storagesync]

---

From: Hugo González Labrador <[email-masked]>
Date: Wed, 02 Dec 2015 11:28:48 +0100

On Wed, Dec 2, 2015, at 11:18 AM, Linhui Sun wrote:
> Hi
>
> On 周三, 12月 2, 2015 at 17:56,  Hugo González Labrador
> <[email-masked]> wrote:
>>
>>
>>
>>>> from my point of view the remoteStorage project addresses a subset
>>>> of the use cases of the  WebDAV specification.
>>>>
>>>> The main difference I can observe is that the specification is
>>>> built on REST instead of XML-based communication.
>>>>
>>>> developer friendly and it is faster to develop. Maybe the
>>>> remoteStorage API becomes the next WebDAV :)
>>>>
>>>> However, the remoteStorage API does not provide a way of
>>>> synchronising data, this task is delegated to the developers. Is
>>>> there a plan to provide such feature based on the outcome of this
>>>> draft?
> Got it. And What I am wondering is that do we need to include those
> capabilities in a base specifications. Since it is hard to standardize
> the capabilities like dedup or delta. Maybe a better way is to define
> those in a separate specification, something like extensions. For a
> base document, we just need to define how to perform sync operations.

Yes, I agree that may be  an extension of this draft could handle the
synchronisation part.

>>
>>>>
>>>> BTW, I want to introduce ClawIO (http://clawio.github.io), a
>>>> research project to benchmark different synchronisation protocols
>>>> against different data backends with special attention to provide a
>>>> common sync API.
>>>>
>>>> A common API for different sync protocols is being created based on
>>>> the architecture specified in this draft (control and data servers)
>>>> and the
>>> I cannot find a distributed architecture in this draft. It seems
>>> that they handle metadata and content data together, just like a
>>> normal web service.
>>
>> than be scaled out. Data and Metadata channels are independent from
>> each other and reside on different servers.
> That is widely employed in popular sync services. And that is also
> beneficial to privacy to some extent. But in the context of sync in
> browser (which is mainly considered in the remoteStorage), I don't
> know whether this is reasonable. But I think we should deploy
> distributed architecture although it will make things complicated.

Of course, the remoteStorage is targeted to browsers, so syncing does
not make too much sense in this case. With the rise of Linux container
micro-service based architectures, the deployment of  such highly
complex systems should become easier and faster.

Best,

Hugo

>>>> The Phase I has implemented the ownCloud Sync Protocol and Phase II
>>>> will implement the SeaFile Sync Protocol. The choice of these
>>>> protocols among others is because they are really opposed to each
>>>> other in terms of syncing (delta vs non-delta, state-based vs log/event/git-
>>>> based sync …), so finding a common approach is more challenging.
>>>>
>>>> Providing a base specification/architecture to measure the
>>>> feasibility of this draft is one of the objectives of the project.
>>>>
>>>> I believe that the work being done here and in ClawIO are
>>>> supplementary to each other and I think mutual collaboration could
>>>> be beneficial for both sides.
>>>>
>>>> > https://www.ietf.org/mailman/listinfo/storagesync or, via email,
>>>> > send a message with subject or body 'help' to       storagesync-
>>>> > [email-masked]
>>>> >
>>>> > You can reach the person managing the list at       storagesync-
>>>> > [email-masked]
>>>> >
>>>> > When replying, please edit your Subject line so it is more
>>>> > specific than "Re: Contents of Storagesync digest..." Today's
>>>> > Topics:
>>>> >
>>>> >available (Michiel de Jong)    2. Re: New version of draft-dejong-
>>>> >remotestorage Internet-Draft       available (Gihan Dias)    3.
>>>> >Re: New version of draft-dejong-remotestorage Internet-Draft
>>>> >available (Fei Song)
>>>> > Storagesync mailing list [email-masked]
>>>> > https://www.ietf.org/mailman/listinfo/storagesync Email had 3
>>>> > attachments:
>>>> > + [Storagesync] New version of draft-dejong-remotestorage Internet-
>>>> >   Draft available   2k (message/rfc822)
>>>> > + Re: [Storagesync] New version of draft-dejong-remotestorage Internet-
>>>> >   Draft available   1k (message/rfc822)
>>>> > + Re: [Storagesync] New version of draft-dejong-remotestorage Internet-
>>>> >   Draft available   2k (message/rfc822)
>>>>
>>>> Storagesync mailing list [email-masked]

---

From: Michiel de Jong <[email-masked]>
Date: Wed, 2 Dec 2015 13:30:46 +0100

Hi all!

Thanks for all your reactions to the remoteStorage Internet-Draft.

We get the question about WebDAV a lot, in the next version we should add a
remark about it in the intro. The folder descriptions returned when you GET
a URL that ends with a / are indeed a deviation from the XML returned by
WebDAV server, and this is just because nowadays JSON is easier to use than
XML for developers, both client-side and server-side.

The fact that we don't require servers to support WebDAV's custom verbs
like PROPPATCH etc. is for three reasons:
1) it's a lot of work to implement this without using an existing WebDAV
library
2) in practice, a lot of WebDAV servers get it wrong, or don't implement
all of WebDAV. It's very annoying for client implementers to have to deal
with servers that e.g. chose not to implement LOCK and UNLOCK.
3) we don't really need all these advanced features on top of standard
HTTP, just supporting GET/PUT/DELETE for resources, and adding a simple
folder description format, is enough for most applications.

Other than that, the remoteStorage Internet-Draft specifies a *lot* more
than just how each HTTP verb should behave:
* requiring support for OAuth implicit-grant flow
* requiring ETag support and nested versioning (i.e. the folder's ETag
changes if anything within that folder changes)
* requiring CORS headers
* requiring a WebFinger announcement for service discovery

It would be easy to add these three things on top of WebDAV, instead of
putting them on top of our minimal GET/PUT/DELETE API definition. In fact,
we could probably separate it into two Internet-Drafts: one for the
'RESTful folders' API which is our simplification of WebDAV, and a second
one for OAuth/ETags/CORS/WebFinger on top of either WebDAV or 'RESTful
folders' or whatever other HTTP-based API you want.

On Wed, Dec 2, 2015 at 11:28 AM, Hugo González Labrador <
[email-masked]> wrote:

> capabilities like dedup or delta. Maybe a better way is to define those in
> a separate specification,
>
>
Thanks for giving these examples - so by 'synchronisation capabilities' you
mean 1) deduplication and 2) delta updates? Anything else or is that an
exhaustive list?

> something like extensions. For a base document, we just need to define how
> to perform sync operations.
>
>

Our Internet-Draft is heavily focused on the world wide web, whose URLs are
not content-addressable, we can't change that. So that architecture is not
very friendly to deduplication, compared to for instance BitTorrent. As you
already said, developers can still dedupe on the server-side when storing
blobs to disk, and can also dedupe on the client side before creating the
resources the client uploads.

As far as I know, delta updates are not supported by the ETag system - you
cannot do a range request to find out if certain bytes within a document
have changed. However, the folder system we define does encourage you, for
instance when you develop a To Do List app, put each task on its own
document, and then query the folder to see which of them changed, instead
of putting them all in one big document and retrieving the whole document
each time.

Cheers,
Michiel.

>
>
> BTW, I want to introduce ClawIO ( <http://clawio.github.io>
> http://clawio.github.io), a research
> ClawIO is fully distributed. Every logical unit is a different server than
> be scaled out. Data and Metadata channels are independent from each other
> and reside on different servers.
>
> beneficial to privacy to some extent. But in the context of sync in browser
> (which is mainly considered in the remoteStorage), I don't know whether
> this is reasonable. But I think we should deploy distributed architecture
> although it will make things complicated.
>
>
> Of course, the remoteStorage is targeted to browsers, so syncing does not
> make too much sense in this case.
> With the rise of Linux container micro-service based architectures, the
> deployment of  such highly complex systems should become easier and faster.
>
> Best,
>
> Hugo
>
>
> Regards,
> Linhui
>
>
>
>
> Regards,
> Linhui
>
> >        <https://www.ietf.org/mailman/listinfo/storagesync>
> > <https://www.ietf.org/mailman/listinfo/storagesync>
> <https://www.ietf.org/mailman/listinfo/storagesync>

---

From: Hugo González Labrador <[email-masked]>
Date: Wed, 02 Dec 2015 13:43:38 +0100

On Wed, Dec 2, 2015, at 01:30 PM, Michiel de Jong wrote:
> Hi all! Thanks for all your reactions to the remoteStorage Internet-
> Draft.
>
> We get the question about WebDAV a lot, in the next version we
> should add a remark about it in the intro. The folder descriptions
> returned when you GET a URL that ends with a / are indeed a
> deviation from the XML returned by WebDAV server, and this is just
> because nowadays JSON is easier to use than XML for developers, both
> client-side and server-side.

I totally agree here, this was going to happen soon or later and it is
really appreciated.

> The fact that we don't require servers to support WebDAV's custom
> verbs like PROPPATCH etc. is for three reasons:
> 1) it's a lot of work to implement this without using an existing
>    WebDAV library
> 2) in practice, a lot of WebDAV servers get it wrong, or don't
>    implement all of WebDAV. It's very annoying for client implementers
>    to have to deal with servers that e.g. chose not to implement LOCK
>    and UNLOCK.
>    HTTP, just supporting GET/PUT/DELETE for resources, and adding a
>    simple folder description format, is enough for most applications.
>
> Other than that, the remoteStorage Internet-Draft specifies a *lot*
> more than just how each HTTP verb should behave:
> * requiring a WebFinger announcement for service discovery It would be
>   easy to add these three things on top of WebDAV, instead of putting
>   them on top of our minimal GET/PUT/DELETE API definition. In fact,
>   'RESTful folders' API which is our simplification of WebDAV, and a
>   second one for OAuth/ETags/CORS/WebFinger on top of either WebDAV or
>   'RESTful folders' or whatever other HTTP-based API you want.

There is one requirement that all synchronisers need to handle: moving
resources. In this spec the alternative of a WebDAV MOVE is not
specified.

It is true that a MOVE could be replaced with a DELETE + UPLOAD but that
is not acceptable in most cases due to the inefficiency of such
operation (cpu, bandwidth consumption ...)

Is there a plan to support such basic feature?

>From the current remoteStorage spec, the ownCloud sync protocol can be
implemented. The missing bit is tracking those remote moves.

Cheers

> On Wed, Dec 2, 2015 at 11:28 AM, Hugo González Labrador
>> __
>>>>>> from my point of view the remoteStorage project addresses a
>>>>>> subset of the use cases of the  WebDAV specification.
>>>>>>
>>> capabilities in a base specifications. Since it is hard to
>>> standardize the capabilities like dedup or delta. Maybe a better way
>>> is to define those in a separate specification,
>>
>
> Thanks for giving these examples - so by
'synchronisation capabilities' you mean 1) deduplication and 2) delta
updates? Anything else or is that an exhaustive list?
>>
>>> something like extensions. For a base document, we just need to
>>> define how to perform sync operations.
>>
>>
> Our Internet-Draft is heavily focused on the world wide web, whose
> URLs are not content-addressable, we can't change that. So that
> architecture is not very friendly to deduplication, compared to for
> instance BitTorrent. As you already said, developers can still dedupe
> on the server-side when storing blobs to disk, and can also dedupe on
> the client side before creating the resources the client uploads. As
> far as I know, delta updates are not supported by the ETag system -
> you cannot do a range request to find out if certain bytes within a
> document have changed. However, the folder system we define does
> encourage you, for instance when you develop a To Do List app, put
> each task on its own document, and then query the folder to see which
> of them changed, instead of putting them all in one big document and
> retrieving the whole document each time.
>
> Cheers, Michiel.
>
>>
>>>>
>>>>>>
>>>>>> against different data backends with special attention to provide
>>>>>> a common sync API.
>>>>>>
>>>>>> A common API for different sync protocols is being created based
>>>>>> on the architecture specified in this draft (control and data
>>>>>> servers) and the
>>>> ClawIO is fully distributed. Every logical unit is a different
>>>> server than be scaled out. Data and Metadata channels are
>>>> independent from each other and reside on different servers.
>> not make too much sense in this case. With the rise of Linux
>> container micro-service based architectures, the deployment of  such
>> highly complex systems should become easier and faster.
>>
>> Best,
>>
>>
>> Hugo
>>
>>
>>> Regards, Linhui
>>>>
>>>>
>>>>> Regards, Linhui
>>>>>> The Phase I has implemented the ownCloud Sync Protocol and Phase
>>>>>> II will implement the SeaFile Sync Protocol. The choice of these
>>>>>> supplementary to each other and I think mutual collaboration
>>>>>> could be beneficial for both sides.
>>>>>>
>>>>>> > https://www.ietf.org/mailman/listinfo/storagesync or, via
>>>>>> > email, send a message with subject or body 'help' to       storagesync-
>>>>>> > [email-masked]
>>>>>> >
>>>>>> >   Internet-Draft available   1k (message/rfc822)
>>>>>> >   Internet-Draft available   2k (message/rfc822)
>>>>>>

---

From: Linhui Sun <[email-masked]>
Date: Wed, 2 Dec 2015 20:59:10 +0800

2015-12-02 20:43 GMT+08:00 Hugo González Labrador <[email-masked]>:

> We get the question about WebDAV a lot, in the next version we should add
> a remark about it in the intro. The folder descriptions returned when you
> GET a URL that ends with a / are indeed a deviation from the XML returned
> by WebDAV server, and this is just because nowadays JSON is easier to use
> than XML for developers, both client-side and server-side.
>
>
> resources. In this spec the alternative of a WebDAV MOVE is not specified.
>
> is not acceptable in most cases due to the inefficiency of such operation
> (cpu, bandwidth consumption ...)
>
I agree with Hugo that MOVE is useful, sometimes if you just rename a file
it will be perfect. But the question I have is that whether we need to make
two documents? Multiple choices is not good for standardization. In my
view, webdav is something that we need to have a very clear decision on
whether to consider it or not.

Regards,
Linhui

> Thanks for giving these examples - so by 'synchronisation capabilities'
> you mean 1) deduplication and 2) delta updates? Anything else or is that an
> exhaustive list?
>
>
>
> Our Internet-Draft is heavily focused on the world wide web, whose URLs
> are not content-addressable, we can't change that. So that architecture is
> not very friendly to deduplication, compared to for instance BitTorrent. As
> you already said, developers can still dedupe on the server-side when
> storing blobs to disk, and can also dedupe on the client side before
> creating the resources the client uploads.

---

From: Hugo González Labrador <[email-masked]>
Date: Wed, 02 Dec 2015 14:03:37 +0100

I propose to come up with a list of advantages and disadvantages of
using WebDAV and compare them against a JSON/REST based approach, like
remoteStorage.

Does it sound good ?

On Wed, Dec 2, 2015, at 01:59 PM, Linhui Sun wrote:
>> I totally agree here, this was going to happen soon or later and it
>> is really appreciated.
>>
>>
>>>    implement all of WebDAV. It's very annoying for client
>>>    implementers to have to deal with servers that e.g. chose not to
>>>    implement LOCK and UNLOCK.
>>> 3) we don't really need all these advanced features on top of
>>>    standard HTTP, just supporting GET/PUT/DELETE for resources, and
>>>    adding a simple folder description format, is enough for most
>>>    applications.
>>>
>>> * requiring ETag support and nested versioning (i.e. the folder's
>>>   ETag changes if anything within that folder changes)
>>> * requiring CORS headers
>>> * requiring a WebFinger announcement for service discovery It would
>>>   be easy to add these three things on top of WebDAV, instead of
>>>   putting them on top of our minimal GET/PUT/DELETE API definition.
>>>   In fact, we could probably separate it into two Internet-Drafts:
>>>   one for the 'RESTful folders' API which is our simplification of
>>>   WebDAV, and a second one for OAuth/ETags/CORS/WebFinger on top of
>>>   either WebDAV or 'RESTful folders' or whatever other HTTP-based
>>>   API you want.
>>
>>
>> There is one requirement that all synchronisers need to handle:
>> moving resources. In this spec the alternative of a WebDAV MOVE is
>> not specified.
>>
>> It is true that a MOVE could be replaced with a DELETE + UPLOAD but
>> that is not acceptable in most cases due to the inefficiency of such
>> From the current remoteStorage spec, the ownCloud sync protocol can
>> be implemented. The missing bit is tracking those remote moves.
> I agree with Hugo that MOVE is useful, sometimes if you just rename a
> file it will be perfect. But the question I have is that whether we
> need to make two documents? Multiple choices is not good for
> standardization. In my view, webdav is something that we need to have
> a very clear decision on whether to consider it or not.
>
> Regards, Linhui
>>
>>
>> Cheers
>>
>>>>>>>> synchronising data, this task is delegated to the developers.
>>>>>>>> Is there a plan to provide such feature based on the outcome of
>>>>>>>> this draft?
>>>>>>> provide that. Is that because remoteStorage does not perform
>>>>>>> like a typical sync services (e.g. dropbox...) or you are saying
>>>>>>> something else?
>>>>> Got it. And What I am wondering is that do we need to include
>>>>> those capabilities in a base specifications. Since it is hard to
>>>>> standardize the capabilities like dedup or delta. Maybe a better
>>>>> way is to define those in a separate specification,
>>>>
>>>
>>>> Yes, I agree that may be  an extension of this draft could handle
>>>> the synchronisation part.
>>>>
>>>>
>>>>
>>>
>>> instance BitTorrent. As you already said, developers can still
>>> dedupe on the server-side when storing blobs to disk, and can also
>>> dedupe on the client side before creating the resources the client
>>> uploads. As far as I know, delta updates are not supported by the
>>> ETag system - you cannot do a range request to find out if certain
>>> bytes within a document have changed. However, the folder system we
>>> define does encourage you, for instance when you develop a To Do
>>> List app, put each task on its own document, and then query the
>>> folder to see which of them changed, instead of putting them all in
>>> one big document and retrieving the whole document each time.
>>>
>>> Cheers, Michiel.
>>>
>>>>
>>>>>>
>>>>>>>>
>>>>>>>> research project to benchmark different synchronisation
>>>>>>>> protocols against different data backends with special
>>>>>>>> attention to provide a common sync API.
>>>>>>>>
>>>>>>>> A common API for different sync protocols is being created
>>>>>>>> based on the architecture specified in this draft (control and
>>>>>>>> data servers) and the
>>>>> beneficial to privacy to some extent. But in the context of sync
>>>>> in browser (which is mainly considered in the remoteStorage), I
>>>>> don't know whether this is reasonable. But I think we should
>>>>> deploy distributed architecture although it will make things
>>>>> complicated.
>>>>
>>>>
>>>> Of course, the remoteStorage is targeted to browsers, so syncing
>>>> does not make too much sense in this case. With the rise of Linux
>>>> container micro-service based architectures, the deployment of
>>>> such highly complex systems should become easier and faster.
>>>>
>>>> Best,
>>>>
>>>>
>>>> Hugo
>>>>
>>>>
>>>>> Regards, Linhui
>>>>>>
>>>>>>
>>>>>>> Regards, Linhui
>>>>>>>> The Phase I has implemented the ownCloud Sync Protocol and
>>>>>>>> Phase II will implement the SeaFile Sync Protocol. The choice
>>>>>>>> of these protocols among others is because they are really
>>>>>>>> state-based vs log/event/git-based sync …), so finding a common
>>>>>>>> approach is more challenging.
>>>>>>>>
>>>>>>>> Also, if there is interest, the remoteStorage API can be added
>>>>>>>> to ClawIO.
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>>
>>>>>>>> Hugo Gonzalez Labrador
>>>>>>>>
>>>>>>>> > email, send a message with subject or body 'help' to
>>>>>>>> >remotestorage Internet-Draft       available (Gihan Dias)
>>>>>>>> >3. Re: New version of draft-dejong-remotestorage Internet-
>>>>>>>> >Draft       available (Fei Song)

---

From: Linhui Sun <[email-masked]>
Date: Wed, 2 Dec 2015 21:32:33 +0800

That's cool for me, a separate section for this might make sense.

Another thing is do we need to include an application-layer chunking here
(not just for a browser sync), since if we want to further extend other
capabilities it is essential.

Regards,
Linhui

2015-12-02 21:03 GMT+08:00 Hugo González Labrador <[email-masked]>:

> I propose to come up with a list of advantages and disadvantages of using
> WebDAV and compare them against a JSON/REST based approach, like
> remoteStorage.
>
> Does it sound good ?
>
>

---

From: Michiel de Jong <[email-masked]>
Date: Wed, 2 Dec 2015 15:05:24 +0100

Cool! I created https://github.com/remotestorage/spec/issues/137 about the
need for  MOVE verb.

Application-level chunking is partially supported by HTTP itself through
`Content-Range` headers (although it's not clear whether these are allowed
on PUT requests as well as on GET requests, see
https://github.com/remotestorage/spec/issues/131). The problem is that HTTP
defines versioning at the document level, you cannot ask a server to
produce or check an ETag for a specific byte-range of a document, only for
the entire document.

A comparison document sounds good! See also
http://www.servicedenuages.fr/en/generic-storage-ecosystem.

Cheers,
Michiel.

On Wed, Dec 2, 2015 at 2:32 PM, Linhui Sun <[email-masked]> wrote:

>> file it will be perfect. But the question I have is that whether we need to
>> make two documents? Multiple choices is not good for standardization. In my
>> On 周三, 12月 2, 2015 at 17:56, Hugo González Labrador <
>> something like extensions. For a base document, we just need to define
>> how to perform sync operations.
>>
>>
>>
>> As far as I know, delta updates are not supported by the ETag system -
>> document have changed. However, the folder system we define does encourage
>> you, for instance when you develop a To Do List app, put each task on its
>> own document, and then query the folder to see which of them changed,
>> instead of putting them all in one big document and retrieving the whole
>> document each time.
>>
>> Cheers,
>> Michiel.
>>
>>
>>
>>
>>
>>
>>

---

From: Linhui Sun <[email-masked]>
Date: Wed, 2 Dec 2015 22:29:56 +0800

Hi

2015-12-02 22:05 GMT+08:00 Michiel de Jong <[email-masked]>:

> Cool! I created https://github.com/remotestorage/spec/issues/137 about
> the need for  MOVE verb.
>
> https://github.com/remotestorage/spec/issues/131). The problem is that
> HTTP defines versioning at the document level, you cannot ask a server to
Actually what I'm saying is a chunking before transmitting (using http), in
this way, they are treated as individual documents from the standpoint of
http. But I don't know whether this is appropriate for remoteStorage, just
a comment.

Regards,
Linhui

>>> 2015-12-02 20:43 GMT+08:00 Hugo González Labrador <[email-masked]
>>> >:
>>>
>>>
>>>
>>>
>>>
>>>
>>> We get the question about WebDAV a lot, in the next version we should
>>> add a remark about it in the intro. The folder descriptions returned when
>>> you GET a URL that ends with a / are indeed a deviation from the XML
>>> returned by WebDAV server, and this is just because nowadays JSON is easier
>>> to use than XML for developers, both client-side and server-side.
>>>
>>>
>>>
>>> not make too much sense in this case.

---

From: Michiel de Jong <[email-masked]>
Date: Wed, 2 Dec 2015 16:05:43 +0100

Ah sure, that's entirely appropriate, remoteStorage treats both the
Content-Type header value and the actual body of a document as opaque
strings of bytes. It doesn't "care" if you use it to store only data blocks
that are chunks of something else.

For instance, you could have a folder on a user's storage that contains
only inode-like JSON-documents, which list the URLs of binary documents
that make up 1Kb blocks of the actual data. Nice for deduping, delta
updates, and also renaming files without reuploading their content.

But yeah, the argument is that *how* to create and manage these chunks, is
then still left up to the application developer (or to another spec on top
of the remoteStorage spec).

Cheers,
Michiel.

On Wed, Dec 2, 2015 at 3:29 PM, Linhui Sun <[email-masked]> wrote:

> Actually what I'm saying is a chunking before transmitting (using http),
> in this way, they are treated as individual documents from the standpoint
> of http. But I don't know whether this is appropriate for remoteStorage,
> just a comment.
>
> Regards,
> Linhui
>
>>
>>> Another thing is do we need to include an application-layer chunking
>>> here (not just for a browser sync), since if we want to further extend
>>> other capabilities it is essential.
>>>
>>> Regards,
>>> Linhui
>>>
>>> 2015-12-02 21:03 GMT+08:00 Hugo González Labrador <[email-masked]
>>> >:
>>>
>>>> implement all of WebDAV. It's very annoying for client implementers to have
>>>> to deal with servers that e.g. chose not to implement LOCK and UNLOCK.
>>>> 2015-12-02 5:14 GMT+08:00 Hugo González Labrador <[email-masked]
>>>> >:
>>>>
>>>> Hi,
>>>>
>>>> provide that. Is that because remoteStorage does not perform like a typical
>>>> sync services (e.g. dropbox...) or you are saying something else?
>>>>

---

From: Linhui Sun <[email-masked]>
Date: Wed, 02 Dec 2015 23:36:58 +0800

Sounds reasonable! How to perform this chunking and how to manage chunks should
be left for the higher level. Thanks for that : )
On 周三, 12月 2, 2015 at 23:05, Michiel de Jong <[email-masked]> wrote:
Ah sure, that's entirely appropriate, remoteStorage treats both the Content-Type
header value and the actual body of a document as opaque strings of bytes. It
doesn't "care" if you use it to store only data blocks that are chunks of
something else.

For instance, you could have a folder on a user's storage that contains only
inode-like JSON-documents, which list the URLs of binary documents that make up
1Kb blocks of the actual data. Nice for deduping, delta updates, and also
renaming files without reuploading their content.

But yeah, the argument is that *how* to create and manage these chunks, is then
still left up to the application developer (or to another spec on top of the
remoteStorage spec).

Cheers,
Michiel.

On Wed, Dec 2, 2015 at 3:29 PM, Linhui Sun < [email-masked] [[email-masked]] > wrote:
Hi
2015-12-02 22:05 GMT+08:00 Michiel de Jong < [email-masked] [[email-masked]] > :
Cool! I created [https://github.com/remotestorage/spec/issues/137] https://github.com/remotestorage/spec/issues/137
[https://github.com/remotestorage/spec/issues/137] about the need for MOVE verb.

Application-level chunking is partially supported by HTTP itself through
`Content-Range` headers (although it's not clear whether these are allowed on
PUT requests as well as on GET requests, see [https://github.com/remotestorage/spec/issues/131] https://github.com/remotestorage/spec/issues/131
[https://github.com/remotestorage/spec/issues/131] ). The problem is that HTTP defines versioning at the document level, you
cannot ask a server to produce or check an ETag for a specific byte-range of a
document, only for the entire document.
Actually what I'm saying is a chunking before transmitting (using http), in this
way, they are treated as individual documents from the standpoint of http. But I
don't know whether this is appropriate for remoteStorage, just a comment.
Regards, Linhui
A comparison document sounds good! See also [http://www.servicedenuages.fr/en/generic-storage-ecosystem] http://www.servicedenuages.fr/en/generic-storage-ecosystem
[http://www.servicedenuages.fr/en/generic-storage-ecosystem] .

Cheers,
Michiel.

On Wed, Dec 2, 2015 at 2:32 PM, Linhui Sun < [email-masked] [[email-masked]] > wrote:
That's cool for me, a separate section for this might make sense.
Another thing is do we need to include an application-layer chunking here (not
just for a browser sync), since if we want to further extend other capabilities
it is essential.

Regards, Linhui
2015-12-02 21:03 GMT+08:00 Hugo González Labrador < [email-masked] [[email-masked]] > :
I propose to come up with a list of advantages and disadvantages of using WebDAV
and compare them against a JSON/REST based approach, like remoteStorage.
Does it sound good ? On Wed, Dec 2, 2015, at 01:59 PM, Linhui Sun wrote:
2015-12-02 20:43 GMT+08:00 Hugo González Labrador < [email-masked] [[email-masked]] > :

On Wed, Dec 2, 2015, at 01:30 PM, Michiel de Jong wrote:
Hi all!
Thanks for all your reactions to the remoteStorage Internet-Draft.
We get the question about WebDAV a lot, in the next version we should add a
remark about it in the intro. The folder descriptions returned when you GET a
URL that ends with a / are indeed a deviation from the XML returned by WebDAV
server, and this is just because nowadays JSON is easier to use than XML for
developers, both client-side and server-side.
I totally agree here, this was going to happen soon or later and it is really
appreciated.
The fact that we don't require servers to support WebDAV's custom verbs like
PROPPATCH etc. is for three reasons:
1) it's a lot of work to implement this without using an existing WebDAV library
2) in practice, a lot of WebDAV servers get it wrong, or don't implement all of
WebDAV. It's very annoying for client implementers to have to deal with servers
that e.g. chose not to implement LOCK and UNLOCK.
3) we don't really need all these advanced features on top of standard HTTP,
just supporting GET/PUT/DELETE for resources, and adding a simple folder
description format, is enough for most applications.
Other than that, the remoteStorage Internet-Draft specifies a *lot* more than
just how each HTTP verb should behave:
* requiring support for OAuth implicit-grant flow
* requiring ETag support and nested versioning (i.e. the folder's ETag changes
if anything within that folder changes)
* requiring CORS headers
* requiring a WebFinger announcement for service discovery
It would be easy to add these three things on top of WebDAV, instead of putting
them on top of our minimal GET/PUT/DELETE API definition. In fact, we could
probably separate it into two Internet-Drafts: one for the 'RESTful folders' API
which is our simplification of WebDAV, and a second one for
OAuth/ETags/CORS/WebFinger on top of either WebDAV or 'RESTful folders' or
whatever other HTTP-based API you want.
There is one requirement that all synchronisers need to handle: moving
resources. In this spec the alternative of a WebDAV MOVE is not specified.
It is true that a MOVE could be replaced with a DELETE + UPLOAD but that is not
acceptable in most cases due to the inefficiency of such operation (cpu,
bandwidth consumption ...)
Is there a plan to support such basic feature?
implemented. The missing bit is tracking those remote moves.
I agree with Hugo that MOVE is useful, sometimes if you just rename a file it
will be perfect. But the question I have is that whether we need to make two
documents? Multiple choices is not good for standardization. In my view, webdav
is something that we need to have a very clear decision on whether to consider
it or not.
Regards,
Linhui
Cheers
On Wed, Dec 2, 2015 at 11:28 AM, Hugo González Labrador < [email-masked] [[email-masked]] > wrote:

On Wed, Dec 2, 2015, at 11:18 AM, Linhui Sun wrote:
Hi
On 周三, 12月 2, 2015 at 17:56, Hugo González Labrador < [email-masked] [[email-masked]] > wrote:
On Wed, Dec 2, 2015, at 08:20 AM, Linhui Sun wrote:
Hi
2015-12-02 5:14 GMT+08:00 Hugo González Labrador < [email-masked] [[email-masked]] > :
Hi,
from my point of view the remoteStorage project addresses a subset of
the use cases of the WebDAV specification.
The main difference I can observe is that the specification is built on
REST instead of XML-based communication.
I personally like much more REST than WebDAV because it is more
developer friendly and it is faster to develop.
Maybe the remoteStorage API becomes the next WebDAV :)
However, the remoteStorage API does not provide a way of synchronising
data, this task is delegated to the developers.
Is there a plan to provide such feature based on the outcome of this
draft?
I'm a little bit confused why you say the remoteStorage does not provide that.
Is that because remoteStorage does not perform like a typical sync services
(e.g. dropbox...) or you are saying something else?
Yes, because it does not offer synchronisation capabilities.
Got it. And What I am wondering is that do we need to include those capabilities
in a base specifications. Since it is hard to standardize the capabilities like
dedup or delta. Maybe a better way is to define those in a separate
specification,
Thanks for giving these examples - so by 'synchronisation capabilities' you mean
1) deduplication and 2) delta updates? Anything else or is that an exhaustive
list?
something like extensions. For a base document, we just need to define how to
perform sync operations.
Yes, I agree that may be an extension of this draft could handle the
synchronisation part.
Our Internet-Draft is heavily focused on the world wide web, whose URLs are not
content-addressable, we can't change that. So that architecture is not very
friendly to deduplication, compared to for instance BitTorrent. As you already
said, developers can still dedupe on the server-side when storing blobs to disk,
and can also dedupe on the client side before creating the resources the client
uploads.
As far as I know, delta updates are not supported by the ETag system - you
cannot do a range request to find out if certain bytes within a document have
changed. However, the folder system we define does encourage you, for instance
when you develop a To Do List app, put each task on its own document, and then
query the folder to see which of them changed, instead of putting them all in
one big document and retrieving the whole document each time.
Cheers,
Michiel.
BTW, I want to introduce ClawIO ( [http://clawio.github.io] [http://clawio.github.io] http://clawio.github.io [http://clawio.github.io] ), a research
project to benchmark different synchronisation protocols against
different data backends with special attention to provide a common sync
API.
A common API for different sync protocols is being created based on the
architecture specified in this draft (control and data servers) and the
I cannot find a distributed architecture in this draft. It seems that they
handle metadata and content data together, just like a normal web service.
ClawIO is fully distributed. Every logical unit is a different server than be
scaled out. Data and Metadata channels are independent from each other and
reside on different servers.
That is widely employed in popular sync services. And that is also beneficial to
privacy to some extent. But in the context of sync in browser (which is mainly
considered in the remoteStorage), I don't know whether this is reasonable. But I
think we should deploy distributed architecture although it will make things
complicated.
Of course, the remoteStorage is targeted to browsers, so syncing does not make
too much sense in this case.
With the rise of Linux container micro-service based architectures, the
deployment of such highly complex systems should become easier and faster.
Best,
Hugo
Regards,
Linhui
Regards,
Linhui
one from the CERN EOS project (management, disk and queue servers).
The Phase I has implemented the ownCloud Sync Protocol and Phase II will
implement the SeaFile Sync Protocol.
The choice of these protocols among others is because they are really
opposed to each other in terms of syncing (delta vs non-delta,
state-based vs log/event/git-based sync …), so finding a common approach
is more challenging.
Providing a base specification/architecture to measure the feasibility
of this draft is one of the objectives of the project.
I believe that the work being done here and in ClawIO are supplementary
to each other and I think mutual collaboration could be beneficial for
both sides.
Also, if there is interest, the remoteStorage API can be added to
ClawIO.
Best regards,
Hugo Gonzalez Labrador
On Tue, Dec 1, 2015, at 09:00 PM, [email-masked] [[email-masked]] wrote:
> [https://www.ietf.org/mailman/listinfo/storagesync] [https://www.ietf.org/mailman/listinfo/storagesync] https://www.ietf.org/mailman/listinfo/storagesync
> [https://www.ietf.org/mailman/listinfo/storagesync] [https://www.ietf.org/mailman/listinfo/storagesync] https://www.ietf.org/mailman/listinfo/storagesync
_______________________________________________
Storagesync mailing list
[email-masked] [[email-masked]]
[https://www.ietf.org/mailman/listinfo/storagesync] [https://www.ietf.org/mailman/listinfo/storagesync] https://www.ietf.org/mailman/listinfo/storagesync
[https://www.ietf.org/mailman/listinfo/storagesync]

---

From: [email-masked]
Date: Thu, 3 Dec 2015 16:02:51 +0800 (GMT+08:00)

+1. This is definitely a basic issue for this group~

-----原始邮件-----
发件人: "Hugo González Labrador" <[email-masked]>
发送时间: 2015年12月2日 星期三
收件人: "Linhui Sun" <[email-masked]>
抄送: storagesync <[email-masked]>, "Michiel de Jong" <[email-masked]>, fkooman <[email-masked]>
主题: Re: [Storagesync] Storagesync Digest, Vol 5, Issue 1

I propose to come up with a list of advantages and disadvantages of using WebDAV and compare them against a JSON/REST based approach, like remoteStorage.

Does it sound good ?

 

 

Hi all!

We get the question about WebDAV a lot, in the next version we should add a remark about it in the intro. The folder descriptions returned when you GET a URL that ends with a / are indeed a deviation from the XML returned by WebDAV server, and this is just because nowadays JSON is easier to use than XML for developers, both client-side and server-side.

I totally agree here, this was going to happen soon or later and it is really appreciated.

The fact that we don't require servers to support WebDAV's custom verbs like PROPPATCH etc. is for three reasons:

2) in practice, a lot of WebDAV servers get it wrong, or don't implement all of WebDAV. It's very annoying for client implementers to have to deal with servers that e.g. chose not to implement LOCK and UNLOCK.

3) we don't really need all these advanced features on top of standard HTTP, just supporting GET/PUT/DELETE for resources, and adding a simple folder description format, is enough for most applications.

Other than that, the remoteStorage Internet-Draft specifies a *lot* more than just how each HTTP verb should behave:

* requiring ETag support and nested versioning (i.e. the folder's ETag changes if anything within that folder changes)

* requiring CORS headers

It would be easy to add these three things on top of WebDAV, instead of putting them on top of our minimal GET/PUT/DELETE API definition. In fact, we could probably separate it into two Internet-Drafts: one for the 'RESTful folders' API which is our simplification of WebDAV, and a second one for OAuth/ETags/CORS/WebFinger on top of either WebDAV or 'RESTful folders' or whatever other HTTP-based API you want.

There is one requirement that all synchronisers need to handle: moving resources. In this spec the alternative of a WebDAV MOVE is not specified. 

It is true that a MOVE could be replaced with a DELETE + UPLOAD but that is not acceptable in most cases due to the inefficiency of such operation (cpu, bandwidth consumption ...)

From the current remoteStorage spec, the ownCloud sync protocol can be implemented. The missing bit is tracking those remote moves.

I agree with Hugo that MOVE is useful, sometimes if you just rename a file it will be perfect. But the question I have is that whether we need to make two documents? Multiple choices is not good for standardization. In my view, webdav is something that we need to have a very clear decision on whether to consider it or not.

Regards,

Linhui

Cheers

On Wed, Dec 2, 2015 at 11:28 AM, Hugo González Labrador <[email-masked]> wrote:

 

Hi

On 周三, 12月 2, 2015 at 17:56, Hugo González Labrador <[email-masked]> wrote:

 

Hi

Hi,

draft?

I'm a little bit confused why you say the remoteStorage does not provide that. Is that because remoteStorage does not perform like a typical sync services (e.g. dropbox...) or you are saying something else?

Got it. And What I am wondering is that do we need to include those capabilities in a base specifications. Since it is hard to standardize the capabilities like dedup or delta. Maybe a better way is to define those in a separate specification,

Thanks for giving these examples - so by 'synchronisation capabilities' you mean 1) deduplication and 2) delta updates? Anything else or is that an exhaustive list?

something like extensions. For a base document, we just need to define how to perform sync operations.

Yes, I agree that may be an extension of this draft could handle the synchronisation part.

Our Internet-Draft is heavily focused on the world wide web, whose URLs are not content-addressable, we can't change that. So that architecture is not very friendly to deduplication, compared to for instance BitTorrent. As you already said, developers can still dedupe on the server-side when storing blobs to disk, and can also dedupe on the client side before creating the resources the client uploads.

As far as I know, delta updates are not supported by the ETag system - you cannot do a range request to find out if certain bytes within a document have changed. However, the folder system we define does encourage you, for instance when you develop a To Do List app, put each task on its own document, and then query the folder to see which of them changed, instead of putting them all in one big document and retrieving the whole document each time.

Cheers,

Michiel.

API.

I cannot find a distributed architecture in this draft. It seems that they handle metadata and content data together, just like a normal web service.

ClawIO is fully distributed. Every logical unit is a different server than be scaled out. Data and Metadata channels are independent from each other and reside on different servers.

That is widely employed in popular sync services. And that is also beneficial to privacy to some extent. But in the context of sync in browser (which is mainly considered in the remoteStorage), I don't know whether this is reasonable. But I think we should deploy distributed architecture although it will make things complicated.

Of course, the remoteStorage is targeted to browsers, so syncing does not make too much sense in this case.

With the rise of Linux container micro-service based architectures, the deployment of  such highly complex systems should become easier and faster.

Best,

Hugo

Regards,

Linhui 

Regards,

Linhui

is more challenging.

both sides.

ClawIO.

Best regards,

Hugo Gonzalez Labrador

Storagesync mailing list

[email-masked]

---

From: Linhui Sun <[email-masked]>
Date: Thu, 3 Dec 2015 16:13:29 +0800

Hi

I've missed some part of your message yesterday, please see my comments
inline.

> you mean 1) deduplication and 2) delta updates? Anything else or is that an
Yes and something like chunking (the one I've mentioned in the previous
message) and bundling. They are typically client capabilities (or referred
as service capabilities). These should be beyond the basic document like
the remoteStorage, but is important and useful to be defined in a companion
document.

> creating the resources the client uploads.
IMO, performing dedup. inside server or client itself is not interesting.
What we want for a sync service is to reduce the number of transmitting
packets between client and server. That requires both client and server
should be aware of which packet should be transmitted. And I think
capabilities like dedup. has nothing to do with whether you use a
distributed architecture.

> of putting them all in one big document and retrieving the whole document
That's a delta update in essence, but is done by user behavior. And could
we make the "query the folder" something like server push (similar issue in
webpush WG)? When there is a change, the server will notify the client.

Regards,
Linhui

---

From: [email-masked]
Date: Thu, 3 Dec 2015 16:40:37 +0800 (GMT+08:00)

Hi Michael,

Based on the description on http://remotestorage.io/ “remoteStorage-enabled apps automatically sync your data across all of your devices”.

However, there is no synchronization in the abstract.

“The protocol supports storing, retrieving, and removing individual documents, as well as listing the contents of an individual folder, and access control is based on bearer tokens”

Therefore, I think:

If synchronization is added into abstract, the MOVE action should be added for sure.

If synchronization is not included, like current version, the basic actions in draft are enough.

-----原始邮件-----
发件人: "Michiel de Jong" <[email-masked]>
发送时间: 2015年12月2日 星期三
收件人: "Linhui Sun" <[email-masked]>
抄送: storagesync <[email-masked]>, fkooman <[email-masked]>, "Hugo González Labrador" <[email-masked]>
主题: Re: [Storagesync] Storagesync Digest, Vol 5, Issue 1

Cool! I created https://github.com/remotestorage/spec/issues/137 about the need for  MOVE verb.

Application-level chunking is partially supported by HTTP itself through `Content-Range` headers (although it's not clear whether these are allowed on PUT requests as well as on GET requests, see https://github.com/remotestorage/spec/issues/131). The problem is that HTTP defines versioning at the document level, you cannot ask a server to produce or check an ETag for a specific byte-range of a document, only for the entire document.

A comparison document sounds good! See also http://www.servicedenuages.fr/en/generic-storage-ecosystem.

Cheers,

Michiel.

Another thing is do we need to include an application-layer chunking here (not just for a browser sync), since if we want to further extend other capabilities it is essential.

Regards,
Linhui

Does it sound good ?

 

 

Hi all!

 

* requiring CORS headers

 

Regards,

Linhui

Cheers

 

Hi

 

Hi

Hi,

draft?

 

 

Cheers,

Michiel.

API.

 

Best,

Hugo

Regards,

Linhui 

Regards,

Linhui

is more challenging.

both sides.

ClawIO.

Best regards,

Hugo Gonzalez Labrador

Storagesync mailing list

[email-masked]

---

From: [email-masked]
Date: Thu, 3 Dec 2015 16:44:08 +0800 (GMT+08:00)

Hi Michiel and Linhui,

I think it will be good to have a boundary for this draft and leave some work for the applicaiton layer~

-----原始邮件-----
发件人: "Michiel de Jong" <[email-masked]>
发送时间: 2015年12月2日 星期三
收件人: "Linhui Sun" <[email-masked]>
抄送: storagesync <[email-masked]>, fkooman <[email-masked]>, "Hugo González Labrador" <[email-masked]>
主题: Re: [Storagesync] Storagesync Digest, Vol 5, Issue 1

Ah sure, that's entirely appropriate, remoteStorage treats both the Content-Type header value and the actual body of a document as opaque strings of bytes. It doesn't "care" if you use it to store only data blocks that are chunks of something else.

For instance, you could have a folder on a user's storage that contains only inode-like JSON-documents, which list the URLs of binary documents that make up 1Kb blocks of the actual data. Nice for deduping, delta updates, and also renaming files without reuploading their content.

But yeah, the argument is that *how* to create and manage these chunks, is then still left up to the application developer (or to another spec on top of the remoteStorage spec).

Cheers,

Michiel.

Hi

Actually what I'm saying is a chunking before transmitting (using http), in this way, they are treated as individual documents from the standpoint of http. But I don't know whether this is appropriate for remoteStorage, just a comment.

Regards,
Linhui

Cheers,

Michiel.

Regards,
Linhui

Does it sound good ?

 

 

Hi all!

 

* requiring CORS headers

 

Regards,

Linhui

Cheers

 

Hi

 

Hi

Hi,

draft?

 

 

Cheers,

Michiel.

API.

 

Best,

Hugo

Regards,

Linhui 

Regards,

Linhui

is more challenging.

both sides.

ClawIO.

Best regards,

Hugo Gonzalez Labrador

Storagesync mailing list

[email-masked]

---

From: Linhui Sun <[email-masked]>
Date: Thu, 3 Dec 2015 16:59:33 +0800

2015-12-03 16:40 GMT+08:00 <[email-masked]>:

> Hi Michael,
>
>
>
> Based on the description on *http://remotestorage.io/*
> <http://remotestorage.io/> “remoteStorage-enabled apps automatically sync
> your data across all of your devices”.
>
>
>
> “The protocol supports storing, retrieving, and removing individual
> documents, as well as listing the contents of an individual folder, and
> access control is based on bearer tokens”
>
>
>
> Therefore, I think:
>
I don't know why MOVE should be tied to the “synchronization”. It seems
that only sync allows MOVE to be used.

Regards,
Linhui

---

From: Michiel de Jong <[email-masked]>
Date: Thu, 3 Dec 2015 10:01:33 +0100

Yes! I'll discuss with Francois how we can make these boundaries clearer.
Maybe we can split along these lines:

* upload/manipulate/query/download protocol (e.g. WebDAV or the
GET/PUT/DELETE operations we currently define)
* CORS or other cross-origin access mechanism, OAUTH or other auth
mechanism, WebFinger or other discovery mechanism, ETag or other versioning
mechanism
* chunking/deduping/delta-updates

And the remoteStorage spec should probably only define the middle part.

On Thu, Dec 3, 2015 at 9:44 AM, <[email-masked]> wrote:

> *发件人:* "Michiel de Jong" <[email-masked]>
> *发送时间:* 2015年12月2日 星期三
> *收件人:* "Linhui Sun" <[email-masked]>
> *抄送:* storagesync <[email-masked]>, fkooman <[email-masked]>,
> "Hugo González Labrador" <[email-masked]>
> *主题:* Re: [Storagesync] Storagesync Digest, Vol 5, Issue 1
>
>>>>> It would be easy to add these three things on top of WebDAV, instead
>>>>> of putting them on top of our minimal GET/PUT/DELETE API definition. In
>>>>> fact, we could probably separate it into two Internet-Drafts: one for the
>>>>> Thanks for giving these examples - so by 'synchronisation
>>>>> capabilities' you mean 1) deduplication and 2) delta updates? Anything else
>>>>> or is that an exhaustive list?
>>>>>
>>>>>
>>>>>
>>>>> instance BitTorrent. As you already said, developers can still dedupe on
>>>>> the server-side when storing blobs to disk, and can also dedupe on the
>>>>> client side before creating the resources the client uploads.
>>>>> BTW, I want to introduce ClawIO ( <http://clawio.github.io/>
>>>>> With the rise of Linux container micro-service based architectures,
>>>>> the deployment of  such highly complex systems should become easier and
>>>>> faster.
>>>>>
>>>>> Best,
>>>>>
>>>>>
>>>>> Hugo
>>>>>
>>>>>
>>>>>
>>>>> Regards,
>>>>> Linhui
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Regards,
>>>>> Linhui
>>>>>
>>>>> will

---

From: [email-masked]
Date: Thu, 3 Dec 2015 17:06:40 +0800 (GMT+08:00)

Hi

If we really want to discuss the Pros and Cons of WebDAV, can we mark these three reasons as disadvantages of WebDAV? any support for that?

For the target of our ISS group, whether the WebDAV can be reused is critical.

-----原始邮件-----
发件人: "Michiel de Jong" <[email-masked]>
发送时间: 2015年12月2日 星期三
收件人: "Hugo González Labrador" <[email-masked]>
抄送: "Linhui Sun" <[email-masked]>, storagesync <[email-masked]>, [email-masked]
主题: Re: [Storagesync] Storagesync Digest, Vol 5, Issue 1

Hi all!

* requiring CORS headers

 

Hi

 

Hi

Hi,

draft?

Cheers,

Michiel.

 

API.

Best,

Hugo
 
Regards,

Linhui 

Regards,

Linhui

is more challenging.

both sides.

ClawIO.

Best regards,

Hugo Gonzalez Labrador

Storagesync mailing list

[email-masked]

---

From: [email-masked]
Date: Thu, 3 Dec 2015 17:15:27 +0800 (GMT+08:00)

Hi Linhui,

Thanks for asking! Maybe I didn’t express clearly. I think MOVE is good and more useful for synchronization.

But if this draft only want to identify the basic elements for remoteStorage and look for the minimum set, MOVE should be further investigated.

After all, it can be replaced by DEL + UPLOAD.

-----原始邮件-----
发件人: "Linhui Sun" <[email-masked]>
发送时间: 2015年12月3日 星期四
收件人: fsong <[email-masked]>
抄送: storagesync <[email-masked]>, fkooman <[email-masked]>, "Michiel de Jong" <[email-masked]>, "Hugo González Labrador" <[email-masked]>
主题: Re: [Storagesync] Storagesync Digest, Vol 5, Issue 1

Hi Michael,

Therefore, I think:

I don't know why MOVE should be tied to the “synchronization”. It seems that only sync allows MOVE to be used.

Regards,
Linhui

---

From: [email-masked]
Date: Thu, 3 Dec 2015 17:31:51 +0800 (GMT+08:00)

Great！

-----原始邮件-----
发件人: "Michiel de Jong" <[email-masked]>
发送时间: 2015年12月3日 星期四
收件人: [email-masked]
抄送: "Linhui Sun" <[email-masked]>, storagesync <[email-masked]>, fkooman <[email-masked]>, "Hugo González Labrador" <[email-masked]>
主题: Re: [Storagesync] Storagesync Digest, Vol 5, Issue 1

Yes! I'll discuss with Francois how we can make these boundaries clearer. Maybe we can split along these lines:

* upload/manipulate/query/download protocol (e.g. WebDAV or the GET/PUT/DELETE operations we currently define)

* CORS or other cross-origin access mechanism, OAUTH or other auth mechanism, WebFinger or other discovery mechanism, ETag or other versioning mechanism

Hi Michiel and Linhui,

-----原始邮件-----
发件人: "Michiel de Jong" <[email-masked]>
发送时间: 2015年12月2日 星期三
收件人: "Linhui Sun" <[email-masked]>
抄送: storagesync <[email-masked]>, fkooman <[email-masked]>, "Hugo González Labrador" <[email-masked]>
主题: Re: [Storagesync] Storagesync Digest, Vol 5, Issue 1

Cheers,

Michiel.

Hi

Regards,
Linhui

Cheers,

Michiel.

Regards,
Linhui

Does it sound good ?

 

 

Hi all!

 

* requiring CORS headers

 

Regards,

Linhui

Cheers

 

Hi

 

Hi

Hi,

draft?

 

 

Cheers,

Michiel.

API.

 

Best,

Hugo

Regards,

Linhui 

Regards,

Linhui

is more challenging.

both sides.

ClawIO.

Best regards,

Hugo Gonzalez Labrador

Storagesync mailing list

[email-masked]

---

From: François Kooman <[email-masked]>
Date: Thu, 3 Dec 2015 10:44:42 +0100

On 12/03/2015 10:06 AM, [email-masked] wrote:
> If we really want to discuss the Pros and Cons of WebDAV, can we
> mark these three reasons as disadvantages of WebDAV? any support for
> that?

Personally, I do not think that is fair.

> 1) it's a lot of work to implement this without using an existing

I do not see a problem in reusing an existing WebDAV library. This has
the added benefit that you get a lot of functionality 'for free' and
that existing test suites are available for testing compatibility. This
is purely a practical argument!

> 2) in practice, a lot of WebDAV servers get it wrong,
> or don't implement all of WebDAV. It's very annoying for client

The point is not to have a fully compliant (whatever that is exactly)
WebDAV server. Either this ML (or remoteStorage specification) could
determine a subset of WebDAV that would be required to be implemented.
E.g. we exhaustively list the verbs that need to be implemented and
their expected behavior. If we keep this a limited as possible and stay
away from experimental features we have a high probability that most
libraries will get it right! If we stay away from locking and ACLs the
library situation looks a lot brighter :)

> 3) we don't really need all these advanced
> features on top of standard HTTP, just supporting GET/PUT/DELETE for
> resources, and adding a simple folder description format, is enough
> for most applications.

Exactly!

> For the target of our ISS group, whether the WebDAV can be reused is 

Well, my point is that there is little reason to reinvent the wheel if
the only benefit is to use JSON instead of XML for folder listings. The
amount of experience and available tooling for WebDAV is already huge!
It would be a waste to throw this away just for liking JSON better.

But maybe there are other reasons using (a limited set of) WebDAV is
impossible or unwanted...

Cheers,
François

---

From: Hugo González Labrador <[email-masked]>
Date: Thu, 03 Dec 2015 10:49:09 +0100

What is the best way to collect the advantages and drawbacks of WebDAV
vs remoteStorage ?

I propose to create a GitHub instead of/in addition to specify the
differences here. I think is clear to show this info in something like a
table on GitHub instead of in a series of mail messages.

I can create the repo if it is okay for you.

Best,Hugo

On Thu, Dec 3, 2015, at 10:31 AM, [email-masked] wrote:
> Great！
>
>
>> -----原始邮件----- *发件人:* "Michiel de Jong" <[email-masked]>
>> *发送时间:* 2015年12月3日 星期四 *收件人:* [email-masked] *抄送:* "Linhui Sun"
>> <[email-masked]>, storagesync <[email-masked]>, fkooman
>> <[email-masked]>, "Hugo González Labrador" <[email-masked]>
>> Yes! I'll discuss with Francois how we can make these boundaries
>> clearer. Maybe we can split along these lines:
>>   mechanism, WebFinger or other discovery mechanism, ETag or other
>>   versioning mechanism

>>>> -----原始邮件----- *发件人:* "Michiel de Jong" <[email-masked]>
>>>> *发送时间:* 2015年12月2日 星期三 *收件人:* "Linhui Sun" <[email-masked]>
>>>> *抄送:* storagesync <[email-masked]>, fkooman
>>>> <[email-masked]>, "Hugo González Labrador"
>>>> <[email-masked]> *主题:* Re: [Storagesync] Storagesync Digest,
>>>> Vol 5, Issue 1
>>>>
>>>> Content-Type header value and the actual body of a document as
>>>> opaque strings of bytes. It doesn't "care" if you use it to store
>>>> only data blocks that are chunks of something else. For instance,
>>>> you could have a folder on a user's storage that contains only inode-
>>>> like JSON-documents, which list the URLs of binary documents that
>>>> make up 1Kb blocks of the actual data. Nice for deduping, delta
>>>> But yeah, the argument is that *how* to create and manage these
>>>> chunks, is then still left up to the application developer (or to
>>>> another spec on top of the remoteStorage spec).
>>>>
>>>>
>>>> Cheers, Michiel.
>>>>
>>>>>> Cool! I created https://github.com/remotestorage/spec/issues/137
>>>>>> about the need for  MOVE verb. Application-level chunking is
>>>>>> partially supported by HTTP itself through `Content-Range`
>>>>>> headers (although it's not clear whether these are allowed on PUT
>>>>>> requests as well as on GET requests, see
>>>>>> https://github.com/remotestorage/spec/issues/131). The problem is
>>>>>> that HTTP defines versioning at the document level, you cannot
>>>>>> ask a server to produce or check an ETag for a specific byte-
>>>>>> range of a document, only for the entire document.
>>>>>
>>>>> Actually what I'm saying is a chunking before transmitting (using
>>>>> http), in this way, they are treated as individual documents from
>>>>> the standpoint of http. But I don't know whether this is
>>>>> appropriate for remoteStorage, just a comment.
>>>>>
>>>>> Regards, Linhui
>>>>>>
>>>>>>> Another thing is do we need to include an application-layer
>>>>>>> chunking here (not just for a browser sync), since if we want to
>>>>>>> further extend other capabilities it is essential.
>>>>>>>
>>>>>>> Regards, Linhui
>>>>>>>
>>>>>>>> I propose to come up with a list of advantages and
>>>>>>>> disadvantages of using WebDAV and compare them against a
>>>>>>>> JSON/REST based approach, like remoteStorage.
>>>>>>>>
>>>>>>>> Does it sound good ?
>>>>>>>>
>>>>>>>>
>>>>>>>>>>> Hi all! Thanks for all your reactions to the remoteStorage
>>>>>>>>>>> Internet-Draft.
>>>>>>>>>>>
>>>>>>>>>>> We get the question about WebDAV a lot, in the next version
>>>>>>>>>>> we should add a remark about it in the intro. The folder
>>>>>>>>>>> descriptions returned when you GET a URL that ends with a /
>>>>>>>>>>> are indeed a deviation from the XML returned by WebDAV
>>>>>>>>>>> server, and this is just because nowadays JSON is easier to
>>>>>>>>>>> use than XML for developers, both client-side and server-
>>>>>>>>>>> side.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> The fact that we don't require servers to support WebDAV's
>>>>>>>>>>> custom verbs like PROPPATCH etc. is for three reasons:
>>>>>>>>>>> 2) in practice, a lot of WebDAV servers get it wrong, or
>>>>>>>>>>>    don't implement all of WebDAV. It's very annoying for
>>>>>>>>>>>    client implementers to have to deal with servers that
>>>>>>>>>>>    e.g. chose not to implement LOCK and UNLOCK.
>>>>>>>>>>> 3) we don't really need all these advanced features on top
>>>>>>>>>>>    of standard HTTP, just supporting GET/PUT/DELETE for
>>>>>>>>>>>    resources, and adding a simple folder description format,
>>>>>>>>>>>    is enough for most applications.
>>>>>>>>>>>
>>>>>>>>>>> Other than that, the remoteStorage Internet-Draft specifies
>>>>>>>>>>> a *lot* more than just how each HTTP verb should behave:
>>>>>>>>>>> * requiring ETag support and nested versioning (i.e. the
>>>>>>>>>>>   folder's ETag changes if anything within that folder
>>>>>>>>>>>   changes)
>>>>>>>>>>> * requiring CORS headers
>>>>>>>>>>>   It would be easy to add these three things on top of
>>>>>>>>>>>   WebDAV, instead of putting them on top of our minimal
>>>>>>>>>>>   GET/PUT/DELETE API definition. In fact, we could probably
>>>>>>>>>>>   separate it into two Internet-Drafts: one for the 'RESTful
>>>>>>>>>>>   folders' API which is our simplification of WebDAV, and a
>>>>>>>>>>>   second one for OAuth/ETags/CORS/WebFinger on top of either
>>>>>>>>>>>   WebDAV or 'RESTful folders' or whatever other HTTP-based
>>>>>>>>>>>   API you want.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> There is one requirement that all synchronisers need to
>>>>>>>>>> handle: moving resources. In this spec the alternative of a
>>>>>>>>>> WebDAV MOVE is not specified.
>>>>>>>>>>
>>>>>>>>>> It is true that a MOVE could be replaced with a DELETE +
>>>>>>>>>> UPLOAD but that is not acceptable in most cases due to the
>>>>>>>>>> inefficiency of such operation (cpu, bandwidth consumption
>>>>>>>>>> ...)
>>>>>>>>>>
>>>>>>>>>> From the current remoteStorage spec, the ownCloud sync
>>>>>>>>>> protocol can be implemented. The missing bit is tracking
>>>>>>>>>> those remote moves.
>>>>>>>>> I agree with Hugo that MOVE is useful, sometimes if you just
>>>>>>>>> rename a file it will be perfect. But the question I have is
>>>>>>>>> that whether we need to make two documents? Multiple choices
>>>>>>>>> is not good for standardization. In my view, webdav is
>>>>>>>>> something that we need to have a very clear decision on
>>>>>>>>>>>>>>>> from my point of view the remoteStorage project
>>>>>>>>>>>>>>>> addresses a subset of the use cases of the  WebDAV
>>>>>>>>>>>>>>>> specification.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The main difference I can observe is that the
>>>>>>>>>>>>>>>> specification is built on REST instead of XML-based
>>>>>>>>>>>>>>>> communication.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I personally like much more REST than WebDAV because it
>>>>>>>>>>>>>>>> is more developer friendly and it is faster to develop.
>>>>>>>>>>>>>>>> However, the remoteStorage API does not provide a way
>>>>>>>>>>>>>>>> of synchronising data, this task is delegated to the
>>>>>>>>>>>>>>>> developers. Is there a plan to provide such feature
>>>>>>>>>>>>>>>> based on the outcome of this draft?
>>>>>>>>>>>>>>> I'm a little bit confused why you say the remoteStorage
>>>>>>>>>>>>>>> does not provide that. Is that because remoteStorage
>>>>>>>>>>>>>>> does not perform like a typical sync services (e.g.
>>>>>>>>>>>>>>> dropbox...) or you are saying something else?
>>>>>>>>>>>>> Got it. And What I am wondering is that do we need to
>>>>>>>>>>>>> include those capabilities in a base specifications. Since
>>>>>>>>>>>>> it is hard to standardize the capabilities like dedup or
>>>>>>>>>>>>> delta. Maybe a better way is to define those in a separate
>>>>>>>>>>>>> specification,
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> capabilities' you mean 1) deduplication and 2) delta
>>>>>>>>>>>>> something like extensions. For a base document, we just
>>>>>>>>>>>>> need to define how to perform sync operations.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> Our Internet-Draft is heavily focused on the world wide web,
>>>>>>>>>>> whose URLs are not content-addressable, we can't change
>>>>>>>>>>> that. So that architecture is not very friendly to
>>>>>>>>>>> deduplication, compared to for instance BitTorrent. As you
>>>>>>>>>>> already said, developers can still dedupe on the server-side
>>>>>>>>>>> when storing blobs to disk, and can also dedupe on the
>>>>>>>>>>> client side before creating the resources the client
>>>>>>>>>>> uploads. As far as I know, delta updates are not supported
>>>>>>>>>>> by the ETag system - you cannot do a range request to find
>>>>>>>>>>> out if certain bytes within a document have changed.
>>>>>>>>>>> However, the folder system we define does encourage you, for
>>>>>>>>>>> instance when you develop a To Do List app, put each task on
>>>>>>>>>>> its own document, and then query the folder to see which of
>>>>>>>>>>> them changed, instead of putting them all in one big
>>>>>>>>>>> document and retrieving the whole document each time.
>>>>>>>>>>>
>>>>>>>>>>> Cheers, Michiel.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> BTW, I want to introduce ClawIO
>>>>>>>>>>>>>>>> (http://clawio.github.io[1]), a research project to
>>>>>>>>>>>>>>>> benchmark different synchronisation protocols against
>>>>>>>>>>>>>>>> different data backends with special attention to
>>>>>>>>>>>>>>>> provide a common sync API.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> A common API for different sync protocols is being
>>>>>>>>>>>>>>>> created based on the architecture specified in this
>>>>>>>>>>>>>>>> draft (control and data servers) and the
>>>>>>>>>>>>>>> I cannot find a distributed architecture in this draft.
>>>>>>>>>>>>>>> It seems that they handle metadata and content data
>>>>>>>>>>>>>>> together, just like a normal web service.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ClawIO is fully distributed. Every logical unit is a
>>>>>>>>>>>>>> different server than be scaled out. Data and Metadata
>>>>>>>>>>>>>> channels are independent from each other and reside on
>>>>>>>>>>>>>> different servers.
>>>>>>>>>>>>> That is widely employed in popular sync services. And that
>>>>>>>>>>>>> is also beneficial to privacy to some extent. But in the
>>>>>>>>>>>>> context of sync in browser (which is mainly considered in
>>>>>>>>>>>>> the remoteStorage), I don't know whether this is
>>>>>>>>>>>>> reasonable. But I think we should deploy distributed
>>>>>>>>>>>>> architecture although it will make things complicated.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Of course, the remoteStorage is targeted to browsers, so
>>>>>>>>>>>> syncing does not make too much sense in this case. With the
>>>>>>>>>>>> rise of Linux container micro-service based architectures,
>>>>>>>>>>>> the deployment of  such highly complex systems should
>>>>>>>>>>>> become easier and faster.
>>>>>>>>>>>>
>>>>>>>>>>>> Best,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hugo
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Regards, Linhui
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards, Linhui
>>>>>>>>>>>>>>>> The Phase I has implemented the ownCloud Sync Protocol
>>>>>>>>>>>>>>>> and Phase II will implement the SeaFile Sync Protocol.
>>>>>>>>>>>>>>>> The choice of these protocols among others is because
>>>>>>>>>>>>>>>> they are really opposed to each other in terms of
>>>>>>>>>>>>>>>> syncing (delta vs non-delta, state-based vs log/event/git-
>>>>>>>>>>>>>>>> Providing a base specification/architecture to measure
>>>>>>>>>>>>>>>> the feasibility of this draft is one of the objectives
>>>>>>>>>>>>>>>> of the project.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I believe that the work being done here and in ClawIO
>>>>>>>>>>>>>>>> are supplementary to each other and I think mutual
>>>>>>>>>>>>>>>> collaboration could be beneficial for both sides.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Also, if there is interest, the remoteStorage API can
>>>>>>>>>>>>>>>> be added to ClawIO.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hugo Gonzalez Labrador
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> > https://www.ietf.org/mailman/listinfo/storagesync or,
>>>>>>>>>>>>>>>> > via email, send a message with subject or body 'help'
>>>>>>>>>>>>>>>> > to       [email-masked]
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > When replying, please edit your Subject line so it is
>>>>>>>>>>>>>>>> > more specific than "Re: Contents of Storagesync
>>>>>>>>>>>>>>>> > digest..." Today's Topics:
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> >1. New version of draft-dejong-remotestorage    Internet-
>>>>>>>>>>>>>>>> >Draft       available (Michiel de Jong)    2. Re: New
>>>>>>>>>>>>>>>> >version of draft-dejong-remotestorage Internet-Draft
>>>>>>>>>>>>>>>> >available (Gihan Dias)    3. Re: New version of draft-dejong-
>>>>>>>>>>>>>>>> >remotestorage Internet-Draft       available (Fei
>>>>>>>>>>>>>>>> >Song)
>>>>>>>>>>>>>>>> > + [Storagesync] New version of draft-dejong-
>>>>>>>>>>>>>>>> >   remotestorage Internet-Draft available   2k
>>>>>>>>>>>>>>>> >   (message/rfc822)
>>>>>>>>>>>>>>>> > + Re: [Storagesync] New version of draft-dejong-
>>>>>>>>>>>>>>>> >   remotestorage Internet-Draft available   1k
>>>>>>>>>>>>>>>> >   (message/rfc822)
>>>>>>>>>>>>>>>> > + Re: [Storagesync] New version of draft-dejong-
>>>>>>>>>>>>>>>> >   remotestorage Internet-Draft available   2k
>>>>>>>>>>>>>>>> >   (message/rfc822)
>>>>>>>>>>>>>>>>

Links:

  1. http://clawio.github.io/

---

From: Michiel de Jong <[email-masked]>
Date: Thu, 3 Dec 2015 10:52:51 +0100

I think we should not (try to) pick one protocol - https://xkcd.com/927/

I think sync *clients* should be "protocol polyglots" in the same way as
https://en.wikipedia.org/wiki/Comparison_of_instant_messaging_clients#Protocol_support

---

From: Hugo González Labrador <[email-masked]>
Date: Thu, 03 Dec 2015 11:05:39 +0100

If clients need to be polyglot, then I don't see the need for remoteStorage or this draft, in the end, they will be other standards that clients need to implement (3rd image of https://xkcd.com/927/) 

There are already a bunch of polyglot sync clients,
https://crosscloud.me/ as an example. The problem with such client is
that they have to deal with N different protocols.

I thought the objective of this draft was to avoid such situation and
let the cloud provider implement a common standard (the one we are
seeking for).

On Thu, Dec 3, 2015, at 10:52 AM, Michiel de Jong wrote:
> I think we should not (try to) pick one protocol -
> https://xkcd.com/927/ I think sync *clients* should be "protocol
> polyglots" in the same way as
> For the target of our ISS group, whether the WebDAV can be reused is
>>
> critical.
>>
Cheers,
François

---

From: François Kooman <[email-masked]>
Date: Thu, 3 Dec 2015 11:09:46 +0100

> I think we should not (try to) pick one protocol - https://xkcd.com/927/

Saying that after writing your own spec ;)

> https://en.wikipedia.org/wiki/Comparison_of_instant_messaging_clients#Protocol_support

Because it worked so well for messaging clients? Yeah, I guess it works
if you live in the year 2000 and don't care about file transfer, voice,
video, offline messaging, receive notifications, sending pictures, group
chats, sync between devices, usable encryption, stickers and the holy
grail: federation. Is this not solved by Signal nowadays?

In theory it is a great idea and I'd agree with you, but in practice not
so much. KISS is key here. I'd recommend building on technology that is
already there, working and proven and free!

Cheers,
François

---

From: Linhui Sun <[email-masked]>
Date: Thu, 3 Dec 2015 18:52:01 +0800

Hi

> already there, working and proven and free!
Which technology are you talking about here?

---

From: François Kooman <[email-masked]>
Date: Thu, 3 Dec 2015 11:53:12 +0100

> Which technology are you talking about here?

WebDAV :)

Cheers,
François

---

From: Linhui Sun <[email-masked]>
Date: Thu, 3 Dec 2015 19:00:48 +0800

> > Which technology are you talking about here?
Ah, as I guessed : )
However I'd prefer HTTP rather than WebDAV. Since I still think WebDAV is
not suitable for today's sync service. The frequency of operations is far
higher than what WebDAV expects. For example, frequently sending propfind
to detect changes is not a good way in my view.

Regards,
Linhui

---

From: Linhui Sun <[email-masked]>
Date: Thu, 3 Dec 2015 19:06:36 +0800

2015-12-03 17:49 GMT+08:00 Hugo González Labrador <[email-masked]>:

> What is the best way to collect the advantages and drawbacks of WebDAV vs
> remoteStorage ?
>
That is a good way to discuss, but I think we should not saying the pros or
cons, it's just about applicability here. A discussion about whether it is
suitable to build on WebDAV is better.

Regards,
Linhui

> *发送时间:* 2015年12月3日 星期四
> *收件人:* [email-masked]
> *抄送:* "Linhui Sun" <[email-masked]>, storagesync <
> [email-masked]>, fkooman <[email-masked]>, "Hugo González
> Labrador" <[email-masked]>

---

From: Markus Unterwaditzer <[email-masked]>
Date: Thu, 3 Dec 2015 13:16:48 +0100

> library situation looks a lot brighter :)

WebDAV's syntax is excessively verbose to allow non-standard properties on
files and folders, non-standard querying methods for `calendar-multiget`,
PROPFIND and friends, and etc. The whole point of XML is extensibility at the
cost of verbosity.

If you take that away from your newly defined subset of WebDAV, what is left? A
new protocol which 

- requires extra requests for creating and deleting folders (while there's
  nothing left that would speak against implicitly creating those). Sure you
  have GET, PUT, DELETE for documents

- has excessively verbose syntax for listing folders, but without the advanced
  filtering capabilities.

- is not backwards-compatible with most client implementations and therefore
  has to be rebranded as a new protocol so users don't try to plug a "full"
  WebDAV client into a server that only implements your subset

Especially that last one. If you're going to slice off 90% of WebDAV's
features, you might as well define a new, clean protocol.

---

From: Ted Lemon <[email-masked]>
Date: Thu, 03 Dec 2015 14:38:05 +0000

> to detect changes is not a good way in my view.

WebDAV is HTTP.   However, your point about propfind is correct--that doesn't scale.   What you want is a separate layer for dealing with synchronization of metadata.   You could use HTTP transport to do this, but not propfind.

--
Sent from Whiteout Mail - https://whiteout.io

My PGP key: https://keys.whiteout.io/[email-masked]

---

From: Jakub Moscicki <[email-masked]>
Date: Thu, 3 Dec 2015 18:41:37 +0000

> WebDAV is HTTP.   However, your point about propfind is correct--that doesn't scale.   

Could you please explain why you think it does not scale? There are schemes in which you do not need to propfind the entire remote tree to discover changes, only a sequence of propfinds in a direct path leading to a changed entity. That’s not optimal for sure (several calls needed, depending on the depth of the change) but if it comes to WebDAV itself, apart from a bloated XML representation, I do not see much problem.

> What you want is a separate layer for dealing with synchronization of metadata.   You could use HTTP transport to do this, but not propfind.

Okey, but what you need to remember the client state and pass it on to the server to calculate the diff state. Any good ideas how to do that?

kuba

--

> My PGP key: https://keys.whiteout.io/[email-masked]
> Storagesync mailing list
> [email-masked]

---

From: Ted Lemon <[email-masked]>
Date: Thu, 03 Dec 2015 19:14:07 +0000

> Could you please explain why you think it does not scale? There are schemes in which you do not need to propfind the entire remote tree to discover changes, only a sequence of propfinds in a direct…

It doesn't scale because you've just described a lockstep process for locating changes; instead of there being one transaction to get the set of changes from a known common version, you have multiple steps, and the number of steps increases at least logarithmically (I am guessing, since I don't know your algorithm) as the number of changes since last sync increases or the size of the tree increases.   If you are syncing continuously this may not be a big deal, but I don't think that's a reasonable assumption.

This also has the problem that you have no separation of metadata and data, and no versioning, which means that you have to have a client-server model, and you can't have a distributed peer model.

> Okey, but what you need to remember the client state and pass it on to the server to calculate the diff state. Any good ideas how to do that?

Keep the metadata sorted (e.g. by modification time), and do a diff whenever a new version is generated.   Versions that have never been shared with other peers should be trimmed once they are not the head version.   When peer A wants to sync with peer B, it announces the version it has and the version it last remembers syncing with peer B.   If Peer B remembers that version, then the difference can be computed trivially, O(n), and is minimal (n*2 at worst), where n is the number of changes.   If Peer B has lost its memory, then it has to send the complete inventory of its head version, which Peer A can then compare with its head to see what's changed.

---

From: "Fei Song" <[email-masked]>
Date: Fri, 4 Dec 2015 09:40:16 +0800

Hi

Sorry for the late reply, Please see it inline~

--------------
Fei Song
>>>>> of the use cases of the? WebDAV specification.
>>>>>
>>>>> developer friendly and it is faster to develop.?Maybe the
>micro-service based architectures, the deployment of ?such highly

Indeed！This is why I said the synchronization should be mentioned at the abstract.
Otherwise, it seems that the remoteStorage draft only focus on browser-based storage.

>>>>> > send a message with subject or body 'help' to? ? ? ?storagesync-
>>>>> > [email-masked]
>>>>> >
>>>>> > You can reach the person managing the list at? ? ? ?storagesync-
>>>>> > [email-masked]
>>>>> >
>>>>> >1. New version of draft-dejong-remotestorage? ? Internet-Draft
>>>>> >available (Michiel de Jong)? ? 2. Re: New version of draft-dejong-
>>>>> >remotestorage Internet-Draft? ? ? ?available (Gihan Dias)? ? 3.
>>>>> >   Draft available? ?2k (message/rfc822)
>>>>> >   Draft available? ?1k (message/rfc822)
>>>>> >   Draft available? ?2k (message/rfc822)
>>>>>

---

From: "Fei Song" <[email-masked]>
Date: Fri, 4 Dec 2015 09:47:16 +0800

Hi Fran?ois,

--------------
Fei Song
>impossible or unwanted...

For your last comment, I wonder do we already have a limited version of WebDAV? 

>Fran?ois

---

From: "Fei Song" <[email-masked]>
Date: Fri, 4 Dec 2015 09:49:03 +0800

I think that would be more efficient, thanks Hugo~

--------------
Fei Song
>>>>>>> about the need for? MOVE verb. Application-level chunking is
>>>>>>>>>>>>>>>>> addresses a subset of the use cases of the? WebDAV
>>>>>>>>>>>>>>>>> specification.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>> the deployment of ?such highly complex systems should
>>>>>>>>>>>>>>>>> > to? ? ? ?[email-masked]
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> >1. New version of draft-dejong-remotestorage? ? Internet-
>>>>>>>>>>>>>>>>> >Draft? ? ? ?available (Michiel de Jong)? ? 2. Re: New
>>>>>>>>>>>>>>>>> >available (Gihan Dias)? ? 3. Re: New version of draft-dejong-
>>>>>>>>>>>>>>>>> >remotestorage Internet-Draft? ? ? ?available (Fei
>>>>>>>>>>>>>>>>> >Song)
>>>>>>>>>>>>>>>>> >   remotestorage Internet-Draft available? ?2k
>>>>>>>>>>>>>>>>> >   (message/rfc822)
>>>>>>>>>>>>>>>>> >   remotestorage Internet-Draft available? ?1k
>>>>>>>>>>>>>>>>> >   (message/rfc822)
>>>>>>>>>>>>>>>>> >   remotestorage Internet-Draft available? ?2k
>>>>>>>>>>>>>>>>> >   (message/rfc822)
>>>>>>>>>>>>>>>>>

---

From: "Fei Song" <[email-masked]>
Date: Fri, 4 Dec 2015 09:55:39 +0800

Hi Hugo,

Just a quick comment, comparison is great. 
Do you think we should launch something like "WebDAV vs remoteStorage" or "advantages vs disadvantages of WebDAV".
or both of them?

--------------
Fei Song

---

From: "Fei Song" <[email-masked]>
Date: Fri, 4 Dec 2015 10:06:18 +0800

Good suggestion, "Pros and Cons" is not a good option for discussion in current phase.

So until now we have
1. Whether it is suitable to build on WebDAV
2. WebDAV vs remoteStorage
3. Advantages vs disadvantages of WebDAV
Maybe option 1 is the result of option 3, just like "Pros and Cons" is the result of option 1.

What do you think, Hugo?

--------------
Fei Song

---

From: "Fei Song" <[email-masked]>
Date: Fri, 4 Dec 2015 11:01:45 +0800

lol, very good pic. So the responsibility of this remoteStorage draft is to be a useful and specific 15th. 

--------------
Fei Song
>On Thu, Dec 3, 2015 at 10:44 AM, Fran?ois Kooman <[email-masked]> wrote:
>

---

From: "Fei Song" <[email-masked]>
Date: Fri, 4 Dec 2015 11:07:31 +0800

Hi

--------------
Fei Song
>grail: federation. Is this not solved by Signal nowadays?

+1, more applications and requirements appear, more clients will be created.
However, these new clients will trigger the process of standard design as well and hope new standard could reduce the workloads

---

From: Linhui Sun <[email-masked]>
Date: Fri, 4 Dec 2015 11:53:23 +0800

2015-12-04 3:14 GMT+08:00 Ted Lemon <[email-masked]>:

> > Could you please explain why you think it does not scale? There are
> schemes in which you do not need to propfind the entire remote tree to
> discover changes, only a sequence of propfinds in a direct path leading to
> a changed entity. That’s not optimal for sure (several calls needed,
> depending on the depth of the change) but if it comes to WebDAV itself,
> apart from a bloated XML representation, I do not see much problem.
>
> It doesn't scale because you've just described a lockstep process for
> locating changes; instead of there being one transaction to get the set of
> changes from a known common version, you have multiple steps, and the
> number of steps increases at least logarithmically (I am guessing, since I
> don't know your algorithm) as the number of changes since last sync
> increases or the size of the tree increases.   If you are syncing
> continuously this may not be a big deal, but I don't think that's a
> reasonable assumption.
>
Agree. In my view, the key problem is not about how to reduce the number of
propfinds but how to come up with an improved method to replace the
existing pattern.

>
> This also has the problem that you have no separation of metadata and
> data, and no versioning, which means that you have to have a client-server
> model, and you can't have a distributed peer model.
>
Separation of metadata and data is better. But I don't understand why we
need a distributed peer model. C/S model could still work fine with such
separation.

>
> > Okey, but what you need to remember the client state and pass it on to
> the server to calculate the diff state. Any good ideas how to do that?
>
> Keep the metadata sorted (e.g. by modification time), and do a diff
> whenever a new version is generated.   Versions that have never been shared
> with other peers should be trimmed once they are not the head version.
>  When peer A wants to sync with peer B, it announces the version it has and
> the version it last remembers syncing with peer B.   If Peer B remembers
> that version, then the difference can be computed trivially, O(n), and is
> minimal (n*2 at worst), where n is the number of changes.   If Peer B has
> lost its memory, then it has to send the complete inventory of its head
> version, which Peer A can then compare with its head to see what's changed.

---

From: Ted Lemon <[email-masked]>
Date: Fri, 04 Dec 2015 16:29:31 +0000

> need a distributed peer model. C/S model could still work fine with such

This is correct.   I want the distributed peer model because it's a better model, not because the client/server model wouldn't work.

---

From: Markus Unterwaditzer <[email-masked]>
Date: Fri, 4 Dec 2015 19:11:10 +0100

On Thu, Dec 03, 2015 at 02:38:05PM +0000, Ted Lemon wrote:
> WebDAV is HTTP.   However, your point about propfind is correct--that doesn't
> scale.   What you want is a separate layer for dealing with synchronization
> of metadata.   You could use HTTP transport to do this, but not propfind.

We do have that separate layer in the form of
https://www.ietf.org/rfc/rfc6578.txt and so far only a handful of FOSS servers
have implemented it. The fully-interopable way to synchronize servers still
includes a fallback to a full PROPFIND.

ETags on folder listings are arguably simpler to implement than a partial
transaction history. remoteStorage requires those, and consequently all servers
must implement it.

---

From: Markus Unterwaditzer <[email-masked]>
Date: Fri, 4 Dec 2015 19:18:50 +0100

> complex systems should become easier and faster.

It is not exclusively targeted at browsers, although the browser's Origin-based
security model is required for sane client identification. You can totally
access remoteStorage accounts using desktop apps, a proof-of-concept exists at
https://github.com/untitaker/vdirsyncer.

Synchronization is a completely unrelated topic. I don't see how it's related
to whether the protocol is targeted at browser applications, in either case
offline sync has to work. And it does.

I don't understand what you mean with "linux container micro-service based
architectures", or how it is related to this discussion.

-- Markus

---

From: Ted Lemon <[email-masked]>
Date: Fri, 04 Dec 2015 19:00:54 +0000

> includes a fallback to a full PROPFIND.

Oh wow, thank you.   I hadn't seen this RFC before.   This is definitely closer to what I want.   I'll have to look at it in a bit more detail to see if I can make it work for my application, but if it wouldn't work exactly as is, it's certainly a big step in the right direction.

> transaction history. remoteStorage requires those, and consequently all servers

I think etags for folders would be a little painful other than for a read-only client.

---

From: Hugo González Labrador <[email-masked]>
Date: Fri, 04 Dec 2015 21:30:21 +0100

> https://github.com/untitaker/vdirsyncer.

That's great to hear. If this is achievable maybe will be good to
clarify this aspect in the spec.

>From the abstract

    This draft describes a protocol by which client-side applications,
    running inside a web browser, can communicate with a data storage
    server that is hosted on a different domain name. This way, the
    provider of a web application need not also play the role of data
    storage provider. The protocol supports storing, retrieving, and
    removing individual documents, as well as listing the contents of an
    individual folder, and access control is based on bearer tokens.

it is difficult to infer that the spec is also targeted to non-web apps
as there are no mentions to server-side or desktop apps.

As far as I know the OAuth authentication flow works in client-side web
applications, have you managed to use such mechanism in your non-web
project ? That would be a really interesting achievement. Maybe this is
the cause why remoteStorage does not mention non-web apps in their
abstract.

Summoning Michiel to clarify this aspect.

> Synchronisation is a completely unrelated topic. I don't see how it's
> related

That was said to give an example of how using Docker Containers can help
to manage non-monolithic applications. Of course, that is not directly
related to sync.

---

From: Markus Unterwaditzer <[email-masked]>
Date: Fri, 4 Dec 2015 22:14:11 +0100

> as there are no mentions to server-side or desktop apps.

(Apologies to skippe and others who've had this discussion with me already)

I'll have to back-paddle a bit.

Yes, it's possible. It is possible AND secure if the server requires stricter
methods for app identification, which is stated in the spec as a MAY. The spec
does focus on web applications, and they're the supported usecase.

The way I've made my particular desktop application work is sort-of a hack, but
may be less fragile if the application has full control over the browser window
(e.g. WebView on Android, or bundled webkit browser for desktop apps)

---

From: Hugo González Labrador <[email-masked]>
Date: Fri, 04 Dec 2015 22:28:32 +0100

> (e.g. WebView on Android, or bundled webkit browser for desktop apps)

Okay, I see. So, web applications are the supported use case by
remoteStorage but there are alternatives, as you have explained, that
make it possible to connect from non web apps using embedded browsers. 

I would like to hear Michiel opinion on this subject to see if
remoteStorage will have non web apps as a supported use case in the
future or if it is out of scope.

Best,

Hugo

---

From: Sebastian Kippe <[email-masked]>
Date: Fri, 4 Dec 2015 22:24:18 +0000

Hi,

> the cause why remoteStorage does not mention non-web apps in their

There are many programs doing this in some way today, usually involving
a Web View component of some kind. Check out Gnome Online Accounts for
example. Or many Android apps. In fact, the remoteStorage.js library has
support for doing it in Cordova apps via the in-app-browser plugin.

Cheers,
Sebastian

---

From: Linhui Sun <[email-masked]>
Date: Sat, 5 Dec 2015 13:28:47 +0800

2015-12-05 0:29 GMT+08:00 Ted Lemon <[email-masked]>:

> This is correct.   I want the distributed peer model because it's a better
> model, not because the client/server model wouldn't work.
>
I'm not very familiar with the distributed peer model (I view it as p2p,
please correct me if I am wrong). From the example you have provided
previously, it seems that it works fine with 2 participants. But I don't
know whether it works well when multiple participants are willing to share
a file/folder. I'm asking this question since we need to know which model
should the iss probably employ.

---

From: Ted Lemon <[email-masked]>
Date: Sat, 05 Dec 2015 05:47:40 +0000

> should the iss probably employ.

The distributed peer model simply means that there is no server.   There are only instances of the database.   Any instance can be modified, and the changes will be propagated to other instances when they sync.   A change made on one instance may propagate to another instance through a third instance.

Importantly, this avoids the situation where the metadata is lost on the server, and as a consequence the data is all removed from the client.   Since there is no server, and no client, but only peers, neither peer is considered authoritative, so a metadata loss on either peer can result in a safe recovery without data loss.

In practice, in most cases there will be one or more instances that are treated as servers, and then one or more that are treated more as clients, but now you can have two or three "servers" and it doesn't matter which one the "client" syncs to; the new information will propagate to the other two "servers."

---

From: Linhui Sun <[email-masked]>
Date: Sat, 5 Dec 2015 14:09:08 +0800

2015-12-05 13:47 GMT+08:00 Ted Lemon <[email-masked]>:

> The distributed peer model simply means that there is no server.   There
> are only instances of the database.   Any instance can be modified, and the
> changes will be propagated to other instances when they sync.   A change
> made on one instance may propagate to another instance through a third
> instance.
>
> Importantly, this avoids the situation where the metadata is lost on the
> server, and as a consequence the data is all removed from the client.
>  Since there is no server, and no client, but only peers, neither peer is
> considered authoritative, so a metadata loss on either peer can result in a
> safe recovery without data loss.
>
> In practice, in most cases there will be one or more instances that are
> treated as servers, and then one or more that are treated more as clients,
> but now you can have two or three "servers" and it doesn't matter which one
> the "client" syncs to; the new information will propagate to the other two
> "servers."
>
It sounds like this could resolve your IMAP issue to some extent. I'm
wondering could we understand this in another way for the practical
application: we have a group of servers and a group of clients. Inside the
group of servers, they use a explicit distributed peer model and any new
info will be propagated to the group members. Between the server group and
client group, it is simply C/S model but a client can sync to arbitrary one
from the server group. However inside the group of clients, they have no
communications since I don't want my client to talk to anyone else.

---

From: Ted Lemon <[email-masked]>
Date: Sat, 05 Dec 2015 14:56:04 +0000

> communications since I don't want my client to talk to anyone else.

Any two devices that share the same server and can update it, whether they use a peer-to-peer protocol to talk to the server or a client-server protocol, are communicating.   However, if you just mean that you don't want to do the work to set up the communication, but you want to use the same protocol in all cases, then yes, this is a good example of a configuration that could work with the peer-to-peer sync model: what peer-to-peer means in this context is simply that it in principle for any two participants can communicate, not that they are configured so that they _actually_ communicate.

---

From: Markus Unterwaditzer <[email-masked]>
Date: Sun, 06 Dec 2015 00:44:09 +0100

>I think etags for folders would be a little painful other than for a

I don't see what's wrong with it.
-- 
Sent from my phone. Please excuse my brevity.

---

From: Linhui Sun <[email-masked]>
Date: Sun, 6 Dec 2015 10:37:45 +0800

> I don't see what's wrong with it.
IMO, etag is designed for client side cache. A typical usage: if the file
on the server has no actual change, the client will use the cache and the
server does not need to send the file content again. While for a storage
service, we also need some some similar mechanism at server side. In this
way, the client do not need to upload unchanged content to the server.

---

From: "Fei Song" <[email-masked]>
Date: Sun, 6 Dec 2015 12:14:08 +0800

Hi,

--------------
Fei Song
>transaction history. remoteStorage requires those, and consequently all servers

ah, do you think it will trigger much extra burdens for the servers?

---

From: "Fei Song" <[email-masked]>
Date: Sun, 6 Dec 2015 12:20:42 +0800

--------------
Fei Song
>offline sync has to work. And it does.

That is why it should be clarified at the beginning of document...emphasize the target usage scenarios.

---

From: "Fei Song" <[email-masked]>
Date: Sun, 6 Dec 2015 17:34:51 +0800

+1

--------------
Fei Song

---

From: "Fei Song" <[email-masked]>
Date: Sun, 6 Dec 2015 18:13:47 +0800

Hi Sebastian

--------------
Fei Song
>support for doing it in Cordova apps via the in-app-browser plugin.

Thanks for sharing. It is great to know these programs.
The extension usage of remoteStorage library might be out of scope of this draft.

---

From: Markus Unterwaditzer <[email-masked]>
Date: Sun, 6 Dec 2015 18:36:46 +0100

> way, the client do not need to upload unchanged content to the server.

There is no "true-and-only" usecase for etags, and no, you do not need anything
more than etags for synchronization. Using a additional "metadata" file, the
client can cheaply determine whether local content changed and whether server
content changed, and that's all that is required for (in this case) file
synchronization. Sync conflicts are a different story, and the solution to that
heavily depends on the kind of data you're syncing.

---

From: Markus Unterwaditzer <[email-masked]>
Date: Sun, 6 Dec 2015 18:39:36 +0100

> Hi,

What do you mean by "it"? ETags are simpler to implement than the above RFC,
and ETags are also required in WebDAV.

- WebDAV: Etags AND a sync protocol
- remoteStorage: just etags

The latter is simpler, although you'll have to fetch the full folder listing
for each sync when something in that folder changed.

---

From: Ted Lemon <[email-masked]>
Date: Sun, 06 Dec 2015 18:26:54 +0000

> and ETags are also required in WebDAV.

Etags are not really much simpler than a metadata versioning sync protocol, and they are a lot less efficient.   Having to fetch the whole directory for every directory that contains a change is expensive, both for the server and the client, unless changes are infrequent.   And because the server is the sole source of truth, it's a lot more fragile.

---

From: Markus Unterwaditzer <[email-masked]>
Date: Sun, 6 Dec 2015 22:54:58 +0100

Etags are *much* simpler to implement on the server side than the WebDAV sync
protocol because they do not require the server to keep sync tokens and
metadata history. Note that the server has to keep metadata of *deleted* files
to correctly implement WebDAV's sync protocol. Etags are less efficient in
terms of network usage of course, but that's the tradeoff to be made.

I don't know what you mean with the last sentence.

---

From: Linhui Sun <[email-masked]>
Date: Mon, 07 Dec 2015 08:17:26 +0800

There is no "true-and-only" usecase for etags, and no, you do not need anything
more than etags for synchronization. Using a additional "metadata" file, the
client can cheaply determine whether local content changed and whether server
content changed, and that's all that is required for (in this case) file
synchronization. Sync conflicts are a different story, and the solution to that
heavily depends on the kind of data you're syncing. I'm a little bit confused here: if I have a metadata file, why do I still need
the etag? The metadata file should contain something equivalent to the etag.

---

From: Markus Unterwaditzer <[email-masked]>
Date: Mon, 7 Dec 2015 01:20:20 +0100

You need to store the etags from the last sync to see which ones have changed.
I'm doing this in vdirsyncer (https://github.com/untitaker/vdirsyncer), a
remoteStorage and CalDAV/CardDAV sync client and probably should do a
high-level blogpost about it.

---

From: Linhui Sun <[email-masked]>
Date: Mon, 07 Dec 2015 08:31:55 +0800

On 周一, 12月 7, 2015 at 08:20, Markus Unterwaditzer <[email-masked]>
wrote:
You need to store the etags from the last sync to see which ones have changed. The server only need to know the latest version in my view. They don't need to
be able to identify last sync or change, just to check whether metadata file is
latest is enough.
I'm doing this in vdirsyncer (https://github.com/untitaker/vdirsyncer), a
remoteStorage and CalDAV/CardDAV sync client and probably should do a
high-level blogpost about it.

On Mon, Dec 07, 2015 at 08:17:26AM +0800, Linhui Sun wrote:
> heavily depends on the kind of data you're syncing. I'm a little bit confused
here: if I have a metadata file, why do I still need

---

From: Markus Unterwaditzer <[email-masked]>
Date: Mon, 7 Dec 2015 01:38:10 +0100

On Mon, Dec 07, 2015 at 08:31:55AM +0800, Linhui Sun wrote:
> The server only need to know the latest version in my view. They don't need to

The metadata file contains all etags *for all files* from the previous sync. It
is stored on the clientside and is never sent to the server.

---

From: Linhui Sun <[email-masked]>
Date: Mon, 07 Dec 2015 08:50:00 +0800

On 周一, 12月 7, 2015 at 08:38, Markus Unterwaditzer <[email-masked]>
wrote:

The metadata file contains all etags *for all files* from the previous sync. It
is stored on the clientside and is never sent to the server. So the local metadata file is for client to decide whether to send a content.
But what if the file has been changed during two sync processes by another
participant who also shares the file.

---

From: Markus Unterwaditzer <[email-masked]>
Date: Mon, 7 Dec 2015 01:54:26 +0100

> participant who also shares the file.

Those are sync conflicts, how to deal with then depends on the kind of data you
deal with.

---

From: Linhui Sun <[email-masked]>
Date: Mon, 7 Dec 2015 09:09:08 +0800

> Those are sync conflicts, how to deal with then depends on the kind of
That works, but do you think this could make the things complicated? It
sounds like different situations have different mechanisms. Why not just
make the metadata exchanged, any disadvantage for doing that?

> > > > There is no "true-and-only" usecase for etags, and no, you do not
> need
> > > anything
> > > > client can cheaply determine whether local content changed and
> whether
> > server
> > > > synchronization. Sync conflicts are a different story, and the
> solution to
> > > that

---

From: Ted Lemon <[email-masked]>
Date: Mon, 07 Dec 2015 01:20:48 +0000

> terms of network usage of course, but that's the tradeoff to be made.

There's nothing hard about keeping metadata history.

> I don't know what you mean with the last sentence.

If you use etags, there's no way to tell whether a file disappeared from the server because it was deleted or because the server got damaged.

---

From: Markus Unterwaditzer <[email-masked]>
Date: Mon, 7 Dec 2015 02:22:53 +0100

> make the metadata exchanged, any disadvantage for doing that?

I don't know what that means.

---

From: Ted Lemon <[email-masked]>
Date: Mon, 07 Dec 2015 01:31:57 +0000

> is stored on the clientside and is never sent to the server.

This may be how your system works, but what I want is for the each peer to remember every version that it has shared with another peer.   This allows two peers to agree on a common version and send only the metadata that's changed since that version, rather than sending a complete metadata snapshot or, as I think would be required in your case, Markus, scanning the WebDAV server to see what's changed.

---

From: Ted Lemon <[email-masked]>
Date: Mon, 07 Dec 2015 01:35:39 +0000

> make the metadata exchanged, any disadvantage for doing that?

Right.   Conflict resolution is a layer above syncing.   Could be automatic, could require human intervention (although that's best avoided if possible).   Ideally you have a way to go back and undo a change if something important was lost or overwritten.   This is another reason why keeping a versioned metadata store is good.

---

From: Linhui Sun <[email-masked]>
Date: Mon, 07 Dec 2015 10:05:56 +0800

On 周一, 12月 7, 2015 at 09:22, Markus Unterwaditzer <[email-masked]>
wrote:
> make the metadata exchanged, any disadvantage for doing that?

I don't know what that means. The question is why you just keep the metadata locally at client side but not
exchange the metadata between client and server to make both of them (both
peers) have the latest metadata store in time.

---

From: Linhui Sun <[email-masked]>
Date: Mon, 07 Dec 2015 10:16:17 +0800

On 周一, 12月 7, 2015 at 09:35, Ted Lemon <[email-masked]> wrote:
> make the metadata exchanged, any disadvantage for doing that?

Right. Conflict resolution is a layer above syncing. Could be automatic, could
require human intervention (although that's best avoided if possible). Concurrent conflict (e.g. Real-time edit) might/should require human
intervention. But the non-concurrent conflict (e.g. modification caused by
different peers but not at the same time) should be automatically resolved by
the system. Ideally you have a way to go back and undo a change if something important was
lost or overwritten. This is another reason why keeping a versioned metadata
store is good.

---

From: Ted Lemon <[email-masked]>
Date: Mon, 07 Dec 2015 02:27:25 +0000

> lost or overwritten. This is another reason why keeping a versioned metadata

Nope, this doesn't always work.   E.g., peer A is acting as the central server, and peers B and C are disconnected, and while disconnected, both modify file F.   When they reconnect, it's not the case that the later modification to file F can be considered to supersede the earlier modification.   Because both clients modified the same version of the file, what you probably want is for the changes that B made and the changes that C made to be merged.   Often this can be done automatically, but not if both changes are to the same part of the file, and are not the same change.

Merges can also be a problem if there are cross-file dependencies.

So the bottom line is that you do need to be able to run a conflict resolution process after an update.  But it's not part of the update--it's a separate step, which may result in _another_ update once it's completed.

---

From: "Fei Song" <[email-masked]>
Date: Mon, 7 Dec 2015 11:17:06 +0800

--------------
Fei Song
>Nope, this doesn't always work.   E.g., peer A is acting as the central server, and peers B and C are disconnected, and while disconnected, both modify file F.   When they reconnect, it's not the ca…

Can we just treat the requests sent by peer B and peer C based on their timestamps (arriving at the server).
If the file must be kept as the same version, different modification reports can be generated independently with FCFS mode.
If this can not be finished automatically, a human intervention will be needed.

>Merges can also be a problem if there are cross-file dependencies.

indeed.

---

From: Ted Lemon <[email-masked]>
Date: Mon, 07 Dec 2015 03:40:15 +0000

> If this can not be finished automatically, a human intervention will be needed.

No, this doesn't work even if you don't care about wiping out changes.   Think about this sequence:

0: File F is at version 0
1: Client A syncs to server
2: Client B syncs to server
3: File f changed on Client A -> Version 0.A
4: File f changed on Client B -> Version 0.B
5: Client B syncs to server, updating file F to version 0.B, modified time = 4
6: Client A syncs to server: what happens?
   a: File updated to version 0.A, modified time=3
   b: Server overwrites version 0.A with version 0.B, changes in 0.A lost
   c: Conflict resolution combines versions 0.A and 0.B, producing version 1, mod time=6

I think C is the only right answer.   We want conflict resolution to be automatic whenever possible, but we do want to do conflict resolution, and not just wipe out changes, because those changes may contain real work that the user on client A did.

---

From: Ted Lemon <[email-masked]>
Date: Mon, 07 Dec 2015 04:02:20 +0000

> These "middle" changes are important. Yeah. they should be stroed at the server side.
> Wiping out changes is just a simple appoach.
> I also think c is a good option. Then we need to consider a suitable time window.
> e.g. how long should be consider as a "simultaneous" updating.
> otherwise F might be updated as Version 1 directly.

In option C, the time doesn't matter at all.   It's the fact that the two versions are different that triggers the conflict resolution process.   So there's no timing race to worry about: the first client to update just succeeds, and the second client to update has to do conflict resolution.   Doesn't matter which client updates first, or when the updates happen.

---

From: Linhui Sun <[email-masked]>
Date: Mon, 7 Dec 2015 14:45:24 +0800

2015-12-07 11:40 GMT+08:00 Ted Lemon <[email-masked]>:

> > If the file must be kept as the same version, different modification
> reports can be generated independently with FCFS mode.
> I think C is the only right answer.   We want conflict resolution to be
> automatic whenever possible, but we do want to do conflict resolution, and
> not just wipe out changes, because those changes may contain real work that
> the user on client A did.
>
Do you mean you want the conflict resolution to be performed every time? If
so, I think this might be a little bit unnecessary since a version conflict
is not very frequently happened (especially for some personal users). The
case you mentioned should definitely be resolved and such offline-to-online
switch could be treated as concurrent conflict in my view.

But a more frequent case is that people update their file just to replace
the previous one, even though there are multiple people working on the same
file. In this case, replacing file according to modification time seems
reasonable. So the key point is how to justify which two/more versions
should trigger the conflict resolution to avoid wiping out real work.

As for the conflict resolution itself, it is very hard to achieve since the
system needs to handle different types of file. GoogleDocs performs well
since it only focuses on the documents. But for a storage service, we don't
know what else types of file will be stored. A popular way I've seen is
just to keep all the conflicted versions (named by different peers) in the
storage.

---

From: "Fei Song" <[email-masked]>
Date: Mon, 7 Dec 2015 15:28:31 +0800

--------------
Fei Song
>switch could be treated as concurrent conflict in my view.

For the collaborative working, perhaps we have to do that.

>should trigger the conflict resolution to avoid wiping out real work.

Sure, in un-collaborative case (just for your own file), it's OK.
When multiple people working on the same file (like a source code file), some work should be saved even they are not included in the final version.
This is indeed useful and may contribute as well.

>just to keep all the conflicted versions (named by different peers) in the

I agree! The target of ISS, I think, should also consider different type of files: documents, video, audio, databases, etc.
How to do the sync among them is very challenging and significant.

---

From: Markus Unterwaditzer <[email-masked]>
Date: Mon, 7 Dec 2015 13:33:20 +0100

> The question is why you just keep the metadata locally at client side but not

I think we have a misunderstanding here.

We have a server storage that has an etag for each file. We have a client
storage that has an etag for each file.

The sync client accesses those *and* a "status" file that contains a list of
all files + etags for *both* sides, as they were after the previous sync.  This
is similar to the algorithm described in
http://blog.ezyang.com/2012/08/how-offlineimap-works/, except that one doesn't
deal with files being modified. As said, this warrants a longer blogpost, but I
don't have the time to write one now.

---

From: Markus Unterwaditzer <[email-masked]>
Date: Mon, 7 Dec 2015 13:36:23 +0100

Note that this is not specific to remoteStorage. This is how WebDAV sync
clients work in practice (not via the sync protocol, which is not popular
enough), it is how remoteStorage sync clients work in practice. At least in
terms of the kind of data they access from server and client, the exact
algorithm may be different.

---

From: Ted Lemon <[email-masked]>
Date: Mon, 07 Dec 2015 17:57:42 +0000

> switch could be treated as concurrent conflict in my view.

It really depends on the use case.

> should trigger the conflict resolution to avoid wiping out real work.

This use case is a common use case for folders that are used informally as a way to transfer files between users.   Typically each version of the file actually has some kind of version in the filename -- e.g., "SoW 2015/11/1 10am Ted" and we just expect the users to manage versions.   In this use model, you definitely don't need to do anything fancy.

However, this is a really broken use model, which exists because the tools don't work well enough to do something sensible.   They don't allow you to track versions, they don't allow multiple committers, and they don't have a mechanism for resolving conflicts when two people change the same version of the file and try to upload the change.

> just to keep all the conflicted versions (named by different peers) in the

I think this is a valid conservative way of approaching the problem.  But what makes sense to me is that the sync service be able to identify conflicts, and that there be a conflict resolution process at a layer above the sync service.   The conflict resolution process could do something trivial like resolving the conflict by making a new file with the same name plus a well-understood notation like "conflict Ted/Sun 15/11/7". 

This would be similar to what people would do if there were no conflict resolution layer.   But if you have a file type for which there is already an automatic conflict resolution process, then the conflict resolution process can just do that resolution.   There's no reason to pick a one-size-fits-all approach to this problem.   What matters is the ability to know that a conflict has occurred based on the versioning information in the metadata.

---

From: Linhui Sun <[email-masked]>
Date: Tue, 8 Dec 2015 17:06:17 +0800

2015-12-08 1:57 GMT+08:00 Ted Lemon <[email-masked]>:

> This use case is a common use case for folders that are used informally as
> a way to transfer files between users.   Typically each version of the file
> actually has some kind of version in the filename -- e.g., "SoW 2015/11/1
> 10am Ted" and we just expect the users to manage versions.   In this use
> model, you definitely don't need to do anything fancy.
>
> However, this is a really broken use model, which exists because the tools
> don't work well enough to do something sensible.   They don't allow you to
> track versions, they don't allow multiple committers, and they don't have a
> mechanism for resolving conflicts when two people change the same version
> of the file and try to upload the change.
>
Yes, and this use case should be considered.

> I think this is a valid conservative way of approaching the problem.  But
> what makes sense to me is that the sync service be able to identify
> conflicts, and that there be a conflict resolution process at a layer above
> the sync service.   The conflict resolution process could do something
> trivial like resolving the conflict by making a new file with the same name
> plus a well-understood notation like "conflict Ted/Sun 15/11/7".
>
> This would be similar to what people would do if there were no conflict
> resolution layer.   But if you have a file type for which there is already
> an automatic conflict resolution process, then the conflict resolution
> process can just do that resolution.   There's no reason to pick a
> one-size-fits-all approach to this problem.   What matters is the ability
> to know that a conflict has occurred based on the versioning information in
> the metadata.
>
That's better. It seems that a separate layer for conflict resolution is
well accepted. But for etag and versioning metadata, which to employ still
needs to be discussed. Any comments for that?

---

From: Ted Lemon <[email-masked]>
Date: Tue, 08 Dec 2015 16:54:01 +0000

> needs to be discussed. Any comments for that?

Both can be made to work in theory, but etag is missing explicit versioning, so in order to get conflict detection and avoid amnesiac servers erasing data on clients, you pretty much need to build up a metadata versioning system anyway.

---

From: Markus Unterwaditzer <[email-masked]>
Date: Tue, 8 Dec 2015 19:59:22 +0100

On Tue, Dec 08, 2015 at 04:54:01PM +0000, Ted Lemon wrote:
> Both can be made to work in theory, but etag is missing explicit versioning,
> so in order to get conflict detection and avoid amnesiac servers erasing data
> on clients, you pretty much need to build up a metadata versioning system
> anyway.

Per-file conflicts can be detected with just etags. In the case of file-sync
this is often the only thing you can do, since conflict-resolution depends on
the filetype. Dropbox stores both copies in such cases. I'd say this is
reasonably efficient for average usecases.

Something like a three-way merge could also be done with just etags, but
doubles storage usage on the clientside.

---

From: Ted Lemon <[email-masked]>
Date: Tue, 08 Dec 2015 21:25:37 +0000

> reasonably efficient for average usecases.

Can you explain how per-file conflicts can be detected using etags?

---

From: Ted Lemon <[email-masked]>
Date: Wed, 09 Dec 2015 14:11:02 +0000

Wednesday, Dec 9, 2015 7:22 AM Markus Unterwaditzer wrote:
> By file-conflict I mean just the condition that both remote (the server) and local (the sync client) have changed since the last sync, and it is therefore undecidable which version of the file to use.

Sure.   This takes care of the most trivial case of a conflict, but does not retain enough information to detect interleaved conflicts, where for example one client makes a change based on version X of the file, a second makes a change based on version X+1, the second one commits, and then the first one commits.   In your case, automatic conflict detection would undo changes in the file that had been made between versions X and X+1, because neither the client nor the server has any record of what the sequence of changes was.

---

From: Jakub Moscicki <[email-masked]>
Date: Wed, 9 Dec 2015 14:21:58 +0000

Unless you want to dig into the type of the file and explore it’s internal structure to merge, the best you can do is to make sure a version on the server is generated for each conflict file and these may be easily accessible for manual merge by the interested users. For this etags are sufficient. From this point of view there is no difference between conflicts which are generated because two clients update the file at the same time (so conflicting uploads overlapping in time) and a case of offline client which edits locally outdated version (newer version available meanwhile on the server) and consequently uploading it when online again (so conflicting uploads not overlapping in time). In both cases it looks to me good enough if they end up as versions on the server for manual user merge/revert.

kuba

--

> On 09 Dec 2015, at 15:11, Ted Lemon <[email-masked]> wrote:
>

---

From: Ted Lemon <[email-masked]>
Date: Wed, 09 Dec 2015 14:35:33 +0000

> Unless you want to dig into the type of the file and explore it’s internal structure to merge, the best you can do is to make sure a version on the server is generated for each conflict file and th…

Most files are text files, which are easy to merge, so yes, I do want to be able to dig into the structure of the file.   Even when the file isn't "easy to merge," e.g. a sound file or a drawing, knowing the order of the changes can be very helpful when someone is doing a manual merge.   And you can build merge tools for commonly-merged files; if conflict detection is a feature of the data store, then there's an incentive to build such tools, whereas if everybody is accustomed to dumb data stores that don't provide this feature, there's no point in making merge easy.

The absence of this feature is a serious problem in my workflows; I don't know if your experience is similar.   I don't really see any point in inventing "yet another" storage sync mechanism that doesn't provide this feature.   E.g., do you really want a calendaring system that automatically re-adds files that have been deleted on one client when a second client connects that hasn't seen the deletion?   An address book that backs out updates?   A mail system that undeletes deleted messages, or deletes messages on the client that were lost on the server due to a crash?

The use case of a small collection of large files used entirely sequentially is easy to address, but (a) not very interesting and (b) not actually a common use model.   It _seems_ common because we don't provide anything better, and so people make do with it, but it doesn't actually fit with what they are doing.   If it did, we wouldn't see the proliferation of manually-labeled versions of the same file that is so common in such data stores.

---

From: Ted Lemon <[email-masked]>
Date: Wed, 09 Dec 2015 14:43:09 +0000

BTW, it occurs to me that at CERN you deal a lot in very large data sets which are created once and never modified, because they are records of physical events.  So the idea of merging might not seem all that important, because you will never do it on these data sets.

However, one thing that a good versioning system allows for that I think you should consider important is the ability to avoid accidentally losing data.   When you have really big data archives, one of the things that you want to do for redundancy is keep multiple copies.   If one instance gets corrupted, you want to be able to detect that, and you want to be able to avoid losing other instances of the file if the file is lost from an instance of a folder.   The most reliable way to avoid that is to keep versioning metadata.   Multiple copies of versioning metadata are very useful for forensic analysis when something goes wrong, as well.

Additionally, while the bulk of the actual _data_ that CERN stores is really big files, the little files matter just as much--the work researchers are doing, particularly collaboratively.   Enabling efficient collaboration on articles in progress, enabling effective sharing of code, and so on, all are very important despite representing a tiny percentage of the total data stored.

---

From: "Marc Blanchet" <[email-masked]>
Date: Wed, 09 Dec 2015 09:44:02 -0500

On 9 Dec 2015, at 9:35, Ted Lemon wrote:

>> Unless you want to dig into the type of the file and explore it’s 
>> internal structure to merge, the best you can do is to make sure a 
>> version on the server is generated for each conflict file and these 
>> may be easily accessible for manual merge by the interested users. 
>> For this etags are sufficient. From this point of view there is no 
>> difference between conflicts which are generated because two clients 
>> update the file at the same time (so conflicting uploads overlapping 
>> in time) and a case of offline client which edits locally outdated 
>> version (newer version available meanwhile on the server) and 
>> consequently uploading it when online again (so conflicting uploads 
>> not overlapping in time). In both cases it looks to me good enough if 
>> they end up as versions on the server for manual user merge/revert.
>
> Most files are text files, which are easy to merge,

I don’t agree with that statement. A lot of files nowadays  are 
complex text files in a complex format (docx, odt, …). Not easy to 
merge. Not at all. (I don’t have statistics, but my guess is that 90% 
of shared files are not plain text files)

> so yes, I do want to be able to dig into the structure of the file.   
> Even when the file isn't "easy to merge," e.g. a sound file or a 
> drawing, knowing the order of the changes can be very helpful when 
> someone is doing a manual merge.   And you can build merge tools for 
> commonly-merged files; if conflict detection is a feature of the data 
> store, then there's an incentive to build such tools, whereas if 
> everybody is accustomed to dumb data stores that don't provide this 
> feature, there's no point in making merge easy.
>
> The absence of this feature is a serious problem in my workflows; I 
> don't know if your experience is similar.   I don't really see any 
> point in inventing "yet another" storage sync mechanism that doesn't 
> provide this feature.   E.g., do you really want a calendaring system 
> that automatically re-adds files that have been deleted on one client 
> when a second client connects that hasn't seen the deletion?   An 
> address book that backs out updates?   A mail system that undeletes 
> deleted messages, or deletes messages on the client that were lost on 
> the server due to a crash?

you are giving a good example of something vastly different. Calendering 
is a very well structured and very limited dataset (compared to free 
text) that was made by design with sync considerations. The latest 
version of vcard was also designed with sync considerations, and it took 
a lot of time to get it right, (I can tell you I was the chair of 
vcarddav when we did this). And a vcard is a pretty simple dataset and 
very structured.

So I think your point just shows the inverse. Merging complex, free, not 
structured-limited-dataset files is very very difficult.

I guess your goal is great, but to me, that is for research. So if 
people want to do this work, I would split it into two working groups:
- one on the IRTF side about generic sync-merge algorithms
- one on the IETF side for the protocol components.

Marc.

>
> The use case of a small collection of large files used entirely 
> sequentially is easy to address, but (a) not very interesting and (b) 
> not actually a common use model.   It _seems_ common because we don't 
> provide anything better, and so people make do with it, but it doesn't 
> actually fit with what they are doing.   If it did, we wouldn't see 
> the proliferation of manually-labeled versions of the same file that 
> is so common in such data stores.
>
>
> --
> My PGP key: 
> https://keys.whiteout.io/[email-masked]
> Storagesync mailing list
> [email-masked]

---

From: Ted Lemon <[email-masked]>
Date: Wed, 09 Dec 2015 14:59:29 +0000

Marc, I agree that generalized merge is a research problem.   However, if the underlying protocol doesn't provide metadata that can be used to detect conflicts, there is no way to do the research.   Saying "don't do this obvious thing that will be needed in order to make things better" is a recipe for stasis.   We already have webdav.   If what we want is the feature set that webdav provides, why are we even talking about this?

---

From: Klaas Freitag <[email-masked]>
Date: Wed, 9 Dec 2015 22:13:38 +0100

> Most files are text files, which are easy to merge, so yes, I do want to be able to dig into the structure of the file.   Even when the file isn't "easy to merge," e.g. a sound file or a drawing, k…
Automatically mergeable or not, I don't think resolving conflicts shoudl
be part of the sync protocol. Of course conflicts must be detected and
hooks should exist to plug in code to work with it, but I doubt it
should be part of the sync layer to interpret and resolve them.

And even if there is an automatic resolving of conflicts for some file
types (that is obviously possible if that is wanted from a higher level
point of view) there always remains the possibility of not at all
automatically mergeable conflicts.

I am also not sure about the importance of the sequence of changes, as
there are situations caused for example by offline work, where changes
happen in parallel, these cause a conflict, and it's not really
important which change was before or after. Important is that both
versions can be found again, but not necessarily in timely order.

I do not understand why Kuba said "Etags are sufficient" in this
context, but maybe I lack context because I am late to the party...

regards,

Klaas

---

From: Jakub Moscicki <[email-masked]>
Date: Wed, 9 Dec 2015 21:25:00 +0000

Hello,

Last time I looked at this in cernbox we had more than 1000 file extensions and quite a large fraction of “typical dropbox” types of files (documents of different sorts, latex source files of articles, iWork files, Office files, images, pdfs, source code (!), etc). We also have an increasing number of physics datasets and we will have many more as we provide direct connection via sync/share to the already existing huge collections of these datasets. For the data files at CERN, yes you are right, there is no point of merging them and they quite often are never modified. But I would also tend to disagree with your statement that most of files are text files — as was explained by Marc Blanchet — in general it is not the case, not only for CERN.

Best regards,

kuba

--

> On 09 Dec 2015, at 15:43, Ted Lemon <[email-masked]> wrote:
>

---

From: Ted Lemon <[email-masked]>
Date: Wed, 09 Dec 2015 22:05:17 +0000

> should be part of the sync layer to interpret and resolve them.

I agree, as I said to Marc.   But the sync protocol has to _support_ the accurate detection of conflicts, or there's no way to resolve them, because you don't know about them (until your change has been overwritten, at which point it's too late).

> automatically mergeable conflicts.

Yes, I think we all agree that this is a problem.   Marc mentioned that he thinks it's a research problem.   I think a general solution is a research problem, and not a very interesting one: all I really want is a way to know that the conflict exists and what the sequence of events was; if an automatic merge is possible, that's great, but not required.

> versions can be found again, but not necessarily in timely order.

Not remembering the order in which the changes occurred is an optimization which gains you almost no efficiency and costs you greatly in terms of functionality.

> context, but maybe I lack context because I am late to the party...

Etags are sufficient if you just want to see if what you have is more recent in time than what's on the server.   They don't allow you to detect interleaved updates.

---

From: Ted Lemon <[email-masked]>
Date: Wed, 09 Dec 2015 22:08:29 +0000

Wednesday, Dec 9, 2015 4:25 PM Jakub Moscicki wrote:
> But I would also tend to disagree with your statement that most of files are text files — as was explained by Marc Blanchet — in general it is not the case, not only for CERN.

I think you and Marc misunderstood what I mean by "text files," but that's immaterial.   Whether merges can be accomplished automatically or not, knowing that a conflict has occurred, and when, and why, is (IMHO) important.

---

From: Linhui Sun <[email-masked]>
Date: Thu, 10 Dec 2015 11:31:27 +0800

Hi,

>From the previous messages, I think we all agree that the conflict
resolution should be another layer's problem, whether to enable an
automatic merging might be a research problem or a further use case. But I
think the problem belongs to this list is whether we need to detect more on
the conflicts (not resolving them). Just knowing there are conflicts OR
knowing what are those conflicts, when and where do they come from (e.g.
the sequence of different conflict versions). Personally I prefer the
latter. Just imagine when we reconnect our device and see multiple conflict
versions but don't know anything about that. How to handle them? Which one
should we begin with?

IMHO, this is a discussion on which is enough, etag or metadata? Conflict
detection is just one of the related use cases.

Regards,
Linhui

2015-12-10 6:08 GMT+08:00 Ted Lemon <[email-masked]>:

> > But I would also tend to disagree with your statement that most of files
> are text files — as was explained by Marc Blanchet — in general it is not
> the case, not only for CERN.
>
> I think you and Marc misunderstood what I mean by "text files," but that's
> immaterial.   Whether merges can be accomplished automatically or not,
> knowing that a conflict has occurred, and when, and why, is (IMHO)
> important.
>
>
> --

---

From: Jakub Moscicki <[email-masked]>
Date: Thu, 10 Dec 2015 07:56:47 +0000

Hi,

I tend to agree with what you said above.

I do not understand why Kuba said "Etags are sufficient" in this
context, but maybe I lack context because I am late to the party…

Meaning:  in the view what you said above the opaque etags (e.g. md5 or uuids) should be sufficient as there is no need to reason about the order of changes or version numbers, etc.

[Plug: all people passionate about discussing these things should consider to come to CS3 in January and do it there :-) ]

Best regards,

kuba

--

regards,

Klaas

_______________________________________________
Storagesync mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/storagesync

---

From: "Fei Song" <[email-masked]>
Date: Sat, 12 Dec 2015 21:13:35 +0800

--------------
Fei Song
>Additionally, while the bulk of the actual _data_ that CERN stores is really big files, the little files matter just as much--the work researchers are doing, particularly collaboratively.   Enabling…

I agree that efficient collaboration and other similar works in sync is important.
Here, I got a question. If the data is used for recording physical event and never modified.
Could we just mark "read only" and prohibit updates for all multiple copies after the first sync is finished.
What is the benefit for using versioning metadata system for this case?

---

From: "Fei Song" <[email-masked]>
Date: Sat, 12 Dec 2015 21:34:39 +0800

>vcarddav when we did this). And a vcard is a pretty simple dataset and 

Good to know that~
If it is so hard for such simple and well-structured vcard case, should we handle this issue by seperating file based on their property?
The situation might be easier. 
Then the user can decide at the very beginning: I want to solve "File level conflicts" or "Interleaved level conflicts"

Fei Song

---

From: "Fei Song" <[email-masked]>
Date: Sat, 12 Dec 2015 21:48:29 +0800

--------------
Fei Song

+1
>Not remembering the order in which the changes occurred is an optimization which gains you almost no efficiency and costs you greatly in terms of functionality.

The order of changes is significant for some cases. In my routine work, e.g. source code, latex file, etc. just belong to these cases.

---

From: "Fei Song" <[email-masked]>
Date: Sat, 12 Dec 2015 22:07:56 +0800

--------------
Fei Song
>detection is just one of the related use cases.

It seems that metadata will be required if the second goal is selected.
I think the conflict discovery (resolution) should be an important work for ISS, since we have not get consensus about whether it should be included in sync protocols.

>> are text files ― as was explained by Marc Blanchet ― in general it is not

---

From: "Fei Song" <[email-masked]>
Date: Sat, 12 Dec 2015 22:11:06 +0800

A quick supplement: the etag is also great if file-level conflict is the target~

--------------
Fei Song