Subject: Re: [Storagesync] Storagesync Digest, Vol 2, Issue 7

From: ZeQi Lai <[email-masked]>
Date: Wed, 9 Sep 2015 12:51:42 +0200

Dear Ted and Marc:

I think WebDAV is originally designed for authoring and versioning. It
allows users to collaboratively edit and manage files on remote web
services. However it does not consider how to make transmission efficient.
For example if we modify a small portion of a large file in local device,
WebDAV needs to upload the full file to the server.

Of course we can combine WebDAV with rsync to improve the transmission
efficiency. However I think this is still inadequate. As we mentioned
previously, in a cloud storage system files are split into chunks. The
rsync algorithm is designed to find the delta content between two
files. rsync works well in file granularity, but we need an approach that
can work in chunk granularity, and find the delta of two file versions even
though they are split into many chunks.

Besides, there are some other techniques (e.g. deduplication, bundling and
compression) that can be used to speed up transmission for cloud storage
services. Deduplication is to avoid the transmission of files that already
exist in the server. These techniques are complemented with rsync.

Overall, we think the existed solution (e.g. WebDAV+rsync) is inadequate.
We hope to design a standard sync protocol combining and extending existed
techniques to efficiently synchronize data between client and server. We
also think the approach to merge parallel changes on two ends is very
important and should be considered in the sync protocol.

Best regards,
Zeqi Lai

On Tue, Sep 8, 2015 at 11:05 PM, <[email-masked]> wrote:

> Send Storagesync mailing list submissions to
>         [email-masked]
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         https://www.ietf.org/mailman/listinfo/storagesync
> or, via email, send a message with subject or body 'help' to
>         [email-masked]
>
> You can reach the person managing the list at
>         [email-masked]
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Storagesync digest..."
>
> Today's Topics:
>
>    1. Re: Storagesync Digest, Vol 2, Issue 6 (ZeQi Lai)
>
>
> ---------- Forwarded message ----------
> From: ZeQi Lai <[email-masked]>
> To: [email-masked]
> Cc: Linhui Sun <[email-masked]>, Yong Cui <[email-masked]>
> Date: Tue, 8 Sep 2015 23:04:57 +0200
> Subject: Re: [Storagesync] Storagesync Digest, Vol 2, Issue 6
> Hi Ted and Marc,
>
> I am Zeqi Lai from Tsinghua University.
>
> Thanks for your suggestions! We would identify this in the problem
> statement later. However, I'd like to express some different viewpoints
> about the rsync over webdav first. Actually, I do not think rsync+WebDAV is
> not as good as it looks for several reasons.
>
> 1) WebDAV is not suitable for large files, e.g. a HD movie. Also WeDAV
> does not work well in mobile/wireless environments (high RTT/loss,
> intermittent connections). Temporary loss of connection may lead to sync
> failure.
>
> 2) There exists other capabilities (i.e. chunking, deduplication,
> bundling, compression) that can be used to speed up data sync. Current
> WebDAV does not consider these useful capabilities.
>
> 3) The limitation of rsync. Another point is that rsync works in the file
> granularity. That requires the entities running on two ends must have
> access to the entire file. However, to achieve better scalability, current
> network-based storage services split files into chunks and store them
> distributedly (chunk granularity). Directly using rsync needs to
> piece together all chunks to construct the whole file and would waste
> massive intra-cluster bandwidth.
>
> In summary, the sync protocol what we'd like to do is something more than
> rsync/webdav. As for the features Ted mentioned (e.g. parallel update), we
> haven't considered them a lot at first. After several recent discussions,
> we think they are really important and worth the effort. We'd like to see
> how many people have interests in those features? How many people think
> those features should be the core issue rather than improving the sync
> efficiency? And then we could update the problem statement and produce an
> appropriate charter : )
>
> Best regards,
> Zeqi Lai
>
> On Thu, Sep 3, 2015 at 9:00 PM, <[email-masked]> wrote:
>
>> Send Storagesync mailing list submissions to
>>         [email-masked]
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>>         https://www.ietf.org/mailman/listinfo/storagesync
>> or, via email, send a message with subject or body 'help' to
>>         [email-masked]
>>
>> You can reach the person managing the list at
>>         [email-masked]
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of Storagesync digest..."
>>
>> Today's Topics:
>>
>>    1. Unsubscribe (David Noveck)
>>
>>
>> ---------- Forwarded message ----------
>> From: David Noveck <[email-masked]>
>> To: [email-masked]
>> Cc:
>> Date: Wed, 2 Sep 2015 15:02:41 -0400
>> Subject: [Storagesync] Unsubscribe
>> On Sep 2, 2015 3:00 PM, <[email-masked]> wrote:
>>
>>> Send Storagesync mailing list submissions to
>>>         [email-masked]
>>>
>>> To subscribe or unsubscribe via the World Wide Web, visit
>>>         https://www.ietf.org/mailman/listinfo/storagesync
>>> or, via email, send a message with subject or body 'help' to
>>>         [email-masked]
>>>
>>> You can reach the person managing the list at
>>>         [email-masked]
>>>
>>> When replying, please edit your Subject line so it is more specific
>>> than "Re: Contents of Storagesync digest..."
>>>
>>> Today's Topics:
>>>
>>>    1. Re: Welcome to the Discussion on Internet Storage Sync
>>>       (Linhui Sun)
>>>    2. Re: Welcome to the Discussion on Internet Storage Sync (Ted Lemon)
>>>    3. Re: Welcome to the Discussion on Internet Storage Sync
>>>       (Marc Blanchet)
>>>    4. Re: Welcome to the Discussion on Internet Storage Sync (Ted Lemon)
>>>
>>>
>>> ---------- Forwarded message ----------
>>> From: Linhui Sun <[email-masked]>
>>> To: Ted Lemon <[email-masked]>
>>> Cc: "Songhaibin (A)" <[email-masked]>, storagesync <
>>> [email-masked]>, "[email-masked]" <[email-masked]>
>>> Date: Wed, 2 Sep 2015 20:40:51 +0800
>>> Subject: Re: [Storagesync] Welcome to the Discussion on Internet Storage
>>> Sync
>>>
>>> 2015-09-02 20:23 GMT+08:00 Ted Lemon <[email-masked]>:
>>>
>>>> On Sep 2, 2015, at 7:40 AM, Linhui Sun <[email-masked]> wrote:
>>>>
>>>> I think maybe we should make the definition of sync clearer. Could we
>>>> explain it as a simple upload/download process. Or the sync is specifically
>>>> referred to the transfer process of Network-based storage services which
>>>> may employ several well-known techniques (e.g. rsync)?
>>>>
>>>>
>>>> Personally I am not interested in a client/server sync protocol.   What
>>>> I would like is a sync protocol more like git, where each device has a
>>>> history of the last time a sync happened, and can tell what has changed
>>>> since the last sync.   When two devices connect, they compare notes and
>>>> figure out what changes have to be made on both sides to reach a new master
>>>> version.   This of course means that some conflict resolution process may
>>>> be necessary.   It also means that if two devices connect to a central
>>>> server on a regular basis, they will be in sync with each other even though
>>>> they have never communicated.
>>>>
>>>> [LH]: That is what we want to do in the IETF and is also what the
>>> current storage services are trying to provide. Simple C/S protocol is not
>>> satisfied.
>>>
>>> I think this is a more useful paradigm than something like rsync, but I
>>>> realize that it’s a lot more complicated, and may be beyond the scope of
>>>> what the folks who proposed this mailing list intended.   Is it something
>>>> you’d be interested in pursuing, or should I go off and do it on my own?
>>>> :)
>>>>
>>>> [LH]: I think rsync should not be considered as an essential part of
>>> such sync protocol. Not only because it is complicated. We usually find the
>>> rsync failed in some scenarios and sometimes causes extra sync traffic.
>>>
>>>
>>>
>>> ---------- Forwarded message ----------
>>> From: Ted Lemon <[email-masked]>
>>> To: Marc Blanchet <[email-masked]>
>>> Cc: Linhui Sun <[email-masked]>, "Songhaibin (A)" <
>>> [email-masked]>, storagesync <[email-masked]>, "
>>> [email-masked]" <[email-masked]>
>>> Date: Wed, 2 Sep 2015 10:29:01 -0400
>>> Subject: Re: [Storagesync] Welcome to the Discussion on Internet Storage
>>> Sync
>>> On Sep 2, 2015, at 8:07 AM, Marc Blanchet <[email-masked]>
>>> wrote:
>>>
>>> I’ve said to the organizers that I think a solution already implemented
>>> and deployed is rsync over webdav for this. It should be looked at
>>> carefully before reinventing a new protocol.
>>>
>>>
>>> rsync is inadequate if you have changes on both ends or more than two
>>> copies of the sync’d data.   An rfc describing the rsync protocol might be
>>> useful to someone, but I don’t think it’s interesting new work.   However,
>>> this question is certainly one that should be answered in the requirements
>>> document.   Do you think rsync is adequate because you think that parallel
>>> updates aren’t important?   It sounds like one motivation for this work is
>>> support for CDNs, and for those something like rsync would be perfectly
>>> adequate, but that’s not the use case I’m interested in.   Frankly, that
>>> seems like a solved problem to me.
>>>
>>>
>>>
>>> ---------- Forwarded message ----------
>>> From: Marc Blanchet <[email-masked]>
>>> To: Ted Lemon <[email-masked]>
>>> Cc: Linhui Sun <[email-masked]>, "Songhaibin (A)" <
>>> [email-masked]>, storagesync <[email-masked]>, "
>>> [email-masked]" <[email-masked]>
>>> Date: Wed, 02 Sep 2015 11:26:53 -0400
>>> Subject: Re: [Storagesync] Welcome to the Discussion on Internet Storage
>>> Sync
>>>
>>>
>>> On 2 Sep 2015, at 10:29, Ted Lemon wrote:
>>>
>>> On Sep 2, 2015, at 8:07 AM, Marc Blanchet <[email-masked]>
>>>> wrote:
>>>>
>>>>> I’ve said to the organizers that I think a solution already
>>>>> implemented and deployed is rsync over webdav for this. It should be looked
>>>>> at carefully before reinventing a new protocol.
>>>>>
>>>>
>>>> rsync is inadequate if you have changes on both ends or more than two
>>>> copies of the sync’d data.   An rfc describing the rsync protocol might be
>>>> useful to someone, but I don’t think it’s interesting new work.   However,
>>>> this question is certainly one that should be answered in the requirements
>>>> document.   Do you think rsync is adequate because you think that parallel
>>>> updates aren’t important?   It sounds like one motivation for this work is
>>>> support for CDNs, and for those something like rsync would be perfectly
>>>> adequate, but that’s not the use case I’m interested in.   Frankly, that
>>>> seems like a solved problem to me.
>>>>
>>>
>>> well, that is my point, so we agree…
>>> - webdav+rsync is providing pretty good solution. therefore, it should
>>> be clearly identified in the problem statement and also what it is not
>>> solving.
>>> - then we can discuss what other features (such as what you described)
>>> can be worth technical work, spec, etc…
>>>
>>> Marc.
>>>
>>>
>>>
>>>> _______________________________________________
>>>> Storagesync mailing list
>>>> [email-masked]
>>>> https://www.ietf.org/mailman/listinfo/storagesync
>>>>
>>>
>>>
>>>
>>>
>>> ---------- Forwarded message ----------
>>> From: Ted Lemon <[email-masked]>
>>> To: Marc Blanchet <[email-masked]>
>>> Cc: Linhui Sun <[email-masked]>, "Songhaibin (A)" <
>>> [email-masked]>, storagesync <[email-masked]>, "
>>> [email-masked]" <[email-masked]>
>>> Date: Wed, 2 Sep 2015 12:59:26 -0400
>>> Subject: Re: [Storagesync] Welcome to the Discussion on Internet Storage
>>> Sync
>>> On Sep 2, 2015, at 11:26 AM, Marc Blanchet <[email-masked]>
>>> wrote:
>>>
>>> well, that is my point, so we agree…
>>> - webdav+rsync is providing pretty good solution. therefore, it should
>>> be clearly identified in the problem statement and also what it is not
>>> solving.
>>> - then we can discuss what other features (such as what you described)
>>> can be worth technical work, spec, etc…
>>>
>>>
>>> Ah, OK, thanks for clarifying!
>>>
>>>
>>> _______________________________________________
>>> Storagesync mailing list
>>> [email-masked]
>>> https://www.ietf.org/mailman/listinfo/storagesync
>>>
>>>
>> _______________________________________________
>> Storagesync mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/storagesync
>>
>>
>
>
> --
> Yours sincerely
> LZQ
>
> _______________________________________________
> Storagesync mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/storagesync
>
>

-- 
Yours sincerely
LZQ

---

From: "Marc Blanchet" <[email-masked]>
Date: Wed, 09 Sep 2015 07:34:17 -0400

> Overall, we think the existed solution (e.g. WebDAV+rsync) is 

I’m not trying to say that webdav+rsync is better or worst. I’m 
saying it should be clearly discussed (and documented if we have a 
problem statement kind of document) so that we know why we are designing 
another protocol.

Marc.

>>>> implemented

---

From: ZeQi Lai <[email-masked]>
Date: Wed, 9 Sep 2015 14:08:28 +0200

Dear Marc:

Thank you very much for your suggestion! We will clearly identify the
problem in the problem statement.

Best regards,
Zeqi Lai

On Wed, Sep 9, 2015 at 1:34 PM, Marc Blanchet <[email-masked]>
wrote:

> I’m not trying to say that webdav+rsync is better or worst. I’m saying it
> should be clearly discussed (and documented if we have a problem statement
> kind of document) so that we know why we are designing another protocol.
>
> Marc.
>
>
>>> Cc: Linhui Sun <[email-masked]>, Yong Cui <[email-masked]
>>> >

-- 
Yours sincerely
LZQ