Subject: Re: [Storagesync] Welcome to the Discussion on Internet Storage Sync

From: "[email-masked]" <[email-masked]>
Date: Tue, 1 Sep 2015 16:52:08 +0800

hi, 
Third-party API and optimizing the sync protocols may be important for Online storage. But, IMO, the most important is improving data upload rate that end users can obtain.

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Hi folks,
Welcome to the mailing list of Internet Storage Sync ([email-masked]). This list is for the discussions of Internet Storage Sync work. You are welcome to introduce yourself here and start related discussions.
We've produced a draft to outline the main problems in existing Internet storage services, please find the document with the following link. Please feel free to comment on it and your valuable comments are more than welcome.
http://datatracker.ietf.org/doc/draft-cui-iss-problem/
A wiki for some useful resources and links could be found. Please feel free to edit it or let us know if we've missed something. 
https://github.com/iss-ietf/iss/wiki/Internet-Storage-Sync/
Regards,
Yong Cui,
Professor, Tsinghua University, Beijing, China
Associate Editor on IEEE TPDS, IEEE TCC
URL: http://www.4over6.edu.cn/cuiyong/index.html

---

From: Linhui Sun <[email-masked]>
Date: Tue, 1 Sep 2015 21:09:23 +0800

Hi Xiaowei,

Thanks for sharing your thoughts. I think the things you mentioned are not
conflicting when considering the standardization of sync protocol. If we
could come up with an open and standard sync protocol, the third-party APIs
could be significantly simplified. And it is reasonable and necessary that
such standard sync protocol should pay more attention to the sync
efficiency to allow users have an ideal upload rate.

Any comments?

Actually we outline five different problems in the problem statement draft.
Problem 4 and 5 reveals why most existing proprietary sync protocols do not
have a satisfactory sync efficiency (e.g. poor performance on upload rate).

1) *Complicated Support for APIs*
2) *Unavailable Cross-service Sync*
3) *Redundant Similar Clients*
4) *Different Capability Configurations and Implementations*
5) *Challenges in Mobile and Wireless Environments *
(More detailed descriptions could be found in:
http://datatracker.ietf.org/doc/draft-cui-iss-problem/)

In addition, I think the most urgent task of this Non-WG list is to have an
agreement on the specified problems and the scope of work. Thus may I ask
the people in this list to share your thoughts on the problems and scope of
work? Which problem do you think is not required or is there any other
problems we haven't mentioned in the draft? Please feel free to comment on
it.

Best regards,
Linhui

2015-09-01 16:52 GMT+08:00 [email-masked] <[email-masked]>:

> hi,
> Third-party API and optimizing the sync protocols may be important for
> Online storage. But, IMO, the most important is improving data upload rate
> that end users can obtain.
>
>
>
>
>
>
>
>
> We've produced a draft to outline the main problems in existing Internet storage services, please find the document with the following link. Please feel free to comment on it and your valuable comments are more than welcome.http://datatracker.ietf.org/doc/draft-cui-iss-problem/
> A wiki for some useful resources and links could be found. Please feel free to edit it or let us know if we've missed something. https://github.com/iss-ietf/iss/wiki/Internet-Storage-Sync/
> Regards,
> Yong Cui,
> ------------------------------
>
> _______________________________________________
> Storagesync mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/storagesync
>
>

---

From: "[email-masked]" <[email-masked]>
Date: Wed, 2 Sep 2015 10:25:08 +0800

hi,
Besides  optimizing the sync protocol, we may consider to avoid the transmission bottlenecks existing in long internet communication, such as latency, packet loss, network outages, inefficient 
protocols, and inter-network friction, etc.

regards
Xiaowei Qin

From: Linhui Sun
Date: 2015-09-01 21:09
To: [email-masked]; storagesync
Subject: Re: [Storagesync] Welcome to the Discussion on Internet Storage Sync
Hi Xiaowei,

Thanks for sharing your thoughts. I think the things you mentioned are not conflicting when considering the standardization of sync protocol. If we could come up with an open and standard sync protocol, the third-party APIs could be significantly simplified. And it is reasonable and necessary that such standard sync protocol should pay more attention to the sync efficiency to allow users have an ideal upload rate.

Any comments?

Actually we outline five different problems in the problem statement draft. Problem 4 and 5 reveals why most existing proprietary sync protocols do not have a satisfactory sync efficiency (e.g. poor performance on upload rate).

1) Complicated Support for APIs
2) Unavailable Cross-service Sync
3) Redundant Similar Clients
4) Different Capability Configurations and Implementations
5) Challenges in Mobile and Wireless Environments 
(More detailed descriptions could be found in: http://datatracker.ietf.org/doc/draft-cui-iss-problem/)

In addition, I think the most urgent task of this Non-WG list is to have an agreement on the specified problems and the scope of work. Thus may I ask the people in this list to share your thoughts on the problems and scope of work? Which problem do you think is not required or is there any other problems we haven't mentioned in the draft? Please feel free to comment on it.

Best regards,
Linhui

---

From: Ted Lemon <[email-masked]>
Date: Tue, 1 Sep 2015 22:53:30 -0400

> protocols, and inter-network friction, etc.

Question: you say "optimizing the sync protocol," which sounds like there is already a sync protocol.   Is that true, or do you mean optimizing the sync protocol after it’s been developed?   I haven’t seen any documents about a sync protocol yet.

---

From: "Songhaibin (A)" <[email-masked]>
Date: Wed, 2 Sep 2015 02:56:00 +0000

IMO, WebDAV can be seen as the IETF sync protocol for version management. I do not know if NFS also supports sync or not.

Best Regards!
-Haibin

From: Storagesync [mailto:[email-masked]] On Behalf Of Ted Lemon
Sent: Wednesday, September 02, 2015 10:54 AM
To: [email-masked]
Cc: Linhui Sun; storagesync
Subject: Re: [Storagesync] Welcome to the Discussion on Internet Storage Sync

On Sep 1, 2015, at 10:25 PM, [email-masked]<mailto:[email-masked]> wrote:
Besides  optimizing the sync protocol, we may consider to avoid the transmission bottlenecks existing in long internet communication, such as latency, packet loss, network outages, inefficient
protocols, and inter-network friction, etc.

---

From: "lilishan48"<[email-masked]>
Date: Wed, 2 Sep 2015 11:08:26 +0800

Dear Haibin,

WebDAV (Web-based Distributed Authoring and Versioning) extends the HTTP protocol to enable users to collaboratively edit and manage files on the remote web servers. WebDAV mainly focuses on the collaborative work on files. I think the ISS may handle large data files more frequently and focus more on the remote data sync capability.
NFS (Network File System) is some kind of distributed file system that enables users to store or access files on a network as is they are manipulating local files. NFS is just a file system without any transfer capability, thus it makes use of RPC protocol to fulfill the file sharing process.

Best Regards,
Lishan

发件人："Songhaibin (A)" <[email-masked]>
发送时间：2015-09-02 10:56
主题：Re: [Storagesync] Welcome to the Discussion on Internet Storage Sync
收件人："Ted Lemon"<[email-masked]>,"[email-masked]"<[email-masked]>
抄送："Linhui Sun"<[email-masked]>,"storagesync"<[email-masked]>

Best Regards!
-Haibin

On Sep 1, 2015, at 10:25 PM, [email-masked] wrote:
Besides  optimizing the sync protocol, we may consider to avoid the transmission bottlenecks existing in long internet communication, such as latency, packet loss, network outages, inefficient 
protocols, and inter-network friction, etc.

---

From: "[email-masked]" <[email-masked]>
Date: Wed, 2 Sep 2015 11:25:04 +0800

hi,Lishan

>>>>>>>I think the ISS may handle large data files more frequently and focus more on the remote data sync capability.

I agree with you.  ISS may focus on large content deliver, especially upstream traffic generated by end users.
Unfortunately, inherent limitations in the Internet‘s architecture make it difficult to achieve desired levels of performance natively on the Internet. Designed as a best-effort network, the Internet provides no guarantees on end-to-end reliability or performance. 
So, IMO, for maximizing upstream throughput,  moving "the destination" closer to the end user may benefit to pre-uploaded content, like CDN.

From: lilishan48
Date: 2015-09-02 11:08
To: Songhaibin (A); Ted Lemon; [email-masked]
CC: Linhui Sun; storagesync
Subject: Re: Re: [Storagesync] Welcome to the Discussion on Internet Storage Sync
Dear Haibin,

Best Regards,
Lishan

 
Best Regards!
-Haibin

---

From: Linhui Sun <[email-masked]>
Date: Wed, 2 Sep 2015 19:40:17 +0800

I think maybe we should make the definition of sync clearer. Could we
explain it as a simple upload/download process. Or the sync is specifically
referred to the transfer process of Network-based storage services which
may employ several well-known techniques (e.g. rsync)?

Best regards,
Linhui

2015-09-02 10:56 GMT+08:00 Songhaibin (A) <[email-masked]>:

> IMO, WebDAV can be seen as the IETF sync protocol for version management.
> I do not know if NFS also supports sync or not.
>
>
>
> Best Regards!
>
> -Haibin
>
>
>
> *From:* Storagesync [mailto:[email-masked]] *On Behalf Of *Ted
> Lemon
> *Sent:* Wednesday, September 02, 2015 10:54 AM
> *To:* [email-masked]
> *Cc:* Linhui Sun; storagesync
> *Subject:* Re: [Storagesync] Welcome to the Discussion on Internet
> Storage Sync
>
>
>
> Besides  optimizing the sync protocol, we may consider to avoid the
> transmission bottlenecks existing in long internet communication, such
> as latency, packet loss, network outages, inefficient
>
> Question: you say "optimizing the sync protocol," which sounds like there
> is already a sync protocol.   Is that true, or do you mean optimizing the
> sync protocol after it’s been developed?   I haven’t seen any documents
> about a sync protocol yet.
>
>
>

---

From: "Marc Blanchet" <[email-masked]>
Date: Wed, 02 Sep 2015 08:07:51 -0400

I’ve said to the organizers that I think a solution already 
implemented and deployed is rsync over webdav for this. It should be 
looked at carefully before reinventing a new protocol.

Marc.

On 1 Sep 2015, at 22:56, Songhaibin (A) wrote:

> IMO, WebDAV can be seen as the IETF sync protocol for version 
> management. I do not know if NFS also supports sync or not.
>
> Best Regards!
> -Haibin
>
> On Sep 1, 2015, at 10:25 PM, 
> [email-masked]<mailto:[email-masked]> wrote:
> Question: you say "optimizing the sync protocol," which sounds like 
> there is already a sync protocol.   Is that true, or do you mean 
> optimizing the sync protocol after it’s been developed?   I 
> haven’t seen any documents about a sync protocol yet.
>

---

From: Ted Lemon <[email-masked]>
Date: Wed, 2 Sep 2015 08:23:47 -0400

On Sep 2, 2015, at 7:40 AM, Linhui Sun <[email-masked] <mailto:[email-masked]>> wrote:
> I think maybe we should make the definition of sync clearer. Could we explain it as a simple upload/download process. Or the sync is specifically referred to the transfer process of Network-based storage services which may employ several well-known techniques (e.g. rsync)?

Personally I am not interested in a client/server sync protocol.   What I would like is a sync protocol more like git, where each device has a history of the last time a sync happened, and can tell what has changed since the last sync.   When two devices connect, they compare notes and figure out what changes have to be made on both sides to reach a new master version.   This of course means that some conflict resolution process may be necessary.   It also means that if two devices connect to a central server on a regular basis, they will be in sync with each other even though they have never communicated.

I think this is a more useful paradigm than something like rsync, but I realize that it’s a lot more complicated, and may be beyond the scope of what the folks who proposed this mailing list intended.   Is it something you’d be interested in pursuing, or should I go off and do it on my own?   :)

---

From: Linhui Sun <[email-masked]>
Date: Wed, 2 Sep 2015 20:39:08 +0800

Hi Marc,

We've refined the wiki and it includes a brief comparison between
WebDAV/rsync and this ISS work (
https://github.com/iss-ietf/iss/wiki/Internet-Storage-Sync/). Actually,
there exists third party software called DropDAV that user could use WebDAV
to sync files with Dropbox servers. As for rsync over WebDAV, it is really
a smart solution which introduce incremental sync in the original WebDAV.

IMO, they are two different things which have different focuses. And I
think it would be an interesting issue to introduce some features of WebDAV
(e.g. version control) in the ISS work to enhance its collaborative work
ability.

2015-09-02 20:07 GMT+08:00 Marc Blanchet <[email-masked]>:

> I’ve said to the organizers that I think a solution already implemented
> and deployed is rsync over webdav for this. It should be looked at
> carefully before reinventing a new protocol.
>
> Marc.
>
>> transmission bottlenecks existing in long internet communication, such as
>> latency, packet loss, network outages, inefficient

---

From: Linhui Sun <[email-masked]>
Date: Wed, 2 Sep 2015 20:40:51 +0800

2015-09-02 20:23 GMT+08:00 Ted Lemon <[email-masked]>:

> On Sep 2, 2015, at 7:40 AM, Linhui Sun <[email-masked]> wrote:
>
> Personally I am not interested in a client/server sync protocol.   What I
> would like is a sync protocol more like git, where each device has a
> history of the last time a sync happened, and can tell what has changed
> since the last sync.   When two devices connect, they compare notes and
> figure out what changes have to be made on both sides to reach a new master
> version.   This of course means that some conflict resolution process may
> be necessary.   It also means that if two devices connect to a central
> server on a regular basis, they will be in sync with each other even though
> they have never communicated.
>
> [LH]: That is what we want to do in the IETF and is also what the current
storage services are trying to provide. Simple C/S protocol is not
satisfied.

I think this is a more useful paradigm than something like rsync, but I
> realize that it’s a lot more complicated, and may be beyond the scope of
> what the folks who proposed this mailing list intended.   Is it something
> you’d be interested in pursuing, or should I go off and do it on my own?
> :)
>
> [LH]: I think rsync should not be considered as an essential part of such
sync protocol. Not only because it is complicated. We usually find the
rsync failed in some scenarios and sometimes causes extra sync traffic.

---

From: Ted Lemon <[email-masked]>
Date: Wed, 2 Sep 2015 10:29:01 -0400

On Sep 2, 2015, at 8:07 AM, Marc Blanchet <[email-masked]> wrote:
> I’ve said to the organizers that I think a solution already implemented and deployed is rsync over webdav for this. It should be looked at carefully before reinventing a new protocol.

rsync is inadequate if you have changes on both ends or more than two copies of the sync’d data.   An rfc describing the rsync protocol might be useful to someone, but I don’t think it’s interesting new work.   However, this question is certainly one that should be answered in the requirements document.   Do you think rsync is adequate because you think that parallel updates aren’t important?   It sounds like one motivation for this work is support for CDNs, and for those something like rsync would be perfectly adequate, but that’s not the use case I’m interested in.   Frankly, that seems like a solved problem to me.

---

From: "Marc Blanchet" <[email-masked]>
Date: Wed, 02 Sep 2015 11:26:53 -0400

On 2 Sep 2015, at 10:29, Ted Lemon wrote:

> rsync is inadequate if you have changes on both ends or more than two 
> copies of the sync’d data.   An rfc describing the rsync protocol 
> might be useful to someone, but I don’t think it’s interesting new 
> work.   However, this question is certainly one that should be 
> answered in the requirements document.   Do you think rsync is 
> adequate because you think that parallel updates aren’t important?   
> It sounds like one motivation for this work is support for CDNs, and 
> for those something like rsync would be perfectly adequate, but 
> that’s not the use case I’m interested in.   Frankly, that seems 
> like a solved problem to me.

well, that is my point, so we agree…
- webdav+rsync is providing pretty good solution. therefore, it should 
be clearly identified in the problem statement and also what it is not 
solving.
- then we can discuss what other features (such as what you described) 
can be worth technical work, spec, etc…

Marc.

---

From: Ted Lemon <[email-masked]>
Date: Wed, 2 Sep 2015 12:59:26 -0400

On Sep 2, 2015, at 11:26 AM, Marc Blanchet <[email-masked]> wrote:
> - webdav+rsync is providing pretty good solution. therefore, it should be clearly identified in the problem statement and also what it is not solving.
> - then we can discuss what other features (such as what you described) can be worth technical work, spec, etc…

Ah, OK, thanks for clarifying!

---

From: Zhenhua Li <[email-masked]>
Date: Wed, 9 Sep 2015 18:09:34 +0800

Dear all,

I'm Zhenhua Li, who has been investigating cloud storage services for
several years at Tsinghua University.
I used to make a horizontal comparison of the network-level efficiency for
cloud storage services, and the resulting paper is published in ACM IMC'14
conference:
http://www.greenorbs.org/people/lzh/papers/[IMC'14]%20Cloud%20Sync%20TUE.pdf

Besides, I also made an in-depth study of the data sync process for
Dropbox-like cloud storage services, and the resulting paper is published
in ACM Middleware'13 conference:
http://www.greenorbs.org/people/lzh/papers/[Middleware'13]%20Dropbox.pdf

In my opinion, ISS (Internet Storage Sync) services lie between rsync and
git/svn/WebDAV for two reasons:

1) rsync cannot satisfy the requirements of ISS even under today's typical
application scenarios (let alone other potential scenarios in the future).
Several technical papers have reported the performance issues of rsync,
including Prof. Yong Cui's MobiCom'15 paper and my IMC'14 and Middleware'13
paper.
Thus, the design of ISS is more complicated that rsync.

2) ISS should be less complicated than git/svn/WebDAV. First, I agree with
others' opinion that ISS is mostly used for syncing (possibly large) files
rather than only small documents.
Second, most files hosted by ISS would not be frequently modified, or are
not expected to be written such as image, audio, or video files.
As a result, the application scenarios of ISS are different from
git/svn/WebDAV.
Of course, ISS can provide some simplified "version rollback" functions as
what Dropbox have done.

My two cents :)

Zhenhua

-- 
Zhenhua Li,   homepage: http://www.greenorbs.org/people/lzh/
Associate Researcher, GreenOrbs - Cloud Computing Group, TNLIST, Tsinghua
University
PostDoc, School of Software, Tsinghua University, Beijing, China
Email: [email-masked], [email-masked]
Phone: +86-13552819836 (@Beijing)

---

From: Zhenhua Li <[email-masked]>
Date: Wed, 9 Sep 2015 18:18:27 +0800

Hi Linhui,

I suggest we consider an additional problem, *Key Metrics for Internet
Storage Sync*.
At the moment, we even don't know what are the key metrics to correctly
evaluate an Internet Storage Sync service/operation.

For your reference :)

Zhenhua

On Tue, Sep 1, 2015 at 9:09 PM, Linhui Sun <[email-masked]> wrote:

> Actually we outline five different problems in the problem statement
> draft. Problem 4 and 5 reveals why most existing proprietary sync protocols
> do not have a satisfactory sync efficiency (e.g. poor performance on upload
> rate).
>
> In addition, I think the most urgent task of this Non-WG list is to have
> an agreement on the specified problems and the scope of work. Thus may I
> ask the people in this list to share your thoughts on the problems and
> scope of work? Which problem do you think is not required or is there any
> other problems we haven't mentioned in the draft? Please feel free to
> comment on it.
>
> Best regards,
> Linhui
>

---

From: Linhui Sun <[email-masked]>
Date: Wed, 09 Sep 2015 21:41:54 +0800

Hi Zhenhua,
Thanks for your comment. It sounds like the output of this problem is an
evaluation document for the sync protocol. Am I right? If so, personally I think
this could be considered later. Currently, we should make it clear why we would
like to propose such a work as Marc suggested.
Comments are welcome.
Best regards, Linhui

On 周三, 9月 9, 2015 at 18:18, Zhenhua Li <[email-masked]> wrote:
Hi Linhui,
I suggest we consider an additional problem, Key Metrics for Internet Storage Sync . At the moment, we even don't know what are the key metrics to correctly
evaluate an Internet Storage Sync service/operation.
For your reference :)
Zhenhua
On Tue, Sep 1, 2015 at 9:09 PM, Linhui Sun < [email-masked] [[email-masked]] > wrote:
Hi Xiaowei,
Thanks for sharing your thoughts. I think the things you mentioned are not
conflicting when considering the standardization of sync protocol. If we could
come up with an open and standard sync protocol, the third-party APIs could be
significantly simplified. And it is reasonable and necessary that such standard
sync protocol should pay more attention to the sync efficiency to allow users
have an ideal upload rate.
Any comments?
Actually we outline five different problems in the problem statement draft.
Problem 4 and 5 reveals why most existing proprietary sync protocols do not have
a satisfactory sync efficiency (e.g. poor performance on upload rate).
1) Complicated Support for APIs
2) Unavailable Cross-service Sync
3) Redundant Similar Clients
4) Different Capability Configurations and Implementations
5) Challenges in Mobile and Wireless Environments
(More detailed descriptions could be found in: http://datatracker.ietf.org/doc/draft-cui-iss-problem/
[http://datatracker.ietf.org/doc/draft-cui-iss-problem/] )
In addition, I think the most urgent task of this Non-WG list is to have an
agreement on the specified problems and the scope of work. Thus may I ask the
people in this list to share your thoughts on the problems and scope of work?
Which problem do you think is not required or is there any other problems we
haven't mentioned in the draft? Please feel free to comment on it.
Best regards, Linhui
2015-09-01 16:52 GMT+08:00 [email-masked] [[email-masked]] < [email-masked] [[email-masked]] > :
hi, Third-party API and optimizing the sync protocols may be important for Online
storage. But, IMO, the most important is improving data upload rate that end
users can obtain.

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------Hi folks,
Welcome to the mailing list of Internet Storage Sync ([email-masked] [[email-masked]]). This list is for the discussions of Internet Storage Sync work. You are welcome to introduce yourself here and start related discussions.
We've produced a draft to outline the main problems in existing Internet storage services, please find the document with the following link. Please feel free to comment on it and your valuable comments are more than welcome.
http://datatracker.ietf.org/doc/draft-cui-iss-problem/ [http://datatracker.ietf.org/doc/draft-cui-iss-problem/]
A wiki for some useful resources and links could be found. Please feel free to edit it or let us know if we've missed something. 
https://github.com/iss-ietf/iss/wiki/Internet-Storage-Sync/ [https://github.com/iss-ietf/iss/wiki/Internet-Storage-Sync/]
Regards,
Yong Cui,
Professor, Tsinghua University, Beijing, China
Associate Editor on IEEE TPDS, IEEE TCC
URL: http://www.4over6.edu.cn/cuiyong/index.html [http://www.4over6.edu.cn/cuiyong/index.html]

--------------------------------------------------------------------------------

_______________________________________________
Storagesync mailing list
[email-masked] [[email-masked]]
https://www.ietf.org/mailman/listinfo/storagesync
[https://www.ietf.org/mailman/listinfo/storagesync]

_______________________________________________
Storagesync mailing list
[email-masked] [[email-masked]]
https://www.ietf.org/mailman/listinfo/storagesync
[https://www.ietf.org/mailman/listinfo/storagesync]

--
Zhenhua Li, homepage: http://www.greenorbs.org/people/lzh/ [http://www.greenorbs.org/people/lzh/]
Associate Researcher, GreenOrbs - Cloud Computing Group, TNLIST, Tsinghua
University
PostDoc, School of Software, Tsinghua University, Beijing, China
Email: [email-masked] [[email-masked]] , [email-masked] [[email-masked]]
Phone: +86-13552819836 (@Beijing)

---

From: Zhenhua Li <[email-masked]>
Date: Wed, 9 Sep 2015 21:58:00 +0800

I agree. The performance metrics are not the first-place problem at the
moment. We can keep it in mind and consider later.

Zhenhua

On Wed, Sep 9, 2015 at 9:41 PM, Linhui Sun <[email-masked]> wrote:

> evaluation document for the sync protocol. Am I right? If so,  personally I
> think this could be considered later. Currently, we should make it clear
> why we would like to propose such a work as Marc suggested.
>
> Comments are welcome.
>
> Best regards,
> Linhui
>
>
>> Thanks for sharing your thoughts. I think the things you mentioned are
>> not conflicting when considering the standardization of sync protocol. If
>> we could come up with an open and standard sync protocol, the third-party
>> APIs could be significantly simplified. And it is reasonable and necessary
>> that such standard sync protocol should pay more attention to the sync