Subject: [Storagesync] Some preliminary investigations on ownCloud

From: Linhui Sun <[email-masked]>
Date: Wed, 25 Nov 2015 22:45:32 +0800

Hi all,

As I mentioned before, I think the developers could benefit from the IETF
standards. The ownCloud (https://owncloud.org/) is just an example. It is
developed for those who do not trust commercial storage services and want
to build their own network-based storage services. The ownCloud is using
WebDAV (RFC4918) to achieve the data sync. IMO, the WebDAV is designed for
distributed work but not for the sync. Thus, I made some preliminary
investigations on how the ownCloud uses WebDAV for sync purposes. A brief
summary of what I've found is in the following, please correct me if I am
wrong.

I installed the ownCloud server (v8.2.1) on the CentOS7, and the client is
a desktop client on Windows.

1. To find whether there is a change to the synced directory, the client
continuously sends PROPFIND to the server at regular intervals (around 34
seconds under my observation). The server will respond a 207 Multi-Status
Response to tell whether the main directory has been changed. To perform
this regular check, the client will open a new TCP connection to send the
PROPFIND, the server will close the existing TCP connection after
responding the 207 Multi-Status Response. For the next check, the client
will open another new TCP connection.

2. Every time adding (or creating) a new file to the local folder, the
client will open a new TCP connection (if there is no connection existing)
to send the file asap. The client will first send several PROPFINDs to find
out which sub-directory has been changed. And then it sends the file using
PUT. The server will respond a 201 Created Response and then terminate the
connection. Currently, I haven't found any application layer chunking, all
the segmentation are performed by TCP.

3. Every time I delete (or rename) a file locally, the client will also
open a new TCP connection to send several PROPFINDs to find out which file
has been removed (or renamed). Then it will send DELETE (or MOVE). The
server will respond a 204 No Content Response (or 201 Created Response) and
then terminate the connection.

4. I open a file and frequently edit and save it (actually this is what I
usually do with the Dropbox). The client will send the whole file to the
server every time I save the file.

To summarize, it seems that the ownCloud makes heavily use of PROPFIND to
achieve the sync process. Each sync operation (e.g. upload, modify and
etc.) will start with sending one or more PROPFINDs. And currently, if I
add a file to the server (directly from the server side via web interface),
the client cannot find the change. I need to interrupt the sync and recover
it to make the client be aware of the change and download the newly added
file. I'm not sure whether this is caused by the sync mechanism or an
improper server configuration. I need to investigate this further and also
how the ownCloud works for multiple clients (or devices).

For ISS, I think ownCloud has demonstrated to some extent that similar IETF
protocols could be deployed and employed. The intention of this message is
to investigate the current state of using WebDAV for sync purposes to see
what needs to be improved here and whether we need new protocols.

Comments are welcome : )

Regards,
Linhui

---

From: Jakub Moscicki <[email-masked]>
Date: Wed, 25 Nov 2015 17:50:20 +0000

Hello,

What kind of outcome are you looking for with this analysis? Some research in this area has already been done or is being done as we speak

e.g. "A study of delta-sync and other optimisation in HTTP/Webdav synchonisation protocols"

see "Technology and Research":

http://cs3.ethz.ch/program.html

It would be interesting to see if there is a potential for collaboration. Or maybe we already have some information you are looking for.

Best regards,

Jakub Moscicki

—

On 25 Nov 2015, at 11:45, Linhui Sun <[email-masked]<mailto:[email-masked]>> wrote:

Hi all,

As I mentioned before, I think the developers could benefit from the IETF standards. The ownCloud (https://owncloud.org/) is just an example. It is developed for those who do not trust commercial storage services and want to build their own network-based storage services. The ownCloud is using WebDAV (RFC4918) to achieve the data sync. IMO, the WebDAV is designed for distributed work but not for the sync. Thus, I made some preliminary investigations on how the ownCloud uses WebDAV for sync purposes. A brief summary of what I've found is in the following, please correct me if I am wrong.

I installed the ownCloud server (v8.2.1) on the CentOS7, and the client is a desktop client on Windows.

1. To find whether there is a change to the synced directory, the client continuously sends PROPFIND to the server at regular intervals (around 34 seconds under my observation). The server will respond a 207 Multi-Status Response to tell whether the main directory has been changed. To perform this regular check, the client will open a new TCP connection to send the PROPFIND, the server will close the existing TCP connection after responding the 207 Multi-Status Response. For the next check, the client will open another new TCP connection.

2. Every time adding (or creating) a new file to the local folder, the client will open a new TCP connection (if there is no connection existing) to send the file asap. The client will first send several PROPFINDs to find out which sub-directory has been changed. And then it sends the file using PUT. The server will respond a 201 Created Response and then terminate the connection. Currently, I haven't found any application layer chunking, all the segmentation are performed by TCP.

3. Every time I delete (or rename) a file locally, the client will also open a new TCP connection to send several PROPFINDs to find out which file has been removed (or renamed). Then it will send DELETE (or MOVE). The server will respond a 204 No Content Response (or 201 Created Response) and then terminate the connection.

4. I open a file and frequently edit and save it (actually this is what I usually do with the Dropbox). The client will send the whole file to the server every time I save the file.

To summarize, it seems that the ownCloud makes heavily use of PROPFIND to achieve the sync process. Each sync operation (e.g. upload, modify and etc.) will start with sending one or more PROPFINDs. And currently, if I add a file to the server (directly from the server side via web interface), the client cannot find the change. I need to interrupt the sync and recover it to make the client be aware of the change and download the newly added file. I'm not sure whether this is caused by the sync mechanism or an improper server configuration. I need to investigate this further and also how the ownCloud works for multiple clients (or devices).

For ISS, I think ownCloud has demonstrated to some extent that similar IETF protocols could be deployed and employed. The intention of this message is to investigate the current state of using WebDAV for sync purposes to see what needs to be improved here and whether we need new protocols.

Comments are welcome : )

Regards,
Linhui

_______________________________________________
Storagesync mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/storagesync

---

From: "Fei Song" <[email-masked]>
Date: Thu, 26 Nov 2015 11:11:48 +0800

Hi all

--------------
Fei Song
>will open another new TCP connection.

I wonder whether these message you got had been encrypt or not.

>how the ownCloud works for multiple clients (or devices).

What do you mean by "directly from the server side via web interface".
If all the configuration were correct, do you want to identify that the sync in 
ownCloud is only unidirectional?

---

From: Linhui Sun <[email-masked]>
Date: Thu, 26 Nov 2015 11:19:53 +0800

Hi Jakub,

2015-11-26 1:50 GMT+08:00 Jakub Moscicki <[email-masked]>:

> Hello,
>
> What kind of outcome are you looking for with this analysis? Some research
> in this area has already been done or is being done as we speak
>
I think this is for the previous confusion that "why do we need the ISS
over the existing WebDAV". As we could find that many open source projects
are using WebDAV now (ownCloud is a popular example), we need to find out
whether WebDAV is sufficient for this kind of applications. IMO, the
developers for those applications might be the active users of the ISS
work, thus finding out what do they lack, what do they want is important.

> http://cs3.ethz.ch/program.html
Actually I'm not talking about the efficiency or optimization here. What I
really want to say is the applicability. If we find the WebDAV is enough
and appropriate, then for ISS, a generalized API to access all services and
for collaboration would be better than a sync protocol.

>
> It would be interesting to see if there is a potential for collaboration.
> Or maybe we already have some information you are looking for.
>
The collaboration is really important but is based on the fact that many
providers are willing to use it. At the current stage, I think we need to
find someone who are willing to deploy and use the possible ISS work, maybe
ownCloud is a potential user : )

Regards,
Linhui

>
> Best regards,
>
> Jakub Moscicki
>
> —
>
>
> On 25 Nov 2015, at 11:45, Linhui Sun <[email-masked]> wrote:
>
> Hi all,
>
> For ISS, I think ownCloud has demonstrated to some extent that similar
> IETF protocols could be deployed and employed. The intention of this
> message is to investigate the current state of using WebDAV for sync
> purposes to see what needs to be improved here and whether we need new
> protocols.
>
> Comments are welcome : )
>
> Regards,
> Linhui
>
>
> [email-masked]

---

From: "Fei Song" <[email-masked]>
Date: Thu, 26 Nov 2015 11:20:25 +0800

Hi Jakub and all,

I can't download the paper you mentioned. Could you please show me a link to it?
For the URL http://cs3.ethz.ch/program.html ,there is no links... 

--------------
Fei Song
>―

---

From: Linhui Sun <[email-masked]>
Date: Thu, 26 Nov 2015 11:27:38 +0800

Hi Fei,

> Hi all
The ownCloud suggests people use HTTPS, but you could also use the HTTP.
And that is what I did.

> ownCloud is only unidirectional?
People could access to the ownCloud service via desktop, mobile client or
web interface (i.e. browser), the point I want to say here is an operation
on the server side cannot be detected automatically. I'm not sure whether
I've configured the server correctly. But the sync is not unidirectional
since after I recovering the sync, the client could download the asap.

> IETF

---

From: "Fei Song" <[email-masked]>
Date: Thu, 26 Nov 2015 11:28:03 +0800

BTW, Based on the last sentence of last email:"The intention of this message is to investigate the current state of using WebDAV for sync purposes to see what needs to be improved here and whether we need new protocols"

The outcome he/she wanted might be just the links like http://cs3.ethz.ch/program.html :)

--------------
Fei Song

---

From: Linhui Sun <[email-masked]>
Date: Thu, 26 Nov 2015 11:28:44 +0800

> For the URL http://cs3.ethz.ch/program.html ,there is no links...
+1

Regards,
Linhui

>
>
> --------------
> Fei Song
> >Hello,
> >
> >What kind of outcome are you looking for with this analysis? Some
> research in this area has already been done or is being done as we speak
> >

---

From: "Fei Song" <[email-masked]>
Date: Thu, 26 Nov 2015 11:36:57 +0800

Hi Linhui,

--------------
Fei Song
>The ownCloud suggests people use HTTPS, but you could also use the HTTP.

Thanks!

>since after I recovering the sync, the client could download the asap.

I see, then there might be an option at the server side 
can sync the changes to the client automatically.

---

From: Linhui Sun <[email-masked]>
Date: Thu, 26 Nov 2015 11:37:13 +0800

2015-11-26 11:28 GMT+08:00 Fei Song <[email-masked]>:

> BTW, Based on the last sentence of last email:"The intention of this
> protocols"
In another word, I think what I want finally might be a discussion about
"What is needed for the ISS: a sync protocol or a generalized API". Sorry
for the poor expression : )

Regards,
Linhui

---

From: "Fei Song" <[email-masked]>
Date: Thu, 26 Nov 2015 12:56:32 +0800

I think what ISS need is a new protocol, here are some thoughts:

1.	The latest version of WebDAV (RFC 4918) was standardized in 2007. We have witnessed that cloud storage or online storage had been widely used. More and more applications had been emerging recently. It is not quite flexible for the developer or service providers to use WebDAV or other competent schemes to meet these new and challenging requirements quickly.
2.	Even if the modifications of WebDAV or other competent schemes are accomplished to achieve different purpose, it is not beneficial for the unification and standardization.
3.	A generalized API is hard to have. Sometimes it is even harder than to design a protocol since there are diverse protocols and ever-changing implementation version. It might be out of the scope of IETF as well…

--------------
Fei Song

---

From: "Fei Song" <[email-masked]>
Date: Thu, 26 Nov 2015 13:03:29 +0800

I will also add this into "recent issues discussed". Here is the latest version:

1. The design targets of WebDAV, rsync and other existing approaches?
2. The potential use cases of ISS, such as client/server, git-like pattern, svn, etc.
3. The efficiency improvements might be the second goal for standardizing ISS protocol
4. CORS headers on storage sync APIs
5. What is needed for the ISS: a sync protocol or a generalized API

--------------
Fei Song

---

From: Jakub Moscicki <[email-masked]>
Date: Thu, 26 Nov 2015 07:29:17 +0000

2015-11-26 11:28 GMT+08:00 Fei Song <[email-masked]<mailto:[email-masked]>>:
BTW, Based on the last sentence of last email:"The intention of this message is to investigate the current state of using WebDAV for sync purposes to see what needs to be improved here and whether we need new protocols"

The outcome he/she wanted might be just the links like http://cs3.ethz.ch/program.html :)
In another word, I think what I want finally might be a discussion about "What is needed for the ISS: a sync protocol or a generalized API". Sorry for the poor expression : )

I think this is a relevant question indeed (well, actually also if there is any standard needed at all in the first place). By the generalized API do you mean a HTTP-style API (like REST)? In that case you may consider WebDAV quite close and the difference between the protocol and the API blurs for practical purposes.

While I agree that WebDAV may have disadvantages there is a good number of installations using this already in our community (research labs mainly in, but not limited to, Europe). I think the interesting point about WebDAV/HTTP is extensibility and maybe it is worth is that these extensions go by a “standard”. However, you should consider that for reasonably efficient sync scenario the server should also exhibit certain behaviour. That is, in case of owncloud for example, a server should be able to efficiently propagate the ETAG changes up the directory tree (like the Merkle Tree) so that the client may use propfind efficiently. This is not a hard spec requirement but otherwise propfinding the entire remote tree each time would be impractically inefficient. So this really goes a little bit beyond just an API.

You should also consider that the OpenCloudMesh (under GEANT umbrella) is an initiative with the intent is to make cross-service sharing very easy. These shares may also be synchronized automatically. I currently have no evidence, however, that other software providers with the exception of owncloud are interested in developing such standard. If there is no bottom up interest from the users then it won’t happen (in my opinion).

With the link I wanted to point you to the fact that what you discuss in this mailing list will be also discussed at the upcoming CS3 event I linked in. The intent is to do this discussing together between service providers, developers and researchers together — so that it does not only end up as an academic exercise but backed up by a critical mass if there is some potential in standardisation, at least in our community. I hope this could be of interest to IETF community, as mentioned on the program page.

Selected papers will be published in FGCS after the event, so please stand by, or attend the event if you want to be part of this discussion here. BTW. the abstract submission deadline is past but one exceptionally good contribution could still possibly be accommodated if submitted rapidly.

I would be nonetheless happy to continue contributing to the discussion in this mailing list.

Best regards,

Jakub Moscicki

--

Regards,
Linhui

--------------
Fei Song
>On 25 Nov 2015, at 11:45, Linhui Sun <[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>> wrote:
>
>Hi all,
>
>[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>

---

From: Jakub Moscicki <[email-masked]>
Date: Thu, 26 Nov 2015 07:39:23 +0000

BTW, here is the public CFP for FGCS: http://cs3.ethz.ch/publications.html

Best regards,

Jakub Moscicki

—

On 26 Nov 2015, at 04:29, Jakub Moscicki <[email-masked]<mailto:[email-masked]>> wrote:

Best regards,

Jakub Moscicki

--

Regards,
Linhui

--------------
Fei Song

---

From: "Fei Song" <[email-masked]>
Date: Thu, 26 Nov 2015 16:02:02 +0800

Thanks for that~

--------------
Fei Song
>With the link I wanted to point you to the fact that what you discuss in this mailing list will be also discussed at the upcoming CS3 event I linked in. The intent is to do this discussing together between service providers, developers and researchers together ― so that it does not only end up as an academic exercise but backed up by a critical mass if there is some potential in standardisation, at least in our community. I hope this could be of interest to IETF community, as mentioned on the program page.
>

---

From: "Fei Song" <[email-masked]>
Date: Thu, 26 Nov 2015 16:21:26 +0800

Hi Jakub,

--------------
Fei Song
>While I agree that WebDAV may have disadvantages there is a good number of installations using this already in our community (research labs mainly in, but not limited to, Europe). I think the intere…

+1, Just like I sent to the mailing list before, WebDAV is a great solution. However, a better one might be more beneficial for the community.

>You should also consider that the OpenCloudMesh (under GEANT umbrella) is an initiative with the intent is to make cross-service sharing very easy. These shares may also be synchronized automaticall…

The term "standard" you mentioned in "software providers are interested in developing such standard" means a protocol or other things?

>With the link I wanted to point you to the fact that what you discuss in this mailing list will be also discussed at the upcoming CS3 event I linked in. The intent is to do this discussing together…

Many thanks for your information. It is great to see the parallel discussions in CS3.

---

From: Linhui Sun <[email-masked]>
Date: Thu, 26 Nov 2015 17:38:04 +0800

2015-11-26 15:29 GMT+08:00 Jakub Moscicki <[email-masked]>:

> I think this is a relevant question indeed (well, actually also if there
> is any standard needed at all in the first place). By the generalized API
> do you mean a HTTP-style API (like REST)? In that case you may consider
> WebDAV quite close and the difference between the protocol and the API
> blurs for practical purposes.
>
Yes. And maybe for the application layer applications, an API is more
realistic and practical at the first stage? I'm not saying we should not
standardize a sync protocol...

>
> While I agree that WebDAV may have disadvantages there is a good number of
> installations using this already in our community (research labs mainly in,
> but not limited to, Europe). I think the interesting point about
> WebDAV/HTTP is extensibility and maybe it is worth is that these extensions
> go by a “standard”. However, you should consider that for
>
Actually I don't know what the "extensibility" means here, are you saying
that we could extend the WebDAV?

> reasonably efficient sync scenario the server should also exhibit certain
> behaviour. That is, in case of owncloud for example, a server should be
> able to efficiently propagate the ETAG changes up the directory tree (like
> the Merkle Tree) so that the client may use propfind efficiently. This is
> not a hard spec requirement but otherwise propfinding the entire remote
> tree each time would be impractically inefficient. So this really goes a
> little bit beyond just an API.
>
And that is what I mean the "applicability", IMO, the WebDAV seems not
suitable for sync purposes. As a result, the developers compromise to
propfind the entire file tree at intervals.

>
> You should also consider that the OpenCloudMesh (under GEANT umbrella) is
> an initiative with the intent is to make cross-service sharing very easy.
> These shares may also be synchronized automatically. I currently have no
> evidence, however, that other software providers with the exception of
> owncloud are interested in developing such standard. If there is no bottom
> up interest from the users then it won’t happen (in my opinion).
>
That's cool. I'm just wondering whether the ISS could get some input from
that...

>
> With the link I wanted to point you to the fact that what you discuss in
> this mailing list will be also discussed at the upcoming CS3 event I linked
> in. The intent is to do this discussing together between service providers,
> developers and researchers together — so that it does not only end up as an
> academic exercise but backed up by a critical mass if there is some
> potential in standardisation, at least in our community. I hope this could
> be of interest to IETF community, as mentioned on the program page.
>
> Selected papers will be published in FGCS after the event, so please stand
> by, or attend the event if you want to be part of this discussion here.
> BTW. the abstract submission deadline is past but one exceptionally good
> contribution could still possibly be accommodated if submitted rapidly.
>
Thanks for that!

>> >It would be interesting to see if there is a potential for
>> collaboration. Or maybe we already have some information you are looking
>> for.
>> >
>> >Best regards,
>> >
>> >Jakub Moscicki
>> >
>> >—
>> >
>> >
>> >As I mentioned before, I think the developers could benefit from the
>> IETF standards. The ownCloud (https://owncloud.org/) is just an example.
>> It is developed for those who do not trust commercial storage services and
>> want to build their own network-based storage services. The ownCloud is
>> using WebDAV (RFC4918) to achieve the data sync. IMO, the WebDAV is
>> designed for distributed work but not for the sync. Thus, I made some
>> preliminary investigations on how the ownCloud uses WebDAV for sync
>> purposes. A brief summary of what I've found is in the following, please
>> correct me if I am wrong.
>> >
>> >4. I open a file and frequently edit and save it (actually this is what
>> I usually do with the Dropbox). The client will send the whole file to the
>> >To summarize, it seems that the ownCloud makes heavily use of PROPFIND
>> to achieve the sync process. Each sync operation (e.g. upload, modify and