Subject: [Storagesync] Mechanisms to synchronize client file systems with Internet-based data storage services <storagesync.ietf.org>

From: Hugo González Labrador <[email-masked]>
Date: Thu, 03 Dec 2015 22:14:46 +0100

>Thursday, Dec 3, 2015 1:41 PM Jakub Moscicki wrote:
>Could you please explain why you think it does not scale? There are schemes in which you do not need to propfind >the entire remote tree to discover changes, only a sequence of propfinds in a direct path leading to a changed >entity. That’s not optimal for sure (several calls needed, depending on the depth of the change) but if it comes to >WebDAV itself, apart from a bloated XML representation, I do not see much problem.

>It doesn't scale because you've just described a lockstep process for locating changes; instead of there being one >transaction to get the set of changes from a known common version, you have multiple steps, and the number of >steps increases at least logarithmically (I am guessing, since I don't know your algorithm) as the number of >changes since last sync increases or the size of the tree increases.   If you are syncing continuously this may not >be a big deal, but I don't think that's a reasonable assumption.

This depends on which part of your system you want to put the
synchronisation logic. The approach Jakub has described delegates the
sync responsibility to clients and the server does not do too much job
and can scale  to handle the blast of possible PROPFIND requests.
On the other hand, as you mentioned, the server can facilitate the sync
job to clients creating snapshots of the system in a given moment and
give the clients a delta. This approach puts much more load on the
server and it is more difficult to scale.
Both types of sync has their own benefits and drawbacks and the election
should be made on the use case you want for your business.

>This also has the problem that you have no separation of metadata and data, and no versioning, which means that >you have to have a client-server model, and you can't have a distributed peer model.

I totally disagree here, you can have your metadata fully detached from
the data and be able to sync. Also, you can expose versions trough
WebDAV creating your own logic or use WebDAV extensions that deal with
that. The RFC https://www.ietf.org/rfc/rfc3253.txt gives GIT-like
version control to WebDAV Resources.

>Okey, but what you need to remember the client state and pass it on to the server to calculate the diff state. Any >good ideas how to do that?

>Keep the metadata sorted (e.g. by modification time), and do a diff whenever a new version is generated.   Versions >that have never been shared with other peers should be trimmed once they are not the head version.   When peer >A wants to sync with peer B, it announces the version it has and the version it last remembers syncing with peer B.   >If Peer B remembers that version, then the difference can be computed trivially, O(n), and is minimal (n*2 at worst), >where n is the number of changes.   If Peer B has lost its memory, then it has to send the complete inventory of its >head version, which Peer A can then compare with its head to see what's changed.

---

From: "Fei Song" <[email-masked]>
Date: Fri, 4 Dec 2015 11:27:05 +0800

Hi Hugo,

I think the condition mentioned by Ted is more generic.(please correct me here)
It would be useful to discuss a common scenario.
Do you have any good approaches (or some examples) to seperate the metadata and data?

--------------
Fei Song
>_______________________________________________
>Storagesync mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/storagesync

---

From: Ted Lemon <[email-masked]>
Date: Fri, 04 Dec 2015 03:54:30 +0000

> I think the condition mentioned by Ted is more generic.(please correct me here)

RFC 3253 would basically meet my use case, but has the problems I mentioned earlier.   Jakub is right that what I suggest would impose a greater load on the core repository if there is one, but this is a problem that github has solved, so I don't think that's really a serious issue--if it works with github's funding model, it probably would work with Box/Google/etc.   But to be fair, I am not personally really concerned about whether it works with Box/Google/etc.   They may already consider what they have "good enough" and not be interested in what we are discussing here.   That was certainly Igor's and EKR's position at the microphone in Yokohama!

One of my big concerns with RFC 3253 is that because it's piggybacked on top of WebDAV, it's really complicated.   It's quite an achievement to do that with WebDAV's native architecture, and I'm willing to be convinced that it's good enough, but I've never heard of an implementation of RFC 3253 that I could use.

Hm, actually I'm not sure it was Jakub who was talking about RFC 3253--the threading got a bit confusing in there.   If I got that wrong, apologies!

--
Sent from Whiteout Mail - https://whiteout.io

My PGP key: https://keys.whiteout.io/[email-masked]

---

From: Hugo González Labrador <[email-masked]>
Date: Fri, 04 Dec 2015 14:47:56 +0100

On Fri, Dec 4, 2015, at 04:27 AM, Fei Song wrote:
> Do you have any good approaches (or some examples) to separate the
> metadata and data?
> 

There are already synchronisation platforms built on top of such
concept.

Usually, an object storage service like Amazon S3/ OpenStack Swift/Ceph
RadosGW is used just to keep binary data in a fast flat namespace.

Then, a supporting component, detached from the data store, keeps track
of which metadata is attached to the data. This metadata includes
mtimes, Etags, paths and platform ACLs among others.

I know some products that use this approach:

- ownCloud when configured to use an Object Storage as the Primary User
Storage. (Metadata is handled by a MySQL database)
- CERN EOS Storage System. (Metadata is handled in high performance
in-memory structures).
- DropBox. (As far as I know it uses S3 behind. For metadata it is
unknown, but probably not on S3).
- ClawIO will have an implementation of this approach in the next phase
using Swift.

---

From: Ted Lemon <[email-masked]>
Date: Fri, 04 Dec 2015 14:34:49 +0000

Friday, Dec 4, 2015 8:47 AM Hugo González Labrador wrote:
> [...]

Oh, that's very interesting.   Thanks for pointing that out.   I'm (a) glad to know that it's not just me, and (b) interested in what ownCloud and Cern have done.   I had looked at ownCloud a while back and decided that I wasn't interested, but now I'm curious again.

---

From: Linhui Sun <[email-masked]>
Date: Fri, 4 Dec 2015 23:18:33 +0800

> unknown, but probably not on S3).
Yes, Dropbox uses its own control server for metadata management and S3 for
content data. See http://annasperotto.org/papers/2012/imc140-drago.pdf

> > >>Could you please explain why you think it does not scale? There are
> schemes in which you do not need to propfind >the entire remote tree to
> discover changes, only a sequence of propfinds in a direct path leading to
> a changed >entity. That’s not optimal for sure (several calls needed,
> depending on the depth of the change) but if it comes to >WebDAV itself,
> apart from a bloated XML representation, I do not see much problem.
> > >
> > >>It doesn't scale because you've just described a lockstep process for
> locating changes; instead of there being one >transaction to get the set of
> changes from a known common version, you have multiple steps, and the
> number of >steps increases at least logarithmically (I am guessing, since I
> don't know your algorithm) as the number of >changes since last sync
> increases or the size of the tree increases.   If you are syncing
> continuously this may not >be a big deal, but I don't think that's a
> reasonable assumption.
> > >
> > >>This also has the problem that you have no separation of metadata and
> data, and no versioning, which means that >you have to have a client-server
> model, and you can't have a distributed peer model.
> > >
> > >>Okey, but what you need to remember the client state and pass it on to
> the server to calculate the diff state. Any >good ideas how to do that?
> > >
> > >>Keep the metadata sorted (e.g. by modification time), and do a diff
> whenever a new version is generated.   Versions >that have never been
> shared with other peers should be trimmed once they are not the head
> version.   When peer >A wants to sync with peer B, it announces the version
> it has and the version it last remembers syncing with peer B.   >If Peer B
> remembers that version, then the difference can be computed trivially,
> O(n), and is minimal (n*2 at worst), >where n is the number of changes.
>  If Peer B has lost its memory, then it has to send the complete inventory
> of its >head version, which Peer A can then compare with its head to see
> what's changed.
> > >

---

From: "Fei Song" <[email-masked]>
Date: Sun, 6 Dec 2015 11:25:18 +0800

Hi Hugo,

It's quite nice to know these. Thanks for sharing~ Especially the example part

--------------
Fei Song

---

From: "Fei Song" <[email-masked]>
Date: Sun, 6 Dec 2015 11:37:37 +0800

Hi Ted,

--------------
Fei Song
>Oh, that's very interesting.   Thanks for pointing that out.   I'm (a) glad to know that it's not just me, and (b) interested in what ownCloud and Cern have done.   I had looked at ownCloud a while…

In this mailing list, there was a topic named “[Storagesync] Some preliminary investigations on ownCloud” initialed by Linhui and contributed by Jakub.
A URL http://cs3.ethz.ch/program.html was also pointed.