Subject: [decade] Metadata, DECADE Data Objects, and you

From: Richard Alimi <[email-masked]>
Date: Mon, 25 Jul 2011 03:58:43 -0700

In his review of the DECADE Requirements draft, Akbar brought up a
very good question regarding metadata associated with data objects.
In particular, he asked whether such metadata was immutable or not. A
couple of the requirements drafts authors discussed a bit, but it
seems clear that there are a couple of questions here that need input
from the larger WG.

To frame the problem space, lets begin with:
(1) Should DECADE Clients be able to specify key/value pairs stored
alongside the data object's content?  The use case we had in mind
would be to allow an application to specify contextual information
associated with an object, for example "this is chunk 123 of that live
soccer stream I am watching" (of course, not in plain English).  It
could be used if the application crashes and resumes, or is stopped
and resumed on a different device (in which case a simple local
checkpoint won't suffice).  The DECADE Client could then request a
list of objects matching "that soccer stream I am watching", and
retrieve the ones it is missing from the DECADE Server.

Possible answers are:
(1a) Yes:  Then we continue on to (2)..
(1b) No: It might be possible for an application to still achieve what
it wants, in a somewhat more limited way.  For example, it could store
additional metadata as the first K bytes of a data object, and (if we
allow range requests) then the application could determine if this is
the desired object without necessarily requesting the full object.  It
is less clear how we might support querying for a list of matching
objects in this case. Alternatively, one could argue that this is the
application's problem, and DECADE should not be in the business of
storing this arbitrary metadata at all.

(2) Should this additional metadata associated with a data object be
immutable?  In particular, should an application be prohibited from
adding/modifying/removing these key/value pairs *after* a data object
has been stored?  It is important to note that data objects (the
contents themselves) in DECADE are already immutable, which lends
itself to some very attractive simplicity.

Regardless of whether metadata is mutable or immutable, we would need
to decide: Should the data object's name (content hash) be a hash over
the data *and* the metadata?
- If yes, this automatically means that one could in a sense change
the metadata by creating a new data object. Note that the actual
operation can be made to be cheap at the DECADE server, though it does
make the operational model somewhat less simple and implementations
more complex.
- If no, then things like cross-server de-duplication (if an
implementation wished to support that) become much trickier since they
need to deal with conflicts in the key/value pairs associated with
data objects.

Now to whether metadata is immutable. Possible answers are:
(2a) Yes: Then I think we are mostly done, aside from the naming issue
above.  We need to be sure to provide a mechanism (perhaps via the
granted tokens) so that a DECADE Server can validate that the DECADE
Client storing the data is indeed attaching the correct metadata
(similar to how it can already validate that the data being written is
the correct data).  An important question here is: are we giving up
flexibility that is needed?
(2b) No: This seems to open a can of worms with regards to locking and
caching (e.g., intermediaries, or even within applications).

My personal opinion is that the can of worms in (2b) isn't worth
dealing with.  The tradeoffs between (2a) and (1b) are less clear to
me.  Thinking through this some more, I have a slight preference for
the simplicity of (1b), but might be convinced otherwise.

Other thoughts?

Thanks,
Rich

---

From: Songhaibin <[email-masked]>
Date: Mon, 25 Jul 2011 13:09:27 +0000

Hi Rich,

Thank you very much for putting out this interesting issue.

In my personal opinion, we should at least allow the extensibility to include metadata although we may not need to support it right now.

My two cents: Can metadata be stored as a separate data object itself( include associated hash value of the content object as one parameter so as to associate them)? In this case, the metadata can be changed without creating a new object for the content itself, but creating a new object for the metadata iteself. And it still needs to resolve the conflicts if there are conflict metadatas associated with the same content object:(

BR,
-Haibin

________________________________________
From: [email-masked] [[email-masked]] on behalf of Richard Alimi [[email-masked]]
Sent: Monday, July 25, 2011 6:58 PM
To: [email-masked]
Subject: [decade] Metadata, DECADE Data Objects, and you

Possible answers are:

Other thoughts?

Thanks,
Rich
_______________________________________________
decade mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/decade

---

From: "Woundy, Richard" <[email-masked]>
Date: Mon, 25 Jul 2011 14:00:52 +0000

Here is my personal opinion -- not as wg chair.

I think the answer may depend on what *type* of metadata is associated with the data object. If the metadata is related to the physical attributes of the object (eg codec parameters for a media file), it may be immutable. If the metadata is related to the management attributes of the object (eg object lifetime, owner contact info), then it may be desirable to update it over time.

-- Rich

----- Original Message -----
From: Songhaibin [mailto:[email-masked]]
Sent: Monday, July 25, 2011 09:09 AM
To: Richard Alimi <[email-masked]>; [email-masked] <[email-masked]>
Subject: Re: [decade] Metadata, DECADE Data Objects, and you

Hi Rich,

BR,
-Haibin

Possible answers are:

Other thoughts?

Thanks,
Rich

---

From: Richard Alimi <[email-masked]>
Date: Sat, 30 Jul 2011 08:10:57 -0700

Just to update the status, we had a discussion about this during the
meeting. The outcome was that decade will not allow arbitrary
(application-specified) metadata associated with data objects.

We will update the draft to reflect this and post an update by the end of
this coming week.

Thanks,

Rich
On Jul 25, 2011 10:01 AM, "Woundy, Richard" <
[email-masked]> wrote:
> I think the answer may depend on what *type* of metadata is associated
with the data object. If the metadata is related to the physical attributes
of the object (eg codec parameters for a media file), it may be immutable.
If the metadata is related to the management attributes of the object (eg
object lifetime, owner contact info), then it may be desirable to update it
over time.
> In my personal opinion, we should at least allow the extensibility to
include metadata although we may not need to support it right now.
>
> My two cents: Can metadata be stored as a separate data object itself(
include associated hash value of the content object as one parameter so as
to associate them)? In this case, the metadata can be changed without
creating a new object for the content itself, but creating a new object for
the metadata iteself. And it still needs to resolve the conflicts if there
are conflict metadatas associated with the same content object:(