Subject: [Cellar] Adding private elements to Matroska

From: Jerome Martinez <[email-masked]>
Date: Fri, 13 Apr 2018 11:35:44 +0200

I am working on a project related to Matroska and I would like to add 
some private elements to Matroska, at different levels (TrackEntry, 
BlockGroup, Segment).
These elements add no specific features for media playback.

As there is no indication about how to add private elements (no element 
is reserved for hosting private sub elements), Matroska conformance 
checkers currently flag new elements as "unknown".

I would like to debate about how to add new elements to Matroska, for 
some specific needs so more flagged "private", and for future needs for 
Matroska (e.g. HDR metadata was added during CELLAR work, what should be 
done in the case CELLAR already released the spec?).

Some questions:
- IMO a new element should not make a new version mandatory, as new 
element are ignored by players, but what should be the behavior of a 
conformance checker? error raising if the element is not known, 
information only? Maybe we need a paragraph about it in specs.
- How to manage private elements? Difficulty is to be sure an element 
identifier used by third party A is not reused by third party B due to B 
not knowing about A.On my side I suggest to just add it to 
ebml_matroska.xml without specific details in spec itself, see 
https://github.com/Matroska-Org/matroska-specification/pull/223/files 
for an example, so everyone is aware of identifiers reserved for a third 
party tool. Should such elements be in a separate document, mp4ra style?

Jérôme

---

From: Tobias Rapp <[email-masked]>
Date: Fri, 13 Apr 2018 13:55:54 +0200

> party tool. Should such elements be in a separate document, mp4ra style?

There has been some discussion about private elements on the list:
https://mailarchive.ietf.org/arch/msg/cellar/GwatcU8PFLwC4FFFWVyriDZtPR8

I think the best option would be to reserve some identifier range for 
private use (similar to Private Use Area in Unicode).

Regarding new to-be standardized elements: The question seems to be when 
to increment the DocTypeVersion value in a muxer application. Only when 
the standard is complete or already when it's clear that the element 
will be added?

Regards,
Tobias

---

From: hubblec4 <[email-masked]>
Date: Fri, 13 Apr 2018 21:13:14 +0200

Hi all,

Am 13.04.2018 um 11:35 schrieb Jerome Martinez:
> As there is no indication about how to add private elements (no 
> element is reserved for hosting private sub elements), Matroska 
> conformance checkers currently flag new elements as "unknown".
>
> some specific needs so more flagged "private", and for future needs 
> for Matroska (e.g. HDR metadata was added during CELLAR work, what 
> should be done in the case CELLAR already released the spec?).
>
> Some questions:
> identifier used by third party A is not reused by third party B due to 
> B not knowing about A.On my side I suggest to just add it to 
> for an example, so everyone is aware of identifiers reserved for a 
> third party tool. Should such elements be in a separate document, 
> mp4ra style?
>
> Jérôme

I have read your PR in Matroska (yet nobody has answered) and it is 
first time reading about RAWcooked. I'm sure there are not much people 
how know about this project.

To add new Matroska elements for this "little" project is in my eyes too 
much. But I can understand how important it is for you and the project.
I wait already a  long time for a PR but It has become quiet around 
Matroska.
When I have more time to work on Matroska Menu I'm afraid it will take a 
lot of time to add new elements.

Then I thought about this problem and maybe is the solution useful.
Inspired by Haali TRACKSETEX which uses the Matroska Tags!

Here my suggestion:

Add a top-level Tag with a SimpleTag with Title RAWCOOKED. Then add for 
all your RAWcooked elements nested-SimpleTags.
In the RAWcooked project is a RawcookedBlockGroupelement(probably very 
often used) which is distributed of the entire file.
To collect all RawcookedBlockGroup values, you must read the entire file 
which takes a while, but with Matroska Tags a reader needs less milli 
seceonds.
You could change all RAWcooked element values without re-muxing the mkv 
file.
You could change the RAWcooked project structure and have also backward 
compatibility.

Every "private" project could add his own Matroska Tags structure. Every 
developer can then decide which projects are supported(Matroska 
reader/Splitter).

For HEVC Hdr data we should add new Matroska elements because this are 
stream base data.

Kind regards
Martin Below

---

From: Michael Bradshaw <[email-masked]>
Date: Fri, 13 Apr 2018 19:27:27 +0000

I concur with Martin. For this particular case, I don't think new elements
should be introduced just for RAWcooked. RAWcooked, could instead add an
attachment or tag that contains its needed metadata.

In general, I'm opposed to private elements in Matroska. If, for some
reason, attachments and tags aren't feasible, I could concede to adding a
new global master element that is comprised of two child binary elements:
one acting as an identifier, and one acting as a payload. That way any
arbitrary private info could be identified stored anywhere in the file. But
I'd prefer to avoid that, if possible.

---

From: Moritz Bunkus <[email-masked]>
Date: Fri, 13 Apr 2018 21:56:24 +0200

Hey,

> arbitrary private info could be identified stored anywhere in the
> file. But I'd prefer to avoid that, if possible.

That would be the equivalent of a tagging system with arbitrary key/value
pairs, albeit without the existing targeting facilities (the
\Segment\Tags\Tag\Targets element). I don't like that approach much,
e.g. because it would mean duplication of data if the same private data
should be applied to multiple tracks.

Additionally the scope of such elements wouldn't be entirely clear: would
there be a difference between that element being a direct child of a
\Segement\Tracks\TrackEntry and being a child of
e.g. \Segment\Tracks\TrackEntry\Audio?

I'd rather see the use of the existing attachments & tags systems for this.

Kind regards,
mosu

---

From: Steve Lhomme <[email-masked]>
Date: Sun, 15 Apr 2018 14:55:37 +0200

Hi,

2018-04-13 11:35 GMT+02:00 Jerome Martinez <[email-masked]>:
> I am working on a project related to Matroska and I would like to add some
> private elements to Matroska, at different levels (TrackEntry, BlockGroup,
> Segment).
> As there is no indication about how to add private elements (no element is
> reserved for hosting private sub elements), Matroska conformance checkers
> currently flag new elements as "unknown".

OK, that's something we need to define. And that should be at the EBML
level, not Matroska.

- how to add new elements that will not be assumed to be invalid IDs
by past parsers/validators ?
- how to define elements that are not meant to be merged in the main
DocType specs (like Matroska v5) but on the side ?

We already have the case of the DivX elements that are currently in
the XML file we use to define all the elements. But they should not be
part of the Matroska specs. It should be a separate XML file that
attaches to the main specs. For example in mkvalidator you can specify
that you are validating a DivX MKV file which allows these extra
elements. Otherwise they are not allowed.

> I would like to debate about how to add new elements to Matroska, for some
> specific needs so more flagged "private", and for future needs for Matroska
> (e.g. HDR metadata was added during CELLAR work, what should be done in the
> case CELLAR already released the spec?).

The HDR metadata make sense in Matroska. The DivX one are specific to
their use (trick playback) and yours also seem to be related to a
particular case that don't make sense elsewhere.

I would not call them private though as they will probably be defined
publicly (the DivX ones are or were at the time they made them).

For truly private elements we could do nothing at all and let people
making use of them dealing with the issues. It depends how we solve
the cases of DivX and RAWCooked. If we add some info in the EBML
header then they could benefit from the same feature.

> Some questions:
> - IMO a new element should not make a new version mandatory, as new element
> are ignored by players, but what should be the behavior of a conformance
> checker? error raising if the element is not known, information only? Maybe
> we need a paragraph about it in specs.

It depends if the element is meant to be merged in the main spec or
not. From the beginning Matroska was designed to be able to play with
as few elements defined as possible. The elements that need to be
added are the ones essential for playback. HDR falls in that category,
if you don't store the information the file won't play correctly.
Chapter elements used by ordered edition also affect playback so would
probably need to bump the Matroska version.

The DivX one is special as it's not necessary for playback, it just
adds an extra playback mode. And it's not used by anyone else and uses
an older DocType version. But I would move it out the current specs.
If anything, to show how it's done to add extensions to Matroska. On
the other hand if EBML header additions are needed, all the existing
files would be invalid...

> identifier used by third party A is not reused by third party B due to B not
> knowing about A.On my side I suggest to just add it to ebml_matroska.xml
> without specific details in spec itself, see
> https://github.com/Matroska-Org/matroska-specification/pull/223/files for an
> example, so everyone is aware of identifiers reserved for a third party
> tool. Should such elements be in a separate document, mp4ra style?

If it's private you have no way of knowing it doesn't collide with
anything else. Wether it's element IDs or a private element with a
sub-ID. You just can't tell if it's not used anywhere else or if it
will never be used. That's another good reason to make it public, even
if you (not YOU) don't want to share what you really do with the data.

>
> Jérôme
>
> _______________________________________________
> Cellar mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cellar

-- 
Steve Lhomme
Matroska association Chairman

---

From: Jerome Martinez <[email-masked]>
Date: Sun, 15 Apr 2018 15:37:52 +0200

On 13/04/2018 21:27, Michael Bradshaw wrote:
> I concur with Martin. For this particular case, I don't think new 
> elements should be introduced just for RAWcooked. RAWcooked, could 
> instead add an attachment or tag that contains its needed metadata.

Current implementation of RAWcooked uses an attachment, this method has 
some advantages, e.g. can survive over a remux if attachment are copied, 
but also some drawbacks, e.g. all is in a single place (maxOccurs for 
"Attachments" element is 1) which may be a problem in some corner cases 
(redundancy, need to keep in memory or elsewhere all RAWcooked data 
before being able to store the file at the end of the encoding...).
As Tags element has no maxOccurs, I could use it for storing data 
corresponding to e.g. the previous cluster (Tags is at Segment level 
only), with a TagName having a "magic value" specific to my project (as 
I do with attachments right now) and TagBinary containing the 
corresponding data, but it is usable only at the Segment level, so not 
generic way to define private content in other Matroska elements.

> In general, I'm opposed to private elements in Matroska.

Even if the outcome for RAWcooked is to avoid to create Matroska 
elements, other projects may decide that this is the only solution for 
them, and if we (as people working on Matroska spec) don't define a way 
to store private elements, people may do as DivX did (if I remember 
well, DivX first decided to create some new Matroska elements, then 
Matroska people put them in spec in order to avoid so get collision), 
which is maybe not the way we want it.

Most modern formats have private elements (MPEG-TS has 
registration_format_identifier in descriptors for knowing what is used 
for table_id>=0x80,  AVC/HEVC uses "user_data_unregistered" in user_data 
followed by uuid_iso_iec_11578 for uniqueness, MP4/MOV uses "uuid" atom 
also followed by an UUID...), even AV1 is planning 
METADATA_TYPE_PRIVATE_DATA metadata type for storing private data, I 
guess it is not for nothing ;-), so I think that planning nothing for 
people wanting to store unspecified content is a good method, if we plan 
that Matroska is even more used that currently, so any company would be 
interested in storing private content as they do for years when they use 
a format.

> (...)

---

From: Steve Lhomme <[email-masked]>
Date: Sun, 15 Apr 2018 15:47:02 +0200

2018-04-15 15:37 GMT+02:00 Jerome Martinez <[email-masked]>:
> Current implementation of RAWcooked uses an attachment, this method has some
> advantages, e.g. can survive over a remux if attachment are copied, but also
> some drawbacks, e.g. all is in a single place (maxOccurs for "Attachments"
> element is 1) which may be a problem in some corner cases (redundancy, need
> to keep in memory or elsewhere all RAWcooked data before being able to store
> the file at the end of the encoding...).
> corresponding to e.g. the previous cluster (Tags is at Segment level only),
> with a TagName having a "magic value" specific to my project (as I do with
> attachments right now) and TagBinary containing the corresponding data, but
> it is usable only at the Segment level, so not generic way to define private
> content in other Matroska elements.
>
>
> Even if the outcome for RAWcooked is to avoid to create Matroska elements,
> other projects may decide that this is the only solution for them, and if we
> (as people working on Matroska spec) don't define a way to store private
> elements, people may do as DivX did (if I remember well, DivX first decided
> to create some new Matroska elements, then Matroska people put them in spec
> in order to avoid so get collision), which is maybe not the way we want it.
>
> registration_format_identifier in descriptors for knowing what is used for
> table_id>=0x80,  AVC/HEVC uses "user_data_unregistered" in user_data
> followed by uuid_iso_iec_11578 for uniqueness, MP4/MOV uses "uuid" atom also
> followed by an UUID...), even AV1 is planning METADATA_TYPE_PRIVATE_DATA
> metadata type for storing private data, I guess it is not for nothing ;-),
> so I think that planning nothing for people wanting to store unspecified
> content is a good method, if we plan that Matroska is even more used that
> currently, so any company would be interested in storing private content as
> they do for years when they use a format.

I think you're comparing to formats that are not easily extensible by
nature (I might be wrong). EBML/Matroska allows private extensions
without breaking anything. DivX did it, WebM did it. It's just the
validation/remuxing of files that may cause problems.

---

From: Moritz Bunkus <[email-masked]>
Date: Sun, 15 Apr 2018 15:49:10 +0200

Hey,

I'm not particularly against private _data_ inside Matroska, but more
against private _elements_ as we already have ways of attaching arbitrary
data (tags & attachments). We could simply define a schema for private tag
names and be done with it.

Kind regards,
mosu

---

From: Steve Lhomme <[email-masked]>
Date: Sun, 15 Apr 2018 16:00:14 +0200

But would we define an element for each level where one can appear
(that would be a lot) ? Or do we define a global element similar to
Void/CRC32 that is open bar for whatever you put it in, but a bit too
loose for validation.

The latter may be a simpler possibility and we use an element ID with
2 or 3 bytes to avoid false positives. Like Michael suggested an ID +
value may be useful too.

The issue would be for readers, they will have to map and signal this
element every time it's found.

At least this approach is a lot simpler that defining a whole system
to put in the EBML header.

---

From: Jerome Martinez <[email-masked]>
Date: Sun, 15 Apr 2018 16:24:04 +0200

On 15/04/2018 15:47, Steve Lhomme wrote:
>> [...]
> nature (I might be wrong).

MP4/MOV has a similar method about element, there are 2^32 possibilities.
But it does not prevent conflicts, as 2 entities may use the same 
element name (2^32 possibilities, but people may limit their choice to a 
subset of possibilities, e.g. easily readable in English), I guess this 
is the reason "uuid" atom appeared later in files.

>   EBML/Matroska allows private extensions

Taking the example of DivX, issue is if 2 entities use accidentally the 
same element name before seeing that the other has chose the same 
element name.

---

From: Jerome Martinez <[email-masked]>
Date: Sun, 15 Apr 2018 16:33:39 +0200

> names and be done with it.

Issue I see here is that tags are at Segment level only, so does not 
permit to be in other elements.
A workaround would to permit Tags (or Tag) elements everywhere (like 
Void and CRC32), but IMO Tags are very specific (in specs: "specific to 
Tracks/Chapters"), not broad enough. I would a dedicated Matroska 
element for that.
While I think to RAWcooked usage because it is my use case, I try to 
think to all potential possibilities and potentially usage may be 
everywhere in Matroska elements (e.g. for MOV file, "uuid" atom can be 
everywhere, from doc: "UUID atoms for specific extensions may be placed 
in any position where a succession of atoms is permitted")

---

From: Tobias Rapp <[email-masked]>
Date: Mon, 16 Apr 2018 08:58:06 +0200

> same element name before seeing that the other has chose the same 

Some months ago when working with AVI/RIFF I had the need to store some 
"sparse" file data. Basically it was an index file with all the metadata 
chunks except for the actual media ("data" chunk), plus some custom 
chunks. There I did choose a "unique" 2-char ASCII prefix for the RIFF 
identifiers in the hope that it would not collide with any chunks in the 
input AVI file.

When Matroska gets more popular I assume something similar to the AVI 
indexing above will be implemented for Matroska and I would be in a 
similar situation to pick some "unique" element identifiers.

In my scenario there is no problem regarding validation on output side, 
as the private elements are ignored when copying elements from the index 
file into the output file.

Regards,
Tobias