Subject: Re: [Cellar] On the multiplicity of Info elements

From: Steve Lhomme <[email-masked]>
Date: Sat, 2 Jan 2016 09:47:47 +0100

2015-12-30 10:18 GMT+01:00 Moritz Bunkus <[email-masked]>:
> Hey,
>
> I only remember the discussion around Tracks being multiple, not
> particularly for the other ones. Our intent way back when was to allow
> muxers to write multiple instances of _the same information_ in
> different places in order to make the file more resilient against
> damage or incomplete downloads with protocols like BitTorrent.

Yes, that's the idea for the Track Info as it's vital to the usability
of the file, as well as the Segment Info. I'm not sure it's used in
practice though.
Since the goal of CELLAR is archiving solutions it may still make sense.

> The same reasoning could be applied to Info. Both elements are
> absolutely crucial to playback; the other level 1 elements safe for the
> clusters simply aren't.
>
>> SeekHead, Info, Cluster, Tracks, and Tags are multiple.
>
> SeekHead and Cluster must be multiple. SeekHead in order to allow moving
> a SeekHead to the end of the file while still referencing it from the
> start (so that normal players will still find it quickly). Cluster for
> obvious reasons.
>
>> And Cues, Attachments, and Chapters are non-multiple.
>
> I have no idea why Tags is multiple and these three aren't.
>
> To me the following would make sense:
>
> - Info, Tracks – multiple but only if each instance contains the same
>   information
>
> - SeekHead, Cluster – multiple without restrictions
>
> - Attachments, Chapters, Cues, Tags – single
>
> Kind regards,
> mosu
>
> _______________________________________________
> Cellar mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cellar
>

-- 
Steve Lhomme
Matroska association Chairman

---

From: Dave Rice <[email-masked]>
Date: Sun, 3 Jan 2016 10:53:11 -0500

> On Jan 2, 2016, at 3:47 AM, Steve Lhomme <[email-masked]> wrote:
> 

Perhaps to declare that an Element may be repeated but must be repeated identically should be a new EBML Element Attributes, so there can be a distinction between the repeatability Segment/Info and the repeatability of SimpleBlock.

>> clusters simply aren’t.

But what should happen when the read finds differences in repeated-but-should-be-identical elements?

For a scenario of two differing Info Elements, VLC and FFmpeg use different Info Elements. Which use is correct? Since the use of repeated-identical elements is resilience a deviation between the two could be expected, so we should suggest how the reader should respond.

>> - Attachments, Chapters, Cues, Tags – single

I can understand Attachments and Tags being multiple as it could allow attachments or tags to be added to a file without having to re-write too many bytes.

Dave

---

From: Steve Lhomme <[email-masked]>
Date: Mon, 4 Jan 2016 08:11:23 +0100

> Perhaps to declare that an Element may be repeated but must be repeated identically should be a new EBML Element Attributes, so there can be a distinction between the repeatability Segment/Info and…

That might be good. After all not elements make sense as repeated
ones. For example in Matroska you don't want a Cluster (timestamped
data) to be repeated.

> But what should happen when the read finds differences in repeated-but-should-be-identical elements?

Good question. Maybe repeated elements should have a CRC ? If a CRC is
wrong (or not found) the parser could look for a copy.

> For a scenario of two differing Info Elements, VLC and FFmpeg use different Info Elements. Which use is correct? Since the use of repeated-identical elements is resilience a deviation between the t…

It was designed for recovery tools. It may not be good to change
players for such cases. It would make them more complex. (unless an
elegant/easy solution is found).

> I can understand Attachments and Tags being multiple as it could allow attachments or tags to be added to a file without having to re-write too many bytes.

Yes. But then there should be a way for the player to know about these
beforehand. Good players scan Matroska files beforehand anyway (unless
it's live streaming).

---

From: "Sebastian G. <bastik>" <[email-masked]>
Date: Mon, 4 Jan 2016 19:59:27 +0100

04.01.2016, 08:11 Steve Lhomme:
>>>> I only remember the discussion around Tracks being multiple,
>>>> not particularly for the other ones. Our intent way back when
>>>> was to allow muxers to write multiple instances of _the same
>>>> information_ in different places in order to make the file more
>>>> resilient against damage or incomplete downloads with protocols
>>>> like BitTorrent.
>>> 
>>> Yes, that's the idea for the Track Info as it's vital to the
>>> usability of the file, as well as the Segment Info. I'm not sure
>>> it's used in practice though. Since the goal of CELLAR is
>>> archiving solutions it may still make sense.
>> 
>> Perhaps to declare that an Element may be repeated but must be
>> repeated identically should be a new EBML Element Attributes, so
>> there can be a distinction between the repeatability Segment/Info
>> and the repeatability of SimpleBlock.
> 
>>>> absolutely crucial to playback; the other level 1 elements safe
>>>> for the clusters simply aren’t.
>> 
>> But what should happen when the read finds differences in
>> repeated-but-should-be-identical elements?
> 
> Good question. Maybe repeated elements should have a CRC ? If a CRC
> is wrong (or not found) the parser could look for a copy.

I like the CRC idea for repeated elements, but it still does not define
how players should behave if they encounter two elements, even with
valid CRCs, not matching each other.

There would have to be a recommendation. "Use always the first
occurrence of an element." or "If an element occurs repeated and its
values differ, the last occurrence is the one that should be used."

Obviously tools that create such files are violating the specifications
since it should not be allowed to create repeated elements with
differing values. On the other hand should it be hard to break playback.
I prefer uniform behavior among players.

>> For a scenario of two differing Info Elements, VLC and FFmpeg use
>> different Info Elements. Which use is correct? Since the use of
>> repeated-identical elements is resilience a deviation between the
>> two could be expected, so we should suggest how the reader should
>> respond.
> 

For differences due to transmission errors a CRC for repeated elements
seems a good solution.

Players have to do something with repeated elements. I don't know what
they do, but there should be a recommended way they should handle such
cases. If a player breaks, that is OK, as long as the file was violating
the specs. A player should behave in an expected way.

>>>> SeekHead and Cluster must be multiple. SeekHead in order to
>>>> allow moving a SeekHead to the end of the file while still
>>>> referencing it from the start (so that normal players will
>>>> still find it quickly). Cluster for obvious reasons.
>>>> 
>>>> - Info, Tracks – multiple but only if each instance contains
>>>> the same information
>>>> 
>> I can understand Attachments and Tags being multiple as it could
>> allow attachments or tags to be added to a file without having to
>> re-write too many bytes.
> 
> Yes. But then there should be a way for the player to know about
> these beforehand. Good players scan Matroska files beforehand anyway
> (unless it's live streaming).
> 

I agree with having a mechanism for players to know about them beforehand.

--
Sebastian

---

From: Steve Lhomme <[email-masked]>
Date: Tue, 5 Jan 2016 09:20:00 +0100

> valid CRCs, not matching each other.

Also what about repeated elements that are not master elements. You'd
have no way of telling which is the best version. So repeated should
probably be master elements. Maybe CRC should be mandatory too (not
sure if real life files already follow this rule). That's the only way
a parser would be able to tell which version is correct, as far as I
can see.

> values differ, the last occurrence is the one that should be used."

Not necessarily. Bogus data could be on the first one. The goal is not
to write a version of the element and then an updated version in the
file. It has to be the same data. If you want to clear the first
version, you should use the Void EBML element. When the file is
originally written, repeated values should be exactly the same. That
means elements that have a file offset relative to a child element
would not be equal. Luckily in Matroska offset positions are always
relative to the Segment, so a Level-0 element.

http://www.matroska.org/technical/specs/notes.html#Position_References

> the specs. A player should behave in an expected way.

IMO what makes sense from a player point of view is to read an
element. If there's a CRC, it's broken and the semantic says the
element can be repeated, then it should look for a valid version.
Otherwise it shouldn't have to wonder which element to use if it
encounters another one. IMO repeated elements only make sense if
there's a CRC (whichever form it may take).

Another rule for repeatable elements: the element MUST be unique at
that level (not multiple).

---

From: Michael Niedermayer <[email-masked]>
Date: Tue, 5 Jan 2016 17:49:58 +0100

> a parser would be able to tell which version is correct, as far as I

i dont disagree but i think "no way" is not correct
its possible that external means like errors from reading data could
be used to detect which repeated versions are bad, also parsing errors
could indicate what is bad and then if there are 3 or more copies
simple majority "voting" on a byte per byte base could be used
to construct a "good" version, this too could be done when all copies
fail CRC checks. No idea if that would be usefull in any actual real
world usecase, but a player or repair tool trying to be exceptionally
resillient to damages could do things like that

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Many things microsoft did are stupid, but not doing something just because
microsoft did it is even more stupid. If everything ms did were stupid they
would be bankrupt already.

---

From: Dave Rice <[email-masked]>
Date: Wed, 6 Jan 2016 02:06:05 -0500

> On Jan 5, 2016, at 3:20 AM, Steve Lhomme <[email-masked]> wrote:
> 

Here is a draft for an EBML Schema Attribute to be used in the definition of EBML Elements for what I’m referring to as Identically-Recurring Elements.

====
A boolean to express if the EBML Element may occur within its Parent Element more than once but that each recurrance within that Parent Element MUST be identical both in storage and semantics. Such Elements are referred to as Identically-Recurring Elements. In this case, identical copies of an EBML Element are permitted to be stored multiple times within the same Parent Element in order to increase data resillience and optimize the use of EBML in transmission. Identically-Recurring Elements SHOULD include a CRC-32 Element as a Child Element. If a Parent Element contains more than one copy of an Identically-Recurring Element which includes a CRC-32 Child Elememnt then the first instance of the Identically-Recurring Element with a valid CRC-32 value should be used for interpretation. If a Parent Element contains more than one copy of an Identically-Recurring Element which does not contain a CRC-32 Child Elememnt then the first instance of the Identically-Recurring Element should be used for interpretation. If the `identical` attribute is not expressed for that Element then that Element is considered to not have a requirement for identical expression within the same Parent Element. The `identical` attribute is only valid if the Element is not set to `multiple`, otherwise the `identical` attribute shall be ignored.
====

The text may be seen in context with the other attributes at https://github.com/MediaArea/ebml-specification/blob/d108c14d1f1c748d1f3f50e58d4057208325f892/specification.markdown#ebml-schema-element-attributes

Some pending questions:
- Should Identically-Recurring Elements recommend or mandate inclusion of a CRC-32? I suggestion recommend and not mandate for reverse compability.
- Is 'Identically-Recurring Elements’ a decent short name for these types of Elements?
- Should the ‘identical’ attribute apply to only Master-elements? I think it could be open. I could see scenarios to place the CRC-32 both at the beginning and end of the Parent Element.
- I’m thinking that ‘multiple’ means the Element may recur within its Parent Element and means that there are multiple semantics. I suggest that ‘identical’ Elements not be ‘multiple’ as an ‘identical’ Element recurs but only with a single and non-multiple semantic meaning. OK?
- If there are multiple copies of identical Elements, I wrote in the draft that if there’s a CRC than the first copy with valid CRC be used, else the first copy be used. Potentially we could say first valid copy be used, but then need to say fully what valid means.
- I agree that an identical Element should not be used to change the semantics over time.

Btw while working on this I found a mkv file from Lavf54.29.104 where a Track Entry element contained two different copies of DefaultDuration. They only differed by 1, but is the first one valid and the second one invalid?

Dave Rice

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Wed, 06 Jan 2016 03:48:30 -0800

> - Should Identically-Recurring Elements recommend or mandate inclusion of a CRC-32? I suggestion recommend and not mandate for reverse compability.

The text should say something about this justification (i.e., when it 
would be reasonable to violate the SHOULD).

---

From: "Sebastian G. <bastik>" <[email-masked]>
Date: Wed, 6 Jan 2016 20:12:16 +0100

06.01.2016, 08:06 Dave Rice:
> Here is a draft for an EBML Schema Attribute to be used in the
> definition of EBML Elements for what I’m referring to as
> Identically-Recurring Elements.
> 
> ==== A boolean to express if the EBML Element may occur within its
> Parent Element more than once but that each recurrance within that

Is 'recurrance' spelled correctly? (Not a native speaker). 'recurrence'
is suggested as search term suggestion.

> Parent Element MUST be identical both in storage and semantics. Such
> Elements are referred to as Identically-Recurring Elements. In this
> case, identical copies of an EBML Element are permitted to be stored
> multiple times within the same Parent Element in order to increase
> data resillience and optimize the use of EBML in transmission.

'resillience' seems to have an error. 'resilience'.

> Identically-Recurring Elements SHOULD include a CRC-32 Element as a
> Child Element. If a Parent Element contains more than one copy of an
> Identically-Recurring Element which includes a CRC-32 Child Elememnt

s/Elememnt/Element

> then the first instance of the Identically-Recurring Element with a
> valid CRC-32 value should be used for interpretation. If a Parent
> Element contains more than one copy of an Identically-Recurring
> Element which does not contain a CRC-32 Child Elememnt then the first

s/Elememnt/Element

'If a Parent Element contains more than one copy of an
Identically-Recurring Element which does not contain a CRC-32 Child
Element "or if CRC-32 Child Elements are present but neither of those
are valid" then the first instance of the Identically-Recurring Element
should be used for interpretation.'

I took your sentence and added (noted by "") the case something would
contain CRCs, but neither of them would be valid. Feel free to ignore
this suggestion. It might be a rare case.

It is certainly possible to have it as separate sentence. Maybe it makes
it easier to read. Maybe even easier to understand.

> instance of the Identically-Recurring Element should be used for
> interpretation. If the `identical` attribute is not expressed for
> that Element then that Element is considered to not have a
> requirement for identical expression within the same Parent Element.
> The `identical` attribute is only valid if the Element is not set to
> `multiple`, otherwise the `identical` attribute shall be ignored. 
> ====
> 
> The text may be seen in context with the other attributes at
> https://github.com/MediaArea/ebml-specification/blob/d108c14d1f1c748d1f3f50e58d4057208325f892/specification.markdown#ebml-schema-element-attributes
>
>  Some pending questions:

Backward compatibility seems to be a higher goal than CRCs. I think it
should be even possible to play a file without checking CRCs, which does
not contradict having CRCs in the first place.

Like Timothy B. Terriberry points out, reasons for not having CRCs on
those elements should be given in the specification.

> - Is 'Identically-Recurring Elements’ a decent short name for these types of Elements?

I have no objection. IRE could be used when talking about those
elements, unless it is within official documents.

> - Should the ‘identical’ attribute apply to only Master-elements? I think it could be open. I could see scenarios to place the CRC-32 both at the beginning and end of the Parent Element.

I have not enough knowledge to make an informed statement.

> - I’m thinking that ‘multiple’ means the Element may recur within its Parent Element and means that there are multiple semantics. I suggest that ‘identical’ Elements not be ‘multiple’ as an ‘identi…

I have not enough knowledge to make an informed statement.

> - If there are multiple copies of identical Elements, I wrote in the draft that if there’s a CRC than the first copy with valid CRC be used, else the first copy be used. Potentially we could say fi…

I'm neutral, but agree that if just valid is used it would have to be
explained what valid means.

> - I agree that an identical Element should not be used to change the semantics over time.

Agreed.

--
Sebastian

---

From: Dave Rice <[email-masked]>
Date: Sun, 10 Jan 2016 13:35:33 -0500

> On Jan 6, 2016, at 2:12 PM, Sebastian G. <bastik> wrote:
> 

fixed

> 'resillience' seems to have an error. 'resilience’.

fixed

>> Identically-Recurring Element which includes a CRC-32 Child Elememnt

fixed

>> Element which does not contain a CRC-32 Child Elememnt then the first

fixed

> not contradict having CRCs in the first place.

I agree.

> those elements should be given in the specification.

Based on Tim and Sebastian’s comments I extended the sentence to be:

"Identically-Recurring Elements SHOULD include a CRC-32 Element as a Child Element; this is especially recommended when EBML is used for long-term storage or transmission.”

Listing reasons to not have a CRC Child Element seems awkward. Realistically I think the primary reason that CRC elements are not used is because many muxers don’t support adding them, although the EBML specification has long contained: "All level 1 elements SHOULD include a CRC-32.” Instead of addressing reasons why not I added some reasons for why with references to ong-term storage and transmission.

> I have not enough knowledge to make an informed statement.

Unless there’s a good reason to do so, I won’t restrict the ‘identical’ attribute to certain Element Types, thus the specifications for EBML formats can decide how to use this.

>> - I agree that an identical Element should not be used to change the semantics over time.

Here is a new version for review:

Definition for a new EBML Schema Attribute: identical
===============
A boolean to express if the EBML Element may occur within its Parent Element more than once but that each recurrence within that Parent Element MUST be identical both in storage and semantics. Such Elements are referred to as Identically-Recurring Elements. In this case, identical copies of an EBML Element are permitted to be stored multiple times within the same Parent Element in order to increase data resilience and optimize the use of EBML in transmission. Identically-Recurring Elements SHOULD include a CRC-32 Element as a Child Element; this is especially recommended when EBML is used for long-term storage or transmission. If a Parent Element contains more than one copy of an Identically-Recurring Element which includes a CRC-32 Child Element then the first instance of the Identically-Recurring Element with a valid CRC-32 value should be used for interpretation. If a Parent Element contains more than one copy of an Identically-Recurring Element which does not contain a CRC-32 Child Element or if CRC-32 Child Elements are present but none are valid then the first instance of the Identically-Recurring Element should be used for interpretation. If the `identical` attribute is not expressed for that Element then that Element is considered to not have a requirement for identical expression within the same Parent Element. The `identical` attribute is only valid if the Element is not set to `multiple`, otherwise the `identical` attribute shall be ignored.
===============

Dave Rice

---

From: Dave Rice <[email-masked]>
Date: Sun, 10 Jan 2016 13:40:50 -0500

> On Jan 10, 2016, at 1:35 PM, Dave Rice <[email-masked]> wrote:
> 
> 

I added this text as a pull request, so that it may be reviewed in context. https://github.com/Matroska-Org/ebml-specification/pull/47
Dave Rice

---

From: Steve Lhomme <[email-masked]>
Date: Mon, 11 Jan 2016 10:46:33 +0100

> resillient to damages could do things like that

True, some "forensic" could be applied to recover the damaged data
when all CRC fails. I don't think the specs should specify such
recovery techniques, nor a general reader should care too much about
that. The first level of CRC (find one that matches) should be good
enough for that level.

---

From: Steve Lhomme <[email-masked]>
Date: Mon, 11 Jan 2016 11:05:26 +0100

>> not contradict having CRCs in the first place.

The backward compatibility is important so that older files are
handled correctly by future (stricter) parsers. In the case where a
recurring element doesn't have any CRC, I don't think future parsers
should discard such elements. So we could still make it mandatory and
keep backward compatibility. There will be plenty of other cases for
old files to not be compliant to the current/future specs and still
play fine.

> Unless there’s a good reason to do so, I won’t restrict the ‘identical’ attribute to certain Element Types, thus the specifications for EBML formats can decide how to use this.

If CRC is mandatory, then it's not possible. If it's not mandatory
it's possible to have unique elements duplicate. But then how do you
know which one to use ? Also having unique elements repeated seem
bogus and some parsers may not like that. It's a contradiction to what
"unique" means.

---

From: Michael Niedermayer <[email-masked]>
Date: Mon, 11 Jan 2016 13:43:24 +0100

> that. The first level of CRC (find one that matches) should be good

i agree

the possibility of such "forensic" techniques should be considered
in the design of the spec though at places where it might be usefull
iam not sure CRCs are the only such place ...

[...]

The worst form of inequality is to try to make unequal things equal.
-- Aristotle

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Mon, 11 Jan 2016 08:18:30 -0800

Dave Rice wrote:
> Listing reasons to not have a CRC Child Element seems awkward.
> Realistically I think the primary reason that CRC elements are not
> used is because many muxers don’t support adding them, although the
> EBML specification has long contained: "All level 1 elements SHOULD
> include a CRC-32.” Instead of addressing reasons why not I added

So this is really, "SHOULD, unless it is too much work for you to bother"?

I say this as someone who has had every SHOULD in one of my own drafts 
systematically dissected with the question, "Why not MUST? When would it 
be okay to violate this SHOULD?" late in the review process. I am trying 
to save you pain further down the line.

---

From: Dave Rice <[email-masked]>
Date: Mon, 11 Jan 2016 16:27:04 -0500

> On Jan 11, 2016, at 11:18 AM, Timothy B. Terriberry <[email-masked]> wrote:
> 
> Dave Rice wrote:
> So this is really, "SHOULD, unless it is too much work for you to bother”?

That does seem honest. ;)

From a sample set of 59,881 Matroska files uploaded to archive.org only 119 of them include CRC-32 Elements. Almost 0.2%! Additionally the webm Document Type lists the CRC-32 Element as unsupported. So I think it’s true that it’s rarely bothered with. The implementation of CRC-32 Elements does add many complications and perhaps demand has been too low to resolve them. Moritz may have comments here.

> I say this as someone who has had every SHOULD in one of my own drafts systematically dissected with the question, "Why not MUST? When would it be okay to violate this SHOULD?" late in the review process. I am trying to save you pain further down the line.

Historically both Matroska and EBML specs have said that "All level 1 elements should include a CRC-32” with a lower-case “should” (that ’should’ was capitalized in a recent review); however the implementation of this recommendation has been minimal.  To change the SHOULD to a MUST, would invalidate approximately 99.8% of Matroska files and 100% of webm files. I realize that this reason alone may not make good language within an RFC.

On the other hand, through discussion here, I do see the use of CRC-32 Elements to be more meaningful especially in the two use cases of the CELLAR working group title: archiving and transmission. 

Perhaps the recommendation for CRC-32 usage (in Level 1 Elements and Identically-Recurring Elements) should be removed from the EBML Specification and moved to the Matroska EBML Schema.

For any webm lurkers, was there any reason (that could be useful here) to set the CRC-32 Element to “Unsupported” within the webm specification?

Other suggestions?

Dave Rice

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Mon, 11 Jan 2016 13:44:03 -0800

Dave Rice wrote:
> implementation of this recommendation has been minimal.  To change
> the SHOULD to a MUST, would invalidate approximately 99.8% of
> Matroska files and 100% of webm files. I realize that this reason

Well, in the IETF (in contrast with the traditional approach of, say, 
MPEG) we are allowed to specify requirements on muxers separately from 
demuxers. So it's perfectly permissible to say a muxer MUST include CRCs 
while simultaneously saying a demuxer MUST be able to read a file 
without them. Then you wouldn't invalidate the existing media, just the 
existing software that produced them.

However, if you don't think it's worth trying to force muxers to change 
their implementation (or don't believe that a SHOULD or MUST in a spec 
will actually cause them to change), then perhaps the most appropriate 
level for this requirement is MAY?

> Perhaps the recommendation for CRC-32 usage (in Level 1 Elements and
> Identically-Recurring Elements) should be removed from the EBML
> Specification and moved to the Matroska EBML Schema.

That might also be reasonable (I hold no strong opinion).

---

From: Moritz Bunkus <[email-masked]>
Date: Tue, 12 Jan 2016 14:36:56 +0100

Hey,

> From a sample set of 59,881 Matroska files uploaded to archive.org
> only 119 of them include CRC-32 Elements. Almost 0.2%! Additionally
> the webm Document Type lists the CRC-32 Element as unsupported. So I
> think it’s true that it’s rarely bothered with. The implementation of
> CRC-32 Elements does add many complications and perhaps demand has
> been too low to resolve them. Moritz may have comments here.

That's pretty much spot on. The CRC32 element was added comparatively
late; a lot of muxers had already been created by that time.

Most requests I've received where more concerned with error recovery
than with error detection. This is something that CRC32 wasn't meant
for; additionally CRC32 elements weren't meant to be written in the
cluster elements which make up the bulk of a Matroska file's data.

Yes, there is a feature request[1] for adding support in mkvmerge and
mkvinfo. It has been created in 2010, but except for those two people
mentioned in the bug I haven't really received any other requests for
such a functionality.

Kind regards,
mosu

[1] https://github.com/mbunkus/mkvtoolnix/issues/543

---

From: Steve Lhomme <[email-masked]>
Date: Thu, 14 Jan 2016 11:39:49 +0100

2016-01-11 22:44 GMT+01:00 Timothy B. Terriberry <[email-masked]>:
> Well, in the IETF (in contrast with the traditional approach of, say, MPEG)
> we are allowed to specify requirements on muxers separately from demuxers.
> So it's perfectly permissible to say a muxer MUST include CRCs while
> simultaneously saying a demuxer MUST be able to read a file without them.
> Then you wouldn't invalidate the existing media, just the existing software
> that produced them.

That sounds like a good way to do it. CRC requirements can only be on
the muxer side, and defined at the semantic level.

> their implementation (or don't believe that a SHOULD or MUST in a spec will
> actually cause them to change), then perhaps the most appropriate level for
> this requirement is MAY?
>

Yes, they should move to the Matroska side of the specs.

---

From: "Sebastian G. <bastik>" <[email-masked]>
Date: Thu, 14 Jan 2016 18:54:30 +0100

> [1] https://github.com/mbunkus/mkvtoolnix/issues/543

Hi,

to said ticket you added two entries [2][3] which mention a problem with
verifying checksums.

Would that be much easier to handle if the CRC would be about the data
that should be present in the file, but are omitted due to default
values aren't written? Compared to creating a checksum of the data that
actually get written and verifying said checksum.

I had a rather strong opinion towards including a CRC that covers the
elements that are written. I'm just curious and if something can be
solved elegantly rather than having to invest too much resources into
it, it might be worth reconsidering. After all it would be defined
within a proper specification.

Back then when I offered the bounty I expected matroska to be used a lot
more often in the future by other people than me. My usecase for
checksums was indeed just error recovery. However I am not against
extending the potential usecases to have applications use it to detect
errors.

[2] https://github.com/mbunkus/mkvtoolnix/issues/543#issuecomment-171036571
[3] https://github.com/mbunkus/mkvtoolnix/issues/543#issuecomment-171422587

--
Sebastian

---

From: Dave Rice <[email-masked]>
Date: Fri, 15 Jan 2016 01:47:18 -0500

> On Jan 12, 2016, at 8:36 AM, Moritz Bunkus <[email-masked]> wrote:
> 
> Hey,
> 

There are certainly Matroska files that use CRC-32 within Clusters.

See: http://archive.org/download/WC14UruVsCost/FIFA.World.Cup.2014.Uruguay.Vs.Costa.Rica.SDTV-WDTeam_1st.mkv
Muxing App Haali DirectShow Matroska Muxer 1.13.138.14
Writing App gdsmux

Or http://archive.org/download/aawqws/Covert.Affairs.S01E05.mkv
Muxing App libebml-0.7.5 & libmatroska-0.7.7
Writing App VirtualDubMod 1.5.10.2 (build 2540/release)

Or http://archive.org/download/archive_actu1_Lara/Lara.S01.E24.mkv
Muxing App libebml v0.7.8 + libmatroska v0.8.1
Writing App Cyberlink Matroska Muxer v1.0.0

That last example uses CRC-32 Elements only within the Cluster Element and not elsewhere.

---

From: Dave Rice <[email-masked]>
Date: Fri, 15 Jan 2016 02:28:46 -0500

> On Jan 14, 2016, at 12:54 PM, Sebastian G. <bastik> wrote:
> 

The Matroska files with CRC Elements store a checksum of what data is stored in the Master-element rather than what is interpreted (via defaults).

For instance in the test file suite, test2.mkv has a CRC value in the EBML Header Element with a value of 0x67277B3A. The rest of the Master-element stores DocType, DocTypeVersion, and DocTypeReadVersion. The other EBML Header Elements of EBMLVersion, EBMLReadVersion, EBMLMaxIDLength, and EBMLMaxSizeLength are not stored but interpreted as the defined default values. The data as stored matched the checksum, not the data as interpreted.

I think switching the CRC to calculate interpreted data rather than stored data would present numerous challenges.

> extending the potential usecases to have applications use it to detect

A few use cases:

If I have a Matroska file in long-term storage and migrate it to new storage (hard drive to new hard drive, or LTO Tape, future format, etc), the CRC values can be verified to ensure the data is transferred authentically and that the data on the new form of storage.

Alternatively if the data is moved across a network the received file can be verified to ensure that it is still valid. This is possible with FLAC files as they have an MD5 in the header and CRCs per frame.

Also if a Matroska file in an archive needs to be changed. For instance to add contextual metadata or a supporting attachment, the file could be edited in place without affecting other Level 1 elements. After the edit the file could still be verified to ensure the Level 1 elements unaffected by the edit have maintained authenticity. A whole file checksum wouldn’t work in this case since the whole file is intended to be different before and after the change, but the parts that aren’t involved in the change can still be verified.

As discussed, CRC elements can allow authority in selecting one of an Identically-Recurring Element if there are many in a damaged file.

There’s also a use case with secure or legal recordings. The CameraV project, https://guardianproject.info/apps/camerav/, uses matroska and checksums in a way to allow verifiable recordings.

---

From: Moritz Bunkus <[email-masked]>
Date: Fri, 15 Jan 2016 09:10:27 +0100

Hey,

> The Matroska files with CRC Elements store a checksum of what data is
> stored in the Master-element rather than what is interpreted (via
> defaults).

That's good as it is and it's the only possible way to handle it.

> I think switching the CRC to calculate interpreted data rather than
> stored data would present numerous challenges.

Worse, it'd be impossible. Each time the specs change wrt. to default
values (e.g. by adding a new element with a default value) the CRC would
implicitly be invalidated as the CRC was calculated over the old set of
elements with a default value.

The problem I'm mentioned on my issue on GitHub is solely in libEBML's
design. A library's design should not force the specs to follow it. The
proposed CRC specs are fine as they are. No need to change them. I'll
just have to fix/extend libEBML accordingly (not easy, but doable).

> > My usecase for checksums was indeed just error recovery.

Just to be clear. "Error recovery" usually means both "error detection"
and "error correction". CRCs are only meant and useful for the
recovery. On their own they don't do anything for correction.

Only when combined with redundancy e.g. in the form of having multiple
"tracks" element with the same segment can they be used for
correction.

Kind regards,
mosu

---

From: Moritz Bunkus <[email-masked]>
Date: Fri, 15 Jan 2016 09:18:31 +0100

Hey,

> There are certainly Matroska files that use CRC-32 within Clusters.

True, I'm just debating their value ;) Sure, you can detect that there's
a bit error in that cluster, but what are you going to do? Drop the
whole cluster? OK, you can inform the user about it and she can watch
the appropriate section in the file in order to determine how bad it is.

My current goal for MKVToolNix is to handle CRCs everywhere:

1. mkvmerge will write them for each level 1 element

2. mkvmerge will check existing CRCs and emit warnings about elements
   where the CRC doesn't match the data

3. mkvpropedit will include them in each level 1 element it changes/adds
   as will the GUI's header/chapter/attachment editors

4. mkvinfo will print existing CRCs and emit warnings about elements
   where the CRC doesn't match the data

Unfortunately all of those require changes in libEBML. The ones required
for 1. and 3. are not that bad (handling of masters in which a CRC
element is already present) and are ABI-compatible.

The changes for 2. and 4. are much more involved, though; for one CRC
elements read from storage must be passed through to the application (at
the moment libEBML tries to handle them transparently while mkvinfo
wants to output the element's presence and data); additionally the
verification is currently broken (the value's correct on disk, though)
and I haven't really understood why that is. It's also very slow. Fixing
this most likely requires changing the ABI, and making it fast/use less
memory bandwidth will require even more ABI-breaking changes.

It'll take some time until this has been implemented.

Kind regards,
mosu

---

From: Michael Niedermayer <[email-masked]>
Date: Fri, 15 Jan 2016 11:59:41 +0100

[...]
> recovery. On their own they don't do anything for correction.

Its possible to correct single bit, burst and some multi bit errors
with CRCs

it depends on the CRC polynom used and on the data length

see: http://guru.multimedia.cx/index.php?s=crc+correcting

[...

Republics decline into democracies and democracies degenerate into
despotisms. -- Aristotle