Subject: [Cellar] CRC support for Matroska in FFmpeg

From: Dave Rice <[email-masked]>
Date: Thu, 6 Oct 2016 17:10:03 -0400

Hi all,

I wanted to point out that, thanks to James Almer, FFmpeg now supports writing CRC values into the Top-Level Elements of Matroska as it "SHOULD". This is good news for the use of Matroska in long-term preservation as Matroska files written in this way have embedded fixity on its components from the point when it was first created.

http://ffmpeg.org/pipermail/ffmpeg-devel/2016-October/thread.html#200586 <http://ffmpeg.org/pipermail/ffmpeg-devel/2016-October/thread.html#200586>

Dave Rice

---

From: Kieran O Leary <[email-masked]>
Date: Thu, 6 Oct 2016 23:06:40 +0100

Hello!

On Thu, Oct 6, 2016 at 10:10 PM, Dave Rice <[email-masked]> wrote:

> Hi all,
>
> I wanted to point out that, thanks to James Almer, FFmpeg now supports
> writing CRC values into the Top-Level Elements of Matroska as it "SHOULD".
> This is good news for the use of Matroska in long-term preservation as
> Matroska files written in this way have embedded fixity on its components
> from the point when it was first created.
>
> http://ffmpeg.org/pipermail/ffmpeg-devel/2016-October/thread.html#200586
>
>

 Wonderful news, yet another preservation friendly feature. Thanks James
Almer!

 I have a few questions in terms of use:

1. Once commited to git head, are these CRC elements written by default?

2. How does one perform a fixity check, is it by decoding using ffmpeg -i
video.mkv -f null -

3. You mention the Top-Level Elements. Are there any aspects of the
Matroska container that do not have CRC values?

Kindest regards,

Kieran.

---

From: Jerome Martinez <[email-masked]>
Date: Fri, 7 Oct 2016 10:56:56 +0200

Le 07/10/2016 à 00:06, Kieran O Leary a écrit :
> Hello!
>
> On Thu, Oct 6, 2016 at 10:10 PM, Dave Rice <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Hi all,
>
>     I wanted to point out that, thanks to James Almer, FFmpeg now
>     supports writing CRC values into the Top-Level Elements of
>     Matroska as it "SHOULD". This is good news for the use of Matroska
>     in long-term preservation as Matroska files written in this way
>     have embedded fixity on its components from the point when it was
>     first created.
>
>     <http://ffmpeg.org/pipermail/ffmpeg-devel/2016-October/thread.html#200586>
>
>
>
>  Wonderful news, yet another preservation friendly feature. Thanks 
> James Almer!
>

It is committed, and enabled by default (there is an option for 
disabling it)

>
> 2. How does one perform a fixity check, is it by decoding using ffmpeg 
> -i video.mkv -f null -

If I understand well the code, there is only writing support, no check 
currently on FFmpeg side.
You need to use e.g. MediaConch (we already test the CRC and report an 
error if wrong).
I guess we could ask for having automatic test of CRC as it is the case 
for FFV1.

> Matroska container that do not have CRC values?

Just to be clear: in the spec, you can choose also to have more precise 
CRC, i.e. at any level, if you want. It permits to find the source of 
the error with more precision. So having "only" top-level is good enough 
(e.g. you have 1 CRC per bunch of frames, in addition to FFV1 CRC per 
slice).
that said, CRC is currently "only" on the Segment element (all elements 
directly below the Segment element), so the EBML header (few bytes) is 
not "protected" (it is a "level 0"), as well as the Segment element 
header. I think that's OK, we can easily detect issues on EBML header 
and Segment element header without CRC and it is also easily fixable.

Jérôme

---

From: Dave Rice <[email-masked]>
Date: Fri, 7 Oct 2016 09:27:34 -0400

> On Oct 7, 2016, at 4:56 AM, Jerome Martinez <[email-masked]> wrote:
> 
>> On Thu, Oct 6, 2016 at 10:10 PM, Dave Rice <[email-masked] <mailto:[email-masked]>> wrote:
>> Hi all,
>> 
>>  Wonderful news, yet another preservation friendly feature. Thanks James Almer! 
>> 
> It is committed, and enabled by default (there is an option for disabling it)
> 
>> 
>> 2. How does one perform a fixity check, is it by decoding using ffmpeg -i video.mkv -f null - 
> 
> If I understand well the code, there is only writing support, no check currently on FFmpeg side.
> You need to use e.g. MediaConch (we already test the CRC and report an error if wrong).
> I guess we could ask for having automatic test of CRC as it is the case for FFV1.

mkvalidator validates embedded CRC values.

>> 3. You mention the Top-Level Elements. Are there any aspects of the Matroska container that do not have CRC values?
> 
> Just to be clear: in the spec, you can choose also to have more precise CRC, i.e. at any level, if you want. It permits to find the source of the error with more precision. So having "only" top-level is good enough (e.g. you have 1 CRC per bunch of frames, in addition to FFV1 CRC per slice).
> that said, CRC is currently "only" on the Segment element (all elements directly below the Segment element), so the EBML header (few bytes) is not "protected" (it is a "level 0"), as well as the Segment element header. I think that's OK, we can easily detect issues on EBML header and Segment element header without CRC and it is also easily fixable.

There is a sample with a CRC-32 at Level 1 (within the EBML Header) in https://github.com/Matroska-Org/matroska-test-files. The EBML specifications going back to Feb. 2003 <https://web.archive.org/web/20030226072615/http://ebml.sourceforge.net/specs/> have recommended that "All level 1 elements should include a CRC-32."

Dave Rice