Subject: [Cellar] matroska and side data vs timecode

From: Dave Rice <[email-masked]>
Date: Tue, 29 Oct 2019 12:04:07 -0400

Hi cellar working group,

I wanted to share some recent and ongoing developments within https://github.com/cellar-wg/matroska-specification <https://github.com/cellar-wg/matroska-specification>.

Some background of recent Matroska side data work:

Recently a pull request at https://github.com/cellar-wg/matroska-specification/pull/287 was merged into the Matroska specification which adds a structure to identify, store, and describe side data to Matroska frames. This adds a block mapping to the TrackEntry that identifies the side data to a registry entry such as:

<Segment>
 <!-- skipped elements here -->
 <Tracks>
  <!-- lots of other track data here -->
  <TrackEntry>
   <BlockAdditionMapping>
    <BlockAddIDValue> {arbitrary value used to link BlockAddID (see the Cluster below) } </BlockAddIDValue>
    <BlockAddIDName>  {a human readable name for that side data}.   </BlockAddIDName>
    <BlockAddIDType>  {a registered ID for that type of side data}  </BlockAddIDType>
   </BlockAdditionMapping>
 </Tracks>
 <!-- then within the Cluster which stores audiovisual data within frames in Blocks, there is -->
 <Cluster>
  <BlockGroup>
   <Block> { some data, probably an audio or video frame } </Block>
   <BlockAdditions>
    <BlockMore>
     <BlockAddID> {arbitrary value used to link BlockAddIDValue (see above) } </BlockAddID>
     <BlockAdditional> { insert side data here, if the BlockAddID value is >=2 then look up what the data is in the BlockAdditionMapping } </BlockAdditional>
    </BlockMore>
   </BlockAdditions>
  </BlockGroup>
 </Cluster>
</Segment>

Within BlockMore, if the BlockAddID = 1 then the Codec Mapping defines what is within BlockAdditional as side data, but if the BlockAddID is >=2, then the identity of the BlockAdditional side data is defined within the BlockAdditionalMapping of the associated TrackEntry.

Active work on Matroska and timecode:

I drafted a pull request at https://github.com/cellar-wg/matroska-specification/pull/348 that builds upon the new side data structure defined above. This pull request registers a BlockAddIDType for a 64 bit binary expression of timecode as defined by SMPTE ST12-1:2014 (which is mostly hours, minutes, seconds, frames, a drop frame flag, and other a few other flags and binary groups). This would allow timecode to be associated with Matroska frames such as:

<Segment>
 <!-- skipped elements here --> 
 <Tracks>
  <!-- lots of other track data here -->
  <TrackEntry>
   <BlockAdditionMapping>
    <BlockAddIDValue>2</BlockAddIDValue>
    <BlockAddIDName>SMPTE ST 12-1 timecode</BlockAddIDName>
    <BlockAddIDType>121</BlockAddIDType>
   </BlockAdditionMapping>
 </Tracks>
 <!-- then within the Cluster which stores audiovisual data within frames in Blocks, there is -->
 <Cluster>
  <BlockGroup>
   <Block> { a video frame } </Block>
   <BlockAdditions>
    <BlockMore>
     <BlockAddID>2</BlockAddID>
     <BlockAdditional> { SMPTE ST12-1:2014 64-bit binary representation of a 07:32:54;18 } </BlockAdditional>
    </BlockMore>
   </BlockAdditions>
  </BlockGroup>
  <BlockGroup>
   <Block> { the next video frame } </Block>
   <BlockAdditions>
    <BlockMore>
     <BlockAddID>2</BlockAddID>
     <BlockAdditional> { SMPTE ST12-1:2014 64-bit binary representation of a 07:32:54;19 } </BlockAdditional>
    </BlockMore>
   </BlockAdditions>
  </BlockGroup>
 </Cluster>
</Segment>

With the BlockAdditions, BlockMore, BlockAddID, and BlockAdditional Elements, I think this would add 18 bytes per timecode expression. Adding timecode to one hour of PAL video (25 frames * 3600 seconds/hour) would add about 1.5 MB to the file.

Implementation Considerations:

Thinking ahead of this work, I think it would be worthwhile to add a dedicated section to the Matroska specification on timecode and describe two methods for adding a reference timecode to a Matroska Segment.

1. Using tags. Make an official tag “TIMECODE” to store a string of the timecode expression that correlates to the Matroska timestamp of 0. This approach is limited to storing timecode that is incremental and continuous. Defaults for the incrementation and the behavior of the timecode could be described as part of the tag definitions and feasibly another tag or tags could be reserved for timecode flags when the timecode behavior is not a default. Defining a use of a “TIMECODE” tag would provide the benefit of standardizing an existing practice in FFmpeg’s Matroska muxer, where rewrapping from a timecode-supportive format to Matroska carries the timecode value over as a tag.

2. Using BlockAdditions as described above. This adds more overhead as each timecode value would be written, but would support non-continuous timecode. In FFmpeg, timecode is sometimes carried as side-data such as from the decklink device via https://git.ffmpeg.org/gitweb/ffmpeg.git/commit/0946c0ec177dc48ef0677f890aa42d95e667c417 <https://git.ffmpeg.org/gitweb/ffmpeg.git/commit/0946c0ec177dc48ef0677f890aa42d95e667c417>. I’d be interested if the Block Addition Mapping described above could be used to carry the timecode side data from the decklink device (or other incoming stream with timecode side data) and store the result within the written Matroska BlockGroups.

Comments welcome.

Thanks much,
Dave Rice

---

From: Derek Buitenhuis <[email-masked]>
Date: Wed, 30 Oct 2019 15:02:48 +0000

Comments inlined below.

On 29/10/2019 16:04, Dave Rice wrote:
> _Implementation Considerations:_
>

To me, requiring timecodes to be incremental and continuous pretty much defeats a good chunk of the point of timecodes, no?

> 2. Using BlockAdditions as described above. This adds more overhead as each timecode value would be written, but would support non-continuous timecode. In FFmpeg, timecode is sometimes carried as side-data such as from the decklink device via https://git.ffmpeg.org/gitweb/ffmpeg.git/commit/0946c0ec177dc48ef0677f890aa42d95e667c417. I’d be interested if the Block Addition Mapping described above could be used to carry the timecode side data from the decklink device (or other incoming stream with timecode side data) and store the result within the written Matroska BlockGroups.

As long as it's an optional element, I don't see a downside to the small overhead - especially considering the large size of many/most mezzanines that carry timecodes.

Unrelated to the above two comments: How do you intent to rectify the historical (and confusing) use of 'timecode' in Matroska to be timestamp with the newly introduced timecode functionality?

Cheers,
- Derek

---

From: Kieran O Leary <[email-masked]>
Date: Wed, 30 Oct 2019 15:39:46 +0000

Hi Derek

On Wed, 30 Oct 2019, 15:03 Derek Buitenhuis, <[email-masked]>
wrote:

> *Implementation Considerations:*
>
> Thinking ahead of this work, I think it would be worthwhile to add a
> dedicated section to the Matroska specification on timecode and describe
> two methods for adding a reference timecode to a Matroska Segment.
>
> 1. Using tags. Make an official tag “TIMECODE” to store a string of the
> timecode expression that correlates to the Matroska timestamp of 0. This
> approach is limited to storing timecode that is incremental and continuous.
> Defaults for the incrementation and the behavior of the timecode could be
> described as part of the tag definitions and feasibly another tag or tags
> could be reserved for timecode flags when the timecode behavior is not a
> default. Defining a use of a “TIMECODE” tag would provide the benefit of
> standardizing an existing practice in FFmpeg’s Matroska muxer, where
> rewrapping from a timecode-supportive format to Matroska carries the
> timecode value over as a tag.
>
> To me, requiring timecodes to be incremental and continuous pretty much
> defeats a good chunk of the point of timecodes, no?
>
Kind of, but all tape capture software that I've encountered just stores
the starting timecode value along with framerate and all other timecodes
are based on this starting value. So it won't represent the source
videotape if there's a timecode jump or reset. But the method that Dave
outlines here would at least allow Matroska to mirror what happens with
Aja/blackmagic capture tools using v210/mov as the capture format.
It's very difficult to find tools or examples of noncontinuous timecode
tracks..

Best,

Kieran O'Leary
Irish Film Institute

---

From: Derek Buitenhuis <[email-masked]>
Date: Wed, 30 Oct 2019 15:49:59 +0000

On 30/10/2019 15:39, Kieran O Leary wrote:
> Kind of, but all tape capture software that I've encountered just stores the starting timecode value along with framerate and all other timecodes are based on this starting value. So it won't represent the source videotape if there's a timecode jump or reset. But the method that Dave outlines here would at least allow Matroska to mirror what happens with Aja/blackmagic capture tools using v210/mov as the capture format.
> It's very difficult to find tools or examples of noncontinuous timecode tracks..

An example would be a digitally produced (in an NLE) mezzanine from a number of ingested sources, where its timecodes map to the timecods in the original sources - I've seen this used fairly often to facilitate figuring out where a particular edit may come from in a final export or mezzanine.

I would need to dig up which NLEs, specifically, produce these.

Cheers,
- Derek

---

From: Dave Rice <[email-masked]>
Date: Wed, 30 Oct 2019 12:08:25 -0400

> On Oct 30, 2019, at 11:49 AM, Derek Buitenhuis <[email-masked]> wrote:
> 
>> Kind of, but all tape capture software that I've encountered just stores the starting timecode value along with framerate and all other timecodes are based on this starting value.

This is most of my experience as well when transferring as well, but to QuickTime files, where a timecode track stores a single sample value and some properties to define the incrementation. These timecode tracks are limited since they are based upon a single value but very easy to edit, whereas storing timecode per frame would require edits throughout the file to (for instance) apply an offset to the entire timecode track.

Still I like the idea of formalizing support for continuous timecode based upon a single value within a tag (as an alternate option to timecode as side data), as it would be very easy to add or manipulate. Also it is what is already done when ffmpeg rewraps a video with timecode to Matroska.

>> So it won't represent the source videotape if there's a timecode jump or reset. But the method that Dave outlines here would at least allow Matroska to mirror what happens with Aja/blackmagic capture tools using v210/mov as the capture format.
I’ve seen some of these too. In QuickTime the timecode track will still store a single value but then the track has an edit list to manage the offsets at the points of discontinuity. Also in stream based tape transfers like dv and m2ts from DV or HDV tape, non-continuous timecode is rather common. Still IIUC support for timecode as side data in ffmpeg is limited to a few formats and devices, so getting an input from decklink would pass along timecode strings as side data, but the dv decoder currently would not pass along such data.
> I would need to dig up which NLEs, specifically, produce these.

Thanks,
Dave

---

From: Tobias Rapp <[email-masked]>
Date: Wed, 30 Oct 2019 17:28:52 +0100

On 30.10.2019 16:39, Kieran O Leary wrote:
> videotape if there's a timecode jump or reset.

Our NOA FrameLector software stores timecode start offset and jumps 
during the capturing process. It is using VITC and/or LTC as sources, 
depending on what is available. The timecode values are stored in an XML 
alongside the media file. When exporting the media file into other 
systems currently only the initial timecode offset is embedded into the 
MXF/MOV output file.

The method of storing timecode as side-data for each frame has the 
disadvantage that all frame side-data must be read when building a 
timecode-to-playtime seeking table in players. Also the timecode 
information is lost when writing an audio-only variant of the file. Thus 
I think it would be better to store timecode data as a separate track, 
similar to how it could be done in MOV.

Best regards,
Tobias