Subject: Re: [Cellar] ReferenceTimecode tracks in Matroska proposal

From: Dave Rice <[email-masked]>
Date: Mon, 11 Jul 2016 23:50:32 -0400

Hi all,

I’m providing an update to this old thread. To summarize conversation:

- debate about Element renames
	There’s some discussion about renaming existing Elements that incorporate the string “TimeCode” to use “TimeStamp”. There’s obvious issues either way as many applications depending on the consistent interoperability of specification-provided Element names. For now I’m using ReferenceTimeCode to distinguish from Matroska’s existing use of the term “Timecode”.

- where to store ReferenceTimecode
	There was some discussion about storage strategy: to store ReferenceTimecode values in BlockGroup so that there may be a timecode per frame or to store ReferenceTimecode data per track in CodecPrivate. Storing ReferenceTimecode values in BlockGroup may make it easier to support non-incremental timecode (for instance I see some m2ts streams with many successive timecode values of 00:00:00;00 then a jump to a much higher value, then incrementing, then a jump to a different number).

For storing a ‘ReferenceTimecode Track’:

Here is an updated proposal:

TrackType:
Add to the TrackType definition to list "0x13: reference timecode,"

CodecID:
Add the following timecode Codec IDs.
Codec ID        | Name               | Description
T_QUICKTIME     | QuickTime Timecode | Timecode data as stored in QuickTime, e.q. tmcd. The CodecPrivate contains all additional data that is stored in the 'tcmd' (timecode) atom in the QuickTime file. See the [QuickTime specification](https://developer.apple.com/library/mac/documentation/QuickTime/RM/MovieBasics/MTEditing/I-Chapter/9TimecodeMediaHandle.html) for the definition of the tmcd atom.

Proposed elements under TrackEntry:

Element Name: ReferenceTimecode
Level:        3
ID:           [EF]
minOccurs:	0
maxOccurs:	1
Default:      -
Type:         m
Description:  ReferenceTimecode settings

Element Name: ReferenceTimecodeTrackUID
Level:        4
ID:           [EF][A1]
minOccurs:	0
maxOccurs:	unbounded
Range:        -
Default:      -
Type:         u
Description:  Specify the TrackUID that the ReferenceTimecode describes. Specifying no track implies that the timecode references all tracks.

Element Name: ReferenceTimecodeSourceLabel
Level:        4
ID:           [EF][A2]
minOccurs:	0
maxOccurs:	1
Range:        -
Default:      -
Type:         8
Description: A label to document or identify the source of the timecode track.

And for ‘tagging' a block with a ReferenceTimecode:

Under BlockGroup:

Element Name: BlockReferenceTimecode
Level:        3
ID:           [EF][A2]
minOccurs:	0
maxOccurs:	1
Range:        -
Default:      -
Type:         s
Description: A signed integer to represent a ReferenceTimecode as a frame number. Players supporting ReferenceTimecode should convert the frame number accordingly to a drop-frame, non-drop-frame display, or HH:MM:SS.mmm display.

Perhaps another element tied to BlockReferenceTimecode to note the preferred ReferenceTimecode display style (drop frame, non dropframe, etc). Also I think there should be a way to determine what to do is many tracks all use this feature, which is prefer for display.

New round of comments is welcome.

Dave Rice

---

From: Jerome Martinez <[email-masked]>
Date: Tue, 12 Jul 2016 08:32:18 +0200

> 	There was some discussion about storage strategy: to store ReferenceTimecode values in BlockGroup so that there may be a timecode per frame or to store ReferenceTimecode data per track in CodecPri…

I don't like the "or" in the storage strategy, at least if it is an 
"exclusive or".
Both strategies exist, sometimes in the same specification (for example, 
GXF container specs use the term "striped" for saying when only the time 
code code of the first frame exists, or if it is per frame).

I think it is important to support both, especially for when you do an 
immediate seek in the file so you don't read first frames and miss the 
only one time code at the beginning of the first cluster if you don't 
parse first frames only for the purpose of finding the time code packet 
(it is a lot easier with MOV because the file offset is in the header so 
a demuxer can seek directly to this file offset and read the value 
before seeking to the expected frame number, this is not the case of 
Matroska).

So we should have a proposal with:
- "stripped" time code (e.g. an MXF time code track, GXF/MOV files with 
only 1 time code value) in the track header
- "non-striped" time codes (e.g. SDTI or MOV files with 1 value per 
frame) in the cluster (note: in other containers, non-striped time codes 
are sometimes spanned across the whole file, sometimes they are in a 
single block at the beginning or at the end of the file).

> Element Name: BlockReferenceTimecode

actually "s" or "u" depends of the CodecPrivate data.
if QuickTime tcNegTimesOK is 1, this is "s", else "u".

I don't like the idea of considering a time code as different than 
another track type.
We should not define the content, it is the job of the "codec" (here: 
QuickTime specs). a time code block must have its own track number (this 
is a track, and it may be linked to 1 or several other track with 
ReferenceTimecodeTrackUID; you can also have several time code tracks so 
the track ID is need for knowing which track we are referring to) and 
time stamp.
I propose that we don't define any format (it is delegated to QuickTime 
spec, so no need to update Matroska specs when Apple updates its spec) 
and that the content is in a Simpleblock.

> New round of comments is welcome.

We could consider an additional element e.g. "CodecPrivateContent" 
(here: which contains the striped time code used for calculation of all 
time codes, or all time codes when time codes are available soon during 
encoding, but in a generic manner it may be used for codec needing 
content data globally, similar to the need of "CodecPrivate") beside 
"CodecPrivate" element (contains only the "init" part of the codec)

---

From: Steve Lhomme <[email-masked]>
Date: Tue, 12 Jul 2016 19:38:17 +0200

2016-07-12 8:32 GMT+02:00 Jerome Martinez <[email-masked]>:
>>         There’s some discussion about renaming existing Elements that
>> incorporate the string “TimeCode” to use “TimeStamp”. There’s obvious issues
>> either way as many applications depending on the consistent interoperability
>> of specification-provided Element names. For now I’m using ReferenceTimeCode
>> to distinguish from Matroska’s existing use of the term “Timecode”.
>>
>>         There was some discussion about storage strategy: to store
>> ReferenceTimecode values in BlockGroup so that there may be a timecode per
>> frame or to store ReferenceTimecode data per track in CodecPrivate. Storing
>> ReferenceTimecode values in BlockGroup may make it easier to support
>> non-incremental timecode (for instance I see some m2ts streams with many
>> successive timecode values of 00:00:00;00 then a jump to a much higher
>> value, then incrementing, then a jump to a different number).
>
>
> Both strategies exist, sometimes in the same specification (for example, GXF
> container specs use the term "striped" for saying when only the time code
> code of the first frame exists, or if it is per frame).
>

That's often the strategy with Matroska. When you can't make
everything work together then have both. Although with time one
solution kills the other one.

> immediate seek in the file so you don't read first frames and miss the only
> one time code at the beginning of the first cluster if you don't parse first
> frames only for the purpose of finding the time code packet (it is a lot
> easier with MOV because the file offset is in the header so a demuxer can
> seek directly to this file offset and read the value before seeking to the
> expected frame number, this is not the case of Matroska).
>
> - "stripped" time code (e.g. an MXF time code track, GXF/MOV files with only
> 1 time code value) in the track header
> - "non-striped" time codes (e.g. SDTI or MOV files with 1 value per frame)
> in the cluster (note: in other containers, non-striped time codes are
> sometimes spanned across the whole file, sometimes they are in a single
> block at the beginning or at the end of the file).

My first impression, as Dave said, is that a Timecode specifies a
frame. This is important especially because video frames may not be
found in their display order (most of the time it's not the case). But
then that means audio frames should have one too. So the question is,
what is a timecode good for ? Is it a permanent stamp to identify when
a frame/instant particularly happen ? Or it is to display and search
precise instants in a file ?

In the case where it's a stamp IMO it's better to keep it as close to
the frame and preferably hard to alter (on purpose or by mistake). It
doesn't really matter the order it's stored in. It's not that hard to
find what you want when you know the Matroska timestamp/timecode
mapping for a track.

In the case of something to search and be displayed incrementally then
it's probably better in a track of it's own. That means to find the
content matching a particular timestamp you first find it in the
track, get the Matroska timestamp of that track and look for the best
corresponding timestamp in other tracks. Maybe once a Cluster, as
Jerome said, is enough for that. That's a bit odd to encode because
everytime you remux a file, with different boundaries the TimeCode
track may or may not be in a Cluster.

Maybe there should be 2 names for timecodes ? So we don't confuse the 2 needs ?

>> T_QUICKTIME     | QuickTime Timecode | Timecode data as stored in
>> QuickTime, e.q. tmcd. The CodecPrivate contains all additional data that is
>> stored in the 'tcmd' (timecode) atom in the QuickTime file. See the
>> [QuickTime
>> specification](https://developer.apple.com/library/mac/documentation/QuickTime/RM/MovieBasics/MTEditing/I-Chapter/9TimecodeMediaHandle.html)
>> for the definition of the tmcd atom.
>>
>> Description:  Specify the TrackUID that the ReferenceTimecode describes.
>> Specifying no track implies that the timecode references all tracks.
>>
> I don't like the idea of considering a time code as different than another
> track type.
> QuickTime specs). a time code block must have its own track number (this is
> a track, and it may be linked to 1 or several other track with
> ReferenceTimecodeTrackUID; you can also have several time code tracks so the
> track ID is need for knowing which track we are referring to) and time
> stamp.
> spec, so no need to update Matroska specs when Apple updates its spec) and
> that the content is in a Simpleblock.
>
>> Description: A signed integer to represent a ReferenceTimecode as a frame
>> number. Players supporting ReferenceTimecode should convert the frame number
>> accordingly to a drop-frame, non-drop-frame display, or HH:MM:SS.mmm
>> display.
>>
>> Perhaps another element tied to BlockReferenceTimecode to note the
>> preferred ReferenceTimecode display style (drop frame, non dropframe, etc).
>> Also I think there should be a way to determine what to do is many tracks
>> all use this feature, which is prefer for display.
>>
> We could consider an additional element e.g. "CodecPrivateContent" (here:
> which contains the striped time code used for calculation of all time codes,
> or all time codes when time codes are available soon during encoding, but in
> a generic manner it may be used for codec needing content data globally,
> similar to the need of "CodecPrivate") beside "CodecPrivate" element
> (contains only the "init" part of the codec)
>
>
> _______________________________________________
> Cellar mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cellar

-- 
Steve Lhomme
Matroska association Chairman

---

From: Jerome Martinez <[email-masked]>
Date: Tue, 12 Jul 2016 21:01:43 +0200

Le 12/07/2016 à 19:38, Steve Lhomme a écrit :
>>> [...]

Here, both strategies are alive for years, it actually depends on the 
continuity in the time code:
- if time code values are continuous, the single value + frame rate is 
enough, "very high compression".
- if time code values are not continues, we need to keep all of them

> precise instants in a file ?

Dave is an archivist, has a specific usage, but he may miss some other 
cases.
(Dave will yell here ;-) )
He uses to see a time code associated to a frame, because this is often 
the case.

As a developer, I see that QuickTime specs, for example, permits a 
different frame rate for the time code track, because the time code 
track is a complete standalone track with its own ID, its own frame 
rate, its own timestamp per "frame". I am afraid that we restrict too 
much the Matrsoka spec and some day we'll not be able to transwrap from 
a different container because someone found that it is useful to use a 
feature which was not making sense before so we did not plan Matroska 
for it.

Let's take a scenario having an issue if we associate time code to a 
frame from another track.
We decide to use FFV1 for video 25fps, and PCM for audio, and a time 
code track at the same frame rate as the video track.
for interleaving purpose, we decide to use the same fps for PCM 48 kHz, 
so 1920 samples per frame.
Great, all fits well, we put the BlockReferenceTimecode in the 
BlockGroup containing the corresponding video frame, and it is synchronized.
Now, someone decides to remove the video stream, and keep only the audio 
and the time code. It makes sense, an audio stream may also have a time 
code, and the time code value changes every 1920 samples (40 ms at 48 
kHz). No more video, so we put the BlockReferenceTimecode in the 
BlockGroup containing the corresponding audio frame, and it is still 
synchronized.
Now, it is decided to compress to FLAC with 1024 samples per FLAC frame 
(here I cheat a bit because it is possible to set a FLAC frame with 1920 
samples, but 1/ it is not mandatory 2/ if we use AC-3 or AAC, it is not 
possible);
ho ho... Now we have a FLAC frame every 21 ms, and a time code expected 
to change every 40 ms. This weird case may happen because the time code 
is associated to a frame rate (change every 1920/48000 second) and FLAC 
compression has decided to have another frame rate (a frame every 
1024/48000 second). Where to put the BlockReferenceTimecode? Impossible 
in the BlockGroup  containing the audio frame (not the same time stamp) 
and if it is alone there is no "Block" associated with the 
BlockReferenceTimecode so no associated time stamp.

Another example is in the case we want to save a time code stream coming 
from a serial interface (e.g. with CDP transport layer), we may have the 
corresponding video only later, so we may want to dump the time code 
alone. How to do if we have a BlockReferenceTimecode element instead of 
content in a Bloc element?

I think to such cases, but in theory a lot more cases are possible in 
containers like MOV, and I prefer to avoid limitations not present in 
other containers.

also: QuickTime time code format (one unsigned integer is a format of 
time code, not the only one), we may specify a lot more different 
formats later (e.g. SMPTE ST-12, SDTI time codes... they are not an 
integer), how will we do if we say that BlockReferenceTimecode is an 
integer?

for these reasons, I argue for having the content of the time code track 
opaque, and embedded in Matroska as any other track (i.e. the content of 
the time code is in the block content as any other stream format).
Link between the time code track and the corresponding video track is 
done by ReferenceTimecodeTrackUID, as in QuickTime.

> Maybe there should be 2 names for timecodes ? So we don't confuse the 2 needs ?

I don't see 2 needs: from my point of view, there is only one need (a 
time code track), and the discussion is about how to technically 
implement 2 technically different cases. This is similar to old 
containers having no "CodecPrivate" elements, and nowadays we have 2 
cases: without CodecPrivate (each I-frame can be decoded after any 
seek), and with CodecPrivate (each I-frame can be decoded only if the 
global data is there). for time code, this is similar (with a slight 
difference: no Block data if there is the data inside the track header 
directly)

> [...]

I think that the first agreement we must have is if we consider a time 
code track as any other track standalone with its own Track ID, time 
stamp... or if it must be associated to a (video) track (and all its 
frames).

***

I propose to keep the proposal from Dave for the track header, remove 
the BlockReferenceTimecode proposal (content is in a Block or 
SimpleBlock), and add an element in the TrackEntry, generic (not only 
for QuickTime time code format, it would be generic as CodecPrivate):

Element Name: CodecPrivateBlock
Level:        3
ID:           [63][A3] (?)
minOccurs:    0
maxOccurs:    1
Range:        -
Default:      -
Type:         b
Description:  Block containing the actual data to be rendered, global to 
whole file, all frames are calculated based on this block content. If 
this element is present, time stamp of this block is 0 and no Block 
element is expected in Clusters for this track.

(I have a doubt about the time stamp: can the Timecode of the first 
Cluster be something different than 0?)

Jérôme

---

From: Thomas Daede <[email-masked]>
Date: Sat, 16 Jul 2016 01:12:49 -0700

> solution kills the other one.

This strategy works when you want to stuff arbitrary data in with
minimal effort in the muxer. However, what is the player supposed to do
when presented with multiple, possibly conflicting copies of this data?

- Thomas

---

From: Steve Lhomme <[email-masked]>
Date: Sat, 16 Jul 2016 15:24:47 +0200

2016-07-12 21:01 GMT+02:00 Jerome Martinez <[email-masked]>:
>>> GXF

At least in Matroska it doesn't exist yet.

>>> only
>>> frame)
> He uses to see a time code associated to a frame, because this is often the
> case.
>
> As a developer, I see that QuickTime specs, for example, permits a different
> frame rate for the time code track, because the time code track is a
> complete standalone track with its own ID, its own frame rate, its own
> timestamp per "frame". I am afraid that we restrict too much the Matrsoka
> spec and some day we'll not be able to transwrap from a different container
> because someone found that it is useful to use a feature which was not
> making sense before so we did not plan Matroska for it.

I agree. Although the plan has always been to be able to put
everything in Matroska. But not necessarily out of Matroska since we
may support things that don't exist elsewhere or with "inferior"
designs.

> Let's take a scenario having an issue if we associate time code to a frame
> from another track.
> We decide to use FFV1 for video 25fps, and PCM for audio, and a time code
> track at the same frame rate as the video track.
> for interleaving purpose, we decide to use the same fps for PCM 48 kHz, so
> 1920 samples per frame.
> Great, all fits well, we put the BlockReferenceTimecode in the BlockGroup
> containing the corresponding video frame, and it is synchronized.
> Now, someone decides to remove the video stream, and keep only the audio and
> the time code. It makes sense, an audio stream may also have a time code,
> and the time code value changes every 1920 samples (40 ms at 48 kHz). No
> more video, so we put the BlockReferenceTimecode in the BlockGroup
> containing the corresponding audio frame, and it is still synchronized.
> ho ho... Now we have a FLAC frame every 21 ms, and a time code expected to
> change every 40 ms. This weird case may happen because the time code is
> associated to a frame rate (change every 1920/48000 second) and FLAC
> compression has decided to have another frame rate (a frame every 1024/48000
> second). Where to put the BlockReferenceTimecode? Impossible in the
> BlockGroup  containing the audio frame (not the same time stamp) and if it
> is alone there is no "Block" associated with the BlockReferenceTimecode so
> no associated time stamp.

But that's why it's important to know why people use timecodes. Is it
used in audio systems as well ? I think it's used in video production
and maybe movie production as well. I think it's a professional thing
not meant to leave the production system (ie end users never see the
timecode). From that it would be interesting to know the exact
use/needs.

My understanding is that it's used to reference the beginning or parts
of a video segment for editing or archiving later. I think it needs to
be precise to the video frame or maybe to the millisecond so editing
matching sound is also included. To create key points or parts of a
video/audio file, we already have Chapters in Matroska. They can be
tagged as well. And since it's outside of the stream it can be more
convenient for editing as well.

In the Segment we also have the DateUTC which is an absolute time.
With that information we can tell exactly when any of the frame
exactly occurred with a nanosecond precision.

I doubt editing/cutting systems (including splitting in mkvmerge) edit
the DateUTC between created files though.

> corresponding video only later, so we may want to dump the time code alone.
> How to do if we have a BlockReferenceTimecode element instead of content in
> a Bloc element?
>
> containers like MOV, and I prefer to avoid limitations not present in other
> containers.
>
> also: QuickTime time code format (one unsigned integer is a format of time
> code, not the only one), we may specify a lot more different formats later
> (e.g. SMPTE ST-12, SDTI time codes... they are not an integer), how will we
> do if we say that BlockReferenceTimecode is an integer?
>
> opaque, and embedded in Matroska as any other track (i.e. the content of the
> time code is in the block content as any other stream format).
> Link between the time code track and the corresponding video track is done
> by ReferenceTimecodeTrackUID, as in QuickTime.
>
>>
> I don't see 2 needs: from my point of view, there is only one need (a time
> code track), and the discussion is about how to technically implement 2
> technically different cases. This is similar to old containers having no
> "CodecPrivate" elements, and nowadays we have 2 cases: without CodecPrivate
> (each I-frame can be decoded after any seek), and with CodecPrivate (each
> I-frame can be decoded only if the global data is there). for time code,
> this is similar (with a slight difference: no Block data if there is the
> data inside the track header directly)

Given what I said above if there's something not covered it's probably
through some legacy track that will be preversed when remuxing/editing
without knowledge of what's inside. If it has its own track and own
timebase I don't think we need an extra element.

>> [...]
>>
>>
>
> I think that the first agreement we must have is if we consider a time code
> track as any other track standalone with its own Track ID, time stamp... or
> if it must be associated to a (video) track (and all its frames).
>
> ***
>
> I propose to keep the proposal from Dave for the track header, remove the
> BlockReferenceTimecode proposal (content is in a Block or SimpleBlock), and
> add an element in the TrackEntry, generic (not only for QuickTime time code
> format, it would be generic as CodecPrivate):
>
> whole file, all frames are calculated based on this block content. If this
> element is present, time stamp of this block is 0 and no Block element is
> expected in Clusters for this track.
>
> (I have a doubt about the time stamp: can the Timecode of the first Cluster
> be something different than 0?)
>
> Jérôme
>
>

---

From: Dave Rice <[email-masked]>
Date: Sat, 16 Jul 2016 09:37:28 -0400

> On Jul 16, 2016, at 9:24 AM, Steve Lhomme <[email-masked]> wrote:
> 
> used in audio systems as well ?

Not as often, but yes. For instance http://web.archive.org/web/20091229093941/http://tech.ebu.ch/docs/tech/tech3285.pdf defines a value called TimeReference in the bext chunk of a wav file. This stores only the initial timecode value.

> I think it's used in video production

In archival work I think the most common use case is to relate a frame of a digital file to the source frame of a videotape from which the file was created. For example, if in reviewing a file someone sees a head clog at timecode 01:15:13:00, then that person could retrieve the tape that was digitized to make the file and go to the same timecode on the tape to assess if the digitization could be improved. It's possible to do this without the timecode by searching by determining the relationship between file frame and tape frame is a lot of guess work without a timecode.

Another user example. A video tape is accompanied by descriptive metadata that relates notes to timecodes of the video tape. Such as:
01:30:05 - discussion about Matroska
01:45:15 - discussion about EBML
02:00:00 - end of meeting
00:00:00 - timecode reset on tape
00:15:00 - discussion of ffv1

When the tape is digitized to a file. It is challenging to align the notes of the tape with the content of the file without using the timecode values to link everything.

> matching sound is also included.

Re: "video frame or maybe to the millisecond". Usually archivists do not have control over the timecode of the source material that they work with. The timecode may be non-drop-frame or drop-frame, it may loop back to 0 after 23:59:59 or increment about 24 hours, it may allow for negative values, etc but in order to be used for the use cases above, the timecode should be sustained as a lossless copy of the timecode of the source.

> To create key points or parts of a

I agree that chapters could be used to accomplish some use cases of timecode, but not sure how to use chapters to make the tape:frame to file:frame relationship.

> exactly occurred with a nanosecond precision.

This seems to conflict with the definition of DateUTC.

>> be something different than 0?)

Dave

---

From: Steve Lhomme <[email-masked]>
Date: Sat, 16 Jul 2016 15:57:38 +0200

> When the tape is digitized to a file. It is challenging to align the notes of the tape with the content of the file without using the timecode values to link everything.

Ah yes, I forgot about timecode reset I have on some DV tapes.

So we need a track with a lossless way to store this information so
it's not lost/changed/delayed when editing. It would be nice to map
these data into the chapters though to make it easier to find the
information without seeking manually. Maybe in Chapters there should
be a way to tell what timecode it corresponds to. It's "timecode
codec" dependent though. Unless we define a basic way to define a
timecode value. I think they can always translate to text in the form
of HH:MM:SS.sss (sss for milliseconds) or HH:MM:SS.ff (frame count in
that second) ? The last differing part should probably be defined as a
fraction (1/25 or 1/1000). I suppose the other parts can just be added
to that fraction too ?

---

From: Kieran O Leary <[email-masked]>
Date: Sat, 16 Jul 2016 15:01:13 +0100

Hi,

I was about to reply when I saw Dave's email, which contains
everything I was going to write anyhow. Regardless, it might be useful
to hear that the archive I work for, and most other archives have
similar use cases.

> When the tape is digitized to a file. It is challenging to align the notes of the tape with the content of the file without using the timecode values to link everything.

+1 We have the exact same experience.
Also, sometimes filmmakers have old reference clips that have burnt in
timecodes on them. If they get back to us years later and want to
order footage, it makes life a lot easier for us if we can just match
the clips using the timecode. It can also be useful for matching a
digital file to its source asset if this wasn't properly documented.
for example, in my digital file, the first image starts at
10:00:00:00, and we have three tapes in the archive but only one
starts at 10:00:00:00, so we have a better chance of tracing
provenance.

Also some subtitle files like EBU-STL use timecode, rather than the
regular duration,  to match image to subtitles. These subtitles are
.stl sidecars, usually accompanying an MXF( an AS-11 in our case) . I
can't imagine these files migrating to matroska , but it's another use
for timecode regardless.

> I agree that chapters could be used to accomplish some use cases of timecode, but not sure how to use chapters to make the tape:frame to file:frame relationship.

Same.

---

From: Steve Lhomme <[email-masked]>
Date: Sat, 16 Jul 2016 16:08:16 +0200

> starts at 10:00:00:00, so we have a better chance of tracing

The more I think about it. The more I think it could just be a
timecode track with a special CodecID but probably just storing UTF-8
text. This way it can be easily displayed by any player that can
display Matroska text subtitles. With a logical mapping to chapters we
would have a machine readable version for fast seeking to a precise
timecode.

> can't imagine these files migrating to matroska , but it's another use

Are these text based as well ?

---

From: Kieran O Leary <[email-masked]>
Date: Sat, 16 Jul 2016 15:31:27 +0100

> would have a machine readable version for fast seeking to a precise

This timecode track would be a subtitle track, rather than a data
track, as is the case with quicktime tmcd and MXF TC? That sounds like
an interesting workaround.
What i've found is that if I have a v210.mov with a quicktime tmcd track, if I:
1. Transcode to ffv1.mkv, the timecode will appear as metadata

Stream #0:1(eng): Video: ffv1 (FFV1 / 0x31564646), yuv422p10le,
720x576, SAR 16:15 DAR 4:3, 25 fps, 25 tbr, 1k tbn, 1k tbc (default)

    Metadata:

      HANDLER_NAME    : Apple Alias Data Handler

      TIMECODE        : 09:58:00:00

      ENCODER         : Lavc57.46.100 ffv1

      DURATION        : 01:51:01.280000000

2. If I transcode that file back to a quicktime, ffmpeg will create a
timecode track for it regardless. with the same starting timecode. I
don't know enough about the nature of those timecode tracks to know
how much has altered though.

> Are these text based as well ?

Unfortunately not. https://trac.videolan.org/vlc/ticket/3424 I think
there's an example here. EBU-STL is different from the other STL
(Spruce?) which is text based as far as I know.

---

From: Jerome Martinez <[email-masked]>
Date: Sat, 16 Jul 2016 17:24:03 +0200

> would have a machine readable version for fast seeking to a precise

I think we go to far and try to reinvent the wheel.
There are already tons of time code formats, as there are tons of video 
format, for different needs.
Changing of container should not force to change of time code format, as 
changing of container should not force to change of video format.
Looks like we are going on something like https://xkcd.com/927/ and I 
don't like that.

So I propose that we don't try to examine all "real" (we'll miss some of 
them here because we are too few people) cases, and we rely on what was 
already done by other people.
QuickTime timecode format exists, so let's use it as is and we just 
define the way we embed such format as it was done with any other video 
or audio format, without trying to change the "bitstream": in MP4, MXF, 
GXF and so on, time code are several bytes stored in a packet which has 
a PTS, separately of any video frame, why would be smarter than the 
others when we decide to bind a time code packet to a video packet and 
create a new format?
The only difference comapred to an audio or video packet is that the 
time code is sometimes "striped", and I proposed a way for storing that.
Comments about 
https://mailarchive.ietf.org/arch/msg/cellar/hCx_TOhokJjClJop4jGdQmh6Yi0 
would be appreciated because I think it resolves the issues we are 
talking about, in avoiding to lose any possibility offered by QuitkTime 
time code format (for example) and I am sure someday someone will 
complain he can not do something he can do with MOV time codes if we 
don't just map 1 to 1 QuikTime time code features.

Jérôme

---

From: Jerome Martinez <[email-masked]>
Date: Sat, 16 Jul 2016 18:04:37 +0200

> would have a machine readable version for fast seeking to a precise

Another issue I am thinking to.
here, we initially talk of QuickTime time code format, because it is the 
most used by archivists.
But some other people (e.g. broadcasters, but also some other archivists 
who got data from broadcasters for archiving purposes) use other time 
code formats, some of them contain extra data (e.g. SMPTE ST-12 contains 
a time code value + colour framing flag + field bit + user bits). Worse: 
time code is only one item of an ancillary data stream (actually you can 
have several time code values for the same video frame, in addition to 
stuff like AFD https://en.wikipedia.org/wiki/Active_Format_Description , 
CDP subtitles 
http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=7290879 , 608 
captions...).
See https://en.wikipedia.org/wiki/Ancillary_data for more details about 
ancillary data streams.
We can not convert any input about time code (and others) in UTF-8 
string, because there are tons of possibilities and it would be 
transcoding instead of transwrapping, we risk to lose data 
(transformation may be not lossless).

I argue for not trying to create a new format about the content (time 
code, ancillary data...), with the corresponding transcoding (which may 
be not lossless) to code, but I argue for "just" specifying something 
that permits transwrapping in a lossless manner, and this is the role of 
the decoder/player to display correctly the content, as it does for 
video and audio.

---

From: Steve Lhomme <[email-masked]>
Date: Sun, 17 Jul 2016 08:26:30 +0200

2016-07-16 18:04 GMT+02:00 Jerome Martinez <[email-masked]>:
> But some other people (e.g. broadcasters, but also some other archivists who
> got data from broadcasters for archiving purposes) use other time code
> formats, some of them contain extra data (e.g. SMPTE ST-12 contains a time
> code value + colour framing flag + field bit + user bits). Worse: time code

The colour and field bits should be extracted by the muxer and put in
the appropriate track(s) where they will be useful.

There are already cases where we extract data to put in different
places. The way it was originally stored is lost. And it's fine as
long as no information is lost.

There are also various subtitles file formats that are parsed to be
merged in Matroska and end up as S_TEXT/UTF8 because that's all the
information they carried. So for timecodes internal or external it
could be the same. What matters is that the Matroska file is easily
handled and no information is lost.

> is only one item of an ancillary data stream (actually you can have several
> time code values for the same video frame, in addition to stuff like AFD
> https://en.wikipedia.org/wiki/Active_Format_Description , CDP subtitles
> We can not convert any input about time code (and others) in UTF-8 string,
> because there are tons of possibilities and it would be transcoding instead
> of transwrapping, we risk to lose data (transformation may be not lossless).

I agree we must not lose information. But we should not try to put
overly complex new structures just because they were originally stored
in a crap format. So we should identify all the existing formats and
see the ones that require more than a simple text/date value per time.

> I argue for not trying to create a new format about the content (time code,
> ancillary data...), with the corresponding transcoding (which may be not
> lossless) to code, but I argue for "just" specifying something that permits
> transwrapping in a lossless manner, and this is the role of the
> decoder/player to display correctly the content, as it does for video and
> audio.
>
>
>

---

From: Steve Lhomme <[email-masked]>
Date: Sun, 17 Jul 2016 14:07:06 +0200

most used by archivists.
> But some other people (e.g. broadcasters, but also some other archivists
who got data from broadcasters for archiving purposes) use other time code
ancillary data streams.

I had a closer look at this. It can handle multiple audio, frame rates,
etc. This things are better extracted and put in a more common way in
Matroska. It might be the same for timecode information.

The issue of Ancillary data was already mentioned a few times as it's
important to keep it for archival. Either raw or without the data that have
been extracted into proper Matroska format. In both cases it can be a
Complex track in Matroska with a Codec that needs to be defined. Whether
it's raw or partial, I think the timecode should be extracted. It's simple
enough information that can be easily displayed by any Matroska player.

> We can not convert any input about time code (and others) in UTF-8
string, because there are tons of possibilities and it would be transcoding
instead of transwrapping, we risk to lose data (transformation may be not
lossless).
> I argue for not trying to create a new format about the content (time
code, ancillary data...), with the corresponding transcoding (which may be
not lossless) to code, but I argue for "just" specifying something that
permits transwrapping in a lossless manner, and this is the role of the
decoder/player to display correctly the content, as it does for video and
audio.