Subject: [Cellar] ReferenceTimecode tracks in Matroska proposal

From: Dave Rice <[email-masked]>
Date: Sun, 10 Jan 2016 17:57:21 -0500

Hi all,

Here is an early draft of Elements for managing ReferenceTimecode tracks. Matroska uses the term 'Timecode' to reference presentation timestamps, whereas other formats (QuickTime, MXF) use 'Timecode' to reference a time reference that is added to the timeline which is inpendent of the presentation (i.e. a QuickTime file with a timecode track that starts at one hour aka 1:00:00;00 would not cause a one hour delay before playback starts). In the context of Matroska I am using ReferenceTimecode (for now) to refer to QuickTime/MXF style timecode tracks.

TrackType:
Add to the TrackType definition to list "0x13: timecode,"

CodecID:
Add the following timecode Codec IDs.
Codec ID        | Name               | Description
T_QUICKTIME     | QuickTime Timecode | Timecode data as stored in QuickTime, e.q. tmcd. The CodecPrivate contains all additional data that is stored in the 'tcmd' (timecode) atom in the QuickTime file.
Others? LTC? VITC?

Proposed elements under TrackEntry:

Element Name: ReferenceTimecode
Level:        3
ID:           [EF]
Mandatory:    -
Multiple:     -
Default:      -
Type:         m
Description:  ReferenceTimecode settings

Element Name: ReferenceTimecodeForm
Level:        4
ID:           [EF][A1]
Mandatory:    mand.
Multiple:     -
Range:        0-2
Default:      0
Type:         u
Description:  Indicates the form of expression used to present the timecode (using HH for hours, MM for minutes, SS for seconds, mmm for milliseconds, ff for frames).
              0: HH:MM:SS.mmm
              1: HH:MM:SS:ff (non-drop frame)
              2: HH:MM:SS;ff (drop frame)

Element Name: ReferenceTimecode24HourMax
Level:        4
ID:           [EF][A2]
Mandatory:    mand.
Multiple:     -
Range:        0-1
Default:      0
Type:         u
Description:  Indicates if the timecode should revert to a zero-value at the time reference of 24 hours. Set to 1 if the ReferenceTimecode should use a maximum value of 24 hours and wrap values back to zero hours at that point.

For storage of ReferenceTimecode values in Blocks, I suggest storing a signed integer (ReferenceTimecode values may be negative). The stored signed integer would be the frame number (players supporting ReferenceTimecode should convert the frame number accordingly to a drop-frame or non-drop-frame display). So if ReferenceTimecodeForm=1, the Block stores 0x00BC, and the frame duration is NTSC, then the displayed value would be 00:00:06:08. The next value 00:00:06:09, etc.

I propose that one block per ReferenceTimecode not be required for simple incrementing ReferenceTimecode, but that the ReferenceTimecode value of the next video frame be considered to be either 1 greater than the previous value (if ReferenceTimecodeForm is 1 or 2) or increased by the duration of the corresponding video frame.

If the ReferenceTimecode is non-continuous then a new ReferenceTimecode value in a new Block with a new Timecode (Matroska meaning of Timecode) could be inserted at the point of the discontinuity.

Comments?
Dave Rice

---

From: Kieran O Leary <[email-masked]>
Date: Mon, 11 Jan 2016 01:07:55 +0000

> Hi all,

Hi,
I don't have much to add except that I'm delighted to see this addressed
and it looks like a great addition to the matroska format. Some comments
further down.

>
> Here is an early draft of Elements for managing ReferenceTimecode tracks.
> Matroska uses the term 'Timecode' to reference presentation timestamps,
> whereas other formats (QuickTime, MXF) use 'Timecode' to reference a time
> reference that is added to the timeline which is inpendent of the
> presentation (i.e. a QuickTime file with a timecode track that starts at
> one hour aka 1:00:00;00 would not cause a one hour delay before playback
> starts). In the context of Matroska I am using ReferenceTimecode (for now)
> to refer to QuickTime/MXF style timecode tracks.
>
> TrackType:
> T_QUICKTIME     | QuickTime Timecode | Timecode data as stored in
> QuickTime, e.q. tmcd. The CodecPrivate contains all additional data that is
> stored in the 'tcmd' (timecode) atom in the QuickTime file.
> Others? LTC? VITC?
>

I have mostly encountered QuickTime tmcd info thus far. I haven't
reformatted any mxfs as mkv yet, but tmcd info within matroska would be
fantastic from my perspective.

> Description:  Indicates the form of expression used to present the
> timecode (using HH for hours, MM for minutes, SS for seconds, mmm for
> milliseconds, ff for frames).
>               0: HH:MM:SS.mmm

I don't know if this flag is explicitly mentioned in a QuickTime
container,but it seems to me to be a great idea to flag this message info.

> Description:  Indicates if the timecode should revert to a zero-value at
> the time reference of 24 hours. Set to 1 if the ReferenceTimecode should
> use a maximum value of 24 hours and wrap values back to zero hours at that
> point.
>
> For storage of ReferenceTimecode values in Blocks, I suggest storing a
> signed integer (ReferenceTimecode values may be negative). The stored
> signed integer would be the frame number (players supporting
> ReferenceTimecode should convert the frame number accordingly to a
> drop-frame or non-drop-frame display). So if ReferenceTimecodeForm=1, the
> Block stores 0x00BC, and the frame duration is NTSC, then the displayed
> value would be 00:00:06:08. The next value 00:00:06:09, etc.
>
> I propose that one block per ReferenceTimecode not be required for simple
> incrementing ReferenceTimecode, but that the ReferenceTimecode value of the
> next video frame be considered to be either 1 greater than the previous
> value (if ReferenceTimecodeForm is 1 or 2) or increased by the duration of
> the corresponding video frame.
>
> If the ReferenceTimecode is non-continuous then a new ReferenceTimecode
> value in a new Block with a new Timecode (Matroska meaning of Timecode)
> could be inserted at the point of the discontinuity.
>
> Comments?
> Dave Rice
>
>
Best,

Kieran.

---

From: Reto Kromer <[email-masked]>
Date: Mon, 11 Jan 2016 11:57:00 +0100

>container,but it seems to me to be a great idea to flag this message info.

As long as I know, in a purist world

   : and ; should designate the field 2
   . and , should not

(Still as long as I know, SMPTE has defined:

   HH:MM:SS:FF:XX

Sadly I don't remember exactly the letters they use for XX which 
designate parts of the frame. However I never saw this in the 
real world.)

Sorry for adding confusion! Reto

AV Preservation by reto.ch
chemin du Suchet 5 | 1024 Ecublens | Switzerland
Web: http://reto.ch | Twitter: @retoch

---

From: Jerome Martinez <[email-masked]>
Date: Mon, 11 Jan 2016 12:36:26 +0100

>                2: HH:MM:SS;ff (drop frame)

I don't see the purpose of such flag.
a time code is not a time stamp, so no "HH:MM:SS.mmm" in the stream 
directly. From my point of view, it is the role of the player to propose 
either HH:MM:SS:ff or HH:MM:SS.mmm display formats for the same content. 
the container should not contain such detail.
non-drop frame vs drop frame: it is already in the private data (not 
only for T_QUICKTIME, such flag is in all timecode formats I have ever 
seen). Is there a need to repeat it?
If you want such piece of information for being able to compute the 
timecode without the private data, you also need the frame rate (as an 
integer).

> Description:  Indicates if the timecode should revert to a zero-value at the time reference of 24 hours. Set to 1 if the ReferenceTimecode should use a maximum value of 24 hours and wrap values bac…

it is already in the private data. Is there a need to repeat it?

>
> For storage of ReferenceTimecode values in Blocks, I suggest storing a signed integer (ReferenceTimecode values may be negative). The stored signed integer would be the frame number

which format of the integer? EBML? if "32-bit signed integer", the issue 
is with (theoretical) huge values: if QuickTime settings are with 
NegativeTimes=0, you can not store the big numbers.

Actually, I don't like the idea to "standardize" the storage, because we 
define the CodecID as "T_QUICKTIME", so we should just raw copy the 
stream from a .mov file, without modification (and without specifying 
it, as we don't specify the AVC stream for video. Reference is QuickTime 
specification).
And with "T_QUICKTIME", we don't define our own format, so adding 
"increased by the duration of the corresponding video frame" in the 
definition does not make any sense: either we define a new timecode 
format, either we have the goal to use QuickTime format, but mixing 
"T_QUICKTIME" and our own definition will create a lot of confusion.

>   (players supporting ReferenceTimecode should convert the frame number accordingly to a drop-frame or non-drop-frame display). So if ReferenceTimecodeForm=1, the Block stores 0x00BC, and the frame duration is NTSC, then the displayed value would be 00:00:06:08. The next value 00:00:06:09, etc.
>

Actually, technically and except when the timecode is "burned" in the 
video frame, the timecode stream does not depend of any "video frame" or 
"video stream". You could have (in theory, totally useless in the 
reality, I know) a 24 fps timecode stream with a 25 fps video stream 
(actually, in .mov file, the time code stream has a frame rate defined 
by the time scale and duration of a frame, and also a "An 8-bit integer 
that contains the number of frames per second for the timecode format. 
If the time is a counter, this is the number of frames for each counter 
tick." so you can have a 24 fps time code at 30 fps beside a video 
stream at 25 fps)
So linking this way timecode stream and video stream is risky. linking 
duration ("time stamp") and timecode is also risky.

I propose to not try to "adapt" QuickTime time code, just dump it as it 
is, without trying to "limit" the possibilities offered by QuickTime: so 
a timecode stream has a value (32-bit signed or unsigned integer) with a 
timestamp (Matroska "timecode") and a duration (useful if we want to 
know if there is only one value or several values).

Example of metadata from a QuickTime file
- time scale from container (mdhd): 30000
- total duration from container (mdhd) (in time scale units): 335335
- sample duration from container (stts): 335335
- tmcd time scale: 30000
- tmcd frame duration (in time scale units): 1001
- tmcd number of frames (time code "frame rate"): 30
- tmcd drop frame flag: no
(note: I don't know the behavior of QuickTime player in case of tmcd 
time scale != mdhd time scale)

so we need to keep such data in Matroska, e.g.:
- CodecPrivate: tcmd content
- no need of extra setting in the track header (drop frame flag, 24H 
flag are in tmcd)
- Matroska TimecodeScale: 30000
- Matroska Timecode of the unique frame: 0
- Matroska BlockDuration of the unique frame: 335335
- Block content: the 4 bytes from QuickTime timecode stream.

> If the ReferenceTimecode is non-continuous then a new ReferenceTimecode value in a new Block with a new Timecode (Matroska meaning of Timecode) could be inserted at the point of the discontinuity.

It is not so easy to insert a new block and being sure it will be 
correctly displayed, due to interlacement (when can we consider that the 
the next block is not present?) without BlockDuration.

Jérôme

---

From: Michael Bradshaw <[email-masked]>
Date: Mon, 11 Jan 2016 10:12:36 -0800

> to refer to QuickTime/MXF style timecode tracks.

Will the current timecode elements (TimecodeScale, Timecode,
ReferenceTimeCode, TrackTimecodeScale, and block's Timecode) be renamed to
"timestamp" or "presentation timestamp" (or something like that)? I hope
so. I know this is tangential to your post, but I want to get a "big
picture" in my mind of where this is going to end up. I'm also slightly
concerned that using the name "ReferenceTimecode"  will be a bit confusing
to developers, given the old ReferenceTimeCode (DivX trick track
extensions) element.

--Michael

---

From: Jerome Martinez <[email-masked]>
Date: Tue, 12 Jan 2016 20:45:38 +0100

Le 10/01/2016 23:57, Dave Rice a écrit :
> [...]

All other time codes I ever met in the digital world (I exclude time 
codes "burned" in the video frame):
- MXF time code track: a single value, I think it could be converted to 
T_QUICKTIME without losing information
- CPL (from IMF) time code track: a single value, I think it could be 
converted to T_QUICKTIME without losing information
- WMV time code track: we need to define a new codec identifier
- SMPTE/SDTI time code track: we need to define a new codec identifier
- SMPTE/System scheme 1 time code: we need to define a new codec identifier
- LTC, VITC, ATC_LTC, ATC_VITC1, ATC_VITC2 in SMPTE ST 291 (in MXF, GXF, 
LXF...): we need to define a new codec identifier for SMPTE ST 291 
ancillary data stream (which can transport captions/subtitles, time 
codes, bar data, acquisition metadata, audio metadata, program 
description... "compound" stream kind)

I don't think we should try to define new codec identifiers for all time 
code formats but T_QUICKTIME for the moment, until someone really needs it.
So I propose to start with T_QUICKTIME only.

Jérôme

---

From: Steve Lhomme <[email-masked]>
Date: Thu, 14 Jan 2016 13:30:42 +0100

2016-01-11 19:12 GMT+01:00 Michael Bradshaw <[email-masked]>:
>> presentation (i.e. a QuickTime file with a timecode track that starts at one
>> hour aka 1:00:00;00 would not cause a one hour delay before playback
> "timestamp" or "presentation timestamp" (or something like that)? I hope so.

Definitely, yes.

> I know this is tangential to your post, but I want to get a "big picture" in
> my mind of where this is going to end up. I'm also slightly concerned that
> using the name "ReferenceTimecode"  will be a bit confusing to developers,
> given the old ReferenceTimeCode (DivX trick track extensions) element.
>
> --Michael
>
> _______________________________________________
> Cellar mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cellar
>

-- 
Steve Lhomme
Matroska association Chairman

---

From: Steve Lhomme <[email-masked]>
Date: Thu, 14 Jan 2016 13:37:32 +0100

>> Proposed elements under TrackEntry:

Anything that's specific to a track type and not audio or video should
be under its own "umbrella" element, similar to Video [E0] and Audio
[E1].

> either HH:MM:SS:ff or HH:MM:SS.mmm display formats for the same content. the
> container should not contain such detail.
> non-drop frame vs drop frame: it is already in the private data (not only
> for T_QUICKTIME, such flag is in all timecode formats I have ever seen). Is
> there a need to repeat it?
> If you want such piece of information for being able to compute the timecode
> without the private data, you also need the frame rate (as an integer).
>
>>
>> the time reference of 24 hours. Set to 1 if the ReferenceTimecode should use
>> a maximum value of 24 hours and wrap values back to zero hours at that
>> point.
>
>
>> signed integer (ReferenceTimecode values may be negative). The stored signed
>> integer would be the frame number
>
>
> which format of the integer? EBML? if "32-bit signed integer", the issue is
> with (theoretical) huge values: if QuickTime settings are with
> define the CodecID as "T_QUICKTIME", so we should just raw copy the stream
> from a .mov file, without modification (and without specifying it, as we
> don't specify the AVC stream for video. Reference is QuickTime
> specification).
> And with "T_QUICKTIME", we don't define our own format, so adding "increased
> by the duration of the corresponding video frame" in the definition does not
> make any sense: either we define a new timecode format, either we have the
> goal to use QuickTime format, but mixing "T_QUICKTIME" and our own
> definition will create a lot of confusion.

I agree, the storage in Blocks is codec specific and should be defined
separately.

Also if the goal is to "tag" frames with a timecode, putting an extra
element in BlockGroup seem more logicial.

>>   (players supporting ReferenceTimecode should convert the frame number
>> accordingly to a drop-frame or non-drop-frame display). So if
>> ReferenceTimecodeForm=1, the Block stores 0x00BC, and the frame duration is
>> NTSC, then the displayed value would be 00:00:06:08. The next value
>> 00:00:06:09, etc.
>>
> Actually, technically and except when the timecode is "burned" in the video
> frame, the timecode stream does not depend of any "video frame" or "video
> stream". You could have (in theory, totally useless in the reality, I know)
> a 24 fps timecode stream with a 25 fps video stream (actually, in .mov file,
> the time code stream has a frame rate defined by the time scale and duration
> of a frame, and also a "An 8-bit integer that contains the number of frames
> per second for the timecode format. If the time is a counter, this is the
> number of frames for each counter tick." so you can have a 24 fps time code
> at 30 fps beside a video stream at 25 fps)
> I propose to not try to "adapt" QuickTime time code, just dump it as it is,
> without trying to "limit" the possibilities offered by QuickTime: so a
> timecode stream has a value (32-bit signed or unsigned integer) with a
> timestamp (Matroska "timecode") and a duration (useful if we want to know if
> there is only one value or several values).
>
> (note: I don't know the behavior of QuickTime player in case of tmcd time
> scale != mdhd time scale)
>
> - no need of extra setting in the track header (drop frame flag, 24H flag
> are in tmcd)
> It is not so easy to insert a new block and being sure it will be correctly
> displayed, due to interlacement (when can we consider that the the next
> block is not present?) without BlockDuration.
>
> Jérôme
>
>

---

From: Moritz Bunkus <[email-masked]>
Date: Thu, 14 Jan 2016 13:38:49 +0100

Hey,

> > "timestamp" or "presentation timestamp" (or something like that)? I hope so.

Please keep in mind that existing Matroska libraries (not just
libMatroska) name their functions/classes after the elements in the
specs. Renaming the elements in the specs requires either that the the
classes/functions in the libraries be renamed as well (breaking
compilation with each and every program out there) or we'll end up with
a severe discrepancy between the naming in the libraries and the specs
(especially if new elements will be introduced after the renaming with
names that were used before the naming!).

I don't like either outcome.

Kind regards,
mosu

---

From: Steve Lhomme <[email-masked]>
Date: Thu, 14 Jan 2016 13:42:19 +0100

> names that were used before the naming!).

The spec file already has a way to override the name of the C++ class
when it doesn't match exactly the one displayed in the specs (usually
shorter or just to keep backward compatibility). We just need to use
this for such elements.
Maybe also add some typedefs to have both old and new names.

---

From: Moritz Bunkus <[email-masked]>
Date: Thu, 14 Jan 2016 13:50:00 +0100

Hey,

> Maybe also add some typedefs to have both old and new names.

What I meant:

- Rename e.g. ClusterTimecode to ClusterTimestamp in the specs
- Leave the class name the same in the library (ClusterTimecode)
- Then introduce elements in the spec dealing with actual timecodes
  (e.g. ReferenceTimecode) which would translate into a class name
  ReferenceTimecode

Now you have the situation that a developer working with such a library
would have several classes with Timecode in their name and some of
those would deal with timestamps and others with timecodes. Highly
confusing to anyone not familiar with our current discussion.

Also: functions like KaxBlockGroup::GlobalTimecode() – those return
timestamps. If not returned it would be very, very confusing for new
users to determine what they actually return, timestamps or timecodes.

Kind regards,
mosu

---

From: Jerome Martinez <[email-masked]>
Date: Thu, 14 Jan 2016 14:52:32 +0100

> element in BlockGroup seem more logicial.

There are several points of view, depending of the background of people.
End user usually considers that a time code is tied to a video frame. 
Note that depending of the end user, he may be interested in the 
QuickTime time code, the VITC time code, the LTC time code, the SDTI 
time code... And these time codes may be different (in addition to 
cryptic names :) ).
Technical person sees that QuickTime time code has its own track number, 
its own frame rate (usually same as the tied video track, but it is not 
mandatory by any spec), its own count of frames, as any video or audio 
track, so it is an independent track. QuickTime file format permits a 
reference (saying that this time code track is tied to video track with 
ID x, looks like that references are not supported by Matroska), not always.

Putting an extra element in BlockGroup from a QuickTime time code track 
means that we do a "transcode" (with potential loss of data, as any 
transcode), not a "transmux" (raw copy of a track). It is possible, with 
pros (e.g. just an element) and cons (less possibilities, sometimes 
input can not be mapped to a specific frame).

Jérôme

---

From: Steve Lhomme <[email-masked]>
Date: Fri, 15 Jan 2016 11:45:34 +0100

>   (e.g. ReferenceTimecode) which would translate into a class name

I see. Such confusion can be easily avoided. We just need to make sure
to provide backward compaibility.

> confusing to anyone not familiar with our current discussion.

We could include the text from the specdata as a comment for each
class name. That should at least avoid some confustion.

> users to determine what they actually return, timestamps or timecodes.

These aren't aren't generated IIRC. So we can edit and document that
manually. Providing both might be a good idea for a future transition.