Subject: [Cellar] Rationale Numbers for timestamps

From: Steve Lhomme <[email-masked]>
Date: Tue, 26 Jul 2016 21:35:15 +0200

As the topic often resurfaces maybe it's time we discuss it in the
context of CELLAR.

There have been talks about ways to do it in the Matroska mailing
before in this thread:
https://lists.matroska.org/pipermail/matroska-devel/2012-October/004294.html
https://lists.matroska.org/pipermail/matroska-devel/2012-October/004295.html
https://lists.matroska.org/pipermail/matroska-devel/2012-October/004296.html
https://lists.matroska.org/pipermail/matroska-devel/2012-October/004299.html

An element to handle it was added (still in specdata.xml but commented)
https://lists.matroska.org/pipermail/matroska-devel/2012-September/004289.html
and later removed
https://lists.matroska.org/pipermail/matroska-devel/2012-December/004317.html

So far there was nothing conclusive but it seems it's not possible to
do it properly in a backward compatible way.

The constraints are simple:
- backward compatible, so the 16 bits timestamp value in the
(Simple)Block must remain unchanged. This value is multiplied by the
scale (global and/or per track) to get the actual timestamp.
- There could be some acceptable rounding between the two coexisting systems.
- The rounding should happen in the current system, the rationale
should always be exact (provided the source is exact)
- A lot of Matroska elements are based on the timestamp scales. So
they may be adjusted accordingly
- We must minimize the elements where the same information is found
twice. Basically only scale elements should be added. Maybe some
elements for the Cluster timestamp might be added as well if
necessary.
- because of the 16 bits range per Cluster differences between a
rationale system and the ns system must be tight.
- we still want Clusters to be 4-5s long to minimize the scope of
errors and buffering before playback. It may also correspond to some
GOP values.
- sample accuracy for audio would be good.

Let the math begin.

-- 
Steve Lhomme
Matroska association Chairman

---

From: Jerome Martinez <[email-masked]>
Date: Wed, 27 Jul 2016 13:13:58 +0900

> - sample accuracy for audio would be good.

Rationale Numbers for timestamps are usually for US style fixed 
framerate, so I think my answer in a previous email still applies.
https://mailarchive.ietf.org/arch/msg/cellar/WQCExNvAZy4goibE3_chnFB1w0c
Please comment about which need would not be fulfilled with this proposal.

Copy/paste:

I don't like the idea to have an incompatible change with such feature,
because this is not a "must have" for 99% of people, and Matroska "v5"
will not be adopted with that.
I think that we can have a workaround for having additional 0.99% happy
without the need of a break in compatibility:
- lot of people want to know the exact frame rate of a fixed frame rate
stream, people muxing from tapes know that the frame rate is fixed
(hardware limitation) so can set this element in the header. putting
frame rate info in the header with numerator and denominator is enough
for doing the difference between 30000/1001 and 29970/1000, and the
current millisecond timestamp is enough (a demuxer aware of the
framerate element can "correct" the timecode from e.g. 0.033s to
1001/30000 second; an old player just play the file with 1 ms
precision). We could add some rules in case a timecode does not fit the
frame rate (e.g. if timecode is 0.031, frame rate info is considered not
applicable)
- people having variable frame rate with very precise timestamp need to
know it before starting the muxing and can use a Timecodescale element
of 1 (so precision of 1 nanosecond, usually enough when there is no
fixed frame rate).
Which real life scenario can not deal with such workaround?

Compared to the concerns in:
https://lists.matroska.org/pipermail/matroska-devel/2012-October/004294.html
It is per track so it resolves most concern
For PCM, the corresponding video framerate is sometimes used and it 
would not be resolved correctly, right (i.e. at 30000/1001 fps, there 
are sometimes 1601 samples and sometimes 1602 samples and it is not 
possible to do the difference with the 16-bit timestamp having the 
millisecond precision)
Note that my proposal is similar to the one in the mail having criticism 
about TimecodeScaleDenominator (proposal of TrackTimeBaseNumerator and 
TrackTimeBaseDenominator)

If there is some issues with this idea, please say it so I stop to bug 
you with this idea.

---

From: Thomas Daede <[email-masked]>
Date: Tue, 26 Jul 2016 21:31:02 -0700

> scale (global and/or per track) to get the actual timestamp.

According to the current spec, there is no per-track TimecodeScale, correct?

- Thomas

---

From: Steve Lhomme <[email-masked]>
Date: Wed, 27 Jul 2016 20:42:15 +0200

> According to the current spec, there is no per-track TimecodeScale, correct?

Correct. It was removed as it was always 1.0.
The original idea what that you could mix PAL video with NTSC audio
(and vice versa) from heterogeneous sources and still be able to keeo
everything in sync regardless of the source. But in practice such
files don't work as intended. You cannot have proper interleaving of
mismatched tracks. So that feature/possibility is gone.

> - Thomas
>
> _______________________________________________
> Cellar mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cellar

---

From: Steve Lhomme <[email-masked]>
Date: Wed, 27 Jul 2016 21:28:35 +0200

2016-07-27 6:13 GMT+02:00 Jerome Martinez <[email-masked]>:
> Rationale Numbers for timestamps are usually for US style fixed framerate,
> so I think my answer in a previous email still applies.
> (hardware limitation) so can set this element in the header.

Let's rewind a decade or more. The timestamps were originally using
floating points on purpose. At the time all sources were analog (film
or video tape). And in the analog world the clock is never perfect.
You cannot expect to have 30000/1001 frames per second and 48000
samples a second for hours without some drift. And I'd rather have the
timing as accurate as the analog source rather than clean it for the
digital world.

In the context of digitizing for archivists I don't know if that's
something that's taken in account. Dave ? For example how is that
solved for movies from the early days where a guy would have to turn a
handle to advance the film. It's definitely not a very precise/clean
clock.

> putting

In video the issue is not too big. But if we want sample precision for
audio, the fraction is a lot bigger (up to 192000 fps with modern pro
hardware). And with a timestamp scale shared among tracks that could
be problematic. If we want sample precision for audio and video we
could end up we'd lose a big range of the 16 bits available per
Cluster. Each of the 65536 Block timestamp would be 1/192,000s
(0.0052ms) giving a max Cluster duration of 0.341333s (65536/192000).
If we want to precision for a 1001/30000 track and a 1/192000 track we
would need a fraction of 1001/(30000*192000) (0.000173ms) resulting in
a Cluster max duration of 0.011s.

If the muxer knows that a track will always pack samples by an amount
X (1152 in MP3 for example) we can regain some duration. But that's
not always the case. Some codecs like Vorbis pack only a small amount
of samples per frame and the amount can vary between frames (In Vorbis
I, legal frame sizes are powers of two from 64 to 8192 samples). The
worst case scenario (I know of) of Vorbis at 192 KHz, frame precision
of the timestamp means we can only have a Cluster duration of 21.84s
(65536*64/192000). Mixed with NTSC video that's
65536*64*1001/30000*192000 = 0.72s max duration for sample precision
per frame (not sample but it's close enough). That's still 21 video
frames in that Cluster. Not too bad.

> of 1 (so precision of 1 nanosecond, usually enough when there is no

I can think of 3 different variable frame rate use cases:

- mixing video (25/29fps) and film sources (24fps) into one final video
- a camera dropping some frames or only storing images when they
change. In this case there's likely a stable main clock like in
constant frame rate.
- a camera that would work like the eye. When there's
adrenaline/danger you capture images faster. In this case there's no
real clock. The timestamps can happen at anytime.

> Which real life scenario can not deal with such workaround?

The question is: why do we want rationale number if we're going to
allow some form of rounding between the tracks that match the
timestamp scale precisely and the one that don't. IMO if there's
rounding let's just keep what we have.

> For PCM, the corresponding video framerate is sometimes used and it would
> not be resolved correctly, right (i.e. at 30000/1001 fps, there are
> sometimes 1601 samples and sometimes 1602 samples and it is not possible to
> do the difference with the 16-bit timestamp having the millisecond
> precision)
> If there is some issues with this idea, please say it so I stop to bug you
> with this idea.
>

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Wed, 27 Jul 2016 12:57:12 -0700

> I, legal frame sizes are powers of two from 64 to 8192 samples). The

There's interaction with CodecDelay here. If you want that to have an 
exact rational sample time, also, it won't be able to be a multiple of 
the block size.

---

From: Kieran O Leary <[email-masked]>
Date: Wed, 27 Jul 2016 21:10:21 +0100

Hi Steve,

> handle to advance the film. It's definitely not a very precise/clean

The preservation format for film is usually image sequences, so
framerate in this instance isn't a massive issue. Things get more
complicated when making an exhibition copy from that image sequence.
I've generally seen tape/digital copies just running too fast at
24fps, or there might be some frame duplication involved in order to
give the impression of 16/18fps in a 24/25fps copy. I haven't heard of
anyone using true variable frame rate for digitial exhibitions of
silent hand cranked films, but it's an interesting topic.
Unfortunately, Digital Cinema Packages do not allow variable frame
rates or frame rates < 24fps so this does not seem to be on many
agendas.

Best,

Kieran,

---

From: Steve Lhomme <[email-masked]>
Date: Sun, 31 Jul 2016 08:41:48 +0200

2016-07-27 21:57 GMT+02:00 Timothy B. Terriberry <[email-masked]>:
> There's interaction with CodecDelay here. If you want that to have an exact
> rational sample time, also, it won't be able to be a multiple of the block
> size.

There's a chance that it's a multiple of the internal block size used
by the codec. But it's not 100% guaranteed.

In any case the rationale timestamp is going to be optional. So in
cases where it's not possible to use it, it won't. It will be the job
of the muxer to decide if, based on the sources, it can be used or
not.

---

From: Steve Lhomme <[email-masked]>
Date: Sun, 31 Jul 2016 09:07:36 +0200

> of the muxer to decide if, based on the sources, it can be used or

Apart from the numerator/denominator needed to express more precisely
the TimecodScale we're going to need additional elements for
timestamps not scaled. For example the timestamp start/stop in
Chapters. Since we want these to be sample accurate they should be
based on the timestap rationale scale. And to do this sample precision
we must not have the "samples per frame" in that timestamp scale.

In the Vorbis+NTSC example that means the rational would be something
like 1001 / (30,000 * 192,000) (paired with a TimecodeScale of 17).

We could add an element to each TrackEntry to define how many samples
are packed per frame (if known). If present that number should be used
to compute the rational timestamp of a Block:
frame timestamp = (Cluster timestamp + Block timestamp * Sample per
Block) * rational timestamp scale

Since it has an impact on the timestamp value put on each Block it
would be backward incompatible. So we may add this refinement in
Matroska v5. The rational timecode scale could probably go in v4
though.

---

From: Moritz Bunkus <[email-masked]>
Date: Sun, 31 Jul 2016 10:00:12 +0200

Hey,

> For example the timestamp start/stop in Chapters. Since we want these
> to be sample accurate…

Chapters are something that I consider to be agnostic to sample
precision. Which track should they match up with? The video track? One
of the audio tracks? All of them? Next it's highly unlikely that a
chapter start falls exactly onto a point in time when both a video frame
as well as a frame of all audio tracks start. Let's just leave chapters
keep their ns precision.

> are packed per frame (if known).

That number varies from block to block in certain codecs,
e.g. Vorbis. There are audio blocks with e.g. 1024 samples and ones with
4096.

Kind regards,
mosu

---

From: Steve Lhomme <[email-masked]>
Date: Sun, 31 Jul 2016 17:01:59 +0200

> as well as a frame of all audio tracks start. Let's just leave chapters

I agree it's per track and with NTSC timing there's a chance you won't
ever fall on audio matching video (except 0 of course). That does not
mean authoring tools may not want to put their chapters exactly on
video frame boundaries. Also if you only have an audio track, let's
say an album in FLAC. You may want to put your CD track markers at the
exact sample. Which may well be inside a FLAC frame. But on
playback/decoding the exact frame can be recovered. Although to be
fair the ns precision, even for a 192 kHz track (~5ns per sample)
won't get much mismatching. So yeah, let's keep that simple.

> e.g. Vorbis. There are audio blocks with e.g. 1024 samples and ones with

If known also means if it's correct. If it can't be told for sure then
it shouldn't be used.

But for this element to work that means all tracks should have it and
default to 1. Because it would be used for timestamps of all frames:
frame timestamp = (Cluster timestamp + Block timestamp * Sample per
Block) * rational timestamp scale

It's odd to put 1 when you don't know the value though. So it should
probably not relate to samples in the name. More like
TrackTimestampScale. It's just a trick to extend the range of Block
timestamps.

---

From: Moritz Bunkus <[email-masked]>
Date: Sun, 31 Jul 2016 17:17:56 +0200

Hey,

> But on playback/decoding the exact frame can be recovered. Although to
> be fair the ns precision, even for a 192 kHz track (~5ns per sample)

You're off by 1000 ;) 192 kHz = 192.000 samples/second = 192
samples/millisecond = 192 samples/1.000.000 nanoseconds ⇒ 5.208
nanoseconds/samples. Not 5ns but 5µs.

With nanosecond precision you can always place a chapter marker
unambiguously on any frame (no matter if it's audio or video, even with
irrational time bases.

Kind regards,
mosu

---

From: Steve Lhomme <[email-masked]>
Date: Sun, 31 Jul 2016 17:29:40 +0200

> nanoseconds/samples. Not 5ns but 5µs.

Oops. I blame it on my calculator that uses the e-x notation for very
small Real numbers.