Subject: [Cellar] Overconstrained timecodes

From: Thomas Daede <[email-masked]>
Date: Sun, 17 Jul 2016 04:59:20 -0700

(note, this is not related to the ReferenceTimecode discussion)

Here are the various non-deprecated elements that can contain timecodes
or are specified in timecode units in the Matroska spec:

- Duration (of segment)
- Cluster
- Block (relative to Cluster)
- BlockDuration
- DefaultDuration
- DefaultDecodedFieldDuration

This mix allows one to specify a file with many possible conflicts.
These include:

- Sum of durations of clusters being longer or shorter than a segment
- Clusters can overlap in time or have gaps
- Blocks can begin before a Cluster begins and after the cluster ends,
or partially extend beyond the end of the Cluster
- BlockDuration may not match the sum of the
DefaultDecodedFieldDurations contained within.
- Internal codec specific data might not match with all of the above.

For these cases, two things need to be defined: 1) whether a muxer can
produce such files, and 2) what a demuxer should do when it encounters
such files.

Unfortunately, unlike other containers, the TimecodeScale in Matroska is
defined only as a single integer duration rather than a numerator and
denominator. This means that for almost all files, the exact
TimecodeScale to place each frame exactly is not possible to represent.
Section 257 provides guidelines on how to represent this, but only for
timecode values of Blocks or Clusters, not for any durations, which
makes creating any of the above situations very easy to do by accident.

- Thomas

---

From: Steve Lhomme <[email-masked]>
Date: Sun, 17 Jul 2016 14:39:13 +0200

> - Sum of durations of clusters being longer or shorter than a segment

A Cluster doesn't have a duration. If there's no DefaultDuration for a
track and no BlockDuration there's no way to tell the duration until
another Block for that track comes. One way to set a duration on a Cluster
(especially the last one of a Segment) is to set a BlockDuration on the
last Block of each Track.

> - Clusters can overlap in time or have gaps

Gaps in a track are fine. You may have dropped frames during a capture.
There's no way to tell there's a discontinuity though.

Data from Clusters may overlap between tracks but in one track it's not
supposed to happen. Each frame (reassembled in display order) replaces the
previous one.

> - Blocks can begin before a Cluster begins and after the cluster ends,

There's no Cluster end. Only end of tracks in that Cluster.

Because the frames are stored in coding order, it's possible that, in a
track, the next Block has an older timestamp and is found in the next
Cluster. I suppose that's what you mean. It's the job of the codec to
reorder the frames after decoding.

If BlockGroup is used it's possible that frame reference what other
frame/Block they need to be decoded. In that case it's possible to edit the
file in a safe way. When using SimpleBlock you lose that information and
can only safely edit at keyframe boundaries.

> - Internal codec specific data might not match with all of the above.

Correct. In general we should specify which take precedence: the container
or the codec. In most cases it should be the container. The same applies to
cropping.

> produce such files,

There's no way to control all muxers are correct and bug free. So one can
always assume such files will exist.

> and 2) what a demuxer should do when it encounters
> such files.

For reordering the codec should take care of it if it's involved in the
processing chain. Otherwise ReferenceBlock and otherwise it's best effort.

> timecode values of Blocks or Clusters, not for any durations, which

I suppose you're referring to the Segment Duration which is in floating
point. It's an informational value that should not be used for precise
operations (mostly only for displaying to the user and approximate a
playlist duration). That's the reason it's a float, so no one tries to use
it as something perfect and accurate.

> makes. creating any of the above situations very easy to do by accident.

> - Thomas
>
> _______________________________________________
> Cellar mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cellar

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Sun, 17 Jul 2016 12:11:02 -0700

> There's no way to tell there's a discontinuity though.

This might be true for video, but it's certainly not true for audio. Yet 
we have code in Firefox to insert silence padding between frames for 
Vorbis audio in WebM to correct for these gaps 
(<https://bugzilla.mozilla.org/show_bug.cgi?id=576539>, see particularly 
the third bullet in comment #6), despite the fact that I can guarantee 
you the original Vorbis encoding did not have any (it's an overlapped 
block codec!). This requires ad-hoc thresholds that are completely a 
function of implementers not being able to compute accurate, or even 
consistent timestamps.

---

From: Thomas Daede <[email-masked]>
Date: Sun, 17 Jul 2016 12:27:20 -0700

On 07/17/2016 05:39 AM, Steve Lhomme wrote:
> Le 17 juil. 2016 13:59, "Thomas Daede" <[email-masked]
> <mailto:[email-masked]>> a écrit :
>>
> another Block for that track comes. One way to set a duration on a
> Cluster (especially the last one of a Segment) is to set a BlockDuration
> on the last Block of each Track.

The spec seems to currently allow BlockDuration to exist on every block,
not just the last block, so I guess is problem is between blocks, not
clusters (though still, the last block of the last cluster could extend
beyond the segment with an incorrect BlockDuration).

What happens if the last block of a cluster's BlockDuration isn't
specified, but another segment never arrives?

> supposed to happen. Each frame (reassembled in display order) replaces
> the previous one.

OK. The draft should disallow this, and probably also specify what
happens if this is seen in a file (entire Cluster rejected, frames
dropped, etc).

> frame/Block they need to be decoded. In that case it's possible to edit
> the file in a safe way. When using SimpleBlock you lose that information
> and can only safely edit at keyframe boundaries.

OK.

> Correct. In general we should specify which take precedence: the
> container or the codec. In most cases it should be the container. The
> same applies to cropping.
> 
> There's no way to control all muxers are correct and bug free. So one
> can always assume such files will exist.

You're right, if the demuxer behavior is fully specified in all
circumstances than it isn't necessary to constrain muxer behavior. I
guess it depends on where you want to draw the line of an "invalid
Matroska file".

> playlist duration). That's the reason it's a float, so no one tries to
> use it as something perfect and accurate.

OK, that should be specified. Though I was not referring to the
precision of TimecodeScale. A muxer is expected to maintain a high
accuracy clock, pick a sufficiently accurate TimecodeScale, and then
round that internal clock to produce timecodes written into the file.
However, if it does high precision duration calculations and then rounds
those into BlockDuration values, then the rounding might go a different
way and the BlockDuration will create a gap or overlap of 1
TimecodeScale unit.

- Thomas

---

From: Steve Lhomme <[email-masked]>
Date: Mon, 18 Jul 2016 08:36:21 +0200

> This might be true for video, but it's certainly not true for audio. Yet
we have code in Firefox to insert silence padding between frames for Vorbis
audio in WebM to correct for these gaps (<
https://bugzilla.mozilla.org/show_bug.cgi?id=576539>, see particularly the
third bullet in comment #6), despite the fact that I can guarantee you the
original Vorbis encoding did not have any (it's an overlapped block
codec!). This requires ad-hoc thresholds that are completely a function of
implementers not being able to compute accurate, or even consistent
timestamps.

That's an implementation detail. It could be nice to have a flag for
discontinuity but if a part the file is damaged you may not have that
luxury when trying to play the file. So the scenario of keeping playback in
sync when there are unexpected discontinuity has to be handled.

I don't know if we should decide how it should be handled. For example in
VLC the audio is stretched when the end of a packet doesn't match the
beginning of the next packet. If it's too long there's silence instead.

IMO having the beginning of the timestamp and the codec knowing the
internal duration, the framework has enough information to decide how to
handle this case best for them.

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Tue, 19 Jul 2016 02:29:03 -0700

> That's an implementation detail. It could be nice to have a flag for

I think you're conflating two issues, probably because I'm explaining 
them badly. Allowing discontinuities in what is normally continuous data 
like audio, with or without a flag, is one issue, and the argument 
around having to support damaged files is reasonable (though if there's 
damage, that's detectable, at least until you remux the damaged sections).

But the point of the comment I highlighted is a different one. And 
that's the practical consequences of having timestamps that cannot 
exactly represent the sample times for most content, including even just 
24 or 30 fps video (to say nothing of NTSC) or 44.1 kHz audio.

As a result, we see "gaps" or overlaps of a few samples quite often in 
real files. It's clear when you have a 4 second gap, that's a 
discontinuity, and it's likely when it's a sample or two you don't. But 
it's not clear where the threshold is between those two cases, and the 
30 ms chosen in that bug was basically arbitrary. It also doesn't handle 
the case where these timestamps are consistently wrong in one direction. 
Even if the difference between the container's reported duration and the 
codec's is small for every frame, the cumulative difference could 
eventually become very large.

So I don't think you can wave this away as an "implementation detail".

> I don't know if we should decide how it should be handled. For example
> in VLC the audio is stretched when the end of a packet doesn't match the

Resampling to correct for one- or two-sample gaps and overlaps sounds 
absolutely horrible (in multiple senses). Not to mention the complexity 
of implementation implied and the likelihood of a project without VLC's 
resources of getting this right (as much as that is even possible).

> handle this case best for them.

Well, okay, but you haven't actually specified how to make that 
decision. It also seems hard to harmonize with this advice from a prior 
mail:

> container or the codec. In most cases it should be the container.

For Matroska versions 1...3, probably the best you can do is try to 
write down some careful rules about how you're supposed to round 
timestamps and durations and other values. But in practice I expect few 
implementations will get this right (if history is any indication). Even 
just the requirement for enough precision to get sample accuracy for 
audio is regularly violated, since the default precision is only at the 
millisecond level.

Going forward, my opinion (as an individual only) is that you should 
seriously reconsider the way timestamps are represented. Especially for 
an archive format. No other modern container in popular use has these 
problems.

---

From: Moritz Bunkus <[email-masked]>
Date: Tue, 19 Jul 2016 19:19:14 +0200

Hey,

I happen to agree with Timothy. There have been legitimate complaints
about Matroska's timestamp handling for as long as I can remember, and
we've never gotten around to actually fixing it properly.

Now might be the time to actually attempt it.

The problem is that such a change would be completely incompatible with
existing players. So we should think really hard about this and either
do it right once or not at all — e.g. introduce new elements, redefine
the interpretation of certain existing ones and deprecate others all at
the same time. Otherwise we'll end up with multiple conflicting pieces
of information about the calculation and validity of timestamps within
the same file.

Such files should not be Matroska v4 but rather v5 already. Why? Because
v4 files are actually used in the wild already, even though we haven't
finalized that version yet. However, suddenly changing the timestamp
handling radically would invalidate the existing files. Not a good move,
obviously.

Anyway. I see this standardization process as the one time we may
actually implement such a wide-ranging change.

I'll try to write up a proposal as a basis for discussion this weekend.

Kind regards,
mosu

---

From: Jerome Martinez <[email-masked]>
Date: Thu, 21 Jul 2016 11:55:43 +0200

On 19/07/2016 19:19, Moritz Bunkus wrote:
> [...]
> existing players.

I don't like the idea to have an incompatible change with such feature, 
because this is not a "must have" for 99% of people, and Matroska "v5" 
will not be adopted with that.
I think that we can have a workaround for having additional 0.99% happy 
without the need of a break in compatibility:
- lot of people want to know the exact frame rate of a fixed frame rate 
stream, people muxing from tapes know that the frame rate is fixed 
(hardware limitation) so can set this element in the header. putting 
frame rate info in the header with numerator and denominator is enough 
for doing the difference between 30000/1001 and 29970/1000, and the 
current millisecond timestamp is enough (a demuxer aware of the 
framerate element can "correct" the timecode from e.g. 0.033s to 
1001/30000 second; an old player just play the file with 1 ms 
precision). We could add some rules in case a timecode does not fit the 
frame rate (e.g. if timecode is 0.031, frame rate info is considered not 
applicable)
- people having variable frame rate with very precise timestamp need to 
know it before starting the muxing and can use a Timecodescale element 
of 1 (so precision of 1 nanosecond, usually enough when there is no 
fixed frame rate).
Which real life scenario can not deal with such workaround?

then we define v5 with the right change about time code (we prepared it 
in advance?) when we get the killer feature ye need for having it 
accepted by lot of players