Subject: [Cellar] Clarification on blocks with a negative timecode

From: Michael Bradshaw <[email-masked]>
Date: Thu, 7 Jan 2016 14:39:54 -0800

Blocks can have a negative timecode in the following situations:

   - Cluster Timecode + Block timecode < 0. This may happen if the
   Cluster's Timecode is near zero and the Block's timecode is sufficiently
   negative.
   - CodecDelay > Cluster Timecode + Block timecode. I have a WebM file
   created by ffmpeg with an Opus audio track that has a CodecDelay of 6.5ms,
   and since the first block's timecode is zero, its computed timecode is
   -6.5ms. mkvinfo shows this frame has having a timecode of 0.000s, but
   ffprobe shows a PTS of -0.007ms (rounding up).

The notes say that "Blocks with a negative Raw Timecode are not valid."[1]
This page, however, does not mention CodecDelay's role in computing
timecodes.

Questions:

   - Should the Specification Notes talk about CodecDelay's role in
   computing a block's timecode?
   - Assuming the answer to the previous questions is "yes," is a block
   with a negative timecode due to a CodecDelay still invalid (i.e. is
   CodecDelay a part of Raw Timecodes)?

--Michael

[1]: http://matroska.org/technical/specs/notes.html

---

From: Ralph Giles <[email-masked]>
Date: Thu, 7 Jan 2016 15:05:41 -0800

On 2016-01-07 2:39 PM, Michael Bradshaw wrote:

>   * Should the Specification Notes talk about CodecDelay's role in

Yes. This is pretty confusing, and I think implementation guidance is
welcome.

>   * Assuming the answer to the previous questions is "yes," is a block

The way we intended this to work is that the CodecDelay is subtracted
from the declared timestamps to get the actual timestamps. This is
specifically to get around the issue of declaring timestamps which work
out to be negative for samples which should be trimmed.

Assuming that's what people have been implementing, CodecDelay on its
own shouldn't make a block invalid.

It's helpful if the spec can specify what implementation must/should do
with blocks or clusters with negative timestamps. Should they Trim to
zero? Ignore the block? Reject the file?

 -r

---

From: Steve Lhomme <[email-masked]>
Date: Mon, 11 Jan 2016 11:19:48 +0100

2016-01-07 23:39 GMT+01:00 Michael Bradshaw <[email-masked]>:
> Cluster Timecode + Block timecode < 0. This may happen if the Cluster's
> Timecode is near zero and the Block's timecode is sufficiently negative.
> CodecDelay > Cluster Timecode + Block timecode. I have a WebM file created
> by ffmpeg with an Opus audio track that has a CodecDelay of 6.5ms, and since
> the first block's timecode is zero, its computed timecode is -6.5ms. mkvinfo
> shows this frame has having a timecode of 0.000s, but ffprobe shows a PTS of
> -0.007ms (rounding up).
>
>
> Should the Specification Notes talk about CodecDelay's role in computing a
> block's timecode?
> Assuming the answer to the previous questions is "yes," is a block with a
> negative timecode due to a CodecDelay still invalid (i.e. is CodecDelay a
> part of Raw Timecodes)?

IMO it should be invalid. Does this 6.5ms before 0.0 have any decoded
data at all ? If so it seems to be an abuse of the CodecDelay
(although the definition doesn't cover that).

> _______________________________________________
> Cellar mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cellar
>

-- 
Steve Lhomme
Matroska association Chairman

---

From: Michael Bradshaw <[email-masked]>
Date: Mon, 11 Jan 2016 10:47:24 -0800

> (although the definition doesn't cover that).

I've attached the file (tos.mkv) and the first 6.5ms of decoded audio
(first_6.5ms.wav). When ffmpeg decodes the MKV, AVStream.skip_samples is 0.
These first 6.5ms of audio (312 samples, to be exact) are not silent. I've
repeated these first 312 samples 400 times (repeated_400_times.wav) so you
can listen and here the data is in fact not silent.

ffmpeg produced this MKV file. I think this might be a good example that
shows it's important to clarify this in the specs.

---

From: Steve Lhomme <[email-masked]>
Date: Thu, 14 Jan 2016 12:08:50 +0100

2016-01-13 20:04 GMT+01:00 Frank Galligan <[email-masked]>:
> I don't think it is invalid. CodecDelay means a track needs to decode this
> data and throw it away as it is invalid data. Then all of the decoded data
> following CodecDelay will be valid.

Yes, that's what I meant by invalid. Although it produces data, it's
actually meant to be played. It's more like "seeding" samples. IIRC
that's how pre-roll in DirectShow works, you can send data with old
timestamps, but only data after the pre-roll timestamp will play.

> Most audio codecs have a "generated" coding delay.

Correct, even MP3 has it. Only Opus needed this to be known at the
container level for reasons that escape me now.

---

From: Frank Galligan <[email-masked]>
Date: Thu, 14 Jan 2016 09:39:37 -0800

On Thu, Jan 14, 2016 at 3:08 AM, Steve Lhomme <[email-masked]> wrote:

> >> > Should the Specification Notes talk about CodecDelay's role in
> computing
> >> > a
> >> > block's timecode?
> >> > Assuming the answer to the previous questions is "yes," is a block
> with
> >> > a
> >> > negative timecode due to a CodecDelay still invalid (i.e. is
> CodecDelay
> >> > a
> >> > part of Raw Timecodes)?
> >>
I think all of the Matroska audio tracks before/without CodecDelay will be
technically rendered wrong, but I think the number of pre-roll samples for
those codecs were small. I think Vorbis was less than 3 milliseconds. With
Opus, the pre-roll could be much larger, to the point that some people
might have noticed AV sync issues. Also trying to splice audio, for use
cases like adaptive would be very hard, if players did not know
about CodecDelay.

So I know when we first started talking about CodecDelay, it wasn't only
for Opus. Opus was the catalyst.

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Thu, 14 Jan 2016 13:20:52 -0800

> container level for reasons that escape me now.

No, Vorbis needed it also, but it was simply never implemented (which 
meant certain aspects of encapsulating Vorbis in MKV were broken without 
it, see below).

There are two use cases:

1) The delay introduced by the encoder is variable in Opus. For example, 
if the encoder knows the SILK layer is never going to be used, it can 
avoid adding the extra delay it requires (a decoder cannot know this in 
advance). Alternatively, if the encoder uses different internal 
resamplers than the reference implementation, those can require 
different delay offsets between CELT/SILK (and thus a different total 
delay).

For Vorbis, the delay was always a single frame (the delay was still 
variable, because that frame could be short or long, but this was 
signaled in the frame itself at the codec layer).

2) Sample-accurate cropping without re-encoding. In this case you want 
to start playing the data from the middle of a frame, and need to know 
how much to discard (this use-case applies to both Opus and Vorbis).

Ogg provided a mechanism to handle both of these, but before CodecDelay, 
MKV did not. That meant that if you muxed an Ogg Vorbis track that used 
start-trimming into a MKV container, the software would silently just 
change the duration of the track. The same problem existed with 
sample-accurate durations at the end of the track (much more common), 
but that's what DiscardPadding is for.

MP3 is simply incapable of representing sample-accurate start and end 
times (though in theory MP3 in MKV could now do so with these elements).

---

From: Steve Lhomme <[email-masked]>
Date: Fri, 15 Jan 2016 11:48:39 +0100

2016-01-14 22:20 GMT+01:00 Timothy B. Terriberry <[email-masked]>:
> No, Vorbis needed it also, but it was simply never implemented (which meant
> certain aspects of encapsulating Vorbis in MKV were broken without it, see
> below).
>
> There are two use cases:
>
> 1) The delay introduced by the encoder is variable in Opus. For example, if
> the encoder knows the SILK layer is never going to be used, it can avoid
> adding the extra delay it requires (a decoder cannot know this in advance).
> Alternatively, if the encoder uses different internal resamplers than the
> reference implementation, those can require different delay offsets between
> CELT/SILK (and thus a different total delay).
>
> variable, because that frame could be short or long, but this was signaled
> in the frame itself at the codec layer).
>
> 2) Sample-accurate cropping without re-encoding. In this case you want to
> start playing the data from the middle of a frame, and need to know how much
> to discard (this use-case applies to both Opus and Vorbis).
>
> Ogg provided a mechanism to handle both of these, but before CodecDelay, MKV
> did not. That meant that if you muxed an Ogg Vorbis track that used
> start-trimming into a MKV container, the software would silently just change
> the duration of the track. The same problem existed with sample-accurate
> durations at the end of the track (much more common), but that's what
> DiscardPadding is for.
>
> MP3 is simply incapable of representing sample-accurate start and end times
> (though in theory MP3 in MKV could now do so with these elements).

OK, so do we agree that negative timecodes should not be "rendered"
(play any audio) ?

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Fri, 15 Jan 2016 06:55:15 -0800

> OK, so do we agree that negative timecodes should not be "rendered"

I think the meaning of CodecDelay is clear: you should throw away the 
first "CodecDelay" worth of samples. What would be the behavior if the 
first block's timecode (before adjusting for CodecDelay) was positive? 
If it was already negative (again, before adjusting for CodecDelay)?

To my mind, this should be thought of as if the first "CodecDelay" worth 
of samples are simply deleted. They don't have a negative timecode. They 
don't exist. After doing that, the behavior should be the same as in the 
no-CodecDelay case. The problem here is that we currently talk about the 
timecode of a block, not of individual samples.

(speaking solely as an individual here)

---

From: Steve Lhomme <[email-masked]>
Date: Sun, 17 Jan 2016 14:09:28 +0100

2016-01-15 15:55 GMT+01:00 Timothy B. Terriberry <[email-masked]>:
> I think the meaning of CodecDelay is clear: you should throw away the first
> "CodecDelay" worth of samples. What would be the behavior if the first
> block's timecode (before adjusting for CodecDelay) was positive? If it was
> already negative (again, before adjusting for CodecDelay)?
>
> To my mind, this should be thought of as if the first "CodecDelay" worth of
> samples are simply deleted. They don't have a negative timecode. They don't
> exist. After doing that, the behavior should be the same as in the

Good point. Maybe it should be more clear in the specs.

The "negative" timecode are a different (but related) topic. IMO they
should not play, either. If you pass such frames in Directshow, they
will not render. It would be odd to add an element to shift timecodes
so that such timecodes are played anyway.

This is also the reason why when you cut a Matroska file, the second
part doesn't start at zero. Otherwise, depending on where you cut, it
may end up with negative timecodes, resulting in unplayed data.