Subject: Re: [Cellar] [Matroska-devel] Depth offsets for subtitles in case of 3D MVC tracks in MKV files

From: Steve Lhomme <[email-masked]>
Date: Sun, 14 Feb 2016 15:50:50 +0100

2016-01-26 14:39 GMT+01:00 Nithin Mathew Kurien <[email-masked]>:
> Hi,
>
> It's like this. Suppose the MVC video track has 8 frames. Also suppose it
> has 5 Offset Metadata Sequences embedded in it at the time of encoding.
> (Sequence entries are in pixels, positive if in front of the screen, and
> negative if behind.)
>
> Frame  1  2  3  4  5  6  7  8
> Seq1    5  1 -8 -2 -9  1 -3 -1
> Seq2    8  1  0  8 -4 -5  4 -1
> Seq3   -2  8 -3 -5  7  5 -5 -4
> Seq4    9  0  0 -5 -2  1 -4 -1
> Seq5   -9  3 -2 -4 -6  0 -3  6
>
> If the M2TS container has 2 subtitle tracks P1, P2 and 1 menu track I1, then
> a mapping can be defined in the M2TS container like this:
> P1 : Seq1
> P2 : Seq3
> I1  : Seq4

OK, so it seems it's both in the header and in the track (the GOP is
part of the stream, I think).

> So basically, the sequences are defined in the MVC track and the mapping is
> defined in the M2TS container. If this mapping is not defined in the
> container, the player would display the subtitle or menu track at the plane
> of the screen. Thus the outcome would be as if a sequence whose all entries
> are 0, is assigned to the subtitle or menu track. This has been a problem
> with the few hardware media players supporting MVC inside MKV, for example
> the Mede8er ones (http://www.mede8erforum.com/index.php?topic=9624.0).
>
> Ideally, a menu track should be displayed in front of the subtitle track,
> which should be in front of the video track. This can be done through the
> proper definitions of sequences and mappings. The free program
> MVCPlanes2OFS.exe from the package BD3D2MK3D
> (http://forum.doom9.org/showthread.php?t=170828) can display all Offset
> Metadata Sequences embedded in a .MVC file.
>
> Thanks and regards,
> Nithin
>
>
> On Tue, Jan 26, 2016 at 6:46 PM, madshi <[email-masked]> wrote:
>>
>> 2016-01-26 13:34 GMT+01:00 Steve Lhomme via Matroska-devel
>> <[email-masked]>:
>> > I'm not sure I understand. Is the offset per track or per frame for
>> > the subtitle ? If it's per track it should go in the Track Info. If
>> > not that means a mechanism to add that information for each frame.
>> > Maybe BlockGroup could help.
>>
>> The 3D Blu-Ray spec says:
>>
>> > P.9.3.3.6 Offset metadata
>> > The offset_metadata() shall be stored in the user data container
>> > (defined in 9.16.3.1) which is contained in MVC scalable nesting
>> > SEI message of the first view component of each GOP of MPEG-4
>> > MVC Dependent view video stream in decoding order.
>>
>> I'm not a big expert on these things, but doesn't that mean it's part
>> of the MVC video bitstream? In that case there's probably no need to
>> store it in separate MKV fields/properties?
>>
>> Best regards, Mathias.
>
>

-- 
Steve Lhomme
Matroska association Chairman

---

From: Steve Lhomme <[email-masked]>
Date: Mon, 29 Feb 2016 08:51:06 +0100

2016-02-28 23:12 GMT+01:00 madshi <[email-masked]>:
> 2016-02-14 15:50 GMT+01:00 Steve Lhomme <[email-masked]>:
> Yes. As Nithin Mathew explained (thanks!), there are SEI 3D depth info
> blocks inside of the video stream. We don't need to do anything about them.
> But there's additional data in the Blu-Ray playlist file(s) which
> assigns specific
> subtitle tracks to an "array index" in the SEI 3D depth info block.
>
> In the meanwhile nevcairiel and I have already added full support for all
> this 3D depth offset stuff to LAV Splitter/Video decoder and madVR, when
> playing back original Blu-Rays. But for Matroska there's no full solution
> possible yet, because the information in the playlist cannot be properly
> stored yet.
>
> It would be great if we could get an official solution for this soon, because
> we'd like to implement it "now", and we have the MakeMKV devs on board
> for this now, too.
>
> Basically I think what we need is one additional item for each subtitle
> track which simply assigns one SEI 3D depth array index to the subtitle
> track. E.g. something like "stereo_subtitle_offset_id", or whatever. The
> name is not important to me. Possible values I think are between 0..31
> for 3D Blu-Rays.

That seems to be heavily relying on the Blu-ray specs. I'd favor a
solution that is more generic and doesn't have to go through "codec
specific" lookup tables.

Otherwise if the codec used for Blu-Rays subtitles is already
specific, we could just use the CodecState to store the SEI value when
it changes.
https://matroska.org/technical/specs/index.html#CodecState

That means custom made text subtitles will not work properly with 3D
Blu-ray content because it doesn't use the same codec.

If we want the general case to work we may introduce an element at the
same level as CodecState that defines the Z-order for the Block. That
means it would have to translate the lookup index in the SEI to the
actual z-order value (if I understand correctly how it works). We need
to define if it's like the CodecState, ie it only changes from scene
to scene, not Block by Block. So it would be a State that needs to be
remembered when needed and only written when it changes.

What is the range of depth allowed in Blu-Rays ? Are they discrete
value or floating values ?

> One additional piece of information would be which array indexes of the
> SEI 3D depth info block are assigned to IG (interactive graphics). This
> could be used by media players or video renderers to decide at which
> 3D depth to draw the user interface (GUI/OSD). FWIW, it could be none,
> one or multiple array indexes. So the information field in the MKV header
> would have to be some sort of list/array. Logically this info would
> probably belong to the video track. This is less important than the
> subtitle assignment, though. But would still be nice to have.
>
> Best regards, madshi.

---

From: Steve Lhomme <[email-masked]>
Date: Mon, 29 Feb 2016 09:25:01 +0100

FW answer

---------- Forwarded message ----------
From: Steve Lhomme <[email-masked]>
Date: 2016-02-29 9:24 GMT+01:00
Subject: Re: [Matroska-devel] [Cellar] Depth offsets for subtitles in
case of 3D MVC tracks in MKV files
To: Hendrik Leppkes <[email-masked]>

2016-02-29 9:17 GMT+01:00 Hendrik Leppkes <[email-masked]>:
> On Mon, Feb 29, 2016 at 8:51 AM, Steve Lhomme <[email-masked]> wrote:
>>
> The "depth" is defined in discrete pixel values in the BD SEI, ie. the
> offset to move it left/right for each eye, so it doesn't have a direct
> meaning of "depth", but just instructions on how to render it.

Ah, interresting. We could still use a z-order value, because that's
what 3D is about. But then it would probably be a float to actually
translate to a wide range of possibilities. But, as seen with the rare
float values in Matroska, in the case where it's actually used (this
case) it would just lead to approximate values of the original format.

> Note that the way this is implemented on Blu-rays, the depth can
> change from frame to frame. Every GOP has a new SEI block which
> includes one offset for every frame in the SEI, so its controlled on a
> frame level, not a scene/GOP level.

So I think we should go with the CodecState approach. Meaning we need
a specific codec ID for such Blu-Ray subtitles. We should not write
the CodecState if the values are the same as the time it was
previously written.

This is also assuming the subtitles are always key frames, so the
(de)coding order is also the display order.

---

From: Steve Lhomme <[email-masked]>
Date: Mon, 29 Feb 2016 09:57:24 +0100

> the CodecState if the values are the same as the time it was

Alternatively if the values are expected to change a lot
BlockAdditional could be used. The data are left to interpretation by
the codec, just like CodecState but doesn't require the blocks to be
added to the Seek entries.

https://matroska.org/technical/specs/index.html#BlockAdditional

---

From: Moritz Bunkus <[email-masked]>
Date: Mon, 29 Feb 2016 10:04:25 +0100

Hey,

> CodecState is per Cluster, though, not per Track,

This is wrong. CodecState is a child element of BlockGroup. It's effect
as it's been meant to be understood so far is to replace CodecPrivate
for that track from that BlockGroup onwards (until another BlockGroup
with CodecState comes along). So it _could_ be used for changing Z-depth
values for individual subtitle frames. However, as it's supposed to
replace CodecPrivate I don't think it would be a good fit for Z-depth
data as Z-depth data is additional data that changes on a frame-by-frame
basis; it doesn't invalidate the existing CodecPrivate data.

CodecState was meant for situations in which e.g. a whole new set of
SEIs/PPSs come along.

I'd prefer new elements, or maybe BlockAddition.

Kind regards,
mosu