Subject: [Cellar] Matroska support for closed captions?

From: Scott Shepherd <[email-masked]>
Date: Wed, 22 Apr 2020 13:48:29 +0000

Hi all,

Are there any plans to expand Matroska capability to contain additional stream types such as timecode and closed captions?

We have some .MXF files (mostly JPEG2000) and are toying with the idea of migrating them to FFV1/.MKV, but currently .MKV can't contain timecode or closed caption streams. While I can't say I fully understand the need for multiple timecode streams (often there are 3 streams in the .MXF), we'd definitely be losing data where closed captions exist.

I know that .MKV can hold "attachments," but attachments don't seem like the right solution here. I also know .MKV can hold subtitles, but what we're seeing in these .MXF files is EIA-608 and EIA-708 closed captions that fail to carry over to .MKV.

Ideally, we'd use ffmpeg to transcode JPEG2000/.MXF to FFV1/.MKV and map all existing streams. Is that possible currently? If not, is it on the roadmap or can it be put on the roadmap? This is one of the main obstacles keeping us from embracing the FFV1/.MKV combination.

Scott Shepherd

---

From: Dave Rice <[email-masked]>
Date: Mon, 27 Apr 2020 11:46:32 -0400

Hi Scott,

> On Apr 22, 2020, at 9:48 AM, Scott Shepherd <sbshepherd=[email-masked]> wrote:
> 
> Hi all,
>  

Feel welcome to open a ticket at https://github.com/cellar-wg/matroska-specification/issues/new.

> We have some .MXF files (mostly JPEG2000) and are toying with the idea of migrating them to FFV1/.MKV, but currently .MKV can't contain timecode or closed caption streams. While I can’t say I fully understand the need for multiple timecode streams (often there are 3 streams in the .MXF), we’d definitely be losing data where closed captions exist.
>  
> I know that .MKV can hold "attachments," but attachments don't seem like the right solution here. I also know .MKV can hold subtitles, but what we’re seeing in these .MXF files is EIA-608 and EIA-708 closed captions that fail to carry over to .MKV. 

For EIA-608 and EIA-708 data I suppose, either a new type of Codec Mapping could be written (and then these would be a new type of subtitle track) or they could be support as side-data with a BlockAdditionalMapping as described at https://www.ietf.org/id/draft-ietf-cellar-codec-04.html#name-block-additional-mapping. I’m unsure which technique the community would recommend but curious for some discussion on it.

[…]

Best Regards,
Dave Rice

---

From: Jerome Martinez <[email-masked]>
Date: Mon, 27 Apr 2020 18:08:11 +0200

> For EIA-608 and EIA-708 data I suppose, either a new type of Codec Mapping could be written (and then these would be a new type of subtitle track) or they could be support as side-data with a Block…

As MXF have JPEG-2000, I guess that captions are in a standalone 
ancillary data (SMPTE ST 291) track.
As a first shot, I would not be in favor of BlockAdditionalMapping 
because the track does not depend on data in another bistream.

then there would (at least) 2 choices:
- demux of the track from MXF and mux of this track as is; advantage is 
that we lose no other data inside the ancillary data, disadvantage is 
that lot of different content (several formats of captions, bar data, 
VBI, time codes, Acquisition Metadata...) are muxed in the "raw" stream, 
more difficult for a demuxer/player.
- demux of the captions from Ancillary data from MXF; advantage is that 
we have 1 track per content stream, disadvantage is that we may lose 
some other "opaque" data from the ancillary data.

That said, we could propose both.

Jérôme

---

From: Devin Heitmueller <[email-masked]>
Date: Mon, 27 Apr 2020 12:53:46 -0400

> That said, we could propose both.

For what it's worth, we have the same general problem within ffmpeg.
File formats like MXF and MP4 treat captions and timecodes as a
separate track, while other formats like TS expect it to be tied to
the individual video frames as side data.  There's no easy answer, and
in particular it's a huge pain when you need to convert from one to
the other (e.g. extracting captions from the SEI in an H.264 TS where
it gets treated as video frame side data, and creating an MP4 where
captions need their own subtitle track).

Devin

-- 
Devin J. Heitmueller - Kernel Labs
http://www.kernellabs.com

---

From: Hendrik Leppkes <[email-masked]>
Date: Mon, 27 Apr 2020 21:15:27 +0200

> captions need their own subtitle track).

Well, its more like TS does not handle closed captions at all, which
is why they get thrown in with the video. Its not a TS concept, but a
video one. If we have the option to carry them seperately, then I
believe we should. It makes it so much easier for everything involved
- and there is precedent for that in eg. MP4 already as well.
You can always go the "TS way" with Matroska as well if you must,
since its entirely container-agnostic, but if container support is
being build, I see no reason to handle them differently then "proper"
subtitles.

- Hendrik