Subject: Re: [Cellar] [Matroska-devel] Colour Format proposal

From: Dave Rice <[email-masked]>
Date: Tue, 17 May 2016 13:52:56 -0400

Hi Steve, Frank, Reto, and all,

> On Apr 18, 2016, at 4:29 AM, Steve Lhomme via Matroska-devel <[email-masked]> wrote:
> 
> Following the Videolan Colorimetry workshop from this week-end, I have
> a better understanding of this whole subject.

I wanted to bump this conversation and topic. I'm curious to hear more of what came out of the Videolan Coloimetry workshop, but I also see that the color format proposed elements are now integrated into https://matroska.org/technical/specs/index.html <https://matroska.org/technical/specs/index.html>, as well as in process in Libav and FFmpeg, and the webm project. I don't mean to rush the specification work but wanted to note that the proposed color elements are in use. I suggest that we give this work a status to say if it is complete or needs more work (and if so, identify what work remains).

> I think it's good if we can have as much information in Matroska as we
> can. But they probably should be split between elements that are
> necessary for accurate playback (list to be defined) and the other
> information like how the content was captured on camera. The first
> group should be in the TrackInfo while the others should be just tags
> (so with string names) that target the whole file or some chapters
> (each scene probably has different capture characteristics).
> 
> From Frank's list I would put those in the TrackInfo:
> - MatrixCoefficients
> - Range
> - TransferFunction
> - Primaries
> 
> - BitsPerChannel
> - ChromaSubsamplingHorz / ChromaSubsamplingVert
> - CbSubsamplingHorz / CbSubsamplingVert
> 
> I'm not sure about the Luminance min/max but maybe it's needed for HDR ?
> Not sure either about arbitrary RGB chromacity values of the
> primaries. Are people deviating from the few norms that are available
> ? And have tools for playback ? That seems to provide the same
> information as the "Primaries" field and it's never good to have 2
> different set of values describing the same thing. You never know if
> you have to write both and which to use for playback especially when
> you need 8 values to make one set to compare...

I'm interested in hearing about the use cases for these (ping to Reto). Are we certain that the "Primaries" element overlaps semantically with the Primary[R|G|B]Chromaticity[X|Y] elements? Perhaps we should define one to overrule the other.

> Are MaxCLL and MaxFALL exclusive ? Are they needed for HDR playback ?
> 
> I think MasteringMetadata is not needed for playback.
> 
> For list of possible values, we shouldn't follow lists from standards
> that are not freely available. We should also have the default value
> be 0 when possible.

For much of the semantics used in the proposed colour elements, '2' rather than '0' has been used to express an 'undetermined' state whereas '0' has a specific meaning. I think it's consistent to keep it this way as it better aligns with existing specification (even if non-free) and existing implementations (free and not free).

> And no "reserved" values.

Same comment as above. If we are deriving a vocabulary from an established existing standard then I think adopting traditional reserved values is okay.
Best Regards,
Dave Rice

> 2016-03-29 19:12 GMT+02:00 Hendrik Leppkes via Matroska-devel
> <[email-masked]>:
>> On Tue, Mar 29, 2016 at 6:11 PM, Frank Galligan via Matroska-devel
>> <[email-masked]> wrote:
>>> 
>>> 
>>> On Mon, Mar 28, 2016 at 5:05 AM, Steve Lhomme <[email-masked]> wrote:
>>>> 
>>>> 2016-03-17 5:46 GMT+01:00 Frank Galligan <[email-masked]>:
>>>>> OK really no comments for a long time.
>>>>> 
>>>>> On Fri, Feb 19, 2016 at 1:45 PM, Michael Niedermayer
>>>>> <[email-masked]> wrote:
>>>>>> 
>>>>>> Hi
>>>>>> 
>>>>>> On Thu, Feb 18, 2016 at 11:50:27AM -0800, Frank Galligan wrote:
>>>>>>> Here is the current proposal, minus the reference to the 265 doc.
>>>>>>> 
>>>>>>> The parent element would be Video [E0].
>>>>>>> 
>>>>>>> 
>>>>>>> Element Name: Colour
>>>>>>> 
>>>>>>> Level:        4
>>>>>>> 
>>>>>>> ID:           [55][B0]
>>>>>>> 
>>>>>>> Mandatory:    -
>>>>>>> 
>>>>>>> Multiple:     -
>>>>>>> 
>>>>>>> Default:      -
>>>>>>> 
>>>>>>> Type:         m
>>>>>>> 
>>>>>>> Description:  Settings describing the colour format.
>>>>>>> 
>>>>>>> 
>>>>>>> Element Name: MatrixCoefficients
>>>>>>> 
>>>>>>> Level:        5
>>>>>>> 
>>>>>>> ID:           [55][B1]
>>>>>>> 
>>>>>>> Mandatory:    -
>>>>>>> 
>>>>>>> Multiple:     -
>>>>>>> 
>>>>>>> Default:      2
>>>>>>> 
>>>>>>> Type:         u
>>>>>>> 
>>>>>>> Description:  The Matrix Coefficients of the video used to derive
>>>>>>> luma
>>>>>>> and
>>>>>>> 
>>>>>>>             chroma values from reg, green, and blue color primaries.
>>>>>>> For
>>>>>>> 
>>>>>>>             clarity, the value and meanings for MatrixCoefficients
>>>>>>> are
>>>>>>> adopted
>>>>>>> 
>>>>>>>             from Table 4 of ISO/IEC 23001-8:2013/DCOR1. (0:GBR, 1:
>>>>>>> BT709,
>>>>>>> 
>>>>>>>             2: Unspecified, 3: Reserved, 4: FCC, 5: BT470BG, 6:
>>>>>>> SMPTE
>>>>>>> 170M,
>>>>>>> 
>>>>>>>             7: SMPTE 240M, 8: YCOCG, 9: BT2020 Non-constant
>>>>>>> Luminance,
>>>>>>> 
>>>>>>>             10: BT2020 Constant Luminance)
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>>> Element Name: BitsPerChannel
>>>>>>> 
>>>>>>> Level:        5
>>>>>>> 
>>>>>>> ID:           [55][B2]
>>>>>>> 
>>>>>>> Mandatory:    -
>>>>>>> 
>>>>>>> Multiple:     -
>>>>>>> 
>>>>>>> Default:      0
>>>>>>> 
>>>>>>> Type:         u
>>>>>>> 
>>>>>>> Description:  Number of decoded bits per channel. A value of 0
>>>>>>> indicates
>>>>>>> that
>>>>>>> 
>>>>>>>             the BitsPerChannel is unspecified.
>>>>>>> 
>>>>>> 
>>>>>> what would this be set to for old 16bit rgb, that is 5 bit red
>>>>>> 6 bit green, 5 bit blue rawvideo.
>>>>>> This maybe does not matter and iam not strongly suggesting to add it,
>>>>>> rather i want to point it out so its not unintentionally forgotten
>>>>> 
>>>>> 
>>>>> I didn't get into RGB here. As you said 565, 551, there are a good
>>>>> amount of
>>>>> combinations. I think DirectShow (or maybe it was DirectDraw) that had
>>>>> R,G,B, and A masks to show which bits belonged to which channel. I also
>>>>> worked with formats like RGBBGR repeating.
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> [...]
>>>>>> 
>>>>>>> Element Name: CbSubsamplingHorz
>>>>>>> 
>>>>>>> Level:        5
>>>>>>> 
>>>>>>> ID:           [55][B5]
>>>>>>> 
>>>>>>> Mandatory:    -
>>>>>>> 
>>>>>>> Multiple:     -
>>>>>>> 
>>>>>>> Default:      -
>>>>>>> 
>>>>>>> Type:         u
>>>>>>> 
>>>>>>> Description:  The amount of pixels to remove in the Cb channel for
>>>>>>> every
>>>>>>> pixel
>>>>>>> 
>>>>>>>             not removed horizontally. This is additive with
>>>>>>> 
>>>>>>>             ChromaSubsamplingHorz. Example: For video with 4:2:1
>>>>>>> chroma
>>>>>>> 
>>>>>>>             subsampling, the ChromaSubsamplingHorz should be set to
>>>>>>> 1
>>>>>>> and
>>>>>>> 
>>>>>>>             CbSubsamplingHorz should be set to 1.
>>>>>>> 
>>>>>>> 
>>>>>>> Element Name: CbSubsamplingVert
>>>>>>> 
>>>>>>> Level:        5
>>>>>>> 
>>>>>>> ID:           [55][B6]
>>>>>>> 
>>>>>>> Mandatory:    -
>>>>>>> 
>>>>>>> Multiple:     -
>>>>>>> 
>>>>>>> Default:      -
>>>>>>> 
>>>>>>> Type:         u
>>>>>>> 
>>>>>>> Description:  The amount of pixels to remove in the Cb channel for
>>>>>>> every
>>>>>>> pixel
>>>>>>> 
>>>>>>>             not removed vertically. This is additive with
>>>>>>> 
>>>>>>>             ChromaSubsamplingVert.
>>>>>> 
>>>>>> What if Cr is subsampled more than Cb ?
>>>>>> That too is rather obscure, but theres code in FFmpeg to handle such
>>>>>> jpegs, so i suspect this case while very rare is not entirely non
>>>>>> existent ...
>>>>> 
>>>>> 
>>>>> I guess we can add that as well if more people really want it. Actually
>>>>> I
>>>>> didn't even have CbSubsampling* elements at first. I only added that to
>>>>> support the 4:2:1 format that was defined in the first enum.
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Element Name: ChromaSitingHorz
>>>>>>> 
>>>>>>> Level:        5
>>>>>>> 
>>>>>>> ID:           [55][B7]
>>>>>>> 
>>>>>>> Mandatory:    -
>>>>>>> 
>>>>>>> Multiple:     -
>>>>>>> 
>>>>>>> Default:      0
>>>>>>> 
>>>>>>> Type:         u
>>>>>>> 
>>>>>>> Description:  How Chroma is subsampled horizontally. (0: Unspecified,
>>>>>>> 1:
>>>>>>> Left
>>>>>>> 
>>>>>>>             collocated , 2: Half)
>>>>>>> 
>>>>>>> Element Name: ChromaSitingVert
>>>>>>> 
>>>>>>> Level:        5
>>>>>>> 
>>>>>>> ID:           [55][B8]
>>>>>>> 
>>>>>>> Mandatory:    -
>>>>>>> 
>>>>>>> Multiple:     -
>>>>>>> 
>>>>>>> Default:      0
>>>>>>> 
>>>>>>> Type:         u
>>>>>>> 
>>>>>>> Description:  How Chroma is subsampled vertically. (0: Unspecified,
>>>>>>> 1:
>>>>>>> Top
>>>>>>> 
>>>>>>>             collocated , 2: Half)
>>>>>>> 
>>>>>> 
>>>>>> iam not sure this is enough to specify all variants
>>>>>> for 4:2:0 alone there are a few different variants
>>>>>> theres mpeg1 style
>>>>>> mpeg2 progressive and interlaced
>>>>>> the mpeg2/mpeg4 style also differs from itself if the image is fliped
>>>>>> right-left
>>>>>> cropping 1 or 2 lines of the top of mpeg2 yuv420 also results in
>>>>>> different variants
>>>>> 
>>>>> 
>>>>> 
>>>>> I also didn't get into interlaced.
>>>>> 
>>>>> 
>>>>> 
>>>>> I don't think we should add enough elements to support every format that
>>>>> was
>>>>> ever produced. Opinions?
>>>> 
>>>> For archival (one of the main goal here) I think we should.
>>>> 
>>>>> I think we should probably strive to support 99% of what is currently
>>>>> produced today. Opinions?
>>>> 
>>>> The problem is that when you leave 1% out, you need to be sure it's
>>>> possible to integrate it later. Usually the best approach is to know
>>>> beforehand how you're going to do it. So in the end it's just like
>>>> covering it. That's just a general remark though.
>>> 
>>> I don't want to exclude anything in the future, but I also don't want to
>>> make something overly complex/over specified to support a format that will
>>> barely see any use.
>>> 
>>> I want to move forward with this, so how about we make a list and then vote
>>> on what we think should be added to the sepc?
>>> 1. Raw RGB
>>> 2. Raw YUV
>>> 3. Interlaced
>>> 4. mpeg1 style
>>> 5. mpeg2 pogessive
>>> 6. mpeg2 interlaced
>>> 7. image flipped right-left
>>> 8. jpeg Cr subsampled more than Cb
>>> 9. non-linear color values (DPX scans)
>>> 
>>> Anything else we should add?
>>> 
>> 
>> Just my 2 cents, but if you want to support everything and the kitchen
>> sink, you should use string fields that can be arbitrarily expanded,
>> instead of integer enumerations.
>> 
>> - Hendrik
>> _______________________________________________
>> Matroska-devel mailing list
>> [email-masked]
>> http://lists.matroska.org/cgi-bin/mailman/listinfo/matroska-devel
>> Read Matroska-Devel on GMane: http://dir.gmane.org/gmane.comp.multimedia.matroska.devel
> 
> 
> 
> -- 
> Steve Lhomme
> Matroska association Chairman
> _______________________________________________
> Matroska-devel mailing list
> [email-masked]
> https://lists.matroska.org/cgi-bin/mailman/listinfo/matroska-devel
> Read Matroska-Devel on GMane: http://dir.gmane.org/gmane.comp.multimedia.matroska.devel

---

From: Steve Lhomme <[email-masked]>
Date: Wed, 18 May 2016 09:15:36 +0200

2016-05-17 19:52 GMT+02:00 Dave Rice <[email-masked]>:
> On Apr 18, 2016, at 4:29 AM, Steve Lhomme via Matroska-devel
> <[email-masked]> wrote:
>
> I wanted to bump this conversation and topic. I'm curious to hear more of
> what came out of the Videolan Coloimetry workshop, but I also see that the

The result can be summarized in the introduction of these 4 enums:
http://git.videolan.org/?p=vlc.git;a=blob;f=include/vlc_es.h;h=9c0feb9f81257a1f628096f914d5ecb79eea4011;hb=HEAD#l195

Since the VLC code is basically for playing or converting videos,
there was no need to support everything under the sun, as it should
with Matroska.

We identified these 4 pieces to be the data needed for proper
rendering of various color formats.

Also playback support is not yet fully implemented, for example
BT.2020 is not there yet.

> color format proposed elements are now integrated into
> https://matroska.org/technical/specs/index.html, as well as in process in
> Libav and FFmpeg, and the webm project. I don't mean to rush the
> specification work but wanted to note that the proposed color elements are
> in use. I suggest that we give this work a status to say if it is complete
> or needs more work (and if so, identify what work remains).
>

Yes, although I haven't found much information on how to support HDR
yet (not handled by VLC).

> I'm interested in hearing about the use cases for these (ping to Reto). Are
> we certain that the "Primaries" element overlaps semantically with the
> Primary[R|G|B]Chromaticity[X|Y] elements? Perhaps we should define one to
> overrule the other.
>

IMO we shouldn't have elements we don't understand, or they should be
explained, possibly with external links.

> I think MasteringMetadata is not needed for playback.

Agreed.

> that are not freely available. We should also have the default value

As for the audio sampling frequency, we should put the default value
for SD content. That is files that are intentionally (or historically)
small so they stay small. The other (more) important rule is that old
files without the value should fall into the default value in the
majority of cases.

> For much of the semantics used in the proposed colour elements, '2' rather
> than '0' has been used to express an 'undetermined' state whereas '0' has a
> specific meaning. I think it's consistent to keep it this way as it better
> aligns with existing specification (even if non-free) and existing
> implementations (free and not free).
>
> Same comment as above. If we are deriving a vocabulary from an established
> existing standard then I think adopting traditional reserved values is okay.
> Best Regards,
> Dave Rice
>
> Read Matroska-Devel on GMane:
> http://dir.gmane.org/gmane.comp.multimedia.matroska.devel
>
>
>
>
> --
> Steve Lhomme
> Read Matroska-Devel on GMane:
> http://dir.gmane.org/gmane.comp.multimedia.matroska.devel
>
>