Subject: Re: [Cellar] Colour Format proposal

From: Dave Rice <[email-masked]>
Date: Thu, 20 Oct 2016 10:32:25 -0400

Hi Frank and others,

> On Feb 11, 2016, at 1:32 PM, Frank Galligan <[email-masked]> wrote:
> 
> 
> On Mon, Feb 8, 2016 at 2:23 PM, Frank Galligan <[email-masked] <mailto:[email-masked]>> wrote:
> 
> 
> On Mon, Feb 8, 2016 at 9:15 AM, Jerome Martinez <[email-masked] <mailto:[email-masked]>> wrote:
> Sorry for the late answer, here are my comments:
> 
> On 22/01/2016 23:54, Frank Galligan wrote:
>> [...]
>> 
>> - I know some people expressed that they don't think starting from FFmpeg for the TransferFunction is a good idea as they might have got something wrong or too subjective. But I just used it as a starting point for a list. I can reorder the list however we want. I also don't think following another list, so we will be compatible with future additions, will give us what we want as that list most likely will be incomplete with something that is defined only in a different list. So should we just start with a list, re-order it (so we are not explicitly following it), then add what is currently missing? Then in the future if someone needs to add something to the list it can be brought up here?
> 
> That was me.
> I am OK with that as long as we are clear that we don't follow FFmpeg list.
> OK, I changed the text to see ISO/IEC 23001-8 document, which has the same values as FFmpeg as well as the 264 ITU doc.
> 
>> 
>> [...]
>> 
>> Element Name: Matrix
>> Level:        5
>> ID:           [55][A1]
>> Mandatory:    -
>> Multiple:     -
>> Default:      2
>> Type:         u
>> Description:  Colour Matrix of the video. (0: IEC 61966-2-1 (sRGB), 1: BT709,
>>               2: Unspecified, 3: Reserved, 4: FCC, 5: BT470BG, 6: SMPTE 170M,
>>               7: SMPTE 240M, 8: YCOCG, 9: BT2020 Non-constant Luminance,
>>               10: BT2020 Constant Luminance) 
> 
> 
> If we don't follow FFmpeg list, what is the purpose of  value 3 and why unspecified is 2 rather than 0?
> That is the way it is specified in ISO/IEC 23001-8 and 264 ITU doc. 
> 
> Using FFmpeg list (which is based on MPEG list, with same oddities) is not bad, and I am not against it, just wondering if it is good to take the same oddities rather than a clean list without oddities.
> I don't think there are any oddities. Looks like FFmpeg's list matches exactly to  ISO/IEC 23001-8 and 264 ITU doc, except for #10 in Primaries and #16 and #17 in Transfer function, which were added here [1]. Also #18 in Transfer, which is HLG [2].
> 
> I think this is the whole issue. Do we follow some list (standardized or not)? What if the list goes stale? What if the list doesn't include algorithms from different lists (E.g. maybe HLG)?
> 
> 
> 
> Does anyone plan to transfer from DPX? If yes, we may need to add now missing elements in the list (e.g. "Printing density") but I am not expert enough for know which one is worth it.
> 
> 
>> 
>> 
>> [...]
>> 
>> 
>> Element Name: ChromaSubsampling
>> Level:        5
>> ID:           [55][A3]
>> Mandatory:    -
>> Multiple:     -
>> Default:      0
>> Type:         u
>> Description:  (0: Unspecified, 1: 4:4:4, 2: 4:4:0, 3: 4:2:2, 4: 4:2:1, 5: 4:2:0,             
>>               6: 4:1:1, 7: 4:1:0, 8: 3:1:1)
> 
> I am more, as some other people, for something more generic, e.g. ChromaSubsamplingHorz and ChromaSubsamplingVert, with a value of the subsampling (0 unspecified and default). I am afraid that someone crazy used e.g. a vertical subsampling of 4 (which can not be defined by X:X:X values), it is possible (and I have a file like that if I remember well) with FFV1.
> 
> 
> 
>> 
>> [...]
>> 
>> Element Name: TransferFunction
>> Level:        5
>> ID:           [55][A7]
>> Mandatory:    -
>> Multiple:     -
>> Default:      2
>> Type:         u
>> Description:  Transfer Function. (0: Reserved, 1: ITU-R BT.709, 2: Unspecified,
>>               4: Gamma 2.2 curve, 5: Gamma 2.8 curve, 6: SMPTE 170M,
>>               7: SMPTE 240M, 8: Linear, 9: Log, 10: Log Sqrt,
>>               11: IEC 61966-2-4, 12: ITU-R BT.1361 Extended Colour Gamut,
>>               13: IEC 61966-2-1, 14: ITU-R BT.2020 10 bit,
>>               15: ITU-R BT.2020 12 bit, 16: SMPTE ST 2084,
>>               17: SMPTE ST 428-1 18: ARIB STD-B67 (HLG))
>> 
>> 
>> Element Name: Primaries
>> Level:        5
>> Mandatory:    -
>> Multiple:     -
>> ID:           [55][A8]
>> Default:      2
>> Description:  (0: Reserved, 1: ITU-R BT.709, 2: Unspecified, 4: ITU-R BT.470M,
>>                5: ITU-R BT.470BG, 6: SMPTE 170M, 7: SMPTE 240M, 8: FILM,
>>                9: ITU-R BT.2020, 10: SMPTE ST 428-1)
> 
> Same remark as with Matrix.
> 
>> 
>> [...]
>> 
> 
> 
> [1] https://github.com/FFmpeg/FFmpeg/commit/c3cd6dd106b1381933e2f24898eeec0d8aa17746 <https://github.com/FFmpeg/FFmpeg/commit/c3cd6dd106b1381933e2f24898eeec0d8aa17746>
> [2] http://www.arib.or.jp/english/html/overview/std-b67.html <http://www.arib.or.jp/english/html/overview/std-b67.html>
> 
> 
> 
> So this is what I have currently:
> The parent element would be Video [E0].
> 
> 
> Element Name: Colour
> Level:        4
> ID:           [55][B0]
> Mandatory:    -
> Multiple:     -
> Default:      -
> Type:         m
> Description:  Settings describing the colour format.
> 
> 
> Element Name: Matrix
> Level:        5
> ID:           [55][B1]
> Mandatory:    -
> Multiple:     -
> Default:      2
> Type:         u
> Description:  ColourMatrix of the video. See ISO/IEC 23001-8 for more
>               information on enumerations. (0: IEC 61966-2-1 (sRGB), 1: BT709,
>               2: Unspecified, 3: Reserved, 4: FCC, 5: BT470BG, 6: SMPTE 170M,
>               7: SMPTE 240M, 8: YCOCG, 9: BT2020 Non-constant Luminance,
>               10: BT2020 Constant Luminance) 
> 
> 
> Element Name: BitsPerChannel
> Level:        5
> ID:           [55][B2]
> Mandatory:    -
> Multiple:     -
> Default:      0
> Type:         u
> Description:  Number of decoded bits per channel. This number may be less for 
>               specific channels depending on the Matrix and ChromaSubsampling. A
>               value of 0 is unspecified.
> 
> 
> Element Name: ChromaSubsamplingHorz
> Level:        5
> ID:           [55][B3]
> Mandatory:    -
> Multiple:     -
> Default:      -
> Type:         u
> Description:  The amount of chrominance pixels to remove for every chrominance
>               pixel not removed horizontally.
> 
> 
> Element Name: ChromaSubsamplingVert
> Level:        5
> ID:           [55][B4]
> Mandatory:    -
> Multiple:     -
> Default:      -
> Type:         u
> Description:  The amount of chrominance pixels to remove for every chrominance
>               pixel not removed vertically.
> 
> Element Name: CbSubsamplingHorz
> Level:        5
> ID:           [55][B5]
> Mandatory:    -
> Multiple:     -
> Default:      -
> Type:         u
> Description:  The amount of Cb chrominance pixels to remove for every Cb
>               chrominance pixel not removed horizontally. This is additive with
>               ChromaSubsamplingHorz.
> 
> 
> Element Name: CbSubsamplingVert
> Level:        5
> ID:           [55][B6]
> Mandatory:    -
> Multiple:     -
> Default:      -
> Type:         u
> Description:  The amount of Cb chrominance pixels to remove for every Cb
>               chrominance pixel not removed vertically. This is additive with
>               ChromaSubsamplingVert.
> 
> 
> Element Name: ChromaSitingHorz
> Level:        5
> ID:           [55][B7]
> Mandatory:    -
> Multiple:     -
> Default:      0
> Type:         u
> Description:  How Chroma is subsampled horizontally. (0: Unspecified, 1: Left 
>               collocated , 2: Half)
> 
> Element Name: ChromaSitingVert
> Level:        5
> ID:           [55][B8]
> Mandatory:    -
> Multiple:     -
> Default:      0
> Type:         u
> Description:  How Chroma is subsampled vertically. (0: Unspecified, 1: Top
>               collocated , 2: Half)
> 
> 
> Element Name: Range
> Level:        5
> ID:           [55][B9]
> Mandatory:    -
> Multiple:     -
> Default:      0
> Type:         u
> Description:  Clipping of the color ranges. (0: Unspecified, 1: Broadcast range,
>               2: Full range (no clipping), 3: Defined by
>               Matrix/TransferFunction)
> 
> 
> Element Name: TransferFunction
> Level:        5
> ID:           [55][BA]
> Mandatory:    -
> Multiple:     -
> Default:      2
> Type:         u
> Description:  Transfer Function. See ISO/IEC 23001-8 for more information on
>               enumerations. (0: Reserved, 1: ITU-R BT.709, 2: Unspecified,
>               3: Reserved, 4: Gamma 2.2 curve, 5: Gamma 2.8 curve,
>               6: SMPTE 170M, 7: SMPTE 240M, 8: Linear, 9: Log, 10: Log Sqrt,
>               11: IEC 61966-2-4, 12: ITU-R BT.1361 Extended Colour Gamut,
>               13: IEC 61966-2-1, 14: ITU-R BT.2020 10 bit,
>               15: ITU-R BT.2020 12 bit, 16: SMPTE ST 2084,
>               17: SMPTE ST 428-1 18: ARIB STD-B67 (HLG))
> 
> 
> Element Name: Primaries
> Level:        5
> Mandatory:    -
> Multiple:     -
> ID:           [55][BB]
> Default:      2
> Type:         u
> Description:  Values that can be represented in the CIE 1931 colour space. See
>               ISO/IEC 23001-8 for more information on enumerations.
>               (0: Reserved, 1: ITU-R BT.709, 2: Unspecified, 3: Reserved,
>               4: ITU-R BT.470M, 5: ITU-R BT.470BG, 6: SMPTE 170M, 7: SMPTE 240M,
>               8: FILM, 9: ITU-R BT.2020, 10: SMPTE ST 428-1)
> 
> 
> Element Name: MaxCLL
> Level:        5
> ID:           [55][BC]
> Mandatory:    -
> Multiple:     -
> Default:      -
> Type:         u
> Description:  Maximum brightness of a single pixel in candelas per square
>               meter (cd/m²).
> 
> 
> Element Name: MaxFALL
> Level:        5
> ID:           [55][BD]
> Mandatory:    -
> Multiple:     -
> Default:      -
> Type:         u
> Description:  Maximum brightness of a single full frame in candelas per square
>               meter (cd/m²).
> 
> 
> Element Name: MasteringMetadata
> Level:        5
> ID:           [55][D0]
> Mandatory:    -
> Multiple:     -
> Default:      -
> Type:         m
> Description:  SMPTE 2086 mastering data.
> 
> 
> Element Name: PrimaryRChromaticityX
> Level:        6
> ID:           [55][D1]
> Mandatory:    -
> Multiple:     -
> Range:        0 <= f <= 1
> Default:      -
> Type:         f
> Description:  Red X chromaticity coordinate as defined by CIE 1931.
> 
> 
> Element Name: PrimaryRChromaticityY
> Level:        6
> ID:           [55][D2]
> Mandatory:    -
> Multiple:     -
> Range:        0 <= f <= 1
> Default:      -
> Type:         f
> Description:  Red Y chromaticity coordinate as defined by CIE 1931.
> 
> 
> Element Name: PrimaryGChromaticityX
> Level:        6
> ID:           [55][D3]
> Mandatory:    -
> Multiple:     -
> Range:        0 <= f <= 1
> Default:      -
> f
> Description:  Green X chromaticity coordinate as defined by CIE 1931.
> 
> 
> Element Name: PrimaryGChromaticityY
> Level:        6
> ID:           [55][D4]
> Mandatory:    -
> Multiple:     -
> Range:        0 <= f <= 1
> Default:      -
> Type:         f
> Description:  Green Y chromaticity coordinate as defined by CIE 1931.
> 
> 
> Element Name: PrimaryBChromaticityX
> Level:        6
> ID:           [55][D5]
> Mandatory:    -
> Multiple:     -
> Range:        0 <= f <= 1
> Default:      -
> f
> Description:  Blue X chromaticity coordinate as defined by CIE 1931.
> 
> 
> Element Name: PrimaryBChromaticityY
> Level:        6
> ID:           [55][D6]
> Mandatory:    -
> Multiple:     -
> Range:        0 <= f <= 1
> Default:      -
> Type:         f
> Description:  Blue Y chromaticity coordinate as defined by CIE 1931.
> 
> 
> Element Name: WhitePointChromaticityX
> Level:        6
> ID:           [55][D7]
> Mandatory:    -
> Multiple:     -
> Range:        0 <= f <= 1
> Default:      -
> Type:         f
> Description:  White point X chromaticity coordinate as defined by CIE 1931.
> 
> 
> Element Name: WhitePointChromaticityY
> Level:        6
> ID:           [55][D8]
> Mandatory:    -
> Multiple:     -
> Range:        0 <= f <= 1
> Default:      -
> Type:         f
> Description:  White point Y chromaticity coordinate as defined by CIE 1931.
> 
> 
> Element Name: LuminanceMax
> Level:        6
> ID:           [55][D9]
> Mandatory:    -
> Multiple:     -
> Range:        0 <= f <= 9999.99
> Default:      -
> Type:         f
> Description:  Maximum luminance. Shall be represented in candelas per square
>               meter (cd/m²).
> 
> 
> Element Name: LuminanceMin
> Level:        6
> ID:           [55][DA]
> Mandatory:    -
> Multiple:     -
> Range:        0 <= f <= 999.9999
> Default:      -
> Type:         f
> Description:  Minimum luminance. Shall be represented in candelas per square
>               meter (cd/m²).
> 
> 
> I removed ChromaSubsampling and added ChromaSubsamplingHorz, ChromaSubsamplingVert, CbSubsamplingHorz, and CbSubsamplingVert.
> 
> This is how I think the elements should be written for the different subsampling types:
> 1: 4:4:4
>     - ChromaSubsamplingHorz and ChromaSubsamplingVert will not be set as there should be no chroma subsampling.
> 
> 2: 4:4:0
>   - ChromaSubsamplingHorz :not set
>   - ChromaSubsamplingVert :1
> 
> 3: 4:2:2
>   - ChromaSubsamplingHorz :1
>   - ChromaSubsamplingVert :not set
> 
> 4: 4:2:1
>   - ChromaSubsamplingHorz :1
>   - ChromaSubsamplingVert :not set
>   - CbSubsamplingHorz :1
>   - CbSubsamplingVert :not set
>   - We could remove CbSubsamplingHorz and CbSubsamplingVert if we didn't care about handling formats where the Cr and Cb channels are different sizes.
> 
> 5: 4:2:0
>   - ChromaSubsamplingHorz :1
>   - ChromaSubsamplingVert :1
> 
> 6: 4:1:1
>   - ChromaSubsamplingHorz :3
>   - ChromaSubsamplingVert :not set
> 
> 7: 4:1:0
>   - ChromaSubsamplingHorz :3
>   - ChromaSubsamplingVert :1
> 
> 8: 3:1:1
>   - ChromaSubsamplingHorz :2
>   - ChromaSubsamplingVert :not set
>   - I'm assuming the luma subsampling can be handled by PixelWidth, and DisaplyWidth.
> 
> Jerome's vertical subsampling of 4
>   - ChromaSubsamplingHorz :not set
>   - ChromaSubsamplingVert :3
> 
> 
> 
> The other issue I want to bring up is the value of "18: ARIB STD-B67 (HLG)" in TransferFunction. Unfortunately, in WebM we will need to use this value sooner than Matroska v4 will be finalized. Should I make this value much higher? Or leave at 18? I think "16: SMPTE ST 2084" and "17: SMPTE ST 428-1" will be standardized across most documents, like 1-15 are. Just not sure if 18 will be HLG.
> 
> Thanks,
> Frank

I wanted to point out a recent ffmpeg-devel discussion http://ffmpeg.org/pipermail/ffmpeg-devel/2016-October/201357.html <http://ffmpeg.org/pipermail/ffmpeg-devel/2016-October/201357.html> pertaining to the implementation of these elements. It asks "Could someone more familiar with the current standardization process chime in to confirm if these Colour elements are finalized or not?" IIUC correctly some of these elements are already integrated into FFmpeg and they are already in use in webm via Google. Is there further work needed on these?
Dave Rice