Subject: Re: [Cellar] 360/VR videos in Matroska

From: Dave Rice <[email-masked]>
Date: Thu, 6 Oct 2016 00:19:16 -0400

> On Sep 4, 2016, at 8:46 AM, Steve Lhomme <[email-masked]> wrote:
> 
> 2016-08-30 18:40 GMT+02:00 Michael Bradshaw <[email-masked]>:
>> Matroska currently doesn't have a way to specify projection information for
>> 360/VR videos. Google's Spherical Video V2 RFC draft was just updated a few
>> days ago to include some new elements specifying projection and viewer pose
>> information[1].
>> 
>> These proposed elements haven't made their way over to the WebM
>> specification (and I haven't asked the WebM team if they intend to
>> incorporate them), but I think it raises a good question about Matroska
>> support for these kinds of things.
>> 
>> I don't know of any past work in specifying projection information in
>> Matroska. Assuming there hasn't been any, is there interest in adding it
>> (I'm assuming the answer is still yes, from those I asked in Berlin)? Do the
> 
> Since I'm currently adding 360° support in VLC the answer is a
> definite YES. So far we only have MP4 sources. For now we only support
> some XML embedded in an MP4 UUID XML360 box. The XML is similar to the
> one defined here:
> https://github.com/google/spatial-media/blob/master/docs/spherical-video-rfc.md#user-content-appendix-1---global-metadata-sample
> 
> There's already hardware that saves directly to this format.
> 
>> elements from the Spherical Video V2 RFC draft look like good candidates?
> 
> It's a good start ;)
> 
> I think the 0.0 default value should be 0x0p+1.
> 
> Also I think the ProjectionType value 0 should be None corresponding
> to the current situation when the field doesn't exist.
> 
> Are the Yaw/Pitch/Roll used in practice ?;

I also suggest documenting `range` values for Elements where it is relevant: ProjectionType, ProjectionPoseYaw, ProjectionPosePitch, ProjectionPoseRoll. Also the options for binary data within ProjectionPrivate could use references.

>> (For the record, I'm just breaking the ice here and getting the conversation
>> started; this isn't a proposal to adopt the new elements just quite yet)
>> 
>> [1]:
>> https://github.com/google/spatial-media/blob/master/docs/spherical-video-v2-rfc.md#webm-matroska
>> 
>> _______________________________________________
>> Cellar mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/cellar
>> 
> 
> 
> 
> -- 
> Steve Lhomme
> Matroska association Chairman
> 
> _______________________________________________
> Cellar mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cellar

---

From: Dave Rice <[email-masked]>
Date: Thu, 13 Oct 2016 22:34:00 -0400

> On Aug 30, 2016, at 12:40 PM, Michael Bradshaw <[email-masked]> wrote:
> 
> Matroska currently doesn't have a way to specify projection information for 360/VR videos. Google's Spherical Video V2 RFC draft was just updated a few days ago to include some new elements specifying projection and viewer pose information[1].
> 
> These proposed elements haven't made their way over to the WebM specification (and I haven't asked the WebM team if they intend to incorporate them), but I think it raises a good question about Matroska support for these kinds of things.
> 
> I don't know of any past work in specifying projection information in Matroska. Assuming there hasn't been any, is there interest in adding it (I'm assuming the answer is still yes, from those I asked in Berlin)? Do the elements from the Spherical Video V2 RFC draft look like good candidates?
> 
> (For the record, I'm just breaking the ice here and getting the conversation started; this isn't a proposal to adopt the new elements just quite yet)
> 
> [1]: https://github.com/google/spatial-media/blob/master/docs/spherical-video-v2-rfc.md#webm-matroska <https://github.com/google/spatial-media/blob/master/docs/spherical-video-v2-rfc.md#webm-matroska>
After hearing Andrew Scherkus’s presentation at Demuxed on the standardizing spatial media, I drafted a pull request at https://github.com/Matroska-Org/matroska-specification/pull/53 for consideration which adds the projection elements (from Michael’s link above) to Matroska’s EBML Schema. This pull request is dependent on another pull request at https://github.com/Matroska-Org/matroska-specification/pull/42 which is an adjustment to Matroska’s EBML Schema based on recent EBML specification updates.

Some considerations from writing this pull request:

There are several Matroska elements that describe enumerated values and the `Projection` Child Elements make use of enumeration lists as well. Currently the practice in Matroska element definitions appear to enumerate in this form “(1: description of the first option, 2: description of another option, 3: one more option)”. Perhaps it’s worthwhile to standardize how enumeration lists are expressed in the EBML Schema, so that rather than:

<element name="ProjectionType" path="1*1(\Segment\Tracks\TrackEntry\Video\ProjectionType)" id="0x7671" type="uinteger" minOccurs="1" minver="4" default="0" range="0-3" webm="1">
  <documentation lang="en">Describes the projection used for this video track. Valid values: (0: Rectangular, 1: Equirectangular, 2: Cubemap, 3: Mesh)</documentation>
</element>

we could have something like

<element name="ProjectionType" path="1*1(\Segment\Tracks\TrackEntry\Video\ProjectionType)" id="0x7671" type="uinteger" minOccurs="1" minver="4" default="0" range="0-3" webm="1">
  <documentation lang="en">Describes the projection used for this video track.</documentation>
  <enum value="0">Rectangular</enum>
  <enum value="1">Equirectangular</enum>
  <enum value="2">Cubemap</enum>
  <enum value="3">Mesh</enum>
</element>

Is breaking down enumeration into a structure like this helpful or over-engineering?

Also although I see the documentation for Projection in webm, but is there a comparable set of documentation for ambisonics in webm? The draft at https://github.com/google/spatial-media/blob/9bc4691f748a553a06b4b297f05ec2a62e53da1b/docs/spatial-audio-rfc.md only references mp4.

Dave Rice

---

From: Aaron Colwell <[email-masked]>
Date: Fri, 14 Oct 2016 16:21:08 +0000

On Thu, Oct 13, 2016 at 7:34 PM Dave Rice <[email-masked]> wrote:

> Matroska currently doesn't have a way to specify projection information
> for 360/VR videos. Google's Spherical Video V2 RFC draft was just updated a
> few days ago to include some new elements specifying projection and viewer
> pose information[1].
>
> (I'm assuming the answer is still yes, from those I asked in Berlin)? Do
> the elements from the Spherical Video V2 RFC draft look like good
> candidates?
>
> (For the record, I'm just breaking the ice here and getting the
> conversation started; this isn't a proposal to adopt the new elements just
> quite yet)
>
> [1]:
> After hearing Andrew Scherkus’s presentation at Demuxed on the
> standardizing spatial media, I drafted a pull request at
> https://github.com/Matroska-Org/matroska-specification/pull/53 for
> consideration which adds the projection elements (from Michael’s link
> above) to Matroska’s EBML Schema. This pull request is dependent on another
> pull request at
> https://github.com/Matroska-Org/matroska-specification/pull/42 which is
> an adjustment to Matroska’s EBML Schema based on recent EBML specification
> updates.
>
> There are several Matroska elements that describe enumerated values and
> the `Projection` Child Elements make use of enumeration lists as well.
> Currently the practice in Matroska element definitions appear to enumerate
> in this form “(1: description of the first option, 2: description of
> another option, 3: one more option)”. Perhaps it’s worthwhile to
> standardize how enumeration lists are expressed in the EBML Schema, so that
> rather than:
>
> <element name="ProjectionType"
> path="1*1(\Segment\Tracks\TrackEntry\Video\ProjectionType)" id="0x7671"
> type="uinteger" minOccurs="1" minver="4" default="0" range="0-3" webm="1">
>   <documentation lang="en">Describes the projection used for this video
> track. Valid values: (0: Rectangular, 1: Equirectangular, 2: Cubemap, 3:
> Mesh)</documentation>
> </element>
>
> <element name="ProjectionType"
> path="1*1(\Segment\Tracks\TrackEntry\Video\ProjectionType)" id="0x7671"
> type="uinteger" minOccurs="1" minver="4" default="0" range="0-3" webm="1">
> Also although I see the documentation for Projection in webm, but is there
> a comparable set of documentation for ambisonics in webm? The draft at
> https://github.com/google/spatial-media/blob/9bc4691f748a553a06b4b297f05ec2a62e53da1b/docs/spatial-audio-rfc.md
> only references mp4.
>
> Dave Rice
>

Thanks for doing this Dave! I just let Andrew know that you are helping us
move this forward. :) We have not created a Matroska equivalent of the
ISOBMFF ambisonics stuff yet, but we definitely would like to nail that
down. It has been on our todo list but video had been the higher priority
lately. I'll chat with Dillon Cower, the primary maintainer of the spatial
audio spec, about this today. I'm assuming this list would be the preferred
venue for the spatial audio discussion.

Aaron

---

From: Steve Lhomme <[email-masked]>
Date: Sun, 16 Oct 2016 16:26:28 +0200

2016-10-14 4:34 GMT+02:00 Dave Rice <[email-masked]>:
> After hearing Andrew Scherkus’s presentation at Demuxed on the standardizing
> spatial media, I drafted a pull request at
> consideration which adds the projection elements (from Michael’s link above)
> to Matroska’s EBML Schema. This pull request is dependent on another pull
> request at https://github.com/Matroska-Org/matroska-specification/pull/42
> which is an adjustment to Matroska’s EBML Schema based on recent EBML
> specification updates.
>
> There are several Matroska elements that describe enumerated values and the
> `Projection` Child Elements make use of enumeration lists as well. Currently
> the practice in Matroska element definitions appear to enumerate in this
> form “(1: description of the first option, 2: description of another option,
> 3: one more option)”. Perhaps it’s worthwhile to standardize how enumeration
> lists are expressed in the EBML Schema, so that rather than:
>
> track.</documentation>

No I think it's really worth. There could be a documentation for each
element, and in various languages.

> Also although I see the documentation for Projection in webm, but is there a
> comparable set of documentation for ambisonics in webm? The draft at

---

From: Reto Kromer <[email-masked]>
Date: Sun, 16 Oct 2016 16:55:18 +0200

Steve Lhomme wrote:

>conversation
>standardizing
>above)
>> request at
>https://github.com/Matroska-Org/matroska-specification/pull/42

+1

>https://github.com/google/spatial-media/blob/
>9bc4691f748a553a06b4b297f05ec2a62e53da1b/docs/spatial-audio-rfc.md
>> only references mp4.
>>
>> Dave Rice
>>
>

---

From: Steve Lhomme <[email-masked]>
Date: Sun, 16 Oct 2016 16:58:35 +0200

I added an "issue" so we don't forget
https://github.com/Matroska-Org/ebml-specification/issues/114

2016-10-16 16:55 GMT+02:00 Reto Kromer <[email-masked]>:
>>> On Aug 30, 2016, at 12:40 PM, Michael Bradshaw <[email-masked]>
>>
>> wrote:
>>>
>>>
>>> Matroska currently doesn't have a way to specify projection
>>
>> information for
>>>
>>> 360/VR videos. Google's Spherical Video V2 RFC draft was just updated
>>
>> a few
>>>
>>> days ago to include some new elements specifying projection and viewer
>>
>> pose
>>>
>>> information[1].
>>>
>>> incorporate them), but I think it raises a good question about
>>
>> Matroska
>>>
>>> Matroska. Assuming there hasn't been any, is there interest in adding
>>
>> it
>>>
>>> (I'm assuming the answer is still yes, from those I asked in Berlin)?
>>
>> Do the
>>>
>>> elements from the Spherical Video V2 RFC draft look like good
>>
>> candidates?
>>>
>>>
>>> started; this isn't a proposal to adopt the new elements just quite
>>
>> yet)
>>>
>>>
>>> [1]:
>>>
>>> to Matroska’s EBML Schema. This pull request is dependent on another
>>
>> pull
>>>
>>> request at
>>
>>> There are several Matroska elements that describe enumerated values
>>
>> and the
>>>
>>> `Projection` Child Elements make use of enumeration lists as well.
>>
>> Currently
>>>
>>> the practice in Matroska element definitions appear to enumerate in
>>
>> this
>>>
>>> form “(1: description of the first option, 2: description of another
>>
>> option,
>>>
>>> 3: one more option)”. Perhaps it’s worthwhile to standardize how
>>
>> enumeration
>>>
>>> path="1*1(\Segment\Tracks\TrackEntry\Video\ProjectionType)"
>>
>> id="0x7671"
>>>
>>> type="uinteger" minOccurs="1" minver="4" default="0" range="0-3"
>>
>> webm="1">
>>>
>>>   <documentation lang="en">Describes the projection used for this
>>
>> video
>>>
>>> track. Valid values: (0: Rectangular, 1: Equirectangular, 2: Cubemap,
>>
>> 3:
>>>
>>> Mesh)</documentation>
>>> </element>
>>>
>>> path="1*1(\Segment\Tracks\TrackEntry\Video\ProjectionType)"
>>
>> id="0x7671"
>>>
>>> type="uinteger" minOccurs="1" minver="4" default="0" range="0-3"
>>
>> webm="1">
>>>
>>>   <documentation lang="en">Describes the projection used for this
>>
>> video
>>>
>>> track.</documentation>
>>> Also although I see the documentation for Projection in webm, but is
>>
>> there a
>>>

---

From: Michael Bradshaw <[email-masked]>
Date: Mon, 17 Oct 2016 09:32:20 -0700

Matroska currently lacks audio channel layout specification (the container
has no way to indicate which channel corresponds to which speaker for
surround-sound layouts). As is, channel layout has to be read/inferred from
audio codec-level data (whether stored in CodecPrivate or somewhere else).
For Opus and Vorbis this isn't much of a problem, but I don't know enough
about other codecs to know if some might be problematic in Matroska.

While we're considering ambisonics in Matroska, I think we should also
consider the broader channel layout issue. If we add ambisonic-related info
to Matroska, I think we also need to add surround-sound channel info to
Matroska. If we don't add surround-sound channel info to Matroska, then
perhaps we shouldn't for ambisonics either and instead require the codec to
determine the channel order (e.g.
https://tools.ietf.org/html/draft-ietf-codec-ambisonics).

Generally I think it might be nice to have this information in Matroska,
but there are *tons* of ways to arrange audio channels and I'm concerned
about specification bloat.

On Sun, Oct 16, 2016 at 7:58 AM, Steve Lhomme <[email-masked]> wrote:

> >> https://github.com/google/spatial-media/blob/master/
> docs/spherical-video
> >> -v2-rfc.md#webm-matroska
> >>>
> >>>
> >>>

---

From: Steve Lhomme <[email-masked]>
Date: Sun, 30 Oct 2016 10:10:19 +0100

Hello Michael,

This issue was already raised in this thread:
https://mailarchive.ietf.org/arch/search/?email_list=cellar&gbt=1&index=HzxRVJouLAtFitvTMWF_ZbHuoKE

The conclusion was... unconclusive. So far we probably lean towards
borrowing the channel mapping from AVI/Microsoft. For ambisonic I'm
not sure that's sufficient though. Also this really makes sense for
PCM/raw codecs. Other ones should have their own internal mapping as
they usually use some channel coupling for better efficiency. Maybe
lossless codecs could also lack channel mapping and rely on the
container.

2016-10-17 18:32 GMT+02:00 Michael Bradshaw <[email-masked]>:
> Generally I think it might be nice to have this information in Matroska, but
> there are *tons* of ways to arrange audio channels and I'm concerned about
> specification bloat.
>