Subject: [Cellar] adding unknownsizeallowed attribute for EBML definitions

From: Dave Rice <[email-masked]>
Date: Sat, 13 Feb 2016 16:36:56 -0500

Current the EBML spec implies that any Master-element may use an unknown Element Data Size and may contain data that is not part of an EBML Element. I propose that this should be constrained in EBML Element definitions.

I drafted another attribute for EBML Element definitions called unknownsizeallowed (better names welcome).

The proposed attribute is not required and I drafted this definition:
> A boolean to express if an EBML Element MAY use an Element Data Size that is Unknown (having all VINT\_DATA bits set to 1). The `unknownsizeallowed` attribute MAY only be applied to Master-elements. If the `unknownsizeallowed` attribute is not used it is assumed that the element not allowed to use an unknown Element Data Size.

With the added definition, I also changed a sentence in the definition of the Master-element Data Type from:
> When EBML is used in transmission or streaming, data that is not part of an EBML Element is permitted to be present within a Master-element.

to
> When EBML is used in transmission or streaming, data that is not part of an EBML Element is permitted to be present within a Master-element if `unknownsizeallowed` is enabled within that Master-element's definition.

The adjustment presumes that non-EBML junk data is only permitted in Master-elements with unknown size. Possibly these two qualities could be separated but I can’t really see a reason to have a Master-element with a known size also contain junk data.

A draft PR is here: https://github.com/Matroska-Org/ebml-specification/pull/55

Comments?

Also if using this, we’ll have to clarify which Matroska/webm elements would use this? I suspect it’s a small percentage of the Master-elements declared for those document types.

Dave Rice

---

From: Steve Lhomme <[email-masked]>
Date: Sun, 14 Feb 2016 16:26:24 +0100

>From the Matrosk point of view, I don't see exactly where this would
be used, or only to restrict the feature to certain elements. But it
seems more like a profile (live streaming or not) rather that
something to set for all Matroska files. For example an archive should
never allow such unknown sizes. This is bad for error recovery. So
should we define 2 versions of the specs ? Or profiles that allow
variations on the values? Such elements would  be marked in the specs
with possible/allowed variations.

I understand that some other EBML derivatives may have the use for the
feature only on specific elements and never some others.

> Also if using this, we’ll have to clarify which Matroska/webm elements would use this? I suspect it’s a small percentage of the Master-elements declared for those document types.

In practice only The Segments and (some) level-1 elements that can't
be held in memory or need low latency transmission.

> Dave Rice
>
> _______________________________________________
> Cellar mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cellar

-- 
Steve Lhomme
Matroska association Chairman

---

From: Dave Rice <[email-masked]>
Date: Sun, 14 Feb 2016 11:41:33 -0500

> On Feb 14, 2016, at 10:26 AM, Steve Lhomme <[email-masked]> wrote:
> 
> be used, or only to restrict the feature to certain elements.

From my understanding, most Master-elements are not intended to ever use an unknown size. For example having an unknown size for the EBML Header or ChapterTrack or Seek doesn't seem to make sense. Since the use of unknown sizes increases the challenges to parse the stream, I propose that the EBML Schema of the docType could specify where the unknown size is allowed.

> But it
> should we define 2 versions of the specs ?

This is feasible on another layer, but I'm distinguishing:
- saying that unknown sizes are only valid for certain defined elements (can be clarified by the element definition)
- recommending not to use unknown sizes in certain circumstances (the role of profiles)

> Or profiles that allow

---

From: Steve Lhomme <[email-masked]>
Date: Sun, 14 Feb 2016 17:47:59 +0100

> - recommending not to use unknown sizes in certain circumstances (the role of profiles)

You're right. The specs should say where it's allowed (mostly not
allowed everywhere) and then profiles could restrict the use even
more. So "Allowed" in the name is good.

---

From: Dave Rice <[email-masked]>
Date: Sun, 14 Feb 2016 12:07:48 -0500

> On Feb 14, 2016, at 11:47 AM, Steve Lhomme <[email-masked]> wrote:
> 

Based on comments from Sebastian and Steve I updated the definition and force pushed the changes to the pull request at https://github.com/Matroska-Org/ebml-specification/pull/55/files#diff-01f3b0f2fbf3e44327e6072f64ddc8a0 <https://github.com/Matroska-Org/ebml-specification/pull/55/files#diff-01f3b0f2fbf3e44327e6072f64ddc8a0>.

Here is the updated definition for unknownsizeallowed (note it is optional, and I suggest that all elements are considered to disallow unknown-size unless unknownsizeallowed is enabled:

> A boolean to express if an EBML Element MAY be used as an `Unknown-Sized Element` (having all VINT\_DATA bits of Element Data Size set to 1). The `unknownsizeallowed` attribute MAY only be applied to Master-elements. If the `unknownsizeallowed` attribute is not used it is assumed that the element is not allowed to use an unknown Element 

Also in the section on Element Data Size, I changed this:
> The end of a Master-element with unknown size is determined by the beginning of the next element that is not a valid sub-element of that Master-element.

to
> Master-elements MAY only use an unknown size is the unknownsizeallowed attribute of the EBML Schema is set to true. The end of a Master-element with unknown size is determined by the beginning of the next element that is not a valid sub-element of that Master-element. An Element with an unknown Element Data Size is referred to as an Unknown-Sized Element.

When we get to implementing Matroska spec, I can do a survey on what master-elements have been seen to implement unknown sizes.

Data Size.

---

From: Steve Lhomme <[email-masked]>
Date: Sun, 14 Feb 2016 18:27:35 +0100

2016-02-14 18:07 GMT+01:00 Dave Rice <[email-masked]>:
> Current the EBML spec implies that any Master-element may use an unknown
> Element Data Size and may contain data that is not part of an EBML Element.
> I propose that this should be constrained in EBML Element definitions.
>
> I drafted another attribute for EBML Element definitions called
> unknownsizeallowed (better names welcome).
>
> A boolean to express if an EBML Element MAY use an Element Data Size that is
> Unknown (having all VINT\_DATA bits set to 1). The `unknownsizeallowed`
> attribute MAY only be applied to Master-elements. If the

How about "only applies to Master-elements" ? And/or "the value MUST
be no for elements that are not Master-elements" ?

> `unknownsizeallowed` attribute is not used it is assumed that the element
> not allowed to use an unknown Element Data Size.
>
>
> When EBML is used in transmission or streaming, data that is not part of an
> EBML Element is permitted to be present within a Master-element.
>
>
> to
>
> When EBML is used in transmission or streaming, data that is not part of an
> EBML Element is permitted to be present within a Master-element if
> `unknownsizeallowed` is enabled within that Master-element's definition.
>
>
> The adjustment presumes that non-EBML junk data is only permitted in
> Master-elements with unknown size. Possibly these two qualities could be
> separated but I can’t really see a reason to have a Master-element with a
> known size also contain junk data.
>
> A draft PR is here:
> https://github.com/Matroska-Org/ebml-specification/pull/55
>
> Comments?
>
>
> From my understanding, most Master-elements are not intended to ever use an
> unknown size. For example having an unknown size for the EBML Header or
> ChapterTrack or Seek doesn't seem to make sense. Since the use of unknown
> sizes increases the challenges to parse the stream, I propose that the EBML
> Schema of the docType could specify where the unknown size is allowed.
>
> But it
> Also if using this, we’ll have to clarify which Matroska/webm elements would
> use this? I suspect it’s a small percentage of the Master-elements declared
> for those document types.
>
>
> Based on comments from Sebastian and Steve I updated the definition and
> force pushed the changes to the pull request at
> https://github.com/Matroska-Org/ebml-specification/pull/55/files#diff-01f3b0f2fbf3e44327e6072f64ddc8a0.
>
> Here is the updated definition for unknownsizeallowed (note it is optional,
> and I suggest that all elements are considered to disallow unknown-size
> unless unknownsizeallowed is enabled:
>
> A boolean to express if an EBML Element MAY be used as an `Unknown-Sized
> Element` (having all VINT\_DATA bits of Element Data Size set to 1). The
> `unknownsizeallowed` attribute MAY only be applied to Master-elements. If
> the `unknownsizeallowed` attribute is not used it is assumed that the
> element is not allowed to use an unknown Element
>
>
> The end of a Master-element with unknown size is determined by the beginning
> of the next element that is not a valid sub-element of that Master-element.
>
> to
>
> Master-elements MAY only use an unknown size is the unknownsizeallowed
> attribute of the EBML Schema is set to true. The end of a Master-element
> with unknown size is determined by the beginning of the next element that is
> not a valid sub-element of that Master-element. An Element with an unknown
> Element Data Size is referred to as an Unknown-Sized Element.
>
>
> When we get to implementing Matroska spec, I can do a survey on what
> master-elements have been seen to implement unknown sizes.
>
> Data Size.

---

From: Dave Rice <[email-masked]>
Date: Sun, 14 Feb 2016 12:42:30 -0500

> On Feb 14, 2016, at 12:27 PM, Steve Lhomme <[email-masked]> wrote:
> 

I used the first suggestion. We already have clarifications that unknown sizes are only allowed for Master-elements in the Element Data Size section.

Squashed and forced. Updated PR at https://github.com/Matroska-Org/ebml-specification/pull/55/files <https://github.com/Matroska-Org/ebml-specification/pull/55/files>.

---

From: Steve Lhomme <[email-masked]>
Date: Sun, 14 Feb 2016 19:10:21 +0100

2016-02-14 18:42 GMT+01:00 Dave Rice <[email-masked]>:
> I used the first suggestion. We already have clarifications that unknown
> sizes are only allowed for Master-elements in the Element Data Size section.
>
> Squashed and forced. Updated PR at
> https://github.com/Matroska-Org/ebml-specification/pull/55/files.
>

Merged.