Subject: Re: [Cellar] EBML Element Parenting

From: Dave Rice <[email-masked]>
Date: Mon, 1 Aug 2016 20:50:28 -0400

> On Jul 31, 2016, at 11:41 AM, Steve Lhomme <[email-masked]> wrote:
> 
> As we're moving towards splitting the Matroska specs in many smaller
> documents, putting not essential things in a document it becomes more
> important to define things differently than the "level" element we
> currently rely on. It makes it very difficult to define a single
> element without having to copy all its parents to explain where it
> should go in the EBML format.
> 
> I think we should set the name of the parent(s) explicitely to avoid
> that confusion. We will also not rely anymore on the order in which
> elements are defined in the specs.
> 
> I did a pull request with this addition:
> https://github.com/Matroska-Org/ebml-specification/pull/83
> 
> This change will probably mean many paragraphs need to be rewritten to
> rely less on the level system.
> 
> I will do the pull request for Matroska as well once it's settled.

Sorry to join this conversation late. I think the current state of the EBML implies two different ways of documenting nesting. The EBML Schema example simply nests the <elements> within one another. For example <https://github.com/Matroska-Org/ebml-specification/blob/master/specification.markdown#ebml-schema-example>:

<element name="Files" level="0" id="0x1946696C" type="master">
    <element name="File" level="1" id="0x6146" type="master" minOccurs=“1" maxOccurs=“unbounded">
        <element name="FileName" level="2" id="0x614E" type="utf-8” minOccurs=“1”/>
   </element>
</element>

The nesting of the EBML Schema implies that FileName occurs within File and File occurs within Files. The use of the ‘level’ attribute is redundant to the structure.

However, later we list out the elements of the EBML Header in a series of tables <https://github.com/Matroska-Org/ebml-specification/blob/master/specification.markdown#ebml-header-elements>. The tables don’t imply the nested structure as the xml does, so the level attribute here is needed, though it is certainly a bit unclear since determining the parent requiring reading backwards to the last master element of a higher value.

I prefer to keep the XML analogy where elements are defined in an XML schema where the arrangement of the elements within one another defines the nesting structure. Then since we can compile our documentation the “Parent” values can be decipher from the XML-based EBML Schema. However this gets to a conversation at the CELLAR working group meeting on whether XML should be considered normative or not.

So I think I would prefer not to have a ‘parent’ attribute but to allow that to be implied by the nesting of elements within an EBML Schema. Then the parent data could be presented for users when we compile documentation from the EBML Schema. For instance this markdown is build from the EBML Schema and lists parent values derived from the structure of the EBML Schema. https://gist.github.com/dericed/dd1710e47dd3b6779463a85e4fc0f788

Dave Rice

---

From: Steve Lhomme <[email-masked]>
Date: Fri, 5 Aug 2016 17:09:43 +0200

2016-08-02 2:50 GMT+02:00 Dave Rice <[email-masked]>:
> Sorry to join this conversation late. I think the current state of the EBML
> implies two different ways of documenting nesting. The EBML Schema example
> simply nests the <elements> within one another. For example:
>
> The nesting of the EBML Schema implies that FileName occurs within File and
> File occurs within Files. The use of the ‘level’ attribute is redundant to
> the structure.

Yes it's redundant and should probably be removed. It only creates
possible issues and gives no extra information. Except when it's 1+
which doesn't have any equivalent in XML and is currently not defined.

> However, later we list out the elements of the EBML Header in a series of
> tables. The tables don’t imply the nested structure as the xml does, so the
> level attribute here is needed, though it is certainly a bit unclear since
> determining the parent requiring reading backwards to the last master
> element of a higher value.
>
> I prefer to keep the XML analogy where elements are defined in an XML schema
> where the arrangement of the elements within one another defines the nesting
> structure. Then since we can compile our documentation the “Parent” values
> can be decipher from the XML-based EBML Schema. However this gets to a
> conversation at the CELLAR working group meeting on whether XML should be
> considered normative or not.

While I support having an up to date machine readable version of the
specs. I don't think it should/will be the actual reference for the
specifications.

Also the XML parent analogy only works if there's a fixed parent and
the element can only be found at that level. How do you define an
element that can be found anywhere under /File/Attribute/* (to use the
path system) (or elements similar to Void and CRC32 which can be
defined for a certain DocType) ? Either we add an attibute in the XML
element or we define how level with a + should be interpreted. Since
it's extra information it's probably better to put it out of the level
(which should go from the XML anyway). How would you define an element
that be put anywhere within a DocType in the XML format ? I agree that
it would be odd to have a "parent" attribute in a non flat XML file.
It's also odd to have a "parent" on a flat XML file.

> So I think I would prefer not to have a ‘parent’ attribute but to allow that
> to be implied by the nesting of elements within an EBML Schema. Then the
> parent data could be presented for users when we compile documentation from
> the EBML Schema. For instance this markdown is build from the EBML Schema
> and lists parent values derived from the structure of the EBML Schema.
> https://gist.github.com/dericed/dd1710e47dd3b6779463a85e4fc0f788
>
> Dave Rice
>

-- 
Steve Lhomme
Matroska association Chairman

---

From: Steve Lhomme <[email-masked]>
Date: Fri, 5 Aug 2016 17:13:13 +0200

> It's also odd to have a "parent" on a flat XML file.

Also there's still the problem how to define a (set of) element(s)
outside of the core Matroska speficications. I can't be the core
specifications including an external file. It should be that external
(XML) that tells where it should be inserted. Is there a common XML
mechanism to inject one file into another ? Does it involve defining
all the elements in the original file to define where we want the
insertion ? With possible mismatch or missing updates.

---

From: Dave Rice <[email-masked]>
Date: Fri, 5 Aug 2016 11:23:49 -0400

> On Aug 5, 2016, at 11:09 AM, Steve Lhomme <[email-masked]> wrote:
> 

It is possible in an XML Schema to define that an element may occur recursively within itself. A few methods to do this are detailed at http://lists.w3.org/Archives/Public/xmlschema-dev/2012Aug/0026.html.

> specs. I don't think it should/will be the actual reference for the

Do you mean that a machine-readable version shouldn't be the reference for human or shouldn't be the reference for tools such as mkvalidator.

> defined for a certain DocType) ?

Good point. The current EBML Schema works around this since the only global elements are in the EBML spec and inherited by but not defined by Matroska.

> Either we add an attibute in the XML
> element or we define how level with a + should be interpreted.

Another option is as it currently is where we have a recursive and global values and if either are set to true then the level attribute means that it may occur at that level or any greater level.

> Since

That's something I think about to. The parent attribute fixes some issues but if we adopt it then perhaps we should un-nest the EBML Schema.
dave

> _______________________________________________
> Cellar mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cellar

---

From: Dave Rice <[email-masked]>
Date: Fri, 5 Aug 2016 11:26:49 -0400

> On Aug 5, 2016, at 11:13 AM, Steve Lhomme <[email-masked]> wrote:
> 
> outside of the core Matroska speficications.

You mean to define a custom extra section to Matroska?

> I can't be the core
> mechanism to inject one file into another ?

Yes, in XML Schema there is a way to define an element to contain xsd:any meaning that any elements may occur within it. Within that section the sub-element should ideally have a namespace that points to another XML Schema that defines that particular section.

> Does it involve defining

It involves the parent format defining where the flexibility can occur so that any valid XML is allowed in a defined area and then the contents of that optionally being defined by another schema.
dave

---

From: Steve Lhomme <[email-masked]>
Date: Fri, 5 Aug 2016 17:30:08 +0200

> Do you mean that a machine-readable version shouldn't be the reference for human or shouldn't be the reference for tools such as mkvalidator.

No programs should use the machine-readable version. So we have to
make sure this machine-readable file generates the specs or vice
versa.

> Another option is as it currently is where we have a recursive and global values and if either are set to true then the level attribute means that it may occur at that level or any greater level.

That sounds hackish. If you want that special element that can be
found in /File/Attributes/* to be a Master element you might be able
to find it inside itself. So it would be marked as recursive. A
separate attribute is safer IMO.

> That's something I think about to. The parent attribute fixes some issues but if we adopt it then perhaps we should un-nest the EBML Schema.

Likely, yes. I suppose for XML tools it's easier to handle a hierarchy
when it's already there. But the current specdata.xml is flat it's not
a big deal (at least in my code).

---

From: Steve Lhomme <[email-masked]>
Date: Fri, 5 Aug 2016 17:34:03 +0200

> You mean to define a custom extra section to Matroska?

Yes, outside of the original spec. For example if a company does like
DivX and creates a set of new elements. We don't want to update the
original Matroska specs for that.

> It involves the parent format defining where the flexibility can occur so that any valid XML is allowed in a defined area and then the contents of that optionally being defined by another schema.

That does not sound very practical/flexible in our case. Or we'd have
to define all Master elements with such a system. And that would have
to be done for all EBML-based formats.

---

From: Steve Lhomme <[email-masked]>
Date: Tue, 23 Aug 2016 20:35:32 +0200

I updated the pull request with something that merges with the current master.

Anyone disagree with the move before I merge it ?

The next steps after that:
- unflatten the XML definition
- remove the level altogether, including all text references and in
the Matroska specs.

---

From: Dave Rice <[email-masked]>
Date: Tue, 23 Aug 2016 14:39:24 -0400

> On Aug 23, 2016, at 2:35 PM, Steve Lhomme <[email-masked]> wrote:
> 

I've been thinking about it and have slowly starting thinking that this is a better approach than I first thought. It does deviate from the XML Schema analogy but maybe the complexity of XML Schemas aren't for the best here. You're right that it would be easier to add extensions (divx) with this approach. It also resolves the issue with xml being normative, since the text and xml versions would be more semantically identical without depending on xml nesting.

> - unflatten the XML definition

What do you mean?

> - remove the level altogether, including all text references and in

Yes.
Dave

---

From: Steve Lhomme <[email-masked]>
Date: Tue, 23 Aug 2016 20:41:51 +0200

>> - unflatten the XML definition

<EBMLSchema docType="files-in-ebml-demo" version="1">
 <!-- Root Element-->
 <element name="Files" parent="/" id="0x1946696C" type="master">
  <documentation lang="en" type="definition">Container of data and
  attributes representing one or many files.</documentation>
 </element>
 <element name="File" parent="/Files" id="0x6146" type="master" minOccurs="1"
  maxOccurs="unbounded">
   <documentation lang="en" type="definition">An attached file.</documentation>
 </element>
 <element name="FileName" parent="/Files/File" id="0x614E" type="utf-8"
   minOccurs="1">
  <documentation lang="en" type="definition">Filename of the attached
  file.</documentation>
 </element>
 <element name="MimeType" parent="/Files/File" id="0x464D" type="string"
   minOccurs="1">
  <documentation lang="en" type="definition">MIME type of the
  file.</documentation>
 </element>
 <element name="ModificationTimestamp" parent="/Files/File" id="0x4654"
   type="date" minOccurs="1">
  <documentation lang="en" type="definition">Modification timestamp of
  the file.</documentation>
 </element>
 <element name="Data" parent="/Files/File" id="0x4664" type="binary"
   minOccurs="1">
  <documentation lang="en" type="definition">The data of the
  file.</documentation>
 </element>
</EBMLSchema>

No more hierarchy that could potentially interfere with the one
defined in "parent". Also as discussed before elements like Void have
no real hierarchy.

---

From: Dave Rice <[email-masked]>
Date: Tue, 23 Aug 2016 15:00:07 -0400

> On Aug 23, 2016, at 2:41 PM, Steve Lhomme <[email-masked]> wrote:
> 

Ah, I would have considered that action as 'flattening', but I support the example that we'll use parent rather than nested elements.

> defined in "parent". Also as discussed before elements like Void have

+1
Dave