Subject: Re: [Cellar] EBML Schema

From: Dave Rice <[email-masked]>
Date: Wed, 3 Feb 2016 01:25:39 -0500

> On Jan 23, 2016, at 2:49 PM, Dave Rice <[email-masked]> wrote:
> 
> 
>> On Jan 17, 2016, at 8:05 AM, Steve Lhomme <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> 2016-01-15 16:35 GMT+01:00 Dave Rice <[email-masked] <mailto:[email-masked]>>:
>>> 
>>>> On Jan 15, 2016, at 7:44 AM, Steve Lhomme <[email-masked] <mailto:[email-masked]>> wrote:
>>>> 
>>>> 2016-01-15 7:27 GMT+01:00 Dave Rice <[email-masked] <mailto:[email-masked]>>:
>>>>> 
>>>>> On Dec 1, 2015, at 10:40 AM, Dave Rice <[email-masked] <mailto:[email-masked]>> wrote:
>>>>> 
>>>>> On Nov 9, 2015, at 1:19 PM, Dave Rice <[email-masked] <mailto:[email-masked]>> wrote:
>>>>> 
>>>>> On Oct 3, 2015, at 9:46 AM, Steve Lhomme <[email-masked] <mailto:[email-masked]>> wrote:
>>>>> 
>>>>> On Aug 28, 2015 17:00, "Dave Rice" <[email-masked] <mailto:[email-masked]>> wrote:
>>>>> 
>>>>>>> On Aug 28, 2015, at 2:50 AM, Moritz Bunkus <[email-masked] <mailto:[email-masked]>> wrote:
>>>>>>> 
>>>>>>> Hey,
>>>>>>> 
>>>>>>> I have no objections, however I don't know a lot about XML schemas in
>>>>>>> the first place (neither about DTDs, to be honest).
>>>>>> 
>>>>>> 
>>>>>> Honestly, I know a lot more about XML Schemas than I do about DTDs. As
>>>>>> wikipedia mentions at
>>>>>> https://en.wikipedia.org/wiki/Document_type_definition <https://en.wikipedia.org/wiki/Document_type_definition>, DTDs have largely
>>>>>> been superseded by XML Schemas. And at this point I think that XML Schemas
>>>>>> may be a more familiar analogy to use.
>>>>>> 
>>>>>> I think XML Schemas also share more in common with specdata.xml than DTDs
>>>>>> do. Schemas use the <element> node and have maxOccurs and minOccurs
>>>>>> attributes (specdata has semantically the same thing with mandatory and
>>>>>> multiple), they both have a similar declaration of element type, element
>>>>>> name and element description. Actually I think a semantically equivalent
>>>>>> version of specdata.xml could be written as an XML Schema.
>>>>>> 
>>>>>> XML Schemas also offer a few advantages for machine readable expressions;
>>>>>> for instance XML Schemas can mandate a particular pattern or regex for a
>>>>>> value.
>>>>>> 
>>>>>>>> I propose the specdata.xml file here
>>>>>>>> 
>>>>>>>> https://github.com/Matroska-Org/foundation-source/blob/master/spectool/specdata.xml <https://github.com/Matroska-Org/foundation-source/blob/master/spectool/specdata.xml>
>>>>>>>> 
>>>>>>>> <https://github.com/Matroska-Org/foundation-source/blob/master/spectool/specdata.xml>
>>>>>>>> is a good basis for the consideration of an EBML Schema. From what I
>>>>>>>> can see, specdata.xml is an expression of the EBML + Matroska
>>>>>>>> specifications to support automated creation of documentation, but the
>>>>>>>> structure of this already shares a lot of similarity to XML Schemas.
>>>>>>> 
>>>>>>> 
>>>>>>> For both documentation (e.g. the table on the matroska.org <http://matroska.org/> specs page is
>>>>>>> generated from this file) and code (libMatroska's class hierarchy is
>>>>>>> generated automatically from this file) actually.
>>>>>> 
>>>>>> 
>>>>>> Does specdata.xml play a role in mkvalidate? I'm thinking of the potential
>>>>>> to have an ebmlvalidator where you can provide the EBML Schema to validate
>>>>>> particular EBML docType.
>>>>> 
>>>>> Well the parsing code is generated from the XML file, so in a way, yes. But
>>>>> it's not parsed "live".
>>>>> 
>>>>>>>> Is there a preference in handling the standardization of Matroska:
>>>>>>>> documenting it in a similar fashion to our work in the EBML spec or to
>>>>>>>> define what an EBML Schema is and consider matroska an expression of
>>>>>>>> it?
>>>>>>> 
>>>>>>> 
>>>>>>> I'm not sure whether or not I understand the implications. But my gut
>>>>>>> feeling is that having a definition for an EBML Schema would benefit
>>>>>>> other formats than Matroska, too, therefore the latter seems the way to
>>>>>>> go.
>>>>>> 
>>>>>> 
>>>>>> I have the same feeling:
>>>>>> - document EBML as a specification that includes rules for defining a
>>>>>> docType in the form of an EBML Schema
>>>>>> - write an EBML Schema (updated specdata.xml) for Matroska and maybe webM
>>>>>> 
>>>>>>>> Are some changes to specdata.xml acceptable? Such as a filename change
>>>>>>>> or changing the name of the <table> element of some attributes?
>>>>>>> 
>>>>>>> 
>>>>>>> Well, like I said above the specdata.xml is used for generating both
>>>>>>> documentation and code. Both should stay viable. If changes to it are
>>>>>>> made then the accompanying tools must be updated as well.
>>>>>>> 
>>>>>>>> Neither the current EBML specs nor the specdata.xml specifically refer
>>>>>>>> to the hierarchical arrangement of the elements, but this could be
>>>>>>>> presumed by their ordering. For instance, could any level 3 element be
>>>>>>>> a child of any level 2 Master-element? I presume not, but I don't
>>>>>>>> think it's clear anywhere what parent-child relationships are
>>>>>>>> feasible. Possibly specdata.xml and/or the EBML Schema Definition
>>>>>>>> could define the relationship between levels of related elements
>>>>>>>> similar to how an XML Schema (XSD) does.
>>>>>>> 
>>>>>>> 
>>>>>>> So far it is understood that an element not marked as a global element
>>>>>>> must only occur as a child of its parent. Its parent is the last element
>>>>>>> located before the child element in the specdata file with a lower level
>>>>>>> than the child element. Or something like that.
>>>>>> 
>>>>>> 
>>>>>> This will need some documentation. That's how I've understood the mkv spec
>>>>>> as well but the definition for how an EBML Schema works should be explicit
>>>>>> about this.
>>>>> 
>>>>> Any more opinion about how to go about (or if to go about) modifying
>>>>> specdata.xml towards becoming an expression of a to-be-defined EBML Schema
>>>>> for matroska and webm? As a summary of proposed changes to specdata.xml
>>>>> 
>>>>> - change to XML Schema conventions where relevant:
>>>>> - use maxOccurs attribute instead of the current Multiple attribute.
>>>>> - use minOccurs attribute instead of the current Mandatory attribute.
>>>>> - move documentation of elements to a sub-element (allows for possible
>>>>> internationalization in the schema and better semantics)
>>>>> - arrange elements in hierarchical form to indicate parent-child
>>>>> relationships (rather than the current practices where all elements are
>>>>> defined at the same level, and you have to parse back in elements to the one
>>>>> with the lower-numbered level attribute to find the parent)
>>>>> 
>>>>> A draft of specdata.xml with these changes is at
>>>>> https://gist.github.com/dericed/f0a4bb0e7dc635ed1347 <https://gist.github.com/dericed/f0a4bb0e7dc635ed1347>. I can continue to work
>>>>> on this and send back changes for advice/approval but if I do so is there
>>>>> someone who could later update the tools that use specdata.xml so that
>>>>> newly-defined EBML Schemas may later to be into use?
>>>>> 
>>>>> 
>>>>> I’m preparing a pull request on specdata.xml but want to update the
>>>>> utilities in spectool at the same time so that spec2data, data2lib, and
>>>>> data2spec still function properly. I’m having trouble getting the spectool
>>>>> utilities to build properly so that I can test them. I was able to build
>>>>> coremake but not sure where to go from here. I’ve read spec2data and have an
>>>>> idea of how it works and am thinking that I could reproduce the spec2data
>>>>> workflow with xsl and then we could have a make file which uses xsltproc to
>>>>> convert an EBML Schema into the Drupal table and library files as needed.
>>>>> Any advice on which route to take: continue trying with getting spectool
>>>>> utilities to build or redo the utilities in xsl?
>>>>> 
>>>>> 
>>>>> I’d like to revive this thread to propose some changes in the definition of
>>>>> the EBML Schema. The EBML Schema is intended to be a machine-readible
>>>>> document that details the semantics and structure of an EBML Document Type.
>>>>> The EBML Schema for a Document Type is essential for understanding the
>>>>> semantics and validating an EBML Document. This proposal builds upon an
>>>>> earlier conversation started last year on matroska-devel,
>>>>> http://lists.matroska.org/pipermail/matroska-devel/2015-August/004765.html <http://lists.matroska.org/pipermail/matroska-devel/2015-August/004765.html>,
>>>>> http://lists.matroska.org/pipermail/matroska-devel/2015-October/004807.html <http://lists.matroska.org/pipermail/matroska-devel/2015-October/004807.html>.
>>>>> 
>>>>> I’m open to other solutions or approaches, but since Matroska is based on
>>>>> EBML which itself is becoming well-defined. I think there is utility in
>>>>> documenting how to create a machine-readible document that describes an EBML
>>>>> Document Type. I think the concept of the XML Schema and XML Document is
>>>>> analogous to EBML Schema and EBML Document and that adopting from the XML
>>>>> community for documenting form may be beneficial.
>>>>> 
>>>>> Thus far we have this section
>>>>> https://github.com/Matroska-Org/ebml-specification/blob/3b30502d2adb6a22c60294c1e7ea68f703e07e50/specification.markdown#ebml-schema <https://github.com/Matroska-Org/ebml-specification/blob/3b30502d2adb6a22c60294c1e7ea68f703e07e50/specification.markdown#ebml-schema>
>>>>> which introduces the concept for the EBML Schema and then provides a list of
>>>>> attributes for defining Elements that make up a specific EBML Document Type
>>>>> https://github.com/Matroska-Org/ebml-specification/blob/3b30502d2adb6a22c60294c1e7ea68f703e07e50/specification.markdown#ebml-schema-element-attributes.
>>>>> This documentation currently reflects the specdata.xml file of the Matroska
>>>>> foundation-source repo, which is here
>>>>> https://github.com/Matroska-Org/foundation-source/blob/master/spectool/specdata.xml.
>>>>> The specdata.xml file is used to create the HTML table of elements on
>>>>> http://matroska.org/technical/specs/index.html and libraries used by
>>>>> mkvalidate and other files. After drafting the EBML Schema and Attribute
>>>>> documentation I found
>>>>> http://matroska.org/technical/specs/notes.html#Table_Columns which covers
>>>>> similar concepts.
>>>>> 
>>>>> I’d like to propose the following changes to the definition of EBML Schema:
>>>>> 
>>>>> - Replace <table> with <EBMLSchema>.
>>>>> 
>>>>> The node <table> makes sense in the context of the HTML table on
>>>>> matroska.org but if there will be greater machine-readible usage I think a
>>>>> rename would clarify.
>>>>> 
>>>>> - Add an XML NameSpace attribute to <EBMLSchema>.
>>>>> 
>>>>> This would add something like
>>>>> xmlns=“http://official_url_of_EBML_Schema_definition.org/EBMLSchema.xsd”.
>>>>> The EBMLSchema.xsd file would be an XML Schema that defines the EBMLSchema
>>>>> document itself. Yes, this uses an XML Schema to define the EBML Schema, but
>>>>> this gives a way to validate the EBML Schemas and docType definitions made
>>>>> by others.
>>>>> 
>>>>> - Add a “docType” attribute to <EBMLSchema>
>>>>> 
>>>>> This would clarify what the EBMLSchema is documenting, thus it would start
>>>>> with <EBMLSchema docType=“matroska”>
>>>>> 
>>>>> - Add “version” attribute to <EBMLSchema>
>>>>> 
>>>>> This would note the highest version number of the docType officially
>>>>> documented by the EBML Schema, so the EBML Schema would be inferred to
>>>>> document the noted docType from version 0 up to the version value listed
>>>>> here.
>>>> 
>>>> Maybe we need something to say it's the final version or a draft.
>>> 
>>> With XML Schemas this is done by having an entirely separate document per version. We could have separate files such as:
>>> 
>>> matroska_v1.xml
>>> matroska_v2.xml
>>> matroska_v3.xml
>>> matroska_v4.xml
>>> matroska_v5RC.xml
>>> webm_v1.xml
>>> 
>>> Or we could add an attribute called something like "unofficial" so that we may have an EBML Schema with non-standardized elements for testing.
>>> 
>>> Any preference at which tactic? An advantage of having versioned EBML Schemas (one doc per version) is that the deprecated elements are simply removed rather than an eternal lingering presence. It may accept complexity in use though since you'd have to use the proper schema that matches both the docType and docTypeVersion.
>> 
>> That makes omission by mistake a possibility, involves more proof
>> reading and it becomes harder to know which elements changed between
>> what versions. It's also harder to process to generate code. Also
>> should the documentation we currently have be duplicated on each
>> document ? I'd rather keep a single document.
>> 
>>>>> - Rename “mandatory” as “minOccurs” and rename “multiple” as “maxOccurs”
>>>> 
>>>> That may be odd when we add "repeatable" elements that must be maxOccurs = 1.
>>> 
>>> With maxOccurs it is usually set via numbers, however 'unbounded' is an accepted maxOccurs value to say that the element may repeat for an infinite amount of times. In addition to "unbounded" we could also add a value "identical" which would mean the same as "unbounded" with the added constraint that every copy be identical.
>> 
>> That sounds good, yes.
>> 
>>>>> This is to match the meaning of minOccurs and maxOccurs in XML Schema,
>>>>> https://www.w3.org/TR/xmlschema-0/#ref6 <https://www.w3.org/TR/xmlschema-0/#ref6>.
>>>>> mandatory=“0” would be equivalent to minOccurs=“0"
>>>>> mandatory=“1” would be equivalent to minOccurs=“1”
>>>>> no mandatory attribute would be equivalent of no minOccurs attribute which
>>>>> is equivalent to minOccurs=“1” (as is true with minOccurs within an XML
>>>>> Schema)
>>>>> 
>>>>> multiple=“0” would be equivalent to maxOccurs =“1"
>>>>> multiple =“1” would be equivalent to maxOccurs =“unbounded”
>>>>> no multiple attribute would be equivalent of no maxOccurs attribute which is
>>>>> equivalent to maxOccurs =“1” (as is true with maxOccurs within an XML
>>>>> Schema)
>>>>> 
>>>>> Moving from mandatory/multiple to minOccurs/maxOccurs can be done with no
>>>>> semantic differences, but we’d have to advantage of allowing for constraints
>>>>> to the number of usage that are not 0, 1, and infinite. For instance, I
>>>>> understand the logic to have 2 SeekHead Elements, but is there a scenario
>>>>> for 3 or more SeekHeads at the same Level within the same EBML Document?
>>>> 
>>>> Limiting the amount of redirection might be a good thing indeed.
>>> 
>>> Are there other elements currently listed as multiple="1" where they are intended for use greater than 1 and less than unbounded. SeekHead was the only one coming to mind.
>> 
>> Not from the top of my mind.
>> 
>>>>> - Store <element> nodes hierarchically rather than all at the same level.
>>>>> 
>>>>> Currently with in the specdata.xml file it is difficult to ascertain the
>>>>> parent of a given element. It can be done by seeking to the previous element
>>>>> where the level of the element is one value lower, but there are exceptions
>>>>> for global elements. I suggest that a structure of
>>>>> <element>
>>>>> <element>...</element>
>>>>> <element>...</element>
>>>>> </element>
>>>>> is more clear than all elements at the same level. I made an example draft
>>>>> of specdata.xml with nested implementing here:
>>>>> https://gist.github.com/dericed/f0a4bb0e7dc635ed1347 <https://gist.github.com/dericed/f0a4bb0e7dc635ed1347>.
>>>>> 
>>>>> I can provide a patch to the current EBML Schema and EBML Schema Attributes
>>>>> as well as incorporate a sample EBML Schema into the specification, but
>>>>> wanted to nudge for comments before proceeding.
>>>> 
>>>> The current flat hierarchy is not good anyway. I will need a bit of
>>>> time to update the spec tools accordingly. But the change is OK for
>>>> me.
>>> 
>>> Thanks. Should I cross-post this to webm lists?
>> 
>> No idea if they use the document but it's worth a try.
> 
> Here are some changes made to the definition of EBML Schema. I placed them into a pull request in github, https://github.com/Matroska-Org/ebml-specification/pull/51/files <https://github.com/Matroska-Org/ebml-specification/pull/51/files>, and will summarize the changes here.
> 
> I removed the attributes for mandatory, multiple, and identical and replaced them with minOccurs and maxOccurs (see in context at https://github.com/MediaArea/ebml-specification/blob/ebml-schema-refactor/specification.markdown#ebml-schema-element-attributes <https://github.com/MediaArea/ebml-specification/blob/ebml-schema-refactor/specification.markdown#ebml-schema-element-attributes>)
> 
> minOccurs:
> An integer to express the minimal number of occurrences that the EBML Element MUST occur within its Parent Element if its Parent Element is used. If the minOccurs attribute is not expressed for that Element then that Element shall be considered to have a minOccurs value of 0. This value of minOccurs MUST be a positive integer. The semantic meaning of minOccurs within an EBML Schema is considered analogous to the meaning of minOccurs within an [XML Schema](https://www.w3.org/TR/xmlschema-0/#ref6 <https://www.w3.org/TR/xmlschema-0/#ref6>).
> 
> maxOccurs
> A value to express the maximum number of occurrences that the EBML Element MUST occur within its Parent Element if its Parent Element is used. This value may be either a positive integer or the term `unbounded` to indicate there is no maximum number of occurrences or the term `identical` to indicate that the Element is an Identically-Recurring Element. If the maxOccurs attribute is not expressed for that Element then that Element shall be considered to have a maxOccurs value of 1. The semantic meaning of maxOccurs within an EBML Schema is considered analogous to the meaning of minOccurs within an [XML Schema](https://www.w3.org/TR/xmlschema-0/#ref6 <https://www.w3.org/TR/xmlschema-0/#ref6>), with EBML Schema adding the concept of Identically-Recurring Elements.
> 
> I also added a section called "Identically-Recurring Elements" to define them since they are enabled by setting maxOccurs to "enabled". See in context at https://github.com/MediaArea/ebml-specification/blob/ebml-schema-refactor/specification.markdown#identically-recurring-elements <https://github.com/MediaArea/ebml-specification/blob/ebml-schema-refactor/specification.markdown#identically-recurring-elements>.
> 
> An Identically-Recurring Element is an Element that may occur within its Parent Element more than once but that each recurrence within that Parent Element MUST be identical both in storage and semantics. Identically-Recurring Elements are permitted to be stored multiple times within the same Parent Element in order to increase data resilience and optimize the use of EBML in transmission. Identically-Recurring Elements SHOULD include a CRC-32 Element as a Child Element; this is especially recommended when EBML is used for long-term storage or transmission. If a Parent Element contains more than one copy of an Identically-Recurring Element which includes a CRC-32 Child Element then the first instance of the Identically-Recurring Element with a valid CRC-32 value should be used for interpretation. If a Parent Element contains more than one copy of an Identically-Recurring Element which does not contain a CRC-32 Child Element or if CRC-32 Child Elements are present but none are valid then the first instance of the Identically-Recurring Element should be used for interpretation.
> 
> 
> I also drafted a new section called "EBML Schema example" as a demonstration of an EBML Schema. This example EBML Schema is simply the EBML Header with some structure to store files (somewhat based on the Attachments Element of Matroska). The section can be seen in a GitHub branch here: https://github.com/MediaArea/ebml-specification/blob/ebml-schema-refactor/specification.markdown#ebml-schema-example <https://github.com/MediaArea/ebml-specification/blob/ebml-schema-refactor/specification.markdown#ebml-schema-example> or is pasted here. Compared to the existing implementation of specdata.xml, this uses hierarchical arrangement and the changes mentioned above.
> <?xml version="1.0" encoding="utf-8"?>
> <EBMLSchema docType="files-in-ebml-demo">
>   <element name="EBML" level="0" id="0x1A45DFA3" type="master" minOccurs="1" maxOccurs="1" minver="1">
>     <documentation lang="en">Set the EBML characteristics of the data to follow. Each EBML Document has to start with this.</documentation>
>     <element name="EBMLVersion" level="1" id="0x4286" type="uinteger" minOccurs="1" default="1" minver="1">
>       <documentation lang="en">The version of EBML parser used to create the EBML Document.</documentation>
>     </element>
>     <element name="EBMLReadVersion" level="1" id="0x42F7" type="uinteger" minOccurs="1" default="1" minver="1">
>       <documentation lang="en">The minimum EBML version a parser has to support to read this EBML Document.</documentation>
>     </element>
>     <element name="EBMLMaxIDLength" level="1" id="0x42F2" type="uinteger" minOccurs="1" default="4" minver="1">
>       <documentation lang="en">The EBMLMaxIDLength is the maximum length in octets of the Element IDs to be found within the EBML Body. An EBMLMaxIDLength value of four is recommended, though larger values are allowed.</documentation>
>     </element>
>     <element name="EBMLMaxSizeLength" level="1" id="0x42F3" type="uinteger" minOccurs="1" default="8" minver="1">
>       <documentation lang="en">The EBMLMaxSizeLength is the maximum length in octets of the expression of all Element Data Sizes to be found within the EBML Body. To be clear EBMLMaxSizeLength documents the maximum 'length' of all Element Data Size expressions within the EBML Body and not the maximum 'value' of all Element Data Size expressions within the EBML Body. Elements that have a Element Data Size expression which is larger in octets than what is expressed by EBMLMaxSizeLength SHALL be considered invalid.</documentation>
>     </element>
>     <element name="DocType" level="1" id="0x4282" type="string" minOccurs="1" default="matroska" minver="1">
>       <documentation lang="en">A string that describes and identifies the content of the EBML Body that follows this EBML Header.</documentation>
>     </element>
>     <element name="DocTypeVersion" level="1" id="0x4287" type="uinteger" minOccurs="1" default="1" minver="1">
>       <documentation lang="en">The version of DocType interpreter used to create the EBML Document.</documentation>
>     </element>
>     <element name="DocTypeReadVersion" level="1" id="0x4285" type="uinteger" minOccurs="1" default="1" minver="1">
>       <documentation lang="en">The minimum DocType version an interpreter has to support to read this EBML Document.</documentation>
>     </element>
>   </element>
>   <element name="Void" global="1" level="0+" id="0xEC" type="binary" minver="1">
>     <documentation lang="en">Used to void damaged data, to avoid unexpected behaviors when using damaged data. The content is discarded. Also used to reserve space in a sub-element for later use.</documentation>
>   </element>
>   <element name="CRC-32" global="1" level="1+" id="0xBF" type="binary" minver="1">
>     <documentation lang="en">The CRC is computed on all the data from the last CRC element (or start of the upper level element), up to the CRC element, including other previous CRC elements. All level 1 elements SHOULD include a CRC-32.</documentation>
>   </element>
>   <element name="Files" level="0" id="0x1946696C" type="master" minver="1">
>     <documentation lang="en">Container of data and attributes representing one or many files.</documentation>
>     <element name="File" level="1" id="0x6146" type="master" minOccurs="1" maxOccurs="unbounded" minver="1">
>       <documentation lang="en">An attached file.</documentation>
>       <element name="FileName" level="2" id="0x614E" type="utf-8" minOccurs="1">
>         <documentation lang="en">Filename of the attached file.</documentation>
>       </element>
>       <element name="MimeType" level="2" id="0x464D" type="string" minOccurs="1">
>         <documentation lang="en">MIME type of the file.</documentation>
>       </element>
>       <element name="ModificationTimestamp" level="2" id="0x4654" type="date" minOccurs="1">
>         <documentation lang="en">Modification timestamp of the file.</documentation>
>       </element>
>       <element name="Data" level="2" id="0x4664" type="binary" minOccurs="1">
>         <documentation lang="en">The data of the file.</documentation>
>       </element>
>     </element>
>   </element>
> </EBMLSchema>
> 
> If this approach seems agreeable for defining machine-readable documents of EBML DocTypes then can next create versions of them according to Matroska and webm.

Bumping this thread. The corresponding pull request is open here: https://github.com/Matroska-Org/ebml-specification/pull/51/files.
Dave Rice

---

From: Steve Lhomme <[email-masked]>
Date: Wed, 3 Feb 2016 10:38:31 +0100

2016-01-23 20:49 GMT+01:00 Dave Rice <[email-masked]>:
>
> On Jan 17, 2016, at 8:05 AM, Steve Lhomme <[email-masked]> wrote:
>
> 2016-01-15 16:35 GMT+01:00 Dave Rice <[email-masked]>:
>
>
> On Jan 15, 2016, at 7:44 AM, Steve Lhomme <[email-masked]> wrote:
>
> 2016-01-15 7:27 GMT+01:00 Dave Rice <[email-masked]>:
>
>
> On Dec 1, 2015, at 10:40 AM, Dave Rice <[email-masked]> wrote:
>
> On Nov 9, 2015, at 1:19 PM, Dave Rice <[email-masked]> wrote:
>
> On Oct 3, 2015, at 9:46 AM, Steve Lhomme <[email-masked]> wrote:
>
> On Aug 28, 2015 17:00, "Dave Rice" <[email-masked]> wrote:
>
> On Aug 28, 2015, at 2:50 AM, Moritz Bunkus <[email-masked]> wrote:
>
> Hey,
>
> https://en.wikipedia.org/wiki/Document_type_definition, DTDs have largely
> https://github.com/Matroska-Org/foundation-source/blob/master/spectool/specdata.xml
>
> For both documentation (e.g. the table on the matroska.org specs page is
> https://gist.github.com/dericed/f0a4bb0e7dc635ed1347. I can continue to work
> http://lists.matroska.org/pipermail/matroska-devel/2015-August/004765.html,
> http://lists.matroska.org/pipermail/matroska-devel/2015-October/004807.html.
>
> https://github.com/Matroska-Org/ebml-specification/blob/3b30502d2adb6a22c60294c1e7ea68f703e07e50/specification.markdown#ebml-schema
> With XML Schemas this is done by having an entirely separate document per
> version. We could have separate files such as:
>
> matroska_v1.xml
> matroska_v2.xml
> matroska_v3.xml
> matroska_v4.xml
> matroska_v5RC.xml
> webm_v1.xml
>
> Or we could add an attribute called something like "unofficial" so that we
> may have an EBML Schema with non-standardized elements for testing.
>
> Any preference at which tactic? An advantage of having versioned EBML
> Schemas (one doc per version) is that the deprecated elements are simply
> removed rather than an eternal lingering presence. It may accept complexity
> in use though since you'd have to use the proper schema that matches both
> the docType and docTypeVersion.
>
>
> With maxOccurs it is usually set via numbers, however 'unbounded' is an
> accepted maxOccurs value to say that the element may repeat for an infinite
> amount of times. In addition to "unbounded" we could also add a value
> "identical" which would mean the same as "unbounded" with the added
> constraint that every copy be identical.
>
>
> That sounds good, yes.
>
> https://www.w3.org/TR/xmlschema-0/#ref6.
> Are there other elements currently listed as multiple="1" where they are
> intended for use greater than 1 and less than unbounded. SeekHead was the
> only one coming to mind.
>
>
> https://gist.github.com/dericed/f0a4bb0e7dc635ed1347.
>
> Here are some changes made to the definition of EBML Schema. I placed them
> into a pull request in github,
> https://github.com/Matroska-Org/ebml-specification/pull/51/files, and will
> summarize the changes here.
>
> I removed the attributes for mandatory, multiple, and identical and replaced
> them with minOccurs and maxOccurs (see in context at
> https://github.com/MediaArea/ebml-specification/blob/ebml-schema-refactor/specification.markdown#ebml-schema-element-attributes)
>
> minOccurs:
> An integer to express the minimal number of occurrences that the EBML
> Element MUST occur within its Parent Element if its Parent Element is used.
> If the minOccurs attribute is not expressed for that Element then that
> Element shall be considered to have a minOccurs value of 0. This value of
> minOccurs MUST be a positive integer. The semantic meaning of minOccurs
> within an EBML Schema is considered analogous to the meaning of minOccurs
> within an [XML Schema](https://www.w3.org/TR/xmlschema-0/#ref6).
>
> maxOccurs
> A value to express the maximum number of occurrences that the EBML Element
> MUST occur within its Parent Element if its Parent Element is used. This
> value may be either a positive integer or the term `unbounded` to indicate
> there is no maximum number of occurrences or the term `identical` to
> indicate that the Element is an Identically-Recurring Element. If the
> maxOccurs attribute is not expressed for that Element then that Element
> shall be considered to have a maxOccurs value of 1. The semantic meaning of
> maxOccurs within an EBML Schema is considered analogous to the meaning of
> minOccurs within an [XML Schema](https://www.w3.org/TR/xmlschema-0/#ref6),
> with EBML Schema adding the concept of Identically-Recurring Elements.
>
> I also added a section called "Identically-Recurring Elements" to define
> them since they are enabled by setting maxOccurs to "enabled". See in
> context at
> https://github.com/MediaArea/ebml-specification/blob/ebml-schema-refactor/specification.markdown#identically-recurring-elements.
>
> An Identically-Recurring Element is an Element that may occur within its
> Parent Element more than once but that each recurrence within that Parent
> Element MUST be identical both in storage and semantics.
> Identically-Recurring Elements are permitted to be stored multiple times
> within the same Parent Element in order to increase data resilience and
> optimize the use of EBML in transmission. Identically-Recurring Elements
> SHOULD include a CRC-32 Element as a Child Element; this is especially
> recommended when EBML is used for long-term storage or transmission. If a
> Parent Element contains more than one copy of an Identically-Recurring
> Element which includes a CRC-32 Child Element then the first instance of the
> Identically-Recurring Element with a valid CRC-32 value should be used for
> interpretation. If a Parent Element contains more than one copy of an
> Identically-Recurring Element which does not contain a CRC-32 Child Element
> or if CRC-32 Child Elements are present but none are valid then the first
> instance of the Identically-Recurring Element should be used for
> interpretation.
>
>
> I also drafted a new section called "EBML Schema example" as a demonstration
> of an EBML Schema. This example EBML Schema is simply the EBML Header with
> some structure to store files (somewhat based on the Attachments Element of
> Matroska). The section can be seen in a GitHub branch here:
> https://github.com/MediaArea/ebml-specification/blob/ebml-schema-refactor/specification.markdown#ebml-schema-example
> or is pasted here. Compared to the existing implementation of specdata.xml,
> this uses hierarchical arrangement and the changes mentioned above.
>

Does the EBML header need to be defined ? Can it be included from
elsewhere ? That's prone to errors and incompatibilities if we go that
way.

>     <documentation lang="en">Set the EBML characteristics of the data to
> follow. Each EBML Document has to start with this.</documentation>
>       <documentation lang="en">The minimum EBML version a parser has to
> support to read this EBML Document.</documentation>
>     </element>
>       <documentation lang="en">The EBMLMaxIDLength is the maximum length in
> octets of the Element IDs to be found within the EBML Body. An
> EBMLMaxIDLength value of four is recommended, though larger values are
> allowed.</documentation>
>     </element>
>       <documentation lang="en">The EBMLMaxSizeLength is the maximum length
> in octets of the expression of all Element Data Sizes to be found within the
> EBML Body. To be clear EBMLMaxSizeLength documents the maximum 'length' of
> all Element Data Size expressions within the EBML Body and not the maximum
> 'value' of all Element Data Size expressions within the EBML Body. Elements
> that have a Element Data Size expression which is larger in octets than what
> is expressed by EBMLMaxSizeLength SHALL be considered
> invalid.</documentation>
>     </element>
>     <element name="DocType" level="1" id="0x4282" type="string"
> minOccurs="1" default="matroska" minver="1">
>       <documentation lang="en">A string that describes and identifies the
> content of the EBML Body that follows this EBML Header.</documentation>
>     </element>
>       <documentation lang="en">The version of DocType interpreter used to
> create the EBML Document.</documentation>
>     </element>
>       <documentation lang="en">The minimum DocType version an interpreter
> has to support to read this EBML Document.</documentation>
>     </element>
>   </element>
>     <documentation lang="en">Used to void damaged data, to avoid unexpected
> behaviors when using damaged data. The content is discarded. Also used to
> reserve space in a sub-element for later use.</documentation>
>   </element>
>     <documentation lang="en">The CRC is computed on all the data from the
> last CRC element (or start of the upper level element), up to the CRC
> element, including other previous CRC elements. All level 1 elements SHOULD
> include a CRC-32.</documentation>
>   </element>
> If this approach seems agreeable for defining machine-readable documents of
> EBML DocTypes then can next create versions of them according to Matroska
> and webm.
>
> Best Regards,
> Dave Rice

-- 
Steve Lhomme
Matroska association Chairman

---

From: Dave Rice <[email-masked]>
Date: Mon, 8 Feb 2016 23:45:15 -0500

> On Feb 3, 2016, at 4:38 AM, Steve Lhomme <[email-masked]> wrote:
> 

If the EBML Header is considered as analogous to the XML Declaration then perhaps it should not be included in an EBML Schema, since the XML Declaration isn’t referred to in XML Schemas. In the XML Specification <https://www.w3.org/TR/xml11/#sec-prolog-dtd> it says "XML 1.1 documents MUST begin with an XML declaration which specifies the version of XML being used.” but later implies that if an XML Declaration is not included that the parser can consider the document to be XML 1.0 (the declaration was not mandatory in XML 1.0).

I squashed some updates into the GitHub pull request at 

This includes making a separate table to define the attribute of <EBMLSchema>: https://github.com/Matroska-Org/ebml-specification/pull/51

| attribute name | required | definition |
|----------------|----------|------------|
| docType        | Yes      | The `docType` lists the official name of the EBML Document Type that is defined by the EBML Schema; for example, `<EBMLSchema docType="matroska">`. |
| version        | Yes      | The `version` lists an incremental non-negative integer that specifies the version of the docType documented by the EBML Schema. Unlike XML Schemas, an EBML Schema documents all versions of a docType's definition rather than using separate EBML Schemas for each version of a docType. Elements may be introduced and deprecated by using the minver and maxver attributes of <element>. |

I also added definitions for minver and maxver attributes for <element>, partly based on conversations with Steve about this at FOSDEM:

| minver         | No       | The `minver` (minimum version) attribute stores a non-negative integer that represents the first version of the docType to support the element. If the `minver` attribute is not used it is assumed that the element has a minimum version of "1". |
| maxver         | No       | The `maxver` (maximum version) attribute stores a non-negative integer that represents the last or most recent version of the docType to support the element. If the `maxver` attribute is not used it is assumed that the element has a maximum version equal to the value stored in the `version` attribute of <EBMLSchema>. |

I also updated the EBML Schema example according to this changes to the specification, so now it looks like:

<?xml version="1.0" encoding="utf-8"?>
<EBMLSchema docType="files-in-ebml-demo" version="1">
  <element name="Files" level="0" id="0x1946696C" type="master">
    <documentation lang="en">Container of data and attributes representing one or many files.</documentation>
    <element name="File" level="1" id="0x6146" type="master" minOccurs="1" maxOccurs="unbounded">
      <documentation lang="en">An attached file.</documentation>
      <element name="FileName" level="2" id="0x614E" type="utf-8" minOccurs="1">
        <documentation lang="en">Filename of the attached file.</documentation>
      </element>
      <element name="MimeType" level="2" id="0x464D" type="string" minOccurs="1">
        <documentation lang="en">MIME type of the file.</documentation>
      </element>
      <element name="ModificationTimestamp" level="2" id="0x4654" type="date" minOccurs="1">
        <documentation lang="en">Modification timestamp of the file.</documentation>
      </element>
      <element name="Data" level="2" id="0x4664" type="binary" minOccurs="1">
        <documentation lang="en">The data of the file.</documentation>
      </element>
    </element>
  </element>
</EBMLSchema>

Please review the changes in context at https://github.com/Matroska-Org/ebml-specification/pull/51.

Best Regards,
Dave Rice

> _______________________________________________
> Cellar mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cellar