Subject: Re: [Cellar] EBML Element Parenting

From: Steve Lhomme <[email-masked]>
Date: Sun, 2 Oct 2016 16:36:36 +0200

2016-09-25 23:31 GMT+02:00 Dave Rice <[email-masked]>:
> Hi,
>
> I finished a review of
> https://github.com/Matroska-Org/ebml-specification/pull/106 with many
> comments there. Overall I like the new version of EBML Path expression based
> on an EBNF-defined convention.
>
> On Sep 25, 2016, at 2:22 PM, Steve Lhomme <[email-masked]> wrote:
>
> Le 25 sept. 2016 18:38, "Dave Rice" <[email-masked]> a écrit :
>>
>> Hi,
>>
>> > On Sep 25, 2016, at 11:51, Steve Lhomme <[email-masked]> wrote:
>> >
>> > I went ahead and did a new Pull Request with an EBNF based notation to
>> > define the path. I merged the minOccurs/maxOccurs as well as it's
>> > simple enough to add it to the "regular expression" in a way anyone
>> > can understand. Just reading that path line of an element gives you a
>> > quick view of where it is found and how often (including recursively).
>> >
>> > https://github.com/Matroska-Org/ebml-specification/pull/106
>>
>> This looks good though I suggest that we have a mechanism to express a
>> parent without the full path. In xpath '//' is used in this way.
>>
>> For instance "//Info/MuxingApp" could mean that MuxingApp could be a Child
>> Element of any Info Element. Subsequently Info could be defined only at
>> "/Segment/Info". Through extrapolating one could see that MuxingApp can only
>> be at "/Segment/Info/MuxingApp" without needed the full path.
>
> We could have something like that but the idea is that part of Matroska or
> different EBML formats could be defined elsewhere. It's a lot easier to
> understand unambiguously if you have the full path. Also if two formats use
> the same name for different things that becomes a problem. You may also have
> math formats using X, Y, Z all over the place.
>
> Yes but it could cut some of the semantic redundancy. For instance
> `Language`, `ChapLanguage`, and `TagLanguage` could be all defined as
> documenting the language of their Parent Element’s context.

Semantically it's correct.

> name=“Language”
> path="//TrackEntry/Language|//ChapterDisplay/Language|//SimpleTag/Language”

But they don't use the same EBML ID.

This is a possibilty that we could allow in other EBML formats though.

> Also in the PR the concepts of minOccur and maxOccur are moved from
> <element> attributes into the path expression such as
> “/EBML/EBMLVersion(1,1)”. This makes it a bit more difficult as one would
> now have to parse the XML and then parse the path. Not a blocking issue, but
> for tools that will depend on the EBML Schema, parsing out queries such as
> ‘find all mandatory elements’ or ‘all non-repeating element’ may be more
> difficult.

Yes, I will bring back the minOccurs/maxOccurs XML attributes. But I
think the values should be kept in the path. This is easier to match
as a regular expression. Although the occurence is defined with {} in
perl (IIRC) and we use () with means grouping in EBNF.

> But that should be easy to add an EBMLElementParent in there.
>
>> > Now after doing that I realized the XML schema wasn't as easy to parse
>> > as before. Let me know what you think. We may have the EBNF notation
>> > for each element definition in the RFC and the XML definition used
>> > different parts of the EBNF definition in various attributes (likely
>> > the ones we have before).
>> >
>> > I struggled a bit to find a way to define the Void properly without
>> > making the EBNF syntax too complex.
>> >
>> > I left out more complex recursive possibilities like /a/b(/c/d)+ for
>> > recursion jumping a level. It is not supported by current EBML parsers
>> > and was never considered anyway.
>> >
>> > I used the classic EBNF base with concatenation using the "," character.
>>
>> Such as "//Tags/SimpleTag,//SimpleTag/SimpleTag"?
>
> No. "a", "b" equals "ab".
>
>> Dave
>>
>> > 2016-09-25 10:58 GMT+02:00 Reto Kromer <[email-masked]>:
>> >> Steve Lhomme wrote:
>> >>
>> >>> I found the POSIX regex
>> >>> https://en.wikipedia.org/wiki/Regular_expression#Standards
>> >>
>> >> The formal standard is:
>> >>
>> >>  ISO/IEC/IEEE 9945:2009
>> >>  Information technology -- Portable Operating System
>> >>  Interface (POSIX) Base Specifications, Issue 7
>> >>
>> >> which contains much more than "just" the regex.
>> >>
>> >> Best regards, Reto
>> >>
>> >>
>> >> AV Preservation by reto.ch
>> >> chemin du Suchet 5 | 1024 Ecublens | Switzerland
>> >> Web: https://reto.ch | Twitter: @retoch
>> >>
>> >> _______________________________________________
>> >> Cellar mailing list
>> >> [email-masked]
>> >> https://www.ietf.org/mailman/listinfo/cellar
>> >
>> >
>> >
>> > --
>> > Steve Lhomme
>> > Matroska association Chairman
>> >
>> > _______________________________________________
>> > Cellar mailing list
>> > [email-masked]
>> > https://www.ietf.org/mailman/listinfo/cellar
>>
>
> Dave Rice

-- 
Steve Lhomme
Matroska association Chairman

---

From: Steve Lhomme <[email-masked]>
Date: Sun, 2 Oct 2016 18:36:28 +0200

I pushed a new version with the various fixes Dave and Reto requested.

On top of that I added a commit that use ABNF/RFC5234 rather than
EBNF. CRC-32 and Void could be improved. But what we gain here is a
standardized way to define repetition and we get rid of an ISO
reference.

Because "*" is already used by ABNF I added the "any" keyword which
replaces the "*" wildcard. The path delimiter is now "\" not to be
confused with the "|" OR operator in ABNF.

It's a little harder to read than EBNF but since it's an IETF standard
it's probably better to use it anyway.

---

From: Reto Kromer <[email-masked]>
Date: Sun, 2 Oct 2016 21:18:35 +0200

Steve Lhomme wrote:

>Yes, I will bring back the minOccurs/maxOccurs XML
>attributes. But I think the values should be kept in the
>path. This is easier to match as a regular expression.
>Although the occurence is defined with {} in perl (IIRC)
>and we use () with means grouping in EBNF.

+1

---

From: Steve Lhomme <[email-masked]>
Date: Sun, 9 Oct 2016 17:45:23 +0200

I updated the pull request #106 with cleaner CRC32 and Void pathes
that actually match the specified ABNF syntax.

Now elements that can be placed anywhere in a path starting from a
fixed path (CRC32 and Void having the special case of the fixed part
being the root) have their own specific syntax. The variable part must
be the last parent in the path.

The ABNF syntax has been used more with the use of DIGIT, ALPHA and ()
grouping of elements "repetition".

The EBMLElementOccurence is now mandatory. This makes the
path/occurence easy to split.

---

From: Steve Lhomme <[email-masked]>
Date: Tue, 11 Oct 2016 08:38:37 +0200

Pull request merged, we can now continue fixing the little issues and
get going with the Matroska specs.

---

From: Dave Rice <[email-masked]>
Date: Thu, 13 Oct 2016 22:27:06 -0400

Hi all,

> On Oct 11, 2016, at 2:38 AM, Steve Lhomme <[email-masked]> wrote:
> 

I added a new commit to Steve’s pull request [1] to update Matroska’s EBML Schema to the newly defined form. Rather than do these updates manually I converted the EBML Schema from Steve’s edit through this xsl [2] (then only had to insert the commented out ‘TimecodeScaleDenominator’ Element). The pull request's updated EBML Schema for Matroska can be viewed at https://github.com/Matroska-Org/matroska-specification/blob/parenting/ebml_matroska.xml.

Dave

[1]: https://github.com/Matroska-Org/matroska-specification/pull/42
[2]: https://gist.github.com/dericed/ddb0eaa6bb4f0983825e715d45ff6141

---

From: Steve Lhomme <[email-masked]>
Date: Sun, 16 Oct 2016 14:16:05 +0200

> I added a new commit to Steve’s pull request [1] to update Matroska’s EBML Schema to the newly defined form. Rather than do these updates manually I converted the EBML Schema from Steve’s edit thro…

Both are merged.