Subject: [Cellar] Position References vs Linked Segments

From: Dave Rice <[email-masked]>
Date: Sun, 3 Jul 2016 20:57:44 -0400

Hi Moritz, Steve,

Many Elements use the concept of Position References to store offsets. This is currently defined at https://www.matroska.org/technical/specs/notes.html#Position_References.

Although this section is linked to the documentation the term Position Reference is never used outside the definition. I propose changing the term from Position Reference to Segment Offset, if I properly understand that the term refers to the offset within from the Segment or from the first Segment of a Linked Segment.

Also, I'm curious if this part of the documentation is correct: "For example to reference a position in the third segment, the position will be: the first segment total size + second segment total size + offset of the element in the third segment."

Suppose we have this structure

Segment1	ElementID	4 bytes
Segment1	DataSize		6 bytes
Segment1	Data 		100 bytes
Segment1	TotalSize		110 bytes

Segment2	ElementID	4 bytes
Segment2	DataSize		7 bytes
Segment2	Data 		200 bytes
Segment2	TotalSize		211 bytes

Segment3	ElementID		4 bytes
Segment3	DataSize			8 bytes
Segment3	OffsetToElement	300 bytes

By the definition a Position Reference for an Element within Segment3 would be:
Segment1:TotalSize + Segment2:TotalSize + Segment3:OffsetToElement = 110 + 211 + 300 = 621

But are you sure that it is not

Segment1:Data + Segment2:TotalSize + Segment3:ElementID + Segment3:DataSize + Segment3:OffsetToElement =
100 + 211 + 4 + 8 + 300 = 623

If each Segment uses the same length of the Element Data Size of the Segment Element then these two options have the same result, but I think the latter option may be correct since the rest of the definition is referencing offsets from the beginning of data within the first Segment (which is the latter equation).

Best Regards,
Dave Rice

---

From: Moritz Bunkus <[email-masked]>
Date: Mon, 4 Jul 2016 12:57:43 +0200

Hey,

> Although this section is linked to the documentation the term Position
> Reference is never used outside the definition. I propose changing the
> term from Position Reference to Segment Offset, if I properly
> understand that the term refers to the offset within from the Segment
> or from the first Segment of a Linked Segment.

+1

> Also, I'm curious if this part of the documentation is correct: "For
> example to reference a position in the third segment, the position
> will be: the first segment total size + second segment total size +
> offset of the element in the third segment."

I'm almost certain that it should be the segment's data sizes that are
added excluding their element ID and size field size.

> Segment1:Data + Segment2:TotalSize + Segment3:ElementID + Segment3:DataSize + Segment3:OffsetToElement =

Meaning I agree that this should be correct. It was probably an
oversight, or we simply weren't careful enough with our wording.

Kind regards,
mosu

---

From: Dave Rice <[email-masked]>
Date: Mon, 4 Jul 2016 10:44:14 -0400

Hi all,

> On Jul 4, 2016, at 6:57 AM, Moritz Bunkus <[email-masked]> wrote:
> 
> Hey,
> 

Based on this I rewrote the Position Reference section. A pull request is reviewable here and the updated section is pasted below. I mentioned calling this concept "Segment Offset" but since many Element names that use this concept have the word "Position" in their name, I kept Position in the name of the concept.

I think the math to get the Segment Position of an Element in a non-first Segment is tricky, since you have to add Segment fragments while skipping non-Segment data such as EBML Headers, so I added an example.

diff --git a/notes.md b/notes.md
index 2db0fc3..e3147c4 100644
--- a/notes.md
+++ b/notes.md
@@ -93,9 +93,38 @@ An Octet refers to a byte made of 8 bits.
 
 Overlay tracks should be rendered in the same 'channel' as the track it's linked to. When content is found in such a track it is played on the rendering channel instead of the original track.
 
-### Position References
+### Segment Position
 
-The position in some Elements refers to the position, in octets, from the beginning of an Element. The reference is the beginning of the first Segment (= its position + the size of its ID and size fields). 0 = first possible position of a level 1 Element in the Segment. When data is spanned over mutiple Segments within a [Linked Segment](#linked-segments) (in the same file or in different files), the position represents the accumulated offset of each Segment. For example to reference a position in the third Segment, the position will be: the first segment total size + second segment total size + offset of the Element in the third segment.
+The Segment Position of an Element refers to the position of that Element's Element ID, measured in octets, from the beginning of the Element Data section of the first Segment Element. In other words the Segment Position of an Element is the distant in octets from the beginning of the first Segment Element minus the size of the Element ID and Element Data Size of that first Segment Element. The Segment Position of the first Child Element of the first Segment is 0.
+
+When data is spanned over mutiple Segments within a [Linked Segment](#linked-segments) (in the same file or in different files), the position represents the accumulated offset of each Segment. For example to reference a position in the third Segment, the position will be: the size of the Element Data of the first Segment + the total size of the second Segment + offset of that Element from the beginning of the third Segment.
+
+These tables present an example of Segment Position. The Segment Element ID is 0x18538067 and the Element Data Size of the Segment Element is (in this case) 0x8B. The Element ID of the MuxingApp Element is 0x4D80. 
+
+         0                             1                             2
+         0  1  2  3  4  5  6  7  8  9  0  1  2  3  4  5  6  7  8  9  0
+         +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
+      0 |1A|45|DF|A3|80|18|53|80|67|8B|15|49|A9|66|86|4D|80|80|57|41|
+     20 |80|
+
+In the above example, the MuxingApp Element's Element ID (0x4D80) is stored at position 15 and the position of the Element Data of the Segment Element is at 10. Thus Segment Position of MuxingApp in the above example is 5.
+
+         0                             1                             2
+         0  1  2  3  4  5  6  7  8  9  0  1  2  3  4  5  6  7  8  9  0
+         +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
+      0 |1A|45|DF|A3|80|18|53|80|67|8B|15|49|A9|66|86|4D|80|80|57|41|
+     20 |80|1A|45|DF|A3|80|18|53|80|67|8B|15|49|A9|66|86|4D|80|80|57|
+     40 |41|80|1A|45|DF|A3|80|18|53|80|67|8B|15|49|A9|66|86|4D|80|80|
+     60 |57|41|80|
+
+In the above example there are three Segments in an EBML Stream with positions of 5, 26, and 37. The MuxingApp Element of the third Segment is at position 57. The Segment Position of the MuxingApp Element of the third Segment is determined by adding up the following values:
+
+name | the size of the Element Data of the first Segment | the total size of the second Segment | offset of that Element from the beginning of the third Segment
+:--------------|:-------------|:--------|:-------
+range          | 10 - 21      | 26 - 42 | 47 - 57
+size           | 11           | 16      | 10
+
+Thus the SegmentOffset of the MuxingApp Element of the third Segment is 11 + 16 + 10 which equals 37.
 
 ### Raw Timecode
 
-- 
2.8.4

Please check my octet counting in the above example and make sure that this math aligns to practice.

Best Regards,
Dave Rice

---

From: Steve Lhomme <[email-masked]>
Date: Mon, 4 Jul 2016 17:03:10 +0200

+1

> oversight, or we simply weren't careful enough with our wording.

Yes, offsets should exclude the EBML header and the Segment header (ID
+ length).

Moritz, when splitting a file with tags or chapters, do you put them
in each segment or refer to them with an offset in one of the segments
?
Do we have a real life case where we reference an element outside of
its own Segment ?

> Kind regards,
> mosu
>
> _______________________________________________
> Cellar mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cellar

-- 
Steve Lhomme
Matroska association Chairman

---

From: Moritz Bunkus <[email-masked]>
Date: Mon, 4 Jul 2016 17:11:16 +0200

Hey,

> in each segment or refer to them with an offset in one of the segments

I think mkvmerge always includes all tags, but chapters are handled
differently depending on whether segment linking is active or not:

1. If it is NOT active (= if the option "--link" is not given) then
   mkvmerge will create stand-alone files. Each frame starts at
   timestamp 0, they do not use NextUID/PrevUID elements, and mkvmerge
   will only include those chapter atoms that lie at least partially
   within the segment. The chapter start/end timestamps will be
   adjusted.

2. If linking is active then NextUID/PrevUID will be set automatically,
   and again only those chapters that lie in the current part will be
   included — but their start/end timestamps won't be modified.

Kind regards,
mosu