Subject: [Cellar] [PATCH 1/1] Make Date similar to Signed Integer

From: Michael Bradshaw <[email-masked]>
Date: Wed, 20 Jul 2016 21:32:37 +0200

I can submit a pull request on GitHub but I'm never sure if I should just
do that directly, post to this list first, or what. So I'm posting to this
first.

Summary of changes: change "Endianness" from "None" to "Big-endian" and
make the length similar to Signed Integer (i.e. from 0 to 8 octets, instead
of just 0 or 8 octets only).

---

>From 45dca191245e87935b1424d82ebede32810ddad4 Mon Sep 17 00:00:00 2001
From: Michael Bradshaw <[email-masked]>
Date: Wed, 20 Jul 2016 21:25:36 +0200
Subject: [PATCH 1/1] Make Date similar to Signed Integer

It has an endianness and there is no technical reason a date can't use
less than 8 octets.
---
 specification.markdown | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/specification.markdown b/specification.markdown
index 44bd57b..396a51c 100644
--- a/specification.markdown
+++ b/specification.markdown
@@ -155,8 +155,8 @@ Definition        | A UTF-8 Element shall contain only
a valid Unicode string as

 Element Data Type | Date
 :-----------------|:-----
-Endianness        | None
-Length            | A Date Element MUST declare a length of either 0
octets or 8 octets. A Date Element with a zero-octet length represents a
timestamp of 2001-01-01T00:00:00.000000000 UTC.
+Endianness        | Big-endian
+Length            | A Date Element MUST declare a length that is no
greater than 8 octets. A Date Element with a zero-octet length represents a
timestamp of 2001-01-01T00:00:00.000000000 UTC.
 Definition        | The Date Element MUST contain a Signed Integer that
expresses a point in time referenced in nanoseconds from the precise
beginning of the third millennium of the Gregorian Calendar in Coordinated
Universal Time (also known as 2001-01-01T00:00:00.000000000 UTC). This
provides a possible expression of time from 1708-09-11T00:12:44.854775808
UTC to 2293-04-11T11:47:16.854775807 UTC.

 Element Data Type | Master-element
--
2.8.0.rc2

---

From: Moritz Bunkus <[email-masked]>
Date: Wed, 20 Jul 2016 21:49:32 +0200

Hey,

Thanks. About this particular change: the Endian change is fine with me.

The number of octets is another matter, though. The Matroska specs
page[1] has always stated that a date is always 8 octets long. At least
one existing library (libebml) forces the size to 8 octets during
writing and even verifies that the size is 8 octets when reading.

There may very well not be a technical reason the integer couldn't be
smaller, but compatibility with existing content dictates that we
shouldn't change it.

> I can submit a pull request on GitHub but I'm never sure if I should
> just do that directly, post to this list first, or what. So I'm
> posting to this first.

Dave usually just submits pull requests. Dealing with them is usually
faster for me on Github than via email. I'd prefer to have them as pull
requests if they're smallish.

However, if there's fundamental stuff to be done or if the issue in
question is still under discussion and you're only posting a basis for
such discussion then it would be better suited for the list, I guess.

I know that's awfully vague. Sorry about that :)

Kind regards,
mosu

[1]  https://www.matroska.org/technical/specs/index.html

---

From: Steve Lhomme <[email-masked]>
Date: Thu, 21 Jul 2016 12:02:28 +0200

> smaller, but compatibility with existing content dictates that we

Good point. I can't recall a good reason why we forced the size to be
8. The closest I can think of is to make sure implementations use 64
bits for dates. As they should with integers.

In libebml EbmlDate is not a subclass of EbmlInteger and the size is
hardcoded to 8. Anything outside of that (and 0) is considered
invalid.

In FFmpeg I found this code:
    if (matroska->date_utc.size == 8)
        matroska_metadata_creation_time(&s->metadata,
AV_RB64(matroska->date_utc.data));

So current implementations wouldn't like this change.

> _______________________________________________
> Cellar mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cellar

-- 
Steve Lhomme
Matroska association Chairman

---

From: Michael Bradshaw <[email-masked]>
Date: Thu, 21 Jul 2016 12:15:27 +0200

> So current implementations wouldn't like this change.

I'm not sure this is going to break implementations any more than using a
zero-sized Date did. The spec used to say "In particular the Signed
Integer, Unsigned Integer, Float, and Date EBML Element Data Types have
definitions which require a length of at least one octet and thus in these
cases an Element Data Size with all VINT_DATA bits set to zero is invalid."
I think that if adding support for zero-sized Date/(Un)signed Integers, and
Float elements didn't cause too much of a ruckus, then I think adding
support for Date elements with 0<= size <= 8 shouldn't be too bad.

---

From: Steve Lhomme <[email-masked]>
Date: Thu, 21 Jul 2016 12:59:57 +0200

2016-07-21 12:15 GMT+02:00 Michael Bradshaw <[email-masked]>:
> zero-sized Date did. The spec used to say "In particular the Signed Integer,
> Unsigned Integer, Float, and Date EBML Element Data Types have definitions
> which require a length of at least one octet and thus in these cases an
> Element Data Size with all VINT_DATA bits set to zero is invalid." I think
> that if adding support for zero-sized Date/(Un)signed Integers, and Float
> elements didn't cause too much of a ruckus, then I think adding support for
> Date elements with 0<= size <= 8 shouldn't be too bad.

For future code it's fine and it would be a nice addition. The problem
is that every tools up to today using either libebml(2) or libavformat
will consider these future files as broken.

So if we're going down that road we should have a strong case for
making the change. What are the benefits of using other sizes than 8
and 0 ? It just restricts the date range.
8 octets: +/- 292.4 years (2^63 / 1,000,000,000 * 60 * 60 * 24* 365)
7 octets: +/- 1.14 years (2^55 / 1,000,000,000 * 60 * 60 * 24* 365)

So unless we have a narrow case where a format would only need dates
between 1999 and 2003 (generous scope), there isn't much use for dates
smaller than 8 octets.

Steve Lhomme

---

From: Jerome Martinez <[email-masked]>
Date: Thu, 21 Jul 2016 15:29:46 +0200

Le 21/07/2016 à 12:59, Steve Lhomme a écrit :
> [...]

I look more at principles: why should we have an exception (comparing to 
other signed integers) here? We could try to avoid exceptions a much as 
possible (here, for example, a conformance checker would have to add a 
specific rule "this is a singed integer, but different than other signed 
integers, must be 8 bytes") and dates could be just a signed integer 
with a point of reference (2001-01-01)
but I have concerns about interoperability too.
We had also such backward compatibility concerns and we resolved that 
with recommendation for encoders and decoders.

I am in favour of making date similar to other signed integer, with a 
temporary exception:
+Length            | A Date Element MUST declare a length that is no 
greater than 8 octets. A Date Element with a zero-octet length 
represents a timestamp of 2001-01-01T00:00:00.000000000 UTC. For better 
compatibility with old demuxers, a muxer SHOULD declare a length of 
either 0 octets or 8 octets.

Then when we break compatibility (Matroska v5?), we remove the exception 
(and in the meanwhile, we send patch to all demuxers for supporting 
dates of any size, should not be a big effort).

Other lines of the proposal LGTM.

---

From: Steve Lhomme <[email-masked]>
Date: Thu, 21 Jul 2016 16:22:13 +0200

other signed integers) here? We could try to avoid exceptions a much as
possible (here, for example, a conformance checker would have to add a
specific rule "this is a singed integer, but different than other signed
integers, must be 8 bytes") and dates could be just a signed integer with a
point of reference (2001-01-01)

I see it the other way around. Because the size is fixed it provides more
error resilience, a good thing for archival.

For example if you encounter an error in a Master element, in libebml we
parse byte by byte until we find an ID that seems like it's in the right
place. But we also check the size that comes after seem correct. If the
size can only be one possible value, there's less chance for a false
positive and allows for better error recovery.

I suppose you might benefit from that in conformance checker or a file
fixer as well.

And the code reusability between a date and an integer is very small. You
can even take shortcuts to read 64 bits in memory, whereas for an EBML
integer you need code for all possible sizes, therefore a bit slower.

So I'm not in favor of any change.

greater than 8 octets. A Date Element with a zero-octet length represents a
timestamp of 2001-01-01T00:00:00.000000000 UTC. For better compatibility
with old demuxers, a muxer SHOULD declare a length of either 0 octets or 8
octets.
> Then when we break compatibility (Matroska v5?), we remove the exception
(and in the meanwhile, we send patch to all demuxers for supporting dates
of any size, should not be a big effort).