Subject: [Cellar] Fwd: [Matroska-devel] EBML data type constraints

From: Dave Rice <[email-masked]>
Date: Sat, 13 Feb 2016 16:09:14 -0500

Hi all,
Working on the Security Considerations section reminded me that there’s some open discussion about the use of null bytes as terminators in UTF-8 and String elements. This is an extract from Matroska-devel last year:

> Begin forwarded message:
> 
> From: wm4 <[email-masked]>
> Subject: Re: [Matroska-devel] EBML data type constraints
> Date: July 2, 2015 at 11:50:13 AM EDT
> To: [email-masked]
> Reply-To: Discussion about the current and future development of Matroska <[email-masked]>
> 
> On Thu, 02 Jul 2015 17:26:07 +0200
> Jerome Martinez <[email-masked]> wrote:
> 
>> Le 02/07/2015 16:21, wm4 a écrit :
>>> On the other hand, allowing zero-padded strings will cause weirdness 
>>> with other languages which do byte strings, and where a zero byte 
>>> won't automatically terminate a string. IMHO this stupid detail should 
>>> just be disallowed (or deprecated).
>> 
>> In that case, I propose:
>> 
>> String - Printable ASCII (0x20 to 0x7E)
>> Muxers MUST NOT append any NULL byte
>> Demuxers SHOULD accept NULL padding when present and consider such bytes 
>> as non existent.
>> 
>> UTF-8 - Printable Unicode string (Unicode character value >=0x20) 
>> conforming to RFC 2279
>> Muxers MUST NOT append any NULL byte
>> Demuxers SHOULD accept NULL padding when present and consider such bytes 
>> as non existent.
> 
> +1
> 
> (This wording doesn't specify what happens if there are null bytes in
> the middle of the string, with trailing non-null bytes - should be
> refined.)

The present wording for string and UTF-8 Element Types is:
> A String Element may either be empty (zero-length) or contain Printable ASCII characters in the range of 0x20 to 0x7E. Octets with all bits set to zero may follow the string value when needed.

 and
> A UTF-8 Element shall contain only a valid Unicode string as defined in [RFC 2279](http://www.faqs.org/rfcs/rfc2279.html). Octets with all bits set to zero may follow the UTF-8 value when needed.

Is the current wording sufficient or should there be further constraints or explanation?
Dave Rice

---

From: Steve Lhomme <[email-masked]>
Date: Sun, 14 Feb 2016 16:33:20 +0100

2016-02-13 22:09 GMT+01:00 Dave Rice <[email-masked]>:
> Hi all,
> Working on the Security Considerations section reminded me that there’s some
> open discussion about the use of null bytes as terminators in UTF-8 and
> String elements. This is an extract from Matroska-devel last year:
>
> Begin forwarded message:
>
> A String Element may either be empty (zero-length) or contain Printable
> ASCII characters in the range of 0x20 to 0x7E. Octets with all bits set to
> zero may follow the string value when needed.
>
>  and
>
> A UTF-8 Element shall contain only a valid Unicode string as defined in [RFC
> 2279](http://www.faqs.org/rfcs/rfc2279.html). Octets with all bits set to
> zero may follow the UTF-8 value when needed.
>
>

Same remark as above, it doesn't take in consideration if there's data
after the first 0x00. Any data after the first 0x00 should be
discarded. This kinda sucks for languages that don't use NULL
terminators. They have to parse every single strings from the back
(and possibly never ever finding NULL). It would be easier to assume
that if the last octet is not 0x00 then there's no 0x00 elsewhere in
the string.

Since EBML has the ability to void elements, I'd rather not keep the
NULL termination at all. Not sure what are the implications with real
world files.

> Dave Rice
>
> _______________________________________________
> Cellar mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cellar
>

-- 
Steve Lhomme
Matroska association Chairman

---

From: Jerome Martinez <[email-masked]>
Date: Sun, 14 Feb 2016 17:05:29 +0100

On 14/02/2016 16:33, Steve Lhomme wrote:
> [...]
> terminators.

as C++ std::string.

>   They have to parse every single strings from the back

And what a demuxer must do if there are actually 0x00 in the string?

 From my understanding of RFC 2279
"Character values from 0000 0000 to 0000 007F (US-ASCII repertoire) 
correspond to octets 00 to 7F (7 bit US-ASCII values)."
0x00 is a valid UTF-8 character (as well as all in the range 0x00-0x1F, 
forbidden by the current EBML proposal)
so 0x00410041 is a valid UTF-8 (or ASCII) string.

>
> Since EBML has the ability to void elements,

Except for 1-byte padding.

>   I'd rather not keep the

Currently I don't understand the reason we could not use 0x00 as padding 
if it is forbidden in the string.
I see 2 policies:

1/ 0x00-0x1F are forbidden as real content (we restrict ASCII and UTF-8)
I don't see a reason muxers must not write 0x00 if demuxers should 
accept 0x00 as padding. It does not hurt.

2/ We do no restrictions and take ASCII and UTF-8 as is (full range)
We consider 0x00 as a character like any other character and it is the 
role of the software to deal with non printable characters.

I have a preference on no restriction when using other standards because 
I consider it is not the role of the container to forbid characters, but 
this preference is not strong.

if we keep invalid 0x00-0x1F, I would change the wording for ASCII and 
UTF-8 string:
Muxers MUST NOT put invalid bytes sequences in UTF-8 (for UTF-8 part only)
Muxers MUST NOT put characters in the range 0x00-0x1F
Demuxers SHOULD accept characters in the range 0x00-0x1F and invalid 
invalid bytes sequences, and consider such bytes as non existent.

> the middle of the string, with trailing non-null bytes - should be

If we talk about "Demuxers SHOULD accept NULL padding when present and 
consider such bytes as non existent.", it does ("padding" does not mean 
that it is at the end of the string).

Jérôme

---

From: Jerome Martinez <[email-masked]>
Date: Sun, 14 Feb 2016 17:16:10 +0100

>>> UTF-8 - Printable Unicode string (Unicode character value >=0x20)

BTW, how should be filled a tag string with a content having line feeds 
(thinking to e.g. a summary of the content in the file, some summaries 
have several paragraphs)?

On the Matroska website, UTF-8 string has no "Unicode character value 
 >=0x20" restriction, I am afraid that we make a lot of files invalid. 
Maybe ">0x00"?

Jérôme

---

From: Steve Lhomme <[email-masked]>
Date: Sun, 14 Feb 2016 17:25:22 +0100

> And what a demuxer must do if there are actually 0x00 in the string?

Not use it.

> so 0x00410041 is a valid UTF-8 (or ASCII) string.

Uh! It could also be the last octect of a multi-octet value. So
indeed, parsing from the back is not even a clean solution. I'd rather
avoid NULL padding if possible.

Dave can you run a test on the archive files you use to see if that
ever happens ? I don't think I've actually wrote such padding in my
own code. Reading is automatic 'cause it can always in C or C++.

> Except for 1-byte padding.

Correct. That is an issue when editing tags. When removing just one
char in a string it's not possible to write to void the remaining
space. What is possible is to add an octet to the length of the
element. If it's maxed out then removing one octet could be an option.
If it's maxed out because it needs all 4 octets for the size, then
it's not possible. But a string of 2^28 size should not ever happen.

> Currently I don't understand the reason we could not use 0x00 as padding if
> it is forbidden in the string.
> I see 2 policies:
>
> I don't see a reason muxers must not write 0x00 if demuxers should accept
> 0x00 as padding. It does not hurt.
>
> We consider 0x00 as a character like any other character and it is the role
> of the software to deal with non printable characters.
>
> I have a preference on no restriction when using other standards because I
> consider it is not the role of the container to forbid characters, but this
> preference is not strong.

I think we should stick to what UTF-8 says. Forcing constraints on
another standard, another piece of code just for the container is not
a good idea IMO.

> if we keep invalid 0x00-0x1F, I would change the wording for ASCII and UTF-8
> string:
> Demuxers SHOULD accept characters in the range 0x00-0x1F and invalid invalid
> bytes sequences, and consider such bytes as non existent.
>
>
>
> Note about :
>
> consider such bytes as non existent.", it does ("padding" does not mean that
> it is at the end of the string).
>
> Jérôme
>
>

---

From: Steve Lhomme <[email-masked]>
Date: Sun, 14 Feb 2016 17:27:08 +0100

2016-02-14 17:16 GMT+01:00 Jerome Martinez <[email-masked]>:
> (thinking to e.g. a summary of the content in the file, some summaries have
> several paragraphs)?
>
>>=0x20" restriction, I am afraid that we make a lot of files invalid. Maybe
> ">0x00"?

I agree, line feeds are perfectly fine. I suppose the "non-printable"
chars should be fine. Even 0x00 is sometimes represented with a [NUL]
character in some fonts.

---

From: Jerome Martinez <[email-masked]>
Date: Sun, 14 Feb 2016 17:33:59 +0100

>> And what a demuxer must do if there are actually 0x00 in the string?

So "consider such bytes as non existent" as in the proposal, right?

> Uh! It could also be the last octect of a multi-octet value.

No!
by design (see UTF-8 encoding scheme).

>   So
> indeed, parsing from the back is not even a clean solution.

1/ choice of the developer
2/ it is clean, by UTF-8 design (mutli-byte always start with highest 
bit set to 1, i.e. b1xxxxxxx)

>   I'd rather

I am OK with it, but in that case is there a reason to reject NULL 
character (valid in both ASCII and UTF8)?

> another standard, another piece of code just for the container is not

so NULL padding is dead.

Due to legacy reasons, I propose to keep NULL padding and 0x00-0x1F 
forbidden for Matroska v1-v3, and consider 0x00-0x1F as any other 
character (normal characters).

Jérôme

---

From: Jerome Martinez <[email-masked]>
Date: Sun, 14 Feb 2016 17:35:58 +0100

> chars should be fine. Even 0x00 is sometimes represented with a [NUL]

What about current "String - Printable ASCII (0x20 to 0x7E), zero-padded 
when needed"
Shouldn't it be changed to "0x01 to 0x7E"?

---

From: Steve Lhomme <[email-masked]>
Date: Sun, 14 Feb 2016 17:42:44 +0100

> So "consider such bytes as non existent" as in the proposal, right?

Correct.

> by design (see UTF-8 encoding scheme).

Ouf. So we're safe to chose between the end-padding or not allowing
padding at all.

> 2/ it is clean, by UTF-8 design (mutli-byte always start with highest bit
> set to 1, i.e. b1xxxxxxx)
>
>>   I'd rather
> I am OK with it, but in that case is there a reason to reject NULL character
> (valid in both ASCII and UTF8)?

If we remove padding, NULL would become valid. That means older
parsers may consider it like padding. It may also truncate the string
if it's found in the middle. Given it would only be valid for UTF-8
Strings, that means only non vital part of the file (ie not codec IDs
and such) so IMO it's ok to break backward compatibility here. Not
that this case would be common...

>>> role
>>> I
> forbidden for Matroska v1-v3, and consider 0x00-0x1F as any other character
> (normal characters).
>
>
> Jérôme
>

---

From: Steve Lhomme <[email-masked]>
Date: Sun, 14 Feb 2016 17:45:40 +0100

2016-02-14 17:35 GMT+01:00 Jerome Martinez <[email-masked]>:
>>> have
>>>> Maybe

The String format is usually for things not meant for user display
such as Codec IDs, languages and MIME types. They are not supported to
have extended and weird characters. So we should stick to printable
ASCII.