Subject: [Cellar] using ISO 639-3 language codes in Matroska

From: Moritz Bunkus <[email-masked]>
Date: Tue, 12 Jan 2016 14:28:03 +0100

Hey,

the Matroska container currently uses the bibliopgraphic versions of
ISO 639-2 codes for marking anything that requires language
information. Back when Matroska was specified ISO 639-2 was the latest
standard available and therefore a good choice.

However, 639-2 is incomplete and has been largely superseded by
639-3[1] which covers pretty much each and every language out there;
citing Wikipedia:

"It provides an enumeration of languages as complete as possible,
including living and extinct, ancient and constructed, major and
minor, written and unwritten."

Over the course of the last couple of years users have often asked my
to extend MKVToolNix to use 639-3 codes intead of 639-2 ones. One
example of a rather common question I get is why people cannot use
e.g. Mandarin as a language; I even have a FAQ entry for that[2]. I've
always told those people that Matroska itself doesn't support
that.

Right now with work being done to extend Matroska for standardization
may be the best time to introduce 639-3 to Matroska.

Problem is I don't know the best way to do this. I see three possible
avenues each with their own sets of pros and cons, and I'd like some
feedback in order to turn this into a proper proposal:

1. Change the specs so that all language elements use 639-3 codes

2. Introduce new elements on the same level as the existing language
   elements that determine the standard the corresponding language
   element uses defaulting to 639-2 if missing

3. Introduce new elements on the same level as the existing language
   elements that contain a 639-3 code

Here are the details:

1. Change the specs so that all language elements use 639-3 codes

Pros: no new elements required. As most of 639-2 is included in 639-3
this should work mostly OK for existing applications; adding support
should be easy for both players and muxers

Cons: 639-3 is not a superset of 639-2 as far as I know (Wikipedia
agrees), though only corner cases should be affected. Matroska uses
bibliographic versions of the 639-2 codes while 639-3's codes are
dervied from the terminology ones of 639-2 (example: German is
currently "ger" in Matroska, 639-3 uses "deu") potentially confusing
players

2. Introduce new elements on the same level as the existing language
   elements that determine the standard the corresponding language
   element uses defaulting to 639-2 if missing

For example for the TrackLanguage element we'd introduce
TrackLanguageStandard, unsigned integer, default value 0; 0 meaning
ISO 639-2 bibliographic, 1 meaning 639-3.

A conforming player would have to look for such a
TrackLanguageStandard element and interpret TrackLanguage as ISO 639-2
bibliographic if TrackLanguageStandard is missing or if it is set to 0
and as 639-3 if it's present and set to 1.

Pros: clear distinction which standard was used; extensible for future
changes

Cons: introduces three new elements; for non-conforming players the
same cons as for 1. apply

3. Introduce new elements on the same level as the existing language
   elements that contain a 639-3 code

For example for the TrackLanguage element we'd introduce
TrackLanguageIso639_3, ASCII string, no default value. Restrictions:
if present a TrackLanguage element SHOULD be written, too, that
corresponds to the language in TrackLanguageIso639_3 for backwards
compatibility (possible exception: if the producing system knows that
only conforming players will ever read such a file).

A conforming player would have to look for a TrackLanguageIso639_3
element. If it's present then this element is used. Otherwise the
player looks for TrackLanguage just as it always has.

Pros: no confusion over the domain of the information in existing
language elements; should work best with non-conforming/older readers

Cons: not extensible for new standards; introduces three new elements;
complex mapping requirements for writing both elements

------------------------------------------------------------

Those are my thoughts. Ideas? Is this even a worth it or should we
just stick with 639-2? Any other ways to add support? Preferred
solutions?

Thanks.

Kind regards,
mosu

[1] https://en.wikipedia.org/wiki/ISO_639-3
[2] https://github.com/mbunkus/mkvtoolnix/wiki/Chinese-not-selectable-as-language

---

From: Jerome Martinez <[email-masked]>
Date: Tue, 12 Jan 2016 15:19:51 +0100

> always told those people that Matroska itself doesn't support

With 639-3, you'll get a problem with people from Hong Kong
https://en.wikipedia.org/wiki/Cantonese
ISO 639-3 line is.. empty :(
note: this is a real use case, I get such issue with my software, and I 
use RFC 5646, so zh-HK for Cantonese aka "Hong Kong, traditional 
characters".
Looks like RFC 5646 takes care of Chinese language issue.

Actually, I don't understand the issue with Mandarin: I understood that 
it is widely accepted to zh-CH for »Chinese (Simplified)« and zh-TW for 
»Chinese (Traditional)«, and current spec says:
"(...) followed by a dash and a country code for specialities in languages"
so looks like "zho-cn" and "zho-tw" are acceptable with current spec.
What is the issue with such string and current Matroska specs?

> may be the best time to introduce 639-3 to Matroska.

as we are moving to IETF, maybe RFC 5646 is another possibility.
I have no strong opinion about ISO 639-2 / ISO 639-3 / RFC 5646, I just 
do the remark that you may solve 1 issue but maybe not all issues.

> currently "ger" in Matroska, 639-3 uses "deu") potentially confusing

"ger" (bibliographic code, ISO 639-2/B) and "deu" (terminological code, 
ISO 639-2/T) are synonym in ISO 639-2, so current players are expected 
to support both (current spec does not say that there is a restriction 
about B or T)
Cons: does not resolve all issues (e.g. Cantonese)

> complex mapping requirements for writing both elements

three new elements? I understand there is only 1 new element in that 
case (TrackLanguageIso639_3)

3b. Introduce new elements on the same level as the existing language 
elements that provides the RFC 5646 language tag.

Pro: more IETF style (reusing an RFC), includes ISO 639-3 (if I 
understand well the RFC).

Cons: more complex

My preference is:
3b
3 (not IETF, remaining issues)
1 (minor compatibility break, remaining issues)
2 (could be a major break for old players if we decide to switch to a 
super new standard totally incompatible with ISO 639-2, remaining issues)

Jérôme

---

From: Reto Kromer <[email-masked]>
Date: Tue, 12 Jan 2016 15:31:23 +0100

Jerome Martinez wrote:

>I have no strong opinion about ISO 639-2 / ISO 639-3 /
>RFC 5646, I just do the remark that you may solve 1 issue
>but maybe not all issues.

RFC 4646 considers also the alphabet, e.g. fr_Latn_CA
(= français, en caractères latins, à la sauce québécoise)

RK

AV Preservation by reto.ch
chemin du Suchet 5 | 1024 Ecublens | Switzerland
Web: http://reto.ch | Twitter: @retoch

---

From: Moritz Bunkus <[email-masked]>
Date: Tue, 12 Jan 2016 15:37:35 +0100

Hey,

> ISO 639-3 line is.. empty :(

Thanks for that information. Wasn't aware of it.

> Actually, I don't understand the issue with Mandarin: I understood that it
> is widely accepted to zh-CH for »Chinese (Simplified)« and zh-TW for

That no application that I know of actually supports that including
MKVToolNix. They all only use the language code.

> "ger" (bibliographic code, ISO 639-2/B) and "deu" (terminological code, ISO
> 639-2/T) are synonym in ISO 639-2, so current players are expected to
> support both (current spec does not say that there is a restriction about B
> or T)

The Matroska specs explicitly say that the current language codes are
the bibliographic versions of ISO 639-2. See "Language Codes" in
http://www.matroska.org/technical/specs/index.html

> three new elements? I understand there is only 1 new element in that case
> (TrackLanguageIso639_3)

There are language elements for chapters and tags, too.

Kind regards,
mosu

---

From: Jerome Martinez <[email-masked]>
Date: Tue, 12 Jan 2016 15:58:06 +0100

> MKVToolNix. They all only use the language code.

So maybe we should also change the definition of current language 
elements (spec says it is possible) at the same time.
Because it looks like it is an implementation issue, not a spec issue, 
but maybe updating MKVToolNix only would make players no more detecting 
the right language (e.g. if they don't support "en-US" and don't 
translate it to "English" as they do with "en", this is a regression)

> http://www.matroska.org/technical/specs/index.html

Oops, I missed it.
(the content of the link is with both versions).

> There are language elements for chapters and tags, too.

Got it. Maybe a global element instead of 3 different elements?

Jérôme

---

From: Dave Rice <[email-masked]>
Date: Tue, 12 Jan 2016 19:15:27 -0500

> On Jan 12, 2016, at 8:28 AM, Moritz Bunkus <[email-masked]> wrote:

[…]

>   elements that contain a 639-3 code

I’d vote for #2 or #1 with a preference to #2. If option #1 I’d suggest that the language elements may use 639-2 OR 639-3. I’d also suggest that the Matroska specification adopt a externally-managed language authority (639-3 is the most obvious choice) and not labor to extend language identification beyond the adopted standard(s). For instance if there is a noticed deficiency in ISO 639-3, I think that the deficiency is most likely in shared interest with other projects and that addressing that deficiency would be better to direct to http://www-01.sil.org/iso639-3/.

If it aids in the discussion here is a histogram of usage for the Language [22][B5][9C] Element for a large Matroska sample set from archive.org.

95254 und
8073 ara
6416 jpn
5461 eng
 990 hin
 869 fre
 651 spa
 612 per
 555 kor
 477 ger
 456 chi
 325 rus
 275 ind
 203 por
 196 dut
 187 ita
 172 tur
 155 tam
 148 rum
 110 dan
 104 pol
 101 swe
  97 heb
  91 lat
  88 fin
  47 nob
  45 nor
  45 ???
  40 unknown
  38 vie
  34 hun
  34 cat
  32 tha
  30 cze
  21 en
  20 tel
  19 ben
  18 deu
  15 gre
  14 urd
   9 ukr
   9 may
   8 pan
   7 abk
   6 slv
   6 Unspecified
   6 
   5 hrv
   5 fra
   4 srp
   4 mar
   4 lit
   4 lav
   4 ina
   4 ice
   4 NE
   3 nno
   3 jav
   3 est
   3 bul
   3 arn
   3 alb
   2 zul
   2 enm
   2 arp
   2 arm
   2 arg
   2 arc
   2 ar
   2 aar
   1 zho
   1 yao
   1 wak
   1 vai
   1 tiv
   1 std
   1 scr
   1 scc
   1 rom
   1 oci
   1 kru
   1 ira
   1 inc
   1 hau
   1 grc
   1 fr
   1 epo
   1 efi
   1 bos
   1 bnt
   1 ava
   1 apa
   1 amh
   1 aka
   1 NAR

Dave Rice

---

From: Jerome Martinez <[email-masked]>
Date: Wed, 13 Jan 2016 08:40:36 +0100

Le 13/01/2016 01:15, Dave Rice a écrit :
> I’d vote for #2 or #1 with a preference to #2. If option #1 I’d suggest that the language elements may use 639-2 OR 639-3. I’d also suggest that the Matroska specification adopt a externally-managed language authority (639-3 is the most obvious choice)

RFC 5646 extends ISO 639-3 so it is not less obvious to consider it.
Additionally, even if it is not used in reality, Matroska spec already 
has some kind of RFC 5646 style (countries) so it may be relevant.
What are the arguments against using 639-3 "extended" with RFC 5646?
less complex, the definition could be:
Language codes can be either the 3 letters ISO-639-3 form (like "fre" 
for french), or such a language code followed by a dash and a country 
code for specialities in languages (like "fre-ca" for Canadian French). 
Country codes are the same as used for internet domains.
("bibliographic ISO-639-2" changed by "ISO-639-3" in the current spec)

>   and not labor to extend language identification beyond the adopted standard(s). For instance if there is a noticed deficiency in ISO 639-3, I think that the deficiency is most likely in shared interest with other projects and that addressing that deficiency would be better to direct to http://www-01.sil.org/iso639-3/.

I provided 1 example (zh-HK), but I guess there are more examples, and 
change requests are a long process.
We may need a "workaround" in the meanwhile.
We could say that ISO 639-3 SHOULD be used and RFC 5646 is used only if 
up to date ISO 639-3 does not fit the need.

Jérôme

---

From: Moritz Bunkus <[email-masked]>
Date: Wed, 13 Jan 2016 09:01:40 +0100

Hey,

> change requests are a long process.  We may need a "workaround" in the
> meanwhile.  We could say that ISO 639-3 SHOULD be used and RFC 5646 is
> used only if up to date ISO 639-3 does not fit the need.

For what it's worth I don't like such a "use this unless XYZ or
ABC". The standard should clearly say what to use. If the RFC is indeed
a superset of 639-3 then I'm in favor of the RFC; we're trying to
replace a standard where users are running into gaps today. If we
replace that with a standard where users will be running into gaps, too,
though less frequently, then we haven't done our job correctly in my
opinion.

Kind regards,
mosu

---

From: Jerome Martinez <[email-masked]>
Date: Wed, 13 Jan 2016 09:13:53 +0100

Le 13/01/2016 09:01, Moritz Bunkus a écrit :
> a superset of 639-3 then I'm in favor of the RFC;

by "unless", I wanted to mean that we should not use the country if it 
is not necessary.
RFC 5646 says:

***
  Use as precise a tag as possible, but no more specific than is
        justified.  Avoid using subtags that are not important for
        distinguishing content in an application.

        *  For example, 'de' might suffice for tagging an email written
           in German, while "de-CH-1996" is probably unnecessarily
           precise for such a task.
***

 From my point of view, RFC 5646 alone is good for all but has a 
conflict with current Matroska specification: Matroska specification 
says that a 3-letter code is mandatory, and RFC 5646 prefers 2-letter 
code when it is same as the 3-lette code (example: "de" and "ger" are 
equivalent, RFC 5646 prefers "de" while Matroska forces "ger").
I was wanting to avoid the 2-letter code in order to avoid a conflict 
with current Matrosa specs, maybe there is a better way to say it.
Anyway, replacing "bibliographic ISO-639-2" by "ISO-639-3" in the 
current Matroska specification (and implementing the country part) is 
maybe the easiest path.

>   we're trying to

+1

Jérôme

---

From: Moritz Bunkus <[email-masked]>
Date: Wed, 13 Jan 2016 09:23:03 +0100

Hey,

> I was wanting to avoid the 2-letter code in order to avoid a conflict with
> current Matrosa specs, maybe there is a better way to say it.

Ah, I misunderstood your intention then. I do agree with your goal.

> current Matroska specification (and implementing the country part) is

That would be what I've described as migration path 1 plus adding
support for the country.

So, one vote for 2 over 1 (Dave), one for 1 (Jérôme).

Kind regards,
mosu

---

From: Reto Kromer <[email-masked]>
Date: Wed, 13 Jan 2016 10:55:22 +0100

Moritz Bunkus wrote:

>> I was wanting to avoid the 2-letter code in order to
>> avoid a conflict with current Matrosa specs, maybe there
>> is a better way to say it.
>
>> Anyway, replacing "bibliographic ISO-639-2" by 
>> "ISO-639-3" in the current Matroska specification (and
>> implementing the country part) is maybe the easiest path.
>
>That would be what I've described as migration path 1 plus
>adding support for the country.
>

Please let me add that my intervention in favour RFC 4646
over RFC 5646 was not meant to be a gag. Actually we use it
in our company:

  https://tools.ietf.org/html/rfc4646

We had to struggle with films in the time when Serbia and
Montenegro was still one nation, yet with two alphabets (and
almost two everything). And currently we are working on
projects in different Asian countries with very artistic,
but often different alphabets. Therefore this can be very
useful in real life. That said, you are completely free to
choose other ways, of course.

Hope this helps! Reto

---

From: Jerome Martinez <[email-masked]>
Date: Wed, 13 Jan 2016 10:59:11 +0100

Le 13/01/2016 10:55, Reto Kromer a écrit :
> over RFC 5646 was not meant to be a gag.

I don't understand: RFC 4646 is obsoleted by RFC 5646.

Jérôme

---

From: Jerome Martinez <[email-masked]>
Date: Thu, 14 Jan 2016 10:54:46 +0100

> So, one vote for 2 over 1 (Dave), one for 1 (Jérôme).

I voted for my 3b (Introduce new elements on the same level as the 
existing language elements that provides the RFC 5646 language tag, in 
other words copy/paste current definition in a new tag then replace 
"bibliographic ISO-639-2" by "RFC 5646" )
then 3 + keeping the country part (Introduce new elements on the same 
level as the existing language elements that provides the ISO-639-3 
language tag, in other words copy/paste current definition in a new tag 
then replace "bibliographic ISO-639-2" by "ISO-639-3")
then 3

the exact list, in priority order:
3b
3 + keeping the country part
3 (not IETF, remaining issues)
1 (minor compatibility break, remaining issues)
2 (could be a major break for old players if we decide to switch to a 
super new standard totally incompatible with ISO 639-2, remaining issues)

Jérôme