Subject: Re: [Cellar] Language codes in Matroska and EBML

From: "Doug Ewell" <[email-masked]>
Date: Tue, 02 Aug 2016 10:56:23 -0700

Thanks to Jérôme and Moritz for pointing to the January discussion
about language coding.

It looks like the discussion originally focused on expanding the
existing data element in Matroska to incorporate ISO 639-3. Later
discussion was on truly adopting BCP 47, and maintaining compatibility
with existing Matroska implementations by creating a new, BCP
47-compliant language element that would take precedence over the
existing one. This seems sensible and would allow data producers to
migrate to the BCP 47 approach over time.

An important detail about BCP 47 is that compliant applications use the
IANA Language Subtag Registry [1] as their canonical source for language
subtags (codes) and other subtags, and NOT the ISO standards directly.
The Registry is maintained carefully through IANA to be compatible with
the ISO standards, except when doing so would cause conflicts or
incompatibility.

For example, the chaos that ensued in 2003 when ISO 3166/MA reassigned
the retired code element [CS] to mean "Serbia and Montenegro," when
numerous applications were still using it to mean "Czechoslovakia,"
would have caused very little trouble if BCP 47 had existed, because a
numeric replacement code would have been chosen for Serbia and
Montenegro to resolve the conflict.

So it's important to steer clear of statements like "RFC 5646 recommends
to use ISO 639 when it is possible" and "RFC 5646 recommends 2-letter
codes when 3-letters codes are not necessary," because what RFC 5646
actually says is to use the subtags that are in the Registry.

Another important point is that parsing a BCP 47 tag to remove all but
the primary language subtag is dead simple: just stop right before the
first hyphen, if any. So "sr-Latin-ME" becomes "sr". There is no need to
worry about whether the primary language subtag is 2 or 3 letters long.
The only exceptions are for the grandfathered tags "i-default" and
"i-mingo" and for private-use tags starting with "x-", all of which are
unlikely to occur in multimedia data. There are well-defined matching
mechanisms for more complex scenarios; see RFC 4647.

BCP 47 (sometimes as "RFC 5646" or its predecessors) is referenced
normatively in more than 60 RFCs, and informatively in another 20, plus
numerous drafts. So there is no question that it is widely supported!

Please feel free to contact the ietf-languages mailing list [2] with any
questions concerning BCP 47.

[1] http://www.iana.com/assignments/language-subtag-registry
[2] http://www.alvestrand.no/mailman/listinfo/ietf-languages

--
Doug Ewell | Thornton, CO, US | ewellic.org