Subject: Re: [apps-discuss] [Ltru] Fwd: Defining a CBOR tag for RFC 5646Language Tags

From: "Peter Occil" <[email-masked]>
Date: Tue, 13 May 2014 23:33:11 -0400

I'm not aware of any use case where having multiple language tags for the 
same plain-text string is useful.  For instance, RDF supports only one 
language tag for each string.  And HTML5 doesn't support multiple languages 
in the Content-Language header field or META tag; instead, for multilingual 
documents, it relies on markup to set the language used for each section. 
But plain-text strings don't admit of HTML-like markup without more.

Moreover, having multiple language tags for plain text leads to the 
additional problem of determining which parts of the text each language tag 
applies to, which is not so easy in the case of your three-language example.

--Peter

-----Original Message----- 
From: Randy Presuhn
Sent: Monday, May 12, 2014 1:44 AM
To: Ira McDonald ; LTRU Working Group ; Carsten Bormann ; Peter Occil ; Ira 
McDonald
Subject: Re: [Ltru] Fwd: [apps-discuss] Defining a CBOR tag for RFC 
5646Language Tags

Hi -

Is representation of multi-lingual strings a concern?

E.g.  "She said 'Bonjour', and then 'Ciao'."

Randy

>From: Ira McDonald <[email-masked]>
>Sent: May 11, 2014 4:50 PM
>To: LTRU Working Group <[email-masked]>, Carsten Bormann <[email-masked]>, 
>Peter Occil <[email-masked]>, Ira McDonald <[email-masked]>
>Subject: [Ltru] Fwd: [apps-discuss] Defining a CBOR tag for RFC 5646 
>Language Tags
>
>Hi,
>
><oops - retrying with the correct LTRU WG address this time, I hope...>
>
>Forwarding this note to the LTRU list for language tag experts to review.
>Please copy your reply to IETF Apps Discuss list (see below).
>
>Cheers,
>- Ira McDonald
>
>
>---------- Forwarded message ----------
>From: Carsten Bormann <[email-masked]>
>Date: Sun, May 11, 2014 at 6:48 PM
>Subject: [apps-discuss] Defining a CBOR tag for RFC 5646 Language Tags
>To: IETF Apps Discuss <[email-masked]>
>Cc: Peter Occil <[email-masked]>
>
>
>If you care about language tags, I hope the subject got your
>attention.  If you don't, please ignore this request for assistance.
>
>Concise Binary Object Representation (CBOR, RFC 7049) is a binary
>format for structured objects.  CBOR has a number of pre-defined data
>types and allows additional data types to be registered as "tags".
>Among the pre-defined data types is a text string (Unicode characters,
>encoded in UTF-8).  No facility is pre-defined for associating a
>Language Tag with such a string.
>
>A new CBOR tag is being proposed for combining a CBOR text string with
>a Language Tag.  The (single-page) proposal is in:
>
>http://peteroupc.github.io/CBOR/langtags.html
>
>The proposal is almost trivially obvious (pair a language tag with an
>UTF-8 string in a two-element array) and looks right to me.  But I'm
>not an expert in Language Tags, and silly mistakes are being made by
>non-experts all the time.
>
>If you are an expert in Language Tags:
>-- Is anything missing or could anything be done in a better way?
>-- Or does this really simply look right?
>
>Responses to me privately or to the list are appreciated.
>
>Grüße, Carsten
>
>_______________________________________________
>apps-discuss mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/apps-discuss

---

From: Dave Cridland <[email-masked]>
Date: Wed, 14 May 2014 08:18:17 +0100

> applies to, which is not so easy in the case of your three-language example.

Many years ago, Mark Crispin and Chris Newman had a proposal for embedding
language tags in invalid UTF-8; I seem to recall they publicly renounced
their proposal rather dramatically in favour of a Unicode Consortium
proposal for embedding the language tags somewhere in Plane 14 - published
as RFC 2482.

The fact it was all initiated in order to support the pressing needs of
ACAP might give you some hints as to why it never really took off, but as a
counter-proposal to language tags in metadata, it might be worth
re-examining.

Dave.

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 14 May 2014 10:26:29 +0200

On 14 May 2014, at 09:18, Dave Cridland <[email-masked]> wrote:

> embedding language tags in invalid UTF-8

In 2010, RFC 6082 deprecated that (and moved RFC 2482 to Historic), saying:

   It is an idea whose
   time never quite came.  It has been superseded by whole-transaction
   language identification such as the MIME Content-language header
   [RFC3282] and more general markup mechanisms such as those provided
   by XML.  The Unicode Consortium has deprecated the language tag
   character facility and strongly recommends against its use.

The application that motivated the new CBOR tag we are talking about happens not to require multi-language strings.
(It seems their usage is mostly about internationalization, i.e., pairing a string with its locale.)

If such a facility were needed, it could be modeled as (shown in CBOR diagnostic notation):

somenewtag([[“en”, "She said ‘“], [“fr”, “Bonjour”], [“en”, "', and then ‘“], [“zh”, “你好”], [“en”, "’.”]])

or even, maximizing compatibility by using the proposed tag as well:

othernewtag([38([“en”, "She said ‘“]), 38([“fr”, “Bonjour”]), 38([“en”, "', and then ‘“]), 38([“zh”, “你好”]), 38([“en”, "’.”])])

(there are many other ways this could be modeled, too).

Grüße, Carsten

---

From: "Peter Occil" <[email-masked]>
Date: Wed, 14 May 2014 06:09:26 -0400

I have one last issue.  I want to clarify that the language tag and the 
tagged string can optionally be annotated with CBOR tags.  For example:

38([tagX("en"), tagY("Hello world")])

I think this will address some of the "scalability" issue that Martin Duerst 
raised.

--Peter

---

From: "Peter Occil" <[email-masked]>
Date: Thu, 15 May 2014 12:35:12 -0400

I have updated my specification on language-tagged strings.

The two things new are:

- The "NOTE" section.
- The sentence "Both the language tag and the arbitrary string can 
optionally be annotated with CBOR tags."

--Peter

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Mon, 19 May 2014 19:25:56 +0000

Hm.  I was thinking:

{ "en": "Hello", "fr": "Bonjour" }

for the case of alternatives.  The wire size between

[ "en": "Hello" ] and { "en": "Hello" }

is zero; perhaps we could kill two birds with one stone?

-- 
Joe Hildebrand

---

From: John Cowan <[email-masked]>
Date: Wed, 14 May 2014 13:38:25 -0400

> proposal for embedding the language tags somewhere in Plane 14 - published

That's correct.  There is a character meaning "Language tag follows"
(U+E0001) and then there are tag versions of the 96 ASCII graphic
characters (U+E0020 through U+E007E), though in fact only letters,
digits, and hyphen would be useful.  U+E007F means "No language tag".

> counter-proposal to language tags in metadata, it might be worth

It might, but don't get your hopes up.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
SAXParserFactory [is] a hideous, evil monstrosity of a class that should
be hung, shot, beheaded, drawn and quartered, burned at the stake,
buried in unconsecrated ground, dug up, cremated, and the ashes tossed
in the Tiber while the complete cast of Wicked sings "Ding dong, the
witch is dead."  --Elliotte Rusty Harold on xml-dev

---

From: Mark Davis ☕️ <[email-masked]>
Date: Wed, 14 May 2014 10:34:23 -0700

> as a counter-proposal to language tags in metadata, it might be worth
re-examining.

The tag characters in Unicode are deprecated, and should not be used.

Mark <https://google.com/+MarkDavis>

 *— Il meglio è l’inimico del bene —*

On 14 May 2014 00:18, Dave Cridland <[email-masked]> wrote:

> Ltru mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ltru
>
>

---

From: "Peter Occil" <[email-masked]>
Date: Mon, 19 May 2014 18:35:06 -0400

As I recall, that is similar to the approach taken by JSON-LD when mapping 
the same text in different languages (there it's called a "language map").

However, one problem that I see is that this requires case-insensitive 
comparison of
keys: if a decoder encounters a language map with two keys that differ only 
in case (for example, "en" and "EN"), which one should it use?  The one 
defined earlier or later?  While there are already CBOR tags in the registry 
that depend on such ordering (such as string references and shared 
references), I believe that defining further tags that depend on this kind 
of ordering of values in the CBOR stream should be avoided as much as 
possible.  Another approach may be to forbid such an ambiguous language map 
or ignore ambiguous keys, but I don't know which is the best approach.

Issues like those should be dealt with in another tag proposal; I don't mind 
if tag 38 (this proposal) is changed to use language maps, but additional 
discussion would have to be necessary.  Until then, the simple 
language-tagged string is enough.

--Peter

-----Original Message----- 
From: Joe Hildebrand (jhildebr)
Sent: Monday, May 19, 2014 3:25 PM
To: Peter Occil ; LTRU Working Group ; [email-masked]
Subject: Re: [apps-discuss] [Ltru] Defining a CBOR tag for RFC 5646Language 
Tags

Hm.  I was thinking:

-- 
Joe Hildebrand

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 20 May 2014 00:47:26 +0200

On 20 May 2014, at 00:35, Peter Occil <[email-masked]> wrote:

> However, one problem that I see is that this requires case-insensitive comparison of
> keys: if a decoder encounters a language map with two keys that differ only in case (for example, "en" and "EN"), which one should it use?  

We generally handle problematic cases like this well by simply disallowing their use by the sender.
It then matters less how a receiver handles incoming improper streams.

Maybe this is another reason to require case-mapping to lower-case.
More aggressively, maybe combine this with at least a “SHOULD” for canonical form as in section 4.5 of RFC 5646?

Grüße, Carsten

---

From: "Peter Occil" <[email-masked]>
Date: Mon, 19 May 2014 18:57:44 -0400

A good idea.  But my point still stands that a language map should be in a 
different tag proposal, if possible.  There may still be other issues with 
changing this proposal from language-tagged strings to language-tagged maps 
that haven't been dealt with yet, but again, I don't mind if tag 38 
eventually does take that course.

--Peter

-----Original Message----- 
From: Carsten Bormann
Sent: Monday, May 19, 2014 6:47 PM
To: Peter Occil
Cc: Joe Hildebrand (jhildebr) ; LTRU Working Group ; [email-masked]
Subject: Re: [apps-discuss] [Ltru] Defining a CBOR tag for RFC 5646Language 
Tags

On 20 May 2014, at 00:35, Peter Occil <[email-masked]> wrote:

> keys: if a decoder encounters a language map with two keys that differ 
> only in case (for example, "en" and "EN"), which one should it use?

We generally handle problematic cases like this well by simply disallowing 
their use by the sender.
It then matters less how a receiver handles incoming improper streams.

Maybe this is another reason to require case-mapping to lower-case.
More aggressively, maybe combine this with at least a “SHOULD” for canonical 
form as in section 4.5 of RFC 5646?

Grüße, Carsten

---

From: "Peter Occil" <[email-masked]>
Date: Mon, 26 May 2014 15:59:15 -0400

Is there any more discussion on this document?  I feel like this CBOR tag is 
ready to be registered.

--Peter

-----Original Message----- 
From: Peter Occil
Sent: Thursday, May 15, 2014 12:35 PM
To: LTRU Working Group ; [email-masked]
Cc: Carsten Bormann
Subject: Re: [apps-discuss] [Ltru] Defining a CBOR tag for RFC 5646Language 
Tags

The two things new are:

- The "NOTE" section.

--Peter