Subject: [apps-discuss] i18n intro, Sunday 14:00-16:00

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 19 Jul 2011 12:48:39 -0600

You might have noticed a curious item on the agenda at 14:00 on Sunday: 
"Apps Area Preparatory Meeting for Internationalization Working Groups".

At that time, I will present an introduction to internationalization, 
assisted by Pete Resnick (who will correct me where I go wrong). The 
intent of this session is to help apps-area folks learn more about 
internationalization, especially in preparation for the PRECIS WG 
meeting on Thursday. The room we've been assigned (2103) holds up to 60 
people so we should have plenty of space, and there is no need to sign 
up if you want to attend.

If this session goes well, Pete and I might offer a more general 
tutorial at a future IETF meeting. Consider Sunday's session a dry run.

See you in Quebec City!

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 19 Jul 2011 13:20:30 -0600

> tutorial at a future IETF meeting. Consider Sunday's session a dry run.

Sorry, I neglected to provide a pointer to my slides:

http://www.saint-andre.com/ietf/i18n-intro.pdf

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: John C Klensin <[email-masked]>
Date: Tue, 19 Jul 2011 16:00:47 -0400

--On Tuesday, July 19, 2011 13:20 -0600 Peter Saint-Andre
<[email-masked]> wrote:

>> You might have noticed a curious item on the agenda at 14:00
>> on Sunday: "Apps Area Preparatory Meeting for
>> Internationalization Working Groups".
>...

197 slides?  YMBK.

And, by the way, in the beginning, depending on where you start
counting, there were BCD, BAUDOT, and a few other things,
including, arguably, Morse Code.  In a different vocabulary
consider ITU Recommendation T.50, aka International Alphabet 5
(IA5 or IRA5), whose original version corresponded to ISO 646,
whose IRV is essentially ASCII.  It wasn't designated as IA5
because "5" was a nice number -- there were four predecessors
including Recommendation S.1 for ITA2.  :-)

More follows offlist...

   john

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 19 Jul 2011 14:33:37 -0600

>> http://www.saint-andre.com/ietf/i18n-intro.pdf

Why must I be kidding? I presented much the same slides at the XMPP WG 
interim meeting back in February, and we had a productive session.

> including Recommendation S.1 for ITA2.  :-)

These slides are not meant to be comprehensive, just a friendly 
introduction to some key topics.

> More follows offlist...

As always, feedback is welcome.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Nathaniel Borenstein <[email-masked]>
Date: Wed, 20 Jul 2011 08:50:10 +0100

>> http://www.saint-andre.com/ietf/i18n-intro.pdf

Yes, this is clearly far too complex an area to cover in only 197 slides, but perhaps as a simplified introduction.....

And no, I'm not joking (for once).

I for one am very grateful that you (Pete*) are doing this, and look forward to finally achieving internationalization enlightenment.  -- Nathaniel

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 20 Jul 2011 10:22:26 -0700

+1.  See you there.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Nathaniel Borenstein
Sent: Wednesday, July 20, 2011 12:50 AM
To: John C Klensin
Cc: [email-masked]
Subject: Re: [apps-discuss] i18n intro, Sunday 14:00-16:00

197 slides?  YMBK.

---

From: John C Klensin <[email-masked]>
Date: Wed, 20 Jul 2011 14:44:47 -0400

--On Wednesday, July 20, 2011 08:50 +0100 Nathaniel Borenstein
<[email-masked]> wrote:

> Yes, this is clearly far too complex an area to cover in only
> 197 slides, but perhaps as a simplified introduction.....

Indeed.   Sadly, I had not looked at the slides and made an
incorrect assumption about the relationship between that many
slides and the presumed time slot.

> I for one am very grateful that you (Pete*) are doing this,
> and look forward to finally achieving internationalization
> enlightenment.  -- Nathaniel

I have yet to achieve i18n enlightenment, despite looking at
probably thousands of slides over the years (and making several
hundred).  But, like other journeys, this one probably starts
with a single step... or 197 slides.  I, too, look forward to
the presentation.

    john

---

From: Dave CROCKER <[email-masked]>
Date: Wed, 20 Jul 2011 14:52:13 -0400

On 7/19/2011 4:33 PM, Peter Saint-Andre wrote:
>> 197 slides? YMBK.
>
> Why must I be kidding? I presented much the same slides at the XMPP WG interim
> meeting back in February, and we had a productive session.

Animation is a powerful enhancements for slides, so this actually, that sounds 
pretty short, given 24 slides per second...

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: Peter Saint-Andre <[email-masked]>
Date: Wed, 20 Jul 2011 13:00:30 -0600

On 7/20/11 12:52 PM, Dave CROCKER wrote:
>> interim
> Animation is a powerful enhancements for slides, so this actually, that
> sounds pretty short, given 24 slides per second...

In fact I've presented ~100 slides in 8-10 minutes before, so this one 
will be in slow motion because we've got 2 hours. :)

/psa

---

From: Dave CROCKER <[email-masked]>
Date: Wed, 20 Jul 2011 15:03:36 -0400

On 7/20/2011 3:00 PM, Peter Saint-Andre wrote:
> In fact I've presented ~100 slides in 8-10 minutes before, so this one will be
> in slow motion because we've got 2 hours. :)

actually, I'm impressed at what this means about how little ego you have in 
doing a presentation, since folks will (have to) devote all their attention to 
the flashing screen rather than pay attention to you...

d/

-- 

   Dave Crocker

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 21 Jul 2011 16:03:12 +0900

Some comments:

Slide 7: there are thousands of languages and scripts:
Thousands of languages: yes; thousands of scripts: no
(http://www.unicode.org/Public/UNIDATA/Scripts.txt currently has 95 
(including 'Common'), so "well over a hundred" would me much more 
appropriate).

Slide 10: we need to encode more than [A-Z][a-z]:
This should probably be: [A-Za-z]

Slide 22: Why not UPPER vs. lower vs. Title ?

Slide 24: You could do ｆｕｌｌ, half

Slide 55: There actually is U+1E9E LATIN CAPITAL LETTER SHARP S. But 
case conversion only goes downwards, not upwards.

Slide 113: There is no such thing as Compat Recomp!

Slide 114: "Fastest processing": That depends on various assumptions, in 
particular on the assumption that you actually implement processing as 
in slide 113 (which typically is not done).

Slide 117: "Compared to NFKC: Produces more false positives during
comparison operations": This is confusing/wrong. If matches are 
positive, then NFKC will match more than NFC, and if some of these 
matches are considered false, then NFKC will produce more false positives.

Slides 114-118: Some of the arguments given for a single NF apply to a 
certain aspect of NFs, e.g. C or D or K.

Slide 123: Good to see that. By the way, I seem to remember both John 
and me begging you for an explanation of why Jabber wants to use NFD a 
few months ago, and I'm not sure I have seen an answer. Now might be a 
good time (if you already sent one, a pointer would be appreciated).

Slide 131: "UTF-8 is the preferred IETF encoding (RFC 3629)":
RFC 3629 is the reference for UTF-8 per se, the IETF preference is 
expressed in RFC 2277. So the text should say
"UTF-8 (RFC 3629) is the preferred IETF encoding (RFC 2277)"
(or some such), and add RFC 2277 to the references.

Slide 132: integers -> bytes (or octets)
(we are really now on a lower, somewhat more physical level, and 
byte/octet is completely adequate here (indeed anything else would be 
needlessly confusing).

Slide 135: This should come close to normalization. I'd move the part on 
encoding earlier, but maybe that's just me.

Slide 168: Fussball vs. Fußball isn't a normalization issue (not even 
NFKC). Of the two HenryIV, IDNA only allows one (2008) or maps to one 
(2003).

Slide 184, middle bullet: I'd add 'occasionally' to put that issue in 
perspective

I remember having seen this talk before, but I don't know where. I 
thought it was very good. I'd be a bit worried if I'd have to spend 2h 
to present it, it's written for a fast pace.

Regards,   Martin.

---

From: Dave Cridland <[email-masked]>
Date: Thu, 21 Jul 2011 15:20:27 +0100

On Thu Jul 21 08:03:12 2011, "Martin J. Dürst" wrote:
> thought it was very good. I'd be a bit worried if I'd have to spend  
> 2h to present it, it's written for a fast pace.

Peter is often used as a benchmark for powerpoint's framerate; he'll  
get through it, I'm sure.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

---

From: Dave CROCKER <[email-masked]>
Date: Thu, 21 Jul 2011 10:45:46 -0400

On 7/21/2011 10:20 AM, Dave Cridland wrote:
> Peter is often used as a benchmark for powerpoint's framerate; he'll get through
> it, I'm sure.

yeah, but will the audience?

d

-- 

   Dave Crocker

---

From: Dave Cridland <[email-masked]>
Date: Thu, 21 Jul 2011 15:47:51 +0100

> yeah, but will the audience?

I can't think what you mean; Peter is an entertaining and engaging  
speaker, especially if you have a sweepstake on the average  
slides/second.

Dave.

---

From: Dave CROCKER <[email-masked]>
Date: Thu, 21 Jul 2011 10:52:12 -0400

On 7/21/2011 10:47 AM, Dave Cridland wrote:
> especially if you have a sweepstake on the average slides/second.

well, yes, that's probably a good way to ensure that folks focus on the content...

d/
-- 

   Dave Crocker

---

From: John C Klensin <[email-masked]>
Date: Thu, 21 Jul 2011 11:22:22 -0400

> Some comments:

I already sent Peter a rather long list of comments, many of
which overlap these, privately (I will forward them to you,
Martin, or others if you are interested).  I have remarks on a
few of these...

> (http://www.unicode.org/Public/UNIDATA/Scripts.txt currently
> has 95 (including 'Common'), so "well over a hundred" would me
> much more appropriate).

Agreed, although I read that as measuring a language-script
product, which would certainly be in the thousands.  It is
always hard to know how much detail a presentation like this
should go into but, if one wanted to push further, it might be
worth noting that the Unicode script list as found at that URL
is controversial -- some would claim that Unicode joins things
together as a single script that should be separated and
identifies things as separate scripts that are mostly
typographical variations plus a few added characters.

> RFC 3629 is the reference for UTF-8 per se, the IETF
> preference is expressed in RFC 2277. So the text should say

2277 and 5198 in both cases.  5198 is already in the references.

> (we are really now on a lower, somewhat more physical level,
> and byte/octet is completely adequate here (indeed anything
> else would be needlessly confusing).

I assumed Peter was trying to make the distinction between
abstract integer code points and their instantiation in an
encoding.  Given that distinction raised issues in EAI and in
3536bis, it is probably worth trying to keep.  Whether the hint
of using "integer" is sufficient for that purpose probably
depends on what Peter says as the slides are flying by.
Conversely, I don't know whether it is worth trying to preserve
the distinction in a talk at this level.

>...
> Slide 168: Fussball vs. Fußball isn't a normalization issue
> (not even NFKC).

Agreed.  But it is an issue with the use of toCaseFold.  One of
the suggestions I made to Peter is that it may be useful to
differentiate between the level of aggressiveness (i.e.,
vunerability to controversy) of lower-casing (or upper-casing)
and applying case folding.

> Of the two HenryIV, IDNA only allows one (2008) or maps to one
(2003).

True of several others of the examples the slides use or hint at.

best,
   john

---

From: Joe Hildebrand <[email-masked]>
Date: Thu, 21 Jul 2011 10:28:59 -0600

> Slide 123: Good to see that. By the way, I seem to remember both John 
and me
> begging you for an explanation of why Jabber wants to use NFD a 
few months
> ago, and I'm not sure I have seen an answer. Now might be a 
good time (if you
> already sent one, a pointer would be appreciated).

Let me try.  First some assumptions:
- Stringprep is currently one of the performance hotspots of some XMPP
servers.
- XMPP does not guarantee that the original form of the address that is
entered by the user or sent on the first hop is transmitted without
modification to other hops in the system.
- As such, many XMPP servers optimize by performing canonicalization at the
edges of their system and even store the canonical version for future
comparison.
- If the spec is written that clients SHOULD perform canonicalization, many
in our community will, particularly if they know that they will get better
performance from the server.

The property of NFK?D that we like is that if you have a string of
codepoints that is already in NFK?D, you can check that the string is in the
correct normalization form without having to allocate memory.  With NFK?C,
you'll have to decompose (allocating memory), recompose (at some finite CPU
cost), then recompose (possibly allocating *again*) just to check if you
have already done the normalization.

For the K portion, I have found John's argument compelling that codepoints
with compatibility decompositions should just be prohibited in our
localparts.  In our resourceparts, I'm of the opinion that we don't need to
compatibility map -- it's fine for all of those codepoints to stay distinct.

The idea is that clients SHOULD normalize, servers double-check inputs from
non-trusted sources (like clients and other servers), then always store and
forward the normalized version.

-- 
Joe Hildebrand

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Thu, 21 Jul 2011 18:57:02 +0200

>have already done the normalization.

The set of strings that is in Normalization Form C is a regular language
see <http://lists.w3.org/Archives/Public/www-archive/2009Feb/0071.html>,
so recognizing NFC strings is just as easy as recognizing NFD strings if
ignore that automata for NFC are bigger and harder to make than for NFD.
It's easier to use the simple heuristic in the specification and then do
what you suggest above for complicated strings, but it's not necessary.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Peter Saint-Andre <[email-masked]>
Date: Thu, 21 Jul 2011 22:00:50 -0600

Folks, thanks for all the feedback. I won't have time to reply 
individually, but I will work to adjust the slides before I present them 
on Sunday.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 22 Jul 2011 15:05:24 +0900

On 2011/07/21 23:20, Dave Cridland wrote:
> Peter is often used as a benchmark for powerpoint's framerate; he'll get
> through it, I'm sure.

Sorry, I meant it the other way round. It's designed to go through at a 
fast pace, so I'm affraid 2h is much too much time.

Regards,   Martin.

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 22 Jul 2011 08:08:25 +0200

On 20 July 2011 21:00, Peter Saint-Andre wrote:

> In fact I've presented ~100 slides in 8-10
> minutes before, so this one will be in slow
> motion because we've got 2 hours. :)

Thanks - almost all points covered.  Maybe JohnK
or Martin could add a slide why using UTF-8 (or
even UTF-6) was no option for IDNA2003, but okay
for EAI.  The "EAI history" in JohnL's blog is
also very good.

-Frank
 <http://xn--xyzzy.dnsalias.org>

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 22 Jul 2011 15:54:26 +0900

Hello John,

On 2011/07/22 0:22, John C Klensin wrote:
> Martin, or others if you are interested).

Yes interested, please send.

> I have remarks on a
> product, which would certainly be in the thousands.

In that case, it would be fine.

> It is
> should go into but, if one wanted to push further,

I didn't.

> it might be

I think "controversial" is too strong, because it suggests that there 
are great fights going on. Not the only way to count scripts, yes 
indeed. But I only counted that to give a rough start value, and I don't 
think there is a reasonable way to define scripts that results in thousands.

> 2277 and 5198 in both cases.  5198 is already in the references.

I'd guess then this should be:

"UTF-8 (RFC 3629, RFC 5198) is the preferred IETF encoding (RFC 2277)"

> encoding.

That would make a lot of sense. But exactly for that reason, it has to 
be bytes. "UTF-8 encodes each codepoint as a squence of 1 to 4 integers" 
just doesn't make sense.

Regards,   Martin.

> Given that distinction raised issues in EAI and in

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 22 Jul 2011 16:50:27 +0900

Hello Joe,

On 2011/07/22 1:28, Joe Hildebrand wrote:
> On 7/21/11 1:03 AM, "Martin J. Dürst"<[email-masked]>  wrote:
>
> Let me try.

Glad to start talking.

> First some assumptions:

Is that an assumption backed by facts or a wild guess?

> edges of their system and even store the canonical version for future

Makes sense.

> performance from the server.

That's the same for NFC and NFD, isn't it? The advantage of NFC is that 
it's designed to more-or-less match what's out there, so you get the 
advantage that there is more stuff that's already canonicalized even if 
if a client doesn't do anything. That gets reinforced by both the IETF 
and the W3C telling everybody to use NFC.

> have already done the normalization.

Nope. That's just how the algorithm was defined, because the 
decomposition component was already around, and NFD was what defined 
canonical equivalence.

As an example, http://www.w3.org/2003/06/xml1.1test/Overview.html has 
some code (both UTF-8 and UTF-16) for compact (memory footprint) and 
reasonably fast NFC check. Please ignore the "XML 1.1" in the title. 
Also please note that this is proof of concept code, and may need 
additional testing and of course upgrade to the newest version of 
Unicode. I don't have any open bug reports, but that may be for other 
reasons than that there are no bugs.

Also, there are various ways to trade off speed against memory (see also 
Björn's mail), but the memory here is just the footprint of the sharable 
data, there's no need for lots of memory per conversion.

> localparts.

Probably just fine for most cases. Potentially a problem for wide/narrow 
in places like Japan.

> In our resourceparts, I'm of the opinion that we don't need to

Yes indeed.

> forward the normalized version.

That's fine. But it doesn't explain the choice of NFD (vs. NFC).

Regards,   Martin.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 22 Jul 2011 17:24:55 +0900

On 2011/07/22 15:08, Frank Ellermann wrote:
> for EAI.

I'll defer to John here. I was pushing strongly for UTF-8 for IDNA2003, 
but thought that pure UTF-8 could never happen for email, because I was 
told so for over 10 years from all sides.

Regards,   Martin.

> The "EAI history" in JohnL's blog is
> also very good.
>
> -Frank
> _______________________________________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss
>

---

From: Joe Hildebrand <[email-masked]>
Date: Fri, 22 Jul 2011 09:26:17 -0600

> Is that an assumption backed by facts or a wild guess?

I can only talk definitively about one server, but yes, I've got data to
back up that assumption for that one server.

> and the W3C telling everybody to use NFC.

Absolutely.  However, my point is that our client community, unlike the MUA
community (for example) is more likely to adopt change.

> reasons than that there are no bugs.

Am I correct that the key bit of your algorithm is:

"""Check for potential combination with starter. The list of recombinations
is calculated so that any combining character that would lead to a change
(full combination, recombination so that the combining character gets
combined in but another combining character is separated, complete
decomposition,...) is listed. 2176 such pairs have been found."""

Can you talk more about that?  Do we expect that future versions of Unicode
will change that set of pairs?

> Probably just fine for most cases. Potentially a problem for wide/narrow

Yup.  That's why the wide/narrow stuff is still on the list of questions to
be addressed with this approach.  I also fully expect that there are N other
codepoints that are marked as having compatibility decomposition but are
actually in widespread use, and someone's going to complain about them one
day.

> That's fine. But it doesn't explain the choice of NFD (vs. NFC).

One other point, the re-composition stuff in the C forms *is* objectively
more difficult to implement, and will *always* take more CPU.  If it's not
required, it should be avoided.  And I haven't seen any compelling arguments
as to why C should be preferred.  What I've seen so far:

- That's the way we've always done it.  Not compelling to me if we're going
to switch away from NFKC in any way.
- Saves a few bytes on the wire.  Not compelling in a world with
compression.  (see: http://xmpp.org/extensions/xep-0138.html)
- More likely to render correctly.  Not with today's font renderers.

Are there any others I'm missing?

-- 
Joe Hildebrand

---

From: Dave Cridland <[email-masked]>
Date: Fri, 22 Jul 2011 16:44:35 +0100

> back up that assumption for that one server.

I have no opinion on NFC versus NFD, but on this specific point it is  
true for our server as well, and I'm aware of at least one other  
server for which stringprep is a key area of CPU usage (Not Joe's).

Dave.

---

From: Peter Saint-Andre <[email-masked]>
Date: Sun, 24 Jul 2011 09:39:27 -0400

FYI, I updated my slides on the plane yesterday:

Undoubtedly, errors remain. Appropriate disclaimers will be expressed.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Mon, 25 Jul 2011 14:37:40 +0900

Hello Joe,

On 2011/07/23 0:26, Joe Hildebrand wrote:
> On 7/22/11 1:50 AM, "Martin J. Dürst"<[email-masked]>  wrote:
>
>>> First some assumptions:

Okay, thanks for the confirmation.

> community (for example) is more likely to adopt change.

Yes, but there's no need to adopt change without a really good reason. 
And there should be no need for XMPP to differ from the rest of the IETF.

BTW, can you tell us what programming language the server is written in? 
Are you using any kinds of libraries for character-related business?

> Am I correct that the key bit of your algorithm is:

In some way, yes.

> Can you talk more about that?

The actual list is in
http://dev.w3.org/cvsweb/~checkout~/charlint/xml1.1test/nf16data.c?content-type=text/plain, 
in the array 'recombiners'.

The first entry is {0x003C, 0x0338}, which is a '<' and a COMBINING LONG 
SOLIDUS OVERLAY. It's here because there is U+226E, NOT LESS-THAN, which 
is canonically equivalent, and therefore the sequence U+003C, U+0338 (or 
any sequence  U+003C, [other combining characters], U+0338 ) isn't in NFC.

Let's look at another entry: {0x00FC, 0x0300}. This is LATIN SMALL 
LETTER U WITH DIAERESIS (ü) followed by COMBINING GRAVE ACCENT. This is 
not in NFC because there is U+01DC, LATIN SMALL LETTER U WITH DIAERESIS 
AND GRAVE.

Another, somewhat more involved example: {0x1EC5, 0x0323}. This is LATIN 
SMALL LETTER E WITH CIRCUMFLEX AND TILDE and COMBINING DOT BELOW. This 
is not in NFC because there is U+1EC7, LATIN SMALL LETTER E WITH 
CIRCUMFLEX AND DOT BELOW, and NFC is U+1EC7 + U+0303 (COMBINING TILDE) 
because combining characters below have a lower combining class and 
therefore are preferred when recombining.

> Do we expect that future versions of Unicode

This is at least in theory possible. The data is from 2003. With input 
from the W3C, the Unicode Consortium created a fairly strong stability 
policies regarding normalization (please see 
http://www.unicode.org/policies/stability_policy.html, under 
"Normalization Stability"). As a result, for new precomposed characters, 
they had to be included in the combining exclusions. That meant that the 
benefit of proposing them as precomposed characters was lost, and there 
were virtually no such characters that made it all the way. It's still 
possible to propose a new script with some precomposed/decomposed 
alternatives, but I wouldn't know about a case where this has been done.

Please note that such additions would also affect decomposition, so a 
software (table) update would be needed independent of whether you go 
for NFC or NFD.

> more difficult to implement,

Yes, but usually, you'd just use a library, and be done with it.

> and will *always* take more CPU.

Not if most of your data is already in NFC, and you can check that quickly.

I do not have actual statistics, but I don't know any major language or 
script where data would from the start be in NFD. On the other hand, 
there are examples such as Korean where essentially everything is in 
NFC, and NFD expands the number of characters by a factor of between 2 
and 3, affecting every single character. For other big languages such as 
Spanish, Portuguese, French, German, Italian, Polish, Japanese,... data 
is also essentially in NFC, although the characters that would be 
decomposed are less frequent than the 100% for Korean.

> If it's not required, it should be avoided.

That's similar to the argument about saving bytes on the wire. It's not 
compelling in a world with efficient libraries and faster and faster 
processors.

Also, please note that the code at 
http://www.w3.org/2003/06/xml1.1test/Overview.html isn't really the only 
way to optimize, and it's just a proof of concept, not really optimized 
based on actual benchmarks. My guess is that there are things that are 
"overoptimized" (i.e. too complicated without a corresponding speed 
gain) and some spots that could still be further optimized. In many 
cases, checking for all-ASCII first can shave off quite a bit of time. 
Also, doing a lookup of the string (which I guess has to happen sooner 
or later anyway) before checking for normalization may speed up things.

> And I haven't seen any compelling arguments

That may not be compelling on the level of XMPP, but on the level of the 
IETF or the Internet, it's a different issue.

Also, under the assumption that actual characters where NFKC makes a 
difference are few and far between, NFKC and NFC are quite close. A 
change from NFKC to NFC may be much smoother than a change from NFKC to NFD.

> compression.  (see: http://xmpp.org/extensions/xep-0138.html)

Agreed.

> - More likely to render correctly.  Not with today's font renderers.

Depends on the exact characters in question. Can go either way.

> Are there any others I'm missing?

See above (libraries, frequency of data in NFC vs. NFD,...).

Regards,   Martin.