Subject: [Idna-update] Heads up -- Arabic Normalization or rendering issue -- draft UTR 53

From: John C Klensin <[email-masked]>
Date: Thu, 05 Oct 2017 13:35:00 -0400

Hi.

I call the attention of this group to draft UTR#53, "UNICODE
ARABIC MARK ORDERING ALGORITHM".   I'm not going to try to
summarize it because people should read it for themselves, but
it _may_ have some or all of the following implications:

* Special normalization rules, based on combining classes and
inconsistent with current Unicode canonical normalization, may
be needed to Arabic script when combining marks are used.
That would presumably have significant impact on IDNA.  On the
other hand, if we continue to require that all strings that are
input to IDNA be in NFC form, this might be irrelevant to even
if it is important for Arabic 

* If certain marks are used, Arabic script requires more
attention to rendering that current Unicode specifications
assume.  That probably would not affect IDNA because we do no
deal with rendering at all.

* Independent of the proposed solution, the document suggests
that a situation exists that is the opposite of "confusability",
i.e., a single valid U-label might look very different, and
possibly even be interpreted as different strings, depending on
whether the label is in NFC form or has has something like this
algorithm applied after normalization.  

I will leave other IDNA implications to the imagination (and
further research), but it is possible that we need to build
contextual rules around combining classes.   On the other hand,
if combining sequences were banned entirely with Arabic script,
as has been proposed a few times, this (real or potential)
problem would disappear for IDNs.  Precis would probably be
another story.

If this really is intended as a normalization add-on or post
processing step, it seems to me it is really a new normalization
form or set of forms (additional to NFC, NFD, NFKC, and NFKD) in
disguise and that treating it as such might be a lot less
problematic than trying to be sure UAOA is applied (if needed)
after each time one of the traditional normalizations is applied.

Note that this document is out for public review.  Anyone having
comments on it should send them directly to Unicode as specified
in http://www.unicode.org/review/pri359/.   The document is
probably worth discussing on this list only with regard to
IDNA-specific effects.

    john

---

From: Mark Davis ☕️ <[email-masked]>
Date: Thu, 5 Oct 2017 19:53:06 +0200

I believe there is a misunderstanding of this. It is the preordering of
marks as input to the purpose of rendering, and has nothing to do with the
interchanged form of text or any of the normalization forms. Others have
already pointed out that the document as it stands does not make that
sufficiently clear.

Mark

Mark

(https://twitter.com/mark_e_davis)

On Thu, Oct 5, 2017 at 7:35 PM, John C Klensin <[email-masked]> wrote:

> _______________________________________________
> IDNA-UPDATE mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/idna-update
>

---

From: "Patrik Fältström" <[email-masked]>
Date: Thu, 05 Oct 2017 10:54:58 -0700

On 5 Oct 2017, at 10:35, John C Klensin wrote:

> If this really is intended as a normalization add-on or post processing step, it seems to me it is really a new normalization form or set of forms (additional to NFC, NFD, NFKC, and NFKD) in disguise and that treating it as such might be a lot less

After having spent 15 seconds thinking about what you describe, I have two thoughts:

1. You are correct in your view related to normalization

2. Can someone *really* knowing arabic script _and_ IDNA speak up?

   Patrik

---

From: "Patrik Fältström" <[email-masked]>
Date: Thu, 05 Oct 2017 10:57:04 -0700

...but if this implies something being rendered as ABC can be stored either as ABC or ACB, then it do have implications.

That said, if you say the document should in next version make the implications and application be more clear, that is of course appreciated, and I am looking forward to a new version.

   Patrik

On 5 Oct 2017, at 10:53, Mark Davis ☕️ wrote:

> I believe there is a misunderstanding of this. It is the preordering of marks as input to the purpose of rendering, and has nothing to do with the interchanged form of text or any of the normalization forms. Others have already pointed out that the document as it stands does not make that sufficiently clear.
>
> Mark
>
> Mark
>

---

From: John C Klensin <[email-masked]>
Date: Thu, 05 Oct 2017 14:22:36 -0400

--On Thursday, October 5, 2017 10:57 -0700 Patrik Fältström
<[email-masked]> wrote:

> ...but if this implies something being rendered as ABC can be
> stored either as ABC or ACB, then it do have implications.

When B and C are combining marks, that has always been possible.
It is one important reason we require that strings be normalized
before IDNA processing and why that is important.
Interestingly, as W3C moves more and more toward "normalize only
when actually needed", it is an additional reason why
(hypothetical) implementations that push whatever they get
through Punycode (without any checking or preprocessing) and
then look that up can get themselves in false-negative trouble.
I haven't identified any such implementations, but have heard
claims that they are, or should be, out there.
 
> That said, if you say the document should in next version make
> the implications and application be more clear, that is of
> course appreciated, and I am looking forward to a new version.

I'll let Mark comment on how likely this is to go anywhere.  I
don't have even a guess.    But, if the answer is that it won't
until and unless the document is clear that it is only about
rendering and has no effect otherwise, I (and others) will
mostly stop worrying.

As Mark's note implies, the present draft uses terms like
"normalization" and "canonicalization" in ways that are
confusing about its objectives.

    john

---

From: Mark Davis ☕️ <[email-masked]>
Date: Thu, 5 Oct 2017 20:22:27 +0200

The rendering steps already has to process the input for any complex
script. See https://www.microsoft.com/typography/OpenTypeDev/USE/intro.htm,
which discusses the fairly complex reordering already done for Indic
scripts. That doesn't mean that the original text is modified in order to
do that; it is just transient processing.

Mark

Mark

On Thu, Oct 5, 2017 at 7:57 PM, Patrik Fältström <[email-masked]> wrote:

> ...but if this implies something being rendered as ABC can be stored
> either as ABC or ACB, then it do have implications.
>
> That said, if you say the document should in next version make the
> implications and application be more clear, that is of course appreciated,
> and I am looking forward to a new version.
>
>    Patrik
>

---

From: Asmus Freytag <[email-masked]>
Date: Thu, 5 Oct 2017 13:39:30 -0700

My strong feedback to the UTC was to make the following point crystal clear:

On 10/5/2017 11:22 AM, Mark Davis ☕️ wrote:
> script. See 
> https://www.microsoft.com/typography/OpenTypeDev/USE/intro.htm, which 
> discusses the fairly complex reordering already done for Indic 
> scripts. That doesn't mean that the original text is modified in order 
> to do that; it is just transient processing.
>
> Mark
>

I'm glad therefore, to see this clarification, but the discussion on 
this list shows that not being careful in scoping the draft was a 
serious omission.

Let's hope that this can be fixed before the wrong idea of the intended 
scope becomes accepted..

A./

More comments:
> Mark
>
>
> //
>
> On Thu, Oct 5, 2017 at 7:57 PM, Patrik Fältström <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     implications and application be more clear, that is of course
>     appreciated, and I am looking forward to a new version.
>
You can do more than that - there's a simple online form that allows you 
to register your input (except that the next meeting of the UTC is just 
about imminent, so time is of the essence). John gave the details:
>
>
>     > On Thu, Oct 5, 2017 at 7:35 PM, John C Klensin
>     <[email-masked] <mailto:[email-masked]>> wrote:
>     >
>     >> in http://www.unicode.org/review/pri359/
>     <http://www.unicode.org/review/pri359/>.  The document is
>     >> IDNA-specific effects

And neither does Unicode - the variability between low-end and high-end 
typography is too great to allow the standard to give anything close to 
comprehensive guidance to implementers in rendering and fonts.

What you will find in Unicode is generally aimed at helping establish 
the *identity* of the coded character. Part of that identity is when to 
use it (over some possibly competing code point). in complex scripts, 
some of the edge cases are not trivial and therefore Unicode tends to 
call them out.
>     >> algorithm applied after normalization.

That is relatively unsurprising. Being in NFC does not mean the absence 
of combining marks; and therefore mixtures of precomposed characters and 
combining marks exist.

For Latin/Greek/Cyrillic it is generally enough to decompose this input 
to NFD to get explicit combining marks presented in an ordering that 
matches the stacking behavior.

In complex scripts the methods used to force ordering of combining marks 
is too simplistic (and at times froze historic mistakes, because not all 
scripts were understood well enough when Normalization had to be made 
stable).

The draft specification simply tells implementers how to arrive at an 
ordering that naturally relates to how rendering engines will need their 
input presented to them.
>     >> contextual rules around combining classes.
>

In the general case contextual rules are needed for combining marks.

You only have to look at the WLE rules and context rules in LGR-2 to see 
that carried out for SEA scripts. (see 
https://www.icann.org/resources/pages/root-zone-lgr-2015-06-21-en ).

For many other scripts, a generic context rule disallowing two 
*non-spacing* combining marks being repeated might make sense. The aim 
is to disallow cases where low-end rendering systems would overprint 
rather than stack, or where stacked diacritics are clipped. For the 
latter, a limit of the total number of diacritics on the same side of 
the base character would also be needed.

For Thai, we had a situation where we could demonstrate that required 
stacking did not take place in some combinations not used in standard 
Thai (but in minority languages with large user populations). We had to 
disallow these for the root.

For languages where bast+accent form members of an alphabet, the 
recommended method should be to allow these only as part of an explicit 
sequence. (The total set of these needed for modern living languages 
even in a script as widely used as Latin is not expected to exceed a few 
dozen at most -- everything else is overkill).

See the discussion in MSR-2 (sections 4 and 5)
https://www.icann.org/en/system/files/files/msr-2-overview-14apr15-en.pdf

>      On the other hand,

If it wasn't for backwards compatibility, for Arabic, IDNA should simply 
disallow ALL combining marks: there is no way to build a secure LGR 
using combining marks in Arabic (in the general case). This is not just 
my opinion after having done the exercise of trying to define the needed 
variant relations, but you find in RFC 5564 and the most recently 
released LGRs for Arabic for the Root Zone and the Second Level.

Perhaps updating RFC 5564 or providing a more general recommendation 
would be a better approach than updating IDNA itself.

---

From: Patrik Fältström <[email-masked]>
Date: Thu, 5 Oct 2017 14:11:20 -0700

> On 5 Oct 2017, at 11:22, John C Klensin <[email-masked]> wrote:
> 

Oh yes, of course. I meant “after the normalization people are supposed to do before IDNA and the strings touch dns”.

I.e. it all depends on in what order all of these transformations take place, and if that is the slightest unclear, or if it impacts “the normalized string used in dns” then we have a so called serious situation.

  Patrik

---

From: John C Klensin <[email-masked]>
Date: Thu, 05 Oct 2017 19:38:04 -0400

--On Thursday, October 5, 2017 14:11 -0700 Patrik Fältström
<[email-masked]> wrote:

> Oh yes, of course. I meant "after the normalization people
> are supposed to do before IDNA and the strings touch dns".
> 
> I.e. it all depends on in what order all of these
> transformations take place, and if that is the slightest
> unclear, or if it impacts "the normalized string used in
> dns" then we have a so called serious situation.

I think IDNA2008 (RFC 5891 Sections 4.1 and 5.2) are actually
crystal clear that, whatever else one does, the string MUST be
in NFC form before entering IDNA processing.  Now, if that is
ignored or people are confused because of all of the IDNA-ish
things we have floating around, that is a separate issue, but
the problem isn't the order of transformations or any ambiguity
in IDNA2008.

   john