Subject: [Idna-update] Proposal - A new normalization mechanism to protect against confusables

From: Rodger Combs <[email-masked]>
Date: Mon, 17 Apr 2017 01:29:57 -0500

Hello, all!

I've been reading a lot about IDN homoglyph attacks lately, and how browser handling is generally deficient in this area in certain edge cases. Most recently, there's been a lot of attention to the issue of domain labels that are entirely within a single script, but contain all glyphs that are confusable with those in another script. Firefox and Chrome display the characters and allow the homoglyph attack, while Safari prevents the recently-discussed case by not allowing Cyrillic in IDN. Chromium's solution is to display Punycode if a label is entirely composed of a hardcoded list of confusable-with-Latin characters. I find all of these behaviors insufficient:

Chromium's new solution relies on its hardcoded list being sufficient, and one could imagine some edge case where a legitimate label is composed entirely of the listed characters.
Firefox's and Chrome's current behavior is clearly problematic, as it allows for the attack.
Safari's behavior completely fails users of Cyrillic, Greek, and Ethiopic scripts, none of which are whitelisted.

I've been thinking of potential solutions to this problem. In particular, I'd like it to protect against attacks on all domains (not just ASCII ones), and also protect against single-script confusables. I've come up with a proposal, and I hope to get some feedback on it to determine what complexities or edge cases I'm sure I'm overlooking.

Proposed change:

The ToASCII and ToUnicode algorithms remain the same, for the purposes of storing and displaying domain labels as strings. However, ToASCII is replaced with a new algorithm for the purposes of making DNS lookups.
A new algorithm, ToDNSASCII, is defined, as follows:
Perform steps 2 and 3 of ToASCII.
Decompose the sequence using Unicode normalization form KD.
Remove all combining code points from the sequence.
Replace all instances of TR39 source-characters with their corresponding target-strings in the sequence.
Perform steps 5 through 8 of ToASCII.
This algorithm effectively produces a "folded" version of the string, similarly to a collation normalization. It is a lossy procedure, so the resulting string must not be treated as a "canonical" name or displayed to the user as if it were.
When sending a label to a resolver, the ToDNSASCII algorithm is used instead of ToASCII. If the resolver returns NXDOMAIN, retry using the original ToASCII algorithm.
When registering domains, writing zone files, or in similar circumstances, perform both ToASCII and ToDNSASCII and store the results of both if they differ.
For all existing registered domains, register ToDNSASCII(ToUnicode(domain)). If this name is already registered, default precedence to the older registration, and notify the registrant of the newer domain.

This meets all of my goals:
Allows any script or mix of scripts in any domain, with the owner able to define any compatible representation as "canonical"
Protects against all homoglyph attacks listed in TR39, including single-script ones (e.g. "l" vs "1", which is entirely ASCII)
Prevents confusion based on diacritics, without preventing their use
Does not break compatibility with current user software

Potential issues include (but are not limited to):
Requires significant work in browsers, e-mail clients, and other software that handles IDN
Requires significant work from registrars and potentially DNS providers, particularly during the initial migration
Reopens the migration can of worms whenever TR39 is updated
Makes DNS's representation of names impossible to convert losslessly back to the canonical Unicode
May create conflicts between domain owners who previously considered their names legitimately distinct, but collide under ToDNSASCII

Personally, I believe the security and usability benefits of this concept would outweigh its costs if it were widely accepted, but I'll be the first to admit that I'm little more than an armchair enthusiast in the relevant topics, and would be entirely unsurprised if there were unresolvable complexities I haven't foreseen, or if the ones I have were impossible to overcome.

Well... there you have it! Any feedback is appreciated!
--Rodger

---

From: Wil Tan <[email-masked]>
Date: Tue, 18 Apr 2017 03:41:00 +1000

Rodger,

There are many issues that you have not listed, but the main showstopper is
this:

On Mon, Apr 17, 2017 at 4:29 PM, Rodger Combs <[email-masked]>
wrote:

>
>    - When sending a label to a resolver, the ToDNSASCII algorithm is used
>    instead of ToASCII. If the resolver returns NXDOMAIN, retry using the
>    original ToASCII algorithm.
>
> Updating resolver behaviour in this fashion is going to open up an ugly
can of worms.

Your mention of ToASCII and ToUnicode indicates that you may be looking at
an outdated set of specifications (IDNA 2003). The updated specifications
are generally known as IDNA 2008, and you may want to start from RFC5894,
then 5890 - 5893.

.wil

---

From: Rodger Combs <[email-masked]>
Date: Mon, 17 Apr 2017 13:20:33 -0500

> On Apr 17, 2017, at 12:41, Wil Tan <[email-masked]> wrote:
> 
> Rodger,
> 
> There are many issues that you have not listed, but the main showstopper is this:
> 
> On Mon, Apr 17, 2017 at 4:29 PM, Rodger Combs <[email-masked] <mailto:[email-masked]>> wrote:
> Updating resolver behaviour in this fashion is going to open up an ugly can of worms.

I'm actually surprised to hear that, since the concept allows for backwards compatibility with existing behavior, both from the user-agent's and the nameserver's perspective. I would've expected a much larger can of worms in the transition process for registrars. What issues would you anticipate with this particular change?

> 
> Your mention of ToASCII and ToUnicode indicates that you may be looking at an outdated set of specifications (IDNA 2003). The updated specifications are generally known as IDNA 2008, and you may want to start from RFC5894, then 5890 - 5893.

Ah, you're quite right; I was looking at RFC3490 and completely failed to notice that it was marked as obsolete. I'll go update the Wikipedia article on the topic (which still has a lot of 2003 terminology).
It looks like the current procedures are sufficiently similar to the 2003 ones that the added steps I proposed could be inserted between the steps in sections 5.3 and 5.4 of RFC5891, rather than replacing step 4 of ToASCII.

---

From: Asmus Freytag <[email-masked]>
Date: Mon, 17 Apr 2017 12:57:36 -0700

The suggestion:

   * Remove all combining code points from the sequence.

is itself a non-starter.

A./

On 4/17/2017 11:20 AM, Rodger Combs wrote:
>
>> On Apr 17, 2017, at 12:41, Wil Tan <[email-masked] 
>> <mailto:[email-masked]>> wrote:
>>
>> Rodger,
>>
>>       * When sending a label to a resolver, the ToDNSASCII algorithm
>>         is used instead of ToASCII. If the resolver returns NXDOMAIN,
>>         retry using the original ToASCII algorithm.
>>
> I'm actually surprised to hear that, since the concept allows for 
> backwards compatibility with existing behavior, both from the 
> user-agent's and the nameserver's perspective. I would've expected a 
> much larger can of worms in the transition process for registrars. 
> What issues would you anticipate with this particular change?
>
>>
>> Your mention of ToASCII and ToUnicode indicates that you may be 
>> looking at an outdated set of specifications (IDNA 2003). The updated 
>> specifications are generally known as IDNA 2008, and you may want to 
>> start from RFC5894, then 5890 - 5893.
>
> Ah, you're quite right; I was looking at RFC3490 and completely failed 
> to notice that it was marked as obsolete. I'll go update the Wikipedia 
> article on the topic (which still has a lot of 2003 terminology).
> It looks like the current procedures are sufficiently similar to the 
> 2003 ones that the added steps I proposed could be inserted between 
> the steps in sections 5.3 and 5.4 of RFC5891, rather than replacing 
> step 4 of ToASCII.
>
>>
>> .wil
>>
>
>
>
> _______________________________________________
> IDNA-UPDATE mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/idna-update

---

From: Rodger Combs <[email-masked]>
Date: Mon, 17 Apr 2017 20:48:17 -0500

> On Apr 17, 2017, at 14:57, Asmus Freytag <[email-masked]> wrote:
> 
> The suggestion:
> 

That step is more or less separable from the rest of the concept, but I'm curious why you say that. Is it common for legitimate domains to differ only by combining code points after NKFD? Note that this would not prevent _display_ of diacritics.

>>>> On Mon, Apr 17, 2017 at 4:29 PM, Rodger Combs <[email-masked]> wrote:

---

From: Rodger Combs <[email-masked]>
Date: Mon, 17 Apr 2017 20:48:17 -0500

---

From: Asmus Freytag <[email-masked]>
Date: Mon, 17 Apr 2017 19:42:09 -0700

On 4/17/2017 6:48 PM, Rodger Combs wrote:
>
> On Apr 17, 2017, at 14:57, Asmus Freytag <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>> The suggestion:
>>
> That step is more or less separable from the rest of the concept, but 
> I'm curious why you say that. Is it common for legitimate domains to 
> differ only by combining code points after NKFD?

Yes.

> Note that this would not prevent _display_ of diacritics.

Your choice of terms tells me that you think combining marks and 
diacritics are the same thing.

A./
>>>> On Mon, Apr 17, 2017 at 4:29 PM, Rodger Combs 
>>>> <[email-masked] <mailto:[email-masked]>> wrote:
>>>>
>>>>       * When sending a label to a resolver, the ToDNSASCII
>>>>         algorithm is used instead of ToASCII. If the resolver
>>>>         returns NXDOMAIN, retry using the original ToASCII algorithm.
>>>>
>>>> looking at an outdated set of specifications (IDNA 2003). The 
>>>> updated specifications are generally known as IDNA 2008, and you 
>>>> may want to start from RFC5894, then 5890 - 5893.
>>>
>>> Ah, you're quite right; I was looking at RFC3490 and completely 
>>> failed to notice that it was marked as obsolete. I'll go update the 
>>> Wikipedia article on the topic (which still has a lot of 2003 
>>> terminology).
>> [email-masked] <mailto:[email-masked]>

---

From: John C Klensin <[email-masked]>
Date: Mon, 17 Apr 2017 23:57:56 -0400

--On Monday, April 17, 2017 01:29 -0500 Rodger Combs
<[email-masked]> wrote:

> Hello, all!
> 
> I've been reading a lot about IDN homoglyph attacks lately,
> and how browser handling is generally deficient in this area
> in certain edge cases. Most recently, there's been a lot of
> attention to the issue of domain labels that are entirely
> within a single script, but contain all glyphs that are
> confusable with those in another script.

Rodger, I hope you understand that is really old news, with
examples introduced into ICANN meetings by 2002.  You should
also not make the mistake of assuming that UTS#39 either covers
all important cases or represents consensus in the IDNA and DNS
communities.

>  Firefox and Chrome
> display the characters and allow the homoglyph attack, while
> Safari prevents the recently-discussed case by not allowing
> Cyrillic in IDN.

One of the sources of this "problem" is that Greek, Latin, and
Cyrillic are very closely related due to the derivation of the
latter two and a lot of borrowed, even if then somewhat
transfored characters, facts known to many schoolchildren.
However, their are not the only set of scripts with such
relationships and the very notion of a "script" and whether two
related writing systems use the same script or different once
can be a little subject and arbitrary, not unlike the question
of whether two groups of people speak different languages or
dialects of the same one.   It is probably also worth noting
that almost every script uses a more or less vertical stroke
and/or a more or less horizontal or diagonal one, and/or some
sort of round thing as a letter or digit.   So, yes, one can
solve some problems by rejecting Cyrillic and even more by
rejecting Cyrillic and Greek script or, in principle, Latin
script and either of the other two.  One can then start looking
for other sets (pairs of more) or scripts and figure out which
ones are most important.  But, sooner rather than later, one
doesn't have IDNs at all but a moderately small collection of
scripts that are preferred, perhaps for political reasons.

> Chromium's solution is to display Punycode if
> a label is entirely composed of a hardcoded list of
> confusable-with-Latin characters.

That is the Greek-Latin-Cyrillic European bias described above.
It is probably unreasonable to compare everything with Latin and
then assume one is done.   One can (and we do) have similarities
or confusion among other sets of scripts, especially those with
relatively recent common origins, as well.

> I find all of these behaviors insufficient

I hope you don't think you are going to find a single, magical,
solution because, due to both the way various scripts developed
from each other and then are used in different ways in different
languages and to issue with human perception and artistic type
styles, there is almost certainly no such solution.

> Chromium's new solution relies on its hardcoded list being
> sufficient, and one could imagine some edge case where a
> legitimate label is composed entirely of the listed characters.
> Firefox's and Chrome's current behavior is clearly
> problematic, as it allows for the attack. Safari's behavior
> completely fails users of Cyrillic, Greek, and Ethiopic
> scripts, none of which are whitelisted.

Yes.  Again, the quest for magical and simple solutions is never
going to yield completely satisfactory results.

> I've been thinking of potential solutions to this problem. In
> particular, I'd like it to protect against attacks on all
> domains (not just ASCII ones), and also protect against
> single-script confusables. I've come up with a proposal, and I
> hope to get some feedback on it to determine what complexities
> or edge cases I'm sure I'm overlooking.
> 
> Proposed change:
> 
> The ToASCII and ToUnicode algorithms remain the same, for the
> purposes of storing and displaying domain labels as strings.
> However, ToASCII is replaced with a new algorithm for the
> purposes of making DNS lookups. A new algorithm, ToDNSASCII,
> is defined, as follows:

See Will's note about IDNA2008.  Note that by forbidding symbols
and everything else that is not analogous to a letter or digit,
it eliminates many cases your analysis doesn't appear to have
even gotten to yet.

> Remove all combining code points from the sequence.

Especially with NFKD, your eliminated a lot of useful and
important characters.  Latin script provides particular lurid
examples: with a few exceptions like "ø" (U+00F8) (and the fact
that it is an exception is actually a problem), you've
eliminated substantially all Latin-script letters that are in
any way "decorated" (diacritical marks of all flavors included
but note Asmus's second note).  As Asmus says, non-starter.

> Replace all instances of TR39 source-characters with their
> corresponding target-strings in the sequence. Perform steps 5
> through 8 of ToASCII.
> This algorithm effectively produces a "folded" version of the
> string, similarly to a collation normalization. It is a lossy
> procedure, so the resulting string must not be treated as a
> "canonical" name or displayed to the user as if it were. When
> sending a label to a resolver, the ToDNSASCII algorithm is
> used instead of ToASCII. If the resolver returns NXDOMAIN,

During the development of IDNA (both versions) the DNS folks
were very clear (for good reason, IMO) that a systems that
relied on looking up one string and, if that failed, looking up
another was a complete non-starter for both performance and DNS
design philosophy reasons.   Variations on what you are
suggesting were considered and rejected for that and other
reasons.

>  When registering
> domains, writing zone files, or in similar circumstances,
> perform both ToASCII and ToDNSASCII and store the results of
> both if they differ. 

This doesn't work either.  It is a variation on the idea of
"variant" as most recently interpreted by ICANN and depends on
both names being registered to the same party.  That words
better for some protocols than others, can be harder to manage
(at least for things other than web sites) than people imagine,
and we've had history of arbitrators and courts separating pairs
of names that the original registrants thought were bound
together forever.  Your idea at least has the advantage over
separate others of guaranteeing that problem labels come in
pairs rather than creating combinatorial explosions at a single
subtree of the DNS.
 
> For all existing registered domains,
> register ToDNSASCII(ToUnicode(domain)). If this name is
> already registered, default precedence to the older
> registration, and notify the registrant of the newer domain.

Unless I misunderstand you, as soon as both strings are
registered, especially to two different parties, you create an
opportunity for exactly the attacks you are trying to prevent.
Or, if you mean "block the proposed new registration" when you
say "notify the registrant", you've blocked a lot of perfectly
reasonable names for no really good reason.  Again, you don't
even need to go outside Latin script to see the problem:
Applying NFKD to "für" gives U+0066 U+0075 U+0308 U+0072.
Then, if I understand your algorithm correctly, you discard the
combining diaeresis, leading to "fur" for which there appear to
be second-level registrations in a number of domains.  And many
people would argue that "für" and "fur" are not even
confusable, at last unless one believes that any decorated
character can be confused with the associated base one and any
other decorated character with the same base.

>...

best,
   john

---

From: Rodger Combs <[email-masked]>
Date: Tue, 18 Apr 2017 00:00:55 -0500

Thanks for the detailed response; I really appreciate the context! I know this is a complex problem, and I hope I haven't been involuntarily implying that it seemed otherwise. I have no delusions of my concepts approaching perfection; merely that with some work, improvements could be made over what browsers currently implement.

A few specific replies inline, but here's a refreshed proposal that I believe addresses the technical concerns you brought up:

Make no change to names as used in DNS.
Instead, each registry maintains an internal database of "folded" variants of domains, and refuses new domains which "fold" to a value matching that of an existing one.

This solves backwards-compatibility, performance, DNS design, and "variant" management problems. Remaining concerns include (but remain distinctly not limited to):

Registries would actually have to do this.
If domain pairs differing only by combining marks are indeed commonly legitimate, then the relevant stripping would need to be removed (see bottom reply).
The incomplete nature and lack of consensus around UTS#39.
The action to take if existing domains collide.
The action to take if existing domains became colliding after future updates to the confusables list.
Exceptions like U+00F8.

At this point, I think the best thing I could do is probably grab a list of domains (.com, perhaps), write up a script to perform this algorithm, and see what collisions come out. If they turn out to be predominantly legitimate, then there's probably no point pursuing this further. If there are very few, or the bulk appear to be illegitimate, then this might be worth hashing out in more detail. Does that seem reasonable?

> On Apr 17, 2017, at 22:57, John C Klensin <[email-masked]> wrote:
> 
> 
> 

Certainly so; it just happens to have had a bit of new attention lately that I noticed. I wasn't aware (but am not surprised) that there's controversy around UTS#39, though.

> scripts that are preferred, perhaps for political reasons.

Yup, aware, and that preference (which is seen in Chromium, Safari, and others today) is what I was seeking to avoid.

> relatively recent common origins, as well.

Yup.

> going to yield completely satisfactory results.

Not completely, but I can hope for better!

> it eliminates many cases your analysis doesn't appear to have

Ah, very nice.

> but note Asmus's second note).  As Asmus says, non-starter.

(I do know diacritics don't include all combining marks, but I'd gotten tired of typing "combining code points", "character" felt incorrect, and "marks" felt weird since CGJ doesn't have a displayed glyph, though now that I look in more detail not-on-mobile I see the FAQ uses "combining marks"...)
I'm still interested in what cases this specific removal actually prevents. Asmus mentioned pairs of legitimate domains differing only in diacritics, which is about the only response here that's actually surprised me so far. Do you know any specific examples of this case?

> suggesting were considered and rejected for that and other

Extremely fair, and combined with the next point, an unresolvable blocker on the entire concept as proposed.

> other decorated character with the same base.

Ah, I was unclear here. I meant that for existing registrations, a registrar would give the folded value to the older registrant only, and block future registrations where the folded value was already registered.
I was indeed explicitly intending for "für" to be considered a collision with an existing registration "fur" and be blocked. If this body believes that decorated characters cannot in general be confused with the base or other decorated variants, then yes, the "remove combining code points" step should be removed.

Thanks,
--Rodger

---

From: "Patrik Fältström" <[email-masked]>
Date: Tue, 18 Apr 2017 08:56:44 +0200

> That step is more or less separable from the rest of the concept, but I'm curious why you say that. Is it common for legitimate domains to differ only by combining code points after NKFD? Note that…

There are many scripts that require use of combining characters for a correct display of the script.

So this is as Asmus wrote a non-starter.

The evaluation of classes of code points have already been dealt with in IDNA2008.

   Patrik

---

From: "Patrik Fältström" <[email-masked]>
Date: Tue, 18 Apr 2017 09:09:05 +0200

There are many initiatives that have started, and failed, and restarted, and it is important when trying to "do something" to note what has been going on, and as John and Asmus writes between the lines not extrapolate from the script(s) you know to other scripts as (also as John explains) not only are the scripts different. They are also added in slightly different ways to the Unicode Character Set.

To try to summarize we have a number of issues here:

1. People seems to use either IDNA2003, IDNA2008 or some home-brew combination of IDNA2003, IDNA2008 and UTS#46. It is quite important that we all agree that IDNA2008 is what defines what characters can be used in domain names.

2. For mapping, case folding etc, we know already from IDNA2003 that this is locale and otherwise context dependent. Some work have been done in IETF to look at this, UTS#46 includes some suggested mappings and normalizations, but also too much other cruft that one simply can not allow if one follows [1] above, use IDNA2008.

3. Policies must be developed by the registries and also browser and otherwise software so that surprises are minimized. This must of course take [1] and [2] into account, but for example the fact that registries have allowed all of IDNA2008 PVALID characters for registration and no additional rules like "all code points [in the same label or whole domain name] must be the same script" is what in this case have created some of the problem. At least such rules might have made the situation easier. The LGR process in ICANN (we can have different views on efficient, effective and correct it has been) was created as the intention to "do the right thing" script by script.

I am extremely happy to "see a new face" being interested in characters and IDN because to be honest, the number of individuals that have been working on these issues have been very limited. And we need to be more people that think, HARD, on these problems. And the problems are serious.

   Patrik

---

From: John C Klensin <[email-masked]>
Date: Tue, 18 Apr 2017 03:59:41 -0400

--On Monday, April 17, 2017 20:48 -0500 Rodger Combs
<[email-masked]> wrote:

> That step is more or less separable from the rest of the
> concept, but I'm curious why you say that. Is it common for
> legitimate domains to differ only by combining code points
> after NKFD?

Yes.

> Note that this would not prevent _display_ of

One of the things that drove the IDNA2008 work was the
discovery, reinforced by considerable experience, that not
having a reversible mapping between what we now call U-label and
A-label forms caused all sort of problems and ambiguities.
Unless I misunderstand your proposal, you are going to
reintroduce that problem.  Do try to understand the IDNA2008
specs.  While you are at it, I suggest reading RFC 4690.

  best,
    john

---

From: Gervase Markham <[email-masked]>
Date: Tue, 18 Apr 2017 09:03:39 +0100

On 18/04/17 06:00, Rodger Combs wrote:
>   * Make no change to names as used in DNS.
>   * Instead, each registry maintains an internal database of "folded"
>     variants of domains, and refuses new domains which "fold" to a value
>     matching that of an existing one.

That is the correct solution, and is what Mozilla is arguing for (with
"fold" being defined by the Unicode confusables list). Now your only
problem is persuading registries to implement it.

Gerv

---

From: Rodger Combs <[email-masked]>
Date: Tue, 18 Apr 2017 03:17:02 -0500

> On Apr 18, 2017, at 01:56, Patrik Fältström <[email-masked]> wrote:
> 

I completely understand this, but it's besides the point. What I'm suggesting at this point is that registries effectively ban new registrations that differ from existing registrations only in combining marks, _not_ that combining marks themselves not be allowed in actual domains, and I never suggested that they not be _displayed_.
The only case this would break would be non-malicious registrations that differ only by combining marks. There's a valid debate to be had over whether or not that's a useful thing to restrict against, but let's not confuse the issue being discussed there.

I'm looking through an all-zones dump for collisions that differ by combining marks only. I can't hope to read through all of these myself, but so far I've run into a couple common classes of collision:

The same registrant setting up their base domain and one or more decorated forms. My original proposal would've addressed this use-case, but even in the new one, this is easily addressed: allow colliding registrations when both are by the same registrant. See vísa.com <http://xn--vsa-rma.com/>, coffrèo.biz <http://xn--coffro-7ua.biz/>.
Deliberate homograph attacks on popular domains with diacritics added. This seems to frequently use í and ì, as they look very similar to the root glyph (i) without close inspection.
Most of these that I've seen are just parked domains. See wikipédia.org <http://xn--wikipdia-f1a.org/>, doubleclìck.net <http://xn--doubleclck-g8a.net/>, doubleclíck.net <http://xn--doubleclck-r8a.net/>, pízza.com <http://xn--pzza-vpa.com/>, tésla.com <http://xn--tsla-bpa.com/>, kíckstarter.com <http://xn--kckstarter-k8a.com/>, wíne.com <http://xn--wne-rma.com/>.
A few have been weird, though. See pokémongo.com <http://xn--pokmongo-d1a.com/>, linkedín.com <http://xn--linkedn-dza.com/>, and páypal.com <http://xn--pypal-xqa.com/> (which looks like it might seriously be a phishing site of some sort).

If this email gets flagged as spam for all those links, I'm gonna laugh.

---

From: Martin J. Dürst <[email-masked]>
Date: Tue, 18 Apr 2017 17:50:51 +0900

> The only case this would break would be non-malicious registrations that differ only by combining marks. There's a valid debate to be had over whether or not that's a useful thing to restrict again…

I don't know of any actual registered examples, but in the languages 
that use accents/diacritics/combining marks, these are usually there to 
make a distinction. The distinction can be related to grammar (e.g. 
German "Buch" (book) vs. "Bücher" (books), or it may distinguish totally 
unrelated things (e.g. German "Durst" (thirst) vs. "Dürst" (a family name)).

How much is grammar and how much is unrelated stuff will depend on the 
language. As far as I understand, for Arabic, it's mostly grammar, for 
German, it's both, and for other languages, it may be most if not all 
unrelated stuff.

Of course, in no language all the letter combinations are words, and the 
chance to find a minimal pair (a pair of words that differs only in 
combining marks) is rather low, in particular if we limit ourselves to 
nouns, names, and other things that work in domain names. However, where 
there is such a pair, native readers will usually be well aware of it.

Regards,   Martin.

---

From: John C Klensin <[email-masked]>
Date: Tue, 18 Apr 2017 05:11:54 -0400

--On Tuesday, April 18, 2017 09:03 +0100 Gervase Markham
<[email-masked]> wrote:

>>   * Instead, each registry maintains an internal database of
>>   "folded" variants of domains, and refuses new domains which
>>     "fold" to a value matching that of an existing one.
> 
> That is the correct solution, and is what Mozilla is arguing
> for (with "fold" being defined by the Unicode confusables
> list). Now your only problem is persuading registries to
> implement it.

Gerv,

Good to know you are still watching this list.
 
I would have said "part of a correct solution", not a complete
solution.   It is fully consistent with something IDNA2008 calls
for although it does not specif9ically require it in that form:
That registries take final responsibility for understanding the
characters they allow and, more generally, what they are doing
(see draft-klensin-idna-rfc5891bis, which should probably be
updated to explicitly call out not allow labels in the same zone
that can be confused with each other.

However, that gets tangled up with two other problems, one of
which makes its applicability harder and that other of which has
turned into a long-simmering dispute:

(1) ICANN has been going in the direction of what they refer to
as "variants" in which similar or confusable strings are
identified.  All but one can then be blocked (consistent with
your proposal) or one or more can be delegated as long as they
are delegated to the same party (see my previous note about one
problem with that).  The idea of delegating similar labels
rather than blocking them seems to me to be in direct
contradiction of your proposal.

(2) Identifying strings as similar and then blocking all but one
of them from registration can cause a lot of false negatives as
people's guesses about how a particular string (e.g., one they
see in printed form) is transposed into the DNS turn out to be
inconsistent with the particular variation the registrant
selected or the registry allowed.   False negatives are, I trust
obviously, much better than false positives, especially from a
security standpoint, but users not being able to find the names
they think they are looking for upsets both them and
registrants.  That issue is one of the things that has driven
the desire for for widespread mapping of multiple characters
onto a single one, even though that causes its own set of
problems, as discussed in one of my earlier notes.

There is another issue that, depending on one's point of view,
either makes the above hard or brings us right back to where you
(wisely, IMO) started in this area.  We have some empirical
evidence that there are registrars and registries out there that
either don't care or even that have built their business models
on selling confusion (e.g., "you need to buy this label because
otherwise it might be confused with one you care about and then
cause you harm").  There is even evidence that this is not
"some" registrars but the majority of them and a lot of
registries who either tolerate them or hide behind readings of
ICANN guidelines and contracts that don't allow them to reject
registrations.  AFAICT, the only solutions available to a
browser developer who considers that behavior dangerous are 

(a) Keep a list of registries who behave in a careful and
reasonable fashion and insist that registrars selling names in
their domains do so too.   Everyone else sees labels displayed
in A-label form, threatening warning messages, strings that you
simply refuse to look up, or a combination.   That was, IIR,
your idea and where the idea that browsers should take
responsibility if ICANN and registries will not started.  Note,
however, that this approach may be of limited use if bad
behavior occurs below the second level of the DNS tree.

(b) Browser vendors, users, and those with security concerns
increase the pressure on ICANN (and, for ccTLDs, relevant
governments) to push back strongly on registries and registrars
who won't take responsibility or who otherwise encourage
problematic collections of labels to be registered.  To me, such
a  change in policy is the single most important step that could
be taken at this stage and it has the advantage of not requiring
changing code (even if it were to require phasing out some
existing labels that now exist only for confusion-causing or
confusion risk minimization purposes.   However, unless and
until there is a major outcry, it seems obvious that this idea
will go nowhere -- there is no profit in blocking a label, at
least unless one can charge a fee for doing so.  Some recent
episodes on the ICANN side in which registries have taken the
position that they can and will go ahead and register labels
that clearly violate IDNA2008 (and, in important cases, even
IDNA2008) registration rules because they think they can make
money (often expressed as "uses want these" and ICANN can't or
won't stop them reinforce that cynical view.

(c) Either the same parties in (b) or others start resorting to
litigation or criminal charges against registries that allow
registrations that actually cause harm.   There may be countries
in which deliberately misleading users is allowed and encouraged
by law and normal commercial practices, but, if there is, I
don't know where it is.  Turning confusing registrations from a
positive-sum game from registries and registrars into a highly
risky one would change the discussion in very significant, and
IMO useful, ways.  Whether efforts should be made to involve
ICANN as a party to those actions -- especially now that it
doesn't have whatever protection the US Government relationship
brought-- is an interesting, but separate question.

Three more observations for Rodger:

(i) As you sort of point out now, UTS#39 does not cover all of
the cases that might reasonably be considered relevant.  Whether
that is a serious defect, opening up systems to attacks it does
not catch, or a collection of edge cases that fall under the
"can't find a perfect solution to all of this but it is best to
do what one can" is a matter of perspective and opinion.
Similarly, focusing on "combining marks" creates some dependency
on seemingly-arbitrary (and/or language-dependent within the
same script) decisions by Unicode about what single-code-point
representations for characters do and do not decompose.  That is
the reason why the U+00F8 case is important: in that character,
the slash is not a diacritical mark so there is a presumption
that registration of a string with "o" does not block one with
"ø" from being registered.  However, because "ö" does
decompose, it would be blocked.   Whether we like it or find it
appropriate for IDNs or not, there are sound Unicode coding
reasons for that distinction, but explaining them to users is a
real issue (see draft-klensin-idna-5892upd-unicode70 for a more
general exploration of the problem, but note that it is
officially expired and that there is an updated replacement that
has not been posted).

(ii) It seems that your latest approach bypasses the problem
that got you (and others) started, which is the ability to
construct entire labels on one script that look a lot like
labels constructed entirely in a different one.    That is still
covered by the principle that registries need to be careful and
take responsibility for not allowing two labels in the same zone
that could pose difficulties of confusion.

(iii) You should be careful about NFKD (or NFKC), since some of
the so-called compatibility equivalents may not be appropriate
substitutions in all cases.  Again, as a way of figuring out
that there are a pair (or more) of labels only one of which
should be allowed to be registered in a given zone, it may be
fine.  As a way to determine which one should be chosen, there
are several subcategories of compatibility equivalents, some of
which seem globally reasonable and others of which include
problematic and usage-specific cases.   IIR, we discussed that
issue in RFC 4690 as well.

   best,
     john

---

From: Gervase Markham <[email-masked]>
Date: Tue, 18 Apr 2017 10:19:08 +0100

> contradiction of your proposal.

I don't think so. I think either bundling or blocking solves the
problem, and am happy with either or both, at the registry's choice. The
key thing is that two visually-identical domains should not be owned by
two different parties.

> problems, as discussed in one of my earlier notes.

Sure. But this is a registry UX problem. Probably the best method and
the most user-friendly one would be bundling of all variants, and if you
type any one of them in, you get the opportunity to register the entire
bundle, ideally at the same price as a single name. Or, if they charge
per-name, you can pick some, and the others will be blocked. That's what
a responsible registry would do, I feel.

> registrations. 

I can imagine this is true. I suspect that only public pressure, and
pressure from their customers (domain owners who don't want to be
attacked) will change this. And the first step in that is putting
responsibility where it belongs.

> their domains do so too.

As you may know, Firefox used to adopt that policy. However, the gTLD
explosion made us abandon it as non-scalable. See:
https://wiki.mozilla.org/IDN_Display_Algorithm

> IDNA2008) 

Typo here?

Gerv

---

From: John C Klensin <[email-masked]>
Date: Tue, 18 Apr 2017 05:26:41 -0400

--On Tuesday, April 18, 2017 17:50 +0900 "Martin J. Dürst"
<[email-masked]> wrote:

>...
> Of course, in no language all the letter combinations are
> words, and the chance to find a minimal pair (a pair of words
> that differs only in combining marks) is rather low, in
> particular if we limit ourselves to nouns, names, and other
> things that work in domain names. However, where there is such
> a pair, native readers will usually be well aware of it.

And, of course, there are lots of domain names that are not
words but that are, instead, abbreviations, acronyms, codes, or
assorted constructions that might be words but aren't.  Consider
both constructions like "MyBigCompany.example" and even popular
TLDs like "com", "biz", "int", and even "arpa" and almost all
ccTLDs.  Even "no mixed script" rules and strings that are not
visually confusable run afoul of other principles.  For example,
there is a large toy store operation, with many stores in the US
and some on other places, that has a trademark on "toys-я-us".
I'm confident that they believe they have a right to register
that string, and its non-hyphenated variation, in the DNS
regardless of whether they have done so or not and that they
have the right (and perhaps obligation) to block registration of
any of those strings and "toys-r-us" etc. from being registered
and used by anyone else.  "toys-я-us" would be blocked by a "no
mixed script labels" rule, but I'm also fairly confident that
they wouldn't have to look very long to find a judge who would
agree with them that any registry rules that prevented them from
using their trademark in an obviously important context was
invalid and could not be enforced.

Again, no perfect solutions here and relying on "words" isn't
even close.

    best,
     john

---

From: John C Klensin <[email-masked]>
Date: Tue, 18 Apr 2017 05:29:27 -0400

--On Tuesday, April 18, 2017 10:19 +0100 Gervase Markham
<[email-masked]> wrote:

>  clearly violate IDNA2008 (and, in important cases, even
>> IDNA2008) 
> 
> Typo here?

Tep.  Second one should have been "IDNA2003".  Clear evidence
that I need a nap.  

Sorry.
    john

---

From: "Patrik Fältström" <[email-masked]>
Date: Tue, 18 Apr 2017 12:10:44 +0200

> I completely understand this, but it's besides the point. What I'm suggesting at this point is that registries effectively ban new registrations that differ from existing registrations only in comb…

This sentence above does not make any sense to me, and/or is very latin/greek/cyrillic centric view on what "combing mark" is (which is though not true for all code points in latin/greek/cyrillic, see for example use of Latin Script in Catalan Language or Turkish).

What I think you are talking about is confusability, and lately we have seen cross script confusability. Which is a completely different issue than combining characters, and more related to what the LGR panels have been working with.

   paf

---

From: Slim Amamou <[email-masked]>
Date: Tue, 18 Apr 2017 19:26:59 +0100

In arabic it's for both too

Le 18 avril 2017 09:50:51 GMT+01:00, "Martin J. Dürst" <[email-masked]> a écrit :
>> The only case this would break would be non-malicious registrations
>that differ only by combining marks. There's a valid debate to be had
>over whether or not that's a useful thing to restrict against, but
>let's not confuse the issue being discussed there.
>

-- 
Slim Amamou | سليم عمامو
http://alixsys.com