Subject: [EAI] Unicode's PRI #185 & EAI: should the UBA be revised to handle bidi email addresses better too?

From: "Aharon (Vladimir) Lanin" <[email-masked]>
Date: Thu, 14 Jul 2011 16:08:58 +0300

Just in case this is news for anyone here, the Unicode Technical
Committee is considering PRI #185: "Revision of UBA for improved
display of URL/IRIs". (UBA stands for Unicode Bidirectional
Algorithm.) The intent of the proposal is to improve the display of
bidirectional IRIs.

The same problems (if not worse) that exist for bidi IRIs also exist
for bidi email addresses. I have submitted a comment on PRI #185
suggesting that it be extended to cover email addresses.

Aharon Lanin

Here is the announcement of PRI #185:

The Unicode Technical Committee has posted a new issue for public
review and comment. Details are on the following web page:

   http://www.unicode.org/review/

Review period for the new item closes on July 27, 2011.

Please see the page for links to discussion and relevant documents.
Briefly, the new issue is:

PRI #185    Revision of UBA for improved display of URL/IRIs

The Unicode Bidirectional Algorithm (UBA), specified in UAX #9, was
designed for handling ordinary text, and predated the rise of the web.
Unfortunately, IRI/URLs are not ordinary text; they are syntactically
complex in ways that don’t work well with the UBA. That causes IRIs
that contain right-to-left text (such as Arabic or Hebrew) to appear
jumbled, to the point where the IRIs are either uninterpretable,
misleading, or ambiguous. In particular the ambiguous displays could
cause security problems.

The background document for this PRI provides a detailed description
of the problem, and proposes a solution. The Unicode Technical
Committee would like feedback on the feasibility of the proposal, and
in particular, on the open issues listed in the background document.

If you have comments for official UTC consideration, please post them by
submitting your comments through our feedback & reporting page:

   http://www.unicode.org/reporting.html

If you wish to discuss issues on the Unicode forum or the Unicode
mail list, then please use the following links to subscribe (if necessary).
Please be aware that discussion comments on the Unicode mail list are
not automatically recorded as input to the UTC. You must use the reporting
link above to generate comments for UTC consideration.

   http://www.unicode.org/forum/
   http://www.unicode.org/consortium/distlist.html

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 14 Jul 2011 15:32:42 +0200

On 07/14/2011 03:08 PM, Aharon (Vladimir) Lanin wrote:
> for bidi email addresses.

Why would the problems be worse for email?

Bad, yes, no disagreement at all. My question is specifically about the 
"if not worse" bit.

> I have submitted a comment on PRI #185

Excellent.

Arnt

---

From: "Aharon (Vladimir) Lanin" <[email-masked]>
Date: Thu, 14 Jul 2011 17:18:33 +0300

It's not an important point. I guess email addresses are just so deceptively
simple. In RTL, [email-masked] (where uppercase
Latin letters stand for RTL characters) is displayed as

[email-masked]

A large number of users will think that they are talking to someone at
a-well-known-bank.com.

Aharon

On Thu, Jul 14, 2011 at 4:32 PM, Arnt Gulbrandsen
<[email-masked]>wrote:

> Bad, yes, no disagreement at all. My question is specifically about the "if
> not worse" bit.
>
>
> ______________________________**_________________
> IMA mailing list
> [email-masked]
> https://www.ietf.org/mailman/**listinfo/ima<https://www.ietf.org/mailman/listinfo/ima>
>

---

From: John C Klensin <[email-masked]>
Date: Thu, 14 Jul 2011 15:31:09 -0400

Aharon,

There are a number of different issues associated with this.  It
is probably useful to try to separate out at least three of them.

(1) Neither the IDNA2008 Standards nor the EMA work take any
position on how domain names or email addresses (including but
not limited to strings that would invoke the various Bidi rules)
are presented to users or provided by them.   So the
suppositions that lead to a display of
"[email-masked]" may be correct, or
not, but the IETF specs don't have an opinion on the matter.
For the record, they also don't formally have an opinion as to
whether, in parts of the world in which family names are
typically written before personal names, etc., the string that
SMTP sees as [email-masked] is displayed as
    
 [email-masked] 
 [email-masked]
 [email-masked]
   or even
 com.example.mail##bloggs.joe
   or some other variation

(where "##" is some relatively arbitrary delimiter).  That
flexibility is even true when the local-part and domain-part
consist exclusively of ASCII characters.  The fact that we don't
see those variations very often depends more on designers
exercising what they consider sensible behavior than on anything
that the standards or working drafts do or do not require.

(2) While many of those involved are not aware that they are
making the assumption and understand its implications even less,
many folks around ICANN are assuming that FQDNs will be script
homogeneous and directionality-property homogeneousl (maybe even
language-homogeneous, but no one really knows what that means).
They also assume that those homogeneity rules can and will be
enforced, either by DNS software (pretty much impossible) or by
administrative mechanisms (probably possible only if a series of
fairly tough conditions exist administratively).  If one does
believe that myth, a-well-known-bank.com.GEORGE.pl is
essentially not possible.  One could need to have either 
 a-well-known-bank.com.george.pl   or
 A-WELL-KNOWN-BANK.COM.GEORGE.PL
and the more mixed cases are, in the same myth, presumably
expected to atrophy.

(3) Many SDOs, including the IETF, have taken the position that
having other SDOs develop modifications, interpretations, or
even extensions to standards developed by the first body is a
really bad idea (see RFC 5704 for a discussion about a topic
unrelated to i18n).  If this piece's interactions with IRIs and
UTR46 are examples, it has apparently become the norm in the
relatioship between UTC and the IETF.  Perhaps that is A Good
Thing but, if some parties rely only on the IETF specs and
others rely on some combination or a "pick what is convenient"
model, the opportunities for interoperability problems, user
astonishment, and other difficulties are considerable.

best,
     john

--On Thursday, July 14, 2011 17:18 +0300 "Aharon (Vladimir)
Lanin" <[email-masked]> wrote:

> It's not an important point. I guess email addresses are just
> so deceptively simple. In RTL,
> [email-masked] (where uppercase
> A large number of users will think that they are talking to
> someone at a-well-known-bank.com.
> 
> Aharon
> 
>>> Just in case this is news for anyone here, the Unicode
>>> Technical Committee is considering PRI #185: "Revision of
>>> UBA for improved display of URL/IRIs". (UBA stands for
>>> Unicode Bidirectional Algorithm.) The intent of the proposal
>>> is to improve the display of bidirectional IRIs.
>>> 
>>> The same problems (if not worse) that exist for bidi IRIs
>>> also exist for bidi email addresses.
>>> 
>> 
>> Bad, yes, no disagreement at all. My question is specifically
>> about the "if not worse" bit.
>> 
>>

---

From: "Aharon (Vladimir) Lanin" <[email-masked]>
Date: Sun, 17 Jul 2011 09:35:13 +0300

> not, but the IETF specs don't have an opinion on the matter.

I am not suggesting that they should. I was just bringing a related
matter to the attention of the people on this list. The fact is that
your garden-variety mailer UI applies the UBA to email addresses
without treating them differently from plain text.

> fairly tough conditions exist administratively).

I personally heartily wish that this were true, but don't put much
faith in it. That's why I think that a UBA change that would prevent
bidi-itis in email addresses would be very significant.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 29 Jul 2011 19:35:40 +0900

[This is a copy of a comment that I submitted via the Unicode Review 
Form and posted on the member-only Unicode mailing list. I'm sending it 
here to have a public record, because this's the mailing list where most 
of the discussion about this draft in the IETF happened, as far as I'm 
aware of.]

Context
=======

I'm an individual Unicode member, but I'll paste this in to the 
reporting form because that's easier. Please make a 'document' out of it 
(or more than one, if that helps to better address the issues raised 
here). I apologize for being late with my comments.

Substantive Comments
====================

On substance, I don't agree with every detail of what Jonathan Rosenne, 
Behdad Esfahbod, Aharon Lanin and others have said, I agree with them in 
general. If their documents/messages are not properly submitted, I 
include them herewith by reference.

The proposal is an enormous change in the Bidi algorithm, changing its 
nature in huge ways. Whatever the details eventually may look like, it 
won't be possible to get everything right in one step, and probably 
countless tweaks will follow (not that they necessarily will make things 
better, though). Also, dealing with IRIs will increase the 
appetite/pressure for dealing with various other syntactical constructs 
in texts.

The introduction of the new algorithm will create numerous compatibility 
issues (and attack surfaces for phishing, the main thing the proposal 
tries to address) for a long period of time. Given that the Unicode 
Consortium has been working hard to address (compared to this issue) 
even extremely minor compatibility issues re. IDNs in TR46, it's 
difficult for me to see how this fits together.

Taking One Step Back
====================

As one of the first people involved with what's now called IDNs and 
IRIs, I know that the problem of such Bidi identifiers is extremely 
hard. The IETF, as the standards organization responsible for 
(Internationalized) Domain Names and for URIs/IRIs, has taken some steps 
to address it (there's a Bidi section in RFC 3987 
(http://tools.ietf.org/html/rfc3987#section-4), and for IDNs, there is 
http://tools.ietf.org/html/rfc5893).

I don't think these are necessarily sufficient or anything. And I don't 
think that the proposal at hand is completely useless. However, the 
proposal touches many aspects (e.g. recognizing IRIs in plain text,...) 
that are vastly more adequate for definition in another standards 
organization or where a high-bandwidth coordination with such an 
organization is crucial (roughly speaking, first on feasibility of 
various approaches, then on how to split up the work between the 
relevant organizations, then on coordination in details.) Without such a 
step back and high-bandwidth coordination, there is a strong chance of 
producing something highly suboptimal.

(Side comment on  detail: It would be better for the document to use 
something like
http://tools.ietf.org/html/rfc3987#section-2.2 rather than the totally 
obscure and no longer maintained 
http://rfc-ref.org/RFC-TEXTS/3987/chapter2.html, in the same way the 
Unicode Consortium would probably prefer to have its own Web site 
referenced for its work rather than some third-party Web site.)

Taking Another Step Back
========================

I mention 'high-bandwidth' above. The Unicode "Public Review" process is 
definitely not suited for this. It has various problems:
- The announcements are often very short, formalistic, and cryptic
   (I can dig up examples if needed.)
- The announcements go to a single list; forwarding them to other
   relevant places is mostly a matter of chance. This should be improved
   by identifying the relevant parties and contacting them directly.
- To find the Web form, one has to traverse several links.
- The submission is via a Web form, without any confirmation that the
   comment has been received.
- The space for comments on the form is very small.
- There is no way to make a comment public (except for publishing it
   separately).
- There is no official response to a comment submitted to the Web form.
   One finds out about what happened by chance or not at all.
   (compare to W3C process, where WGs are required to address each
    comment formally, and most of them including the responses are
    public)
- The turnaround is slow. Decisions get made (or postponed) at UTCs
   only.
Overall, from an outsider's point of view, the review process and the 
review form feel like a needle's ear connected to a black hole.

[I very much understand that part of the reason the UTC works the way it 
works is because of its collaboration with ISO/IEC committees. And I 
don't think any other standards organization has a perfect process. But 
what's appropriate for one part of the UTCs work may not be appropriate 
for other parts of its work (such as the matter at hand).]

Conclusion
==========

I herewith very strongly recommend that the UTC, besides using the 
upcoming meeting to advance discussion on the technical issues that the 
proposal raises,
a) Postpone the decision to adopt any of the proposed changes, 
independent of details, until such time as point b) is implemented and 
executed.
b) Swiftly take the necessary steps for a much better, high-bandwith 
coordination of this topic and the various issues it encompasses, both 
using the existing liaison mechanism and using public discussion on an 
appropriate forum (e.g. one of the relevant IETF mailing lists 
(idna/eai/iri)).
c) Seriously work on improving the process for soliciting and addressing 
comments from the public and relevant stakeholders.

Regards,    Martin.