Subject: [sipcore] SIP and Internationalisation

From: Jonathan Hansford <[email-masked]>
Date: Fri, 10 May 2013 12:08:44 +0100

Hi, 

I am currently looking at a number of Internet protocols for
international use and am looking to see how much commonality there is
between them. A number are, or are in the process of becoming, IDNA2008
aware (e.g. DNS - RFC 5890, SMTP - RFC 6530/1, XMPP -
draft-ietf-xmpp-6122bis-07). 

I have been trying to identify what
support for internationalization there is within SIP and can find no
reference to it. Can you advise what level of support (if any) SIP
currently provides for countries whose native language is not
(adequately) supported by US ASCII and/or whether there are any plans to
address this in the future? 

Thanks, 

Jonathan Hansford

---

From: Gunnar Hellstrom <[email-masked]>
Date: Fri, 10 May 2013 17:08:40 +0200

Jonathan,
Most items in SIP are UTF-8 coded, so I think there is good preparedness 
for internationlaization.
I think you should coordinate your activities with this draft, that is 
going in the other direction - trying to find out the minimal common set 
to be used in addressing in multiple services. It might need your 
results as a separate section and its current information may be answer 
on your current question.
https://datatracker.ietf.org/doc/draft-saintandre-username-interop

Regards

Gunnar Hellström

On 2013-05-10 13:08, Jonathan Hansford wrote:
> between them. A number are, or are in the process of becoming, 
> IDNA2008 aware (e.g. DNS - RFC 5890, SMTP - RFC 6530/1, XMPP - 
> I have been trying to identify what support for internationalization 
> there is within SIP and can find no reference to it. Can you advise 
> what level of support (if any) SIP currently provides for countries 
> whose native language is not (adequately) supported by US ASCII and/or 
> whether there are any plans to address this in the future?
>
> Thanks,
>
> Jonathan Hansford
>
>
>
> _______________________________________________
> sipcore mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sipcore

---

From: Peter Saint-Andre <[email-masked]>
Date: Fri, 10 May 2013 09:22:21 -0600

On 5/10/13 9:08 AM, Gunnar Hellstrom wrote:
> Jonathan, Most items in SIP are UTF-8 coded, so I think there is
> good preparedness for internationlaization.

In my experience, no one is ever fully prepared for
internationalization. :-)

> I think you should coordinate your activities with this draft, that
> is going in the other direction - trying to find out the minimal
> common set to be used in addressing in multiple services. It might
> need your results as a separate section and its current information
> may be answer on your current question. 

That is a very early effort, and I deliberately punted on the i18n
aspects. If the work is compelling, I can address i18n in a future
version.

>> international use and am looking to see how much commonality
>> there is between them. A number are, or are in the process of
>> becoming, IDNA2008 aware (e.g. DNS - RFC 5890, SMTP - RFC 6530/1,
>> XMPP - draft-ietf-xmpp-6122bis-07).

In addition to the IDNA2008 work, I strongly recommend that you look
at the output of the PRECIS WG (which generalizes the IDNA2008
approach to identifiers other than domain names, such as the localpart
of a SIP URI):

http://datatracker.ietf.org/wg/precis/

>> I have been trying to identify what support for
>> internationalization there is within SIP and can find no
>> reference to it. Can you advise what level of support (if any)
>> SIP currently provides for countries whose native language is not
>> (adequately) supported by US ASCII and/or whether there are any
>> plans to address this in the future?

I can't speak to that directly, but I'm happy to help.

Peter

- --
Peter Saint-Andre
https://stpeter.im/

---

From: Peter Saint-Andre <[email-masked]>
Date: Fri, 10 May 2013 09:51:20 -0600

> internationalization. :-)

Someone asked me this off-list, but it's useful to discuss it in public...

Thinking that UTF-8 solves your internationalization problems is like
thinking that TLS solves your security problems. I hate to break it to
you, but it's not that simple. Yes, using UTF-8 encoding is better
than being stuck with ASCII, but there are many many challenging
issues with internationalization.

The following slides might help orient folks who are new to i18n:

https://stpeter.im/files/i18n-intro.pdf

Peter

- -- 
Peter Saint-Andre
https://stpeter.im/

---

From: Adam Roach <[email-masked]>
Date: Fri, 10 May 2013 11:16:54 -0500

Jonathan:

As far as I know, there has been no serious work on SIP's i18n story so 
far. The closest we come is the use of UTF-8 everywhere. While an 
international character encoding is a necessary part of the story, it's 
far from sufficient.

I've known for quite some time that SIP's IDN story is conspicuous by 
its absence. I've never really had the time to poke at existing, 
deployed implementations to see what they do when you feed unicode 
characters into domain names. Given the lack of specification, I doubt 
most of them do anything useful. Some might accidentally kind of work, 
depending on what punycode support their underlying DNS libraries 
provide, but the chances of this being done in a way that yields success 
seems extremely slim.

The other area we need to worry about, as Gunnar and Peter hinted at, is 
usernames. The big issue here (as I understand it) is the application of 
stringprep for the purpose of measuring username equality.

I'd been keeping my eyes, very loosely, on the IRI work in hope that we 
could leverage its output in a future SIP internationalization effort, 
since it would hopefully provide guidance on both the IDN and stringprep 
issues. Sadly, that work seems to have been terminated before producing 
useful output, so we'll have to make our own way through this morass.

There are probably some similar issues for password normalization, 
although I'm not sure that's as much of an issue, since passwords, by 
their nature, are generated in a more repeatable way than usernames are. 
I would expect that people who have spent far more time than I have 
thinking about the problem have some ideas here.

Aside from URIs and Digest credentials, there is mercifully little in 
the way of protocol fields intended to be user-visible for SIP. A 
handful, like Organization and Subject, were specified in a way that 
implied that they should be user-renderable; however, in practice, no 
one ever uses these fields (and, in any case, I don't think they pose 
any particular i18n challenges, as these fields are not 
machine-interpretable, and do allow UTF-8 characters).

In practice, most implementations them do a lousy job of l10n (e.g. for 
response Reason-Phrases) , but I don't think we're missing anything 
protocol-wise; this seems to be a failing of the implementations themselves.

I think that's the vast majority of what you need to worry about in 
terms of the core SIP protocol. Now, there are some protocols we use 
(e.g., BFCP, TLS, CCMP, S/MIME) that we would rely on having sufficient 
internationalization support, but I would be inclined to consider those 
a separable effort.

In any case, if you can find enough critical mass of people with the 
energy and time to move this work forward, I would welcome it 
wholeheartedly. I believe that, the way the charter is currently 
written, this work is likely to fall under the purview of the SIPCORE 
working group.

/a

---

From: Peter Saint-Andre <[email-masked]>
Date: Fri, 10 May 2013 11:08:02 -0600

Hi Adam, a few corrections and amplifications inline.

On 5/10/13 10:16 AM, Adam Roach wrote:
> Jonathan:
> 
> As far as I know, there has been no serious work on SIP's i18n
> story so far. The closest we come is the use of UTF-8 everywhere.
> While an international character encoding is a necessary part of
> the story, it's far from sufficient.
> 
> I've known for quite some time that SIP's IDN story is conspicuous
> by its absence. I've never really had the time to poke at
> existing, deployed implementations to see what they do when you
> feed unicode characters into domain names. Given the lack of
> specification, I doubt most of them do anything useful. Some might
> accidentally kind of work, depending on what punycode support their
> underlying DNS libraries provide, but the chances of this being
> done in a way that yields success seems extremely slim.

It's not just punycode (RFC 3492) but the framework that re-uses
punycode. The original framework was IDNA2003 (RFC 3490), which has
been superseded by RFCs 5890-5894. However, the world is migrating to
IDNA2008 in a rather slow fashion, so many applications still use
IDNA2003.

> The other area we need to worry about, as Gunnar and Peter hinted
> at, is usernames. The big issue here (as I understand it) is the
> application of stringprep for the purpose of measuring username
> equality.

IDNA2003 used stringprep, which was tied to Unicode 3.2 (the current
version is 6.2). IDNA2008 ditched stringprep so that it could be agile
with respect to Unicode versions. The PRECIS WG is doing the same for
non-DNS identifiers.

> I'd been keeping my eyes, very loosely, on the IRI work in hope
> that we could leverage its output in a future SIP
> internationalization effort, since it would hopefully provide
> guidance on both the IDN and stringprep issues. Sadly, that work
> seems to have been terminated before producing useful output, so
> we'll have to make our own way through this morass.

The IRI WG was trying to update RFC 3987, but that effort ran out of
steam. However, RFC 3987 still exists and is usable.

> although I'm not sure that's as much of an issue, since passwords,
> by their nature, are generated in a more repeatable way than
> usernames are. I would expect that people who have spent far more
> time than I have thinking about the problem have some ideas here.

See RFC 4103 (SASLprep) and draft-ietf-precis-saslprepbis.

Peter

- -- 
Peter Saint-Andre
https://stpeter.im/

---

From: Peter Saint-Andre <[email-masked]>
Date: Fri, 10 May 2013 11:10:46 -0600

> See RFC 4103 (SASLprep) and draft-ietf-precis-saslprepbis.

Er, RFC 4013.

/psa

---

From: [email-masked] (Dale R. Worley)
Date: Fri, 10 May 2013 14:14:25 -0400 (EDT)

> From: Jonathan Hansford <[email-masked]>
> 

Here's the state of SIP practice that I see:

There isn't much of SIP that is visible to the user, so little of it
has internationalization concerns.

As far as I know, the most significant area is the "name-addr"
construction, which is the enhanced URI that is used to address phone
calls.  That is often used at the receiving phone to display the
caller's name a phone number.  Typical use is something like:

      INVITE sip:[email-masked] SIP/2.0
      From: Alice <sip:[email-masked]>;tag=1928301774
      To: Bob <sip:[email-masked]>

The display-name part, "Alice" and "Bob" is defined to be a series of
UTF-8-encoded Unicode characters (with certain restrictions), so it is
already fully internationalized.  It also has the virtue that while
the protocol carries it, the protocol doesn't *interpret* it.

The protocol treats URIs (e.g., sip:[email-masked]) as sequences of
Unicode characters with very limited variation allowed before one URI
becomes "different" from another.  Generally, we do not expect users
to enter URIs or interpret them, but rather select entries from
address books.

The domain name part (atlanta.com) is part of the URI, and so is
restricted to Ascii characters.  But we have a system for encoding
non-Ascii domain names into Ascii characters.  I'm not familiar with
the details, but it's the same problem as encoding non-Ascii domain
names into HTTP URLs.

The part that is least well worked out (IMO) is the user-part of the
SIP URI; in these examples, "bob" and "alice".  As part of a URI,
user-parts have to be composed of Ascii characters, but there is an
escaping system, where instead of an ordinary character you write
"%xx", where "xx" is two hex digits.

It turns out that what the escape *means* is not well specified, other
than that if you replace a syntactically acceptable Ascii character
with its hex escape, the resulting URI must compare equal to the
original URI.  But it seems that there is an intention that a
non-Ascii character can be represented by writing a series of escapes
representing the bytes in the character's UTF-8 representation.  But
it's likely that at least some systems use the escapes %80 to %FF to
represent the characters U+0080 to U+00FF.

Dale

---

From: Paul Kyzivat <[email-masked]>
Date: Fri, 10 May 2013 15:14:32 -0400

I'll echo Adam, but admit that I know very little about 
internationalization - clearly less than Adam, much less Peter.

I do know enough to recognize that there is something lacking.
If we have a volunteer with the knowledge and motivation to work on it, 
then we should be be able to support the work within sipcore.

I expect this hasn't surfaced as much of a problem because most of the 
use of sip is with phone numbers.

	Thanks,
	Paul

---

From: Peter Saint-Andre <[email-masked]>
Date: Fri, 10 May 2013 15:40:38 -0600

> then we should be be able to support the work within sipcore.

I doubt that we'll find a volunteer with both the motivation and the
knowledge. Instead, I expect that we'll need to find someone with the
motivation and then train that person to have the knowledge. In the XMPP
community, Joe Hildebrand and I had to train ourselves (with much help
from more knowledgeable people like Pete Resnick). There is a lot to
learn in this space (personally I feel like I'm still scratching the
surface), and the motivation will need to be accompanied by the
willingness to do a great deal of self-learning.

> use of sip is with phone numbers.

Probably so.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Torrey Searle <[email-masked]>
Date: Mon, 13 May 2013 09:34:55 +0200

If I understand correctly, the domain name part does have a significant
problem as it's defined as follows:

 ALPHA          =  %x41-5A / %x61-7A   ; A-Z / a-z

hostport         =  host [ ":" port ]
host             =  hostname / IPv4address / IPv6reference
hostname         =  *( domainlabel "." ) toplabel [ "." ]
domainlabel      =  alphanum
                    / alphanum *( alphanum / "-" ) alphanum

Which would prevent the newer domain names in non-latin alphabets from
being legally used in SIP.  Or would SIP require the use of the "ToAscii"
format defined rfc3490?

---

From: Peter Saint-Andre <[email-masked]>
Date: Mon, 13 May 2013 06:24:25 -0600

On 5/13/13 1:34 AM, Torrey Searle wrote:
> being legally used in SIP.  Or would SIP require the use of the
> "ToAscii" format defined rfc3490?

Yes, it would require the punycoded form (an A-label in RFC 5890).

Peter

-- 
Peter Saint-Andre
https://stpeter.im/