Subject: How to indicate that the Subject: headers is non-English

From: Scott Petrack <[email-masked]>
Date: Wed, 25 Aug 1999 18:50:13 -0400 (EDT)

I want to send a Subject header in Japanese, Cyrillic, unicode, etc.
The other SIP request headers seem to me invariant, but I need a way to
indicate the language. Can I use a Content-Language: header in a request?
Is there some other mechanism?

Thanks,

Scott

---

From: "Adam B. Roach" <[email-masked]>
Date: Thu, 26 Aug 1999 10:15:54 -0500 (CDT)

>Is there some other mechanism?

I don't know whence it originates, but I've often seen a format
like:

=?<encoding-name>?Q?<header_text>?=

e.g. From: =?ISO-8859-1?Q?Bj=F6rn_Nilsson?= <[email-masked]> 

Can anyone identify the source of this syntax? Does it seems
like a good sort of thing to allow in SIP headers?

--
Adam Roach, Ericsson Inc. |  Ph: +1 972 583 7594 | 1010 E. Arapaho, MS L-04
[email-masked]   | Fax: +1 972 669 0154 | Richardson, TX 75081 USA

---

From: "Dan Swinehart" <[email-masked]>
Date: Thu, 26 Aug 1999 08:49:47 PDT

I think you're looking for

RFC2047: MIME Part Three: Message Header Extensions for
Non-ASCII Text.

Dan Swinehart

---

From: Anders Kristensen <[email-masked]>
Date: Thu, 26 Aug 1999 18:20:25 +0100

> like a good sort of thing to allow in SIP headers?

That would be MIME, more precisely RFC 2047: MIME Part Three: Message
Header Extensions for Non-ASCII Text.

However, this specifies the character set, so that it doesn't have to be
US-ASCII. This is not needed for SIP, as SIP messages are always UTF-8
encoded Unicode, and hence can include pretty much any characters.  I am
not really convinced about the need for an explicit indication of
language. Scott, can you give any details as to why you need this? Isn't
it sufficient to just render the characters?

Cheers,
Anders

-- 
Anders Kristensen <[email-masked]>,
http://www-uk.hpl.hp.com/people/ak/
Hewlett-Packard Labs, Bristol, UK

---

From: Jonathan Lennox <[email-masked]>
Date: Thu, 26 Aug 1999 15:52:17 -0400 (EDT)

> Is there some other mechanism?

According to RFC 2543:

        Subject  =  ( "Subject" | "s" ) ":" *TEXT-UTF8

Subject headers are always UTF-8 (RFC 2279, the US-ASCII-transparent
transformation of Unicode), so you can encode any language you like, except
possibly for questions of Han unification ambiguity (Chinese vs. Japanese
Kanji).  Is there any reason you'd need an encoding *other* than Unicode on
the wire?

Looking over the spec, all free-text strings in SIP (Subject, Organization,
display names, response reasons, etc.) are UTF-8.

I imagine this is a feature which should be tested at the next bakeoff,
since I don't think I've ever seen anyone use non-ASCII characters in SIP
messages.

-- 
Jonathan Lennox
[email-masked]

---

From: Scott Petrack <[email-masked]>
Date: Thu, 26 Aug 1999 19:46:20 -0400 (EDT)

Thanks to everyone for the pointers and answers.  I was actually asked
this by a customer. And you are right that I was asking about both 
character sets and language. I suppose if I were really clever I could
think up a good example. Something like:

Subject: Boot

where I need to say if I am writing German or English. But I personally
can certainly live with just rendering the characters.

Scott

On Thu, 26 Aug 1999, Anders Kristensen wrote:

> Scott, can you give any details as to why you need this? Isn't

---

From: Jonathan Lennox <[email-masked]>
Date: Thu, 26 Aug 1999 21:25:37 -0400 (EDT)

On Thu, August 26 1999, "Scott Petrack" wrote to "confctrl" saying:

> And you are right that I was asking about both 

You could use the Unicode Plane 14 Language Tags
<http://www.unicode.org/unicode/reports/tr7.html>, if you really needed to.

-- 
Jonathan Lennox
[email-masked]

---

From: Henning Schulzrinne <[email-masked]>
Date: Fri, 27 Aug 1999 09:09:56 -0400

See RFC 2482.

---

From: Jonathan Rosenberg <[email-masked]>
Date: Mon, 30 Aug 1999 10:55:46 -0400

How would the application know what language was being written anyway?
Would I have to select "Spanish" before saying that the subject of the
call is "Una fiesta"? What happens if the subject contains a mix of
words from different languages (very common)? In any case, I'm not sure
what the receiving side would do with this information. 

So, I think the right thing to do is just render them as sent, and not
worry about what language its actually in.

Its worth noting that supporting UTF-8 is pretty much a freebie in
proxy/redirect/registrar servers, since they don't need to render or
parse any text that is allowed to be UTF-8. A UTF-8 string with
non-ASCII characters is still an array of null terminated bytes, so your
normal strcmp, strcpy, and so on, will still work fine.

-Jonathan R.

-- 
Jonathan D. Rosenberg                       Lucent Technologies
Member of Technical Staff                   101 Crawfords Corner Rd.
High Speed Networks Research                Holmdel, NJ 07733
FAX:   (732) 834-5379                       Rm. 4C-526
EMAIL: [email-masked]
URL: http://www.cs.columbia.edu/~jdrosen

---

From: Henning Schulzrinne <[email-masked]>
Date: Tue, 31 Aug 1999 09:54:30 -0400

Jonathan Rosenberg wrote:
> words from different languages (very common)?

UTF-8 does allow labeling of substrings by language, so this could be
done, but I agree that this is not likely to be of much use in practice.
One reasonable exception I can think of is that it might be used for
automatic call routing. I suppose a tool could make this part of the
configuration - if I use an English-language tool, the Subject text
would be marked as such. Doesn't deal with me speaking German, but this
is probably not that common. I believe there are even AI tools that
automatically determine the language, but those would probably be better
used at the language-aware ACD than at a client.

>  In any case, I'm not sure

Agreed, although the in-band marking would be harmless if used.

> normal strcmp, strcpy, and so on, will still work fine.

The only exception is strcmp for > or < comparison, e.g., if you want to
sort a caller list.

-- 
Henning Schulzrinne   http://www.cs.columbia.edu/~hgs

---

From: Anders Kristensen <[email-masked]>
Date: Tue, 31 Aug 1999 16:43:41 +0100

Henning Schulzrinne wrote:
> automatic call routing.

One concern might be whether many unicode libraries support the language
tagging mechanism. I'm pretty sure the Java Unicode reader with which
I'm familiar doesn't support getting (or setting) these language tags. 
One alternative might be to assume that the Subject language is the
Accept-Language language with the highest quality value. This is a hack
but it doesn't seem *completely* inappropriate to assume that the
language someone wants response phrases in is also the language in which
(s)he would formulate subjects(?)

Anders

---

From: Kevin Payne <[email-masked]>
Date: Wed, 01 Sep 1999 17:59:22 +1000

> The only exception is strcmp for > or < comparison, e.g., if you want to

The only problem with using strcmp is that it compares the octets, not
the characters that they represent.  Since Unicode allows characters to
be encoded in multiple ways, to compare a word with an "ä", you would
need to take care of the encoding of the character as a single Unicode
character 'ä' or the two characters '¨' and 'a'.

Kevin

---

From: Henning Schulzrinne <[email-masked]>
Date: Wed, 01 Sep 1999 09:28:46 -0400

> character 'ä' or the two characters '¨' and 'a'.

Is there a "canonical" encoding that is somehow preferred? Is there a
wstrcmp() that does this automatically? The only problem I could see is
that if somebody REGISTERs with the two-character version and a call
comes in with the single-character version. Explaining to the customer
why this failed would be rather tricky, since they "look" the same.

---

From: Kevin Payne <[email-masked]>
Date: Thu, 02 Sep 1999 15:44:25 +1000

> why this failed would be rather tricky, since they "look" the same.

Exactly.  AFAIK there is no standardised way of encoding the characters
(then again I'm not an I18N guru).   

As well as the REGISTER example, I can also see problems occuring if a
user wanted to redirect (or accept) calls according to Subject, From
etc.  Kind of embarrasing if your boss's calls always go to voicemail or
get rejected...

It would be really inefficient to have to convert every string from
UTF-8 to a wchat_t string and then process, even if you did so I'm not
sure that this takes care of the multiple encoding problem. 

Kevin

---

From: Kevin Payne <[email-masked]>
Date: Fri, 03 Sep 1999 12:06:00 +1000

> sure that this takes care of the multiple encoding problem.

It would appear that the W3C have addressed this problem and there is a
Uniform Character Model for use on the web, at:
   http://www.w3.org/TR/WD-charmod. 

(This is a working document so of course can't be referenced by a RFC
:-)

In particular, the document refers to using "Unicode Canonical
Composition (Normalization Form C)", which is documented in Unicode
Technical Report #15, another draft report :-(

  http://www.unicode.org/unicode/reports/tr15/tr15-10.html

There is even a PERL 5 script available to normalise (or normalize :-)
UTF-8 data, available at

   http://www.w3.org/International/charlint/ 

To reduce the overhead of normalising UTF-8 data at every place that it
may be compared, it makes sense for SIP to specify that the input
endpoints normalise the data at the edge of network, the various strcmps
will then work much more efficiently.

Kevin