Subject: [Ietf-calsify] Character set restriction in section 4.3.11 Text

From: bernard.desruisseaux at oracle.com (Bernard Desruisseaux)
Date: Mon Aug 21 11:39:48 2006

In section "4.3.11 Text" of RFC 2445 it says:

 > Formal Definition: The character sets supported by this revision of
 > iCalendar are UTF-8 and US ASCII thereof. The applicability to other
 > character sets is for future work. The value type is defined by the
 > following notation.

I don't see any reason why it states that only UTF-8 and US-ASCII
are supported by this revision. Section "3.2 Parameter" doesn't
specify any restriction on the values allowed for the "charset"
parameter.

I propose to remove this statement from section 4.3.11.

Proposed new text:

 > Formal Definition: The value type is defined by the following notation.

Cheers,
Bernard

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Mon Aug 21 11:47:40 2006

> I propose to remove this statement from section 4.3.11.

Can you explain why you feel that it is important for new work to support 
any character set other than UTF-8?  If any revision is made, it should be 
to delete US-ASCII and make UTF-8 mandatory.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

---

From: alexey.melnikov at isode.com (Alexey Melnikov)
Date: Mon Aug 21 11:51:30 2006

Mark Crispin wrote:

> Can you explain why you feel that it is important for new work to 
> support any character set other than UTF-8?

+1

>   If any revision is made, it should be to delete US-ASCII and make 
> UTF-8 mandatory.

However we can't do this, as absence of the charset parameter means 
US-ASCII.
So any iCalendar object in UTF-8 MUST have charset=utf-8.

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Mon Aug 21 11:55:59 2006

On Mon, 21 Aug 2006, Alexey Melnikov wrote:
>>   If any revision is made, it should be to delete US-ASCII and make UTF-8 
>> mandatory.
> However we can't do this, as absence of the charset parameter means US-ASCII.

That's unfortunate.  I admit that I am not up to speed on this 
specification; is there really no way to fix this problem?

If there is some way of defining the charset as always being UTF-8, then 
charset parameters can be removed from the specification as extraneous.

-- Mark --

http://panda.com/mrc

---

From: alexey.melnikov at isode.com (Alexey Melnikov)
Date: Mon Aug 21 12:10:40 2006

> specification; is there really no way to fix this problem?

To clarify, the default charset=US-ASCII comes from MIME specs.
I am not sure if text/calendar can change the default character set 
used. Somebody with better MIME expertise should clarify this.

> If there is some way of defining the charset as always being UTF-8, 
> then charset parameters can be removed from the specification as 
> extraneous.

Yes, that would be nice.

On a related note, according to my interpretation of MIME, the following 
text in draft-ietf-calsify-rfc2445bis-01 is not quite correct:

 >3.1.4.  Character Set
 >
 >   There is not a property parameter to declare the character set used
 >   in a property value.  The default character set for an iCalendar
 >   object is UTF-8 as defined in [RFC2279].
 >
 >   The "charset" Content-Type parameter can be used in MIME transports
 >   to specify any other IANA registered character set.

IMHO, charset is required for UTF-8 case.

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Mon Aug 21 12:44:03 2006

On Mon, 21 Aug 2006, Alexey Melnikov wrote:
> I am not sure if text/calendar can change the default character set used. 
> Somebody with better MIME expertise should clarify this.

If the message is not in email, then the specification can do whatever it 
wants even if it inherits some syntax from MIME.

Even if it is in email, I think that the CALENDAR subtype can define 
independent semantics from TEXT/PLAIN.  Note that CHARSET is meaningless 
in TEXT/HTML.

If you're really worried, then use APPLICATION/CALENDAR.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: alexey.melnikov at isode.com (Alexey Melnikov)
Date: Mon Aug 21 13:52:59 2006

Mark Crispin wrote:

> If the message is not in email, then the specification can do whatever 
> it wants even if it inherits some syntax from MIME.
>
> Even if it is in email,

I think it is.

> I think that the CALENDAR subtype can define independent semantics 
> from TEXT/PLAIN.  Note that CHARSET is meaningless in TEXT/HTML.

The advantage of not assuming charset=utf-8 for text/calendar is that 
applications that don't recognize text/calendar can still do something 
meaningful with it (like displaying it to the user):
[from RFC 2046]

4.1.4.  Unrecognized Subtypes

   Unrecognized subtypes of "text" should be treated as subtype "plain"
   as long as the MIME implementation knows how to handle the charset.
   Unrecognized subtypes which also specify an unrecognized charset
   should be treated as "application/octet- stream".

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Mon Aug 21 14:22:57 2006

Well, then, which is more important; having calendar fall back to 
TEXT/PLAIN behavior (in which case you need CHARSET), or having calendar 
objects be required to be UTF-8?

Another thing that you could do is that TEXT/CALENDAR could require that 
;CHARSET=UTF-8 be present as a parameter.  Declare that a TEXT/CALENDAR 
which omits CHARSET, or has any charset other than UTF-8, is undefined and 
not to be interpreted according to the calendar specification.

By the way, all of these are to be treated as random ideas.  I'm not 
pushing for any of this as a solution, although I believe that it would be 
highly desirable to prevent any new application from getting into the 
multi-charset business.

-- Mark --

---

From: lists at block-online.eu (Oliver Block)
Date: Mon Aug 21 15:08:44 2006

> In section "4.3.11 Text" of RFC 2445 it says:

3.3.11 of draft-ietf-calsify-rfc2445bis-01

>  > Formal Definition: The value type is defined by the following notation.

<quote>
   The specification for any future subtypes of "text" must specify
   whether or not they will also utilize a "charset" parameter, and may
   possibly restrict its values as well.  For other subtypes of "text"
   than "text/plain", the semantics of the "charset" parameter should be
   defined to be identical to those specified here for "text/plain",
   i.e., the body consists entirely of characters in the given charset.
   In particular, definers of future "text" subtypes should pay close
   attention to the implications of multioctet character sets for their
   subtype definitions.
</quote>

That might have been the reason for the definition. It continues:

<quote>
   The charset parameter for subtypes of "text" gives a name of a
   character set, as "character set" is defined in RFC 2045.  The rules
   regarding line breaks detailed in the previous section must also be
   observed -- a character set whose definition does not conform to
   these rules cannot be used in a MIME "text" subtype.

   An initial list of predefined character set names can be found at the
   end of this section.  Additional character sets may be registered
   with IANA.
</quote>

(RFC2046, Section 4.1.2, p. 7,8)

It would be helpful at least to mention that the used character set must 
registered charsets (RFC2978).

Best Regards,

Oliver

-- 
Leben ist mehr als schneller - weiter - h?her
http://www.nak-nrw.de/p_6_4.html

---

From: lists at block-online.eu (Oliver Block)
Date: Mon Aug 21 15:23:22 2006

> registered charsets (RFC2978).

Sorry for that English.

---

From: bernard.desruisseaux at oracle.com (Bernard Desruisseaux)
Date: Mon Aug 21 19:29:45 2006

Hi Mark,

I don't feel that it is important for new work to support any
character set other than UTF-8, but I'm not sure whether rfc2445bis
should be considered as new work though!

In my proposal I only quoted section 4.3.11 which makes clear that
UTF-8 and US-ASCII are the only character sets supported by iCalendar.
I should probably have quoted the other sections where character sets
are discussed, that is, sections 2.3, 3.2, and more importantly 4.1.4:

Section 4.1.4 Character Set

 > to specify any other IANA registered character set.

Section 4.3.11 says that only UTF-8 and US-ASCII are supported.
While section 4.1.4 says that you can specify any IANA registered
character set in the "charset" parameter...

The result is that some iCalendar applications currently output
iCalendar objects in ISO-8859-1.

I see two options:

a) We modify section 4.3.11 as I proposed. But now I realize
    that we would also need to modify all the ABNF rules that
    makes reference to the numerical value of specific characters
    (e.g., LF = %x0A in US-ASCII but %x25 in EBCDIC). Yuk!

b) We modify sections 3.2, 4.1.4 and 4.3.11 to be in sync and
    clearly specify which character sets are supported by iCalendar:

     b.1) UTF-8 only.
     b.2) UTF-8 and US-ASCII only.
     b.3) UTF-8, US-ASCII, and ISO-8859-1 only.

At this point I favor option (b.3). It looks to me as the easiest
way to clarify the specification without breaking most (if not all)
existing iCalendar applications.

Cheers,
Bernard

Mark Crispin wrote:
> support any character set other than UTF-8?  If any revision is made, it 
> should be to delete US-ASCII and make UTF-8 mandatory.
> 
> -- Mark --
> 
> http://panda.com/mrc

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Mon Aug 21 21:25:52 2006

> iCalendar objects in ISO-8859-1.

How many of these are there?  Is this widespread?

> At this point I favor option (b.3).

I strongly oppose any proposal that grants special status to ISO-8859-1.

Leaving aside the fundamental wrongness of grantly clemency to lazy 
European implementors without doing the same for lazy Russian, Chinese, 
Japanese, Korean, etc. implementors, ISO-8859-1 (Latin-1) is deprecated. 
As all of us should be well-aware of by now, the approved ISO 8859 variant 
for use in the EU is ISO-8859-15 (Latin-9 a.k.a. Latin-0).

If ISO-8859-1 were to be exempted, even if just for legacy purposes, 
sooner or later some ding-dong will complain about "ignorant Americans" 
whose "broken" specifications "deny" Europe use of several European 
characters and even the EU currency sign.  Clearly a conspiracy by George 
Bush, the NRA, Wal-Mart, Coca-Cola, and the Walt Disney company... ;-)

We don't know that there aren't any lazy implementor out there that have 
done an iCalendar application using KOI8-R, or GBK, or ISO-2022-JP (more 
likely, Shift-JIS), or KSC.  If we take a firm stand and say "iCalendar is 
Unicode only", we would have moral authority to stand firm should such 
lazy implementors complain.  We'd lose it if we had already made a special 
exception for ISO-8859-1.

We don't always avoid mistakes; but this is one we can definitely 
recognize now and avoid.

Thus, I believe that our choices are:
  . prohibit all charsets other than UTF-8 and US-ASCII (and, to be
    honest, I'd prefer to prohibit US-ASCII as well)
  . allow any charset

If ISO-8859-1 has truly reared its ugly head into calendar and can't be 
exterminated, then I regrettably conclude that "allow any charset" is the 
only viable alternative.

-- Mark --

http://panda.com/mrc

---

From: alexey.melnikov at isode.com (Alexey Melnikov)
Date: Tue Aug 22 03:04:55 2006

Mark Crispin wrote:

> European implementors without doing the same for lazy Russian, 
> Chinese, Japanese, Korean, etc. implementors, ISO-8859-1 (Latin-1) is 
> deprecated. As all of us should be well-aware of by now, the approved 
> ISO 8859 variant for use in the EU is ISO-8859-15 (Latin-9 a.k.a. 
> Latin-0).
>
> sooner or later some ding-dong will complain about "ignorant 
> Americans" whose "broken" specifications "deny" Europe use of several 
> European characters and even the EU currency sign.  Clearly a 
> conspiracy by George Bush, the NRA, Wal-Mart, Coca-Cola, and the Walt 
> Disney company... ;-)
>
> We don't know that there aren't any lazy implementor out there that 
> have done an iCalendar application using KOI8-R, or GBK, or 
> ISO-2022-JP (more likely, Shift-JIS), or KSC.  If we take a firm stand 
> and say "iCalendar is Unicode only", we would have moral authority to 
> stand firm should such lazy implementors complain.  We'd lose it if we 
> had already made a special exception for ISO-8859-1.

I agree.

>    honest, I'd prefer to prohibit US-ASCII as well)

I wouldn't go as far as prohibiting US-ASCII, but I agree otherwise.

> If ISO-8859-1 has truly reared its ugly head into calendar and can't 
> be exterminated, then I regrettably conclude that "allow any charset" 
> is the only viable alternative.

---

From: cyrus at daboo.name (Cyrus Daboo)
Date: Tue Aug 22 06:31:59 2006

Hi Mark,

> How many of these are there?  Is this widespread?

Outlook - need I say more?

-- 
Cyrus Daboo

---

From: cbryant-ical at corp.usa.net (Chris Bryant)
Date: Tue Aug 22 06:54:54 2006

What version of Outlook?  Outlook 2003 seems to encode properly in UTF-8 
from what I've looked at.

I agree that if we need to make an exception for 8859-1, then we need to 
allow any charset, but I don't think we should allow any charsets other than 
UTF-8 and US-ASCII.

Chris

----- Original Message ----- 
From: "Cyrus Daboo" <[email-masked]>

> Hi Mark,
>
> --On August 21, 2006 9:25:28 PM -0700 Mark Crispin 
> <[email-masked]> wrote:
>

---

From: alexey.melnikov at isode.com (Alexey Melnikov)
Date: Thu Aug 24 05:13:13 2006

Mark Crispin wrote:

> TEXT/PLAIN behavior (in which case you need CHARSET), or having 
> calendar objects be required to be UTF-8?
>
> Another thing that you could do is that TEXT/CALENDAR could require 
> that ;CHARSET=UTF-8 be present as a parameter.

I think this is the best way.

> Declare that a TEXT/CALENDAR which omits CHARSET, or has any charset 
> other than UTF-8, is undefined and not to be interpreted according to 
> the calendar specification.

I wouldn't go as far as saying that they are not covered by the calendar 
specification.
For example I find nothing wrong with a calendar that is entirely in 
US-ASCII and has to CHARSET parameter.

> pushing for any of this as a solution, although I believe that it 
> would be highly desirable to prevent any new application from getting 
> into the multi-charset business.

Right.