Subject: [Ietf-calsify] Status of simplification work

From: TimHare at comcast.net (Tim Hare)
Date: Tue May 23 18:27:42 2006

It's been quite a while since I saw any of the simplification topics
addressed on this mailing list. I know some of the work seems to be being
addressed at the Interops hosted by the Calendaring & Scheduling Consortium,
but it's been a while since I've seen notice of a draft, or discussion of
issues here on the mailing list. 

I wonder if someone could post a quick summary of where the work is in
regard to Calsify, which of the substantial issues appear to have some
resolution, etcetera.

Thanks

Tim

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osafoundation.org/pipermail/ietf-calsify/attachments/20060523/cc003448/attachment.htm

---

From: lear at cisco.com (Eliot Lear)
Date: Wed May 24 07:42:03 2006

Hello Tim from the Calconnect Roundtable in Boston,

[CHAIR HAT OFF]

I believe the authors are working to deliver updates based on
discussions in Dallas.  This having been said I'm curious as to what you
believe needs to be simplified.  We've heard here at CalConnect some
issues relating to floating times and RRULEs.  What are your pet peeves?

Eliot

---

From: gsexton at mhsoftware.com (George Sexton)
Date: Wed May 24 11:57:40 2006

I just joined the list, so I don't know if this has been addressed.

Section 4.1 of RFC-2445 states:

.. Lines of text SHOULD NOT be longer than 75 octets...

Since the preferred character set is UTF-8, this is pretty difficult to 
do. To actually limit lines correctly would require an intrinsic 
function in the language to determine the length of each character, and 
if the current length + current char length > 75,wrap the line.

I program mostly in Java, and I don't know any intrinsic functions that 
give you the length in bytes of a specific Unicode character.

In the spirit of simplification, if the wrap recommendation is kept then 
it should be revised to say 75 characters.

-- 
George Sexton
MH Software, Inc.
Voice: +1 303 438 9585
URL:   http://www.mhsoftware.com/

Eliot Lear wrote:
> _______________________________________________
> Ietf-calsify mailing list
> [email-masked]
> http://lists.osafoundation.org/mailman/listinfo/ietf-calsify
>
>   

-- 
George Sexton
MH Software, Inc.
Voice: +1 303 438 9585
URL:   http://www.mhsoftware.com/

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Wed May 24 12:27:25 2006

On Wed, 24 May 2006, George Sexton wrote:
> Since the preferred character set is UTF-8, this is pretty difficult to do. 
> To actually limit lines correctly would require an intrinsic function in the 
> language to determine the length of each character, and if the current length 
> + current char length > 75,wrap the line.
>
> I program mostly in Java, and I don't know any intrinsic functions that give 
> you the length in bytes of a specific Unicode character.

Do you really need an intrinsic function, which it is so easy to write 
your own?

The following small C function translates a UCS-4 codepoint, including 
those codepoints which are not in Unicode (that is, those with values 
0x110000 - 0x8fffffff), into the corresponding number of UTF-8 octets.  It 
returns 0 for a codepoint which is not in UCS-4.

unsigned long utf8_size (unsigned long c)
{
   if (c < 0x80) return 1;
   else if (c < 0x800) return 2;
   else if (c < 0x10000) return 3;
   else if (c < 0x200000) return 4;
   else if (c < 0x4000000) return 5;
   else if (c < 0x80000000) return 6;
   return 0;
}

A simpler version, covering only Unicode codepoints, is:

unsigned long utf8_size (unsigned long c)
{
   if (c < 0x80) return 1;
   else if (c < 0x800) return 2;
   else if (c < 0x10000) return 3;
   else if (c < 0x110000) return 4;
   return 0;
}

> In the spirit of simplification, if the wrap recommendation is kept then it 
> should be revised to say 75 characters.

I suspect that the 75 octet length limitation is for buffer purposes in 
transport (as in certain email gateways which one hopes are long extinct) 
and not any sort of "character" limitation.

"Characters" are not particularly useful to limit in any case; that notion 
is hopelessly tied to fixed-width fonts and the notion that a "character" 
corresponds to a fixed-length field of real estate.  You can only go so 
far with that notion, even if you accomodate East Asian characters being 
double-width ("two characters"), and various marks applied to characters 
being zero-width ("zero characters").

A "glyph" limit might be useful, but even that is tied up with the notion 
of fixed-width fonts and is invalidated by scripts such as Arabic.

Of course, to do any of this you have to know how text is actually drawn 
on the screen, which tends not to be something done at the level of 
protocols.  Octet counts, on the other hand, are.

Consequently, I recommend that the wrap recommendation be retained, and 
that sample code such as the above be proved for the benefit of those 
people who have trouble understanding how to determine the octet length of 
a Unicode character in UTF-8.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: gsexton at mhsoftware.com (George Sexton)
Date: Wed May 24 13:47:06 2006

Thanks for the function. Clearly you know a great deal more than I do 
about UTF-8 as a character set.

 >> double-width ("two characters"), and various marks applied to 
characters

I always tend to think of characters as the smallest atomic part that a 
string can be broken into. If I'm doing character based I/O, then a 
character is the smallest component I can write to the stream (all hail 
recursive definitions). I rarely, if ever concern myself with 
typesetting issues.

Mark Crispin wrote:
>> Since the preferred character set is UTF-8, this is pretty difficult 
>> to do. To actually limit lines correctly would require an intrinsic 
>> function in the language to determine the length of each character, 
>> and if the current length + current char length > 75,wrap the line.
>>
>> I program mostly in Java, and I don't know any intrinsic functions 
>> that give you the length in bytes of a specific Unicode character.
>
> 0x110000 - 0x8fffffff), into the corresponding number of UTF-8 
> octets.  It returns 0 for a codepoint which is not in UCS-4.
>
>> In the spirit of simplification, if the wrap recommendation is kept 
>> then it should be revised to say 75 characters.
>
> I suspect that the 75 octet length limitation is for buffer purposes 
> in transport (as in certain email gateways which one hopes are long 
> extinct) and not any sort of "character" limitation.
>
> "Characters" are not particularly useful to limit in any case; that 
> notion is hopelessly tied to fixed-width fonts and the notion that a 
> "character" corresponds to a fixed-length field of real estate.  You 
> can only go so far with that notion, even if you accomodate East Asian 
> characters being double-width ("two characters"), and various marks 
> applied to characters being zero-width ("zero characters").
>
> A "glyph" limit might be useful, but even that is tied up with the 
> notion of fixed-width fonts and is invalidated by scripts such as Arabic.
>
> Of course, to do any of this you have to know how text is actually 
> drawn on the screen, which tends not to be something done at the level 
> of protocols.  Octet counts, on the other hand, are.
>
> Consequently, I recommend that the wrap recommendation be retained, 
> and that sample code such as the above be proved for the benefit of 
> those people who have trouble understanding how to determine the octet 
> length of a Unicode character in UTF-8.
>
> -- Mark --
>

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Wed May 24 15:49:30 2006

On Wed, 24 May 2006, George Sexton wrote:
> string can be broken into. If I'm doing character based I/O, then a character 
> is the smallest component I can write to the stream (all hail recursive 
> definitions). I rarely, if ever concern myself with typesetting issues.

Unicode breaks that assumption.

For example, consider the German glyph commonly known as "umlaut-a". 
That can be either the Unicode character U+00E4 LATIN SMALL LETTER A WITH 
DIAERESIS or it can be *two* Unicode characters: U+0061 LATIN SMALL LETTER 
A and U+0308 COMBINING DIAERESIS.

In UTF-8, the former is the octet string
 	0xc3 0xa4
and the latter is;
 	0x61 0xcc 0x88
(assuming my notepad calculations are correct).  Of course, if you were 
using good old ISO 8859, it would just be
 	0xe4

The point being is that you can't make any sort of assumptions about
characters or bytes.

Some glyphs (which are what most people mean when they say "character") 
weigh in as being more than a dozen Unicode characters!

Thus, you have to think of "octets", "Unicode characters", and "glyphs".

Now, it is true that many people (myself included) make simplifying 
assumptions for the purpose of their application.  But they have to keep 
track of these assumptions since when (not if) the assumption is broken 
they have to know where to go back and fix....  :-(

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum

---

From: Arnaud.Quillaud at Sun.COM (Arnaud Quillaud)
Date: Thu May 25 02:46:38 2006

> -----Message d'origine-----
> De : [email-masked] 
> [mailto:[email-masked]] De la part de 
> George Sexton
> Envoy? : mercredi 24 mai 2006 20:58
> ? : Eliot Lear
> Cc : ietf-calsify
> Objet : Re: [Ietf-calsify] Status of simplification work
> 
> 

I don't think you need to worry about that.

Section 4.1 "Content Lines" specifies a folding technique and also states that

<<
When parsing a content line, folded lines MUST **first** be unfolded
   according to the unfolding procedure described above. 

So even if you split a 4 bytes UTF8 character by inserting a space+CRLF in the middle, the parser at the other end should remove this space+CRLF even before interpreting the stream as UTF-8.

Arnaud Q

> > discussions in Dallas.  This having been said I'm curious 
> as to what 
> > you believe needs to be simplified.  We've heard here at CalConnect 
> > some issues relating to floating times and RRULEs.  What 
> are your pet 
> > peeves?
> >
> > Eliot

---

From: reinhold at kainhofer.com (Reinhold Kainhofer)
Date: Thu May 25 06:01:58 2006

Am Donnerstag, 25. Mai 2006 11:46 schrieb Arnaud Quillaud:
> So even if you split a 4 bytes UTF8 character by inserting a space+CRLF in
> the middle, the parser at the other end should remove this space+CRLF even
> before interpreting the stream as UTF-8.

Exactly, however, this makes the files invalid UTF-8 encoded text-files, so 
editing them in a text editor will typically not work and mess up the events. 
In KOrganizer, we try to split the lines at a word boundary (i.e. at a 
whitespace "character") between 70 and 75 bytes and if that's not possible, 
we use the glyph boundary before 75 bytes. This makes the ics files valid 
text files that are easy to read in a text editor.

However, this should not got into a standard definition, but maybe it should 
be included in a "best practices" FAQ?

Cheers,
Reionhold

- -- 
- ------------------------------------------------------------------
Reinhold Kainhofer, Vienna, Austria
email: [email-masked], http://reinhold.kainhofer.com/
 * Financial and Actuarial Mathematics, TU Wien, http://www.fam.tuwien.ac.at
 * K Desktop Environment, http://www.kde.org/, KOrganizer maintainer
 * Chorvereinigung "Jung-Wien", http://www.jung-wien.at/

---

From: cyrus at daboo.name (Cyrus Daboo)
Date: Thu May 25 06:31:09 2006

Hi Arnaud,

--On May 25, 2006 11:46:22 AM +0200 Arnaud Quillaud 
<[email-masked]> wrote:

> So even if you split a 4 bytes UTF8 character by inserting a space+CRLF
> in the middle, the parser at the other end should remove this space+CRLF
> even before interpreting the stream as UTF-8.

But that breaks the utf-8 data stored in a file or sent over the wire. As 
far as I am concerned that is wrong - i.e. you MUST do line folding such 
that the resulting data is valid utf-8 (or valid in whatever charset is 
being used).

When I wrote a line folder I arranged it so that it tried to break at 75 
octets, but if that was in the middle of a utf-8 character sequence it 
would step back until it found the starting octet of that sequence and 
break before that.

The current spec does make some hint to that by referring to 'octets' and 
'characters': i.e. 'no longer than 75 octets' and 'split between two 
characters'.

-- 
Cyrus Daboo

---

From: chuck at evdb.com (Chuck Norris)
Date: Thu May 25 06:44:00 2006

It's a stupid question probably, but is it even possible to consider  
removing the fixed line-length requirement?  I know our parser  
doesn't actually care how many octets are in a line, and works fine  
no matter how long the lines are.

Chuck

On May 25, 2006, at 9:00 AM, Reinhold Kainhofer wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.3 (GNU/Linux)
>
> iD8DBQFEdaqETqjEwhXvPN0RAhn8AJ43goJFF7oFtwV2nilMgKi3/OU4cACgghc9
> +ayCXSOIcQKHsW+U5Chg8Rs=
> =WqyL
> -----END PGP SIGNATURE-----

---

From: reinhold at kainhofer.com (Reinhold Kainhofer)
Date: Thu May 25 06:53:13 2006

Am Donnerstag, 25. Mai 2006 15:43 schrieb Chuck Norris:
> removing the fixed line-length requirement?

There is no fixed line-length. Lines SHOULD be not longer than 75 bytes, but 
you can fold them anywhere (or not at all, if you have important reasons).

> I know our parser 

ics content is also meant to be sent via email (among other means of data 
exchange), and some applications even send the iCalendar data as the body of 
the mail (i.e. not as an attachment). In order to prevent problems with some 
MTAs, the folding of long lines should not be removed.

Cheers,
Reinhold
- --

---

From: gsexton at mhsoftware.com (George Sexton)
Date: Thu May 25 06:58:03 2006

> MTAs, the folding of long lines should not be removed.

Good luck. We have problem in our application because Outlook removes 
what it feels are "extra line-breaks". If you actually viewed an ICS 
file that was copied into outlook, you would find that it would wrap 
some lines onto others.

As much as I laud the intent, when the most common corporate MUA in the 
US is totally broken you might as well throw in the towel on that.

---

From: gsexton at mhsoftware.com (George Sexton)
Date: Thu May 25 06:59:59 2006

Actually, I think its a pretty reasonable question. The wrap requirement 
only serves one useful purpose. It makes the files readable in a text 
editor.

Any suggestion that the 75 octet limit on lines it makes the I/O easier 
is incorrect. First of all, anyone that's doing line I/O using fixed 
width buffers is just waiting for a buffer overflow attack.

In this time of multi-megabyte attachments, flash, JPGs, and other 
binary trash floating across the Internet, suggesting that a 75 
character line limit is going to accommodate some antique piece of 
communications hardware just seems unlikely.

Personally, I'm really back to where I started. Octets should be changed 
to characters. This will simplify writing correct software and result in 
a file that is human readable in a text editor (or Email client...).

The point here is simplification. Proving that the exact language of the 
current specification can be met by adding a function, and some error 
prone character break/line wrapping logic does nothing. It will result 
in a portion of the spec that people either don't comply with, or that 
they do wrong. Neither of these improves interoperability.

Chuck Norris wrote:
> removing the fixed line-length requirement?  I know our parser doesn't 
> actually care how many octets are in a line, and works fine no matter 
> how long the lines are.
>
> Chuck
>

---

From: Arnaud.Quillaud at Sun.COM (Arnaud Quillaud)
Date: Thu May 25 08:41:16 2006

> Envoy? : jeudi 25 mai 2006 15:31
> ? : Arnaud Quillaud; George Sexton; Eliot Lear
> Cc : ietf-calsify
> Objet : Re: RE : [Ietf-calsify] Status of simplification work
> 
> 
> Hi Arnaud,
> 
> > So even if you split a 4 bytes UTF8 character by inserting a 
> > space+CRLF in the middle, the parser at the other end should remove 
> > this space+CRLF even before interpreting the stream as UTF-8.
> 

You are correct. It makes the distinction (although, as explained by Mark Crispin, this won't guarantee that your editor will show the correct information).

Arnaud Q

PS: for those who care, libical does split correctly (at the character level, not at the glyph level).

---

From: Arnaud.Quillaud at Sun.COM (Arnaud Quillaud)
Date: Thu May 25 10:13:20 2006

If we really want to change anything regarding this folding (and hence break backward compatibility with older ical parser/generator), I would vote for simply removing the 75 octets limit, not change it to 75 characters.

The spec itself does not mention why it puts such a limit. It would be interesting to know the historical reason that made it appear.

About the email legacy MTA problem:
The ical spec is supposed to be "formatted as a registration for a MIME media type". MIME is already describing and taking care of the legacy SMTP server problem. If an application is generating emails containing ical attachments (or ical directly in the body), it should use one of the content-transfer-encoding defined by MIME. 

About ical being more readable:
The fact that ical is text based is great for debugging purpose. But do you usually really care about lengthy text fields when debugging ?
What is the real life use case for this readability of long lines ?
Does the XML specification require folding of lines > 75 octets ? If not, did its addoption suffer from it ?

That said, Im not sure it would be wise to break backward compatibility for this.

Arnaud Q

> Envoy? : jeudi 25 mai 2006 16:00
> ? : Chuck Norris
> Cc : [email-masked]

---

From: lists at block-online.eu (Oliver Block)
Date: Thu May 25 12:40:41 2006

Hi, 

I am also new to this list.

Am Donnerstag, 25. Mai 2006 15:59 schrieb George Sexton:
> The point here is simplification.

Simplification for whom?

> It will result 
> they do wrong.

By which reason are they doing it wrong?

Changing a spec seems to me as the most comfortable way of not solving a 
problem one faces.
The question should be: Are there good reasons for that rule? Does a change to 
75 characters lead to anything, or does it just mislead? What would you say 
if anybody suggests to change from 75 octets to 75 words (the linguistic 
unit).

Interesting discussion though.:)

Best Regards,

Oliver

---

From: gsexton at mhsoftware.com (George Sexton)
Date: Thu May 25 14:31:08 2006

>> The point here is simplification.

I thought vendors who were producing or consuming iCal files. That's the 
point of this list isn't it?

The proposed change simplifies things because programmers can simply write;

for (int i=0; i < data.length; ) {
    stream.write(data[i++]);
    if ((i%75==0) && (i<data.length))
       stream.write("\n");
}

Research has found that the number one method of reducing the number of 
defects in a program is to reduce the number of lines in that program. 
Doing octets requires:

int iCurrentLine=0,iCharLength;
for (int i=0; i < data.length; i++) {
   iCharLength=ObscureUnicodeToByteLengthConversion(data[i]);
    if (iCurrentLine+iCharLength)>75 {
       stream.write("\n");
       iCurrentLine=0;
    }
    stream.write(data[i]);
    iCurrentLine+=iCharLength;
}

 The second version is about twice as long as the 1st version, requires 
two additional variables, and requires a function call on every 
character. Given a function length for the character to byte length 
converter of 10 lines, that makes the second version almost 3 times longer.

5 lines= simpler

20 lines=more complicated
> By which reason are they doing it wrong?
A lot of people don't wrap at all. Of the rest I would bet that over 90% 
who are writing multi-byte UTF-8 are wrapping at 75 characters.
> Changing a spec seems to me as the most comfortable way of not solving a 
I'll ignore the implied insult that I'm actually not clever enough to 
code a correct implementation.

> The question should be: Are there good reasons for that rule? 
I too am waiting to hear the good reason. So far, the only winner is it 
makes the file easier to read in a text editor.

> Does a change to 
More complexity with no advantage. An average page of English text is 
500 words. Breaking on 75 words would yield 7 lines. b This would give 
the complexity of the output code having a word counter, without the 
benefit of actually making the text viewable in an editor without 
scrolling. It also assumes that the implementer can write generic code 
for any language that will identify linguistic unit separators. Doubtful.

> Interesting discussion though.:)
My mistake. I thought this list was made up of people who are interested 
in improving interoperability. It appears to be a debating society where 
members compete to prove their cleverness. So far, to sum up the responses:

An obscure function that can be used to implement an algorithm 3 times 
longer than the simpler.
A suggestion that people who want to work with characters are doing so 
in misguided attempts to handle typesetting issues.
An incorrect suggestion that it doesn't matter if you interpose a 
newline character between the bytes of a UTF-8 character.
A suggestion that my desire stems from my inability to correctly 
implement the requirement.
A pointless alternative suggestion offered up in the spirit of debate.

I'm suggesting improving interoperability by increasing the probability 
that an implementer of average skill will get this part right. Do you 
want a standard that only extremely skilled programmers will have any 
chance of delivering a correct implementation, or do you want a standard 
that most people will be able to do correctly?

I'm really, really glad that I'm not putting up any money to participate 
in this comedy.

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Thu May 25 15:22:25 2006

On Thu, 25 May 2006, George Sexton wrote:
> I'm suggesting improving interoperability by increasing the probability that 
> an implementer of average skill will get this part right. Do you want a 
> standard that only extremely skilled programmers will have any chance of 
> delivering a correct implementation, or do you want a standard that most 
> people will be able to do correctly?

This may not be a popular view, but it bears stating:

Given my druthers, I would prefer to have a standard that only an 
extremely skilled programmer is capable of delivering any implementation 
at all.  There is a much greater chance of getting multiple quality, 
interoperable, implementations if the incompetant are completely locked 
out.

I have been asked, on multiple occasions, effectively to write the 
protocol engine for an application that some cheapskate contracted to the 
lowest bidder...and to do this service for free.  On other occasions, I 
have been asked protocol questions that are so elementary that it is 
painfully clear that the individual never read the specification (or even 
one of the excellent pedagogical texts that describe the protocol). 
Worse, he makes it clear that he has no intention of doing so; he expects 
me to answer the questions that come up when empirical testing won't 
deliver the answer.

-- Mark --

---

From: gsexton at mhsoftware.com (George Sexton)
Date: Thu May 25 15:29:26 2006

Mark Crispin wrote:
>> I'm suggesting improving interoperability by increasing the 
>> probability that an implementer of average skill will get this part 
>> right. Do you want a standard that only extremely skilled programmers 
>> will have any chance of delivering a correct implementation, or do 
>> you want a standard that most people will be able to do correctly?
>
> extremely skilled programmer is capable of delivering any 
> implementation at all.  There is a much greater chance of getting 
> multiple quality, interoperable, implementations if the incompetant 
> are completely locked out.

As stated, it's a good idea. Practically speaking though, even skilled 
developers will have a much higher defect rate implementing a complex 
specification.

---

From: TimHare at comcast.net (Tim Hare)
Date: Thu May 25 17:45:13 2006

Wow - I guess I wasn't specific enough in my questions, the "answers" became
a discussion on coding solutions in one particular language.

1. Have the recurrence issues been resolved? If so, what is the simplified
support for recurrence rules or were they dropped entirely?

2. At one point there was a discussion about dropping timezones,
transmitting all time data in UCT and leaving timezone conversion as a UI
issue (but there were some unresolved issues regarding recurring time sets
which crossed daylight-savings-time boundaries, I think)

3. It's been a while since I saw a notice of a draft posted - what is the
latest draft related to this work?

4. IF any of the developers for major implementations still read this list,
or if CalConnect would care to take another survey, can we get a show of
hands whether non-folded lines break an implementation?  And conversely, are
there known implementations that naively assume that the line-ending
sequence always means the end of a component?

And to whoever asked the XML question (I've forgotten): I believe the XML
spec ignores whitespace although that doesn't mean that all parsers do. XML
discussions, however, are explicitly not part of the WG's charter.

Tim Hare
Interested Bystander, Non-Inc.

---

From: sroberts at uniserve.com (Sam Roberts)
Date: Thu May 25 23:48:22 2006

>    if ((i%75==0) && (i<data.length))

Sorry, but that assumes that your data buffer is 16 bit unicode
characters, which might be true for you in java but isn't true in the
languages I work in. My library is written with its buffers being in
utf-8, so when I split at the 75 octet boundary, it really is the 75
octet boundary, and not necessarily at the 75 character boundary.

If it was changed to 75 characters, the effect would be to complicate my
code for much the same reasons the current definition complicates your
code, I'd have to figure out how many "characters" are in the buffer.

Anyhow, I don't have a problem with the SHOULD being removed.

I do have a problem with it being changed from octets to 'characters',
it would make it the only one of the mail/mime/etc. specs that works
that way.

Cheers,
Sam

---

From: sroberts at uniserve.com (Sam Roberts)
Date: Thu May 25 23:57:51 2006

> that the resulting data is valid utf-8 (or valid in whatever charset is 

Has anybody noted a similar issue with wrapping at escape sequence
boundaries in TEXT values?

I split at exactly 75 octets, and I've recently had reports that if
there is a '\' 'n' sequence (an escaped newline) that happens to fall on
the boundary that some tools will not read in the data correctly.

Are these equivalent:

-----
DESCRIPTION:a newline \
 n just passed
-----

and 

-----
DESCRIPTION:a newline \n
  just passed
-----

Seems to me that they are, and are both valid utf-8, and obey the folding rules!

Sam

---

From: gsexton at mhsoftware.com (George Sexton)
Date: Fri May 26 08:14:06 2006

> code, I'd have to figure out how many "characters" are in the buffer.
In my example, data is an array of characters. The Java I/O classes 
handle the conversion to the appropriate file encoding automatically.

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Fri May 26 08:48:22 2006

On Fri, 26 May 2006, George Sexton wrote:
>> languages I work in.
> In my example, data is an array of characters. The Java I/O classes handle 
> the conversion to the appropriate file encoding automatically.

There is no such thing as an "array of characters" when you are talking 
about UTF-8.

For that matter, if Java uses UTF-16, there is no such thing as an "array 
of characters" there either.  Characters outside the BMP in UTF-16 are 
represented by surrogates that occupy two array positions.

The only way in Unicode in which there can be an "array of characters" is 
if you use UCS-2 (which only allows the characters in the BMP) or UCS-4. 
However, as I pointed out earlier, even if you use UCS-4 you still can not 
assume that it is alright to break a Unicode string at a Unicode character 
boundary since there are combining characters which add to the glyph 
formed by the previous character.

Given my earlier example of umlaut-a in German; if you insert a break 
between the "a" and the umlaut in a decomposed character (or worse, 
truncate at that point!) you have changed the actual glyph (which is what 
most people mistakenly call "characters").  In some languages, this is 
more than just a missing diacritical mark -- it is a completely different 
(and unrelated) letter!

Once you start using Unicode, you have to give up such quaint notions as 
"array of characters".  I know that this sounds bad to you; it is indeed 
very bad news to programmers who have relied upon traditional methods of 
character manipulation for decades.  I sympathize, having gone through 
this process myself.

Another quaint notion that has to be discarded is that a character is 
representable in a bit-field; that is, that the maximum codepoint value is 
a power of 2 minus one.  That is no longer true in Unicode.  Unicode has 
17 planes of 16-bit codepoints; thus the maximum Unicode codepoint value 
is 0x10ffff.  Insiders say that Unicode values are "20 bits plus 1".

There is no "simple" way to program Unicode.  The sooner you discard such 
notions from the past, and get accustomed to doing things the Unicode way, 
the happier you will be in the long run.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

---

From: gsexton at mhsoftware.com (George Sexton)
Date: Fri May 26 09:11:27 2006

Mark Crispin wrote:
> There is no such thing as an "array of characters" when you are 
> talking about UTF-8.

This is true. I've never argued that there was. In fact, other people 
have argued that there was and I have disagreed. Specifically, the 
person who said that you could shove a new-line between UTF-8 bytes of a 
character.

>
> For that matter, if Java uses UTF-16, there is no such thing as an 
> "array of characters" there either.  Characters outside the BMP in 
> UTF-16 are represented by surrogates that occupy two array positions.

This is totally false. Consider a typical Java declaration:

char[] buf=new char[1024];

declares an array of Unicode 16 characters. If you would like to argue 
the mechanics of what the virtual machine does under the hood, that's 
fine. But from the programming language standpoint Java has arrays of 
characters. I can directly index and access any character.

To elaborate a little on the I/O end of things, if I declare an Output 
stream, and set the encoding to be UTF-8 then when I shove a character 
down that stream the encoding issues are taken care of automatically. If 
I set the encoding to ISO-8859-1, and send an accented character down 
it, then I will get the correct encoding.

> The only way in Unicode in which there can be an "array of characters" 
> is if you use UCS-2 (which only allows the characters in the BMP) or 
> UCS-4. However, as I pointed out earlier, even if you use UCS-4 you 
> still can not assume that it is alright to break a Unicode string at a 
> Unicode character boundary since there are combining characters which 
> add to the glyph formed by the previous character.
>
> truncate at that point!) you have changed the actual glyph (which is 
> what most people mistakenly call "characters").  In some languages, 
> this is more than just a missing diacritical mark -- it is a 
> completely different (and unrelated) letter!
>
> Once you start using Unicode, you have to give up such quaint notions 
> as "array of characters".  I know that this sounds bad to you; it is 
> indeed very bad news to programmers who have relied upon traditional 
> methods of character manipulation for decades.  I sympathize, having 
> gone through this process myself.

Again, my feeling is so what? You really aren't seeing the whole 
picture. Let me try to step you through this:

The current RFC-2445 says that lines SHOULD be wrapped at 75 octets and 
a new-line inserted.

Explicitly wrapping the lines at 75 octets if the data is actually 
multi-byte UTF-8 characters will have negative results. The output may 
not be valid UTF-8 in its folded form. I personally suspect that 
breaking UTF-8 characters may introduce other artifacts (extraneous new 
lines).

My original complaint was that octets was a problem with UTF-8. You've 
pointed this out yourself. If sub-dividing characters is a problem 
sub-dividing bytes is a bigger one.

Your original response was to supply a function that would allow me to 
calculate the length of each individual character so that I could write 
a stream that would output correctly formatted UTF-8 characters and stay 
within the 75 octet limit.

So, again, given what you're saying is true about various accent characters.

Given what the spec says about wrapping lines on arbitrary byte 
boundaries, what is the solution?

I haven't heard you argue for removing the wrapping requirement.

Is this your stand now?

> There is no "simple" way to program Unicode.  The sooner you discard 
> such notions from the past, and get accustomed to doing things the 
> Unicode way, the happier you will be in the long run.
I think I get the "Unicode Way" just fine thank you. The problem is the 
spec has a brain damaged requirement that is a problem when using 
multi-byte characters.

If you could stop arguing both sides of the problem, perhaps we could 
come to a solution.

---

From: gsexton at mhsoftware.com (George Sexton)
Date: Fri May 26 09:25:49 2006

In case you've forgotten your original position:

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Fri May 26 09:36:18 2006

I decline to waste any more time on this discussion.

I wish you good luck.  You will need it.

-- Mark --

http://panda.com/mrc

---

From: lists at block-online.eu (Oliver Block)
Date: Fri May 26 10:53:00 2006

Hi George,

> code a correct implementation.

:) (not intended)

> I too am waiting to hear the good reason.

You never ask for the reason! My guess is: the formal reason is that it is a 
registered mime media type. There may be technical or others.

> > Does a change to
> > 75 characters lead to anything, or does it just mislead? What would you
> > say if anybody suggests to change from 75 octets to 75 words (the
> > linguistic unit).
>
> More complexity with no advantage.

If a character can consist of 1, 2 or three bytes/octets and you change from 
octet to character it is the same. Even if exaggerated. The result is: it 
will not be 75 octets.

> in improving interoperability.

interoperability between ...?

> that an implementer of average skill will get this part right.

OK for me.

> I'm really, really glad that I'm not putting up any money to participate

And you can be sure that you will not be charged by me.:)

Best Regards,

Oliver

---

From: helge.hess at opengroupware.org (Helge Hess)
Date: Sat May 27 19:51:42 2006

On 26. Mai 2006, at 18:11 Uhr, George Sexton wrote:
> declares an array of Unicode 16 characters.

Are you aware of the difference between UTF-16 and UCS-2? AFAIK Java  
(like Cocoa) uses the former which makes Mark's statement totally  
correct. A Java character (char basetype) is not the same like a  
Unicode character.
But thats Java/Unicode 101 and doesn't really belong on this list?

> I haven't heard you argue for removing the wrapping requirement.

I'm kinda confused. Its a SHOULD, not a requirement?

Anyway, if the item is sent over a transport which doesn't support  
longer-than-75-lines (like email) you can still apply content- 
transfer-encodings?

Summary: I think the 75-char requirement should be dropped, its of no  
use? iCalendar is the content type and transport requirements are  
already covered by the transports (eg MIME).

Helge
-- 
http://docs.opengroupware.org/Members/helge/
OpenGroupware.org

---

From: dave at cridland.net (Dave Cridland)
Date: Sun May 28 13:43:46 2006

> longer-than-75-lines (like email) you can still apply content- 

As an aside, just to clarify, email has no such requirement. It has a 
recommendation for limiting line lengths to 78, which is largely a 
matter of ancient history, although it still applies to a large 
degree for western glyphs, which are more likely to be displayed on 
the non-wrapping 80 column displays that spawned the recommendation, 
and it's been reinforced again by format=flowed, which removes any 
counter-argument.

It does have a requirement for line lengths lower than 1000 octets 
including the CR LF delimiter when sent with DATA, as do the MIME 
encodings 7bit and 8bit. These vanish as well with suitable usage of 
binary MIME (RFC3030, etc). This limit has nothing to do with 
display, this relates to ancient mail gateways, I believe, although 
Mark Crispin can probably shed a lot more light on this than I can, 
having worked with mail at the time this limit was being made.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

---

From: lear at cisco.com (Eliot Lear)
Date: Mon May 29 05:49:57 2006

Tim,

[Again, with my chair hat off]

Sorry for my delayed posting (I am on holiday).  In the future, I
suggest that each such question below or new questions be separated by
subject so that they don't get lost in the shuffle.  Here is my opinion
about the below.

> support for recurrence rules or were they dropped entirely?
They have not been resolved.
> which crossed daylight-savings-time boundaries, I think)

This has not been resolved.
> latest draft related to this work?
A draft will be posted prior to the Montreal meeting.  Bernard has his
hands full, but as we are able to close on the above issues they will
either make that draft or the next.
> sequence always means the end of a component?
I do not know the answer to this question for the calendar clients but
as someone who has written calendars I didn't think that folding the
text isn't all that big a deal, and not that hard to test, and I'm not a
great programmer.  Again, chair hat off. YMMV.

Eliot

---

From: Dave.Thewlis at calconnect.org (Dave Thewlis)
Date: Mon May 29 06:15:10 2006

Tim,

With respect to the Recurrence and Timezone subjects, CalConnect has 
published problem statements and recommendations in both of those areas 
which are available on our web site.  Whether they will be adopted in 
whole or in part by the Calisfy WG or the draft editors is another 
matter, but the issues have at least been studied and possible 
approaches to them proposed.

Dave Thewlis
CalConnect

-- 
*Dave Thewlis, Executive Director
Calconnect - The Calendaring and Scheduling Consortium*
+1 707 840 9391 (voice) ? +1 707 498 2238 (mobile)
http://www.calconnect.org ? [email-masked] 
<mailto:[email-masked]>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osafoundation.org/pipermail/ietf-calsify/attachments/20060529/05a7dff4/attachment.html

---

From: Dave.Thewlis at calconnect.org (Dave Thewlis)
Date: Mon May 29 06:38:35 2006

Apologies for not including pointers to the documents I mentioned in my 
previous e-mail.

There's a section on the CalConnect website called "Work Products" and 
the current documents are all listed; go to http://www.calconnect.org 
and select Work Products from the sidebar.  They are all in PDF format. 

Alternatively you can go directly to

http://www.calconnect.org/publications/icalendarrecurrenceproblemsandrecommendationsv1.0.pdf

http://www.calconnect.org/publications/icalendartimezoneproblemsandrecommendationsv1.0.pdf

http://www.calconnect.org/publications/timezoneregistryandservicerecommendationsv1.0.pdf

Dave Thewlis

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osafoundation.org/pipermail/ietf-calsify/attachments/20060529/b3bfa751/attachment.htm

---

From: cyrus at daboo.name (Cyrus Daboo)
Date: Wed May 31 07:47:46 2006

Hi Sam,

> it would make it the only one of the mail/mime/etc. specs that works

2445 says a maximum of 75 octets per line and split between two characters. 
As I read it that meant splitting at the utf-8 'character sequence' 
boundary at or below an octet count of 75.

As to the mail/mime etc specs, they only deal with ASCII data when doing 
folding/unfolding (which only occurs for header fields) - so there is no 
problem there. However, the current efforts to allow non-ascii in message 
headers etc will have this problem - I've not checked to see whether they 
have already tried to address that. I might ask on their list about this 
and see what others think...

-- 
Cyrus Daboo

---

From: gsexton at mhsoftware.com (George Sexton)
Date: Wed May 31 11:54:36 2006

Cyrus Daboo wrote:
> Hi Sam,
>
> --On May 25, 2006 11:47:47 PM -0700 Sam Roberts 
> <[email-masked]> wrote:
>
How many of the other specs specify UTF-8 as the default encoding?

---

From: lists at block-online.eu (Oliver Block)
Date: Wed May 31 14:24:47 2006

> How many of the other specs specify UTF-8 as the default encoding?

That is not the point.

The point is: either there are reasons to fold at 75 (or less) or there are no 
reasons for that. If there are reasons the rule should be kept IMHO.

It could be extended that UTF-8 characters SHOULD NOT be folded between 
multibyte characters. Another RFC states that lines should not/must not be 
folded after whitespace.

Best Regards,

Oliver

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Wed May 31 14:34:34 2006

On Wed, 31 May 2006, Oliver Block wrote:
> The point is: either there are reasons to fold at 75 (or less) or there 
> are no reasons for that. If there are reasons the rule should be kept 
> IMHO.

It is up to the proponents of removing folding to demonstrate that there 
is no reason to fold.

Put another way, it is necessary to prove that nobody depends upon folding 
now; and furthermore that no individual, who is not privy to this 
discussion, will in good faith create an application which depends upon 
folding based upon current documents.

Without such proof, the question is moot.  It seems to me that the 
concensus of experienced protocol implementors is that it is not 
particularly difficult to fold, even when UTF-8 is involved.

> multibyte characters.

Perhaps you should have meant to say something to the effect against 
folding in the middle of a multi-octet UTF-8 character.  But that isn't 
what you said.

You can always fold between characters; the prohibition is on folding in 
the middle of a character.

Also, character != octet != byte.

-- Mark --