Subject: [Ietf-calsify] draft-ietf-calsify-rfc2445bis-01.txt / UTF-8

From: lists at block-online.eu (Oliver Block)
Date: Tue Aug 15 05:55:31 2006

Hello List,

maybe some of you do remember a "discussion" we had about UTF-8 and folding.

<quote>
3.1.4.  Character Set

   There is not a property parameter to declare the character set used
   in a property value.  The default character set for an iCalendar
   object is UTF-8 as defined in [RFC2279].
</quote src="draft-ietf-calsify-rfc2445bis-01.txt">

I'd like to call your attention to the following paragraph I recently read, 
which is taken from "The Unicode Standard 4.0" and available online.

<quote>
Utility programs are not prevented from operating on "mangled" text. For 
example, a UTF-8 file could have had a CRLF sequences introduced at every 80 
bytes by a bad mailer program. This could result in some UTF-8 byte sequence 
being interrupted by CRLFs, producing illegal byte sequences. This mangled 
text is no longer UTF-8. It is permissible for a conformant program to repair 
such text, recognizing that the mangled text was originally well-formed UTF-8 
byte sequences. However, such repair of mangled data is a special case, and 
it must be not be used in circumstances where it would cause security 
problems.
</quote src="The Unicode Standard 4.0, p. 61">

draft-ietf-calsify-rfc2445bis-01.txt specifies UTF-8 as default character set, 
and implies _valid_ UTF-8, of course. But for the mentioned problems on 
folding in that discussion it might be helpful to hint that folding within  
UTF-8 byte sequences produces _illegal_ byte sequences which are no longer 
UTF-8.

Best Regards,

Oliver Block

-- 
Leben ist mehr als schneller - weiter - h?her
http://www.nak-nrw.de/p_6_4.html

---

From: kainhofer at kainhofer.com (Reinhold Kainhofer)
Date: Wed Aug 16 00:29:53 2006

> UTF-8 byte sequences produces _illegal_ byte sequences which are no longer

Sure, but rfc 2445 says that lines first have to be unfolded and then 
interpreted as UTF-8.

Cheers,
Reinhold

---

From: gsexton at mhsoftware.com (George Sexton)
Date: Wed Aug 16 07:16:08 2006

I think the point is that until it's unfolded, it's not a valid UTF-8 
document which is also required by the spec.

Reinhold Kainhofer wrote:
> _______________________________________________
> Ietf-calsify mailing list
> [email-masked]
> http://lists.osafoundation.org/mailman/listinfo/ietf-calsify
>

-- 
George Sexton
MH Software, Inc.
Voice: +1 303 438 9585
URL:   http://www.mhsoftware.com/

---

From: lists at block-online.eu (Oliver Block)
Date: Wed Aug 16 08:13:30 2006

Am Mittwoch, 16. August 2006 09:29 schrieben Sie:
> > draft-ietf-calsify-rfc2445bis-01.txt specifies UTF-8 as default character
> > set, and implies _valid_ UTF-8, of course. But for the mentioned problems
> > on folding in that discussion it might be helpful to hint that folding
> > within UTF-8 byte sequences produces _illegal_ byte sequences which are
> > no longer UTF-8.
>

Yes. My thought was if this point deserves to be emphasized a bit more. I 
would agree that RFC2445 already requires valid UTF-8.

Best Regards,

Oliver

---

From: bernard.desruisseaux at oracle.com (Bernard Desruisseaux)
Date: Mon Aug 21 11:29:31 2006

I propose to add the following statement in section 4.1 Content Lines:

  A multi-octet character MUST NOT be split across lines.

Old text:

 > Lines of text SHOULD NOT be longer than 75 octets, excluding the line
 > break. Long content lines SHOULD be split into a multiple line
 > representations using a line "folding" technique. That is, a long
 > line can be split between any two characters by inserting a CRLF
 > immediately followed by a single linear white space character (i.e.,
 > SPACE, US-ASCII decimal 32 or HTAB, US-ASCII decimal 9). Any sequence
 > of CRLF followed immediately by a single linear white space character
 > is ignored (i.e., removed) when processing the content type.

New text:

 > Lines of text SHOULD NOT be longer than 75 octets, excluding the line
 > break. Long content lines SHOULD be split into a multiple line
 > representations using a line "folding" technique. That is, a long
 > line can be split between any two characters by inserting a CRLF
 > immediately followed by a single linear white space character (i.e.,
 > SPACE, US-ASCII decimal 32 or HTAB, US-ASCII decimal 9).  A multi-
 > octet character MUST NOT be split across lines.  Any sequence of
 > CRLF followed immediately by a single linear white space character
 > is ignored (i.e., removed) when processing the content type.

Cheers,
Bernard

---

From: cyrus at daboo.name (Cyrus Daboo)
Date: Mon Aug 21 13:45:26 2006

Hi Bernard,

>   A multi-octet character MUST NOT be split across lines.

+1

-- 
Cyrus Daboo

---

From: alexey.melnikov at isode.com (Alexey Melnikov)
Date: Mon Aug 21 13:47:28 2006

+1

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Mon Aug 21 14:18:43 2006

Another +1.

However, before we do this, I want to make sure that we are referring to 
"characters" in the context of "Unicode codepoints"; and thus a 
"multi-octet character" refers to a maximal-length UTF-8 representation of 
a Unicode character (4 octets).

If, on the other hand, we are referring to the UTF-8 representation of a 
composite character (that is, one composed of a base character and one or 
more combining characters), we need to back off or at least require 
precomposed forms.  I've seen some decomposed forms that are quite 
lengthy.

>>   A multi-octet character MUST NOT be split across lines.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: lear at cisco.com (Eliot Lear)
Date: Tue Aug 22 02:03:54 2006

Mark,

> Another +1.
>
> However, before we do this, I want to make sure that we are referring
> to "characters" in the context of "Unicode codepoints"; and thus a
> "multi-octet character" refers to a maximal-length UTF-8
> representation of a Unicode character (4 octets).
>
> If, on the other hand, we are referring to the UTF-8 representation of
> a composite character (that is, one composed of a base character and
> one or more combining characters), we need to back off or at least
> require precomposed forms.  I've seen some decomposed forms that are
> quite lengthy.

Can you please propose the exact alternate text you would like?  This
will help move things along.

Thanks,

Eliot

---

From: alexey.melnikov at isode.com (Alexey Melnikov)
Date: Tue Aug 22 03:39:40 2006

>  A multi-octet character MUST NOT be split across lines.

+1.

---

From: lists at block-online.eu (Oliver Block)
Date: Tue Aug 22 05:08:26 2006

Am Montag, 21. August 2006 23:18 schrieb Mark Crispin:
> precomposed forms.  

But those are IMHO _character_ sequences, aren't they?

Best Regards,

Oliver

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Tue Aug 22 08:09:09 2006

> But those are IMHO _character_ sequences, aren't they?

I hope that those are "character sequences" and not "characters".  That's 
what they are in Unicode.  But we have a long history in the IETF of using 
words informally without agreement as to the precise meaning.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Tue Aug 22 08:52:10 2006

> will help move things along.

It's difficult to propose alternative text when I don't know, precisely, 
what the intent is.

Am I correct that the intent is to prohibit breaking up a UTF-8 sequence 
for a single Unicode codepoint, and that there is no intent to prohibit 
breaking up a sequence of character + combining characters?

What is the purpose for the prohibition?

Clearly, we have to fit within a buffer, such as the 75 octet limit for a 
MIME encoded-word.  But MIME encoded-word has rules for concatenation when 
the desired text is longer than can fit in a single encoded-word.

If the purpose is to assist dumb software that blats to a terminal 
emulator, I see little/no hope unless the buffer is huge.  IIRC, some 
UTF-8 sequences for character + combining character (take a look at some 
of the Arabic decompositions) are larger than what would fit in a MIME 
encoded-word.

If the software handles concatenation properly, then I don't see what the 
problem is even if the break is in the middle of a single UTF-8 sequence. 
If it doesn't, I see no hope of the software working right due to 
combining characters.

I won't object if the idea is to "cover the 98% case, and accept that the 
other 2% is broken."  But I haven't heard that stated as such yet.

-- Mark --

http://panda.com/mrc

---

From: cyrus at daboo.name (Cyrus Daboo)
Date: Tue Aug 22 09:08:22 2006

Hi Mark,

> breaking up a sequence of character + combining characters?

Yes.

> What is the purpose for the prohibition?

To ensure that the text data generated after folding remains valid UTF-8. 
Obviously inserting CR-LF-SPACE into the middle of a multi-octet UTF-8 
sequence invalidates that sequence.

> MIME encoded-word.  But MIME encoded-word has rules for concatenation
> when the desired text is longer than can fit in a single encoded-word.

PS I believe the terminology Bernard originally proposed was copied from 
one of the MIME specs - of course that does not mean further clarification 
should not be done...

-- 
Cyrus Daboo

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Tue Aug 22 11:00:47 2006

>> breaking up a sequence of character + combining characters?

OK.

> sequence invalidates that sequence.

But this only if the sequence is interpreted to treat the CR-LF-SPACE as 
significant text, as opposed to merely a protocol artifact to do 
continuation, correct?

Isn't it supposed to unfold first, before interpreting the contents?

Once again, what problem are you trying to sold?

I think that you are trying to solve the problem that a dumb application, 
which does not properly unfold first, would see a broken UTF-8 sequence. 
An example would be something that outputs this data as raw text.

If that's the problem, it should be stated as such.  But if that is the 
problem, then the problem is still not solved because of the combining 
character problem.  Unless you just seek a 98% solution.  In which case, 
that too should be stated.

-- Mark --

---

From: lists at block-online.eu (Oliver Block)
Date: Tue Aug 22 11:51:25 2006

> An example would be something that outputs this data as raw text.

On one hand you are complaining about not folding properly, but on the other 
hand you do not complain about delivering invalid UTF-8.

What I'd like to know is: are your concerns about decompositions (or composite 
characters) legitimate. I did not find anything in the unicode specs, yet. 
But maybe you did!?

Best Regards,

Oliver

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Tue Aug 22 11:56:26 2006

> hand you do not complain about delivering invalid UTF-8.

I don't understand this comment.

Folding is a defined protocol concept.  The question is about what, if 
any, circumstances folding is prohibited.

The proposal is to prohibit folding within a UTF-8 sequence for a Unicode 
character.  The question is if folding should also be prohibited prior to 
Unicode composing characters.

> characters) legitimate. I did not find anything in the unicode specs, yet.

I don't understand this comment either.  Do you claim that nobody uses any 
of the Unicode composing characters?

-- Mark --

---

From: lists at block-online.eu (Oliver Block)
Date: Tue Aug 22 12:05:04 2006

Am Dienstag, 22. August 2006 20:56 schrieben Sie:
> > What I'd like to know is: are your concerns about decompositions (or
> > composite characters) legitimate. I did not find anything in the unicode
> > specs, yet. But maybe you did!?
>
> I don't understand this comment either. ?Do you claim that nobody uses any

No, you've misunderstood me. I mean, if a composite character is a multi-octet 
sequence.

Regards,

Oliver

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Tue Aug 22 12:33:07 2006

> No, you've misunderstood me. I mean, if a composite character is a multi-octet

German lower-case umlaut-a can be transmitted as U+00e4 or U+0061 U+0308.

In UTF-8, this is either C3 A4 and 61 CC 88.  Both of these are multi 
octet sequences.

Questions:
  (a) is it alright to fold between C3 and A4?
  (b) is it alright to fold between 61 and CC?
  (c) is it alright to fold between CC and 88?

If the answer to (a) is "yes", presumably it is also "yes" for (b) and 
(c).  If not, why not?

If the answer to (a) is "no", presumably it is also "no" for (c).  If not, 
why not?

If the answer is (a) is "no", is it also "no" for (b)?  Why?  Or, why not?

The statement "nobody ever uses U+0061 U+0308 to send German umlaut-a" is 
not an acceptable answer.  The fact that there are some precomposed 
characters in Unicode does not mean that we can presume that all 
characters that will be used in email are precomposed.

So, to understand the problem properly, we should assume text with the 
decomposed form.

So, why is it any more wrong to fold in the middle of a UTF-8 sequence, as 
in (a) and (c) above, than it is to fold in the middle of a composed 
character sequence?

Doesn't an application need first to remove folding from a text prior to 
any other processing (including UTF-8 processing)?

Given that the combining characters modify the previous character, isn't 
is wrong to separate them into two UTF-8 strings by folding, even if the 
strings themselves appear to be valid UTF-8?

What does it mean when a string begins with a combining character? 
Does it modify the space of the line of the fold instead of the previous 
string?  If the two strings must be catenated prior to any interpretation, 
what is the harm of folding within a UTF-8 sequence?

If it is "alright" to allow a composed character sequence to be truncated, 
does it remain "alright" if the truncation results in some obscenity or 
blasphemy instead of the original piety?

These questions need to be considered, and answered, carefully.  The more 
that I think about these questions, the less confident that I am that I 
know the answers.

-- Mark --

---

From: lists at block-online.eu (Oliver Block)
Date: Tue Aug 22 15:49:54 2006

Am Dienstag, 22. August 2006 21:32 schrieben Sie:
> > No, you've misunderstood me. I mean, if a composite character is a
> > multi-octet sequence.
>

That's right. C3 A4 and CC 88 are multi-octet sequences. 61 is 01100001 
(binary) and IMHO multi-octet bytes begin (1st byte) with 110..., 1110..., 
11110..., as you know. 

A well-formed multi-byte sequence (as called by the unicode standard), 
consisting of 3 bytes start with E0, E1..EC, ED, EE..EF. 
(Table 3-6, p. 78)

>   (a) is it alright to fold between C3 and A4?

No. (multi-octet sequence.)

>   (b) is it alright to fold between 61 and CC?

Yes. 0x61 is not part of a multi-octet sequence.

>   (c) is it alright to fold between CC and 88?

No. CC is part of the multi-octet sequence CC 88

> If the answer to (a) is "no", presumably it is also "no" for (c).  If not,

I agree on that.

> If the answer is (a) is "no", is it also "no" for (b)?  Why?  Or, why not?

0x61 is IMHO not part of a multi-octet sequence.

[,,,]

> that I think about these questions, the less confident that I am that I

That's the same with me. But having no answer does _sometimes_ indicate that 
the questions need to be thought over.

This answer took me two hours. I hope I passed the exam. :)

Sincerely,

Oliver

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Tue Aug 22 16:28:46 2006

Unfortunately, your answers are circular.

I understand that you assert
  (a) it is not alright to fold in the middle of a UTF-8 sequence
      ("multi-octet sequence" is ambiguous and imprecise)
but that
  (b) it is alright to fold between a character and a combining character.

However, you also give assertion (a) as the answer the "why" question for 
(a) and (b).

Why is it not alright to fold in the middle of a UTF-8 sequence?

Why is it alright to fold between a character and a combining character?

What is wrong with the assertation:
 	A proper interpretation of the text is impossible until
 	all folding is removed and the strings are catenated.
 	Therefore, folding may appear anywhere, even in the
 	middle of a UTF-8 sequence.
or, alternatively:
 	A proper interpretation of a subtext is impossible unless
 	all UTF-8 sequences and combining characters appear in
 	that subtext.  Therefore, folding may not in the middle
 	of a UTF-8 sequence or separating the UTF-8 sequences of
 	any (and all) combining characters from the character
 	being combined.

Why is one, or the other, of the above two assertations inferior to your 
pair of assertations?

I'm sorry for being such a troublemaker, and to be honest I really don't 
know which of these is best.  But someone's got to do it.  Whatever 
decision is made, we need to justify why that decision and not the 
alternatives.

-- Mark --

---

From: bernard.desruisseaux at oracle.com (Bernard Desruisseaux)
Date: Tue Aug 22 19:13:57 2006

[
   For those not familiar with "combining character sequence" here's
   how it is defined by Unicode: A character sequence consisting of
   either a base character followed by a sequence of one or more
   combining characters, or a sequence of one or more combining
   characters.
]

Let me try to put this another way. We need to decide and justify:

1- Whether we want to allow "multi-octet characters" to be split
    across lines.

Answer: No.
Why   : Otherwise the resulting text would end up being invalid
         in the specified encoding.

2- Whether we want to allow "combining character sequences" to be
    split across lines.

Answer: Yes.
Why   : (1) I'm assuming that it is valid for a "combining
         character" to be preceded by the LF character (but I
         don't know this for a fact...), and thus the resulting
         text would still be valid in the specified encoding
         (but would sure "look" different).

         (2) A "combining character sequence" could probably be
         longer than 75 octets in *theory*! But I'm sure we would
         never see this in practice though...

With this approach:

- You could open an iCalendar object specified in the charset 'X'
   in any application that support 'X' without errors.

- You would still need to unfold the iCalendar object to be able
   to "interpret" all the characters properly.

What do you think?

Cheers,
Bernard

Mark Crispin wrote:
> However, you also give assertion (a) as the answer the "why" question 
> for (a) and (b).
>

---

From: McQuilWP at pobox.com (Bill McQuillan)
Date: Wed Aug 23 01:59:09 2006

I think Bernard has pointed out one issue that occurred to me also--since
the information is UTF8 text, an application that does not understand an
iCalendar object could still read it, for instance a text editor. The
question becomes how would it react to a line break in the middle of a
composed character.

After browsing in the Unicode standard I found this sentence in Annex # 14
- Line Breaking Properties:

   Combining character sequences are treated as units for the purpose of
   line breaking.

If the text editor assumes this property, there will likely be some loss of
information.

-- 
Bill McQuillan <[email-masked]>

---

From: lists at block-online.eu (Oliver Block)
Date: Wed Aug 23 04:47:03 2006

>    to "interpret" all the characters properly.

That's exactly how I understand your preceding proposal. Even if it did not 
mention combining character sequences. And that's how I understood Mark in 
his agreement on that issue.

Best Regards,

Oliver

---

From: bernard.desruisseaux at oracle.com (Bernard Desruisseaux)
Date: Wed Aug 23 08:08:33 2006

Hi Bill,

The question is "Do we actually care?".

I would say that as long as the iCalendar stream is a valid
UTF-8 document we are ok.

Cheers,
Bernard

---

From: cyrus at daboo.name (Cyrus Daboo)
Date: Wed Aug 23 08:14:52 2006

Hi Bernard,

> UTF-8 document we are ok.

Agreed - my main concern is ensuring that the stream conforms to UTF-8 abnf 
- i.e. the characters are all valid. I don't really care whether the 'raw' 
unfolded text displays well or not in a text editor. After all, even in the 
absence of combining characters, the folding can occur in the middle of a 
word and the result may be nonsensical or even worse appear to be two valid 
words giving a different meaning to what was meant.

-- 
Cyrus Daboo