Subject: RE: CALL FOR CONSENSUS (was RE: [lemonade] on MTI conversions)

From: "Burger, Eric" <[email-masked]>
Date: Fri, 14 Apr 2006 14:25:49 -0400

Part of me wants to say, "Live with it, it is HTML to text."

Another part of me wants to say, "Character set conversion is so
trivial, let's go with that."

Yet another part of me is inclined to say, "Do HTML to text *and*
character set conversions."

I am leaning towards option 1 (HTML->text) as so far I hear one voice of
dissent and I am worried about option 2 (character set conversions)
leading to months of debate over which character sets need to be
supported and how to deal with non-standard, but common, character set
encodings.

Thoughts?

-----Original Message-----
From: Dave Cridland [mailto:[email-masked]] 
Sent: Wednesday, April 12, 2006 9:26 AM
To: Burger, Eric
Cc: Enhancements to Internet email to support diverse service
enivronments
Subject: Re: CALL FOR CONSENSUS (was RE: [lemonade] on MTI conversions)

On Wed Apr 12 11:48:07 2006, Burger, Eric wrote:
> Would character set conversions exercise parameters, as well?
> 
> 
It would only exercise parameters, in fact.

Dave.
-- 
           You see things; and you say "Why?"
   But I dream things that never were; and I say "Why not?"
    - George Bernard Shaw

_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade

---

From: Mark Crispin <[email-masked]>
Date: Fri, 14 Apr 2006 11:41:16 -0700 (PDT)

> character set conversions."

What about the fourth choice:

Do HTML to text.  As a server implementation option, also do character set 
conversions.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

---

From: Randall Gellens <[email-masked]>
Date: Fri, 14 Apr 2006 13:50:03 -0700

>  supported and how to deal with non-standard, but common, character set

Sounds reasonable to me.  Text to HTML is fairly simple, while 
charsets tend to be more complicated then they seem.
-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
Whenever people agree with me I always feel I must be wrong.
                                              --Oscar Wilde

---

From: "Robert A. Rosenberg" <[email-masked]>
Date: Sat, 15 Apr 2006 00:38:05 -0400

At 14:25 -0400 on 04/14/2006, Burger, Eric wrote about Re: CALL FOR 
CONSENSUS (was RE: [lemonade] on MTI conversio:

>I am worried about option 2 (character set conversions) leading to 
>months of debate over which character sets need to be supported and 
>how to deal with non-standard, but common, character set encodings.

While I do not feel I am that qualified to offer an opinion on if 
option 2 should be done and if it is what character sets should be 
supported, I will repeat my prior comment that one that would 
probably be included (ISO-8859-1 <-> Unicode) can not be 100% 
successful (ie: Accurate) in all cases due to there being 3 separate 
glyph mappings that are marked in Email as ISO-8859-1.

These are:

  1) True ISO-8859-1 where the x80-x9f code point range is used for 
the <soapbox> useless (so far as Email and other data storage as 
opposed to being sent to display devices) C1 control characters that 
the ISO wants to be there for historical reasons instead of usable 
glyphs </soapbox>.

  2) Windows-1252 (AKA CP-1252) which is the native code used on 
Windows Computers where this 32 codepoint range is used for useful 
glyphs in lieu of the useless ISO mandated C1 Control Codes. Many 
Windows Email and Web Design programs blindly mark files with these 
codepoints as ISO-8859-1 even when this is a lie (ie: The file has 
data with these codepoints). Note: I am speaking generically when I 
say the "Native Code" since based on the language that is selected 
for Windows Support, you can/would get the Windows-125x superset 
versions of the ISO-8859-x codes as well as Unicode having replaced 
the old 256 codepoint codes. OTOH - The use of a ISO-8859-x Charset 
in Email I think outweighs the number of UTF-8 marked messages.

  3) Macintosh Source Email marked as ISO-8859-1 (or other ISO-8859-x 
Charsets). This represents a problem since the xa0-xff mappings 
to/from MacRoman is accurate except for a few glyphs that are not in 
MacRoman and which thus represent glyphs that do not exist in either 
Windows-1252 or ISO-8859-1. The major problem is that the Mac mapped 
glyphs in its version of ISO-8859-1 does not agree with the positions 
of the Windows-1252 x80-x9f glyphs (although I think all 32 glyphs in 
that codepoint range are in the Mac version - just in different 
locations in that range).

Windows-1252 can easily be handled by just assuming an codepoints in 
the x80-x9f  codepoint range that occur in a message marked as 
ISO-8859-1 should be treated as if the charset was designated as 
Windows-1252 when transforming to Unicode (and when going from 
Unicode to ISO-8859-1 just map any occurrences of the 32 "extra" 
glyphs to their Windows-1252 positions).

The case of the Macintosh version of ISO-8859-1 is a separate issue 
due to the missing/extra glyphs in the xa0-xff range as well as the 
non-Windows-1252 arrangement of the 32 glyphs mapped to the x80-9f 
codepoint range.

---

From: Ned Freed <[email-masked]>
Date: Fri, 14 Apr 2006 16:23:24 -0700 (PDT)

> charsets tend to be more complicated then they seem.

OK, I gotta ask. Does the HTML->text thing allow me to change

     text/html; charset=gb18030

to

     text/plain; charset=utf-8

And if not, why not?

				Ned

---

From: Randall Gellens <[email-masked]>
Date: Mon, 17 Apr 2006 14:53:29 -0700

At 12:38 AM -0400 4/15/06, Robert A. Rosenberg wrote:

>   I will repeat my prior comment that one that would probably be 
> included (ISO-8859-1 <-> Unicode) can not be 100% successful (ie: 
> Accurate) in all cases due to there being 3 separate glyph mappings 
> that are marked in Email as ISO-8859-1.

I agree with comments from Ned and others that we don't need (and it 
would be dangerous for us to attempt)  to specify heuristics for 
recognizing and dealing with mislabeled charsets.

However, something very much like Robert's description of the problem 
could be included as an informative aside ("Implementers may wish to 
note that, historically, clients have incorrectly labeled as 
ISO-8859-1 at least two different....")  It seems to be helpful 
information that would be nice to have documented.

-------------- Randomly-selected tag: ---------------
The best way to have a good idea is to have lots of ideas.
                                           --Linus Pauling

---

From: Ned Freed <[email-masked]>
Date: Mon, 17 Apr 2006 17:14:53 -0700 (PDT)

> information that would be nice to have documented.

I have no problem with including an informational note about these issues.

				Ned

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 18 Apr 2006 12:44:08 +0100

> charsets tend to be more complicated then they seem.

Randy, I think you underestimate the complexity of this.
For example, how does one convert <table> element to text? How to deal 
with column widths in percents and pixels, etc.?

---

From: Randall Gellens <[email-masked]>
Date: Tue, 18 Apr 2006 16:14:50 -0700

At 12:44 PM +0100 4/18/06, Alexey Melnikov wrote:

>  I think you underestimate the complexity of this.
>  For example, how does one convert <table> element to text? How to 
> deal with column widths in percents and pixels, etc.?

You're right, Alexey, I was overlooking aspects of this that add 
significant difficulty, such as tables.  I was thinking of the 
Qpopper code that converts HTML to plain text.  It does a decent job 
on the basic stuff (paragraphs, attributes, entities, etc.)  I don't 
think it handles tables, though.
-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
Nothing in the world is more dangerous than sincere ignorance and
conscientious stupidity.
          --Martin Luther King, Jr.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 24 Apr 2006 15:20:12 +0200

> with column widths in percents and pixels, etc.?

And how <iframe src="data:...">? But I've been using something like this 
myself for years, and doing without all the odd html stuff is perfectly 
reasonable for mail. The occasional spam is badly formatted.

Not a problem in practice, and IMO also not a problem for an m-t-i 
conversion feature. The chap who wanted to sell me screws and whatever 
in bulk this morning would have to send his spam without tables, or 
else his spam doesn't fit the lemonade profile.

Look at the code in gnumail (an objective-c application for the mac, but 
the code is perfectly readable for the rest of you, too).

Arnt

---

From: "Robert A. Rosenberg" <[email-masked]>
Date: Sat, 29 Apr 2006 01:16:42 -0400

At 14:53 -0700 on 04/17/2006, Randall Gellens wrote about Re: CALL 

>information that would be nice to have documented.

Although I do not consider my expertise level to be anywhere near 
those of others on this list, I have no objection to my comments 
appearing as an informational note (it can serve as my contribution 
to this issue and the resultant RFC <g>).

I would like to slightly amend my comments on translation of 
something MARKED AS ISO-8859-1 into Unicode. Since this ONLY has to 
do with Text and HTML email, any codepoint (or HTML numeric entry) in 
the x80-x9f range has a 0% probability to be a C1 Control Character 
and a 100% probability to be intended to reference a displayable 
glyph (especially in the HTML case). Thus ASSUMING that the 
appearance of such a codepoint or numeric entry should be mapped to 
the correct Unicode glyph codepoint that matches the Windows-1252 
mapping value in the x80-9f range will produce the CORRECT code 
except in the Macintosh MacRoman<->ISO-8859-1 case (along with the 
few MacRoman characters that get assigned codepoints in the xA0-FF 
range due to MacRoman not having the correct ISO-8859-1 glyph and 
replacing them with glyphs that do not exist in ISO-8859-1 [or the 
Windows-1252 extra glyphs that replace the C1 Control Characters]).

If needed, I can supply a list of these non-standard mappings in the 
xA0-FF range or you can look at the table at the top of the article 
at <http://ppewww.ph.gla.ac.uk/~flavell/iso8859/iso8859-mac.html>. 
Unfortunately I can not put my hands on an equivalent article listing 
the x80-x9f mappings right now (although Randall Gellens probably can 
since I seem to remember that it was in some Mac Eudora 
documentation).

The Windows-1252 mappings for the x80-x9f range can be found at 
<http://czyborra.com/charsets/iso8859.html> as a sub-table under 
ISO-8859-1.