Subject: RE: CALL FOR CONSENSUS (was RE: [lemonade] on MTI conversions)

From: "Burger, Eric" <[email-masked]>
Date: Sat, 15 Apr 2006 21:07:10 -0400

This is why I am not a fan of mandating the character set conversion to
be our conversion.  It would require us to document which mapping works,
and that is WAY outside of our charter.

-----Original Message-----
From: Robert A. Rosenberg [mailto:[email-masked]] 
Sent: Saturday, April 15, 2006 12:38 AM
To: Enhancements to Internet email to support diverse
serviceenivronments
Subject: RE: CALL FOR CONSENSUS (was RE: [lemonade] on MTI conversions)

At 14:25 -0400 on 04/14/2006, Burger, Eric wrote about Re: CALL FOR 
CONSENSUS (was RE: [lemonade] on MTI conversio:

>I am worried about option 2 (character set conversions) leading to 
>months of debate over which character sets need to be supported and 
>how to deal with non-standard, but common, character set encodings.

While I do not feel I am that qualified to offer an opinion on if 
option 2 should be done and if it is what character sets should be 
supported, I will repeat my prior comment that one that would 
probably be included (ISO-8859-1 <-> Unicode) can not be 100% 
successful (ie: Accurate) in all cases due to there being 3 separate 
glyph mappings that are marked in Email as ISO-8859-1.

These are:

  1) True ISO-8859-1 where the x80-x9f code point range is used for 
the <soapbox> useless (so far as Email and other data storage as 
opposed to being sent to display devices) C1 control characters that 
the ISO wants to be there for historical reasons instead of usable 
glyphs </soapbox>.

  2) Windows-1252 (AKA CP-1252) which is the native code used on 
Windows Computers where this 32 codepoint range is used for useful 
glyphs in lieu of the useless ISO mandated C1 Control Codes. Many 
Windows Email and Web Design programs blindly mark files with these 
codepoints as ISO-8859-1 even when this is a lie (ie: The file has 
data with these codepoints). Note: I am speaking generically when I 
say the "Native Code" since based on the language that is selected 
for Windows Support, you can/would get the Windows-125x superset 
versions of the ISO-8859-x codes as well as Unicode having replaced 
the old 256 codepoint codes. OTOH - The use of a ISO-8859-x Charset 
in Email I think outweighs the number of UTF-8 marked messages.

  3) Macintosh Source Email marked as ISO-8859-1 (or other ISO-8859-x 
Charsets). This represents a problem since the xa0-xff mappings 
to/from MacRoman is accurate except for a few glyphs that are not in 
MacRoman and which thus represent glyphs that do not exist in either 
Windows-1252 or ISO-8859-1. The major problem is that the Mac mapped 
glyphs in its version of ISO-8859-1 does not agree with the positions 
of the Windows-1252 x80-x9f glyphs (although I think all 32 glyphs in 
that codepoint range are in the Mac version - just in different 
locations in that range).

Windows-1252 can easily be handled by just assuming an codepoints in 
the x80-x9f  codepoint range that occur in a message marked as 
ISO-8859-1 should be treated as if the charset was designated as 
Windows-1252 when transforming to Unicode (and when going from 
Unicode to ISO-8859-1 just map any occurrences of the 32 "extra" 
glyphs to their Windows-1252 positions).

The case of the Macintosh version of ISO-8859-1 is a separate issue 
due to the missing/extra glyphs in the xa0-xff range as well as the 
non-Windows-1252 arrangement of the 32 glyphs mapped to the x80-9f 
codepoint range.

_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade

_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade

---

From: Dave Cridland <[email-masked]>
Date: Sun, 16 Apr 2006 19:29:45 +0100

> and that is WAY outside of our charter.
I'm not convinced that character set conversion requires documenting 
clients incorrectly marking the character set. I appreciate that this 
is, to a degree, ignoring the real world, however I don't think we 
ought to be mandating heuristics to handle software errors of this 
nature in a standards track proposal.

Saying that servers MUST handle iso-8859-1 -> utf-8 is surely within 
our charter, and as far as I'm aware that mapping works if the 
character set used really is iso-8859-1 to begin with. If it isn't, 
then the client is buggy, and how servers deal with buggy content is 
entirely up to them.

As for which character sets, I'd be perfectly content with utf-8 as 
sole (required) output, and an almost random selection of character 
sets as input. (Probably the top 5 or so used in real email. ASCII 
ought to be fairly trivial).

I'm also still pondering Ned Freed's point that character set 
conversion is a reasonable (and possibly essential) thing to include 
within HTML conversion - if this is the case, we need to document 
character set conversions anyway, and my suggestion is essentially a 
text/plain=>text/plain conversion. (In other words, the parameters 
form part of the type).

On the other hand, what constitutes a text/html to text/plain 
conversion? Are we mandating, in effect, that all mailservers wishing 
to do content-conversion include a full W3C TR conformant text-based 
HTML renderer? I think we probably are, and that strikes me as 
considerable effort, not "fairly simple".

Dave.
-- 
           You see things; and you say "Why?"
   But I dream things that never were; and I say "Why not?"
    - George Bernard Shaw

---

From: Ned Freed <[email-masked]>
Date: Sun, 16 Apr 2006 16:58:11 -0700 (PDT)

> nature in a standards track proposal.

I agree 100%. The problem of mislabelled material is insoluable in general - if
it wasn't why would we need the labels to begin with?

If we're to head down this slippery slope, we might as well deal with
inappropriate file names on parts, and for that matter incorrect media types
are not exactly unheard of.

> then the client is buggy, and how servers deal with buggy content is

Exactly.

> text/plain=>text/plain conversion. (In other words, the parameters

I'm by no means sure they are esential, but I'm by no means sure they aren't.

> considerable effort, not "fairly simple".

I don't we really need to go that far. But I do think that finding the right
language to describe what's required may be tricky.

				Ned

---

From: Dave Cridland <[email-masked]>
Date: Mon, 17 Apr 2006 09:50:38 +0100

> I'm by no means sure they are esential, but I'm by no means sure 
I'm not sure either way, but the more I think about it, the more I 
suspect that, at the very least, a "pure" text/html => text/plain 
translation would have to mandate that the text/plain type included a 
charset parameter identical to that in the text/html type. Leaving it 
out implies the client is requesting a character set transform to 
ASCII.

> language to describe what's required may be tricky.
It's not what I feel is required to satisfy the actual use-case - 
dragging the essential information from HTML into plain text - but I 
suspect that the only way of specifying what's requiredis to 
reference the W3C's documents. That in itself is fine, but pretty 
much rules out any possibility of advancing to Draft, because there's 
no realistic hope that email servers might provide a full text/plain 
renderer for HTML, when as far as I'm aware, nobody else ever has. I 
might be pessimistic, or Lynx or Links might actually be strictly 
conforming to the W3C's relevant TRs, however.

On the other hand, if we try to specify what is, and is not, 
important information in an HTML document, I fear we'll be not only 
*way* outside charter - indeed, outside remit for the IETF - but it's 
really not somewhere I think we want to go anyway.

Dave.

---

From: Cyrus Daboo <[email-masked]>
Date: Mon, 17 Apr 2006 09:51:06 -0400

Hi Eric,

> and that is WAY outside of our charter.

Based on my experience the problem with mislabeled character sets is likely 
to be much less of an issue to deal with than the problem of malformed HTML 
in text/html bodies.

In either case I think both character set and html->plain should be 
mandated as they are very common operations that most client authors have 
dealt with one way or another. Really you can't do html->plain without 
addressing character set conversion one way or another - and it is worth 
noting that the character set conversion in the html case may not be 
triggered solely by the MIME content header parameter - there can also be a 
character set in the HTML and that and the MIME one may not be the same!

-- 
Cyrus Daboo

---

From: Ned Freed <[email-masked]>
Date: Mon, 17 Apr 2006 08:03:52 -0700 (PDT)

> out implies the client is requesting a character set transform to

Well, sure, since the default charset for text/plain is US-ASCII.

> > I don't we really need to go that far. But I do think that finding
> > the right language to describe what's required may be tricky.

> conforming to the W3C's relevant TRs, however.

I guess I still don't see the problem. Lossy and imperfect conversions of
various sorts have been specified in an assortment of standards going back to
1984 (e.g., X.400-1984) if not earlier. Perfection isn't required; the only
tricky thing is deciding how much imperfection is acceptable and how much
specificity is needed.

I'm far from convinced that language simply specifying a "best effort"
conversion be done is unacceptable here.

> really not somewhere I think we want to go anyway.

Agreed that that is a path to madness. But this is only an issue if we're
required to open this particular cans of worms to proceed, and I'm still
not hearing a justification for why we have to open it.

				Ned