Subject: UTF-8 and literal packets

From: Jon Callas <[email-masked]>
Date: Tue, 23 Mar 2004 07:09:22 -0800

[Editor hat off, representative of implementer hat on]

We at PGP have been talking with David Shaw about an issue we're having 
with UTF-8. The problem is that there are a number of times where 
someone takes text that is not UTF-8, but something like 8859-n, passes 
it into either GnuPG or PGP, sends it to the other, and then we end up 
displaying it wrong. Abstractly, this is not a problem that can be 
solved entirely. Heck, both my mailer and my web browser have menus 
where I can select what character set/encoding to assume something is 
in.

We would like to have a modification to the literal packet, where there 
is a type 'u' packet which is identical to the type 't' packet except 
that in this packet the implementation is saying, "By gum, I *know* 
this really, really contains UTF-8 in it. Trust me. Really."

We've already tested this and both GnuPG and PGP handle a literal 'u' 
packet like 'b', which only has the potential drawback of artifically 
CRLF endings. This gives us a way, however, to get proper layering in 
the sort of systems that we interact with.

Any objections?

	Jon

---

From: "Hal Finney" <[email-masked]>
Date: Wed, 24 Mar 2004 11:43:38 -0800

Werner writes:
> I'd say don't deprecate anything, the 'u' is merely a flag that the
> message is for sure utf-8 encoded.  It's up to the MUA[1] to create
> and display a message correctly, that flag might be helpful.  Maybe a
> "If it is definitely known that the message is utf-8 encoded, the 'u'
> flag SHOULD be used."

I'm concerned that it's going to be hard to clearly specify when 't'
should be used and when 'u' should be, and likewise how to distinguish
the handling of 't' from 'u'.  Should 't' prompt the recipient to try
to guess what character set it is, i.e. prompt the user to choose
a charset?  That's going to make things inconvenient for all the old
't' encrypted files around.

And what about the problem with 'u' causing CRLF's to be left in files
when decrypted by legacy versions?  That would suggest that in at
least some cases (like plain ASCII text) it would be better to use 't'.
This is true even though plain ASCII is legal UTF-8 and so the rule you
suggest would suggest to use 'u'.

It's not that this is a bad idea, it's just that it's hard to codify
the unfortunately messy situation that we are in now.

Hal