Subject: [openpgp] Character encodings

From: Wyllys Ingersoll <[email-masked]>
Date: Tue, 17 Mar 2015 15:38:31 +0000

One area that I think needs some attention is the character encoding and
charsets for encrypted text messages.

4880 says that everything should be UTF-8.  However, the reality is that
UTF8 is not used everywhere and there are lots of clients that compose
messages in their native preferred character set (Latin5, Greek, Kanji,
etc) and its very difficult as an implementor to figure it out after the
fact without some indication from the sender.

The literal packet format only specifies 3 possible values - binary, UTF8,
or plain.  The ASCII Armor header may specify a different charset (though
unfortunately very few agents add the "Charset" PGP header).
Additionally, if the message had MIME headers, there may be yet another
charset indicated in MIME that differs from the ASCII Armor charset and the
literal packet data format byte.

If the encrypting PGP software knows what character encoding was used to
compose the original message, there should be some way to communicate this
in the message that would be definitive so that the decrypting software can
present it the way it was originally intended.  As an implementor, this is
one of the trickiest areas to get right so that the end user sees the
messages as it was originally intended.

---

From: Stephen Paul Weber <[email-masked]>
Date: Tue, 17 Mar 2015 10:50:23 -0500

>etc)

But a conforming OpenPGP implementation should convert this to UTF-8 before 
encrypt/sign if using text packets.

-- 
Stephen Paul Weber, @singpolyma
See <http://singpolyma.net> for how I prefer to be contacted
edition right joseph

---

From: Wyllys Ingersoll <[email-masked]>
Date: Tue, 17 Mar 2015 15:55:33 +0000

Right, but they don't.  The application doing the encryption really doesn't
know what the original character encoding was, so doing the conversion can
result in lost or corrupted messages.

I think the only way to get it right is if the message composer (i.e. mail
client usually) is tightly coupled with the PGP engine so that it can pass
along the character encoding when the encryption request is made.

---

From: Tim Bray <[email-masked]>
Date: Wed, 18 Mar 2015 08:00:56 +1300

This would be a huge step backward. The proportion of text on the internet
that is UTF-8 is monotonically increasing toward 100%. Thank goodness.
On Mar 18, 2015 4:38 AM, "Wyllys Ingersoll" <[email-masked]> wrote:

> _______________________________________________
> openpgp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/openpgp
>
>

---

From: Wyllys Ingersoll <[email-masked]>
Date: Tue, 17 Mar 2015 19:26:55 +0000

In my experience it's not getting any better for PGP messages that are not
composed in a basic text editor.  Users composing messages on a mobile
devices, for example, do not always default to UTF8, they use the
system-wide character encoding setting (or the charset encoding specified
by the composing app itself).

For example, iOS Apple basically says if you don't know the original
encoding, you have to basically "guess" by trying various encodings until
you find one that works.
https://developer.apple.com/library/ios/documentation/Cocoa/Conceptual/Strings/Articles/readingFiles.html#//apple_ref/doc/uid/TP40003459-SW4
Fortunately, it usually only takes a few tries to get it right if its not
UTF8.

I agree that UTF-8 should be preferred and enforced wherever possible. But
in cases where it is not, it would help if the sender was able to provide a
hint as to what the encoding actually is, and do so in a standardized
manner that can be easily implemented.

On Tue, Mar 17, 2015 at 3:00 PM Tim Bray <[email-masked]> wrote:

>> The literal packet format only specifies 3 possible values - binary,
>> UTF8, or plain.  The ASCII Armor header may specify a different charset
>> (though unfortunately very few agents add the "Charset" PGP header).

---

From: Jon Callas <[email-masked]>
Date: Tue, 17 Mar 2015 12:44:31 -0700

One of the things that OpenPGP doesn't do very well that needs to be fixed is layering.

We have the notion of text versus binary because at one time that kinda made sense. Back when FTP was high-tech, you could get better usability by knowing that something was text so that you could translate between SIXBIT, EBCDIC, RAD50, ASCII, and other codings that only used upper case in their names (because lower-case was also high-tech in those days).

We don't have those problems any more. We have slightly different problems, but we also have solutions to those. If you want to send a text message that has a strange encoding, there are ways to do that. Wyllys Ingersoll and others have noted this.

Just get rid of the notion of text. Make it be all binary. Push the problem up a layer in the software stack -- they have to deal with it anyway, and all OpenPGP can do is make it worse.

	Jon

---

From: Daniel Kahn Gillmor <[email-masked]>
Date: Tue, 17 Mar 2015 19:17:02 -0400

> result in lost or corrupted messages.

Character encodings are a problem for message signatures too.  The fact
that the messages don't embed any knowledge of this means that it's
possible to subtly alter some textual content.

See the "message tampering through header substitution" section here for
a quickly-contrived example (someone with better knowledge of existing
character sets can probably craft a more clever substitution):

  https://dkg.fifthhorseman.net/notes/inline-pgp-harmful/

> along the character encoding when the encryption request is made.

Or have the composer only send well-structured data that includes
framing information (and have the reciever know how to parse the framing
information).

This has to happen *inside* the signature (and inside the encryption,
for that matter).  For PGP/MIME, that work has already been done: the
MIME structure is the framing information.

If you pick some other sort of framing, you're kind of on your own
afaict -- i don't know how many other standards there are out there.

      --dkg

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 17 Mar 2015 19:33:09 -0400

On Tue, Mar 17, 2015 at 3:44 PM, Jon Callas <[email-masked]> wrote:

> We have the notion of text versus binary because at one time that kinda
> made sense. Back when FTP was high-tech, you could get better usability by
> knowing that something was text so that you could translate between SIXBIT,
> EBCDIC, RAD50, ASCII, and other codings that only used upper case in their
> names (because lower-case was also high-tech in those days).
>
> We don't have those problems any more. We have slightly different
> problems, but we also have solutions to those. If you want to send a text
> message that has a strange encoding, there are ways to do that. Wyllys
> Ingersoll and others have noted this.
>
> Just get rid of the notion of text. Make it be all binary. Push the
> problem up a layer in the software stack -- they have to deal with it
> anyway, and all OpenPGP can do is make it worse.
>

+1

It is all just binary blobs for the end-to-end crypto layer.

The biggest mistakes in the Internet are all due to naive attempts to solve
problems for the end user by converting their bits from one format to
another.

Remember when every time you used FTP you had to do every file transfer
twice because the first time you forgot to set the flag to Binary
transport? The main reason email is a problem is that mail gateways do
idiot character transformations like line wrapping and get things wrong.

The layering issue is key. SMTP and HTTP both conflate the application
layer headers and content metadata. These should be separated out. Now it
is too late to do that for regular mail but we can start fixing it for
encrypted.

The content-type, character set, subject line etc. should all be considered
content metadata. In fact From, To, CC should as well because those are not
what SMTP uses to route on, the SMTP values are used.

This also solves the problem of not leaking unnecessary information.

Yes, I understand that using SMTP in its current form will leak information
as well. But there are ways to start fixing that.

---

From: Christoph Anton Mitterer <[email-masked]>
Date: Wed, 18 Mar 2015 23:32:50 +0100

On Tue, 2015-03-17 at 12:44 -0700, Jon Callas wrote: 
> Just get rid of the notion of text. Make it be all binary.
Agreed 100%,.. OpenPGP should never to any conversions (e.g. for
signature verifications), hinting or anything else with respect to "text
encoding".

The best thing that can happen is that nothing gets worse (cause even if
the OpenPGP implementation would do everything right, the MUA or any
other application on top/below may still mangle up data).

The worst thing that can happen, is that one could trick
users/implementations into taking things as signed in a form which they
were not intended to be signed, e.g. I deliberately only wanted to have
the file with \n EOLs to be signed, but not any \n\r. In such case
however, if a "text mode" is identified, the peer's application would
also trust that.
With character encodings things are probably even worse.

All should be binary.

Cheers,
Chris.