Subject: [ISSUE] UTF8 literal packets proposal

From: David Shaw <[email-masked]>
Date: Thu, 27 May 2004 17:11:26 -0400

Things have been quiet here recently.  The UTF8 text literal packet
idea (using a 'u' instead of 't') had a bit of discussion.  Here's a
formal proposal, with text:

In section 5.9 change this:

    If it is a 't' (0x74), then it contains text data, and thus may
    need line ends converted to local form, or other text-mode
    changes.

To this:

    If it is a 't' (0x74), then it contains text data, and thus may
    need line ends converted to local form, or other text-mode
    changes.  The character set used is not specified.

And add:

   If it is a 'u' (0x75), then it contains text data known to be UTF-8
   encoded, and thus may need line ends converted to local form,
   character set conversion, or other text-mode changes.

I think this best describes what was discussed here.  Notably, it does
not say that 't' should not be UTF-8.  't' can be whatever text people
want to put in it.  All 'u' means is that this text is *known* to be
UTF-8.  Implementations can handle this however they like (convert
text in and out, always use 't', etc.)

David

---

From: Jon Callas <[email-masked]>
Date: Thu, 27 May 2004 15:20:44 -0700

I already have it in the next release -- here's what I have:

---
If it is a 'b' (0x62), then the literal packet contains binary data. If 
it is a 't' (0x74), then it contains text data, and thus may need line 
ends converted to local form, or other text-mode changes. The tag 'u' 
(0x75) means the same as 't', but also indicates that implementation 
believes that the literal data contains UTF-8 text.
---

David said off-list that it was perfect (I intended to reply here, 
too), and did only to him. This is edited from that reply.

	Jon

From: Jon Callas <[email-masked]>
Subject: Re: [ISSUE] UTF8 literal packets proposal
Date: Thu, 27 May 2004 15:20:44 -0700
To: OpenPGP <[email-masked]>
Sender: [email-masked]

I already have it in the next release -- here's what I have:

---
If it is a 'b' (0x62), then the literal packet contains binary data. If 
it is a 't' (0x74), then it contains text data, and thus may need line 
ends converted to local form, or other text-mode changes. The tag 'u' 
(0x75) means the same as 't', but also indicates that implementation 
believes that the literal data contains UTF-8 text.
---

David said off-list that it was perfect (I intended to reply here, 
too), and did only to him. This is edited from that reply.

	Jon

To: OpenPGP <[email-masked]>
From: Jon Callas <[email-masked]>
Subject: Bis11 sent out
Date: Thu, 27 May 2004 14:59:18 -0700
Sender: [email-masked]

It includes removing Elgamal signatures, the 'u' literal flag, and lots 
of text regularization.

	Jon

Date: Thu, 27 May 2004 17:11:26 -0400
From: David Shaw <[email-masked]>
To: [email-masked]
Subject: [ISSUE] UTF8 literal packets proposal
Mail-Followup-To: [email-masked]
Sender: [email-masked]

To this:

And add:

David

From: "Adrian 'Dagurashibanipal' von Bidder" <[email-masked]>
To: [email-masked]
Subject: Re: Converting mailbody to UTF-8 before encryption?
Date: Thu, 27 May 2004 08:31:16 +0200
Sender: [email-masked]

--Boundary-02=_0sYtAXeTXEYaYLr
Content-Type: text/plain;
  charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

On Wednesday 26 May 2004 18.30, Holger Sesterhenn wrote:
> Hi,
>
> regarding to chapter 3.4 of the rfc2440bis-10 "the character set for
> text is the UTF-8 encoding of Unicode".
>
> Does this mean I have to convert e.g. the text body (text/plain) of a
> mail to UTF-8 before encryption?

Hmmm. I was going to say that PGP/MIME avoids this problem entirely by=20
requiring all data to be ascii-only (using base64 or qp to achieve=20
this). But looking through 3156, I see that
 - 8bit data for encryption-only messages is allowed and
 - it seems not to specify if text or binary mode should be used in=20
encryption. (I did only quickly scroll through, so maybe I just missed=20
that bit)

The kmail OpenPGP plugin seems to use binary mode (and even so, uses a=20
7bit-clean encoding inside the encryption), so this would indeed not be=20
an issue. Dunno about other implementations.

greetings
=2D- vbi

=2D-=20
COFFEE.EXE Missing---Insert Cup and Press Any Key.

--Boundary-02=_0sYtAXeTXEYaYLr
Content-Type: application/pgp-signature
Content-Description: signature

--Boundary-02=_0sYtAXeTXEYaYLr--

Date: Wed, 26 May 2004 18:30:43 +0200
From: Holger Sesterhenn <[email-masked]>
Organization: Utimaco Safeware AG, NL Aachen
To: [email-masked]
Subject: Converting mailbody to UTF-8 before encryption?
Sender: [email-masked]

Hi,

regarding to chapter 3.4 of the rfc2440bis-10 "the character set for
text is the UTF-8 encoding of Unicode".

Does this mean I have to convert e.g. the text body (text/plain) of a
mail to UTF-8 before encryption? Until today I thought this concerns
only user ids, URIs, notations etc and data to be encrypted is handled
as a binary stream.

This question sounds silly but I have some problems to handle the
decrypted data of "OpenPGP compliant" programs sent by famous mail
clients...

One mail client plugin does convert every mail to UTF-8 (e.g. german
"umlauts" or the euro currency symbol) before encryption. There is no
special "charset" line in the ASCII armored body because UTF-8 is the
default. You can configure the mail client to use "iso-8859-1",
"iso-8859-15", "utf-8" or whatever. Nothing changes. The encrypted data
is still UTF-8.

What should another client do on decryption? Convert back from UTF-8?
Handle the byte stream as is?

Here some examples:

CryptoEx does convert on encryption not on clear signing
PGP does not convert at all
GnuPG does not convert at all

I am a little bit confused.

-- 
Best Regards,

Holger Sesterhenn

content-class: urn:content-classes:message
Subject: RE: Literal packets and canonicalization
Date: Mon, 24 May 2004 17:41:11 -0400
From: "Hasnain Mujtaba" <[email-masked]>
To: "Jon Callas" <[email-masked]>
Cc: <[email-masked]>
Sender: [email-masked]

Jon,

I'm writing an OpenPGP toolkit and my main concern is interoperability
with GPG, PGP, and NAI Ebusiness Server on Windows and UNIX. 

By default, I am not canonicalizing text prior to encryption. In his
response to my question, David clarified the down side of this approach.

I will give my users the option to treat text as input, as do GPG and
PGP, in which case I will canonicalize the line endings and will leave
the trailing white spaces in there. 

Thank you for the detailed response.

-----Original Message-----
From: Jon Callas [mailto:[email-masked]] 
Sent: Monday, May 24, 2004 5:20 PM
To: Hasnain Mujtaba
Cc: [email-masked]; David Shaw
Subject: Re: Literal packets and canonicalization

On 21 May, 2004, at 8:22 AM, Hasnain Mujtaba wrote:

>
> I understand that line endings need to be converted to CRLF. But what 
> about trailing whitespace? When canonicalizing the literal data prior 
> to encryption, do we have to strip off trailing whitespaces? From the 
> RFC, it seems that for signatures the canonicalization must do both, 
> i.e convert line endings and strip off trailing whitespaces. But I am 
> not sure if the same needs to be done for encrypting text data.
>

What problem are you trying to solve? Are you asking in the abstract, 
or are you writing some program that's going to code up OpenPGP data 
blobs?

In the abstract, it's hard to give an answer. My personal opinion is 
that stripping trailing whitespace in something that you know is text 
is rarely a bad idea. But you know, there are lots of things that I 
think are a good idea that I can still come up with a dozen specific 
examples in which you'd break that rule of thumb.

Are you writing a text editor that reads and writes OpenPGP-coded 
files? In that case, if I were you, I'd always strip trailing 
whitespace. But I'd probably also stick a preference somewhere that 
allows the user to choose -- with the default on.

Are you writing an OpenPGP toolkit? Then no, you shouldn't strip white 
space preemptively. But if it were me, I'd make it an option in the 
toolkit, but this time with the default to be not strip.

(For signing, I'd probably do the toolkit with default stripping -- 
there's nothing that says you *can't* do it in OpenPGP -- but give an 
option to not do it, no matter what the standard ends up saying. I 
would do it because stripping will make more signatures verify 
correctly, and if trailing spaces are important, there's always binary 
mode. But I digress.)

Are you writing some file copy plus crypto thing? If so, then no only 
should you not strip white space, but it probably shouldn't be an 
option, except in so much as how it is part of a format translation 
subsystem. People don't like file copies to modify data much at all. 
They're kinda used to FTP text mode (which is where OpenPGP text mode 
comes from, historically), but that's about it. There's no reason why 
you can't take a text file, and render it as a PDF in ten-point Courier 
and then sign that, putting both pieces into an XML wrapper. I'm sure 
there's a couple of people who would think it's kinda cool, even. Other 
people will complain, but there's no reason you can't canonicalize 
through PDF.

As an abstract question, we can debate it forever. As a specific 
question, I've given two scenarios where I'd do it differently.

As a developer, be bold and have some courage of your convictions. 
What's your opinion and why? The reason we have have MAY and SHOULD not 
just MUST is to allow for taste, judgement, and opinion. No matter what 
you want to do, there's at least one person on this list who could 
argue that it's right, and at least one who'd argue it's wrong. And 
then there's people like me, who can do both in the same message with 
hypotheticals.

What are you really considering?

	Jon

Cc: <[email-masked]>, "David Shaw" <[email-masked]>
From: Jon Callas <[email-masked]>
Subject: Re: Literal packets and canonicalization
Date: Mon, 24 May 2004 14:20:17 -0700
To: "Hasnain Mujtaba" <[email-masked]>
Sender: [email-masked]

On 21 May, 2004, at 8:22 AM, Hasnain Mujtaba wrote:

>
> I understand that line endings need to be converted to CRLF. But what 
> about trailing whitespace? When canonicalizing the literal data prior 
> to encryption, do we have to strip off trailing whitespaces? From the 
> RFC, it seems that for signatures the canonicalization must do both, 
> i.e convert line endings and strip off trailing whitespaces. But I am 
> not sure if the same needs to be done for encrypting text data.
>

What problem are you trying to solve? Are you asking in the abstract, 
or are you writing some program that's going to code up OpenPGP data 
blobs?

In the abstract, it's hard to give an answer. My personal opinion is 
that stripping trailing whitespace in something that you know is text 
is rarely a bad idea. But you know, there are lots of things that I 
think are a good idea that I can still come up with a dozen specific 
examples in which you'd break that rule of thumb.

Are you writing a text editor that reads and writes OpenPGP-coded 
files? In that case, if I were you, I'd always strip trailing 
whitespace. But I'd probably also stick a preference somewhere that 
allows the user to choose -- with the default on.

Are you writing an OpenPGP toolkit? Then no, you shouldn't strip white 
space preemptively. But if it were me, I'd make it an option in the 
toolkit, but this time with the default to be not strip.

(For signing, I'd probably do the toolkit with default stripping -- 
there's nothing that says you *can't* do it in OpenPGP -- but give an 
option to not do it, no matter what the standard ends up saying. I 
would do it because stripping will make more signatures verify 
correctly, and if trailing spaces are important, there's always binary 
mode. But I digress.)

Are you writing some file copy plus crypto thing? If so, then no only 
should you not strip white space, but it probably shouldn't be an 
option, except in so much as how it is part of a format translation 
subsystem. People don't like file copies to modify data much at all. 
They're kinda used to FTP text mode (which is where OpenPGP text mode 
comes from, historically), but that's about it. There's no reason why 
you can't take a text file, and render it as a PDF in ten-point Courier 
and then sign that, putting both pieces into an XML wrapper. I'm sure 
there's a couple of people who would think it's kinda cool, even. Other 
people will complain, but there's no reason you can't canonicalize 
through PDF.

As an abstract question, we can debate it forever. As a specific 
question, I've given two scenarios where I'd do it differently.

As a developer, be bold and have some courage of your convictions. 
What's your opinion and why? The reason we have have MAY and SHOULD not 
just MUST is to allow for taste, judgement, and opinion. No matter what 
you want to do, there's at least one person on this list who could 
argue that it's right, and at least one who'd argue it's wrong. And 
then there's people like me, who can do both in the same message with 
hypotheticals.

What are you really considering?

	Jon

Date: Fri, 21 May 2004 13:31:24 -0400
From: David Shaw <[email-masked]>
To: Hasnain Mujtaba <[email-masked]>
Cc: [email-masked]
Subject: Re: Literal packets and canonicalization
Mail-Followup-To: Hasnain Mujtaba <[email-masked]>, [email-masked]
Sender: [email-masked]

On Fri, May 21, 2004 at 11:22:17AM -0400, Hasnain Mujtaba wrote:
> 
> I understand that line endings need to be converted to CRLF. But
> what about trailing whitespace? When canonicalizing the literal data
> prior to encryption, do we have to strip off trailing whitespaces?
> From the RFC, it seems that for signatures the canonicalization must
> do both, i.e convert line endings and strip off trailing
> whitespaces. But I am not sure if the same needs to be done for
> encrypting text data.

It's a bit of a question.  RFC-2440 says you must do both line ending
conversion and trailing whitespace removal.  In practice, GnuPG does
this, and PGP does not (it only does line ending conversion).

For the upcoming 2440bis, there has been some discussion here what the
best thing to do is, but I don't think it is settled yet.

My opinion is that we should behave as PGP currently behaves, and only
do line ending conversion.  Just so I'm clear, this is what I
advocate:

* For clearsigned text, do line ending conversion and remove trailing
  whitespace.

* For 't' literal packets, do line ending conversion only.

David

content-class: urn:content-classes:message
Subject: RE: Literal packets and canonicalization
Date: Fri, 21 May 2004 11:22:17 -0400
From: "Hasnain Mujtaba" <[email-masked]>
To: "David Shaw" <[email-masked]>, <[email-masked]>
Sender: [email-masked]

I understand that line endings need to be converted to CRLF. But what about trailing whitespace? When canonicalizing the literal data prior to encryption, do we have to strip off trailing whitespaces? From the RFC, it seems that for signatures the canonicalization must do both, i.e convert line endings and strip off trailing whitespaces. But I am not sure if the same needs to be done for encrypting text data.
 
Thanks

	-----Original Message----- 
	From: [email-masked] on behalf of David Shaw 
	Sent: Thu 5/6/2004 4:49 PM 
	To: [email-masked] 
	Cc: 
	Subject: Re: Literal packets and canonicalization

	On Thu, May 06, 2004 at 04:27:49PM -0400, Hasnain Mujtaba wrote:
	>
	> Thank you for the answer David. If, as the RFC states, we canoncialize
	> the data before storing it in the literal packet, then the
	> implementation is tampering with the file before performing the
	> operation, say encryption.
	
	"Tampering" is perhaps a little strong.  The canonicalization is part
	of the standard, so while it may be tampering, it's legal
	tampering. ;)
	
	> When I use GPG to encrypt and decrypt a text file, the checksums of
	> the source text file and the decrypted file are the same. So, the
	> file in not being canonicalized prior to encryption?
	
	Maybe, maybe not.  If you are encrypting and decrypting that text file
	on the same platform, then you would expect to end up with the same
	file since the data is canonicalized on the way in, and
	decanonicalized on the way out.
	
	Alternately, if you specified the text file as binary, then no
	canonicalization is done.
	
	GnuPG uses the "--textmode" switch to turn on canonicalization.  PGP
	has a checkbox for it named something like "Input is text".
	
	David

Date: Thu, 6 May 2004 16:49:02 -0400
From: David Shaw <[email-masked]>
To: [email-masked]
Subject: Re: Literal packets and canonicalization
Mail-Followup-To: [email-masked]
Sender: [email-masked]

On Thu, May 06, 2004 at 04:27:49PM -0400, Hasnain Mujtaba wrote:
> 
> Thank you for the answer David. If, as the RFC states, we canoncialize
> the data before storing it in the literal packet, then the
> implementation is tampering with the file before performing the
> operation, say encryption.

"Tampering" is perhaps a little strong.  The canonicalization is part
of the standard, so while it may be tampering, it's legal
tampering. ;)

> When I use GPG to encrypt and decrypt a text file, the checksums of
> the source text file and the decrypted file are the same. So, the
> file in not being canonicalized prior to encryption?

Maybe, maybe not.  If you are encrypting and decrypting that text file
on the same platform, then you would expect to end up with the same
file since the data is canonicalized on the way in, and
decanonicalized on the way out.

Alternately, if you specified the text file as binary, then no
canonicalization is done.

GnuPG uses the "--textmode" switch to turn on canonicalization.  PGP
has a checkbox for it named something like "Input is text".

David

content-class: urn:content-classes:message
Subject: RE: Literal packets and canonicalization
Date: Thu, 6 May 2004 16:27:49 -0400
From: "Hasnain Mujtaba" <[email-masked]>
To: "David Shaw" <[email-masked]>, <[email-masked]>
Sender: [email-masked]

Thank you for the answer David. If, as the RFC states, we canoncialize
the data before storing it in the literal packet, then the
implementation is tampering with the file before performing the
operation, say encryption. When I use GPG to encrypt and decrypt a text
file, the checksums of the source text file and the decrypted file are
the same. So, the file in not being canonicalized prior to encryption?

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of David Shaw
Sent: Thursday, May 06, 2004 3:56 PM
To: [email-masked]
Subject: Re: Literal packets and canonicalization

On Thu, May 06, 2004 at 03:10:17PM -0400, Hasnain Mujtaba wrote:
> 
> Hello,
> 
> I was reading section "5.9. Literal Data Packet" of RFC2440 and I had
a
> question: What are the concequences of not canonicalizing text data
> before storing it in a literal packet and using the literal packet to
> form either an encrypted packet or signature packet? 

The file should decrypt properly, and (at least in PGP and GnuPG)
signatures should verify properly regardless of the canonicalization.

> What if the sender marks all literal data as binary 'b', even if the
> literal data is text?   

The bad thing that will happen is that recipients on platforms that
have a different text line ending convention than the sender will see
somewhat mangled text in the output.

For example: Macs generally end lines with CR.  Unix machines
generally end lines with LF.  Sending data from one to the other
without the benefit of canonicalization results in one very long
"line" with occasional CRs or LFs in there.  Some text editor/viewer
programs do heuristics to detect and fix this problem, but it's
generally better to canonicalize which lets the OpenPGP program handle
it automatically.

David

Date: Thu, 6 May 2004 15:56:18 -0400
From: David Shaw <[email-masked]>
To: [email-masked]
Subject: Re: Literal packets and canonicalization
Mail-Followup-To: [email-masked]
Sender: [email-masked]

On Thu, May 06, 2004 at 03:10:17PM -0400, Hasnain Mujtaba wrote:
> 
> Hello,
> 
> I was reading section "5.9. Literal Data Packet" of RFC2440 and I had a
> question: What are the concequences of not canonicalizing text data
> before storing it in a literal packet and using the literal packet to
> form either an encrypted packet or signature packet? 

The file should decrypt properly, and (at least in PGP and GnuPG)
signatures should verify properly regardless of the canonicalization.

> What if the sender marks all literal data as binary 'b', even if the
> literal data is text?   

The bad thing that will happen is that recipients on platforms that
have a different text line ending convention than the sender will see
somewhat mangled text in the output.

For example: Macs generally end lines with CR.  Unix machines
generally end lines with LF.  Sending data from one to the other
without the benefit of canonicalization results in one very long
"line" with occasional CRs or LFs in there.  Some text editor/viewer
programs do heuristics to detect and fix this problem, but it's
generally better to canonicalize which lets the OpenPGP program handle
it automatically.

David

content-class: urn:content-classes:message
Subject: Literal packets and canonicalization
Date: Thu, 6 May 2004 15:10:17 -0400
From: "Hasnain Mujtaba" <[email-masked]>
To: <[email-masked]>
Sender: [email-masked]

Hello,

I was reading section "5.9. Literal Data Packet" of RFC2440 and I had a
question: What are the concequences of not canonicalizing text data
before storing it in a literal packet and using the literal packet to
form either an encrypted packet or signature packet? 

What if the sender marks all literal data as binary 'b', even if the
literal data is text?   

Thanks
Hasnain. 

----
The information contained in this electronic mail and any attached
document is the confidential and proprietary business information of
Forum Systems, Inc. It is intended solely for the addressed recipient
listed above. It may not be distributed in any manner without the
express written consent of Forum Systems, Inc.