Subject: RE: Literal packets and canonicalization

From: "Hasnain Mujtaba" <[email-masked]>
Date: Fri, 21 May 2004 11:22:17 -0400

I understand that line endings need to be converted to CRLF. But what about trailing whitespace? When canonicalizing the literal data prior to encryption, do we have to strip off trailing whitespaces? From the RFC, it seems that for signatures the canonicalization must do both, i.e convert line endings and strip off trailing whitespaces. But I am not sure if the same needs to be done for encrypting text data.
 
Thanks

	-----Original Message----- 
	From: [email-masked] on behalf of David Shaw 
	Sent: Thu 5/6/2004 4:49 PM 
	To: [email-masked] 
	Cc: 
	Subject: Re: Literal packets and canonicalization

	On Thu, May 06, 2004 at 04:27:49PM -0400, Hasnain Mujtaba wrote:
	>
	> Thank you for the answer David. If, as the RFC states, we canoncialize
	> the data before storing it in the literal packet, then the
	> implementation is tampering with the file before performing the
	> operation, say encryption.
	
	"Tampering" is perhaps a little strong.  The canonicalization is part
	of the standard, so while it may be tampering, it's legal
	tampering. ;)
	
	> When I use GPG to encrypt and decrypt a text file, the checksums of
	> the source text file and the decrypted file are the same. So, the
	> file in not being canonicalized prior to encryption?
	
	Maybe, maybe not.  If you are encrypting and decrypting that text file
	on the same platform, then you would expect to end up with the same
	file since the data is canonicalized on the way in, and
	decanonicalized on the way out.
	
	Alternately, if you specified the text file as binary, then no
	canonicalization is done.
	
	GnuPG uses the "--textmode" switch to turn on canonicalization.  PGP
	has a checkbox for it named something like "Input is text".
	
	David

---

From: David Shaw <[email-masked]>
Date: Fri, 21 May 2004 13:31:24 -0400

On Fri, May 21, 2004 at 11:22:17AM -0400, Hasnain Mujtaba wrote:
> 
> I understand that line endings need to be converted to CRLF. But
> what about trailing whitespace? When canonicalizing the literal data
> prior to encryption, do we have to strip off trailing whitespaces?
> From the RFC, it seems that for signatures the canonicalization must
> do both, i.e convert line endings and strip off trailing
> whitespaces. But I am not sure if the same needs to be done for
> encrypting text data.

It's a bit of a question.  RFC-2440 says you must do both line ending
conversion and trailing whitespace removal.  In practice, GnuPG does
this, and PGP does not (it only does line ending conversion).

For the upcoming 2440bis, there has been some discussion here what the
best thing to do is, but I don't think it is settled yet.

My opinion is that we should behave as PGP currently behaves, and only
do line ending conversion.  Just so I'm clear, this is what I
advocate:

* For clearsigned text, do line ending conversion and remove trailing
  whitespace.

* For 't' literal packets, do line ending conversion only.

David

---

From: Jon Callas <[email-masked]>
Date: Mon, 24 May 2004 14:20:17 -0700

On 21 May, 2004, at 8:22 AM, Hasnain Mujtaba wrote:

>
> I understand that line endings need to be converted to CRLF. But what 
> about trailing whitespace? When canonicalizing the literal data prior 
> to encryption, do we have to strip off trailing whitespaces? From the 
> RFC, it seems that for signatures the canonicalization must do both, 
> i.e convert line endings and strip off trailing whitespaces. But I am 
> not sure if the same needs to be done for encrypting text data.
>

What problem are you trying to solve? Are you asking in the abstract, 
or are you writing some program that's going to code up OpenPGP data 
blobs?

In the abstract, it's hard to give an answer. My personal opinion is 
that stripping trailing whitespace in something that you know is text 
is rarely a bad idea. But you know, there are lots of things that I 
think are a good idea that I can still come up with a dozen specific 
examples in which you'd break that rule of thumb.

Are you writing a text editor that reads and writes OpenPGP-coded 
files? In that case, if I were you, I'd always strip trailing 
whitespace. But I'd probably also stick a preference somewhere that 
allows the user to choose -- with the default on.

Are you writing an OpenPGP toolkit? Then no, you shouldn't strip white 
space preemptively. But if it were me, I'd make it an option in the 
toolkit, but this time with the default to be not strip.

(For signing, I'd probably do the toolkit with default stripping -- 
there's nothing that says you *can't* do it in OpenPGP -- but give an 
option to not do it, no matter what the standard ends up saying. I 
would do it because stripping will make more signatures verify 
correctly, and if trailing spaces are important, there's always binary 
mode. But I digress.)

Are you writing some file copy plus crypto thing? If so, then no only 
should you not strip white space, but it probably shouldn't be an 
option, except in so much as how it is part of a format translation 
subsystem. People don't like file copies to modify data much at all. 
They're kinda used to FTP text mode (which is where OpenPGP text mode 
comes from, historically), but that's about it. There's no reason why 
you can't take a text file, and render it as a PDF in ten-point Courier 
and then sign that, putting both pieces into an XML wrapper. I'm sure 
there's a couple of people who would think it's kinda cool, even. Other 
people will complain, but there's no reason you can't canonicalize 
through PDF.

As an abstract question, we can debate it forever. As a specific 
question, I've given two scenarios where I'd do it differently.

As a developer, be bold and have some courage of your convictions. 
What's your opinion and why? The reason we have have MAY and SHOULD not 
just MUST is to allow for taste, judgement, and opinion. No matter what 
you want to do, there's at least one person on this list who could 
argue that it's right, and at least one who'd argue it's wrong. And 
then there's people like me, who can do both in the same message with 
hypotheticals.

What are you really considering?

	Jon