Subject: Lack of clarity in dash-escaped?

From: Ben Laurie <[email-masked]>
Date: Wed, 30 Nov 2005 10:40:40 +0000

"   Also, any trailing whitespace -- spaces (0x20) and tabs (0x09) -- at
    the end of any line is removed when the cleartext signature is
    generated."

Does this mean they should not be included in the signature, or also
that they should be stripped from the dash-escaped text?

"   The line ending (i.e. the <CR><LF>) before the '-----BEGIN PGP
    SIGNATURE-----' line that terminates the signed text is not
    considered part of the signed text."

Does this mean that one should insert an extra <CR><LF> before the
terminating line? I notice that at least some implementations do not.

Cheers,

Ben.

-- 
http://www.apache-ssl.org/ben.html       http://www.thebunker.net/
**  ApacheCon - Dec 10-14th - San Diego - http://apachecon.com/ **
"There is no limit to what a man can do or how far he can go if he
doesn't mind who gets the credit." - Robert Woodruff

---

From: [email-masked] (Daniel A. Nagy)
Date: Wed, 30 Nov 2005 14:09:46 +0100

> that they should be stripped from the dash-escaped text?

It doesn't matter.

> terminating line? I notice that at least some implementations do not.

Yes, it does. The '-----BEGIN PGP SIGNATURE-----' must begin in a new line.
The line termination before it is not part of the signature. AFAIK, all
implementations treat it that way.

-- 
Daniel

---

From: Ian G <[email-masked]>
Date: Wed, 30 Nov 2005 13:19:31 +0000

> that they should be stripped from the dash-escaped text?

They should not be included in the signature
calculations.  It is an open question as to
whether they should stripped from the text or
not, up to each application.  I would;  but
Jon has posted on good reasons why it is not
the job of the application to change the doc
that is being processed (signed).

I suppose the above text could change "generated"
to "calculated" to make it clearer?  That is, if
my interpretation is the consensus.

> terminating line? I notice that at least some implementations do not.

No, there is no need to insert anything, just
not include the <CR><LF> that must preceed the
'-----' line in the signature calculation.  In
this case I would say that there definately
should not be an extra line ending inserted,
as that is changing the document in a way that
is not reversable.

This is an artifact of the old days where
optimisations were cute, and it was I suppose
considered smart to save on calculating over the
last newline.  It is particularly dumb, because
everything else about the job is line based,
but this is an exception with no value other
than creating an exception and thus annoying
the coders.  None that I ever heard, leastways.

But there was a discussion about it a long time
ago, and no consensus to fix it up, so we are
stuck with the exception.

IMO!

iang

---

From: Ben Laurie <[email-masked]>
Date: Wed, 30 Nov 2005 13:42:34 +0000

> that is being processed (signed).

I would agree, if you ended up signing the unchanged document :-)

As it is, since I think the idea of stripping them is stupid, I don't
really care either way, but it would be good if the document were clear.

> my interpretation is the consensus.

I think you have to say "...at the end of the line is not included in
the signature calculation" to remove the ambiguity (if leaving them in
the text is intended).

> as that is changing the document in a way that

No - what you are proposing is not reversible. Always adding a <CR><LF>
_is_ reversable.

Cheers,

Ben.

---

From: Ben Laurie <[email-masked]>
Date: Wed, 30 Nov 2005 13:35:38 +0000

>> that they should be stripped from the dash-escaped text?

Of course it matters!

> implementations treat it that way.

That doesn't answer the question.

---

From: [email-masked] (Daniel A. Nagy)
Date: Wed, 30 Nov 2005 15:05:33 +0100

> the coders.  None that I ever heard, leastways.

I disagree. The current system allows for texts where the last line has no
line ending. It means, that signed text files can be converted back and
forth between clear-signed and one-pass signed representations, without
losing the signature. This is a very neat feature of OpenPGP, that still
makes a lot of sense.

The following implementation actually uses it (requires java plugin):
http://pgp.epointsystem.org/tool

Unlike hushmail, it does not encrypt clearsigned documents, but if you
decrypt a signed text message, it will output the clearsigned version, which
you can verify. See http://pgp.epointsystem.org/tool-help for usage.

-- 
Daniel

---

From: Ian G <[email-masked]>
Date: Wed, 30 Nov 2005 14:09:21 +0000

> implementations treat it that way.

It does?  I stand corrected.  I see what you are
getting at - some documents will not have a newline
on (mostly application/windows style documents) and
then it will need to have that added.  Which creates
a problem in deciding whether one was added or not
afterwards when stripping the signature.

Hmmm...

Looking back through old signed contracts it appears
that Cryptix adds an extra line always but PGP 5.5
does not, and GPG does not unless needed (see checks
below).

OK, so clarification needed.  What is the rule here -
when is a line added:

     - always?

     - only when needed to correct the line based content?

Well spotted, Ben!

iang

==================== GPG does not add a line unless needed!

galland$ gpg -at --sign -u DSS3 moo
gpg: WARNING: using insecure memory!
gpg: please see http://www.gnupg.org/faq.html for more information

You need a passphrase to unlock the secret key for
user: "Ian Grigg DSS3 <[email-masked]>"
1024-bit DSA key, ID DABCCA96, created 2000-03-26

galland$ cat moo.asc

one
two
galland$ cat moo
one
two
galland$ dd if=moo of=poo bs=1 count=7
7+0 records in
7+0 records out
7 bytes transferred in 0.000147 secs (47585 bytes/sec)
galland$ cat poo
one
twogalland$ gpg -at --sign -u DSS3 poo
gpg: WARNING: using insecure memory!
gpg: please see http://www.gnupg.org/faq.html for more information

You need a passphrase to unlock the secret key for
user: "Ian Grigg DSS3 <[email-masked]>"
1024-bit DSA key, ID DABCCA96, created 2000-03-26

galland$ cat poo.asc

one
two
galland$ cat poo
one
twogalland$

---

From: [email-masked] (Daniel A. Nagy)
Date: Wed, 30 Nov 2005 14:58:12 +0100

> >> that they should be stripped from the dash-escaped text?

No, it does not. Either way, the signature will verify. It is up to the
implementation to strip or not to strip empty line endings. It can even
change them (e.g. add a few spaces and tabs after some of the line endings).

Canonization is an idempotent operation. Documents that are the same after
canonization are considered identical as far as text signaures go.

For pure ascii text, a printed clearsigned document, if typed back in, should
verify. Unfortunately, with unicode this is no longer the case, as there is a
whole bunch of different letters that look exactly the same (e.g. "o" and
"о" -- the latin and cyrillic "o", repsectively). Actually, this is problem
for legal applications.

-- 
Daniel

---

From: David Shaw <[email-masked]>
Date: Wed, 30 Nov 2005 09:32:58 -0500

> that they should be stripped from the dash-escaped text?

A goal of cleartext signatures is that they can survive this sort of
whitespace mangling in transport, so it's actually unimportant if the
whitespace is removed from the dash-escaped text or not.  Either way,
they are not part of the signature, so even implementations that
differ on this point can still verify each others signatures.

> terminating line? I notice that at least some implementations do not.

I think always writing an extra CRLF is the wrong thing to do, if it
is not present in the original document.  The text from the draft here
just says that the last CRLF (the CRLF that is required to be put the
BEGIN PGP SIGNATURE line at beginning of a line) in the document is
not part of the signed text.

As I see it, reasonable behavior here would be to copy in the original
document (with line ending conversion done) faithfully, then add the
terminating "CRLF-----BEGIN PGP SIGNATURE".  If the last line of the
original document had a line ending (transformed to CRLF here), then
the clearsigned result would include it in the hash and the output:

  this is my last line of text  <-- there was an existing CRLF here
  <--- this is the CRLF from "CRLF-----BEGIN PGP SIGNATURE"
  -----BEGIN PGP SIGNATURE-----

In this case, the hash ends with "this is my last line of textCRLF".

If the last line of the original document didn't have a line ending,
then there would be no blank line, as the CRLF from ----BEGIN PGP
SIGNATURE----- effectively ends that line.

  this is my last line of text  <--- this is the CRLF from "CRLF-----BEGIN PGP SIGNATURE"
  -----BEGIN PGP SIGNATURE-----

In this case, the hash ends with "this is my last line of text" (no
CRLF).

This preserves the existence (or not) of the final line ending in the
clearsigned document.

David

---

From: Ian G <[email-masked]>
Date: Wed, 30 Nov 2005 14:33:07 +0000

> really care either way, but it would be good if the document were clear.

The issue that it is meant to solve is that
many applications end up adding whitespace
to the ends of lines.  Not just mailers, but
editors, cut&paste, etc.

So the "do not include trailing whitespace"
is an attempt to make the cleartext signature
more robust within the text world it inhabits.
For me, it is a good feature, it makes the
contracts that I send around more reliable;
but always recognising that there are limits
to this game, as the same tools that do this
also do silly things like slicing up lines
and doubling up on newlines.

> the signature calculation" to remove the ambiguity (if leaving them in

I agree with that.  So I'd agree that this
is a late change that might slip in before
the end if we still have time (always looking
over shoulder to see if the final date is
about to run us over...):

     Also, any trailing whitespace -- spaces (0x20) and tabs (0x09) -- at
     the end of any line is not included in the signature calculation."

Maybe add:

     It is an application decision whether to remove trailing whitespace.

(or maybe not.  This horse has died so many times.)

> No - what you are proposing is not reversible. Always adding a <CR><LF>

Right.  See response to Daniel, who pointed out
the fuller understanding.  So if that is the rule,
we need:

     The line ending (i.e. the <CR><LF>) before the '-----BEGIN PGP
     SIGNATURE-----' line that terminates the signed text is not
     considered part of the signed text."

to change to:

     On signing, a line ending (i.e. the <CR><LF>) is always
     inserted at the end and before the '-----BEGIN PGP
     SIGNATURE-----' line that terminates the signed text.
     This line ending is not part of the signed text nor the
     signature calculation.  It should be stripped when the
     signature is verified and an unsigned document revealed.

Or somesuch.  But I'd like to hear from some of
the old timers to hear whether that is the rule
here.  The alternative is that we simply define
the document as always having a trailing newline,
and that may be non-reversable.

iang

---

From: Ian G <[email-masked]>
Date: Wed, 30 Nov 2005 14:55:55 +0000

> change them (e.g. add a few spaces and tabs after some of the line endings).

I think what you mean is that the whitespace
should not be included in the calculation,
but it doesn't matter whether they are stripped
from the document itself.

> canonization are considered identical as far as text signaures go.

(I've always seen it written as canonicalisation
or canonicalization.)  So the question is whether
we canonicalise the document or simply canonicalise
the signature processing.  I think the answer is
that it is open to the application.

> "Ð¾" -- the latin and cyrillic "o", repsectively). Actually, this is problem
> for legal applications.

It is an issue, yes.  We discussed this a while
back and came to the conclusion that *only* ascii
whitespace was to be stripped/ignored, as the
alternate was too hard to define.  That's why
the specific characters to be stripped are in
the spec - to stop people looking for cyrillic
spaces or different sized spaces.

iang

---

From: [email-masked] (Daniel A. Nagy)
Date: Wed, 30 Nov 2005 16:50:13 +0100

> from the document itself.

Yes, that is precisely what I mean.
 
> spaces or different sized spaces.

Unfortunately, unlike canonical ascii, there is no one-to-one correspondence
between unicode characters and glyph visuals. It would be nice to have some
canonical form for unicode text which is human-readable, yet has a unique
binary representation, but it's not easy and not the job of this WG, IMHO.

-- 
Daniel