Subject: Re: Outstanding question - rule on cleartext signing last line

From: [email-masked] ("Hal Finney")
Date: Tue, 27 Dec 2005 09:38:05 -0800 (PST)

I'd suggest that clearsigning is aimed more at applications which will
present the signed document to users as it was received "on the wire",
so that they can read it usefully even with the signature in place.
For that purpose the important thing is to know exactly how the signature
will be verified so that signers can create valid documents.  I think
we agree that the spec is adequate for this purpose.

For data interchange, you want to start with a document on one machine
and sign or encrypt it, send it to another machine and verify/decrypt,
getting back the same document.  For that purpose the binary formats
work well.

The problem arises if you want to clearsign but still get data
interchange.  The sender needs to know not only how the data will get
verified, but also how it will be stored if the signature is stripped
away.  We didn't spec this because it didn't seem to fit that well into
the clearsign model, which is oriented around presenting data with the
signatures in place.

So what are the rules for storing a clearsigned message, with signature
stripped away?  Would we just want to store the byte stream that
was hashed (the portion of it that was from the message and not the
signature bytes)?  Or would we want to switch to native line terminators
when we store?  Would we strip or keep trailing whitespace when we store?

If we did spec this, it would then by implication tell a sender how
to at least approximate data interchange.  Problems might arise if line
terminators are getting changed.  And if we strip trailing whitespace when
we store, then it would be impossible to send a document with trailing
whitespace and have it be received that way.  OTOH the reason we strip
trailing whitespace for verification is because some mail systems corrupt
that part, so retaining trailing whitespace on store will not guarantee
data interchange either.

In short, clearsigning is not really appropriate for data interchange
because of how many ways the message could be corrupted in transit.

But if we do spec it, I would definitely NOT then put in the spec that
a sender was FORCED to offer a data interchange option!  That is far,
far outside anything we should be imposing on an implementation, in a
data format spec.

Hal Finney

---

From: Ian G <[email-masked]>
Date: Tue, 27 Dec 2005 18:25:07 +0000

Hi Hal,

that is fine;  in which case I'd just suggest
that the spec have a simple one liner in there
that suggests that the cleartext format is not
required to be reversible / for data interchange:

   An implementation MAY produce a signature-stripped
   version of the document, but the format is not
   intended for data interchange, and there are
   some minor artifacts such as with the terminating
   newline.

(Or something hopefully shorter.)

I admit I couldn't think of a case where we wanted
to reverse / interchange.  Still, the spec is less
than clear on this point, and for programmers who
don't follow all the background, a note would be
useful.

Whichever.  I really don't mind which way it goes,
but the spec isn't helpful here.  Programmers will
and do expect the cleartext signing format to be
reversable, and it isn't.

iang

---

From: Ben Laurie <[email-masked]>
Date: Tue, 27 Dec 2005 18:25:38 +0000

> getting back the same document.  For that purpose the binary formats

Not for text documents. It seems to me that it is a reasonable
expectation that if I clearsign a text document, I should be able to
recover the document from the clear signature. Currently that isn't
possible with certainty.

I do not see the harm in fixing this issue.

-- 
http://www.apache-ssl.org/ben.html       http://www.thebunker.net/

"There is no limit to what a man can do or how far he can go if he
doesn't mind who gets the credit." - Robert Woodruff

---

From: David Shaw <[email-masked]>
Date: Tue, 27 Dec 2005 14:29:44 -0500

> the clearsign model, which is oriented around presenting data with the

I think the concern is not so much that we need the ability to data
interchange, but that there is a point of ambiguity in the standard
such that

  Un-Clearsign ( Clearsign ( A ) ) != A

in some cases, specifically around the lack or presence of a line
ending on the final line of A.

David

---

From: [email-masked] (Daniel A. Nagy)
Date: Tue, 27 Dec 2005 22:27:16 +0100

> ending on the final line of A.

I think, clearsigned and one-pass signed text documents should work the
same way. Data interchange AND human-readability together are worth
pursuing. I think, it is perfectly reasonable to expect that

Canonize ( Un-Clearsign ( Clearsign ( A ) ) ) == Canonize ( A )

-- 
Daniel

---

From: David Shaw <[email-masked]>
Date: Tue, 27 Dec 2005 17:25:32 -0500

> Canonize ( Un-Clearsign ( Clearsign ( A ) ) ) == Canonize ( A )

That is a much larger task than the current final-line-ending
question.  Canonical text for a literal packet type 't' and canonical
text for clear signing are not the same.

In a 't', line endings are converted to CRLF.  In a clear signature,
line endings are converted to CRLF and trailing whitespace is removed.
Thus, you can convert a clear signature to a signed 0x01 document, but
not necessarily from a signed 0x01 to a clearsigned document.

I don't think it is reasonable to expect interchange here.  If text
interchange is desired, then there is a form for that (literal 't').
Clear signed is intended for different situations where trailing
whitespace may well not survive transport - clear text
canonicalization is a lossy format because of this.  The fact that a
clear signed document can be converted to a signed 0x01 document is an
interesting curiosity, but not much more than that.

I have a mild interest in fixing the clearsign end-of-document
ambiguity, but I don't want to start changing the canonicalization
rules for either 't' or clear text.

David

p.s. Incidentally, speaking of clear signed data, there are
differences in practice in what trailing whitespace is removed.  Some
programs remove space and tab.  Some programs remove only space.
Currently, a trailing tab in a clearsigned document is an effective
way to break interoperability.

---

From: David Shaw <[email-masked]>
Date: Tue, 27 Dec 2005 18:59:06 -0500

> ending on the final line of A.

Okay, obviously I didn't express that well.  I didn't mean to say that
we don't need data interchange, but yet we do need data interchange.

What I'm trying to say is that we don't need full data interchange,
but it might be nice to fix (or even just document?) this particular
issue.  Given different implementations of end of line at end of
document, verifying a clearsigned document across implementations can
add or subtract blank lines at the end.

David

---

From: Jon Callas <[email-masked]>
Date: Tue, 27 Dec 2005 23:20:26 -0800

I have some general opinions about this issue.

(1) I don't think the spec should change. The reason that I don't  
think it should change is that we're in last call. We want to tidy  
this thing up. Changes to the spec are most likely going to get  
something that is "right" but causes a decade of interoperability  
twists that we will "resolve" by declaring old versions to be bogus  
and the new behavior that no one does to be right. Then a half-dozen  
years from now, we'll go back to whatever it says now.

If you don't believe me, this is *exactly* what happened with the  
whole blank-trimming thing.

Now, then, I am not opposed to having a clarification in the spec.  
I'll merely allude to my standard comment that OpenPGP Formats is not  
a How To Write An OpenPGP Application which everyone's heard at least  
once. And yes, I know there's a big fat fuzzy grey line between  
clarifications and telling people how to code.

(2) I don't see how in the general case clearsigning can be a  
reversible operation. There are several reasons for this. We have a  
way to sign data reversibly. That is binary-mode signing. Binary-mode  
signing is not clearsigning. We also have an higher-level  
abstraction, text-mode signing. Text-mode signing is not, in the  
general case, reversible. The reason is that OpenPGP takes this  
abstract text object and then translates it into the native text  
format of the receiver's system. If one of us is on a unix box, and  
one of us is on Windows, then it's very difficult for us to talk  
about what the heck reversibility is. Ambiguities about text-mode are  
not new to OpenPGP. They date back at least as far as FTP, and they  
are a good thing, not a bad thing.

Clearsigning is an abstraction built on top of text mode that further  
abstracts the signature. Text-mode signing is not human-readable. It  
still involves packets. Clearsigning is human-readable and that's the  
whole point.

I believe that good human interaction is a better virtue than  
reversibility. Let me give an example.

Consider these three text files:

"a"     (the file containing the letter 'a')
"a\n"   (the file containing an 'a' and followed by a line-end)
"a\n\n" (the same, but with two line-ends)

If the first two look the same when clearsigned, this is not a bug,  
it's a feature. The point of clearsigning is that it be a pleasant  
experience for the user who wants to read the text, while making it  
so that a signature can be verified. I expect the third one to have  
one more blank line than the second one. I don't care how many more  
it has than the first one.

I'm on a unix system and if I 'more' each of the first two, they come  
out the same on my terminal. If I 'cat' them, they do not, and I  
believe that this is braindamaged, even if it is "right" and even  
after you explain to me in detail that it's "right." It may be right,  
but it's still braindamaged.

Whatever GnuPG, PGP, etc. are doing are okay. I see lots of  
clearsigned messages and nothing ever rubs my aesthetics the wrong  
way. The purpose of clearsigning is to make it easy to read by  
humans. Abstract properties like reversibility are secondary; if you  
want the thing to be reversible, then use a detached signature, for  
Pete's sake!

If there is some application out there that were to compress a  
zillion newlines into some reasonable number like one, two, or three  
in the output of a clearsigned message, then more power to them!  
There's nothing in OpenPGP that says you can't do that.

(3) I am firmly against any tweak to the spec, including commentary,  
that requires any implementation that's got versions past 1.0.1 to  
have to make a code or behavior change. It's far, far too late for that.

Whatever is out there is good enough. If we can make life easier for  
the likes of Ben and Rachel and other people who are making new  
systems, we should. But this should not burden any existing system  
one iota.

	Jon

---

From: [email-masked] (Daniel A. Nagy)
Date: Wed, 28 Dec 2005 10:02:41 +0100

In my opinion, the spec is clear enough about how to verify clearsigned
messages and that is all that must be specified. I agree with Jon that it is
ultimately up to the applications how they pre-process the text to be
signed, as long as the human-readable part does not change (you sign what
you see).

I don't want to standardize either behavior: in my implementation
(ePointPGP) cleartext signatures are reversible and it is actually
important. I don't want the standard to declare me wrong. I don't want the
standard to declare widely used implementations such as GPG wrong either. I
regard it primarily as a back-end, and if someone wants to make reversible
clearsigned documents and be sure about it, it's still possible with
detached text signatures and perhaps some trivial pre-processing. The end
result can be put together as a clearsigned document. The behavior won't
change if Werner decides to tweak clearsigning a bit further in a future
version of GPG.

-- 
Daniel

---

From: Ian G <[email-masked]>
Date: Wed, 28 Dec 2005 11:43:34 +0000

David Shaw wrote:
> I don't think it is reasonable to expect interchange here.

I'm also of that view, mildly.  The purpose of
cleartext signing is so that we can pass around
signed documents and have them survive as such,
with a human-reachable import.

In my time using this feature with contracts, I
have never come across a case where interchange
/ reversibility is needed.  In fact, to me, it
is a nuisance that verification of the signature
causes a stripped file to be produced.

I'm scratching my head here and I can't quite see
it.  Can anyone?

> way to break interoperability.

Right.  Also there is UTF.  That's why we banged the
drum a while ago and got this inserted:

     Also, any trailing whitespace -- spaces (0x20) and tabs (0x09) -- at
     the end of any line is removed when the cleartext signature is
     generated.

The characters to be removed are specified precisely.
If something is not removing tabs, then that's a bug.
A victory against high stress levels in developers.

iang

---

From: Ian G <[email-masked]>
Date: Wed, 28 Dec 2005 12:59:35 +0000

Jon Callas wrote:
> (1) I don't think the spec should change.

> reversible operation.

> ... Clearsigning is human-readable and that's the  
> whole point.

> that requires any implementation that's got versions past 1.0.1 to  have 
> to make a code or behavior change. It's far, far too late for that.

OK.

Suggested change from this (1st para, chapter 7):

     It is desirable to sign a textual octet stream without ASCII
     armoring the stream itself, so the signed text is still readable
     without special software. In order to bind a signature to such a
     cleartext, this framework is used.  (Note that RFC 3156 defines
     another way to sign cleartext messages for environments that support
     MIME.)

To this (changes are in the parenthesized Note):

     It is desirable to sign a textual octet stream without ASCII
     armoring the stream itself, so the signed text is still readable
     without special software. In order to bind a signature to such a
     cleartext, this framework is used.  (Note that this binding is not
     intended to be reversible.  RFC 3156 defines another way to sign
     cleartext messages for environments that support MIME.)

Changes are in the parenthesized Note and amount to the insertion
of the following CAPS:

    Note that THIS BINDING IS NOT INTENDED TO BE REVERSIBLE.  RFC ...

iang

---

From: Ben Laurie <[email-masked]>
Date: Wed, 28 Dec 2005 15:28:15 +0000

Jon Callas wrote:
> (1) I don't think the spec should change. The reason that I don't think
> it should change is that we're in last call. We want to tidy this thing
> up.

That's a crazy argument. If we're in last call and changes are needed,
then what that tells us is we shouldn't be in last call, not that we
shouldn't make changes!

This argument was used in DNSSEC to squash changes that are now coming
back with a vengeance years later. All that invoking "last call so don't
touch it" did was introduce about 5 years delay into the process.

Which is not to say I care passionately about reversability. I would,
however, like to remove the ambiguity.

Can we say implementations SHOULD always add a newline, but they MAY
omit it if the plaintext ends with a newline (for backwards
compatibility)? That way nothing is broken, but future behaviour is defined.

---

From: Jon Callas <[email-masked]>
Date: Wed, 28 Dec 2005 13:34:56 -0800

On 28 Dec 2005, at 7:28 AM, Ben Laurie wrote:

>> think

But, Ben, that's precisely my argument. When you're in last call, you  
should either make only small changes or blow the thing up, and go  
back to a non-last-call state.

There are no huge problems in clearsigning. It works. It's been  
deployed for 15 years now, if we include pre-OpenPGP.

> compatibility)? That way nothing is broken, but future behaviour is  

I'm happy to make that change. In fact, I just did.

	Jon

---

From: Jon Callas <[email-masked]>
Date: Wed, 28 Dec 2005 13:36:22 -0800

>    Note that THIS BINDING IS NOT INTENDED TO BE REVERSIBLE.  RFC ...

I added that in, too.

	Jon

---

From: David Shaw <[email-masked]>
Date: Wed, 28 Dec 2005 23:01:13 -0500

> I'm happy to make that change. In fact, I just did.

Forgive me, please, but I understood the rule before.  With this
change, I don't think I understand what is expected any longer.  Is
this added newline hashed into the signature?

Wouldn't "always" adding a newline cause text (with no line ending)
to become:

  -----BEGIN PGP SIGNED MESSAGE-----
  Hash: SHA1

  text

  -----BEGIN PGP SIGNATURE-----
  xxxxxx
  -----END PGP SIGNATURE-----

and text\n (with a line ending) to become:

  -----BEGIN PGP SIGNED MESSAGE-----
  Hash: SHA1

  text

  -----BEGIN PGP SIGNATURE-----
  xxxxxx
  -----END PGP SIGNATURE-----

If someone could post an example of what is now expected for "text"
and "text\n", I'd appreciate it.

GPG never adds a final newline to the hashed data, even if the
original document doesn't have one.  If there is a final newline in
the document, it is removed and not hashed into the signature.

Does this change to the draft make GPG noncompliant?

David