Subject: cleartext signatures - trailing white space - recap

From: Ian Grigg <[email-masked]>
Date: Thu, 11 Mar 2004 19:40:30 -0500

I think the *key* email extracts are below on this issue,
but this is my choice, and I've been harsh in stripping
things out, to concentrate.  I also may have missed some
mails, and left out a couple that seemed to have no traction.
(Apologies to those!  Please repost if necessary.)

David Shaw introduced the issue, as well as the issue of
space-stripping in text-mode, which has received relatively
little attention.  It may be that if we can crack the nut
of cleartext signature canonicalization, then the text-mode
sig falls out easily.

iang

===================== David Shaw, 2004.02.11
2) 2440 says of the cleartext signature:

         Also, any trailing whitespace (spaces, and tabs, 0x09) at the
	end of any line is ignored when the cleartext signature is
	calculated.

    Again, PGP through 8 implements this differently than 2440 says,
    where trailing spaces are removed, but trailing tabs are not
    (again, PGP 2.x behavior).

===================== Jon Callas, 2004.02.20
  How about if we remove any whitespace things, and just
  canonicalize line ends? It sounds like Unicode
  whitespace may be a huge can of worms. Alternatively,
  we could just say trim anything that's <= 0x20, which
  is a simple enough thing that solves some obvious
  attacks with backspacing and bare CRs to overstrike.

===================== Ian Grigg 2004.02.20
My vote would be to trim whitespace and normalise
line endines to CR/NL, where whitespace is <=0x20:

     Also, any trailing whitespace (characters <= 0x20) at the
     end of any line is ignored when the cleartext signature is
     calculated.

I think there should be a comment in there that
indicates what to do with Unicode, just to show
we thought about it, and not waste people's time
asking the question when they are implementing.
Something like:

     Unicode whitespace, where defined, SHOULD NOT be ignored.

Or,

     No Unicode whitespace characters are defined.

Leaving open the possibility of defining them in
an update?

===================== Hal Finney, 2004.02.20
{comprehensive list of unicode spaces, elided}

Therefore I think all of these should be hashed
even if they do occur at the end of a line.

...

The only one left is IDEOGRAPHIC SPACE, which I suspect is the default
space character in ideographic languages (although it's possible they use
ordinary SPACE).  I could imagine it being put at the end of a line by
accident, by a Chinese typist or poorly designed word processing program,
so I'd suggest that it should be stripped before hashing.

This is the only one I would suggest adding, along with SPACE.

===================== Derik Atkins, 2004.03.08
* Trailing White Space: The issue is that some e-mail gateways strip
trailing white space on lines when processing mail messages.  This
cause signature validation failure at later date.  The question is
whether this is an issue that needs to be addressed.

One proposal is to strip EOL characters where the character <= 0x20.  From
the floor it was pointed out that this could cause problems from two things.
1) there are some control characters that may be part of the text stream
(such as page feeds) that should not be stripped and 2) for some languages
escape characters for local language processing might produce characters
that are in this character range and thus produce corruption of the text.

One suggestion was to do the standard MIME time canonicalization and ignore
the rest of the issues.  If the message is changed by stripping spaces in a
gateway, then the message correctly fails validation.

As no text has been proposed or was proposed from the floor the issue was
punted back to the authors to propose some text.

=====================

---

From: Ian Grigg <[email-masked]>
Date: Thu, 11 Mar 2004 20:34:04 -0500

Comments on some areas, below, assuming continued
debate.  Proposals in next email.

> This is the only one I would suggest adding, along with SPACE.

My view on this is that a) Hal's summary is
very good, but not necessarily complete, and
b) I'm not sure we have the wherewithall to
be able to predict even the characters that
are there.

So, I would say that any Unicode whitespace
that are encounted SHOULD NOT be treated as
whitespace, in this context.

Later implementations may divine more
clearly what to do here, in which case
they might be encouraged to create an Armor
Header that states how to treat.  Otherwise,
the default is that Unicode characters have
no special treatment, for simplicity, IMHO.

> (such as page feeds) that should not be stripped and

I'm unsure what to make of this - any comments?

If page feeds shouldn't be stripped, then maybe
backspaces shouldn't be stripped, and we are
back to space/tabs being stripped?

 > 2) for some languages

The way I see this is:

If a character set outside Unicode is being used,
then that should be indicated in the Armor Headers,
and then interpreted properly such that corruption
is not present.  If not, it will also muck up on
line endings CR/NL.

Elsewise it is in Unicode, and the rules apply.

> gateway, then the message correctly fails validation.

I'm not quite sure how MIME canonicalization works,
but the issue is wider than mail, things like
cut&paste are widely used for cleartext signed
documents, and these tools tend to add spaces
on the end.

iang (proposal to follow)

---

From: Ian Grigg <[email-masked]>
Date: Thu, 11 Mar 2004 20:40:56 -0500

> punted back to the authors to propose some text.

The issue was more that consesnus hadn't been achieved,
and that text which was proposed was indicative only.

But here goes: it comes down to, as I see it, one of
two choices (full example at end):

OR:

     Also, any trailing whitespace (0x20, 0x09) at the
     end of any line is ignored when the cleartext signature is
     calculated.

Note 1.  I'd like "space" defined as a 0x20 to remove the possibility
of Unicode spaces.

Note 2.  Before it said (spaces, and tabs, 0x09) which is odd, as
tabs are 0x09, unless I have it crossed.  Where vertTabs (0x0B)
supposed to be in there?

Note 3.  I'm ambivalant on the choice.

Note 4.  And, to clarify Unicode, I suggest adding:

     No exception for Unicode whitespace is defined,
     and all Unicode characters SHOULD NOT be ignored.

iang

PS: while we're at it, it might be better if a new
section is added, 7.2, and the paras re-ordered as
below.  Just a thought...  I also took the liberty
of joining the "From " on one line, para 2, line 6.

7.1. Dash-Escaped Text

     The cleartext content of the message must also be dash-escaped.

     Dash escaped cleartext is the ordinary cleartext where every line
     starting with a dash '-' (0x2D) is prefixed by the sequence dash '-'
     (0x2D) and space ' ' (0x20). This prevents the parser from
     recognizing armor headers of the cleartext itself. An implementation
     MAY dash escape any line, SHOULD dash escape lines commencing
     "From " (note the space), and MUST dash escape any line commencing
     in a dash. The message digest is computed using the cleartext
     itself, not the dash escaped form.

     When reversing dash-escaping, an implementation MUST strip the
     string "- " if it occurs at the beginning of a line, and SHOULD warn
     on "-" and any character other than a space at the beginning of a
     line.

7.2. Canonical Line Endings

     As with binary signatures on text documents, a cleartext signature
     is calculated on the text using canonical <CR><LF> line endings.
     The line ending (i.e. the <CR><LF>) before the '-----BEGIN PGP
     SIGNATURE-----' line that terminates the signed text is not
     considered part of the signed text.

     Also, any trailing whitespace (characters <= 0x20) at the
     end of any line is ignored when the cleartext signature is
     calculated.  No exception for Unicode whitespace is defined,
     and all Unicode characters SHOULD NOT be ignored.

8. ...

---

From: Werner Koch <[email-masked]>
Date: Fri, 12 Mar 2004 11:23:34 +0100

>      Also, any trailing whitespace (characters <= 0x20) at the

Please don't define whitespace this way.  I know software using
control characters to separate fields (e.g. STX (0x02) or FS (0x1c))
in a line. Ignoring them in a signature (at the end of a line) might
very well change the content of the message (even if those fields are
empty).  

SPACE, LF, CR and TAB are the whitespace characters we have always
used in PGP and so should it be - that is also what most
programmers[1] understand under whitespace (cf. K&R).  VT and FF would
also belong to them, but given that we did not used them in PGP, I's
feel better not to add them now.

> Note 1.  I'd like "space" defined as a 0x20 to remove the possibility

Agreed.

>      and all Unicode characters SHOULD NOT be ignored.

With a list of white space caracters along with their encoding values,
we won't need that.

>      and all Unicode characters SHOULD NOT be ignored.

     Also, any trailing whitespace (the characters <SPACE> (0x20),
     <TAB> (0x09), <CR> (0x0D) and <LF> (0x0A)) at the end of any line
     is removed and replaced by the sequence <CR><LF> for the purpose
     of cleartext signature calculation.

  Werner

[1] Well, speaking of C programmers; don't know about Java.

---

From: "Brian G. Peterson" <[email-masked]>
Date: Fri, 12 Mar 2004 06:44:44 -0600 (CST)

>      of cleartext signature calculation.

I would modify Werner's proposed text like so, for grammatical clarity:

       When calculating a cleartext signature, trailing whitespace
       (defined as the characters <SPACE> (0x20), <TAB> (0x09),
       <CR> (0x0D) and <LF> (0x0A)) at the end of any line shall be
       removed and replaced by the sequence <CR><LF> for the purpose
       of calculating the cleartext signature.  This protects against
       a failure of a valid signature due to many conversions that may
       occur in text editing tools or in transport.

I think it is important to clarify both the definition and a little of the
reasoning behind this for readers and implementers of the RFC.

Regards,

    - Brian

---

From: Ian Grigg <[email-masked]>
Date: Fri, 12 Mar 2004 08:33:40 -0500

>        occur in text editing tools or in transport.

( See comments to Werner - it's not so easy to
combine the "line ending" processing with the
"whitespace" processing. )

> reasoning behind this for readers and implementers of the RFC.

I agree;  this is one area where a little more
commentary can help solve a lot of little
annoying compatibility issues.

iang

---

From: Ian Grigg <[email-masked]>
Date: Fri, 12 Mar 2004 08:31:16 -0500

> very well change the content of the message (even if those fields are

OK, that makes for two votes for an explicit short
list of whitespace characters.

> programmers[1] understand under whitespace (cf. K&R).

I think there is a difference between whitespace and
line endings, as far as OpenPGP cleartext signatures
are concerned, at least.

The issue comes when you get files that are garbled
in their line endings:

    line< ><CR><CR><LF>

Or

    line<CR> <LF><CR><LF>

and various other combinations.  In the past, when
coding up that sort of thing, I've adopted the strategy
of saying that any change in the nature of the line
endings is treated immediately as a panic (caller to
fix).  E.g., there is at least one manifest error, and
trying to determine the error as being either a line
ending error or a whitespace error makes for too many
complications in the code.

In sum, I'm not sure that we want to define whitespace
in this immediate context as including the legal line
ending characters...  Comments?

 > VT and FF would

If we are going for a list of characters, the shorter
the better, in general.  It seems more likely that
these characters VT, FF, have defined meaning within
the text than are likely to be added later by
transmission gremlins.

> With a list of white space caracters along with their encoding values,

I know we don't need it, but without an explicit
mention of Unicode, I suspect there will be a
an endless stream of questions, and also, people
will start including their Unicode whitespace
chars because there is no explicit guidance...

> [1] Well, speaking of C programmers; don't know about Java.

Perl uses this definition of whitespace:

   \s      A whitespace character      [ \t\n\r\f]

which includes form feeds as 0x0c (I think).

Java uses the java.lang.Character.isWhitespace()
method, which probably depends on the character
set!

I don't know about Python, or Microsoft languages.

This underscores is that the ID should NOT rely
on any languague's definition of whitespace, and
should seek to define explicitly what is meant.

iang

---

From: Stuart A Yeates <[email-masked]>
Date: Fri, 12 Mar 2004 13:54:04 +0000

> which includes form feeds as 0x0c (I think).

Modern versions of perl also have a unicode compliant definition of 
whitespace (and other things) see:
http://perl.active-venture.com/pod/perlretut-morecharacter.html

> method, which probably depends on the character

java.lang.Character.isWhitespace() operates solely on chars, which are 
unicode.

> I don't know about Python, or Microsoft languages.

Modern versions of Python do unicode, I believe.

cheers
stuart

-- 
Stuart Yeates            [email-masked]
OSS Watch                                  http://www.oss-watch.ac.uk/
Oxford Text Archive                             http://ota.ahds.ac.uk/
Humbul Humanities Hub                         http://www.humbul.ac.uk/

---

From: Werner Koch <[email-masked]>
Date: Fri, 12 Mar 2004 15:56:50 +0100

> line endings, as far as OpenPGP cleartext signatures

Yes, but LF is also often used as a whitespace character.

>     line< ><CR><CR><LF>

Our lineending is <CR><LF> and thus I would remove the space and the
first <CR>.

>     line<CR> <LF><CR><LF>

It is easier for systems where the line ending is just one character
;-) In practise most systems do that and use LF (is it CR on Macs?).

> endings is treated immediately as a panic (caller to

However we are talking about what we need to hash and we explicitly
want to convert the line for this purposes - that is what textmode is
about.

> ending characters...  Comments?

1. Determine the length of the line (according to the convention of
   the system the application runs).  
2. Trim trailing whitespace;  including CR and LF.
3. Append CR,LF
4. Hash the line

Agreed.

> an endless stream of questions, and also, people

Okay, so lets add it.

  Werner

---

From: David Shaw <[email-masked]>
Date: Fri, 12 Mar 2004 10:37:54 -0500

> Our lineending is <CR><LF> and thus I would remove the space and the

[..]

> 2. Trim trailing whitespace;  including CR and LF.

I strongly agree with this.  The whole point of trimming whitespace in
cleartext signatures is to protect the signature from accidental
whitespace damage via email and cut and paste and the like.  Leaving
any whitespace at the end of the line sort of defeats that point.

We can perhaps discuss what characters count as whitespace, but I
think that all of them should be removed.

David

---

From: Ian Grigg <[email-masked]>
Date: Fri, 12 Mar 2004 11:53:20 -0500

> ;-) In practise most systems do that and use LF (is it CR on Macs?).

Life would indeed be easier if everyone
just got the religion, yes.  Macs switched
to BSD for OSX, since about 2001, so they
now use LF for many things.  No doubt there
is some legacy CR Mac stuff floating around.

Java tries to use LF internally, but sometimes
there are messy conversions and assumptions
applied in some routines that interface with
the outside world.

> want to convert the line for this purposes - that is what textmode is

Reflecting on this, it matters less how it is
defined in the ID on this particular point,
as there will be a signature verify failure
regardless if the recovery goes wrong.

So I withdraw my previous comments w.r.t.
CR/NL being in the list.

>    the system the application runs).

Which system?  The OS?  The language?  The
message delivery program?  How about this:

1.  Convert bytes into character stream
     (characters being Unicode).
2.  Determine the nature of the line endings
     (by reference to the system convention,
     to inspection:  CR, LF, CR/LF).
3.  Break into an array of lines, separating
     on the line ending in 2.
4.  Trim all trailing whitespace (sp,tab,cr,lf).
5.  Add cr/lf to every line except the last.
6.  Concatonate the array.
7.  Hash.

Mind you, we are now into implementation, which
goes beyond the ID unless people think an
explanatory psuedocode note would help.

iang

PS: btw, I just got a bounce of a mail sent to you
(and CC'd to the PGP users group) in 1997.  It came
from some german mail list at
[email-masked]
... what's happening over there?

---

From: Jon Callas <[email-masked]>
Date: Fri, 12 Mar 2004 14:07:56 -0800

> chars because there is no explicit guidance...

All OpenPGP text is Unicode, so there's no need for *another* explicit 
mention of this. The bottom 128 characters of Unicode are ASCII, and 
the bottom 256 character of Unicode are ISO Latin-1.

What we colloquially say that a space is 0x20, that also means 0x0020 
and 0x00000020 as well.

While I agree that the ability for people to ask exasperating questions 
knows no bounds, there's such a thing as too much explicitness.

	Jon