Subject: Re: cleartext signed messages - UTF-8 - stripping the whitespace

From: Ian Grigg <[email-masked]>
Date: Tue, 06 Jan 2004 12:29:22 -0500

Adrian 'Dagurashibanipal' von Bidder wrote:
> 
> On Monday 05 January 2004 16:24, Ian Grigg wrote:

...
> > 3.  What was the original deep dark motivation
> >     for stripping whitespace from the end of lines
> >     anyway?
> >
> > 4.  Do we care if UTF-8 has some weird whitespace/
> >     line endings?
> 
> IIRC from previous discussions (I wasn't around for years when PGP was
> introduced to the world...): some mailers (MUAs and MTAs) used to strip
> whitespace occasionally or do other weird things.

Thinking about it, I've come across editors and
desktops that do something similar, they add spaces
to the end of lines in arbitrary fashions, and
sometimes modify newlines (take away, add) at the
end of files (but newlines are adequately protected
already in the draft).

> Those old mailers would probably either treat all non-ASCII whitespace and
> line-endings as normal characters, or not be 8-bit clean anyway and so cause
> problems in any case. So the answer to (4) is probably a clear no.

So, the upshot is that only the defined US-ASCII
whitespace chars should be included in the
canonicalisation:

    ....
    Also, any trailing whitespace (spaces, and tabs, 0x09) at the end of
    any line is ignored when the cleartext signature is calculated.

In which case it might be worth adding a comment
to that effect.  Because of the difficulties of
predicting the future here, I'd suggest the following:

    Also, any trailing whitespace (0x20, 0x09) at the end of
    any line is ignored when the cleartext signature is calculated.
    Implementations MAY elect to clean line endings of whitespace
    in the final signed form of the document, including UTF-8 forms.

Thus, when we hit upon some troublesome mailer that
mangles non-english language Word-prepared UTF-8 documents,
the OpenPGP plugin can pre-emptively clean it up and
still be in accord with the standard.

If applications start adding UTF-8 whitespace afterwards,
then we have more of a problem.  I guess at that point,
implementations can agree to add additional characters
to the "ignore" list.

iang

---

From: "Adrian 'Dagurashibanipal' von Bidder" <[email-masked]>
Date: Mon, 5 Jan 2004 20:57:18 +0100

[...]

> 2.  Are there UTF-8 whitespace encodings that
>     are not in the definitions above?
>
>     I.e., not in "spaces, and tabs, 0x09" .
>

cheers
-- vbi

-- 
featured product: PostgreSQL - http://postgresql.org

---

From: Ian Grigg <[email-masked]>
Date: Mon, 05 Jan 2004 10:24:34 -0500

I'm working on some code that deals with cleartext
signed messages over UTF-8 [1].

In debugging the treatment of UTF-8, I looked at
the definition of mods that OpenPGP does to the
cleartext (paras 3,5) for signature treatment [2]:

    As with binary signatures on text documents, a cleartext signature
    is calculated on the text using canonical <CR><LF> line endings.
    The line ending (i.e. the <CR><LF>) before the '-----BEGIN PGP
    SIGNATURE-----' line that terminates the signed text is not
    considered part of the signed text.

    ....

Here are the questions:

1.  In UTF-8, are there such things as line
    endings that are not of <CR> and/or <LF> form?

5.  Are we explicitly ignoring these?

    Hence the question 3, perhaps the answer can
    guide us....

It seems the easiest thing is to say that we
explicitly do not include any UTF-8 characters
in the above discussion.  And add a clarifying
comment to that effect.

Perhaps also something to the effect of:

  Implementations
  MAY strip whitespace (including any UTF-8 whitespace
  that is recognised) from line endings before signing,
  so that the resultant cleartext signed message will
  not include any complex lines.

(That's essentially what I try and do in my code,
but I recognise that this goes beyond the standard....)

Alternatively, if someone can nail what UTF-8 does
in whitespace, it might be possible to put in more
consideration.

iang

[1] Code is based on Edwin Woudt's OpenPGP in Java
as found at
http://www.cryptix.org/products/openpgp/index.html
The application is Ricardian Contracts in Spanish.

[2] Looking at at section 7.1, Dash-Escaped Text, of the
current draft: draft-ietf-openpgp-rfc2440bis-09.txt
http://carmen.cselt.it/internet-drafts/draft-ietf-openpgp-rfc2440bis-09.txt