Subject: Meeting in Seoul: Timeslot acquired. Second call for agenda items.

From: Derek Atkins <[email-masked]>
Date: Wed, 11 Feb 2004 14:34:26 -0500

I've acquired a one-hour timeslot in Seoul:

  This is to confirm that OPENPGP is currently scheduled on Tuesday, March 2 at
  1300-1400

  Other groups scheduled at that time are:
  webdav, l3vpn, dnsop, nsis

The current agenda has two items:

1) 2440bis status
   - current state  (jon via derek, unless jon can make it)
   - list of issues (derek)

2) updating the charter/milestones
   (derek)

If you have other items to discuss, please let me know so I can add
them to the agenda.  I need to submit the agenda by Feb 24th.

-derek

-- 
       Derek Atkins                 617-623-3745
       [email-masked]             www.ihtfp.com
       Computer and Internet Security Consultant

---

From: David Shaw <[email-masked]>
Date: Wed, 11 Feb 2004 15:50:55 -0500

I'm not going to be in Seoul, but one thing I think would be good to
be discussed, whether on this list, in Seoul, or both, is the textmode
end-of-line and whitespace issues.

This is not a severe interoperability problem by any means, but is an
annoyance that pops up now and again, and I think contributes in a
small way to the poor interoperability reputation that OpenPGP has.

As I understand it, there are actually two different problems here.
I'll take them in order:

1) 2440 says of the canonical text signature (sigclass 0x01):

        The signature is calculated over the text data with its line
	endings converted to <CR><LF> and trailing blanks removed.

   This is different than what every version of PGP though 8 does.
   These implementations do the <CR><LF> line endings, but do not
   remove trailing blanks (essentially PGP 2.x behavior).

2) 2440 says of the cleartext signature:

        Also, any trailing whitespace (spaces, and tabs, 0x09) at the
	end of any line is ignored when the cleartext signature is
	calculated.

   Again, PGP through 8 implements this differently than 2440 says,
   where trailing spaces are removed, but trailing tabs are not
   (again, PGP 2.x behavior).

I've seen comments that these details were inadvertent errors in 2440
that would have or should have been fixed, and requests to the WG the
change these two details in 2440bis to match the historical PGP
behavior (after all, PGP 5 predates 2440 and there is a huge installed
base of PGP 5-8).  I've also seen comments that the WG mustn't change
the published standard this many years after the fact to match
behavior already declared noncompliant.

As for me, I don't really have strong feelings for any particular
outcome of this.  What I do care about is that once 2440bis is
published, that it is clear what the "right" way to do things is and
that there will not be questions later.  I don't want it suggested
that the issue wasn't looked at.

Jon Callas pointed out in
<http://www.imc.org/ietf-openpgp/mail-archive/msg03753.html>, that if
programs would just put their particular variation on canonical text
in the literal data packet accompanying the signature, then a lot of
the problems just go away.  This is a good point, and is, in fact,
what both PGP and GnuPG do, which is one reason why #1 hasn't been a
larger problem.  Unfortunately, this practice does not handle
canonical text detached signatures or cleartext signatures, as these
have no literal data packet.  On the brighter side, this practice
means that changing one program's behavior to match the other would
not be a major backwards compatibility problem.

David

---

From: Ian Grigg <[email-masked]>
Date: Wed, 11 Feb 2004 19:50:38 -0500

> end-of-line and whitespace issues.

I'd agree that it needs to be tightened up.

> small way to the poor interoperability reputation that OpenPGP has.

Right.  For many cases, I guess this may be
a nuisance.  For my case, it is critical
functionality, and from time to time, we've
had to re-roll contracts to cope with this,
and/or put in complicated heuristics into
code to verify using multiple methods.

(The preferred path at the moment is to change
the source file such that whitespace is removed,
but that's not really possible for most apps.)

> 	end of any line is ignored when the cleartext signature is

Two notes:

a) it would be useful if the whitespace were
clarified to eliminate potential Unicode white
space.

<http://www.imc.org/ietf-openpgp/mail-archive/msg07714.html>

b) it would make some sort of sense, if any
real changes were made, to unify the definition
of whitespace to be the same in both.  But this
is minor, as one would think that we were not
anticipating any functional change to 2440bis
at this stage.

> behavior already declared noncompliant.

The two implementations that I deal with both implement
2440bis as the (best available) standard.  If 2440 has
to be changed, then a lot of implementations would
have to be as well;  it's long since past that it was
routine to treat the behaviour of pgp as a de facto
standard.  Also, anecdotally, I think gpg has a
pretty significant market share these days, so we
would have to quote what behaviour it follows if
there was to be any change in 2440.

> that the issue wasn't looked at.

I agree.

 > <http://www.imc.org/ietf-openpgp/mail-archive/msg03753.html>,
 > ....

I agree with the thrust of this.  Text should be
sent in the interchange format, and signatures
should be immediately verifiable on the data as
delivered.  It does mean that an implementation
should send data with whitespace trimmed, though.
And, that verification MAY try additional trimming
if it hasn't been already trimmed by a non-conforming
signer.

It would be nice to get such explanation into 2440bis.

iang

---

From: David Shaw <[email-masked]>
Date: Tue, 17 Feb 2004 11:56:32 -0500

On Wed, Feb 11, 2004 at 07:50:38PM -0500, Ian Grigg wrote:

> >I've seen comments that these details were inadvertent errors in
> >2440 that would have or should have been fixed, and requests to the
> >WG the change these two details in 2440bis to match the historical
> >PGP behavior (after all, PGP 5 predates 2440 and there is a huge
> >installed base of PGP 5-8).  I've also seen comments that the WG
> >mustn't change the published standard this many years after the
> >fact to match behavior already declared noncompliant.
> 
> 
> The two implementations that I deal with both implement 2440bis as
> the (best available) standard.  If 2440 has to be changed, then a
> lot of implementations would have to be as well; it's long since
> past that it was routine to treat the behaviour of pgp as a de facto
> standard.  Also, anecdotally, I think gpg has a pretty significant
> market share these days, so we would have to quote what behaviour it
> follows if there was to be any change in 2440.

That's all fine with me.  Like I said, I don't really have strong
feelings for what the outcome of this is.  I just want there to be
*one* outcome.

David

---

From: Jon Callas <[email-masked]>
Date: Thu, 19 Feb 2004 11:27:30 -0800

> <http://www.imc.org/ietf-openpgp/mail-archive/msg07714.html>

If someone will let me know what they are, I'll put them in.

	Jon

---

From: Ian Grigg <[email-masked]>
Date: Fri, 20 Feb 2004 10:16:07 -0500

> If someone will let me know what they are, I'll put them in.

The above thread seems to have been renumbered to be
this one:

http://www.imc.org/ietf-openpgp/mail-archive/msg04791.html
http://www.imc.org/ietf-openpgp/mail-archive/msg04793.html

Here's a summary:

UTF-8 character sets can include whitespace in
seemingly arbitrary fashions.  As the UTF-8
space is very large, it seems unreasonable for
the OpenPGP WG to attempt to track what is
whitespace from a complete, UTF-8, point of
view.

However, whitespace is declared as being stripped,
and the implication in the rfc2440 document is
that this is declared to be common us-ascii
formats, explicitly.

So a way through this may be to say, in the
context of cleartext signatures:

     a) us-ascii whitespace is always stripped from
        the end of lines in signature calculation,
        both in signing and verification.

     b) UTF-8 whitespace may be stripped from the
        end of lines in signing, but in this case, the
        document should be transmitted in its clean
        interchange form, with these characters
        cleaned from the end of lines, so that the
        recipient can calculate (verify) correctly.

        Ref: Jon Callas,
        http://www.imc.org/ietf-openpgp/mail-archive/msg03753.html
        which would merit being included as a policy
        within the rfc2440 (that is, cleartext sig
        preparation should change documents into the
        interchange format, and thus strip whitespace
        proactively).

     c) us-ascii whitespace is defined to be, for
        the purposes of RFC2440 cleartext signature
        calculations, a) above, to be:

          1.  space (0x20), tab (0x09), nl (0x0a),
              cr (0x0d) ...
          2.  everything <= 0x20, or
          3.  or...

        (pick one) - the essence being here that the
        document does not define which it is.

Assuming this is acceptable to the WG, the above
messages include some example suggested text
(now 04791, 04793;  which google previously recorded
as 07728, 07718).

Alternatively, if the WG has decided to define
UTF-8 whitespace and strip it from cleartext, is
that something that happened in Seoul?  Are there
notes on this?

iang

---

From: Adrian von Bidder <[email-masked]>
Date: Fri, 20 Feb 2004 17:00:07 +0100

>         recipient can calculate (verify) correctly.

s/transmitted/signed/ perhaps?

Perhaps just state that the behaviour of OpenPGP implementations wrt non-ascii 
whitespace at EOL is undefined, thus signing such texts should be avoided (by 
removing/encoding such ws)?

cheers
-- vbi

-- 
Dogs just don't seem to be able to tell the difference between important 
people and the rest of us.

---

From: Jon Callas <[email-masked]>
Date: Fri, 20 Feb 2004 14:59:00 -0800

> that something that happened in Seoul?  Are there

Seoul hasn't happened yet. It happens in a couple of weeks. I'm not 
going to be there, so we'll all need an update of what gets discussed 
there.

Before then, however, I have a proposal:

How about if we remove any whitespace things, and just canonicalize 
line ends? It sounds like Unicode whitespace may be a huge can of 
worms. Alternatively, we could just say trim anything that's <= 0x20, 
which is a simple enough thing that solves some obvious attacks with 
backspacing and bare CRs to overstrike.

	Jon

---

From: Jon Callas <[email-masked]>
Date: Fri, 20 Feb 2004 14:54:49 -0800

>    remove trailing blanks (essentially PGP 2.x behavior).

I'm happy to consider this a bug, myself, if consensus says that's what 
the behavior should be. This is probably a case where there's code in 
there that was written by either Derek or Peter Gutmann and no one's 
changed it since then. :-)

> behavior already declared noncompliant.

No, this is not an inadvertent error. It was an intentional change.

PGP 5 did predate 2440 and there are a number of small changes between 
its behavior and what's in 2440. Moreover, this version is now seven 
years old and has not been supported in so long that I can't even 
remember how long it has been unsupported. It is not 2440-compliant, 
can't be because it predated 2440, is not supported by its producer, 
and really isn't germane to any discussion of 2440. Please stop 
bringing it up. It's dead. Thank you, I feel much better now.

The 2440 change in text signatures (adding in whitespace trimming) was 
one of a number of small things there that were debated as to what the 
right thing should be, rather than what went before. There are many 
good reasons for removing trailing whitespace at the end of anything 
that's text mode. It's the sort of thing that gets mangled easily and 
undetectably, as well as a covert channel. (I come from an era in which 
is was common practice for text editors to trim trailing whitespace 
when saving a file, and consider it a feature rather than a bug.)

However, I'd be perfectly happy to settle it once and for all by saying 
only normalize line ends, even. We can just not worry about the 
whitespace. In short, if the consensus here is that however 
well-meaning that change was, it was a bad idea, it's easy to fix. I 
can see that Unicode issues might turn this into a swamp in ways that 
just trimming spaces and tabs isn't.

> Jon Callas pointed out in...

I'll repeat a broader opinion I have on this. I think that text mode 
should be an assurance that the signed data is in a particular format, 
so that it can be handled accurately, not a commentary on how to 
canonicalize.

The reasons I think this is a good idea are that it be an assurance 
include:

* If it's just an assurance, then you verify a signature the same way 
for text mode and binary mode. This is better for the code in a lot of 
ways.

* If it's just an assurance, and the assurance is wrong (because of 
bugs or incompatibilities), then you end up with a good signature over 
data that isn't exactly in the format you wanted. This is a much better 
failure mode than getting good data, but not knowing how to verify the 
data. It's a fail-soft situation.

	Jon

---

From: Ian Grigg <[email-masked]>
Date: Fri, 20 Feb 2004 21:04:49 -0500

Jon Callas wrote:

> ... It sounds like Unicode whitespace may be a huge can of worms.

 From the little research I did, I couldn't find
any help in "defining Unicode whitespace," so,
yes, I agree it will be a huge can of worms.
(Unicode in any context is a mess, but it seems
clear that we need something, so UTF-8 it is.)

> Alternatively, we could just say trim anything that's <= 0x20, which is 
> a simple enough thing that solves some obvious attacks with backspacing 
> and bare CRs to overstrike.

Good point, that would get my vote.  I'd prefer the
"chars <= 0x20" test as it is very clear, easy to
code, and it resolves what one does when some
spurious unprintable char we haven't thought about
comes into play.  I hadn't thought about backspaces...

(other mail:)
 > right thing should be, rather than what went before. There are many good
 > reasons for removing trailing whitespace at the end of anything that's
 > text mode. It's the sort of thing that gets mangled easily and
 > is was common practice for text editors to trim trailing whitespace when
 > saving a file, and consider it a feature rather than a bug.)

It may be that the era is not dead.  There are
a number of scenarios that either add or change
whitespace on the ends of lines;  cut&paste
does it in circumstances which are too boring
to research.

 > whitespace. In short, if the consensus here is that however well-meaning
 > that change was, it was a bad idea, it's easy to fix. I can see that
 > Unicode issues might turn this into a swamp in ways that just trimming
 > spaces and tabs isn't.

My vote would be to trim whitespace and normalise
line endines to CR/NL, where whitespace is <=0x20:

     Also, any trailing whitespace (characters <= 0x20) at the
     end of any line is ignored when the cleartext signature is
     calculated.

I think there should be a comment in there that
indicates what to do with Unicode, just to show
we thought about it, and not waste people's time
asking the question when they are implementing.
Something like:

     Unicode whitespace, where defined, SHOULD NOT be ignored.

Or,

     No Unicode whitespace characters are defined.

Leaving open the possibility of defining them in
an update?

iang