Subject: Dash-escaping clarification

From: David Shaw <[email-masked]>
Date: Fri, 7 Mar 2003 08:46:34 -0500

Section 7.1 (Dash-Escaped Text) of bis-07 says, in part, that
dash-escaped text is "the ordinary cleartext where every line starting
with a dash '-' (0x2D) is prefixed by the sequence dash '-' (0x2D) and
space ' ' (0x20)."

Since the most common use of dash-escaped text is in email, both PGP
and GnuPG (by default) also dash-escape lines starting with the word
"From " (with the space).  This is for the usual mbox-inspired
reasons.  If the "From " line isn't escaped, then some downstream mail
system may escape it, thus breaking the signature.

Nothing in the draft seems to discourage dash-escaping more than just
the lines beginning with a dash.  Still, I am concerned with the
receiving side not knowing that these other lines may be escaped as
well (they may match on a dash-space-dash at the beginning of the
line, rather than dash-space).  A sentence saying something like "Any
other line MAY be dash-escaped as well at the discretion of the
sender" would be very helpful here.

David

---

From: Jon Callas <[email-masked]>
Date: Fri, 07 Mar 2003 10:33:12 -0800

On 3/7/03 5:46 AM, "David Shaw" <[email-masked]> wrote:

Added in for bis08.

    Jon

---

From: "Michael Young" <[email-masked]>
Date: Fri, 7 Mar 2003 14:27:57 -0500

- > ... A sentence saying something like "Any
- > other line MAY be dash-escaped as well at the discretion of the
- > sender" would be very helpful here.
- 
- Sounds good, but as David points out, this may break existing receivers.
- See if yours can verify this.  (PGP6.5.3 silently accepts it.
- GnuPG1.2.1 emits warnings on each line; it cannot verify this
- signature, but if I remove the blank input line above, it can.)

---

From: Adam Back <[email-masked]>
Date: Fri, 7 Mar 2003 22:16:45 +0000

Hmm I don't think it would be a good idea to allow dash-escaping of
literally anything because then you can't reverse the transformation
and '- ' is commonly used for bullet points where as the other cases
----- separated nested PGP signatures and other content types are not
common, and 'From ' is a prexisting common exception.

Isn't it rather just:

- dash escape nested -----BEGIN PGP MESSAGE----- (et al)
- dash escape From

Adam

---

From: Adam Back <[email-masked]>
Date: Fri, 7 Mar 2003 22:25:36 +0000

Let me modify that.  I just tried a few combinations with PGP and it
dash escapes leading '-', and 'From ' but pretty much nothing else.

So to describe existing behavior we could say:

- dash escape leading '-'
- dash escape 'From '

and that's it.

Adam

---

From: David Shaw <[email-masked]>
Date: Fri, 7 Mar 2003 19:38:20 -0500

> common, and 'From ' is a prexisting common exception.

I don't see how there would be a problem in reversing the
transformation.

- bullet point
becomes
- - bullet point
and reverses back to
- bullet point

random text
becomes
- random text
and reverses back to
random text

David

---

From: David Shaw <[email-masked]>
Date: Fri, 7 Mar 2003 19:39:58 -0500

> - dash escape leading '-'

I disagree.  Tomorrow will bring some other thing that needs to be
protected against modification.  A single rule that "anything may be
dash-escaped" is simpler and more general than two specific rules.

David

---

From: David Shaw <[email-masked]>
Date: Fri, 7 Mar 2003 20:40:58 -0500

> - signature, but if I remove the blank input line above, it can.)

The point is that future receivers will know that such a thing is
possible.  They still don't have to support it - it's a MAY.

It's hard to support something before it has been documented ;)

David

---

From: Adam Back <[email-masked]>
Date: Sat, 8 Mar 2003 01:53:03 +0000

See Michael Young's example: doing so breaks existing otherwise
compliant implementations.

'From ' is pretty much the only thing interesting to protect in an
email message, as it is a separator between email messages.

The '-' escape was just a way to protect nested signatures etc to
avoid confusing the parser of the outer signatures.

The rule's been that way since at least 1992, and I haven't seen any
new chars needing quoting.  So it's survived the test of time.

Adam

---

From: "Jeroen C. van Gelderen" <[email-masked]>
Date: Fri, 7 Mar 2003 21:00:28 -0500

> - dash escape nested -----BEGIN PGP MESSAGE----- (et al)

I think the idea has always been to escape all dashes so that the 
receiver can indiscriminately remove the <dash><space> prefix from 
every line in a received message. It is precisely this approach which 
makes it possible to allow arbitrary sentences to be dash-escaped 
*without* the transformation being irreversible.

Cheers,
-J

---

From: "Jeroen C. van Gelderen" <[email-masked]>
Date: Fri, 7 Mar 2003 21:14:52 -0500

> compliant implementations.

They probably can be fixed. And since OpenPGP isn't a finished standard 
yet this might be less of a problem in practice than it appears on 
paper.

> email message, as it is a separator between email messages.

It is also a protocol-specific hack. I strongly support simpler 
solutions if they achieve the same purpose. Allowing arbitrary lines to 
be escaped is simpler. OpenPGP too complex as-is.

> new chars needing quoting.  So it's survived the test of time.

That doesn't mean that a simpler rule isn't a better idea. In fact, 
ever since 1992, PGP has allowed for arbitrary lines to be escaped. So 
one could argue that the current OpenPGP standard is too restrictive.

At the very least we can argue that the OpenPGP way of defining 
escaping isn't sufficiently obvious that PGP get it wrong.

Cheers,
-J

---

From: "Jeroen C. van Gelderen" <[email-masked]>
Date: Fri, 7 Mar 2003 21:41:02 -0500

> possible.  They still don't have to support it - it's a MAY.

Erm, not the way I read it. A compliant implementation MAY generate 
arbitrary dash escapes at the sender's discretion. A compliant receiver 
MUST thus be able to handle this as it is a valid OpenPGP message. You 
can't expect the sender to perform a capability check with the receiver 
before sending the message.

In any case, I'd like to make sure that if we allow the sending of 
arbitrary dash-escapes we also REQUIRE clients to be able to handle 
this. Otherwise we are introducing yet another complication in an 
already overly complex protocol. Even the x86 instruction set looks 
clean compared to the current OpenPGP spec. (And, yes, that is a vote 
against v3 support.)

Come to think of it, in good PGP tradition, we could REQUIRE acceptance 
now and add MAY arbitrarily escape in a year or so ;)

> It's hard to support something before it has been documented ;)

That definitely is true. But OpenPGP kinda documents the pre-existing 
PGP. And it seems that the GnuPG people did test their implementation 
against PGP which prompted allowing arbitrary escapes, albeit with a 
warning.

Cheers,
-J

---

From: David Shaw <[email-masked]>
Date: Fri, 7 Mar 2003 22:00:23 -0500

> before sending the message.

Sorry, my error.  You are completely right.

> against PGP which prompted allowing arbitrary escapes, albeit with a 

Exactly.  There happens to be a minor detail of the GnuPG code so that
blank lines that are escaped are not accepted (i.e. a line with only
"- "), but that is easily remedied.

David

---

From: "Michael Young" <[email-masked]>
Date: Fri, 7 Mar 2003 22:46:39 -0500

Subject: Re: Dash-escaping clarification

I didn't really generate the example to argue against David's suggestion.
In fact, I rather like it.  I provided it mostly for other implementors to
try out (and to see for myself what PGP and GPG would do).

The two most prevalent clients do accept it (almost... as David notes,
the GPG behavior on blank lines really feels like a bug, not a feature).

As for other things that might get chewed up by mail systems... I wonder
whether anything would rewrite MIME headers, were they embedded
in clearsigned text?  (I won't be generating a test case for this :-).