Subject: [openpgp] "OpenPGP Simple"

From: Stephen Paul Weber <[email-masked]>
Date: Sun, 15 Mar 2015 12:57:44 -0500

One of the big obstacles to OpenPGP deployments that I've faced over time is 
the perception that it's "too complicated", mostly based on the sheer size 
of the current RFC.  There are two things going on here:

1) Sections of the RFC define what you might call "extras", such as the 
ASCII Armor (including a checksum unused elsewhere in the spec)
2) There are a lot of backwards-compatibility things (old-style lengths, 
lots of different algorithms)

One of the things I've tried to work on to help in some of my use cases is 
a modular description for a subset of OpenPGP that is (hopefully) easier to 
immediately grok and/or implement.  It is at 
<https://github.com/singpolyma/openpgp-spec>

Is there any prior art on IETF specs having a "full" and "simple" form where 
full implementations can read any output of simple ones, but not always 
vice-versa?  Given the (necessary) size of OpenPGP as a whole, it seems like 
this might be worth considering.

-- 
Stephen Paul Weber, @singpolyma
See <http://singpolyma.net> for how I prefer to be contacted
edition right joseph

---

From: Werner Koch <[email-masked]>
Date: Sun, 15 Mar 2015 20:38:41 +0100

On Sun, 15 Mar 2015 18:57, [email-masked] said:
> One of the big obstacles to OpenPGP deployments that I've faced over
> time is the perception that it's "too complicated", mostly based on
> the sheer size of the current RFC.  There are two things going on

FWIW, having implemented both OpenPGP and CMS/X.509 (aka S/MIME) I can
only tell how easy it was to implement and maintain OpenPGP in contrast
to the S/MIME.  Up until ECC support, only one RFC and not several every
few years changing huge RFCs with so much room for interpretation that
you can't implement them without looking at older standards and actual
implementations.

> 2) There are a lot of backwards-compatibility things (old-style
> lengths, lots of different algorithms)

Actually there are not many algorithms.  If you know two (with 64 bit
and 128 block length) you know all of them ;-).  CMS hides a lot of
details by refering to BER or DER encoding and that is really hard to
test.

> Is there any prior art on IETF specs having a "full" and "simple" form
> where full implementations can read any output of simple ones, but not
> always vice-versa?  Given the (necessary) size of OpenPGP as a whole,
> it seems like this might be worth considering.

You mean notes in the margin to easier see MAY parts?  I doubt that the
RFC format can provide this.  Having two separate official documents
raises the danger that they are not consistent.  Your annotated/edited
version of OpenPGP is likely the best thing to do.  It is similar to
reading a set of RFCs compared to reading Stevens - it is much easier to
grok his books than to start from scratch with the RFCs.

Shalom-Salam,

   Werner

-- 
Die Gedanken sind frei.  Ausnahmen regelt ein Bundesgesetz.

---

From: Wyllys Ingersoll <[email-masked]>
Date: Mon, 16 Mar 2015 01:55:56 +0000

> details by refering to BER or DER encoding and that is really hard to

Agree.  RFC 4880 is not terribly hard to implement in code if you focus on
the common use cases ("MUST" ciphers and modes) and ignore the optional
edge cases that very few people use in real practice.  CMS and ASN.1 are
gross and painfully hard to implement by comparison.

---

From: Jon Callas <[email-masked]>
Date: Mon, 16 Mar 2015 10:58:07 -0700

> 1) Sections of the RFC define what you might call "extras", such as the ASCII Armor (including a checksum unused elsewhere in the spec)

Certainly the ASCII Armor checksum is something that could go, since we don't need to worry so much about modem line noise. :-) But you have to know enough to ignore it.

> 2) There are a lot of backwards-compatibility things (old-style lengths, lots of different algorithms)
> 
> One of the things I've tried to work on to help in some of my use cases is a modular description for a subset of OpenPGP that is (hopefully) easier to immediately grok and/or implement.  It is at <https://github.com/singpolyma/openpgp-spec>

One of the things that I did once was to streamline an implementation my receiving a number of things well, like old-style packets and lengths, but only generating one thing (like five-byte lengths) on the grounds that if you move to generating a simplified format, you can then start doing usage surveys on when to phase out the old stuff.

But yes, having a simplified profile would be a great thing to do, I think.

	Jon

---

From: Stephen Paul Weber <[email-masked]>
Date: Mon, 16 Mar 2015 13:08:55 -0500

>edge cases that very few people use in real practice.

I agree.  I've implemented large parts of RFC4880 several times, with only 
a few catches.  This is just feedback I've gotten from others I've tried to 
work with, especially specs where I've suggested they use OpenPGP instead of 
inventing their own crypto (I think the first community I tried this with 
was Salmon/MagicSig).

---

From: Stephen Paul Weber <[email-masked]>
Date: Mon, 16 Mar 2015 13:12:13 -0500

>One of the things that I did once was to streamline an implementation my 
>receiving a number of things well, like old-style packets and lengths, but 
>only generating one thing (like five-byte lengths) on the grounds that if 
>you move to generating a simplified format, you can then start doing usage 
>surveys on when to phase out the old stuff.

Yes.  Last time I checked, gnupg < 2 (which is still the default on most of 
my systems) only generates old-style headers, whereas my implementations 
tend to only generate new-style.  Of course, usually one wants to consume 
both, if creating even a fairly-complete implementation, because of gnupg.

---

From: Werner Koch <[email-masked]>
Date: Mon, 16 Mar 2015 20:45:47 +0100

On Mon, 16 Mar 2015 19:12, [email-masked] said:

> Yes.  Last time I checked, gnupg < 2 (which is still the default on
> most of my systems) only generates old-style headers, whereas my

That depends on how you invoke gpg and whether the new packer headers
are required.  That is required for PGP-2 compatibility.

Salam-Shalom,

   Werner

---

From: David Leon Gil <[email-masked]>
Date: Mon, 16 Mar 2015 14:15:28 -0700

Re Jon's comment above: The five-octet, new-format thing is what the
End-to-End extension does.

On Monday, March 16, 2015, Werner Koch <[email-masked]> wrote:

> On Mon, 16 Mar 2015 19:12, [email-masked] <javascript:;> said:
>

Is there an option to completely disable this? Relatedly, is there any
option to not use new-format partial lengths?

Partial lengths are really a nuisance to parse.

---

From: David Shaw <[email-masked]>
Date: Mon, 16 Mar 2015 17:30:55 -0400

On Mar 16, 2015, at 5:15 PM, David Leon Gil <[email-masked]> wrote:
> 
> Re Jon's comment above: The five-octet, new-format thing is what the End-to-End extension does.
> 
> Is there an option to completely disable this?

Not in the current code, but you can of course patch it.

> Relatedly, is there any option to not use new-format partial lengths?

A partial length is needed to handle content as a stream - say some program that generates gigabytes of data (like a backup).  Something large enough that you really don't want to have to buffer the whole thing before encrypting it.

> Partial lengths are really a nuisance to parse.

No argument there...

David

---

From: David Leon Gil <[email-masked]>
Date: Mon, 16 Mar 2015 15:03:45 -0700

On Monday, March 16, 2015, David Shaw <[email-masked]> wrote:

> On Mar 16, 2015, at 5:15 PM, David Leon Gil <[email-masked]
> <javascript:;>> wrote:
> >
> > On Monday, March 16, 2015, Werner Koch <[email-masked] <javascript:;>>
> wrote:
> A partial length is needed to handle content as a stream - say some
> program that generates gigabytes of data (like a backup).  Something large
> enough that you really don't want to have to buffer the whole thing before
> encrypting it.
>

I agree on that. But I think this really requires a different mode of
operation / standard. See, e.g.,
https://www.imperialviolet.org/2014/06/27/streamingencryption.html I'm not
sure that an OpenPGP WG is the right place to do this.

(Which is why I said that the scope should be limited to email-length
messages.)

--

But, e.g., GnuPG (always?) emits partial lengths, even for data it already
knows the length of (and is << even 2^20 bytes).

---

From: Christoph Anton Mitterer <[email-masked]>
Date: Tue, 17 Mar 2015 05:08:40 +0100

On Sun, 2015-03-15 at 12:57 -0500, Stephen Paul Weber wrote: 
> of the current RFC.
Uhm? Compared to other standards it seems to be rather simple.

Apart from that, not everyone needs to (neither should one) re-invent
the wheel.
I don't say that there should be no more than one implementation of
OpenPGP, but having too many is just bad for security.
That being said - we already have a number of implementations which
cover the standard quite well - what's wrong with [contributing to|
using] them?

> ASCII Armor (including a checksum unused elsewhere in the spec)
As some others already said, using e.g. MIME would be better - but ASCII
armors aren't just used when sending mail (people use it e.g. to post
their raw key on a website or things like that).
So if ASCII Armors are dropped, the standard should refer to something
else that is used instead - just to prevent that people are using all
different froms of base64 encoding, uuencode, etc.

> lots of different algorithms)
Agree with that.

> vice-versa?
I'd say that this is rather a bad idea.
For many standards where this was done it caused more troubles than
good.
It's also security-wise a problem:
Who decides what's not critical enough for the base standard? Is it
guaranteed that implementations that don't implement the "full" standard
are still secure?

And often it just means that simply no one ever implements the
extensions to the base standard (think of the dozens of JPEG2000
profiles). If something is so special that it wouldn't be needed in the
base standard, one should perhaps question whether it's needed at all.

Cheers,
Chris.

---

From: Werner Koch <[email-masked]>
Date: Tue, 17 Mar 2015 12:18:20 +0100

> A partial length is needed to handle content as a stream - say some program that generates gigabytes of data (like a backup).  Something large enough that you really don't want to have to buffer th…

And to support > 4GiB files on systems without LFS support.

Salam-Shalom,

   Werner

---

From: Derek Atkins <[email-masked]>
Date: Tue, 17 Mar 2015 11:11:04 -0400

David Shaw <[email-masked]> writes:

> program that generates gigabytes of data (like a backup).  Something
> large enough that you really don't want to have to buffer the whole
> thing before encrypting it.

This is exactly what it was created for, when you cannot know the size
ahead of time.  Sometimes you *can* know the size, even for large
objects (e.g. you could stat() the file to get the filesize), but
obviously that doesn't work in all cases.

>> Partial lengths are really a nuisance to parse.

I'm sorry.  I blame Colin for trying to be too clever, and me not
working hard enough to talk him out of it.  ;)

> David

-derek

-- 
       Derek Atkins, SB '93 MIT EE, SM '95 MIT Media Laboratory
       Member, MIT Student Information Processing Board  (SIPB)
       URL: http://web.mit.edu/warlord/    PP-ASEL-IA     N1NWH
       [email-masked]                        PGP key available

---

From: David Shaw <[email-masked]>
Date: Tue, 17 Mar 2015 11:34:43 -0400

> And to support > 4GiB files on systems without LFS support.

Right, good point.  I think it's safe to say there are enough uses for partial length / streaming that it should be kept.  Not all the world is email.

What if the encoding was really simple - something like 4 bytes always, and the leftmost bit would mean "partial".  So any packet 2^31 or less could be encoded in one piece, but we could still do partial for those situations that needed it.  We could have any number of partial lengths, but it would always end with a non-partial final length.

David

---

From: Werner Koch <[email-masked]>
Date: Tue, 17 Mar 2015 21:47:38 +0100

> option to not use new-format partial lengths?

No.  In fact I started with GnuPG before OpenPGP and one of the first
additions to the RFC-1991 PGP-2 format was my own version of partial
lengths.  This has been removed in favor of the OpenPGP and PGP-5
partial length format.  

You really really want partial length.  A major problem with PGP-2 was
that it required large temporary files for certain operations - you
can't do backups this way.

Up until DKIM all Internet protocols have been carefully designed to
allow streaming of data (ie. the Unix way) and the IETF should continue
to stress the importance of this.

> Partial lengths are really a nuisance to parse.

Yeah the OpenPGP encoding is a bit over-engineered to save octets.  But
even its partial format is easier to implement than the HTTP
counterpart.

Shalom-Salam,

   Werner