Subject: Adding in BZ2 compression?

From: Jon Callas <[email-masked]>
Date: Thu, 03 Jul 2003 16:47:52 -0700

I have a request for an algorithm number for bz2 compression. The
implementer in question has promised on a stack of holy books only to use it
along with compression prefs. Anyone object strongly?

    Jon

---

From: Derek Atkins <[email-masked]>
Date: 03 Jul 2003 21:56:49 -0400

Just to play devil's advocate, why do we need yet another compression
algorithm?

-derek

-- 
       Derek Atkins, SB '93 MIT EE, SM '95 MIT Media Laboratory
       Member, MIT Student Information Processing Board  (SIPB)
       URL: http://web.mit.edu/warlord/    PP-ASEL-IA     N1NWH
       [email-masked]                        PGP key available

---

From: Werner Koch <[email-masked]>
Date: Fri, 04 Jul 2003 10:58:09 +0200

> along with compression prefs. Anyone object strongly?

I don't see a real advantage.  bz2 does only make sense for large
files and thus it won't hurt to first but the bz2 compressed data into
an appropriate MIME container and then apply OpenPGP.

If it is a problem that the signature can only be applied on the
compressed data, one should use the PGP/MIME approach and don't
combine signature and encryption in one OpenPGP message.

Decompressing bz2 requires huge amounts of memory and thus it can't be
implemented for small devices.  If we assume that people are going to
use their keys also on PDAs, we will run into problems with the
preferences.

Shalom-Salam,

   Werner

-- 
Werner Koch                                      <[email-masked]>
The GnuPG Experts                                http://g10code.com
Free Software Foundation Europe	                 http://fsfeurope.org

---

From: "Michael Young" <[email-masked]>
Date: Fri, 4 Jul 2003 12:30:26 -0400

From: "Derek Atkins" <[email-masked]>
> Just to play devil's advocate, why do we need yet another compression

OK... I'll argue the other side for you ;-).

Why did we need any more ciphers after adding 3DES?  Performance, for one.
BZ2 produces smaller output for many common forms of input.
If it didn't have any obvious advantage, I'd agree with you; but, it does.
It's certainly more useful than two stylistic variants of the same thing,
which is what we have now.

Turning the question around: what's the harm?

Adding another optional compression algorithm won't create any problems
we don't already have.  (ZLIB is supported by GnuPG but not PGP.)
The preference system is the architected solution to this problem.
As David notes, moving from one implementation to another requires
being able to rewrite preferences, but that's always been the case.

Now, given that compression can be done by the end-user, it's hard to
argue that *any* algorithm is a have-to-have.  (This seems to be the
essence of Werner's argument, although he makes it in terms of
PGP/MIME, which I don't find compelling for the pure OpenPGP spec.)
But since the architecture includes compression, I'd say that
including a popular and more powerful algorithm is worthwhile.

---

From: Werner Koch <[email-masked]>
Date: Fri, 04 Jul 2003 19:07:54 +0200

> which is what we have now.

Yep, I'd prefer to drop compression algo 1 because 2 is better defined
and provides a clean way to specify the used window size.  For
backward compatibility it can not be done though.

> PGP/MIME, which I don't find compelling for the pure OpenPGP spec.)

OpenPGP is mostly used for email, so PGP/MIME makes a lot of sense.
The encapsulation provided by MIME is much more flexible than that of
OpenPGP.

---

From: Jon Callas <[email-masked]>
Date: Mon, 07 Jul 2003 18:03:58 -0700

> Just to play devil's advocate, why do we need yet another compression

I'm not the person asking for them, so I may be inadequately explaining
this. Nonetheless, here goes.

The argument of "no new algorithms unless one is broken" makes more sense
for things like ciphers than it does for compression. Compression in OpenPGP
doesn't have the same sort of security implications that a cipher does, and
so one doesn't need to be as conservative about it.

The practical technical reason for bz2 is that it compresses better. In a
test I saw, it compresses 12% better than Deflate, and 7% better than zlib.
That test was with a backup of a server. Here's the actual results:

Original tar file: 1,739,950,080 bytes (1.62 GB)    100%
.tar.pgp file    : 730,450,065 bytes   (696.6 MB)   43%
.tar.gz file     : 694,085,841 bytes   (661.9 MB)   40%
.tar.bz2 file    : 648,270,622 bytes   (618.2 MB)   38%

The practical product reason is that there are a number of storage archival
systems that are adding in crypto. Many are encrypting some compressed data
with PKCS7 or some home-brew thing. I've been getting questions about using
OpenPGP as an archival primitive, especially since it includes compression.

I would like to be responsive to this, and say that OpenPGP is a great
system to use for encryption and compression, and why sure, just code it up
this way. It would similarly pain me to have to say that OpenPGP isn't
suitable for encryption and compression of large amounts of data, please go
shop at Gimble's.

Yes, I know that there are potential interoperability issues when keys get
migrated around, but I also of the opinion that when an implementation
imports a key, it should make sure that the preferences reflect what it
supports.

    Jon

---

From: Len Sassaman <[email-masked]>
Date: Thu, 17 Jul 2003 15:13:32 -0700 (PDT)

> so one doesn't need to be as conservative about it.

Putting multiple options for anything into a protocol does practical
implications in the context of an anonymity system, however. (I'm not
saying this should prevent adding a new compression algorithm if it serves
a purpose -- it's just something to keep in mind.)

> imports a key, it should make sure that the preferences reflect what it

Amen. Can that be explicitly stated in the next draft?

(And does PGP do this yet?)

--Len.

---

From: David Shaw <[email-masked]>
Date: Thu, 17 Jul 2003 20:58:46 -0400

> Amen. Can that be explicitly stated in the next draft?

Yes, please.

David

---

From: Jon Callas <[email-masked]>
Date: Sun, 20 Jul 2003 02:41:39 -0700

> a purpose -- it's just something to keep in mind.)

I was basically against it until it was explained to me why people wanted
it, and I ended up thinking, "Hmmm, we don't have a compression system in
there that's newer than 1977, and customers are often right."

> Amen. Can that be explicitly stated in the next draft?

I'm under the impression that it already says "SHOULD" in there. I don't
think it should be any stronger. It's a feature of OpenPGP that it's small.
I don't want to force someone who wants to embed OpenPGP in something like a
pager network (yeah, yeah, these days pagers play videos) to have to do
everything in PGP or GPG.

OpenPGP is not supposed to mandate all the features a good desktop program
should have. 

    Jon

---

From: David Shaw <[email-masked]>
Date: Sun, 20 Jul 2003 11:48:07 -0400

On Sun, Jul 20, 2003 at 02:41:39AM -0700, Jon Callas wrote:

> I was basically against it until it was explained to me why people
> wanted it, and I ended up thinking, "Hmmm, we don't have a
> compression system in there that's newer than 1977, and customers
> are often right."

I rather like the idea of OpenPGP as an archival primitive.

> I'm under the impression that it already says "SHOULD" in there. I
> don't think it should be any stronger. It's a feature of OpenPGP
> that it's small.  I don't want to force someone who wants to embed
> OpenPGP in something like a pager network (yeah, yeah, these days
> pagers play videos) to have to do everything in PGP or GPG.
> 
> OpenPGP is not supposed to mandate all the features a good desktop
> program should have.

The current draft says:

  Since a self-signature contains important information about the
  key's use, an implementation SHOULD allow the user to rewrite the
  self-signature, and important information in it, such as preferences
  and key expiration.

How about adding:

  Note that without the ability to rewrite a self-signature,
  interoperability issues may occur when the same key is used in more
  than one implementation.  Implementations may wish to check keys
  upon import to ensure that the preferences on the key match the
  reality of the implementation.

That doesn't mandate anything, but does call attention to the problem.
I guess the last line could be a SHOULD if there was a desire to make
it stronger.

David

---

From: Jon Callas <[email-masked]>
Date: Sun, 20 Jul 2003 14:35:14 -0700

> I rather like the idea of OpenPGP as an archival primitive.

That's what sold me, too.

> I guess the last line could be a SHOULD if there was a desire to make

I put in:

It is good practice to verify that a self-signature imported into an
implementation doesn't advertise features that the implementation doesn't
support, rewriting the signature as appropriate.

    Jon

---

From: David Shaw <[email-masked]>
Date: Mon, 21 Jul 2003 12:38:06 -0400

> support, rewriting the signature as appropriate.

Excellent.  That works for me.

David

---

From: Derek Atkins <[email-masked]>
Date: 22 Jul 2003 13:54:01 -0400

> That's what sold me, too.

Ok.  I have no real objections to adding the algo -- I'm just worried
about the interop issues.

Also, we still need to show multiple implementations to progress to
DRAFT standard, so if we're adding new features it might take longer
to do so.  Have we had any OpenPGP Bakeoffs?

-derek
-- 
       Derek Atkins                 617-623-3745
       [email-masked]             www.ihtfp.com
       Computer and Internet Security Consultant

---

From: Jon Callas <[email-masked]>
Date: Wed, 23 Jul 2003 14:01:56 -0700

> about the interop issues.

There are no interop issues because it's a MAY feature that presently no one
implements. :-) Implementations are already supposed to handle both ZIP and
zlib prefs, so it shouldn't be an issue.

(Incidentally, here at PGP, we presently implement only ZIP, but decided to
add in both zlib and bz2 for a future major relase.)

> to do so.  Have we had any OpenPGP Bakeoffs?

I think so, but I wasn't on top of this for a while. I'm willing to help
push this now.

    Jon