Subject: [openpgp] Deprecating compression support

From: Justus Winter <[email-masked]>
Date: Mon, 18 Mar 2019 13:07:21 +0100

Hello,

I propose to deprecate compression support in OpenPGP.  The reasons
for this are:

  - Compression makes it impossible to reason about the size of a
    decrypted message, requiring the use of a streaming interface even
    for seemingly small messages, e.g. emails.  Experience has shown
    that downstream users struggle with the correct use of streaming
    interfaces.

  - Compression allows the construction of quines.

  - Compression interacts badly with encryption, see e.g. CRIME,
    BREACH, and hiding of EFAIL-style CFB gadgets [0].

  - The downstream application is in a better position to decide whether
    and how to compress data that is then encrypted using OpenPGP.

  - Compression make the standard more complex, and enlarges the
    trusted computing base of implementations.

I realize that we cannot suddenly drop decompression support, but I
would suggest to stop emitting compressed data packets.  If this
proposal gathers traction, I would be happy to suggest a change to the
standard.

Cheers,
Justus

0: Section 5.3 of https://efail.de/efail-attack-paper.pdf

---

From: Vincent Breitmoser <[email-masked]>
Date: Mon, 18 Mar 2019 13:25:23 +0100

I support this motion.

 - V

Justus Winter <[email-masked]> wrote:
> _______________________________________________
> openpgp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/openpgp

---

From: "Neal H. Walfield" <[email-masked]>
Date: Mon, 18 Mar 2019 14:22:25 +0100

Hi Justus,

Thanks for bringing this up.

On Mon, 18 Mar 2019 13:07:21 +0100,
Justus Winter wrote:
> I propose to deprecate compression support in OpenPGP.
> ...

I propose that we change the standard to say:

  - Compliant implementations MUST NOT emit compressed data packets.

  - For backwards compatibility, compliant implementations MAY
    decompress compressed data packets.

Obviously, we should deprecate the "Preferred Compression Algorithms"
subpacket as well.

Note: removing compression support was one of the original goals:

  Subject: [openpgp] details of 4880bis work
  From: Stephen Farrell <[email-masked]>
  Message-ID: <[email-masked]>
  Date: Fri, 10 Apr 2015 12:38:09 +0100
  Archived-At: <http://mailarchive.ietf.org/arch/msg/openpgp/ll36RGS81vXSXkVey0cR0zZ7WkI>

  ...

  g) remove compression entirely

  ...

:) Neal

---

From: "Derek Atkins" <[email-masked]>
Date: Mon, 18 Mar 2019 09:31:55 -0400

Hi,

>   - Compliant implementations MUST NOT emit compressed data packets.

I would lessen this a tiny bit and say SHOULD NOT.  I can still imagine
cases where one MAY want to emit a compressed data packet, especially if
the sender and receiver have some out-of-band knowledge.

>     decompress compressed data packets.

I would also change this to SHOULD, because IMHO backwards compatibility
is important and there are decades of compressed data out there.

>   Archived-At:
> <http://mailarchive.ietf.org/arch/msg/openpgp/ll36RGS81vXSXkVey0cR0zZ7WkI>
>
>   ...
>

I am not against removing it, but it should be deprecated first IMHO.

> :) Neal

-derek

-- 
       Derek Atkins                 617-623-3745
       [email-masked]             www.ihtfp.com
       Computer and Internet Security Consultant

---

From: Vincent Breitmoser <[email-masked]>
Date: Mon, 18 Mar 2019 14:55:29 +0100

> is important and there are decades of compressed data out there.

With v4 keys, yes. But not with v5 keys. We could say MUST NOT compress when
encrypting to v5 keys, or perhaps backport this a bit if we say MUST NOT
compress with AEAD. Either will be a clear cut.

> the sender and receiver have some out-of-band knowledge.

Keep in mind anything we keep as a SHOULD will definitely have to be supported
by implementations forever on the receiving side. It would be nice if future
implementations were able to also drop that attack surface altogether, together
with v4 key or non-AE encryption support.

 - V

---

From: James Howard <[email-masked]>
Date: Mon, 18 Mar 2019 16:34:55 +0000

Hello, long-time lurker, first time caller.  As I read this, this morning, I fail to see the advantages.  For instance, let’s look at a couple of these issues:

Compression makes it impossible to reason about the size of a

decrypted message,

It’s hard not to look at this and say, good!  The idea of encryption is the hide information and burying information about the length of the message can only be an improvement.

I am not sure how valid this is.  For instance, in the paper cited, we see the following comments:

[F]or OpenPGP, we needed to develop more complex exploit techniques upon malleability gadgets because the data is typically compressed before encryption
OpenPGP’s plaintext compression significantly complicates our attack.
The difficulty here is to guess a certain amount of compressed plaintext bytes in order to fully utilize the CFB gadget technique. Not knowing enough compressed plaintext bytes is hardly a countermeasure, but makes practical exploitation a lot harder.

The problems described in the paper are not compression, but rather the sender drops known plaintext right into the start of the stream.  I mean, it’s not rocket science and in the, e.g., Facebook example, it could be addressed by adding a nonce string to the start of the message.  There are certain streams of thought which have advocated this for years, anyway.  

Then there’s this:

Now, I will admit to misinterpreting this (I think) and assumed you meant compressing after application of PGP.  That’s, of course, silly, but since I did the work of showing AES cyphertext to be basically incompressible, I will send it out, anyway.  See the attached Excel spreadsheet for results of pre/post-encryption compression on the Canterbury Corpus.  Nothing here surprising, but data is good!

—James

From: openpgp <[email-masked]> on behalf of Justus Winter <[email-masked]>
Date: Monday, March 18, 2019 at 11:25
To: <[email-masked]>
Subject: [openpgp] Deprecating compression support

Hello,

for this are:

    interfaces.

standard.

Cheers,

Justus

openpgp mailing list

[email-masked]

---

From: Justus Winter <[email-masked]>
Date: Mon, 18 Mar 2019 18:15:56 +0100

James Howard <[email-masked]> writes:

> Hello, long-time lurker, first time caller.  As I read this, this
> morning, I fail to see the advantages.  For instance, let’s look at a
> couple of these issues:
>
> It’s hard not to look at this and say, good!

What I meant here is that the receiver (i.e. not an adversary) cannot
know in advance how big a decrypted message is by looking at the
encrypted message.  This means that the only safe interface for an
OpenPGP library is a streaming one, and this is perceived as difficult
to use, and/or to increase the complexity of implementations.

> The idea of encryption is the hide information and burying information
> about the length of the message can only be an improvement.

However, compress-then-encrypt leaks the entropy of the plaintext, see
e.g. CRIME, BREACH.

> > [F]or OpenPGP, we needed to develop more complex exploit techniques
> > upon malleability gadgets because the data is typically compressed
> > before encryption OpenPGP’s plaintext compression significantly
> > complicates our attack.  The difficulty here is to guess a certain
> > amount of compressed plaintext bytes in order to fully utilize the
> > CFB gadget technique. Not knowing enough compressed plaintext bytes
> > is hardly a countermeasure, but makes practical exploitation a lot
> > harder.

You conveniently left out the very next paragraph, which I will cite
here:

> We show how the compression structure can be exploited to create
> exfiltration channels. Interestingly, with the compression in place,
> we can create exfiltration chan- nels even more precisely and remove
> the random data blocks from the resulting plaintext.

> Now, I will admit to misinterpreting this (I think) and assumed you
> meant compressing after application of PGP.  That’s, of course, silly,

I meant compress-then-encrypt, of course.

Justus

PS: Your MUA is terrible, it doesn't preserve quotations.

---

From: Jon Callas <[email-masked]>
Date: Mon, 18 Mar 2019 13:45:54 -0700

I like the basic proposal. I think that deprecation is better than banning, and consequently we ought to be doing it with SHOULDs and SHOULD NOTs rather than MUSTs, as that’s banning rather than deprecating.

However, I want to add that implementations today can deprecate it on their own and no changes in the standard need not be there. If an implementation creates a key with compression preferences set to no compression then any other implementation is bound by the standard not to compress. I advocate this as well. As OpenPGP sits today, any implementer can unilaterally not only deprecate, but eliminate compression. In fact, I’ll go so far as to advocate that all implementers, even GnuPG ought to just start making keys by default that eschew compression. More on this below.

My rationale for getting rid of compression is different. I think that the security-based reasons in this thread are largely unconvincing and often just not quite correct. I don’t want to discuss that in this thread, because I agree with the destination — let’s move away from compression, and am afraid that if we debate the reasons for it we distract from the result we mutually want. We should deprecate compression, but not because of security, but because of simplicity, and in short a case of “that was then, this is now."

OpenPGP has many things in it that are needlessly complex, and compression is one of them. (Other needless complexities include the way that packet sizes are computed, and others.) Much of this comes from its heritage as a utility, and a utility that was designed for DOS and BBSes. The considerations of networking and communications in 1991 are not at all what we should have for 2021. 

One of these differences is that zip-style compression is just not as useful now. Obviously, the wins for compression are on things that are the largest. Most of the large things we transfer today are already compressed, and thus compressing them again is mostly a waste of cpu cycles. There are some file formats that have zip-style compression built in. For example, GIF and PNG graphics are explicitly compressed. Other formats like JPEG are a different sort of compression but have the same characteristic that it is reasonably mathematically pseudo-random and not readily compressed further. Ditto for audio formats like MP3 and AAC, while FLAC and ALAC are like GIF and PNG in that they’re losslessly compressed. 

Today, the large things we send are media, and therefore there’s no need to compress them. The things we send that are easily compressible tend to be smaller.

In the cases where large files are sent that can be compressed, the best solution is for that system to compress the file itself. Remember, OpenPGP’s history comes from DOS programs on small computers and they didn’t even have a way to pipe a tar or zip command to an encryption program. That’s not the case today.

Historically, compression has been the largest share of the time it takes to process a file with an OpenPGP implementation, and thus we are doing something by default that burns CPU for little gain. We should stop doing it by default. Again, the best way to do this is to create keys that have a compression preference of none. We can do this now.

Over a decade ago, the PGP program had a feature of the “PGP Zip” file. This was literally nothing more than a tar file that was run through gzip, and then encrypted with OpenPGP without compression. PGP itself had some nice viewers that let someone manipulate the container just as other directory compression systems did, but any unix system could handle the file by just decrypting with gnupg and piping that to tar. Later on, as I remember (correct me if I’m wrong, Derek, or someone), we shifted to bz2. The same strategy can be improved as compression tools improve.

The compression inside of OpenPGP is also hard to implement correctly. The default compression, the “DEFLATE” option is a modified implementation of ZIP-style programs from the era of the late 1980s. As those programs coalesced into default (not precisely standard, merely ubiquitous) implementations, they went in one direction and that era’s PGP stayed with the variant. When we did RFC 2440, we added in ZLIB (RFC 1950) encryption so that an implementation wouldn’t have to hack the compression software. This is another reason to move away from compression inside OpenPGP. RFC 4880 added in an option for BZ2 (it was the new hotness at the time), and these days the cool kids are using 7-Zip as it's even better than bzip. The best way to keep up with advances in compression is piping from a compressor to some OpenPGP implementation, rather than continuing to chase advancements.

The underlying reality is even worse. Go look at section 5.6 of 2440, and there are interoperability hints for working with PGP2 because it had further limitations in it for internal table sizes. This is another reason to get rid of it for simplicity. It flat isn’t needed.

Simply not having compression aids implementation, as well. A number of years ago, I was putting together a Javascript implementation and there were no good compression libraries at the time, so we just created keys that said “no compression, please” and ran with it. Every implementation can handle decoding an OpenPGP object that is not compressed, so there are fewer bits of backwards compatibility to transition away from it.

Lastly, I’ll note that everything I’ve said here could be applied to banning encryption in 4880-bis, as well. I am not opposed to a ban, but I think deprecation is better, particularly since every implementation can just stop doing compression by default and everything will work just fine. Repeating myself, I even advocate this. Everyone who makes some OpenPGP program can stop today, and likely should. I’ll even promote that to SHOULD. 

(The biggest reason against an outright ban is that there are a lot of systems that use OpenPGP in the midst of some internal process, like moving around large, sensitive files. There are thus likely a whole lot of shell scripts somewhere that someone’s got to find and change, and I wouldn’t be surprised if something breaks if there’s a sudden shift. If we simply start creating keys with preference of no compression, it gets maximum quick uptake, and can even be improved with some options in gnupg.conf or equivalent.)

So to sum up, I completely support deprecating compression, because it improves and simplifies the standard. Today, compression by default is of limited benefit, and it simplifies implementation and understanding of OpenPGP to phase it out. I am an enthusiastic supporter of implementations just making the changes now, without a document change. I’m also in favor of deprecating. I’m not opposed to a ban, but I think it’s unneeded.

	Jon

---

From: Andrey Jivsov <[email-masked]>
Date: Mon, 18 Mar 2019 15:03:57 -0700

I agree with deprecation at a reasonable pace. SHOULD looks right.

I would primarily like to correct what PGP Zip is. It is a TAR file with
standard OpenPGP compression. The compression layer was not provided by
gzip (that would not have worked in Windows environment). PGP Zip doesn't
need a license  / is free. Therefore, we should expect that OpenPGP
compression is likely used by users of commercial PGP on Windows. In
business world, many users have PGP installed because their employer wants
every machine to use PGP disk, and so they get PGP Zip available to them.

On Mon, Mar 18, 2019, 1:46 PM Jon Callas <joncallas=
[email-masked]> wrote:

> I like the basic proposal. I think that deprecation is better than
> banning, and consequently we ought to be doing it with SHOULDs and SHOULD
> NOTs rather than MUSTs, as that’s banning rather than deprecating.
>
> However, I want to add that implementations today can deprecate it on
> their own and no changes in the standard need not be there. If an
> implementation creates a key with compression preferences set to no
> compression then any other implementation is bound by the standard not to
> compress. I advocate this as well. As OpenPGP sits today, any implementer
> can unilaterally not only deprecate, but eliminate compression. In fact,
> I’ll go so far as to advocate that all implementers, even GnuPG ought to
> just start making keys by default that eschew compression. More on this
> below.
>
> My rationale for getting rid of compression is different. I think that the
> security-based reasons in this thread are largely unconvincing and often
> just not quite correct. I don’t want to discuss that in this thread,
> because I agree with the destination — let’s move away from compression,
> and am afraid that if we debate the reasons for it we distract from the
> result we mutually want. We should deprecate compression, but not because
> of security, but because of simplicity, and in short a case of “that was
> then, this is now."
>
> OpenPGP has many things in it that are needlessly complex, and compression
> is one of them. (Other needless complexities include the way that packet
> sizes are computed, and others.) Much of this comes from its heritage as a
> utility, and a utility that was designed for DOS and BBSes. The
> considerations of networking and communications in 1991 are not at all what
> we should have for 2021.
>
> One of these differences is that zip-style compression is just not as
> useful now. Obviously, the wins for compression are on things that are the
> largest. Most of the large things we transfer today are already compressed,
> and thus compressing them again is mostly a waste of cpu cycles. There are
> some file formats that have zip-style compression built in. For example,
> GIF and PNG graphics are explicitly compressed. Other formats like JPEG are
> a different sort of compression but have the same characteristic that it is
> reasonably mathematically pseudo-random and not readily compressed further.
> Ditto for audio formats like MP3 and AAC, while FLAC and ALAC are like GIF
> and PNG in that they’re losslessly compressed.
>
> Today, the large things we send are media, and therefore there’s no need
> to compress them. The things we send that are easily compressible tend to
> be smaller.
>
> In the cases where large files are sent that can be compressed, the best
> solution is for that system to compress the file itself. Remember,
> OpenPGP’s history comes from DOS programs on small computers and they
> didn’t even have a way to pipe a tar or zip command to an encryption
> program. That’s not the case today.
>
> Historically, compression has been the largest share of the time it takes
> to process a file with an OpenPGP implementation, and thus we are doing
> something by default that burns CPU for little gain. We should stop doing
> it by default. Again, the best way to do this is to create keys that have a
> compression preference of none. We can do this now.
>
> Over a decade ago, the PGP program had a feature of the “PGP Zip” file.
> This was literally nothing more than a tar file that was run through gzip,
> and then encrypted with OpenPGP without compression. PGP itself had some
> nice viewers that let someone manipulate the container just as other
> directory compression systems did, but any unix system could handle the
> file by just decrypting with gnupg and piping that to tar. Later on, as I
> remember (correct me if I’m wrong, Derek, or someone), we shifted to bz2.
> The same strategy can be improved as compression tools improve.
>
> The compression inside of OpenPGP is also hard to implement correctly. The
> default compression, the “DEFLATE” option is a modified implementation of
> ZIP-style programs from the era of the late 1980s. As those programs
> coalesced into default (not precisely standard, merely ubiquitous)
> implementations, they went in one direction and that era’s PGP stayed with
> the variant. When we did RFC 2440, we added in ZLIB (RFC 1950) encryption
> so that an implementation wouldn’t have to hack the compression software.
> This is another reason to move away from compression inside OpenPGP. RFC
> 4880 added in an option for BZ2 (it was the new hotness at the time), and
> these days the cool kids are using 7-Zip as it's even better than bzip. The
> best way to keep up with advances in compression is piping from a
> compressor to some OpenPGP implementation, rather than continuing to chase
> advancements.
>
> The underlying reality is even worse. Go look at section 5.6 of 2440, and
> there are interoperability hints for working with PGP2 because it had
> further limitations in it for internal table sizes. This is another reason
> to get rid of it for simplicity. It flat isn’t needed.
>
> Simply not having compression aids implementation, as well. A number of
> years ago, I was putting together a Javascript implementation and there
> were no good compression libraries at the time, so we just created keys
> that said “no compression, please” and ran with it. Every implementation
> can handle decoding an OpenPGP object that is not compressed, so there are
> fewer bits of backwards compatibility to transition away from it.
>
> Lastly, I’ll note that everything I’ve said here could be applied to
> banning encryption in 4880-bis, as well. I am not opposed to a ban, but I
> think deprecation is better, particularly since every implementation can
> just stop doing compression by default and everything will work just fine.
> Repeating myself, I even advocate this. Everyone who makes some OpenPGP
> program can stop today, and likely should. I’ll even promote that to
> SHOULD.
>
> (The biggest reason against an outright ban is that there are a lot of
> systems that use OpenPGP in the midst of some internal process, like moving
> around large, sensitive files. There are thus likely a whole lot of shell
> scripts somewhere that someone’s got to find and change, and I wouldn’t be
> surprised if something breaks if there’s a sudden shift. If we simply start
> creating keys with preference of no compression, it gets maximum quick
> uptake, and can even be improved with some options in gnupg.conf or
> equivalent.)
>
> So to sum up, I completely support deprecating compression, because it
> improves and simplifies the standard. Today, compression by default is of
> limited benefit, and it simplifies implementation and understanding of
> OpenPGP to phase it out. I am an enthusiastic supporter of implementations
> just making the changes now, without a document change. I’m also in favor
> of deprecating. I’m not opposed to a ban, but I think it’s unneeded.
>
>         Jon
>
>

---

From: Peter Gutmann <[email-masked]>
Date: Tue, 19 Mar 2019 00:57:51 +0000

Jon Callas <joncallas=[email-masked]> writes:

>One of these differences is that zip-style compression is just not as useful
>now. 

There is one situation in which PGP was at one point widely used but for which
I don't have much current data (more recent than about five years ago) and
that's banking and medical EDI (so X12/HL7), where compression is a required
feature, because you're moving potentially multi-megabyte text files via FTP
to/from mainframes over slow links.

Outside of that rather specialised use, there's no reason to use compression.

>ZIP-style programs from the era of the late 1980s. 

To give some background to this, the original PGP use LHarc because it was
sort-of free.  When the first InfoZip implementations came out, Phil correctly
guessed that it would become the universal standard based on the popularity of
MSDOS Pkzip.  However, what went into PGP 2 was a pre-pre-pre-release of Zip,
whose format didn't settle down completely until after PGP was released.  I
think the only reason why deflateInit() still accepts -13 as a compression
parameter is to trigger the PGP 2-compatibility mode in the compressor.

>there are interoperability hints for working with PGP2 because it had further
>limitations in it for internal table sizes. 

That was due to 16-bit segments on the 8086, even then you had all sorts of
hacks to get things to work.

Peter.

---

From: Jon Callas <[email-masked]>
Date: Mon, 18 Mar 2019 23:03:07 -0700

> On Mar 18, 2019, at 5:57 PM, Peter Gutmann <[email-masked]> wrote:
> 

That’s precisely the use case I was thinking of when I mentioned scripts that would be annoying to update. There’s likely a lot of those still around.

> parameter is to trigger the PGP 2-compatibility mode in the compressor.

Thanks! That helps clarify the context.

> hacks to get things to work.

Oh, I’m sure!

There was so much cleverness and good technology in there at the time, it’s just now we’re 28 years later, and we can move on.

	Jon

---

From: Vincent Breitmoser <[email-masked]>
Date: Tue, 19 Mar 2019 10:41:02 +0100

> I agree with deprecation at a reasonable pace. SHOULD looks right.

I'm unsure why both Jon and you think we need a slower "reasonable pace", when
we have not only one but two very sharp points to make this cut as clean as can
be? Both AEAD and the v5 key format already break compatibility. Why pull
something over the edge that we want to phase out anyways?

 - V

---

From: Justus Winter <[email-masked]>
Date: Tue, 19 Mar 2019 16:21:24 +0100

> something over the edge that we want to phase out anyways?

Agreed.  I prepared a patch:

Author: Justus Winter <[email-masked]>
Date:   Tue Mar 19 16:11:03 2019 +0100

    Deprecate compression support.
    
    Forbid compression if encrypting a message to version 5 keys, or if
    AEAD is used.  Discourage compression otherwise.  Drop the dubious
    paragraph about security benefits of compression.
    
    Motivations:

    

    

diff --git a/middle.mkd b/middle.mkd
index 44d1246..4f88eb2 100644
--- a/middle.mkd
+++ b/middle.mkd
@@ -129,21 +129,21 @@ and a public-key signature algorithm.  The sequence is as follows:
 
 ## Compression
 
-OpenPGP implementations SHOULD compress the message after applying the
-signature but before encryption.
+OpenPGP implementations MUST NOT compress the message if the list of
+recipients includes a version 5 key, or the message is encrypted using
+AEAD.
+
+Otherwise, OpenPGP implementations SHOULD NOT compress the message
+after applying the signature but before encryption.
+
+For backwards compatibility, OpenPGP implementations MAY handle
+compressed data packets.
 
 If an implementation does not implement compression, its authors
 should be aware that most OpenPGP messages in the world are
 compressed.  Thus, it may even be wise for a space-constrained
 implementation to implement decompression, but not compression.
 
-Furthermore, compression has the added side effect that some types of
-attacks can be thwarted by the fact that slightly altered, compressed
-data rarely uncompresses without severe errors.  This is hardly
-rigorous, but it is operationally useful.  These attacks can be
-rigorously prevented by implementing and using Modification Detection
-Codes as described in sections following.
-
 ## Conversion to Radix-64
 
 OpenPGP's underlying native representation for encrypted messages,

---

From: Andre Heinecke <[email-masked]>
Date: Wed, 20 Mar 2019 00:47:25 +0100

Hi,

> Hello,

Hello to you, too. Here is a parable:

Kleopatra goes into a Bar:
Kleo (linux): Hey look how small my archives are now using LZMA!!
Kleo (win): Uhm LZMA is that a disease? I don't know it!
KMail (linux): Ha, I have KArchive. K7Zip is the Best!
GpgOl (win): Uhm 7zip, does that mean that I have to unzip it seven times?
KMail (linux): No stupid, you just have to get your user to install additional 
software.
GpgOL (win): Awww. shucks. I thought we were user friendly nowadays.
GpgOL (win): Haha! But now!  I have winrar.
Kleo (linux): What? Winrar? "Win" and rar? go @!§13 yoursellf.
KMail (linux): Yeah KArchive can handle that.. *sunglasses*
Enigmail (win): Eh, guys I'm a crypto plugin, what are you talking about? I 	
			can no longer decrypt your messages! What are you doing?!!
Kleo (win): I can do ZIP! (Imagine Ralph Wiggum)

And even in this scenario how would they even communicate which compression 
algos are used. Because it is _not standartized_
OpenPGP has been around for ages. Software relies on the pecularities of it.
If I would have to design a cryptographic message format from scratch I 
probably also would not include compression in there.

But the existing software relies on that. We are not designing a new standard. 
We are designing an improvement on RFC4880.

I don't want to rewrite major parts of KMail / GpgOL / Kleopatra with some 
weird unstandartized custom compression stuff. It's a part of OpenPGP. IMO it's 
a good part.

It's nice that you guys have new ideas but we have literally millions and 
possibly billions of users already and we have to be f'''ing rock solid in our 
implementations and in our standards.

Best Regards,
Andre
-- 
GnuPG.com - a brand of g10 Code, the GnuPG experts.

g10 Code GmbH, Erkrath/Germany, AG Wuppertal HRB14459
GF Werner Koch, USt-Id DE215605608, www.g10code.com.

---

From: Vincent Breitmoser <[email-masked]>
Date: Wed, 20 Mar 2019 14:14:49 +0100

> weird unstandartized custom compression stuff.

I'm confused. This thread is about deprecating compression algorithms, noone
mentioned adding "custom compression stuff".

Is your point that you would want to keep compression as a feature in those
applications, hence removing it from OpenPGP would mean you'd have to keep it
around unstandardized / on a different layer?

 - V

---

From: Andre Heinecke <[email-masked]>
Date: Wed, 20 Mar 2019 14:17:53 +0100

> around unstandardized / on a different layer?

Exactly. The User Experience of Kleopatra on Windows is that you right click a 
folder and encrypt that folder. You need compression there.
Currently it is standardized. You propose to remove the standard so it will be 
unstandartized and no longer interoperable.

Regards,
Andre

GnuPG e.V., Rochusstr. 44, D-40479 Düsseldorf.  VR 11482 Düsseldorf
Vorstand: W.Koch, M.Gollowitzer, A.Heinecke.    Mail: [email-masked]
Finanzamt D-Altstadt, St-Nr: 103/5923/1779.   Tel: +49-2104-4938799

---

From: Andre Heinecke <[email-masked]>
Date: Wed, 20 Mar 2019 15:53:58 +0100

Hi,

Vincent said that my mail was too rambling. I agree. Here is the short 
Version:

* Compression is good and necessary.
* Standardized compression is good.
* I don't want to use unstandardized compression only because you do not want 
to implement compression in sequoia.
* There is no reason the change the standard. Compression has been around for 
ages. It is _standard_ and working well.

-----------

>    that downstream users struggle with the correct use of streaming

Whose expericence? Not mine. It's OpenPGP and S/MIME. You have to handle this 
even if you don't like it. Compression does not change this.

> - Compression allows the construction of quines.

Yeah that sucks. But a downstream application would still be affected. e.g. a 
MUA or Kleopatra if you move compresssion out of the standard.

>    BREACH, and hiding of EFAIL-style CFB gadgets [0].

*rolleyes* Was OpenPGP affected? Nope.

>    and how to compress data that is then encrypted using OpenPGP.

Nope. Because the downstream application does not know anything about the 
sending application because it is not standardized in you proposal.

  >  trusted computing base of implementations.

Nope. Removing it makes handling it more complex because we are working with a 
well established standard. So you still need to handle old messages so you 
still need to handle compression. But Oh! Now you also have to handle non-
standard compression. Fun! Complexity -> Increase!

Regards,
Andre

---

From: "Daniel A. Nagy" <[email-masked]>
Date: Wed, 20 Mar 2019 19:18:14 +0100

Hi,

I strongly agree with this. Compression is a very useful safety belt
making statistical cryptanalysis substantially harder even if the used
cipher begins showing weakness.

Cheers,

Daniel

---

From: Bart Butler <[email-masked]>
Date: Wed, 20 Mar 2019 18:47:27 +0000

I tend to agree with Andre. Compression still has its place. For example, we are an email provider and receive millions of email per day, most of which are text and thus highly compressible, and due to the volume it's actually an appreciable amount of storage space (read: money) that we save via compression. There's also no BREACH/CRIME risk in this particular use case and in many other use cases for PGP.

>From the library perspective decompression support is certainly here to stay basically forever for legacy messages at the very least, so I don't see a lot of benefit to implementation developers from dropping support either. If you have a streaming interface, you'll still need to support streaming decompression.

One thing we could add to the compression section would be a SHOULD for ZLIB as opposed to the legacy ZIP and bzip2, which is slow and doesn't have much of a reason to exist in my opinion. There's no real guidance right now to which is preferred. I'd also be happy deprecating creation of new messages with ZIP and bzip2 if we wanted to go there, though maybe someone has a use case that I'm not aware of.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

> -   Compression is good and necessary.
> -   Standardized compression is good.
> -   I don't want to use unstandardized compression only because you do not want

> -   There is no reason the change the standard. Compression has been around for
>     ages. It is standard and working well.
>     

> rolleyes Was OpenPGP affected? Nope.
> 

> -----------------

> Finanzamt D-Altstadt, St-Nr: 103/5923/1779. Tel: +49-2104-4938799_______________________________________________
> openpgp mailing list
> [email-masked]

---

From: Jon Callas <[email-masked]>
Date: Wed, 20 Mar 2019 12:36:34 -0700

> On Mar 19, 2019, at 2:41 AM, Vincent Breitmoser <[email-masked]> wrote:
> 

I’m going to quibble over “two” and say that we have one — simplification of an overall implementation.

The second one has had comments on before. I’ll pick up some more in another missive relating to things people have said today.

To address your point, as I said in my long missive, you can do this today. No changes are needed to the protocol. All you have to do is put a compression preference on your key that says no compression, and then you won’t get compression. (Well, to be completely correct, if someone compresses then they’re non-compliant to the standard.) Repeating myself, I support and encourage implementations to do that by default.

Since you can do it today, there’s no need to rush. There are other people who have their own legitimate needs for it.

Does this make sense?

	Jon

---

From: Jon Callas <[email-masked]>
Date: Wed, 20 Mar 2019 13:57:11 -0700

As mentioned in my previous note, I feel like I need to comment on the security rationales, rather than the software engineering rationale. Here’s what was given before. Andre Heinecke also had very good comments.

> On Mar 18, 2019, at 5:07 AM, Justus Winter <[email-masked]> wrote:
> 

What experience? References?

I have a raised eyebrow on this reason because the vast majority of every PGP message created in the last 28 years had compression on. I don’t know of any problems. I don’t know of any downstream uses struggling with streaming interfaces. Moreover, this is a criticism of streaming. 

Moreover, in the common case (an email), there are many plaintext transformations that go on. For example, there’s quoted-printable, hexadecimal expansion, base64 encoding, and lots of other things that go on in the plaintext that make it hard to know the length of the final plaintext. You’re right that it’s hard to know the final length of a message, but removing compression doesn’t solve the problem, nor does it make it appreciatively better. 

Are you suggesting that OpenPGP only allow definite lengths?

To me, this seems like a non sequitur even if you back it up with evidence. 

>  - Compression allows the construction of quines.

So what? How is this a problem? Text editors allow the construction of quines, too. 

This is an assertion, but it doesn’t say how it’s a problem, nor does it state how removing compression would remove the problem. 

My suggestion — moving compression to an upper layer — also allows quines, since it allows someone to encrypt a bit string that’s been compressed, and therefore might have a quine in it.

>    BREACH, and hiding of EFAIL-style CFB gadgets [0].

This isn’t strictly correct.

There are a number of attacks on interactive encryption protocols that use differences in different compressed plaintext to learn something about the internal structure of the plaintext. This is obviously bad.

However, *static* encryption, like OpenPGP doesn’t have this problem.

Here’s a challenge I give.

Create two plaintexts, P and P’ where P’ = compress(P). Pick any compression function and any plaintext. Now, encrypt them both, so we have E_1 = encrypt(P) and E_2 = encrypt(P’). Show that there is an advantage to an attacker for recovering P’ from E_2 over recovering P from E_1.

I assert that if you can, then your cipher is flawed and you need to replace it. There is nothing magical about compressed plaintext that makes it easier to recover.

With regards to EFAIL, that is a huge problem that is far more to do with MIME encoding than anything else. It applies to S/MIME as well as OpenPGP, and S/MIME typically does CBC mode as opposed to CFB and typically does not compress. Thus the connection is tenuous, because EFAIL applies not only to compressed data with CFB mode, but uncompressed data with CBC.

>    and how to compress data that is then encrypted using OpenPGP.

I kinda agree with you, in this one. I’d say upstream rather than downstream because I think that the generator of encrypted objects is upstream, and the thing that processes the output is downstream. But I agree with what I think you’re saying.

Nonetheless, this is a feature that’s been there since Day One, and there are many systems that rely on OpenPGP compressing. In that case, the other application has made a decision and this proposal would override their decision on the grounds that they should be allowed to make a decision.

This is why while I like deprecating compression, I think we need to move with care.

>    trusted computing base of implementations.

I don’t disagree here, and this is essentially my rationale. However, to be fair, this is a part of the system that is used on nearly every encryption and there haven't been many (or any?) problems. Removing something is also risky and introduces other chances for errors, too.

This is why taking action *now* by just stopping to use compression is a very good idea. It shows that there won’t be compatibility issues, and the more that the actual use goes down, the more than an argument to remove makes sense. 

The worst thing possible would be that no-compression gets tied into things like V5 keys (which we need) and AEAD modes (which we also need) is that it would blunt adoption of the good new technologies by adding on and tying things together. This is a classic problem that many systems have, that the goodness is blunted because it drags in something about which gentlepersons can disagree. Okay, enough soapbox.

I think we should start moving away from default compression, starting now. I think we should move slowly with the standard and collect data about use, which would support removal in a future update.

	Jon

---

From: Marcus Brinkmann <[email-masked]>
Date: Thu, 21 Mar 2019 01:09:39 +0100

Hi Andre,

isn't gpgtar (which I assume is still used by Kleopatra on Windows to
encrypt folders) also "unstandardized" by OpenPGP?  For all practical
purposes, compressed archive formats are well-established across all
platforms, so it's unclear to me what kind of problems you fear.

Thanks,
Marcus

-- 
Dipl.-Math. Marcus Brinkmann

Lehrstuhl für Netz- und Datensicherheit
Ruhr Universität Bochum
Universitätsstr. 150, Geb. ID 2/461
D-44780 Bochum

Telefon: +49 (0) 234 / 32-25030
http://www.nds.rub.de/chair/people/mbrinkmann

---

From: Gregory Maxwell <[email-masked]>
Date: Thu, 21 Mar 2019 00:30:07 +0000

> I assert that if you can, then your cipher is flawed and you need to replace it. There is nothing magical about compressed plaintext that makes it easier to recover.

We've been here before:

https://mailarchive.ietf.org/arch/msg/openpgp/rG-X9rp2jlbyACoosnbxRXjCeys

I buy the combining encryption with compression being useful
argument... but at the same time, openpgp compression is increasingly
far from the state-of-the-widespread-art (e.g. xz) and there probably
isn't much interest in updating it to chase the state of the art
compression (and for short human texts, I think recent machine
learning progress look like they're resulting in significantly higher
amounts of compression, -- just no one has productionized that work
yet).

---

From: Werner Koch <[email-masked]>
Date: Thu, 21 Mar 2019 09:32:52 +0100

On Wed, 20 Mar 2019 18:47, bartbutler=[email-masked]
said:

> One thing we could add to the compression section would be a SHOULD
> for ZLIB as opposed to the legacy ZIP and bzip2, which is slow and

I was about to write the same.

Salam-Shalom,

   Werner

-- 
Die Gedanken sind frei.  Ausnahmen regelt ein Bundesgesetz.

---

From: Jon Callas <[email-masked]>
Date: Thu, 21 Mar 2019 15:37:46 -0700

> On Mar 20, 2019, at 5:30 PM, Gregory Maxwell <[email-masked]> wrote:
> 
> 

I almost brought up that case, as I find it interesting and compelling, but it’s not the case we’re dealing with here, and I thought it would be some combination of trollish and a straw man, so I didn’t.

This is a case where there’s an issue. For those wanting a TL;DR, someone created a text ballot that got encrypted with OpenPGP and sent in an email. Because the encrypted ballot a single question and compressed to a different size, so you could tell who someone voted for by the size of the ballot (all other things being equal). [1]

This is a problem that’s worth looking at. (Also, I feel like I am a broken record here.) There is an existing workaround for this issue, and that’s to put a no-compression preference on the key people encrypt to. 

I believe this is yet another reason why the implementations might want to move towards compression off by default. 

That wasn’t what was said at the time — at the time, the debate was that because of this problem, compression should be banned in the standard, which is not only more problematic (as we’re seeing now, there are a lot of people who depend on it), and the slower way to handle things. 

I am quite sure you’re frustrated, and I am too. I agree that we ought to phase out compression, and there are a number of reasons for it. This is one of those reasons. I’m sure that I frustrate you, too, because I consider that case to be a minor reason. We are, however, on the same side on outcomes. Let’s concentrate on the outcome we agree with.

	Jon

[1] Getting rid of encryption does not completely solve the ballot problem; you still need to design the ballot properly. Here’s the counter example: Consider a ballot where someone writes in either “Alice” or “Bob.” This ballot leaks the choice because Alice’s votes are two bytes longer in length. A ballot that ships “[ ] Alice [ ] Bob” and asks to put an X between the appropriate brackets is much better ballot construction. It’s still possible to get that one wrong, too, either by accident or on purpose.

---

From: Werner Koch <[email-masked]>
Date: Fri, 22 Mar 2019 10:19:07 +0100

On Wed, 20 Mar 2019 18:47, bartbutler=[email-masked]
said:

> doesn't have much of a reason to exist in my opinion. There's no real

So what about this minor change:

@@ -3322,8 +3321,9 @@ ## Compression Algorithms
  100--110  Private/Experimental algorithm
 
 Implementations MUST implement uncompressed data.  Implementations
-SHOULD implement ZIP.  Implementations MAY implement any other
-algorithm.
+SHOULD implement ZLIB.  For interoperability reasons implementations
+SHOULD be able to decompress using ZIP.  Implementations MAY implement
+any other algorithm.
 
 ## Hash Algorithms

Salam-Shalom,

   Werner

---

From: Ronald Tse <[email-masked]>
Date: Fri, 22 Mar 2019 10:32:55 +0000

Agree. Thanks Werner!

_____________________________________

Ronald Tse
Ribose Inc.

On Mar 22, 2019, at 5:19 PM, Werner Koch <[email-masked]<mailto:[email-masked]>> wrote:

On Wed, 20 Mar 2019 18:47, bartbutler=[email-masked]<mailto:bartbutler=[email-masked]>
said:

## Hash Algorithms

Salam-Shalom,

  Werner

--
Die Gedanken sind frei.  Ausnahmen regelt ein Bundesgesetz.
_______________________________________________
openpgp mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/openpgp

---

From: Derek Atkins <[email-masked]>
Date: Fri, 22 Mar 2019 10:17:58 -0400

+1 from me!
Thanks, Werner.

-derek

---

From: Bart Butler <[email-masked]>
Date: Fri, 22 Mar 2019 16:58:27 +0000

+1 from me too, thanks!

> Werner Koch [email-masked] writes:
> 

> > Hash Algorithms

> > ----------------

---

From: Jon Callas <[email-masked]>
Date: Fri, 22 Mar 2019 15:01:09 -0700

> On Mar 22, 2019, at 2:19 AM, Werner Koch <[email-masked]> wrote:
> 

+1. Thanks, Werner.

	Jon

---

From: Jon Callas <[email-masked]>
Date: Fri, 22 Mar 2019 15:03:50 -0700

> On Mar 18, 2019, at 1:45 PM, Jon Callas <joncallas=[email-masked]> wrote:
> 

[…]

> … banning encryption in 4880-bis, as well. 

Obviously, I meant compression, not encryption. Oops. Thanks to the friend who pointed this out.

	Jon

---

From: ilf <[email-masked]>
Date: Sat, 23 Mar 2019 18:07:23 +0100

So can we change GnuPG to default-preference Uncompressed?

Jon Callas:
> To address your point, as I said in my long missive, you can do this 
> today. No changes are needed to the protocol. All you have to do is 
> put a compression preference on your key that says no compression, and 
> then you won’t get compression. (Well, to be completely correct, if 
> someone compresses then they’re non-compliant to the standard.) 
> Repeating myself, I support and encourage implementations to do that 
> by default.

-- 
ilf

If you upload your address book to "the cloud", I don't want to be in it.

---

From: Daniel Kahn Gillmor <[email-masked]>
Date: Sat, 23 Mar 2019 23:13:30 +0100

> So can we change GnuPG to default-preference Uncompressed?

[ switching this implementation-specific message to
  [email-masked], please respect Mail-Followup-To: ]

The current implementation of GnuPG creates OpenPGP certificates with
this preference listing:

     Compression: ZLIB, BZIP2, ZIP, Uncompressed

Are you suggesting that we change it to:

     Compression: Uncompressed, ZLIB, BZIP2, ZIP

or to:

     Compression: Uncompressed

?

Setting aside the question of defaults, for your own OpenPGP certificate
right now you can do this with any modern version of GnuPG, if you're
willing to poke at the command line with some arcana:

For the less severe change:

    gpg --edit-key $FINGERPRINT showpref 'setpref S9 S8 S7 S2 H10 H9 H8 H11 H2 Z0 Z2 Z3 Z1'  save

For the more severe change:

    gpg --edit-key $FINGERPRINT showpref 'setpref S9 S8 S7 S2 H10 H9 H8 H11 H2 Z0' save

then of course you'll need to re-publish the cert via whatever your
standard publication mechanism is (keyservers, WKD, keybase, etc).

        --dkg

---

From: "Neal H. Walfield" <[email-masked]>
Date: Wed, 27 Mar 2019 10:50:23 +0100

Hi,

Werner wrote on [email-masked] that he views 4880bis as done,
and the recent proposals as "severe last minute changes".

Is there general consensus that 4880bis is done?

Is further discussion of the changes to 4880bis no longer desired?

Are new proposals no longer desired?

Thanks!

Neal

On Tue, 26 Mar 2019 22:36:43 +0100,
Werner Koch wrote:
>  Those folks who are trying to get severe
> last minute changes into a revision of a standard may be better off to
> start their protocol from scratch.

---

From: "Neal H. Walfield" <[email-masked]>
Date: Wed, 27 Mar 2019 11:46:01 +0100

Hi Andre,

On Wed, 20 Mar 2019 14:17:53 +0100,
> unstandartized and no longer interoperable.

You talk about encrypting a folder.  So I assume you mean multiple
files.  As far as I know, 4880 doesn't support this out of the box.
For instance, an OpenPGP message can only have one literal data
packet:

  https://tools.ietf.org/html/rfc4880#section-11.3

Are you using some container format?  PGP/Mime with multiple
attachments?  Something else?

Thanks,

Neal

---

From: "Neal H. Walfield" <[email-masked]>
Date: Wed, 27 Mar 2019 11:57:40 +0100

Hi Andre,

On Wed, 20 Mar 2019 15:53:58 +0100,
> to implement compression in sequoia.

For the record, Sequoia implements all three compression algorithms
that 4880 specifies.  Also, I don't think Justus suggested that we
remove compression from 4880bis, because we don't want to implement it
in Sequoia, but due to a number of security-centric reasons.  I
welcome constructive criticisms.

> Here is where annoyed rambling about unproductive suggestions starts:

Calling Justus's suggestion "unproductive" is an insult.  If it is so
obviously wrong, then it should be easy to provide a short, convincing
response that ends the discussion.  Unless you think that Justus is a
troll.  In that case, please don't feed the trolls.

I've tried to act in good faith, and appeal solely to technical
arguments.  I hope that we can all work towards this, at least on this
list.

Thanks,

Neal

---

From: Andre Heinecke <[email-masked]>
Date: Wed, 27 Mar 2019 11:59:32 +0100

Hi,

> attachments?  Something else?

As you know we are using tar archives. We take pains that our tar 
implementation is compatible with the tar format used by PGP.

Marcus already made that point. "Why are you not using tar.gz.gpg" if you use 
something out of the standard anyway?

Here I said that this will move something that was standartized (the 
compression) out of the standard and leave it to the application. 

But if you dislike my example with a folder, which I chose to underline that 
compression is needed. Please ignore it and respond to the other points made 
in this thread like compression for mails or just replace "folder" in my 
example with "file" and there you go.

I find this kind of discussion about my example unhelpful and distracting on 
this list. If you want to further discuss this please do it off list.

Regards,
Andre

---

From: "Neal H. Walfield" <[email-masked]>
Date: Wed, 27 Mar 2019 12:24:41 +0100

Hi Andre,

On Wed, 27 Mar 2019 11:59:32 +0100,
Andre Heinecke wrote:
> As you know we are using tar archives.

Actually, I didn't know, which is why I asked.

I've used Kleopatra once or twice to see how it worked, and I've never
used KMail or GpgOl.

> compression is needed.

I'm sorry that I gave you the impression that I disliked your example.

> Please ignore it and respond to the other points made 

Your tone makes it sounds like I've personally attacked you.  I'm not
sure where I did that.  If so, I apologize and I will try and keep at
least my side of the discussion more focused on the technical
discussion.

Thanks,

Neal

---

From: Benjamin Kaduk <[email-masked]>
Date: Sat, 30 Mar 2019 11:49:36 -0500

> least my side of the discussion more focused on the technical

At risk of being accused of piling on,  I'll note that the tone also came
across as something of an escalation, to me.  (I don't know whether that
was the intent or not.)

As for "unhelpful and distracting", technical topics about how openpgp
implementations work seem to be helpful things to have in this forum, in
general.  Yes, it might be distracting from another topic that Andre might
want to focus on, in this case, but that is not really grounds to shut down
discussion of it.

-Ben

---

From: Bart Butler <[email-masked]>
Date: Sat, 30 Mar 2019 17:12:01 +0000

I would agree with Werner that it's essentially done, with minor tweaks being the only thing really remaining.

That said, I'd say limiting the maximum AEAD chunk size would be classified under 'minor tweak' in my book.

-Bart