Subject: Re: [openpgp] AEAD Chunk Size

From: Benjamin Kaduk <[email-masked]>
Date: Mon, 1 Apr 2019 11:10:23 -0500

On Sat, Mar 30, 2019 at 10:16:46PM +0100, Neal H. Walfield wrote:
> Hi Ben,
> 
> Thanks for your note.
> 
> At Sat, 30 Mar 2019 10:04:38 -0500,
> Benjamin Kaduk wrote:
> > I also have a use case for authentication of large chunks of data at rest:
> > they allow me to use a cheap bulk storage service that provides
> > (best-effort) replication and archiving but has poor physical security.  So
> > I encrypt my data to myself and put it in storage, but when I get it  back
> > I need to know that it's valid.  I can imagine at least one case where
> > knowing exactly which chunk was corrupted would save effort; it may be a
> > toy example but perhaps it is illustrative of a broader case.  Note that
> > there are algorithms to compute pi to arbitrary precision, and even to
> > compute the Nth digit thereof without coputing the previous digits.  If I
> > need to have random-access inquiries into the value of pi, I could
> > precompute using softare I trust and do this self-encryption thing, and
> > when a chunk is bad I can recompute only that chunk and still trust that I
> > only ever use values generated by my trusted implementation.
> 
> Just to be clear: when you say "large chunks of data at rest," you're
> not arguing that large AEAD chunks are better, are you?  It seems to
> me that if you use small chunks, at least in your example, you have
> less work to do when you discover a corrupted chunk.

Thanks for spotting that; my "large chunks of data at rest" was meant to
just be large quantities of data (e.g., TB or more), with no relationship
to the chunk size used in the encryption thereof.

-Ben

---

From: Conrado P. L. Gouvêa <[email-masked]>
Date: Tue, 2 Apr 2019 09:50:12 -0300

(Sorry, I'm a lurker in this group, but wanted to chime in...)

On Fri, Mar 29, 2019 at 10:45 AM Peter Gutmann
<[email-masked]> wrote:
> That was due to broken email apps.  If I can convince your email app to
> forward the plaintext of a decrypted message to me, you lose no matter what
> encryption mechanism you use.
>
> Admittedly CBC/CFB made this easier, but it was the email apps that needed
> fixing, not PGP.

I believe that's not true. The second EFail attack was caused by PGP
allowing removing the MDC or downgrading packets. This would not
happen if PGP handled authentication correctly.

>
> I'm not saying it's not worth addressing, but before endlessly debating
> solutions we need to figure out what problem we're solving.  "We have this
> cool AEAD mode lying around and want to apply it to something" isn't a
> problem, or at least not a problem that a PGP user cares about solving, it's
> something interesting for geeks to play with.
>
> In the last five years or so I've received approximately zero PGP-encrypted
> emails, and I'm one of the people who helped write the thing.  OTOH I've
> probably encrypted gigabytes of data with it, almost always symmetric-key with
> the key communicated out of band.  In none of those cases would blocked auth
> protection have been useful.
>

This whole discussion has nothing to do with AEAD itself. With
"manual" encrypt-then-authenticate, the issue of chunk size and
releasing unauthenticated plaintext is the same.

Even for data at rest: many people store the data in cloud providers,
where it could be tinkered with by attackers.
If the problem is to be resilient to file corruption, then the answer
is error-correcting codes, not ignoring authentication. (Though I
agree that's an important issue that is mostly overlooked in crypto)

Conrado

---

From: Conrado P. L. Gouvêa <[email-masked]>
Date: Tue, 2 Apr 2019 10:12:29 -0300

On Sat, Mar 30, 2019 at 11:59 PM Peter Gutmann
<[email-masked]> wrote:
> I'm not saying remove it, just get some data to support making a decision in
> some way.  In particular, AEAD is a good thing, but there's no evidence that
> chunking with AEAD, which complicates things greatly, is useful or necessary.
>

I know you're tired of hearing about it... but EFail.
Even if PGP used AEAD, but without chunks, EFail would probably still
happen. If the AEAD data is arbitrarly large, then implementations
would be forced to provide a streaming API that discloses
unauthenticated plaintext, and the same thing would happen.

Unfortunately I'm not aware of other examples, though I'm pretty sure
they must exist... But why should we wait for more of this issues to
happen before fixing the underlying cause, if we can fix it now? (And
"now" meaning many years hence, since the standard will take a while
to be adopted).

Adam Langley has a good post about it:
https://www.imperialviolet.org/2014/06/27/streamingencryption.html
And many examples of cryptographers claiming releasing unauthenticated
plaintext is dangerous:
https://crypto.stackexchange.com/questions/41087/is-there-an-upper-limit-to-plaintext-size-in-xsalsa20poly1305/51439
https://crypto.stackexchange.com/questions/51537/delayed-tag-checks-in-aes-gcm-for-streaming-data

Conrado

---

From: Jon Callas <[email-masked]>
Date: Tue, 2 Apr 2019 10:42:50 -0700

> On Apr 2, 2019, at 6:12 AM, Conrado P. L. Gouvêa <[email-masked]> wrote:
> 

No, no, it’s okay, because this why I was saying, “Let’s not talk about Efail.” The AEAD discussion is good, and there are many reasons to upgrade to allow its use. If one of those reasons is complex, then having that be the major reason means that there’s a counter-argument that is essentially, “if this isn’t the silver bullet claimed, then maybe we shouldn’t do it,” and worse, it’s a completely reasonable counter-argument. 

There’s one more small issue around AEAD that I’ll bring up in another note.

	Jon

---

From: "Neal H. Walfield" <[email-masked]>
Date: Wed, 10 Apr 2019 10:47:54 +0200

Hi Bart,

Thanks for this mail.  This much more succinct mail then I have been
able to write captures my concerns with Jon's line of thinking.

@Jon: I hope you'll take the time to respond to Bart's questions.

Thanks!

:) Neal

On Sun, 31 Mar 2019 06:11:25 +0200,
Bart Butler wrote:
> On Saturday, March 30, 2019 4:09 PM, Jon Callas <[email-masked]> wrote:
> > > On Mar 29, 2019, at 7:17 PM, Bart Butler bartbutler=[email-masked] wrote:
> > > Hi Jon,
> > > As others have noted, there is a lot of confusion on this thread, some of which you touched in your AEAD Conundrum message, like when we say AEAD should not release unauthenticated plaintext, do we mean the entire message or the chunk?
> > 
> 
> > That is precisely the question. But the bigger question is whether you care about that. Sometimes it matters, sometimes it does not.
> > 
> 
> > For example, let’s suppose that I have a very large blob on mass storage. Media failures happen. If there’s a bad block on a disk, do you want to lose the whole file because of it? Sometimes you want to throw your hands up in the air. This is most common in an interactive protocol, and in general the answer is yes. For example, an SSL connection to a server, if there’s funny business going on, you want to blow up the connection and try over again. On the other hand, if you had an archival thing (e.g. tar file with some historic documents), you want to recover as much as you can.
> > 
> 
> > OpenPGP is in general the latter case rather than the former. I believe it’s less important to have strict semantics on failures because it’s usually storage.
> > 
> 
> 
> I agree. I would say my point is that with sufficiently small chunks, the user/decrypter can choose what kind of failure behavior is appropriate. Large chunks robs the decrypter of that.
> 
> > > Another piece of confusion is that Efail isn't a single vulnerability, it was several vulnerabilities related (at best) thematically.
> > 
> 
> > I understand Efail. Trust me.
> 
> I do. I was bit excessively pedantic defining everything because I felt that this thread has suffered from ambiguity in several terms.
> 
> > 
> 
> > > So to be very specific, for the purpose of the following discussion, the advantage of smaller AEAD chunks is specifically to prevent Efail-style ciphertext malleability/gadget attacks, and the prohibition on releasing unauthenticated plaintext is applied to individual chunks, which is sufficient to foil this kind of attack in email.
> > 
> 
> > I disagree. If you want to prevent something like Efail, you want larger chunks. Assuming that you believe that early release matters.
> > 
> 
> > Let’s rewind here, and not talk about Efail, let’s talk about the real issue. If you want the entire blob to have all-or-nothing semantics, then you want the fewest number of chunks as is reasonable. If you have attacker-controlled inputs, then every joint between the chunks is a vulnerability.
> 
> OK, I think this is the part that I don't understand. Why does it matter what chunking scheme is used here? If my app requires all-or-nothing semantics, I would program my app to enforce that all chunks must pass and not release plaintext unless that happened, with no truncation, etc. So why would every joint be a vulnerability?
> 
> > > What value does large-chunk AEAD actually provide? What I'm getting from the AEAD Conundrum message is that it's a way for the message encrypter to leverage the "don't release unauthenticated chunks" prohibition to force the decrypter to decrypt the whole message before releasing anything. Why do we want to give the message creator this kind of power? Why should the message creator be given the choice to force her recipient to either decrypt the entire message before release or be less safe than she would have been with smaller chunks?
> > 
> 
> > Let me summarize the conundrum: If you want strict AEAD no-release semantics, you want a fewer number of chunks.
> 
> I guess this is my fundamental question. You can force no-release semantics at the application level for any chunk size scheme, right?
> 
> > 
> 
> > > Coming back to Neal's point, it's really hard to see any sort of value in really large AEAD chunks, because the performance overhead is negligible at that point and the only security 'benefit' that I can see is the encrypter trying to use the spec to force the decrypter to not stream, which does not seem like something at all desirable.
> > 
> 
> > Okay, here’s another thing that’s a pet peeve of mine. We’re arguing security and you brought up performance. I never mentioned performance, the people who want large chunks haven’t brought it up. They want large chunks because they perceive it to be more secure..
> > 
> 
> > If you respond to a security request with a performance answer, you literally don’t know what you’re talking about. So let’s toss that aside.
> 
> I apologize, I was not trying to create a strawman here, but I am completely at a loss for what the benefit of large chunks is.
> 
> >     Elided out of this, and possibly important is that “support” includes chunks smaller than that size. I should have said that, but I wanted it to be as stark as possible. So let me repeat it and abstract it with some variables:
> >     
> 
> >     (1) MUST support up to <small-chunk-size> chunks.
> >     
> 
> > 
> 
> > (2) SHOULD support up to <larger-chunk-size> chunks, as these are common..
> > (3) MAY support larger up to the present very large size.
> > (4) MAY reject or error out on chunks larger than <small-chunk-size>, but repeating ourselves, SHOULD support <larger-chunk-size>.
> > 
> 
> > Clauses (3) and (4) set up a sandbox for the people who want very large chunks. They can do whatever they want, and the rest of us can ignore them.. Why get rid of that? It doesn’t add any complexity to the code. It lets the people who want the huge ones do them in their own environment and not bother other people.
> > 
> 
> > My concern is over (1) and (2) and specifically that there’s both <small> and <large> sizes.
> > 
> 
> > I think that’s an issue. If there are two numbers we are apt to end up with skew before settling on one, so it’s better to agree on just one. That’s the real wart in my proposal.
> > 
> 
> 
> I'm OK with eliminating (2) and just using the MAY part to take care of any legacy 256K messages OpenPGP.js users might have. As I said, we don't have any of these messages in production yet and I'd err on the side of a cleaner spec. I just really want to understand the benefit of large chunks for security and right now I clearly do not.
> 
> [1.2 OpenPGP digital signature <application/pgp-signature (7bit)>]
> No public key for 99054465EF331E44 created at 2019-03-31T06:11:24+0200 using (unknown algorithm 22)
> [2  <text/plain; us-ascii (7bit)>]
> _______________________________________________
> openpgp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/openpgp

---

From: "Neal H. Walfield" <[email-masked]>
Date: Wed, 10 Apr 2019 15:03:43 +0200

On Tue, 02 Apr 2019 19:42:50 +0200,
> No, no, it’s okay, because this why I was saying, “Let’s not talk about Efail.” The AEAD discussion is good, and there are many reasons to upgrade to allow its use. If one of those reasons is compl…

I agree that EFail is not the only reason to consider AEAD.  And, I
think that the complexity counter argument is a convincing one.  Like
Ferguson, Schneier and Kohno said in "Cryptography Engineering":

  The more complex a system, the more likely it has security problems.
  Indeed, we like to say that complexity is the worst enemy of
  security.

But, once we decide that we want AEAD, I think it is fair to apply the
same counter argument to any proposals.

In our case, parameterizing the chunk size adds complexity.  It
standardizes not a single algorithm, but a family of algorithms.  I've
already shown how this parameterization can be abused.  But whether
you think my attack is relevant or not, I think we agree that the
burden of justification ought to be on those defending the complexity,
i.e., allow multiple chunk sizes, or allow large chunk sizes.

Currently, I think the only extant argument in favor of large chunks
is your argument in <[email-masked]>
or a variant thereof:

  I believe that the more you believe tight security is necessary, then
  the more *willing* you ought to be to allow people with special needs
  to go off in the weeds on their own.

But, we don't prohibit people from experimenting!  That's why we have
a private name space.  If someone really has such special needs, they
can use, e.g., Tag 61, for an AEAD variant with large chunks.  If it
turns out those needs are not so special, we can standardize that
variant.

(Tobias has proposed foregoing chunking.  That's a different argument,
and one that I think is not interesting for us since it prevents
streaming.)

If there are other unaddressed arguments in favor of large chunk
sizes, please state them.  If I missed them, please repeat them and
accept my apologies.

Thanks!

---

From: Derek Atkins <[email-masked]>
Date: Fri, 12 Apr 2019 09:36:58 -0400

[ Coming back to this late after traveling for a while.  My apologies if I
am re-opening wounds... ]

Marcus Brinkmann <marcus.brinkmann=[email-masked]>
writes:

> The main question here is: What should a conforming application look like?
>
> The current behaviour of GnuPG is that it will process internally (e.g.,
> through the decompression and signature verification layer) and output
> externally unauthenticated plaintext.  If an AEAD chunk is modified by
> an attacker, GnuPG will detect the modification and cancel the
> operation, but only at the end of each chunk.  Due to the asynchronous
> buffer management in GnuPG, quite often some part of the modified chunk
> has then already been processed and output, depending on the particular
> state of the buffers, the buffer size and the chunk size.  This
> behaviour increases the surface for chosen ciphertext attacks and
> possibly adaptive chosen plaintext attacks (if an oracle is exposed).

In my mind, this sounds like the implementation is broken.  If it
releases AEAD plaintext before the end of the AEAD chunk then it is
non-conforming and should be considered broken.

-derek

-- 
       Derek Atkins                 617-623-3745
       [email-masked]             www.ihtfp.com
       Computer and Internet Security Consultant

---

From: "Neal H. Walfield" <[email-masked]>
Date: Fri, 12 Apr 2019 15:46:06 +0200

On Fri, 12 Apr 2019 15:36:58 +0200,
> non-conforming and should be considered broken.

I fully agree with you.

Given this position, it seems to me that all implementations will
necessarily fail on very large chunks (e.g., 4 exabytes).  So, why
even allow them [1]?  It seems to me that these permissible options
just create a temptation to create broken implementations.

Thanks,

:) Neal

  [1] https://mailarchive.ietf.org/arch/msg/openpgp/n1-e2shFQMqLphGY2VUvxdr9T40

---

From: Derek Atkins <[email-masked]>
Date: Fri, 12 Apr 2019 09:37:47 -0400

[ sorry for coming at this late -- I've been traveling ]

"brian m. carlson" <[email-masked]> writes:

> On Thu, Mar 28, 2019 at 07:19:39PM -0700, Jon Callas wrote:
>> Like some interim replies, particularly Bart Butler, I thought we
>> had a rough consensus and that it was approximately:
>> 
>> * MUST support 16KB chunks.
>> * SHOULD support 256K chunks, as these are common (Protonmail).
>> * MAY support larger up to the present very large size.
>> * MAY reject or error out on chunks larger than 16KB, but repeating
>> ourselves, SHOULD support 256K.
>
> I think this is fine. It covers what's requires for interoperability, it
> makes prudent suggestions, and it allows people leeway if they know
> they're the only one consuming their data.

+1.  I support this proposal.

-derek

---

From: "Derek Atkins" <[email-masked]>
Date: Fri, 12 Apr 2019 10:10:21 -0400

Hi,

>> Marcus Brinkmann <marcus.brinkmann=[email-masked]>
[snip]
> just create a temptation to create broken implementations.

Not necessarily; it could buffer to disk (just like PGP2 did, and even
PGP3/5 in certain circumstances).  In my mind, buffering to disk is not
the same as releasing the plaintext.  So it COULD still be conformant,
even with 4 exabytes.

-derek

---

From: Jon Callas <[email-masked]>
Date: Mon, 15 Apr 2019 17:00:58 -0700

> On Mar 30, 2019, at 9:11 PM, Bart Butler <bartbutler=[email-masked]> wrote:
> 

[...]

> I agree. I would say my point is that with sufficiently small chunks, the user/decrypter can choose what kind of failure behavior is appropriate. Large chunks robs the decrypter of that.

We are mostly in violent agreement, I do believe. I feel like I'm saying something like "a quarter is a coin with George Washington on one side and an eagle on the other" and you're saying "a quarter is a coin with an eagle on one side and George Washington on the other." We're talking about the same coin, with a slightly different point of view.

I wouldn't use a term like "rob" because that assigns value to the condition. I think there are places where rejection matters and is a Good Thing. I think there places where it is not a good thing and is even a Bad Thing. That's why I was using terms like "strict semantics" and a lot of conditionals.

I don't want to bury my lede any deeper than this. What I'm saying is:

* The more you want strict AEAD semantics of no-release, the fewer chunks you want.
* It seems to me that the people who most believe in strict AEAD release are also the ones who are arguing for smaller packets. These seem to be in opposition to each other. I've been confused through this discussion because the rationales seem in opposition and confused. I don't get it, and I want to understand; you all are smart people whom I respect, so if I'm confused, maybe I'm not getting something.

We might differ in that I have a nuanced opinion about AEAD rejection. I think that there are places where it matters, and places where you don't. For example, in networking, particularly the parts of the network stack where you can easily get a forged packet. You want to reject that packet as early as possible. Moreover, these places are always using very small packets. (I'm going to wave my hand and say that under a megabyte is "very small" for these purposes.)

But in archival storage, you *don't* want to reject something because there's a media error, you want to recover as much as possible. You might even be required to do so by law. I have real-world anecdotes if you want to hear them.

On a network, rejection is a good thing. You reply a NAK to the sender and they retransmit. In archival storage, there's no retransmitting on a media error. That's the case where it's a Bad Thing, and in fact, it might even be better to use CFB mode and an MDC than AEAD. It also might not, and much depends on which AEAD mode one used.

Nonetheless, if you believe in strict semantics, you also likely want the fewest number of chunks. If there is more than one chunk, you have to stage the output, you have to process *everything* (unless you're going to say that the timing side-channel is not important)

Sometimes this is not possible. Ironically, the place where it's most possible is in storage, where it's the least needed. In online protocols, 

> I guess this is my fundamental question. You can force no-release semantics at the application level for any chunk size scheme, right?

Yes, you can, provided that there's a way to report that back, and your caller checks the return value. 

I suppose this really means no, you can't force it, because the library writer can't force the application code to check the error return.

I have heard that some issues that we're Not Going To Talk About had among the issues improper checking GnuPG's report of an MDC failure was an issue in at least one place.

> I apologize, I was not trying to create a strawman here, but I am completely at a loss for what the benefit of large chunks is.

From a standpoint of debate technique, coming up with a strawman makes your whole side of it weaker because attacking a strawman is attacking a strawman. It makes it look like you don't understand, when you actually have a different issue. I think it has added to the confusion I have been suffering from. The chunk size question is about adjusting security parameters, and thus when you say, "it won't help performance" I can't help but think that we're not discussing the same thing at all, as I'm talking security, and you're talking performance.

Good to put that to bed. Back to the chunk size debate.

I don't know the specific benefits, either. I heard people asking for it, and I'm defending the idea for them.

I believe that an underlying difference between your thinking and mine is that you're looking at this as an application writer, and I'm looking at it like a protocol / API that has many clients, some of whom (and the largest ones) aren't written yet.

Moreover, there are a lot of people who use OpenPGP for a lot of things that we don't know about. As Peter Gutmann pointed out there are a lot of EDI systems, back ends of financial systems, and so on that internally use OpenPGP implementations. They're not here. I'm trying to watch out for them. 

There are also people around who want to do something and for a lot of reasons find it difficult to speak up. I'm not editor any more, Werner is and I have every faith in him. Sometimes, though, old habits die hard.

I tend to see the AEAD packet format as being a successor to the existing streaming, indefinite length things. That allows chunking up to 2^30 and while absurdly large, it has never been an issue. 

In my head, I think why not allow up to that, since it would preserve anyone's weird thing?

On the other side, implementers need guidance. Today, the guidance is folklore with all the issues that go with it. It's better not to have folklore. But, if we basically said, "do what you're doing today" then we'd be looking at 8K chunks, as that's what GnuPG does today.

The clauses I suggested about MAY support larger / MAY give larger the finger seemed to be a compromise that would work because it gives you the guidance you need; it lets whoever these people are the ability to do what they want; and lastly should there be a consensus that it needs to be larger in the real world, a consensus of implementers can change it without a new document. It seemed to me that everyone wins.

Yet I thought I perceived that you not only wanted to win, but you wanted to salt the earth in the other people's territory. Fixing an upper bound on memory has a long history of Famous Last Words going back to the old clichéd "640K is more than enough for anyone." The gods punish hubris.

Okay -- let's sort all this out. I really think we are ALMOST done here.

Here's what I stated before.

> I'm OK with eliminating (2) and just using the MAY part to take care of any legacy 256K messages OpenPGP.js users might have. As I said, we don't have any of these messages in production yet and I'd err on the side of a cleaner spec.

Me too. I think saying 256K is fine. I have an intuition it ought to be at least as large as the largest Jumbo Frame, and that's 9K so round to 16K. Let me restate the proposal.

(1) MUST support up to <chunk-size> chunks.
(3) MAY support larger up to the present very large size.
(4) MAY reject or error out on chunks larger than <chunk-size>

And it seems that 256K is the proposal for <chunk-size>. Are we agreed on all that?

> I just really want to understand the benefit of large chunks for security and right now I clearly do not.

If you believe that no-release is a Good Thing, then you want fewer chunks, ideally only 1 chunk. That's it. That's the ONLY reason.

I believe that no-release can be a Good Thing, but rarely is for OpenPGP's primary use case. As I said in my other missive, I don't think that it's even possible in the general case. Networking packets, yes -- both possible and desirable. Files, no -- neither possible nor desirable.

	Jon

---

From: "Derek Atkins" <[email-masked]>
Date: Mon, 15 Apr 2019 20:16:02 -0400

Jon,

On Mon, April 15, 2019 8:00 pm, Jon Callas wrote:
[snip]

> Me too. I think saying 256K is fine. I have an intuition it ought to be at
> least as large as the largest Jumbo Frame, and that's 9K so round to 16K.
> Let me restate the proposal.
>

I would rather 256K not be the One Required Size.  It's larger than the
RAM size on some of my systems.  I would still prefer 16K (or even 8K).

-derek

---

From: Jon Callas <[email-masked]>
Date: Mon, 15 Apr 2019 18:23:56 -0700

> On Apr 15, 2019, at 5:16 PM, Derek Atkins <[email-masked]> wrote:
> 
> Jon,
> 

Bart, Neal? What do you guys think?

I spent part of the afternoon verifying GnuPG's chunk size. I started off using --list-packets to get the size of something, and then moved to using pgpdump. gpg --list-packets reports the combined length of all the partials as if it was one big thing, but pgpdump shows different. I found that out after I sent my last thing. I thought it was using single-chunk lengths even with some moderately large files. Only when it was showing single size for something over a gigabyte did I go grab pgpdump. 

Playing around, GnuPG aggressively uses 8K chunks, and even when I encrypted something below 8K in length, it used a 4K chunk, a 2K chunk, and then tied off the rest. 

Why not pick 8K?

Arguments in favor:

* It's what is done now.
* As noted in Bart and Neal's comments, there isn't going to be a huge performance difference.
* It works in small implementations, as Derek says.

Arguments against -- 
* I have no idea. Someone who isn't me should say something even like "I want it."

Anyone else?

	Jon

---

From: Heiko Stamer <[email-masked]>
Date: Tue, 16 Apr 2019 19:39:17 +0200

On Apr 16, 2019, at 03:23, Jon Callas wrote:

> I spent part of the afternoon verifying GnuPG's chunk size.

Thank you for analysis of GnuPG's behavior. DKGPG uses a fixed
size of 64K for encryption now. However, I will change to 8K,
if that is the desired and agreed default size.

Bests,
Heiko.

---

From: Bart Butler <[email-masked]>
Date: Wed, 17 Apr 2019 00:05:48 +0000

Hi Jon, 

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Monday, April 15, 2019 5:00 PM, Jon Callas <[email-masked]> wrote:

> > On Mar 30, 2019, at 9:11 PM, Bart Butler bartbutler=[email-masked] wrote:
> 

> I wouldn't use a term like "rob" because that assigns value to the condition. I think there are places where rejection matters and is a Good Thing. I think there places where it is not a good thing…

I said 'rob' because I think fundamentally that the release semantics should be something that is decided by the decrypter, not the encrypter, as only the decrypter knows what kind of release semantics are safe or not. For example, I have a 32 MB PGP/MIME message. I want to show a preview in my email client. If we use 8K chunks, I can read the first chunk, know that it hasn't been messed with, and display it safely. If the spec allows a 32 MB chunk, as an application developer I have some choices:

1. I can load the entire 32 MB and be really slow/bandwidth intensive
2. I can not show a preview for this message
3. I can ignore release semantics and do it anyway, risking the Problem That Shall Not Be Named

All of the options are terrible for a UX perspective. Meanwhile, if the chunk size is capped, this makes it easy, and if I, as an application developer, need strict release semantics for the entire file/message, I can do that too.

Now, with your proposal, the other implementations and I can come to some agreement that hey, we just aren't going to allow chunks meaningfully higher than the cap, what you call "normative" agreement. That's fine, but I'm worried that these norms don't tend to be well-documented (I'm not sure the MAY in the RFC will be sufficient), and someone somewhere is going to write an implementation at some point which exclusively uses big chunks. When they do, our implementations will reject them, and then their users will complain to the app developers, who will in turn complain to the implementers.

I'm certainly not so arrogant to assume I can anticipate all future needs here. But I think it's telling that we can come up with several negative consequences of allowing the large chunks and the only benefit is something that can be achieved at the application layer (or as an option at the implementation layer even) if desired anyway.

I also think that forcing no-release semantics via packet structure is misguided because app developers/implementors are likely to ignore if it becomes too annoying. That is, I anticipate some implementers just allowing some kind of unsafe mode that releases plaintext early with no integrity checks if this comes up (essentially streaming not along AEAD chunk boundaries), and I'm in general uncomfortable with choosing to build a feature whose failure case is "massive security hole", not to mention one that we've seen before with the Problem That Shall Not Be Named. Do we want to allow people to create messages which *cannot* be safely streamed when we have the choice not to do this with zero functional downside? Strict release semantics can always be enforced at the implementation or application level.

> -   The more you want strict AEAD semantics of no-release, the fewer chunks you want.
> -   It seems to me that the people who most believe in strict AEAD release are also the ones who are arguing for smaller packets. These seem to be in opposition to each other. I've been confused through this discussion because the rationales seem in opposition and confused. I don't get it, and I want to understand; you all are smart people whom I respect, so if I'm confused, maybe I'm not getting something.

This feeling is completely mutual. I respect everyone in this discussion and know that all of you are smart people. I will try to rephrase what I think is the fundamental question here, and it's not what release semantics should be--those can be enforced in lots of places, and as you said, vary by use case, which is very compatible with my views. I think the fundamental question here is this: 

*Should we allow creation of valid messages which cannot be streamed and attempt to force strict no-release at the protocol layer?*

I think, in the absence of a compelling reason to, the answer to this is a pretty clear no.

>     But in archival storage, you don't want to reject something because there's a media error, you want to recover as much as possible. You might even be required to do so by law. I have real-world anecdotes if you want to hear them.
>     

>     Nonetheless, if you believe in strict semantics, you also likely want the fewest number of chunks. If there is more than one chunk, you have to stage the output, you have to process everything (unless you're going to say that the timing side-channel is not important)

Why do you have to stage the output in the multi-chunk case? The only difference in the multi-chunk case is that I'd check AEAD tags multiple times instead of just at the end. There's no reason why I'd have to do anything with the output differently than a single chunk if I embrace strict no-release. I could buffer it the exact same way I was buffering the single chunk and the application/consumer doesn't have to know there is any difference.

Fundamentally, multi-chunk just gives you options. There is nothing stopping an implementation from doing strict no-release. 

> Yes, you can, provided that there's a way to report that back, and your caller checks the return value.

You (as an implementation) could just not return the plaintext until the entire message was read. There's nothing stopping implementations from having a strict no-release mode.

> I suppose this really means no, you can't force it, because the library writer can't force the application code to check the error return.

Well, the library can always just not return the plaintext if we don't think it's safe. I just don't think it's the encrypter's business to be deciding what is safe or not for the decrypter.

> I have heard that some issues that we're Not Going To Talk About had among the issues improper checking GnuPG's report of an MDC failure was an issue in at least one place.

Sure, but this could have been configured as a hard failure. The apps didn't configure it as a hard failure because that would have collided with UX/application concerns, and I fear that that collision will occur again if we allow it to, with likely the same result.

> There are also people around who want to do something and for a lot of reasons find it difficult to speak up. I'm not editor any more, Werner is and I have every faith in him. Sometimes, though, ol…

I'm sympathetic to all of this, and I don't want to put anyone on the spot. It would be really great if anyone who has a use case for large chunks speaks up though, either through this thread or privately to me, Jon, or anyone else they feel comfortable speaking with, because I do not want anyone's voice to not be heard, and if there is a use case for large chunks I do want to hear about it before this decision is finalized.

> I tend to see the AEAD packet format as being a successor to the existing streaming, indefinite length things. That allows chunking up to 2^30 and while absurdly large, it has never been an issue.

Well, except that streaming this old stuff is unsafe if ciphertext modification is a threat.

> The clauses I suggested about MAY support larger / MAY give larger the finger seemed to be a compromise that would work because it gives you the guidance you need; it lets whoever these people are…

For the record, I'm pretty much OK with this, I just think it's opening us up to future problems that it would be best to avoid.

> Yet I thought I perceived that you not only wanted to win, but you wanted to salt the earth in the other people's territory. Fixing an upper bound on memory has a long history of Famous Last Words…

I'm sorry I gave that impression or was overly strident. I consider this a rare opportunity to fix something before it becomes a problem rather than afterward with a bunch of legacy baggage in tow. I have no interest in "winning" this argument for it's own sake--I would be happy to get a counter-argument for large chunks that made me think "yes, there is a use case and that's why we want to risk having these future problems".

> And it seems that 256K is the proposal for <chunk-size>. Are we agreed on all that?

As some respondents would like 8K or 16K, I'm fine with doing that instead of 256K. I would like to check with the maintainers of our libraries to find out if there's any reason I'm ignoring that would favor one or the other before committing though.

> If you believe that no-release is a Good Thing, then you want fewer chunks, ideally only 1 chunk. That's it. That's the ONLY reason.

I think I discussed this to death above so I won't add to the word count here.

-Bart

---

From: Bart Butler <[email-masked]>
Date: Thu, 18 Apr 2019 17:28:31 +0000

Hi all,

Jon and I got on the phone today and discussed the AEAD chunk situation a little more. The compromise we reached is:

(1) MUST support chunks up to and including 8KiB.
(2) SHOULD NOT emit chunks larger than 8KiB.
(3) SHOULD reject chunks larger than 8KiB.

Jon likes this because it doesn't fully shut the door on experimentation. I like this because it strengthens the norm and puts the burden of justification for violating the norm on the person asking for huge chunks rather than the implementer. I expect (though please correct me if not) that Derek, Neil, and others working on embedded or otherwise constrained system will like this because the normative limit is low enough for most embedded/constrained systems to do streaming when incremental no-release semantics are desired. And I hope Werner likes this because GnuPG is already doing 8KiB chunks, so the work involved in changing GnuPG's implementation should be minimal :)

Does anyone have any objections to or comments regarding this language?

Thanks,

Bart

---

From: "Derek Atkins" <[email-masked]>
Date: Thu, 18 Apr 2019 13:41:22 -0400

No objections from me!
Thanks Bart!
-derek

On Thu, April 18, 2019 1:28 pm, Bart Butler wrote:
> Hi all,
>
> Jon and I got on the phone today and discussed the AEAD chunk situation a
> little more. The compromise we reached is:
>
> Jon likes this because it doesn't fully shut the door on experimentation.
> I like this because it strengthens the norm and puts the burden of
> justification for violating the norm on the person asking for huge chunks
> rather than the implementer. I expect (though please correct me if not)
> that Derek, Neil, and others working on embedded or otherwise constrained
> system will like this because the normative limit is low enough for most
> embedded/constrained systems to do streaming when incremental no-release
> semantics are desired. And I hope Werner likes this because GnuPG is
> already doing 8KiB chunks, so the work involved in changing GnuPG's
> implementation should be minimal :)
>
> Bart_______________________________________________
> openpgp mailing list
> [email-masked]

---

From: Werner Koch <[email-masked]>
Date: Tue, 23 Apr 2019 10:11:16 +0200

On Thu, 18 Apr 2019 17:28, bartbutler=[email-masked]
said:

> hope Werner likes this because GnuPG is already doing 8KiB chunks, so

I am not sure about the context.  Are you talking about the partial
length encoding or about the AEAD chunk size, a modification of AEAD to
allow detection of transmission errors before the end of the data?

GnuPG 2.3 creates AEAD chunks not larger than 128 MiB.  This can be
changed with an option down to 64 bytes.  However such a values is only
useful for regression testing as it slows down the performance.  I may
consider to change the default to 1 MiB but not lower.

Let me repeat that the whole discussion on the size of the AEAD chunks
is mostly off topic because the chunks are _only_ here to allow
detection of transmission errors before Gigabytes of data have been
processes.  This was the reason why I suggested to Brian the addition of
a chunking mode for AEAD.

Whether the received data is authentic can only be asserted by checking
the signature and that can obviously only be done after all AEAD chunks
have been decrypted.

Those implementations wanting to show a preview can do so regardless of
any AEAD validation etc.  They should just make clear to the user that
this is an unauthenticated and possible corrupted preview of the data.

For all other purposes I propose to use a different protocol on top of
OpenPGP a (e.g MIME) and not to overload OpenPGP with unneeded stuff.
Or well, start from scratch and use a different name for it.

Salam-Shalom,

   Werner

-- 
Die Gedanken sind frei.  Ausnahmen regelt ein Bundesgesetz.

---

From: Benjamin Kaduk <[email-masked]>
Date: Sat, 27 Apr 2019 22:32:07 -0500

Hi Werner,

> is mostly off topic because the chunks are _only_ here to allow

I'm willing to believe that the reason you state is why it was originally
added.  But the volume of discussion we've had already makes me pretty
skeptical that it remains the only reason people might want to have AEAD
chunks, so your argument would be much stronger if you made some reference
to the technical subjects that have been raised and why you believe they
are not relevant.

Thanks,

Ben