Subject: Re: [openpgp] AEAD Chunk Size

From: "Neal H. Walfield" <[email-masked]>
Date: Fri, 01 Mar 2019 09:45:22 +0100

At Thu, 28 Feb 2019 10:39:17 -0800,
Jon Callas wrote:
> > On Feb 28, 2019, at 12:51 AM, Neal H. Walfield <[email-masked]> wrote:
> >> So I think that a SHOULD is the right way to put it. I care less
> >> about what the SHOULD limit should be, but a small hard limit sounds
> >> like a bad idea.
> > 
> > Do you still think a hard upper limit is a bad idea?
> 
> No, I don’t think a hard upper limit is a bad idea. In the text you quote above, I said a *small* hard upper limit. And no, I don’t know what small means. I think 16K is small. 256K might be small. AES-GCM has issues about 64G. One might argue that’s a reasonable large upper limit. Me, I think anything in the few megabytes to a gigabyte-ish is just fine.
> 
> Let me rewind a bit and get to my real point which I don’t think I’m making well.
> 
> When one creates a standard, one needs to be careful with
> parameters, particularly the MUSTs. Seemingly sensible things can
> have downstream effects that convince people to use some other
> protocol. Worse, someone’s angry blog post about something can
> quickly go into “not even wrong” territory and embed itself into
> folklore and you can’t get it out. There are plenty of bad ideas
> that someone else has a really reasonable use case for.

There are always going to be haters.

> ...

> This has never caused a problem that I’m aware of. I’m sure it caused a problem that none of us are aware of, and the implementors just solved it on their own. It is this experience that has me wondering about what the restrictions out to be.
> 
> The best way to deal with it in a standard is to have non-normative guidance. It is non-normative guidance that I was suggesting. Let me write an example bit of non-normative guidance below:
>  ...

I think that security concerns should be our first priority.  And, any
flexibility increases our bug potential.  As such, I'm not convinced
that we shouldn't use a fixed chunk size.  If in a few years, if we
decide that the size is wrong, then we can just define a new mode,
which uses a different chunk size.  In my opinion, it should be
impossible represent something that we want to disallow.

If we are really worried about the haters, then let's just deflect the
criticism: let's use 16 kiB chunks like TLS, as Marcus previously
pointed out:

  https://mailarchive.ietf.org/arch/msg/openpgp/t79iRZ80KHuVTEyVVLAoCLl4Rwc

From RFC 8446:

  5.2.  Record Payload Protection

    length:  The length (in bytes) of the following
      TLSCiphertext.encrypted_record, which is the sum of the lengths of
      the content and the padding, plus one for the inner content type,
      plus any expansion added by the AEAD algorithm.  The length
      MUST NOT exceed 2^14 + 256 bytes.  An endpoint that receives a
      record that exceeds this length MUST terminate the connection with
      a "record_overflow" alert.

  https://tools.ietf.org/html/rfc8446#section-5.2

(A record the approximate equivalent of a chunk.)

> The Partial Body Lengths that OpePGP has had from the beginning have no restrictions on them. There’s non-normative guidance that pretty much says you shouldn’t even use them, but there’s no restriction on size.

I just reread and I don't think it suggests that Partial Body Lengths
should not be used.

  https://tools.ietf.org/html/rfc4880#section-4.2.2.4

Can you point me to this guidance?  This is the first time I hear that
chunking is considered to be a bad idea.

Thanks,

:) Neal

---

From: "Neal H. Walfield" <[email-masked]>
Date: Fri, 01 Mar 2019 09:58:28 +0100

At Thu, 28 Feb 2019 19:11:06 +0000,
Bart Butler wrote:
> "Re. Neal's request, I didn't have the numbers anymore, so I unscientifically created some new ones
> 
> It looks like either Chrome or we have actually fixed the performance issues there with c=8 since the last time I tested it, as for most message sizes the performance is the same as c=12 or even a bit faster, except 128KB-256KB, which fits in one chunk with c=12 and there c=8 is about 1.1x - 1.5x slower.
> 
> In Firefox, for >=64KB messages, c=8 is still about 1.5x - 2.5x slower than c=12, however, it looks like most of the overhead of smaller chunks comes from the streams polyfill, not the web crypto API. So that should probably be possible for us to fix."

Thanks for providing these numbers!  It's particularly interesting to
hear how the performance has changed over time.

> I see your logic on 16 kiB but in particular for file encryption I think there's some value to being able to go bigger in the future (L2 cache sizes have changed in the last 20 years as well and will probably change in the next two decades), and to keep the size byte to be able to configure this rather than have an entirely new packet. I think a SHOULD for 16 kiB though is totally appropriate now though for the reasons you mentioned.

Although I have a slight preference for 16 kiB due to performance
concerns, my first priority is security.  I'd like, at the minimum, a
hard ceiling, but I'd prefer no chunk size parameter at all.  I
appreciate backwards compatibility, and I'm willing to go with
renaming "chunk size" to "magic value" and fixing that to the C
corresponding to 256 kiB chunks.

---

From: "Neal H. Walfield" <[email-masked]>
Date: Fri, 01 Mar 2019 10:01:47 +0100

At Thu, 28 Feb 2019 19:44:41 +0000,
Bart Butler wrote:
> I can't because I do not know--it's an experimental feature in OpenPGP.js. I do know that some users of the library are using it internally in closed systems, and we'd prefer not to break decryption for their existing messages but also prefer not to keep supporting an obsolete draft. You could argue that they shouldn't have used an experimental feature for production but given how overdue AEAD is for PGP I find it difficult to blame them myself.
> 
> ProtonMail doesn't use V5 keys yet at all, as we exist within the federated email ecosystem and it would break compatibility. So this is not coming from us personally, just on behalf of the OpenPGP.js community in general.

For continuity, in:

  Message-ID: <[email-masked]>
  https://mailarchive.ietf.org/arch/msg/openpgp/JukTMAMY-RUoHsHxVxyZNht96R4

I proposed changing the chunk size to a magic value whose value is the
current C value for 256 kiB chunks, and to fix the chunk size to 256
kiB.

---

From: Tobias Mueller <[email-masked]>
Date: Sun, 03 Mar 2019 19:36:08 +0100

Hi,

On Fri, 2019-03-01 at 09:45 +0100, Neal H. Walfield wrote:
> that we shouldn't use a fixed chunk size.

Note that introducing "chunks" diverts from authenticated encryption and
increases the bug potential.  That is, an implementation may more easily
release plaintext although the ciphertext has been modified.
The AE mode is OpenPGP's chance to become a protocol which enjoys the
strong security guarantees of AE and catch up with S/MIME.

By fixing a "chunk size" you take away the ability to benefit from AE
for messages bigger than that size.
Implementations could easily collect all chunks and only release the
plaintext once all chunks check out successfully. But that could go
wrong. And depending on implementations to get things right and clients
to use those implementations correctly is exactly what enabled Efail to
become an issue. I think it'd be much nicer if the protocol already
ensures that my emails do indeed enjoy protection against modification
rather than me having to rely too much on clients getting it right.

Having said that, I understand the desire for fixing a chunk size to
reduce complexity for implementers.  My desire as a user is to have a
strong and resilient protocol.  As such I prefer producing messages that
enjoy strong protection against modification.  That includes my emails
or backups larger than 16kB, 256kB, or whatever size you come up with.

Is there another way to do real AE?

Cheers,
  Tobi

---

From: Sebastian Schinzel <[email-masked]>
Date: Mon, 4 Mar 2019 09:49:54 +0100

> or backups larger than 16kB, 256kB, or whatever size you come up with.

Chunking breaks plaintexts of arbitrary size into many smaller "chunks"
and adds an authentication tag to each chunk. The advantage of smaller
chunks is that the plaintext can be cached until the chunk's auth tag is
validated. That's to guarantee that no unauthenticated plaintext is
released. (Leaving truncation aside.)

Your reasoning regarding proper AE is correct, but you are drawing the
wrong conclusions. You want small chunks to do proper AE! This implies
no limit to the overall size of the plaintext.

I also don't see any reason to keep the variable chunk size. We should
fix it to something between 16kB and 64kB.

Best,
Sebastian

---

From: "Neal H. Walfield" <[email-masked]>
Date: Thu, 07 Mar 2019 17:11:18 +0100

Hi Werner,

Thanks for following up.

At Fri, 01 Mar 2019 15:50:15 +0100,
Werner Koch wrote:
> On Wed, 27 Feb 2019 11:51, [email-masked] said:
> > Consequently, I propose not only imposing a reasonable ceiling on the
> > chunk size that even small embedded devices with a cortex M0 could
> > handle, but to simply fix the parameter to 16 KiB.  It's not clear to
> 
> Without sufficient storage a smaller chunk size does not help you in any
> way.  You can still run a truncation attack and by that time the
> preceding chunks have already been processed in some way because, well,
> there was no way to store the entire message.  Without the final chunk
> you have an incomplete and thus unauthenticated message because the
> sender authenticated the entire message and not certain parts of it.

It's true that a smaller chunk size doesn't provide protection against
truncation attacks.  And, I agree with you that for the receiver to
protect itself against truncation attacks, it can only release the
plaintext after it has authenticated the whole message, which it can
only do if it buffers the whole message.  But, I strongly disagree
that a smaller chunk size does not help us in any way: even in the
face of truncation attacks, AE dramatically increases the security of
streaming.

AE provides ciphertext integrity [1].  Ciphertext integrity means that
the ciphertext has been authenticated.  That is, used correctly, AE
would stop an attack that modifies the ciphertext, like EFAIL.

  [1] See for instance Chapter 9 of A Graduate Course in Applied
  Cryptography by Dan Boneh and Victor Shoup for an explanation:

  https://crypto.stanford.edu/~dabo/cryptobook/BonehShoup_0_4.pdf

Since ciphertext integrity is guaranteed at the chunk granularity, the
ability to buffer a single chunk is sufficient to ensure ciphertext
integrity.

Now, if we admit steaming, and historically, OpenPGP has been
streaming friendly, then chunks must be small enough to fit in the
receiver's available buffer space.  This is essential, because the
decryption function must reject chunks that it can't authenticate [2];
releasing unauthenticated plaintext break's AE's security guarantees.

  [2] RFC 5116, Section 2.2: "[The authenticated decryption operation]
  has only a single output, either a plaintext value P or a special
  symbol FAIL that indicates that the inputs are not authentic."

  https://tools.ietf.org/html/rfc5116#section-2.2

On the other hand, if if we prohibit streaming, well, no, someone is
going to do it anyway.  So, we should just accommodate it as best we
can.

Given streaming, for me, the logical consequence is that we need to
somehow figure out an appropriate chunk size (or sizes).  Since the
sender can rarely make a sensible choice, and the chunk size doesn't
change AEAD's security properties, I've concluded that the most
sensible choice maximizes the number of devices that can decrypt
messages, which means making the chunk size as small as possible
without adversely impacting performance, e.g., 16 kiB or 256 kiB.

(Note: using a small chunk size doesn't mean that an implementation
has to release a chunk as soon as it is authenticated.  An
implementation can frustrate truncation attacks by ensuring that at
least, say, 25 MB of data remain buffered as long as it has not
encountered the end of the file.  FWIW, this is what Sequoia does.)

Let's assume that there are cases where we don't care about ciphertext
integrity, and a larger chunk size is better for some reason.  To
accommodate these cases, we could allow implementations to emit
unauthenticated data if a large chunk size is used.  We can even fix
the threshold in the specification so that senders can be sure that
messages that need ciphertext integrity are processed in a more secure
manner.

Unfortunately, some basic testing using a hacked version of Sequoia
suggests that an attacker can just take a message with a chunk size <
X and change the chunk size to be larger than X, and Sequoia will
happily emit the first chunk, because the chunk size parameter is only
checked after the first chunk is authenticated!  Ouch.

But, it's worse.  We can further defeat any opportunistic buffering
without appending a lot of garbage by wrapping the AEAD packet in a
Compression packet that is jammed at the end with a decompression
bomb.  Thus, the size of the message is only marginally larger than
the original message!

For me, this means that if we want AEAD's security properties, and I
think we do, we can't allow implementations to emit unauthenticated
plaintext.  Although we can write in the specification that
implementations MUST return "EBUYABIGGERCOMPUTER" if the chunk size is
too large, implementors will almost certainly ignore this, because
security must not get in the way of a user getting her job done.  In
fact, there is already an OpenPGP implementation that does this.  The
better way, I think, is to make it hard to do it wrong.  And that
means using a small, fixed chunk size.

If my argumentation and authority are too weak, perhaps Adam Langley
can convince you.  In his 2015 talk, he addresses exactly this problem
and why it is essential to support chunking even though it enables
truncation attacks.  The relevant discussion starts at 18:20, but the
most important bit (quoted below) starts at 22:43:

  Let's say the attacker does cut here.  They can only cut between
  chunks, because you'll figure that you.  But, they cut here.  You
  will know that it is not the correct end, because you never saw this
  tag on the AD.  But, if you are piping it straight into bash, the
  attacker still managed to truncated things.  The only way around
  this is to buffer the whole thing to disk before you release any
  plain text.  And, now you might think, "well, if I'm doing that, why
  did I put all this effort into chunking?"

  And, I'd like to do a quick aside here on what I call Adam's rule of
  weak crypto.  Another common thing that people come up with is---and
  suggest---is that they've done a careful analysis and they only need
  n-bit security where n is 80, or, if it is particularly weak, 40,
  and they have a very carefully argued plan about why what they are
  protecting is almost meaningless.  Like, they almost wouldn't bother
  with cryptography and they've got very tight bandwidth overheads,
  like bluetooth LE is a very common one.  And, so they are going to
  do 40-bit security or something similar.  And, they're correct.
  Their argument is perfectly reasonable.

  They still shouldn't do it.  They reason is that the Internet is
  built out of whatever was laying around.  It's like children with
  lego.  Things are never designed, you'll carefully reasoned argument
  about why this is completely correct will be discarded almost
  immediately.  And in the future, someone you've never met, and may
  never met, and may never talk to, will pick up your think and go:
  "oh, it's cryptography, right, this is fine, I'll build on it".  And
  then 10 years down the road, we find we have some massive
  societal-wide flaw, in some very important system, because they
  built it out of your weak system, never understanding the basis.

  So, you should still do [AEAD chunking], because if your argument is
  that we need to buffer everything, because of truncation attacks, so
  we're going to buffer everything, so we're going to have one huge
  chunk, because it doesn't make any difference, someone will come
  along in the future, they will discard your very carefully reasoned
  argument, they will go: "oh, I'm running on an embedded system, I
  don't have enough disk space, my messages are too large", and they
  will stream it, and it is still your fault, even though they screwed
  up.  So, for the people who are going to stream it, you should do as
  well as you can, which is [AEAD chunking].

  Yahoo Trust Unconference: TLS, Adam Langley (2015)
  https://finance.yahoo.com/video/yahoo-trust-unconference-tls-adam-223046696.html?guccounter=1

> Let me repeat it again: The chunking was introduced for just one
> purpose: To be able to detect rare transmission errors earlier than at
> the end of the message.

I agree that AEAD helps detect transmission errors earlier.  But, AEAD
does so much more than that.  In particular, it prevents attacks like
EFAIL.  It seems to me that it's worth adapting to this new threat.

Even if everyone accepts that the AEAD packet should only be used for
detecting transmission errors early, then in the very least, we need
to rename it.  AEAD has a specific technical meaning, and as currently
realized, 4880bis makes it very hard to meet those expectations.  This
will come back and bite us in the foot.

> For large pipes large chunks are a sensible
> choice.

I don't understand this argument.  The buffer size ought to be
independent of the chunk size.  AEAD doesn't say that as soon as a
chunk is authenticated it must be released; if you want, your
implementation can buffer 128 megabytes of data even if AEAD uses 16
kilobyte chunks.

I'd like to summarize your concerns as I understand them.  Please
correct me, add anything that you haven't yet mentioned, or fill in
something that I left out.

  1. Because AEAD doesn't protect against truncation attacks, streaming
     is as insecure as it ever was.

  2. The sole purpose of AEAD chunking is to detect transmission
     errors early.

  3. Small chunk sizes are bad for IPC.

My reactions:

  1. Although AEAD doesn't protect against truncation attacks, it does
     increase protection, because it would protect against ciphertext
     modifications as done in EFAIL.

  2. Perhaps when AEAD was introduced that was the sole reason, but
     with a small tweak, AEAD can be used to increase security in
     other settings.

  3. You don't have to release chunks as soon as they are
     authenticated, so you can make your buffer really large, if you
     like.

If you are adverse to changing packet 20, then let's just mark 20 as
reserved and create a new packet, 21, which changes the semantics.

Thanks,

Neal

---

From: "Neal H. Walfield" <[email-masked]>
Date: Thu, 07 Mar 2019 18:09:19 +0100

Hi Tobi,

On Sun, 03 Mar 2019 19:36:08 +0100,
> rather than me having to rely too much on clients getting it right.

You're afraid of bugs.  So am I.  Dynamically resizing buffers is
error prone.  Even computing AEAD's chunk size is hard:

  #include <stdio.h>

  int chunk_size(int c) {
    return 2 << (c + 6);
  }

  int
  main(int argc, char *argv[]) {
    printf("%d\n", chunk_size(32));
  }

  $ gcc -Wall a.c
  $ ./a.out
  128

(Note that gcc doesn't even produce a warning!)

But, that's not all.  If there is the possibility of failing due to a
lack of buffer space, implementors won't bother.  And they won't
bother for a very good reason: users want to get work done.  If a
security policy prevents them from getting their work done, they will
disable it.  Or, an implementation will ignore it.  And, because it is
only a problem if there is an attack, users will be thankful for the
more "powerful" solution.

The secure implementation must be easy for both developers and users.
And the secure implementation is a small, fixed sized buffer, even if
that means the client may have to worry about truncation.

> strong and resilient protocol.

Perhaps you're not a typical user :D.

> Is there another way to do real AE?

Chunked AE is still real AE: it still has ciphertext integrity; it is
just susceptible to truncation attacks.  That's not to say that
truncation attacks are not a serious issue, but I'd rather have
truncation attacks than truncation attacks *and* ciphertext modication
attacks.

:) Neal

P.S.  Consider realloc.  It is only marginally easier to write this
wrong code:

  p = realloc(p, 2 * size);
  if (! p) {
      // Oops, p has been leaked!
      return ENOMEM;
  }

than this code:

  void *temp = realloc(p, 2 * size);
  if (! temp) {
      free(p);
      return ENOMEM;
  }
  p = temp;

but I see lots of instances of the former when reading code.

---

From: Werner Koch <[email-masked]>
Date: Fri, 01 Mar 2019 15:50:15 +0100

Hi!

[The mail got stuck somewhere in the IETF mail system - resending]

If you like to adhere in your _implementation_ to some _API_ proposal,
go ahead and use it but the API is not a _protocol_ thing.

Let me repeat it again: The chunking was introduced for just one
purpose: To be able to detect rare transmission errors earlier than at
the end of the message.  For large pipes large chunks are a sensible
choice.

Changing the protocol in a way as suggested by you is not an option
anymore.  We can change recommendations, though.

Salam-Shalom,

   Werner

-- 
Die Gedanken sind frei.  Ausnahmen regelt ein Bundesgesetz.

---

From: Vincent Breitmoser <[email-masked]>
Date: Mon, 11 Mar 2019 12:43:49 +0100

> The chunking was introduced for just one purpose: To be able to detect rare
> transmission errors earlier than at the end of the message.

..really? All this is just to save a few cpu cycles in the rare cases of data
corruption that should have been handled by other layers (filesystem / transport
layer) in the first place? Why even bother?

That said, I don't believe this is the long agreed upon and tiredly repeated
fact you phrased it to be.  Consistenly specced, chunked AE contributes to
better behavior on average in implementations wrt emitting or processing
authenticated data. I fully agree with Adam's argument about this that Neal
quoted.

> anymore.

Oh. Why?

 - V

---

From: Sebastian Schinzel <[email-masked]>
Date: Wed, 13 Mar 2019 07:32:14 +0100

> sender authenticated the entire message and not certain parts of it.

Chosen ciphertext attacks and truncation attacks are two different
attack classes, with different assumptions on the plaintext format and
the necessary attacker capabilities.

Neal's proposal to mandate a small and fixed chunk size can solve
ciphertext malleability for future OpenPGP applications. Waving this
proposal off, just because it won't also solve truncation attacks, does
not make sense.

Best
Sebastian

---

From: Derek Atkins <[email-masked]>
Date: Wed, 13 Mar 2019 09:14:37 -0400

Vincent Breitmoser <[email-masked]> writes:

> ...really? All this is just to save a few cpu cycles in the rare cases of data

No, it is more than that.  Imagine using OpenPGP to encrypt a full
filesystem to tape backup.  You necessarily want to be able to chunk
that as you are saving (and restoring).

-derek
-- 
       Derek Atkins                 617-623-3745
       [email-masked]             www.ihtfp.com
       Computer and Internet Security Consultant

---

From: "Neal H. Walfield" <[email-masked]>
Date: Wed, 13 Mar 2019 14:27:17 +0100

On Wed, 13 Mar 2019 14:14:37 +0100,
> that as you are saving (and restoring).

I don't think Vincent is disputing the validity of Werner's use case
per se.  I think he is saying the marginal utility of that is tiny
(it's "just a performance improvement") relative to ciphertext
integrity (a security property).

---

From: Derek Atkins <[email-masked]>
Date: Thu, 14 Mar 2019 08:36:33 -0400

Neal,

> integrity (a security property).

IMHO it is a bit of both.  Again, looking at this historically (having
written the original streaming code back in ~1995), the idea was to
provide early-notification of cryptographic failure without having to
buffer the whole complete message.  Granted, this was written before
AEAD techniques came into the public eye, but I don't think I would have
done it much differently at the time.

There were at the time systems with limited RAM and/or temp space, but
we still wanted to be able to encrypt & sign LARGE data sets.  Again,
think of a 'tar -czf - / | pgp ... | <network or tape>' as the model we
were using at the time.

If you are on tape, you CAN get "transmission" errors.  Think: Bit Rot.
It's a real thing.  Spinning rust can also have failures, but it's a bit
less common.  Even RAM can have bit errors -- this is why ECC (error
correction, not elliptic curve) RAM is used in servers!

Protecting against transmission error, bit rot, etc is just common
sense!

But the ability to do this in a space that can't "hold" the full data
set is exactly why we have streaming capability.

-derek

---

From: "Neal H. Walfield" <[email-masked]>
Date: Thu, 14 Mar 2019 14:47:14 +0100

On Thu, 14 Mar 2019 13:36:33 +0100,
> Protecting against transmission error, bit rot, etc is just common

I agree that this is extremely useful.

> set is exactly why we have streaming capability.

And, I also want to preserve the ability to stream data.

AEAD catches not only these errors, but also providers ciphertext
integrity.

Are you arguing like Werner that catching transmission errors is
enough and that we shouldn't bother with ciphertext integrity?

---

From: Werner Koch <[email-masked]>
Date: Thu, 14 Mar 2019 15:06:15 +0100

> enough and that we shouldn't bother with ciphertext integrity?

I never said this.  My point was that you are discussing a certain
programming pattern on how to implement AEAD modes and I remarked that
the OpenPGP standard is about a protocol and not an implementation.

BTW, OpenPGP provides ciphertext integrity for more than 15 years.
Experience showed that transmission errors are the major cause for false
MDC triggered alarms.  We want to detect them earlier and not only at
the end of the transmission to support real world use cases.  The move
from CFB+SHA1 to OCB can also be seen in the light of required
performance improvements.

Salam-Shalom,

   Werner

---

From: "Derek Atkins" <[email-masked]>
Date: Thu, 14 Mar 2019 10:17:41 -0400

> enough and that we shouldn't bother with ciphertext integrity?

I don't see how these two are mutually exclusive.

Each chunk can provide protection against both a transmission error and
ciphertext integrity (per chunk).  A simple counter in the chunk header
can protect against splicing attacks, so an attacker could not remove a
middle chunk or otherwise swap chunk orders.  So the only issue is
truncation, where an attacker prevents transmission at the end.

Obviously the receiver/verifier needs to know how to handle the case of a
failure at a chunk or truncation level.  What that means, of course, is up
to the application.

But I don't see how using AEAD per chunk can do anything but help.

-derek

---

From: "Neal H. Walfield" <[email-masked]>
Date: Fri, 15 Mar 2019 12:40:12 +0100

Hi Derek,

Thanks for your comments.

On Thu, 14 Mar 2019 15:17:41 +0100,
> I don't see how these two are mutually exclusive.

These issues aren't mutually exclusive, and I apologize if I gave you
the impression that I thought they were.  I think everyone agrees that
catching transmission errors is desirable.  At least, I do :).  The
sticking point, AFAICT, is that Werner disagrees that the
specification should guarantee cipher text integrity when streaming.

I'm currently convinced that streaming authenticated plaintext is only
possible if we use small chunk sizes.  If we allow large chunk sizes
(e.g., 4 exabytes, which is what the current draft allows), then there
will be cases where an implementation can stream unauthenticated
plaintext, but not authenticated data, and, because it can, it will.
And this is even though pretty much everyone including the IETF (see
RFC 5116 [1]) agrees that AEAD must only emit authenticated plaintext.

  [1] https://tools.ietf.org/html/rfc5116#section-2.2

Implementations will stream unauthenticated plaintext for
interoperability purposes.  Failing to decrypt, because the device
doesn't have enough resources, will prevent users from getting their
work done.  And, if security prevents users from getting their work
done, then the security mechanism will be worked around.

Implementations will stream unauthenticated plaintext to keep their
code simple.  If we allow implementations to stream unauthenticated
plaintext, then streaming authenticated data becomes a special case.
And, no one likes to maintain two code paths.

Will this happen in practice?  IT'S ALREADY HAPPENED.  There is
already an OpenPGP implementation written by one of 4880bis's editors
that emits unauthenticated plaintext.  Now, this puts pressure on
other implementations to follow suit, because users will inevitably
ask why FooGP can decrypt some messages that BarGP can't.

I think the only way to prevent this is to ensure that unauthenticated
streaming provides as few advantages over authenticated streaming as
possible.  This means ensuring that the amount of data to be buffered
is small, and making the implementation work as simple as possible by
using a fixed chunk size.

When I originally thought about this problem, I thought that
introducing a soft threshold could be an option.  But, I've since
concluded that this doesn't help.  Although a sender could use a small
chunk size to ensure that a compliant receiver doesn't emit
unauthenticated data, an attacker can change the chunk size in the
intercepted message, and successfully conduct an EFAIL-style attack.
(See [2].)

  [2] https://mailarchive.ietf.org/arch/msg/openpgp/WvaisWDd9WzoKXL1FyHDeZNtZ2U

Derek, I hope this clarifies my position.  Do you now also agree that
ensuring cipther text integrity when streaming is desirable?  If not,
what concerns do you have?

Thanks!

:) Neal

---

From: "Neal H. Walfield" <[email-masked]>
Date: Fri, 15 Mar 2019 12:48:19 +0100

On Thu, 14 Mar 2019 15:06:15 +0100,
Werner Koch wrote:
> I never said this.

I was referring to this:

  On Fri, 01 Mar 2019 15:50:15 +0100,

> My point was that you are discussing a certain

I agree that the OpenPGP standard is about a protocol.  My suggestions
are about fixing a security hole in the proposed protocol.

> BTW, OpenPGP provides ciphertext integrity for more than 15 years.

But not for streaming, which is what this discussion is about.

> performance improvements.

Performance concerns are secondary to security concerns.  But anyway,
your argument that large chunks are a sensible choice for large pipes
makes no sense, because the protocol doesn't require the
implementation to release chunks as soon as they are authenticated;
they can be buffered.

---

From: Tobias Mueller <[email-masked]>
Date: Sun, 17 Mar 2019 20:33:11 +0100

Hi,

On Wed, 2019-03-13 at 07:32 +0100, Sebastian Schinzel wrote:
> proposal off, just because it won't also solve truncation attacks,
> does not make sense.
I don't understand why you bring up malleability.

As far as I understand Werner, he is concerned with the proposal still
forcing clients to buffer the whole message if the implementation wants
to release authenticated data only. Which is how AE is defined. So
fixing any chunk size does not work.  But I may stand corrected.

Cheers,
  Tobi

---

From: Tobias Mueller <[email-masked]>
Date: Sun, 17 Mar 2019 20:33:59 +0100

Hi,

> And, I also want to preserve the ability to stream data.
but the proposal does not add to that ability but removes the ability to
use one chunk, only.

Cheers,
  Tobi

---

From: Tobias Mueller <[email-masked]>
Date: Sun, 17 Mar 2019 20:35:19 +0100

Hi Sebastian,

On Mon, 2019-03-04 at 09:49 +0100, Sebastian Schinzel wrote:
> wrong conclusions. You want small chunks to do proper AE!
Can you mention what definition of AE you are referring to?
I guess you meant to add that you will need to come up with a secure
scheme to identify the last chunk and implement that properly. And that
you then you will need to buffer all the plaintext until the final chunk
has successfully checked out. Because otherwise you wouldn't get
"proper" AE as in either releasing plaintext or an error.

> The advantage of smaller
> chunks is that the plaintext can be cached until the chunk's auth tag
> is validated. That's to guarantee that no unauthenticated plaintext is

Two things: Firstly, you write "can be cached" rather than "must be
cached".
Unless you relax the security goals of the AEAD protected message.
Secondly, you can release unauthenticated plaintext of an AEAD protected
message of arbitrary size if you don't want to hold all the plaintext of
a decrypted ciphertext. Regardless of the size of the message or chunk.
Hence, there is no advantage of using a small chunk size if you want to
have an AEAD protected message. As in, if you intend to have proper AE
which only releases the full plaintext or an error.

Unless you need the concept of partially authenticated plaintext, the
only reason for using chunks is to detect failures early in the
decryption process rather than at the end. Again, if don't you want your
full message to enjoy the protections AE gives you, then you may be able
to afford partially authenticated messages. I haven't seen anybody
presenting a use-case for those. And even then it seems far fetched to
impose that concept onto each and every OpenPGP user as the current
proposal does.

Cheers,
  Tobi

---

From: Tobias Mueller <[email-masked]>
Date: Sun, 17 Mar 2019 21:00:51 +0100

Hi,

>   [1] https://tools.ietf.org/html/rfc5116#section-2.2
Maybe we're hitting a terminology issue here.
For me, a plaintext is authenticated if the whole ciphertext could be
successfully authenticated. Which seems to be very well in line with the
definition you've linked to.
Now, if you modify a (long) ciphertext that has been broken into chunks
near the end and a decryption routine reveals the first parts of the
decrypted ciphertext, would you agree that revealing those parts of the
plaintext does not meet the definition that you've linked to?
And do you further agree that you would need to find a way to prevent
any plaintext to be revealed unless the full message has been
authenticated correctly in order to match the definition that you've
mentioned?

Cheers,
  Tobi

---

From: Tobias Mueller <[email-masked]>
Date: Sun, 17 Mar 2019 21:01:32 +0100

Hi Neal,

> error prone.  Even computing AEAD's chunk size is hard:
point taken. Programming is hard. Programming C is even harder. Tooling
could be improved. We ought to write a spec that achieves a security
goal while making it less likely that people implement the spec wrongly.

You've made a point but you haven't really taken a stance on the point
that I raised. Which boils down to your proposal forcing everybody to
use chunking which in turn can be implemented wrongly while providing no
benefit in return.

For the sake of the argument, let's assume a safe aead_decrypt oracle. I
think that any CAESAR candidate provides such a function that is
sufficiently well suited.

I believe that the potential for releasing plaintext although the
ciphertext has not been authenticated is higher if I need to call that
oracle multiple times, invent a secure scheme for detecting truncation,
and implement that scheme correctly.

Your proposal removes the option for the message producer to decide
whether it wants to have the recipient (which may very well be the same
entity) rely on an implementation which gets all that right. Because
with the status quo, one can opt for producing one chunk which is
trivially protected against truncation and other attacks.  Which is
probably what I expect when I opted for protecting my message with AE.
And as far as I understand the current spec, it would not be directly
illegal for an implementation to release plaintext of a chunk early. So
my only chance as a message producer to not rely on implementations to
not release plaintext early is to use one chunk. (uff. so many
negations. Sorry about that).
Unless I understand your proposal wrong. But then my question is, how
would I encode a message that is not susceptible to (at least)
truncation attacks? Again, assuming the above mentioned oracle.

I don't fully understand why you are mentioning dynamically resizing
buffers. Strictly speaking, you need space for a single digit number of
blocks (!) to decrypt an EAX or OCB message. And you are free to release
unauthenticated plaintext as much as you want, much like your current
proposal.
If you want to process the message in 16kB pieces only, I don't see why
you couldn't. Even with a 5 exabyte message (or chunk).
If you don't want to reveal the plaintext, you could even mask the
output and only offer to remove the mask after the last chunk has
successfully checked out.

I can understand that the concerns around relying on correct chunking
can be dismissed, because several protocols exist that have successfully
(for all we know so far) done such a thing, so implementations can copy
and paste existing solutions.
I think, though, that it has not been appreciated enough that chunking
carries risks, because it relies on the implementation to not release
plaintext early. And, as we all know, releasing unauthenticated
plaintext has lead to EFail.  If using proper AEAD is the response to
EFail then your proposal to remove the option of not having to rely on
clients not releasing early plaintext is not helping implementations to
do the right thing, i.e. not release unauthenticated plaintext.

> more "powerful" solution.
Right.
One way of addressing your concern is to simply not allow chunking. Then
there wouldn't be a more "powerful" solution. As such I like your
proposal, but I would amend it and state that the chunk size is the size
of the packet minus the overhead.  We're not discussing that, though. 
Instead, your proposal is about  *removing*  the option for creating one
chunk and  *forcing*  everybody to use multiple chunks and thus bear the
risks of an implementation to get the chunking wrong.
Additionally, one could argue that because you force the concept of
partially authenticated plaintext onto every consumer, implementations
might see no problem with releasing that plaintext early. That, in turn,
can be considered worse than not releasing it, because it
prevents living up to the semantics of true AE which is to not release
any plaintext if the ciphertext has been modified.

> that means the client may have to worry about truncation.
Fair enough. Although I don't understand why you say "may" rather than
"must". In my view, typical applications for OpenPGP do not have
application level protection against truncation. I'm thinking Email and
Backups. Neither of these can reliably detect that the message has been
truncated.  And the application can't use the partial output either.
Are partially authenticated plaintexts important to you?
What's your use-case for those?
Could you realise your use-case with producing multiple OpenPGP messages
instead?

Anyway, that tradeoff which you mention, has not been part of your
proposal.  With one chunk, the ciphertext is trivially protected against
truncation.  With multiple chunks, you need to come up with a secure
scheme and implement that correctly.  This is far from trivial.
The change you've proposed does not indicate that the potential for
getting this wrongly has been considered.
And it does not alert consumers that they should use the OpenPGP message
format only for an application that can deal with truncated messages.
Note that I don't believe the currently described chunking mechanism is
broken. But the risk of implementing that wrongly is non-zero.

I wonder whether it makes sense to encourage not releasing any plaintext
if the ciphertext has been modified more strongly.

> Perhaps you're not a typical user :D.
Interesting point.
That may be true.
But I guess it's reasonable to expect a protection of the encrypted
message to be as close to AE as possible when I'm using an AE cipher.
As such, I expect to either get (the full) plaintext or an error.
What do you think does the typical user expect when using an AE cipher?
Do you think that the typical user has use for partially authenticated
plaintexts?
And if so, why would they not produce separate messages rather than one?

> just susceptible to truncation attacks.
First of all, you'd still need to collect all plaintexts before
releasing them to get "real AE". Simply because you must either return
plaintext or an error. I am not aware of a definition of AE that
includes partial plaintexts.
I bet that we haven't yet exhausted all possible ways which could lead
to an attack. For example, I wonder whether certain applications will
suffer from a timing attack because the attacker can determine which
chunk they were able to tamper with. Of course, fantasizing about yet
unknown attacks quickly yields esoteric scenarios and we better spend
our time on currently known problems. And again, we know that it's hard
to not release plaintext although the ciphertext has been modified.
By using one chunk I can choose to not rely on my recipient's
implementation to get the chunking right, simply because there is no
chunking involved.
Your proposal takes that option away and forces me to rely on the spec
and the implementation having gotten the chunking right.

As I've already mentioned, chunking is relatively easy and there are
other things that implementations can screw up more easily instead.  So
it's fair to dismiss that concern.
But it still feels weird to not have a way to express a truly AE
protected message. That makes me concerned that users would turn to
S/MIME instead.

To move forward, let me ask you:
Do you agree that calling something "authenticated encryption" raises
the expectation of getting either the full plaintext or an error rather
than partial plaintexts and an error?
Do you appreciate the need for a one chunk ciphertext (this question is
intentionally generic, i.e. regardless of difficulties implementing
that)?
Do you think we should encourage implementations to not release
plaintext unless the whole ciphertext has been authenticated?
Do you have use-cases for partially authenticated plaintexts?
Would separating the AEAD modes into non-chunked and chunked help you?

>   That's not to say that
Are you saying that leaving the option to do one-chunk AE, as it
currently exists, enables both truncation and ciphertext modification
attacks?  I don't understand why it would.

> but I see lots of instances of the former when reading code.
I hope you've filed the relevant bugs ;-)

Just in case someone is finding this in the archives:
GCC has __attribute__((cleanup)) and GLib has some convenience macros
around it.

Cheers,
  Tobi

---

From: Werner Koch <[email-masked]>
Date: Mon, 18 Mar 2019 07:50:22 +0100

>   > purpose: To be able to detect rare transmission errors earlier than at

Sorry, can you please explain how you can read out of this that I 
said: "we shouldn't bother with ciphertext integrity".

Shalom-Salam,

   Werner

---

From: "Neal H. Walfield" <[email-masked]>
Date: Mon, 18 Mar 2019 09:23:37 +0100

Hi Werner,

On Mon, 18 Mar 2019 07:50:22 +0100,
> said: "we shouldn't bother with ciphertext integrity".

I'm extremely happy to hear that I've misunderstood your position, and
that our goals are actually aligned.

Can you please explain how to get ciphertext integrity when streaming?

To be clear: for me, truncation is an orthogonal issue; my primary
concern is how to ensure that no implementations ever release
plaintext derived from modified ciphertext.

If you think that the current proposal achieves this, I've presented a
few critiques in <[email-masked]>
(https://mailarchive.ietf.org/arch/msg/openpgp/WvaisWDd9WzoKXL1FyHDeZNtZ2U):

  1. If messages can't be decrypted, because a chunk won't fit in
     memory, but unauthenticated streaming would otherwise "work,"
     implementors will do the latter, because security must not get in
     the way of a user getting her job done.

  2. If unauthenticated streaming is allowed or tacitly encouraged, it
     will be done first, because the set of messages that
     unauthenticated streaming can decrypt is a superset of those that
     authenticated decryption can handle, and it will perhaps be done
     exclusively because introducing a second code path that, modulo
     security concerns, is functionally equivalent to the code path
     for unauthenticated streaming is extra work, and extra
     complexity, which developers will try to avoid.

  3. If implementations revert to unauthenticated streaming for large
     chunk sizes, an attacker can conduct an EFAIL-style attack by
     changing the chunk size.

Thanks!

:) Neal

---

From: "Neal H. Walfield" <[email-masked]>
Date: Mon, 18 Mar 2019 09:51:47 +0100

On Sun, 17 Mar 2019 20:35:19 +0100,
> Two things: Firstly, you write "can be cached" rather than "must be

I think Sebastian is using "can" to mean "it is possible", not "may".

That is, 

  The advantage of smaller
  chunks is that *it is possible* to cache the plaintext until the chunk's auth tag
  is validated *whereas it is not necessarily possible to cache the
  plaintext with larger chunks*.

Neal

---

From: Tobias Mueller <[email-masked]>
Date: Mon, 18 Mar 2019 10:49:22 +0100

Hi,

> EFAIL.  It seems to me that it's worth adapting to this new threat.
I have the feeling that this is the spot where we are close to hitting
the underlying misunderstanding.

AFAICS, Werner is talking about chunking. You are talking about AEAD. I
have the feeling that it's helpful for the discussion to carefully
distinguish those.

You seem to imply that any chunked scheme is proper AEAD. But that's not
true. It's not difficult to make a scheme with chunks, and the spec
probably has already done so, but it's not trivial.

Cheers,
  Tobi

---

From: Tobias Mueller <[email-masked]>
Date: Mon, 18 Mar 2019 10:49:51 +0100

Hi Neal,

>   plaintext with larger chunks*.
I think I see what you mean.  But assuming that partial plaintexts are
of little to no value, because you have used an AEAD cipher after all,
then you can achieve the very same thing even today, with a message of
arbitrary size, because you can stream out partial plaintext.

Now you seem to find some value in partially authenticated plaintexts.
But I haven't understood why you find them so appealing that you want to
force them onto each and every user.

Cheers,
  Tobi

---

From: "Neal H. Walfield" <[email-masked]>
Date: Mon, 18 Mar 2019 11:53:14 +0100

Hi Tobias,

On Sun, 17 Mar 2019 21:00:51 +0100,
> Maybe we're hitting a terminology issue here.

I think we do.

> definition you've linked to.

4880bis defines a chunking mechanism based on AEAD: the message is
split into multiple chunks.  In 4880bis, AEAD operates on a per-chunk
basis.  The chunking algorithm provides mechanisms for ensuring chunks
can't be reordered, detecting the end of the message, etc.  Using AEAD
to decrypt a chunk authenticates that chunk's ciphertext; for a given
chunk, the decryption operation will either return the correct
plaintext, or it will return an error.  This is exactly what RFC 5116
requires.  RFC 5116 doesn't discuss chunking; chunking is not AEAD.

The chunking mechanism in 4880bis (assuming implementors follow RFC
5116 with respect to AEAD on chunks) ensures that plaintext is only
released from authenticated ciphertext.  It does not protect against
truncation attacks.  Since OpenPGP decided to support streaming in RFC
2440 (which introduced One Pass Signature packets), this is inline
with OpenPGP's philosophy.  I don't think removing support for
streaming is useful.

You seem to think that AEAD's guarantees must apply to the whole
message.  I disagree.  I agree it is useful, but it is not possible
when streaming.

I think that even if we add a bit that says: "don't stream",
implementations will ignore it.  Moreover an attacker can just clear
it, and then the receiver streams it anyways.

> plaintext does not meet the definition that you've linked to?

No.  AEAD is applied to chunks.

> authenticated correctly in order to match the definition that you've

No.  AEAD is applied to chunks.  When a chunk is decrypted, it either
returns authenticated plaintext, or an error.

Referring to <[email-masked]>
On Sun, 17 Mar 2019 21:01:32 +0100,
> goal while making it less likely that people implement the spec wrongly.

I agree.  Less flexibility in the spec is better, which is what I
want.

You rightly argue, I think, that AEAD without chunking is easier to
implement.  But, OpenPGP supports streaming, which is incompatible
with AEAD without chunking.  Consequently, streaming using AEAD
without chunking means releasing unauthenticated (as I have defined it
above) plaintext.

Releasing unauthenticated plaintext is much worse than releasing only
an authenticated prefix.  Releasing unauthenticated plaintext allows
for an EFAIL style attack; releasing an authenticated prefix does not.
Releasing an authenticated prefix prevents ciphertext malleability.

> use chunking which in turn can be implemented wrongly while providing no

Only releasing plaintext from authenticated chunks prevents
EFAIL-style attacks.  That is not no benefit, that is not a little
benefit, that is a huge benefit.

> and implement that scheme correctly.

I agree that the added complexity that chunking imposes is not worth
it *if* the only reason to introduce chunking is to detect
transmission errors early.  But, it is not.  It prevents modifications
to the ciphertext.

> trivially protected against truncation and other attacks.

You say that it is trivially protected, but this is not the case,
because exactly in those cases implementors will just stream
unauthenticated data.  They should only be able to stream data from
authenticated chunks.

> truncation attacks? Again, assuming the above mentioned oracle.

OpenPGP allows streaming.  The implementation doesn't provide a
mechanism to prevent streaming.  I'm not sure it would be a good idea
to provide one, as I don't think there is a sensible way for the
sender to decide when to set that bit.

> blocks (!) to decrypt an EAX or OCB message.

The size of the message is not known apriori if partial body length
encoding is used.  Even an upper bound is not known if compression is
used.

---

From: Vincent Breitmoser <[email-masked]>
Date: Mon, 18 Mar 2019 13:23:07 +0100

Hi,

a lot that Tobias is bringing up here resonates with me, I feel like we should
be thinking more about the fully vs partially authenticated use cases, not just
chunking on its own.

(As an aside, I'm not convinced the early integrity check should have much
bearing on this discussion.  Transmission errors are (and should be) handled on
other layers in mostly all cases, and noticing errors earlier than at the end of
data that was going to be buffered anyways is not that big of a gain. In cases
where this is a concern (like uh, tape drives?), a tool should be used that is
actually meant for the job, e.g. par2)

Ideally, a receiver won't ever output unauthenticated plaintext, hence ideally
all of the chunking discussions would be moot. What chunking brings to the table
is to give the *sender* of a message the option to *allow* the *receiver* to
emit partially authenticated plaintext, trading a truncation vulnerability for
the ability to process data on a smaller buffer size than the entire plaintext.
This is useful for $large amounts of data, or streamed workflows with unknown
data sizes.

While following the discussion I've gone back and forth a couple of times
between favoring the case for fully authenticated plaintext, or for supporting
streamed workflows with fixed-size chunks (while sacrificing truncation). Both
seem equally valid to me. However, I can't see a good use case for variable size
chunking: it adds complexity to spec and implementations in particular on the
receiving side, and pushes the onus on reasoning about chunk sizes to the
implementations, which is basically impossible in the face of interoperability
concerns.

I'd like to bring up a new proposal then: Support either no chunking, or
fixed-size chunking. The advantage would be that the sender's position on
authentication is made more explicit: If they don't do chunking, they expect the
receiver to fully buffer and authenticate before processing, which could
currently only be achieved implicitly via a large chunk size. If they use the
fixed-size chunking, they explicitly offer the option to emit partially
authenticated plaintext.

 - V

---

From: Peter Pentchev <[email-masked]>
Date: Mon, 18 Mar 2019 14:29:14 +0200

> fixed-size chunking, they explicitly offer the option to emit partially

Not rejecting your idea outright, just curious how it would work with
e.g. instant messengers where there is streaming of irregularly-sized
packets coming at irregular intervals (and the users would expect that
Things Just Work(tm) for both two-character messages and five-megabyte
files sent over the wire).

G'luck,
Peter

-- 
Peter Pentchev  roam@{ringlet.net,debian.org,FreeBSD.org} [email-masked]
PGP key:        http://people.FreeBSD.org/~roam/roam.key.asc
Key fingerprint 2EE7 A7A5 17FC 124C F115  C354 651E EFB0 2527 DF13

---

From: Vincent Breitmoser <[email-masked]>
Date: Mon, 18 Mar 2019 13:39:27 +0100

> files sent over the wire).

I would expect the interface to use non-chunked mode for a simple "buffer in,
buffer out" interface ("secret box"), and chunked mode ("secret stream") for
streamed data.  In your example, a messenger would probably encrypt text
messages by passing an entire buffer, while images or larger data could be
streamed.

This is a useful distinction to have in general, streaming APIs are much harder
to use correctly than simple buffered ones. See also libsodium.

 - V

---

From: "Neal H. Walfield" <[email-masked]>
Date: Mon, 18 Mar 2019 14:50:20 +0100

On Mon, 18 Mar 2019 13:23:07 +0100,
> This is useful for $large amounts of data, or streamed workflows with unknown

When using chunking correctly, the emitted plaintext is not partially
authenticated; it is fully authenticated: all of the bytes are right.
As such, I'd prefer to use the term "authenticated prefix" in place of
"partially authenticated plaintext".

Some questions that come to mind:

  - Should an implementation ever be allowed to emit unauthenticated
    plaintext?

    Is it okay to have a --do-it-anyway flag?  How to we prevent
    implementors from doing it anyways?

  - What should an implementation do if it is passed a large message
    (say n times the available RAM) and chunking is disabled?

    Should implementations be smart enough to detect that they can't
    process this apriori?  Should they wait for the OOM-killer?
    Should they panic?  High-level languages don't make it easy to
    recover from memory allocation failures.  For instance on Rust, an
    out of memory error results in a panic.  It would be a shame if
    the application crashed, or the sender could DoS the receiver.

It seems to me, if we are going to handle the complexity of chunking,
then we should just embrace it, and not make it even more complex.  If
an application wants to protect itself against truncation attacks,
then it can buffer the output, or the openpgp implementation can have
a flag.

> However, I can't see a good use case for variable size

I fully agree with this.

> fixed-size chunking, they explicitly offer the option to emit partially

I'm concerned about an attacker's ability to twiddle the chunking bit.
Do you have any thoughts on how to prevent this?

---

From: "Derek Atkins" <[email-masked]>
Date: Mon, 18 Mar 2019 09:57:37 -0400

Hi Neal,

> Hi Werner,
[snip]
> plaintext derived from modified ciphertext.

With or without AEAD?

>      chunk sizes, an attacker can conduct an EFAIL-style attack by

Which implementation?  The sender or the receiver?  Also, who is the
attacker?  The Sender?  Or a third party?

Note that the way I see Chunking + AEAD working together is that there is
a 1:1 mapping of Chunk to AEAD block.  I realize this is probably a change
in the way it's currently working (and it's been 20 years since I wrote
the original chunking code), but I feel that AEAD and Chunking go
hand-in-hand.  Specifically, I see chunking as a way to splice a bunch of
separate AEAD blocks together into a chain to ensure you don't drop or
reorder the AEAD chunks.  The only attack is a DoS if the end is
truncated, and the earlier (valid) chunks have already been released.

In other words, AEAD is used for each chunked block as a single, coherent
piece.  Also I don't see how an attacker (third party) could leverage this
at all.  They couldn't change the chunk size enroute.  Granted, if the
attacker is the sender they could do much worse things, so you just need
to ensure that, as you say, unauthenticated plaintext isn't released.  But
that cannot happen given the context above.  Each chunk is either
authenticated, or an error occurs.  On error, it wouldn't be released.

Chunk sizes SHOULD be limited to ensure processing capability, which is
where this conversation started.  Of course, this assumes chunked mode. 
It should still be allowed to have a single AEAD "chunk" if the sender
wants to buffer the message.

-derek

---

From: Vincent Breitmoser <[email-masked]>
Date: Mon, 18 Mar 2019 15:23:06 +0100

> "partially authenticated plaintext".

I'm not sure calling individual bytes "authenticated" is valid if they have been
authenticated as part of a larger message. But I don't mind the terminology much
one way or another.

>     implementors from doing it anyways?

I don't think we can do too much there, one way or another.

>     (say n times the available RAM) and chunking is disabled?

They can decide between failing hard, buffering to disk, or taking
responsibility for emitting unauthenticated output. Note that this choice is
much easier at implementation time if the chunk size isn't variable - either you
support chunking, or you don't.

> Do you have any thoughts on how to prevent this?

Good point. I guess we could simply stick with the "empty chunk at the end"
mechanism, which implementations have to have anyways for chunked AE?

 - V

---

From: "Neal H. Walfield" <[email-masked]>
Date: Mon, 18 Mar 2019 15:23:23 +0100

On Mon, 18 Mar 2019 14:57:37 +0100,
> > plaintext derived from modified ciphertext.

Well, whatever :).  As I understand Werner, he agrees that ciphertext
integrity while streaming is desirable, and, I guess, thinks that is
achievable with the current draft.  I don't think it is possible, and
I've raised a few concerns in my prior mails.  I hope Werner can
address those concerns.

> Which implementation?  The sender or the receiver?

I'm assuming that the sender is using the best we have to offer, i.e.,
chunked AEAD.

I'm assuming that the receiver emits unauthenticated plaintext in some
situations.

> Also, who is the

I'm thinking of something like EFAIL.  So, a third-party attacker who
modifies the ciphertext.

> The only attack is a DoS if the end is
> at all.  They couldn't change the chunk size enroute.

I hacked Sequoia to emit unauthenticated data, and I was able to
change the chunk size and still get the plaintext of at least the
first chunk of a message.  So, my concern is: if a receiving
implementation releases unauthenticated plaintext for large chunks
rather than fail with ENOMEM, then a third-party attacker can change
the chunk size of an intercepted message, and cause the receiver to
process unauthenticated data, even though a small chunk size was
originally used.

> Each chunk is either

The question for me is: what does an implementation do if it can't
buffer the message?  Does it really throw an error?  There already
exists at least one implementation written by an editor listed on
4880bis that processes unauthenticated plaintext.

---

From: Vincent Breitmoser <[email-masked]>
Date: Mon, 18 Mar 2019 15:27:43 +0100

> mechanism, which implementations have to have anyways for chunked AE?

Actually, using any fixed value as chunk index as additional data would work
fine, perhaps reserving all 0xFF for that (and boldly assuming that we'll never
have 2^65-1 chunks in an actual message).

 - V

---

From: Tobias Mueller <[email-masked]>
Date: Mon, 18 Mar 2019 20:51:32 +0100

Hi,

On Mon, 2019-03-18 at 11:53 +0100, Neal H. Walfield wrote:
> requires. 
I beg to differ. Because, as you mention:

>  RFC 5116 doesn't discuss chunking; chunking is not AEAD.
> 
Chunking is not AEAD. It's a protocol on top of AEAD messages that you
have to come up with. And then you have to implement it correctly. The
security guarantees that AEAD gives you, do not automatically apply to
your chunking scheme.
As you've said: Chunking is not AEAD. Hence, it cannot automatically be
in line with what RFC5116 demands.

> message.  I disagree.  
I'm glad you're saying this.
And yes, I think that proper AE means that the full message enjoys the
security guarantees of AE. Also because I am not aware of definitions
covering partially authenticated plaintext. And I think that RFC5116
leans more towards full messages rather than trying to define security
guarantees for partial plaintext.  I further think getting as close to
proper AE as possible is a goal worth pursuing.

> I agree it is useful, but it is not possible
> when streaming.
If you absolutely must stream, then there is no way that you can buffer
the whole message, otherwise you wouldn't stream.  I claim, however,
that in the vast majority of use-cases you don't have the requirement of
having to stream.  As in, purely from a functional perspective, not from
an implementation perspective.  Hence, imposing the concept of streaming
onto everybody somehow does not feel right.
I'd like to note, though, that it is possible to not reveal the
plaintext no matter how large the message is, though.  You can mask the
output you release, e.g. XOR it or apply CTR mode, and provide the key
to remove the mask only when the ciphertext has checked out correctly.

>From the proposal you made it seem you think we should not even try to
provide a format for a non-streaming message.  Would you describe that
as correct?

> implementations will ignore it.
Hm. I'd classify this as a wilful violation of the spec rather than an
accident while implementing it.
Once you assume that implementations are doing things wilfully wrongly,
it gets messy.
I mean... where do we stop making compromises in the security of the
spec because we believe someone will wilfully ignore the spec? We rely
on the client not actively misinterpreting the spec. Like.. not making
secret key material available.

Cheers,
  Tobi

---

From: "Neal H. Walfield" <[email-masked]>
Date: Mon, 18 Mar 2019 22:03:17 +0100

On Mon, 18 Mar 2019 20:51:32 +0100,
> in line with what RFC5116 demands.

The chunks use AEAD.  So 5116 applies to the chunks.  That means, for
a given chunk, either authenticated plaintext is returned or failure.
I don't see a contradiction.

> covering partially authenticated plaintext.

TLS is used by a few billion people.  Let's just do what they do...

> And I think that RFC5116
> guarantees for partial plaintext.

RFC 5116 only talks about AEAD.  But these trade-offs are specifically
addressed by David McGrew, the author of 5116, here:

  https://mailarchive.ietf.org/arch/msg/cfrg/_iVbSmVLW7nY82k8xZmuVFIX6cs

  Fragmenting the plaintexts before applying AEAD requires that each
  AEAD message be independently authenticated, and thus this technique
  has more data/encapsulation overhead.

In his terminology, fragmentation is approximately the same as
chunking in our terminology.  It seems to me, the he says AEAD is
applied to fragments/chunks, not the whole message, which is the
terminology that I've been using.

The whole thread, starting here,
https://mailarchive.ietf.org/arch/msg/cfrg/-lj3IEm9agpfgUOb2yX9JNrtrm8
is an interesting read.  And, I think it generally supports my
position.

> I further think getting as close to

I think that if we accept your position, OpenPGP will have less
practical security.

You've repeatedly said that if an implementation doesn't want
authenticated plaintext, it is free to release unauthenticated
plaintext (e.g., "you are free to release unauthenticated plaintext as
much as you want",
[email-masked]).
That's much, much worse than a possible truncation attack.  And, David
McGrew agrees:

  https://mailarchive.ietf.org/arch/msg/cfrg/_iVbSmVLW7nY82k8xZmuVFIX6cs

  robustness against decryption misuse is a valuable property for an
  AEAD algorithm to have.  Nonetheless, it would be better if the
  protocol using AEAD actually performed some sort of plaintext
  fragmentation, if it can accommodate the overhead, because the
  security would be better.

> onto everybody somehow does not feel right.

Let's consider email, which is a common use case for OpenPGP.  I like
that K-9 just downloads the first few kilobytes of each message.  I
want K-9 to be able to continue to do that even when everyone is
encrypting their email.  For that, I need an authenticated prefix.

Likewise, with a dropbox-like application, I'd like to be able to
preview the content.  Again, I need an authenticated prefix.

I want to be able to stream archives and backups.

Pretty much the only case that I can think of that chunking is not
useful is for verifying software updates.

> to remove the mask only when the ciphertext has checked out correctly.

I don't see how that helps with streaming.  Basically any further
processing needs to wait until the whole message has been decrypted a
second time...

> provide a format for a non-streaming message.  Would you describe that

We should provide exactly one variant.  Additional variants must
justify their existence, and I don't see the huge value add for
"proper AEAD".  In fact, it seems dangerous as I think it will
encourage decryption misuse.

> secret key material available.

It's a simple question of: "can I use the software to get my work
done"?  If the security is in the way, then the security will be
disabled.  "Proper AEAD" will be in the way often enough that it will
get disabled, and decrease the security of the whole system.

Sorry,

Neal

---

From: Bill Frantz <[email-masked]>
Date: Mon, 18 Mar 2019 14:11:41 -0700

On 3/18/19 at 6:50 AM, [email-masked] (Neal H. Walfield) wrote:

>If

When processing streamed messages, you have already bought into 
the idea that you may be processing early data in the message 
before the later data has even been sent.

To protect against truncation attacks you can borrow an idea 
from the database people and not commit your changes until you 
have a complete message.

Cheers - Bill

------------------------------------------------------------------------
Bill Frantz        |"Insofar as the propositions of mathematics 
refer to
408-356-8506       | reality, they are not certain; and insofar 
they are
www.pwpconsult.com | certain, they do not refer to reality.” 
-- Einstein

---

From: Andre Heinecke <[email-masked]>
Date: Mon, 18 Mar 2019 22:53:44 +0100

Hi,

You know I would love to follow all discussions here but this thread feels 
somehow like a denial of service attack. I can't get though it.

I need a tl;dr; here. Honestly. Today I saw replies in three different branches 
of this thread and I think it just got out of hand.

My tl;dr; is (and I'm sure you'all will correct me if I'm wrong):

1. The AEAD chunk size in the current proposal is too variable.
2. Neal proposed using a fixed chunk size.
2.3. Some people had points that spoke against it.
3. There was a compromise offered with a smaller "variable chunk size".

Then you wen't off the rails and repeatedly talked by and around each other and 
off topic. Here is where you lost me. This thread now has some epic (rambling) 
proportions.

Can we somehow get some clear "votable" proposals out of this thread?
I mean really. This stuff does not have to be so complicated as you make it 
out?! Just write in as few (that is a _good_ thing) words as possible what 
your proposal ist now. Maybe we could then vote on it or something like that?
"Chunk size should be between 512byte and 128MiB" does not sound so 
complicated to me.

I mean I really do not want to disrespect your efforts but I think there is 
valuable time of OpenPGP implementors wasted in this thread repeating 
themselfs and I really want to move forward with 4880bis

Best Regards,
Andre

-- 
GnuPG.com - a brand of g10 Code, the GnuPG experts.

g10 Code GmbH, Erkrath/Germany, AG Wuppertal HRB14459
GF Werner Koch, USt-Id DE215605608, www.g10code.com.

GnuPG e.V., Rochusstr. 44, D-40479 Düsseldorf.  VR 11482 Düsseldorf
Vorstand: W.Koch, M.Gollowitzer, A.Heinecke.    Mail: [email-masked]
Finanzamt D-Altstadt, St-Nr: 103/5923/1779.   Tel: +49-2104-4938799

---

From: Werner Koch <[email-masked]>
Date: Tue, 19 Mar 2019 08:30:22 +0100

On Mon, 18 Mar 2019 14:11, [email-masked] said:

> To protect against truncation attacks you can borrow an idea from the
> database people and not commit your changes until you have a complete
> message.

Right.  And you need to do that anyway because authenticated encryption
doesn't tell you anything about the origin of the message and thus you
need to check the signature of the message after it has been completely
decrypted (at least with OpenPGP and CMS).  Anyone can send malicious
content and AE doesn't protect against processsing such content.

Salam-Shalom,

   Werner

---

From: "Neal H. Walfield" <[email-masked]>
Date: Tue, 19 Mar 2019 09:32:04 +0100

On Tue, 19 Mar 2019 08:30:22 +0100,
> content and AE doesn't protect against processsing such content.

I agree with you that AEAD + signed chunks is even better.  But,
streaming AEAD still provides a significant improvement relative to
the status quo: it protects against EFAIL-style exfiltration attacks.

---

From: Werner Koch <[email-masked]>
Date: Tue, 19 Mar 2019 11:01:20 +0100

> the status quo: it protects against EFAIL-style exfiltration attacks.

Neal: Please stop spreading false information.  Our MDC already protects
against this (old) kind of attack.  We implemented the new AE algorithms
merely for cryptographic cleanness and speed.

Shalom-Salam,

   Werner

---

From: "Neal H. Walfield" <[email-masked]>
Date: Tue, 19 Mar 2019 11:09:49 +0100

On Tue, 19 Mar 2019 11:01:20 +0100,
> merely for cryptographic cleanness and speed.

Thanks, I guess I've misunderstood the issue.  Can you please explain
how the MDC helps protect against ciphertext modification in the
streaming case, as that it still not clear to me.

Thanks!

:) Neal

---

From: Derek Atkins <[email-masked]>
Date: Tue, 19 Mar 2019 12:21:36 -0400

Neal,

> I've raised a few concerns in my prior mails.  I hope Werner can

I admit that I have not studied the current draft closely, so my
comments might be out of whack, but my expectation is that the AEAD and
Chunking are done hand-in-hand.  The chunk metadata (number/size/etc)
should be protected by the AEAD.

So let's say you have a message that gets processed as:

[Chunk 1] [Chunk 2] [Chunk 3/final]

In the normal case, the receiver will process Chunk 1, then Chunk 2, and
then Chunk 3, and depending on the implementation MAY emit the Chunk 1
plaintext (which HAS BEEN AUTHENTICATED) before it processes Chunk 2
and/or Chunk 3.

What can an attacker do?

* If they modify the chunk size, the AEAD on that chunk will fail and
  therefore the data should not be emitted.  Note that this COULD cause
  the receiver to emit Chunk 1 and then "stop".

* If they cause Chunk 3 to disappear, they could get the receiver to emit
  Chunks 1 and 2 and then stop with.  But again, Chunk1 and Chunk2 are
  still authenticated.

* If they cause Chunk 2 to disappear, the numbering will be out of order
  and the receiver can emit an error.

* They can insert fake chunks.  Let's say they cause the following:
  [Chunk 1] [Chunk 2'] [Chunk 2] [Chunk 3'] [Chink 3/final]

  In this case, the attacker is inserting Chunk 2' and Chunk 3',
  claiming to be chunks 2 and 3 respectively.  So the question is, what
  happens here?  Well, the chunking here is expecting encryption (at
  least I think it should be expecting encryption).  So they would
  authenticate Chunk 1 (and possibly emit it), but when processing Chunk
  2' they would fail on the AEAD because the attacker doesn't know the
  key.

  The chunking should not allow any other type of data here.

Am I missing another attack?

[snip]
> I'm assuming that the receiver emits unauthenticated plaintext in some

In what situations would the receiver emit unauthenticated plaintext?

> I'm thinking of something like EFAIL.  So, a third-party attacker who

EFAIL was more that that -- it was also leveraging the fact that USERS
of OpenPGP would merge the contents of authenticated and
non-authenticated data when presented in e.g. a MIME context, such that
the processor could not differentiate between the protected and
unprotected content.

> process unauthenticated data, even though a small chunk size was

Sure, but the first chunk of the message was authenticated when it was
emitted.  The SECOND chunk failed, and I bet it stopped processing!

This is what SHOULD happen.

> 4880bis that processes unauthenticated plaintext.

I think we have different definitions of unauthenticated plaintext here.

In a chunked message, plaintext is authenticated PER CHUNK.  So once
Chunk 1 is processed, it is considered authenticated, and the
implementation is free to emit it, even if Chunk 2 fails.

If Chunk 2 fails its AEAD check (but Chunk 1 processed successfully), do
you consider the plaintext of Chunk 1 unauthenticated?  If so, why?

-derek

---

From: "Neal H. Walfield" <[email-masked]>
Date: Tue, 19 Mar 2019 19:10:18 +0100

Hi Derek,

Thanks for your analysis.  I think the AEAD chunking algorithm is
sound, if it is used correctly.

My issue is that I don't think it is possible to use the chunking
algorithm correctly for large chunk sizes.  For instance, what should
an implementation do if it encounters a chunk size of 16 TB (and there
really can be >16 TB of data using a small decompression bomb)?
Should it be allowed to emit unauthenticated plaintext?

Let's assume that we allow implementations to emit unauthenticated
plaintext for large chunk sizes.  An attacker who intercepts a message
can change the chunk size to be above this limit.  Now, clearly this
will fail the authentication check, but we just allowed
implementations to release the plaintext for large chunk sizes.  So,
if my analysis and experiments are correct, then using AEAD will never
provide ciphertext integrity for the first chunk against an active
attacker.

Let's assume that we don't allow implementations to ever emit
unauthenticated plaintext.  Well, if they encounter a large chunk that
they can't buffer, they MUST fail.  Ouch!  User's will work around
that and implementations will help.  (As I've pointed out, one already
has.  I don't know if it was an oversight.)

So, my conclusion is, we must prohibit implementations from emitting
unauthenticated plaintext *and* remove any incentives to do so.  For
me, this means a small, fixed chunk size.

Does this clarify my concerns?

Thanks!

:) Neal

At Tue, 19 Mar 2019 12:21:36 -0400,
Derek Atkins wrote:
> _______________________________________________
> openpgp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/openpgp
>

---

From: Marcus Brinkmann <[email-masked]>
Date: Tue, 19 Mar 2019 19:53:08 +0100

> In what situations would the receiver emit unauthenticated plaintext?

(Neal will probably respond, but let me just state that the current
implementation in GnuPG actually manages the output buffer
asynchronously from AEAD chunk processing, so that it will emit
plaintext from unauthenticated, partially processed chunks.)

> the processor could not differentiate between the protected and

That's true for some variants of the EFAIL attack (direct exfiltration
of unmodified ciphertexts with MIME wrapping), but not for all (crypto
gadgets exploiting ciphertext malleability and MDC error tolerance).

Thanks,
Marcus

---

From: Nickolay Olshevsky <[email-masked]>
Date: Tue, 19 Mar 2019 21:57:56 +0300

Hi Neal & others,
Tried to follow the discussion as much as possible, but probably missed some pages.

Independent on chunk size, we get to the issue of large message transfer, which in practice is used widely (large corporations, banks, media companies, whatever else).
The case with small chunk size and damaged last chunk is actually (almost) the same as the case with the single large chunk - you output, say, 20GB of data and then realize that it is broken. The same would happen with streamed transfer of large signed objects.

If you have data stored in file/memory/database you may first check it, and then decide whether it is ok or not, spending resources on two passes.
But while streaming large amounts of data which doesn’t fit to memory there is no choice - you still need to process it in smaller blocks, without guarantee that error will not come later on. Underlying compression or other known data structure comes in handy - it will serve as additional layer of security (not 100% of course) for transmission errors or transfer intervention.
In any case you first output a load of data and then must discard it.

So, my opinion is that we may leave things as they are, or just extend with recommendations based on performance. 
Given the fact that now not 100% of implementations support AEAD even partially.

> On Mar 19, 2019, at 21:10, Neal H. Walfield <[email-masked]> wrote:
> 
> Hi Derek,
>

---

From: Derek Atkins <[email-masked]>
Date: Wed, 20 Mar 2019 09:24:22 -0400

Hi,

> Should it be allowed to emit unauthenticated plaintext?

Aha, and we have come around full circle again.

The chunk size needs to be small enough that the receiver can process
it.  If it's too big, then it cannot be processed and the receiver
either must buffer it or will decide to release the plaintext prior to
the chunk being completed.

[snip]
> me, this means a small, fixed chunk size.

There is no way to prohibit the implementation from emitting
unauthenticated plaintext.  Implementations will do what they want/need
to do.

I still don't think we need a fixed chunk size.  Different use cases may
dictate different ideas.  It's a tradeoff, of course.  The hope would be
the receiver can signal to the sender what it should do.

I DO believe that recommended chunk sizes should be smaller than, say
4TB (let alone exabytes).  I am happy to have the range be anywhere from
1KB to 128MB (give or take), but I still don't think we should outright
prohibit smaller or larger.  Considering the chunk size should be part
of the protected data, I don't see how an attacker could modify it, only
a sender that doesn't pay attention.

Specifically, if the sender and receiver have some out-of-band knowledge
about each other I still think it's perfectly reasonable for them to
make their own choices.

> Does this clarify my concerns?

I think so, and we ARE on the same page, I think.

-derek

---

From: Tobias Mueller <[email-masked]>
Date: Tue, 26 Mar 2019 18:40:35 +0100

Hi Neal,

On Tue, 2019-03-19 at 11:09 +0100, Neal H. Walfield wrote:
> Can you please explain
RFC 4880 currently states in § 5.13:

    Any failure of the MDC indicates that the message has been modified
    and MUST be treated as a security problem. 

Further down in the security section (§ 14):

    An implementation MUST treat an
    MDC failure as a security problem, not merely a data problem [...]

    Consequently, it is important that:

     1. Implementers treat MDC errors and decompression failures as
        security problems.

There the MDC, if the spec is followed, indicates a security problem. I
guess that conversely, applications using OpenPGP thus ought to exercise
care when using that data which was flagged as having a security
problem.  So that's how the MDC protects against modification of the
ciphertext.
Is that clear to you now?

Obviously, that didn't prevent consumers from ignoring that reported
security problem. But the spec cautions the use of the unauthenticated
data.  That's arguably too weak and it should probably say to not use unauthenticated data at all.  And implementations probably shouldn't have released plaintext early.  Or, if the application really needs to access early plaintext, because, say, it's running out of buffering capacity, then it needs to be able to deal with the fact that the data may turn out to be bad.
AFAIU, the MDC itself is not the most elegant or efficient construction,
but works just about well enough to detect modified ciphertext.

As an aside which you all of you are probably aware of: The other,
bigger, problem with SEIP packets is that they it can be downgraded [1]
which then, AFAIU, served as a another motivation to move to AEAD [2].

While "authenticated prefixes" (if used correctly) would render certain
ciphertext modifications powerless, they still leave an application
susceptible to attacks if it is not prepared to defend against them. 
Spontaneous examples that I can come up with are the script in the
backup that does "cp /foo /bar" where the attacker manages to cut off
the "bar" and causes havoc or a PDF with multiple xref tables where the
attacker manages the remove the last one. I'm sure there are more and
better examples.  I appreciate that the bar is higher for an attacker to
mount this attack if you can only cut off at the end rather than modify
anything at free will.

At the end of the day, though, the application layer has to be able to
defend against such an attack.  Now the question becomes whether
applications using OpenPGP are in such a position.  Assuming that most
of them are would make things easy, but may be a bit far fetched.  Yet,
forcing streaming onto every user and encouraging the release of
plaintext while the whole message has not yet been authenticated seems
to make that assumption.  I thus dislike the proposed removal of the
option of not dividing a message into separately authenticated pieces.
I appreciate that the authenticated prefixes reduce the attack surface
of Efail style attacks when you really can't hold all of the message. 
But they don't prevent the root cause of those attacks. Not using
unauthenticated plaintext (or parts thereof) would.

To that end, I like Vincent's idea of moving the discussion towards
fixing a chunk size for chunked mode but leaving support for a non-
chunked mode.

Cheers,
  Tobi

1: https://mailarchive.ietf.org/arch/msg/openpgp/JLn7sL6TqikUf-cD34lN7kof7_A
2: https://mailarchive.ietf.org/arch/msg/openpgp/9uJYltQjfOaTKlnarJoDYnLXNzc

---

From: Tobias Mueller <[email-masked]>
Date: Tue, 26 Mar 2019 18:45:09 +0100

Hi Neal,

On Mon, 2019-03-18 at 22:03 +0100, Neal H. Walfield wrote:
> > > > For me, a plaintext is authenticated if the whole ciphertext
> > > > could
> > > 
> > > be
> > > > successfully authenticated. Which seems to be very well in line
> > > > with
> > > 
> > > the

Let's carefully revisit the RFC5116. It starts with:

   There is a single output:

      A ciphertext C, which is at least as long as the plaintext, or

      an indication that the requested encryption operation could not be
      performed.

Note that there is *a single output* rather than multiple and that it
doesn't allow for releasing partial plaintexts or authenticated
prefixes.
Do you see that any chunking protocol on top of that which is allowed
for releasing plaintext early is not immediately covered by this
definition?

Let the RFC be crystal clear:

    In particular, partially encrypted or
    partially decrypted data MUST NOT be returned.

This contradicts with handing out decrypted chunks, partial plaintexts,
authenticated prefixes, or whatever name you might want to give them, as
those are partial decryptions of the original message that was protected
in full with an AEAD scheme. If I meant to encode several individual
messages which have a right to be decrypted on their own, I might as
well have produced several individual messages.

> TLS is used by a few billion people.  Let's just do what they do...
I don't fully understand how you're making the leap here from
definitions of AE to a transport layer protocol. But that's a very
interesting point.

First of all, be aware that TLS records use a variable size. That's much
unlike what you're proposing. 
Additionally, TLS is a transport layer protocol rather than a syntax
which people will use to archive their emails and backups, which OpenPGP
is to some.
Finally, TLS requires the application to be safe from truncation. From
the source you've referenced yourself <
https://crypto.stanford.edu/~dabo/cryptobook/BonehShoup_0_4.pdf> on the
"Cookie Cutter" attack in § 9.8:
"TLS assumes that the application layer will defend against this
attack".
I think you can make that assumption for applications using OpenPGP, but
if you do then stating that in the spec would be more honest about the
security OpenPGP will provide rather than silently taking the one-chunk
option away without warning the applications which intend to use the
protocol.

The TLS record protocol needed to be changed a few times to respond to
newly discovered threats.  I am too humble to assume that the design of
the OpenPGP chunking protocol has been gotten right on the first
attempt. By removing the option to not use the chunking protocol, you
prevent a safe usage of the protocol in the future.

Do you still think we should blindly "just do what they do"? What would
that even mean?

> is an interesting read.  And, I think it generally supports my
I thought your position was that chunking a message and applying AE to
those automatically preserves the security properties which we have
proofs for.  The position in the thread, as far as I can see, is, that a
custom chunking protocol has unknown security properties.
I'm glad that this was a misunderstanding.

> I think that if we accept your position, OpenPGP will have less
AFAICS, there is no need to accept anything here. Right now, with
4880bis, it is possible to create a message with exactly one chunk. You
are proposing to remove that option.

Did you rather mean to say that "if we keep providing the option of
using proper AE for the messages then we will have less practical
security"?
And I would even agree to some extent.
But I'm not yet convinced that fully trusting the OpenPGP chunking
mechanism leads to a safer future than allowing clients to not rely on
the correct definition and implementation of that mechanism.

> encrypting their email.  For that, I need an authenticated prefix.
Just a quick reminder that nobody is about to take that option away.
The proposal at hand is to remove the option of not having to produce
several chunks.

Reg. Email, let me cite the Efail paper: 

   We think it is safe to turn off streaming in the email context
   because the size of email ciphertexts is limited and can be handled
   by modern computers.

> preview the content.  Again, I need an authenticated prefix.
I don't object. The application could, probably based on the file type,
determine reasonably safe boundaries and encrypt those.
Assuming that the safe boundary for all files is the same seems
dangerous.

> I want to be able to stream archives and backups.
I'm not so sure here. A  nc -l -p 1234 | unsafe_decrypt | tar -f-  can
have severe consequences if partial plaintexts are involved.  I can
understand that some users are enjoying the guarantees of AE, that is,
either to decrypt the whole message or nothing. With your proposal, the
user is at risk of either the chunking algorithm not being sound, the
algorithm not being implemented correctly, or the application (tar, in
this case) not being safe against recovering from a truncated message.
And the truncation is the best case, that is, we assume that the
implementation did everything right. We haven't yet considered a minor
flaw in the implementation that would allow, say, re-using nonces, re-
ordering of chunks, or changing the size of the chunks.
With your proposal, how would such a user construct an OpenPGP message
that is less risky?

> useful is for verifying software updates.
I agree that chunking is useful more often than it's necessary.

> processing needs to wait until the whole message has been decrypted a
Of course. Implementations are free to provide a different API. I merely
presented an example of how one could expose the functionality of the
spec.
Anyway, if you desire a property P and that property depends on all the
bits of the ciphertext, you need to wait until you have gotten around to
process all those bits. One could argue that this has been the semantic
ever since the MDC has been introduced. Now with AEAD we get a formal
description of that semantic.
And it helps an OpenPGP library to support streaming, because it can
safely release that memory to the caller or update the message in-place.

> encourage decryption misuse.
Fair enough.
The value add is that we have proofs for the security properties of AEAD
encrypted messages. AFAICS, we don't have those for the custom chunking
protocol. I'll be happy to be told otherwise. Note that proven schemes
exist which have "intermediate tags", such as POET or COLM. Alas, none
of them is defined as an option for OpenPGP (yet?).
Another benefit is being able to not use the chunking once it turns out
that it's not good enough anymore.

So we know that using AE has security benefits, that libraries can
implement that safely, and that most use cases do not require messages
to be actually streamed or that they're better off defining partial
plaintexts for themselves. Given that applications can produce multiple
message if they desire, the one-chunk option is strictly better than the
multiple-chunk option.  Removing the option of using one chunk and
*forcing* every consumer to use streaming seems mis-guided.

> get disabled, and decrease the security of the whole system.
I get your point. You don't want the security properties of AE. That's
fair. And you assume that implementations will actively misinterpret the
spec. While this is a bit pessimistic it may indeed reflect real-life.
Do you think that defining a non-streaming mode and a streaming mode
would work?  Then clients wouldn't need to feel bad about releasing
plaintext early, but can use the streaming mode instead.

Cheers,
  Tobi

---

From: Derek Atkins <[email-masked]>
Date: Wed, 27 Mar 2019 11:50:32 -0400

Tobias Mueller <[email-masked]> writes:

[snip]
> for releasing plaintext early is not immediately covered by this

In my mind, each chunk is its own AEAD ciphertext.  So the chunking is
happening *during* AEAD encryption, and not after encryption.  I.e., the
chunking and AEAD encryption should be tied together such that the chunk
header is part of the AEAD protection and the chunk data is the AEAD
encrypted data.

This approach does, IMHO, map directly into the RFC definition.

-derek

---

From: "Neal H. Walfield" <[email-masked]>
Date: Wed, 27 Mar 2019 21:11:12 +0100

Hi Derek,

On Wed, 20 Mar 2019 14:24:22 +0100,
> the receiver can signal to the sender what it should do.

I've spent some time thinking about use cases for different chunk
sizes, and I can't come up with any modulo some, IMHO, insignificant
performance tweaks.  Can you please give some examples of use cases
that would profit from different chunk sizes?

> a sender that doesn't pay attention.

If I understand you correctly, you would support a SHOULD restriction
on the the range, but not a MUST restriction.

What should / would you recommend an implementation do if it
encounters a chunk that it can't buffer?  I see two choices: report an
error, or release unauthenticated plaintext.

Please don't misunderstand my questions: I sincerely am interested in
your answers to these questions.

Thanks!

:) Neal

---

From: Justus Winter <[email-masked]>
Date: Thu, 28 Mar 2019 13:30:16 +0100

Hello,

I'm interested in building robust systems.  To do that, I need to
consider memory consumption of both my code, and the code that I'm
using, say in libraries.  I consider that important in general, but
even more so if the application in question is handling sensitive
data, and transporting it over untrusted networks, therefore it is
necessarily exposed to potentially malicious data.

In the context of processing OpenPGP data, currently there is no
relation between the size of the encrypted message and the size of the
decrypted message.  This is due to compression.  Recently, we
discussed compression on this list, and it does not look like we will
be phasing it out anytime soon.

https://mailarchive.ietf.org/arch/browse/openpgp/?index=dFS9kySMJCcOlR9Svys9QwP7I7Y&gbt=1

For me, using an unbounded amount of memory is not an option for a
component processing OpenPGP data if we want to build robust systems
on top.

Therefore, we need to process OpenPGP data in bounded space.  Since
there can be no relation between encrypted and decrypted message size
due to compression, the only option I see is to provide a streaming
API, which let's us process data in constant space.

[Now, when I say constant space, implementations could still decide to
use, say, 30 megabytes of buffer space.  Then, most emails will fit
into this buffer, and we can detect truncated messages before we hand
out one byte to the downstream application.  This is what we do in
Sequoia.  Note, however, that the consumer decides how much data to
buffer before releasing the first data, and not the producer.  If we
decide to even allow 128 megabyte chunks, than the producer can
*force* the consumer to allocate 128 megabytes, or either not process
the message or do it unsafely.]

Now, as efail demonstrated, we need to protect against ciphertext
modifications, and we need to do it in a way that does not bring back
the problems with requiring unbounded space that we're trying to
address with streaming in the first place.

Therefore, we need to use chunking and authenticate message prefixes.
We need to use chunks that are reasonably small, and this size should
preferably not be configurable.  We should consider performance
constraints and pick one suitable size.  Configurable chunk sizes
bring complexity and increase the attack surface, as was pointed out
in this thread.

The only argument for a configurable chunk size that came out of this
thread is to be able to fit the entire message into one chunk.

I appreciate the desire to protect against truncation.  But,
truncation is pretty common when we transmit data, so I'd argue that
application developers are more likely to expect and gracefully deal
with truncated data than with ciphertext being manipulated or the PGP
implementation consuming unbounded amounts of memory.

Now, you may say that even if the PGP implementation doesn't buffer
the plaintext, the downstream consumer must buffer it in order to
detect truncation.  But that is not always true.  As pointed out in
this thread, you can use some kind of transaction scheme to only
commit data once it has been confirmed to be not truncated.

I have implemented AEAD in Sequoia, and I have evaluated the
implementations in GnuPG and RNP.  Every implementation is either
unsafe, not robust, or does not implement the proposal.

What is proposed in RFC4880-bis06 can not be implemented safely.  If the
working group produces a standard that cannot be implemented safely, I
consider that a grave failure of the standardization effort.

Thanks,
Justus

---

From: Bill Frantz <[email-masked]>
Date: Thu, 28 Mar 2019 11:54:23 -0700

>component processing OpenPGP data if we want to build robust systems

Can't you follow Jon's advice:

On 3/20/19 at 12:36 PM, joncallas=[email-masked] 
(Jon Callas) wrote:

>To address your point, as I said in my long missive, you can do 
>this today. No changes are needed to the protocol. All you have 
>to do is put a compression preference on your key that says no 
>compression, and then you won’t get compression. (Well, to be 
>completely correct, if someone compresses then they’re 
>non-compliant to the standard.) Repeating myself, I support and 
>encourage implementations to do that by default.

You can then treat any message that uses compression as 
malicious and refuse to process it.

Cheers - Bill

-------------------------------------------------------------------------
Bill Frantz        | When it comes to the world     | Periwinkle
(408)356-8506      | around us, is there any choice | 16345 
Englewood Ave
www.pwpconsult.com | but to explore? - Lisa Randall | Los Gatos, 
CA 95032

---

From: Jon Callas <[email-masked]>
Date: Thu, 28 Mar 2019 12:33:39 -0700

> On Mar 27, 2019, at 8:50 AM, Derek Atkins <[email-masked]> wrote:
> 

This is exactly what I presumed would be done — each chunk is an AEAD segment. I presumed that one would probably put a chunk number as Additional Data, and that the nonce context would carry over from one chunk to the next in some reasonable way.

That’s directly analogous to the present chunking mechanism, which uses CFB as a stream.

	Jon

---

From: Jon Callas <[email-masked]>
Date: Thu, 28 Mar 2019 14:27:27 -0700

> On Mar 28, 2019, at 5:30 AM, Justus Winter <[email-masked]> wrote:
> 

[…]

> decrypted message.  This is due to compression.  

This isn’t precisely true. Certainly, compression is the biggest factor here, but it is not the only one. There are many factors that make it hard to know the finished size of an OpenPGP message a priori. These include ASCII armor, TEXT mode plaintext, and others. It only gets worse inside the plaintext where there is typically in emails quoted-printable, further base64, both, and other bits of brain damage caused by the accretion of many things that were good ideas at the time.

> component processing OpenPGP data if we want to build robust systems

Okay, prior to this working group, when there was running code without a consensus rough or not, this problem existed. Even with compression, PGP 2 ran on DOS machines with a max of 640K of RAM. There are many similarly constrained systems that run OpenPGP implementations.

> API, which let's us process data in constant space.

I’m not quite sure what you mean when you say “bounded space” because one interpretation of that is obviously false. OpenPGP has always supported being able to process messages where the encryptor does not know the size ahead of time. That’s why we have indeterminate lengths and chunking.

I presume you mean that the implementation has to have constraints on its resources. This is certainly true; there are no unconstrained systems. It’s also true that there are going to be messages that your implementation can’t process well. For example, RFC 4880 allows a partial body length (a chunk) to be 2^30 octets, and that could be irritating to handle.

One of (perhaps unstated) goals of OpenPGP is that it allow for highly constrained implementations. This was a huge consideration in both 2440 and 4880. There are things that are designs the way they are because the working group felt strongly that things have to be one-pass. There were many debates about the MDC that boil down to it (and this is also the reason why HMAC wasn’t used, but there more history there, including that while HMAC existed when the MDC was designed, we did not yet have a proof of security for it.

> the message or do it unsafely.]

Or the consumer could return an error and say it can’t decode it.

> address with streaming in the first place.

Efail is primarily a problem with MIME encoding and layering violations. It works just as much with S/MIME as OpenPGP. Perhaps I’m missing something, but I don’t see how Efail is relevant to resource bounds.

> bring complexity and increase the attack surface, as was pointed out

I’m with you on a lot of this, but I don’t know what you mean by “configurable”? Do you mean that there should be one chunk size only? If so, what size do you propose? 32 Meg or thereabouts (2^25 is in that ball park)? If so, would that mean that all messages smaller than your chunk size would be a single chunk? 

> thread is to be able to fit the entire message into one chunk.

That’s not the way I understand the discussion. The way I understand it, there are people who desire to have single-chunk messages of a rather large size. At present, the non-AEAD chunks can be any power of 2 up to 2^30 (but the first one has to be at least 2^9). I don’t see the request for variable (is that the same thing as configurable?) chunk sizes to be anything other than the analogue of the present situation.

> implementation consuming unbounded amounts of memory.

Does this mean that you think that message truncation is an error that OpenPGP doesn’t need to guard against?

That’s the way I interpret the first line in the paragraph above (“I appreciate … But,…”). If so, that’s counter to the long-standing consensus of the working group. It’s the whole reason we have MDCs and the reason why they were aggressively pushed in the implementations and non-MDC packets browbeaten into doing MDCs. See the non-normative discussion in section 5.13 of 4880.

> commit data once it has been confirmed to be not truncated.

I think I understand. Are you noting that because of the one-pass nature of OpenPGP, it’s possible to process arbitrary amounts of data and not know that there’s an error until the end? This is certainly true of MDCs, because of the one-pass desire. If you make an implementation that has AEAD chunks, it’s possible that you could be processing correct chunks for an indefinite amount of time, and then get an AEAD failure that calls into question the integrity of the whole stream that led up to that.

Is that what you’re pointing out?

> unsafe, not robust, or does not implement the proposal.

Tell us more. What problems did you find?

> consider that a grave failure of the standardization effort.

Okay, you’ve lost me.

What can’t be implemented safely and why?

In my reading of this, I think I have identified two points you’re making.

(1) It’s possible for a chunk to be larger than reasonable processing resources.
(2) It’s possible for a long stream to have an error in the last chunk that signals an error wayyyyyy in the past.

Handling (1) is reasonably easy. Return an error. This situation exists today. It’s possible to make partial bodies of a gigabyte each, and an implementation may not be able to handle that. Return an error.

Handling (2) is also easy, you return an error. This might be unsatisfying, because the error might be in the past, and lots of stuff already handled. Is this your objection?

	Jon

---

From: "Neal H. Walfield" <[email-masked]>
Date: Thu, 28 Mar 2019 23:04:56 +0100

At Thu, 28 Mar 2019 14:27:27 -0700,
Jon Callas wrote:
> Okay, prior to this working group, when there was running code
> without a consensus rough or not, this problem existed. Even with
> compression, PGP 2 ran on DOS machines with a max of 640K of
> RAM. There are many similarly constrained systems that run OpenPGP
> implementations.

Until now, OpenPGP didn't require buffering data.  A decrypted AEAD
chunk MUST only be released when it has been authenticated.  In the
current proposal, AEAD chunks are potentially unbounded (well, up to 4
exabytes...) in size.  No one can decrypt such chunks without
cheating, i.e., releasing unauthenticated plaintext.

> I’m not quite sure what you mean when you say “bounded space”
> because one interpretation of that is obviously false. OpenPGP has
> always supported being able to process messages where the encryptor
> does not know the size ahead of time. That’s why we have
> indeterminate lengths and chunking.

[snip]
>  For example, RFC 4880 allows a partial body length (a chunk) to be
>  2^30 octets, and that could be irritating to handle.

indeterminate lengths and partial body chunking don't prohibit the
implementation from releasing the text early; there is no requirement
to buffer the whole chunk.

> One of (perhaps unstated) goals of OpenPGP is that it allow for
> highly constrained implementations. This was a huge consideration in
> both 2440 and 4880.

We are arguing within this tradition.  We want highly constrained
systems to be able to process (nearly) all OpenPGP messages in a
*conforming* manner.  AEAD mandates buffering whole chunks, because
unauthenticated plaintext MUST NOT be released.  By limiting chunks to
some kilobytes in size, highly constrained systems will be able to
process these messages without releasing unauthenticated plaintext.

> Efail is primarily a problem with MIME encoding and layering violations. It works just as much with S/MIME as OpenPGP. Perhaps I’m missing something, but I don’t see how Efail is relevant to resour…

EFAIL is also about ciphertext modification.  If an implementation
never releases unauthenticated plaintext, then ciphertext modification
is not possible.  Highly constrained systems can do this if the chunk
size is limited to something they can handle.

> I’m with you on a lot of this, but I don’t know what you mean by “configurable”? Do you mean that there should be one chunk size only? If so, what size do you propose? 32 Meg or thereabouts (2^25 i…

Configurable means that the chunk size is not fixed by the standard.
There are two proposals right now: we either fix the chunk size to
16kb (support by Justus, Sebastian and I) or to 256kb (preferred by
Bart from Proton Mail, since that is what they are using in practice).

> That’s the way I interpret the first line in the paragraph above (“I appreciate … But,…”). If so, that’s counter to the long-standing consensus of the working group. It’s the whole reason we have M…

Either you stream or you protect against truncation.  I don't see how
you can do both.  That doesn't mean you can't detect truncation and
signal that it occurred.  Indeed, that is a property that MDC and
signatures provide.  But, then it is up to the application to recover.
(Perhaps the application, with or without the help of the OpenPGP
implementation, simply buffers the whole message or a sufficiently
large prefix.  But, that is built on top.  We want OpenPGP to remain
streamable.)

> Tell us more. What problems did you find?

Sequoia is written in Rust, and in Rust out of memory errors are
fatal.  Handling a 4 exabyte chunk would result in an out of memory
error, and termination of the process.

Both GnuPG and RNP process unauthenticated plaintext.  They release
the plaintext for further processing before it is authenticated by the
AEAD tag.

> Handling (1) is reasonably easy. Return an error. This situation exists today. It’s possible to make partial bodies of a gigabyte each, and an implementation may not be able to handle that. Return…

I don't understand why partial bodies are relevant here.  They are
quite easy to implement using a small buffer even given large partial
body chunks.

The issue with returning an error in the face of large AEAD chunks is
that it is completely gratuitous.  If the sender had only used a small
chunk size, it would have been possible to authenticate the
ciphertext!

What is the advantage of large AEAD chunks (serious question!)?  It
seems their only effect is to encourage the release of unauthenticated
plaintext.

> Handling (2) is also easy, you return an error. This might be unsatisfying, because the error might be in the past, and lots of stuff already handled. Is this your objection?

Supporting stream means allowing for truncation.  We want streaming,
and we are prepared to handle truncation should it occur.

Thanks,

:) Neal

---

From: Bart Butler <[email-masked]>
Date: Thu, 28 Mar 2019 23:47:48 +0000

Hi Neal,

Maybe a reasonable compromise is, as we discussed a while ago, simply an upper bound on the chunk size rather than fixing it to a single value. What's a reasonable upper bound for constrained systems? 4MB?

And for others: what IS the use case for really large chunks? In the limit of the message being one giant chunk, AEAD becomes essentially our current MDC and loses all additional value it has in terms of streaming, which is detecting errors/modifications early rather than at the last possible moment. Why allow producers to undermine this, given its the reason for its existence?

I agree with Jon that it's difficult to foresee future use cases, which is why I don't favor removing the size entirely--if some compelling use case for larger sizes appears, or for having a variety of small sizes (more likely IMHO), then great.

But I have yet to see a compelling use case for really large AEAD chunks, and two compelling reasons not to have them (embedded systems and general poor UX in terms of not being able to error out early). So do we really not want to do some kind of reasonable upper limit in the RFC? It would be one sentence in the RFC and minimal work for existing implementations to cap the AEAD chunk size to reasonable value.

-Bart

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

> conforming manner. AEAD mandates buffering whole chunks, because

---

From: Marcus Brinkmann <[email-masked]>
Date: Fri, 29 Mar 2019 00:58:30 +0100

The main question here is: What should a conforming application look like?

The current behaviour of GnuPG is that it will process internally (e.g.,
through the decompression and signature verification layer) and output
externally unauthenticated plaintext.  If an AEAD chunk is modified by
an attacker, GnuPG will detect the modification and cancel the
operation, but only at the end of each chunk.  Due to the asynchronous
buffer management in GnuPG, quite often some part of the modified chunk
has then already been processed and output, depending on the particular
state of the buffers, the buffer size and the chunk size.  This
behaviour increases the surface for chosen ciphertext attacks and
possibly adaptive chosen plaintext attacks (if an oracle is exposed).

The criticism here is mainly targeted at the following reasoning: If the
AEAD chunk size is not bounded to reasonable length, any conforming
implementation will have to implement a mode of operation that is
similar to the one currently implemented in GnuPG.  As a consequence, to
decrease development cost, most implementations will _only_ have this
one implementation (as opposed to two implementations, one for large
chunks and one for reasonably short chunks).

This assumption is strengthened by the observation that GnuPG is
de-facto norm-setting, due to its popularity and the lack of
standardization activity.

Again: What should a conforming implementation look like?  The current
draft proposal will lead, I think, to a world where most conforming
OpenPGP implementations will never benefit from the non-malleability
properties of AEAD cipher modes.  A proposal restricting the chunk
length to a reasonable value will lead, I hope, to a world where most
OpenPGP implementations will benefit from the non-malleability of AEAD.

There are many other areas where the OpenPGP standard does not specify
reasonable lengths, but implementations do impose such restrictions.  In
my opinion, this is a mistake rather than a precedent to follow.

Thanks,
Marcus

-- 
Dipl.-Math. Marcus Brinkmann

Lehrstuhl für Netz- und Datensicherheit
Ruhr Universität Bochum
Universitätsstr. 150, Geb. ID 2/461
D-44780 Bochum

Telefon: +49 (0) 234 / 32-25030
http://www.nds.rub.de/chair/people/mbrinkmann

---

From: Bill Frantz <[email-masked]>
Date: Thu, 28 Mar 2019 17:10:23 -0700

On 3/28/19 at 4:47 PM, 
bartbutler=[email-masked] (Bart Butler) wrote:

>Maybe a reasonable compromise is, as we discussed a while ago, 
>simply an upper bound on the chunk size rather than fixing it 
>to a single value. What's a reasonable upper bound for 
>constrained systems? 4MB?

The Arduino Uno, which the web site says is the most popular 
Arduino in the line 
<https://store.arduino.cc/usa/arduino-uno-rev3>, has:

  Flash Memory 32 KB (ATmega328P) of which 0.5 KB used by bootloader
  SRAM 2 KB (ATmega328P)
  EEPROM 1 KB (ATmega328P)

So it might be able to use a chunk up to 1KB without having to 
do the kind of pipelining that leads to security bugs and messy code.

YMMV!

In general, when asked about the smallest target for crypto 
algorithms, I think it was in the cryptography mailing list, the 
consensus was an 8 bit microprocessor with 64K of addressing 
would always be the target. People will always find a use for 
cheaper and smaller processors and 8 bits with 64K addresses 
seems to be where we have settled.

Cheers - Bill

-----------------------------------------------------------------------
Bill Frantz        | Truth and love must prevail  | Periwinkle
(408)356-8506      | over lies and hate.          | 16345 
Englewood Ave
www.pwpconsult.com |               - Vaclav Havel | Los Gatos, 
CA 95032

---

From: Jon Callas <[email-masked]>
Date: Thu, 28 Mar 2019 19:19:39 -0700

I wrote a point-by-point reply and decided that that’s not productive. I’m going to try to cut to the chase on this, so forgive me if I have dodged an important point. I’m happy to come back to it later.

Like some interim replies, particularly Bart Butler, I thought we had a rough consensus and that it was approximately:

* MUST support 16KB chunks.
* SHOULD support 256K chunks, as these are common (Protonmail).
* MAY support larger up to the present very large size.
* MAY reject or error out on chunks larger than 16KB, but repeating ourselves, SHOULD support 256K.

I think it’s also reasonable to just simplify it by making be MUST support up to 256K. I don’t like getting rid of the SHOULD clause, thus making it be either 16K or off in squishy land, but I wouldn’t wring my wrists, I’d merely be a bit unhappy.

I could also get behind a hard limit of 2^30 on the grounds that that’s what we had for partial body lengths, but I understand the comment that there are things like multi-terabyte S3 buckets and out and out forbidding those to be single-chunk is a bit churlish, but only a bit.

If we really want to tie a bow around everything, then define some notation or other marker so that an implementation can mark in a key what the max chunk size is.

How’s that?

Now with some other points, I found myself going especially Socratic because the missives that you and Justus wrote weren’t *actionable*. When I was editor, a thing I would say from time to time is at about 90% of the time someone would say, “Please change X to Y for reason Z” it would be a no-brainer to do so. Most of the remaining 10% would end up with some reasonable debate than then an answer. But if someone said, “I think X is wrong” then 90% of the time nothing would happen because there’s nothing actionable there, meaning that there’s no clear point for the editor to be doing. All of these standards are at some level compromises and there’s an old aphorism that the best compromise is one where everyone is equally unhappy.

In my particular case, I thought I mostly agreed with where you wanted to go, but I completely disagreed with some of your premises, and disagreed with some of the reasoning that got us to the conclusion I agreed with. That’s why I was being so Socratic. I thought before the missives today that we had a rough consensus and that I agreed with y’all at some basic level. Then I thought we really disagreed, and I gradually realized that we agree on the destination, but not on the path of how to get there.

If I may give some advice, it is better to make sure that your requests are actionable. If they’re not, then you’re less likely to get what you want, if for no other reason than the rest of us don’t know what you want, we’re having to guess. The basic argument that you were making — that implementations such as yours need tight constraints — is one we all can get behind. The other side here — they want to do something kinda wacky (like terabyte chunks) — can be pushed off into the land of MAYs. 

Okay, anyway.

How’s that proposal I wrote above? Does it work for you? Does it need tweaks? 

	Jon

---

From: Peter Gutmann <[email-masked]>
Date: Fri, 29 Mar 2019 03:06:15 +0000

Jon Callas <joncallas=[email-masked]> writes:

>Okay, prior to this working group, when there was running code without a
>consensus rough or not, this problem existed. Even with compression, PGP 2
>ran on DOS machines with a max of 640K of RAM.

The constraint there was 64K, not 640K, the x86 segment size.  Also, the way
PGP then handled it was to use intermediate files for everything, every single
transformation wrote its own intermediate file to disk.  Including, in the
worst case, prepending a single literal byte to the data.

That's probably not a good example to cite :-).

Peter.

---

From: Peter Gutmann <[email-masked]>
Date: Fri, 29 Mar 2019 03:17:03 +0000

Neal H. Walfield <[email-masked]> writes:

>Until now, OpenPGP didn't require buffering data.  A decrypted AEAD chunk
>MUST only be released when it has been authenticated.  In the current
>proposal, AEAD chunks are potentially unbounded (well, up to 4 exabytes...)
>in size.  No one can decrypt such chunks without cheating, i.e., releasing
>unauthenticated plaintext.

This has been considered before, e.g. with S/MIME's authenticated encryption:

https://tools.ietf.org/html/rfc6476#section-6

and so far doesn't seem to have caused any major problems.  That is, it's not
that there's a perfect solution, it's that actual problem situations seem to
be pretty rare.

If you want to do it right, you'd really want some formal academic treatment
rather than guessing at chunk sizes and what may or may not be needed, i.e.
typical message size X, typical chunk size Y gives these security bounds.  PGP
is typically used to encrypt data at rest (make the chunk size the file size)
or short email messages (chunk size doesn't matter, it's short).  That leaves
a remainder of large emails, which we know exist but don't know how frequent
they are or how often they're sent or from what sorts of systems.

Without hard data on what's actually needed, we're just bikeshedding... while
blindfolded.

Peter.

---

From: Peter Gutmann <[email-masked]>
Date: Fri, 29 Mar 2019 03:29:58 +0000

I wrote:

>PGP is typically used to encrypt data at rest (make the chunk size the file
>size)

Another thing with that particular case, if you get a MAC failure decrypting
data at rest do you really care? It's more likely a bit-flip somewhere than
someone trying to tamper with your archived sales records from 2003, and I
suspect most people would rather have slightly corrupted data than no data at
all.

That's the nice thing about the standard block cipher modes, they recover from
errors.  In... oh, approximately 100% of the encrypted data I have lying
around, I'll happily ignore any auth errors, I just want the plaintext back.
So while it's fun and geeky to dream up something using the latest trendy AEAD
modes, is it something that (a) is necessary and (b) people who aren't geeks
care about?

Peter.

---

From: Peter Gutmann <[email-masked]>
Date: Fri, 29 Mar 2019 03:33:03 +0000

Bill Frantz <[email-masked]> writes:

>The Arduino Uno, which the web site says is the most popular Arduino in the
>line
>So it might be able to use a chunk up to 1KB without having to do the kind of
>pipelining that leads to security bugs and messy code.

And where does the PGP code and data memory itself fit in all this?

Peter.

---

From: Bill Frantz <[email-masked]>
Date: Thu, 28 Mar 2019 21:39:54 -0700

>And where does the PGP code and data memory itself fit in all this?

Well, we had a version of PGP running on an original IBM PC. 
With careful implementation, you might get the code into 32K 
program memory using the 2K R/W memory for buffers and working 
memory. You also might slip implementing all of the SHOULDs and 
perhaps some of the inappropriate MUSTs. You would probably have 
to also always make the tradeoff for space and not performance.

Remember, the original question was asked by an enbedded system 
developer. How small do they go? If they're looking at Raspberry 
Pi size machines, then they really have it comparatively easy.

Cheers - Bill

---

From: "Neal H. Walfield" <[email-masked]>
Date: Fri, 29 Mar 2019 10:07:54 +0100

On Fri, 29 Mar 2019 03:19:39 +0100,
> * MAY reject or error out on chunks larger than 16KB, but repeating ourselves, SHOULD support 256K.

I still think this is a bad idea.  And my discussions offline with
Justus and Marcus suggest that they also don't agree with this.  As I
understand Sebastian's position, he also doesn't agree:

  https://mailarchive.ietf.org/arch/msg/openpgp/SfVIqqwtYnBSOyhmdh8DnoFtlEw

Bart and Sunny are also for reducing the range of allowable chunk sizes:

  https://mailarchive.ietf.org/arch/msg/openpgp/VN-lWmTK6JXxPKnluO7IbQJT52Y

The proposal that I put forward removed the chunk size byte and
standardized a single, fixed chunk size.  I suggested 16kb, but given
Bart's arguments, I was willing to compromise on 256kb:

I also suggested using a different packet tag, and marking 20 as
reserved to avoid breaking users of the current AEAD packet:

  https://mailarchive.ietf.org/arch/msg/openpgp/2SitJh7vA4cHvWGkbAMSzf1Zkio
  https://mailarchive.ietf.org/arch/msg/openpgp/WvaisWDd9WzoKXL1FyHDeZNtZ2U

> I could also get behind a hard limit of 2^30 on the grounds that that’s what we had for partial body lengths, but I understand the comment that there are things like multi-terabyte S3 buckets and o…

This is the bit that I don't understand.  Clearly you see some
advantage to having large chunk sizes.  I've asked several times on
this list, but no one can tell me what concrete advantages large chunk
sizes have.  (Tobias has suggested completely removing chunking, which
does protect against truncation attacks, but it sacrifices the ability
to work with O(1)-sized buffers, i.e., streaming.)  Can you please
tell me why large chunk sizes are helpful?

> If we really want to tie a bow around everything, then define some notation or other marker so that an implementation can mark in a key what the max chunk size is.

Please don't add even more complexity!  Again, what are the advantages
of choosing one chunk size over another?

Ferguson, Schneier and Kohno write in "Cryptography Engineering":

  "There are already enough insecure fast systems; we don't need
  another one" (2.9)

and

  "The more complex a system, the more likely it has security
  problems...  complexity is the worst enemy of security" (2.10)

I feel like making the chunk size configurable adds complexity whose
only justification is some hand wavy "well, in the future it might
help...".  Please help me understand why large/variable chunk sizes
are useful.

> Now with some other points, I found myself going especially Socratic
> because the missives that you and Justus wrote weren’t
> *actionable*. When I was editor, a thing I would say from time to
> time is at about 90% of the time someone would say, “Please change X
> to Y for reason Z” it would be a no-brainer to do so. Most of the
> remaining 10% would end up with some reasonable debate than then an
> answer. But if someone said, “I think X is wrong” then 90% of the
> time nothing would happen because there’s nothing actionable there,
> meaning that there’s no clear point for the editor to be doing. All
> of these standards are at some level compromises and there’s an old
> aphorism that the best compromise is one where everyone is equally
> unhappy.

I don't understand why you think I haven't been concrete :/.  In my
original email, I provided a concrete patch to 4880bis:

  https://mailarchive.ietf.org/arch/msg/openpgp/XH098WlJe8lkOypIaB1IXTde2sY

But, let me try to use your framing:

I propose that we remove the chunk size parameter from the AEAD header
and fix it to a small value (e.g., 64 KB or 256 KB), because 1.) no
one has shown that a large chunk size is useful, 2.) large chunk sizes
encourage implementations to release unauthenticated plaintext, which
is a serious security concern [1], 3.) if an implementation releases
unauthenticated plaintext for large chunks, then an attacker can
always convince it to release unauthenticated plaintext for the first
chunk in a message [2], 4.) making the chunk size variable increases
complexity, which is a security concern.

  [1] https://mailarchive.ietf.org/arch/msg/openpgp/fmQgRm94jhvPLEOi0J-o7A8LpkY
  but RNP appears to do the same.
  [2] https://mailarchive.ietf.org/arch/msg/openpgp/WvaisWDd9WzoKXL1FyHDeZNtZ2U,
  starting at "Let's assume..."

> In my particular case, I thought I mostly agreed with where you wanted to go, but I completely disagreed with some of your premises, and disagreed with some of the reasoning that got us to the conc…

As I understand your position, you want to allow implementations a
broad degree of flexibility in choosing the chunk size.  Can you
please explain why this is useful?  I've reread your messages, but
beyond what appears to me to be some hand wavy performance argument,
and an apparent misunderstand (chunk size != message size) [3],
neither of which are convincing, I can't find any other arguments.
I'm sorry if you did and I didn't understand.

  [3] I believe that if we limit it to 16K, we *will* regret that, for
  performance or other reasons. And while some of the upper sizes are
  presently absurdly large, one never knows what happens a couple
  decades from now.

  Moreover, forbidding it says that we state now that no one could
  ever have a reasonable use; my experience is that categorical
  statements like that are just asking the fates to bite us in an
  uncomfortable place. Amazon S3 increased their max object size to
  5TB a few years ago. I’m not saying it should be that large, but I
  think this is a pretty convincing argument that 16K is too small.

  https://mailarchive.ietf.org/arch/msg/openpgp/PLbg_sFze2H1INrwz_l1Uz1tPGE

> If I may give some advice, it is better to make sure that your requests are actionable. If they’re not, then you’re less likely to get what you want, if for no other reason than the rest of us don’t know what you want, we’re having to guess. The basic argument that you were making ― that implementations such as yours need tight constraints ― is one we all can get behind. The other side here ― they want to do something kinda wacky (like terabyte chunks) ― can be pushed off into the land of MAYs. 

I'm a bit confused.  In my original mail, I included a PR:

  https://mailarchive.ietf.org/arch/msg/openpgp/XH098WlJe8lkOypIaB1IXTde2sY

Was that not actionable enough?  Should I link to the PR more often?

I feel that I need answers to two questions regarding the "allow large
chunk size" position.  I recently asked them, but I have gotten a
response.  Here they are again:

  What should / would you recommend an implementation do if it
  encounters a chunk that it can't buffer?  I see two choices: report
  an error, or release unauthenticated plaintext.

  https://mailarchive.ietf.org/arch/msg/openpgp/m9hd4FwdvIUWKt1MWQfzBq3JahY

Regarding our implementation: it doesn't actually have tight resource
constraints.  Our primary goal is to write a secure implementation.
From that goal, we derived the constraint that we must always work in
a bounded amount of space.  The current version of AEAD doesn't allow
this without potentially rejecting messages that other implementations
can process by being insecure (releasing unauthenticated plaintext),
which we don't want to do.

But, my concerns are not only about my implementation.  I'm concerned
about the ecosystem.  And, the current proposal encourages insecure
implementations as demonstrated by GnuPG and RNP processing
unauthenticated plaintext.  I think the standard should not
proactively make it harder to write a secure implementation.  And that
is what I see the AEAD chunk size doing.

Thanks,

Neal

P.S. I hope it is clear that I'm trying to engage in a constructive
manner.  I sincerely haven't understand your position, and I really
want to.

---

From: "Neal H. Walfield" <[email-masked]>
Date: Fri, 29 Mar 2019 10:23:27 +0100

On Fri, 29 Mar 2019 04:17:03 +0100,
> that there's a perfect solution, it's that actual problem situations seem to

The advice seems pretty good:

   The exact solution to these issues is somewhat implementation-
   specific, with some suggested mitigations being as follows:
   implementations should buffer the entire message if possible and
   verify the MAC before performing any decryption.  If this isn't
   possible due to streaming or message-size constraints, then
** implementations should consider breaking long messages into a
** sequence of smaller ones, each of which can be processed atomically
** as above.  If even this isn't possible, then implementations should
   make obvious to the caller or user that an authentication failure has
   occurred and that the previously returned or output data shouldn't be
   used.  Finally, any data-formatting problem, such as obviously
   truncated data or missing trailing data, should be treated as a MAC
   verification failure even if the rest of the data was processed
   correctly.

It seems to me that we can take this advice at the protocol layer and
largely avoid the security concerns.  We just always use small
chunks.

> they are or how often they're sent or from what sorts of systems.

I'm having trouble imagining why a larger chunk size would ever be
better in either of these cases.

  - File encryption: smaller chunk size means finding errors faster

  - Large emails: smaller chunk size means that it is possible to
    preview the email, which is helpful on mobile connections.

Please help me understand when a large chunk size could be better.

Thanks,

:) Neal

---

From: Marcus Brinkmann <[email-masked]>
Date: Fri, 29 Mar 2019 10:52:27 +0100

Hi,

Just to set the record straight: I made two very specific actionable
proposal on this very list 9 months ago.

* Limit the maximum chunk size to a small value:

https://mhonarc.domainunion.de/archive/html/ietf-openpgp/2018-06/msg00029.html

* Forbid outputting unauthenticated plaintext:

https://mhonarc.domainunion.de/archive/html/ietf-openpgp/2018-06/msg00030.html

Also, I think it is instructive to look at the history of the chunk size
and how we got here in the first place. This is the original proposed
text by Brian M. Carlson:

> An implementation MUST support chunk size octets with values from 0 to
10.  An implementation MAY support other chunk sizes.  Chunk size
octets with values larger than 127 are reserved for future extensions.

https://gitlab.com/bk2204/rfc4880bis/commit/353520abd5be34d9980a0f1ea77a07ba1837d03a

This is what the editor put into the draft standard without discussion:

> An implementation MUST support chunk size octets with values from 0 to
56.  Chunk size octets with other values are reserved for future
extensions.

https://mhonarc.domainunion.de/archive/html/ietf-openpgp/2017-07/msg00010.html

His reasoning was this: "Given that larger values are optional,
implementations will need limit C to 10.  I consider this too low for
practical purposes.  We should require all implementations to support
the same range. Given that we have a 64 bit counter the maximum value
for C should be 57 - I would even say 56 so that we avoid signed and
signed problems in the number of octets."

So, here is an actionable item: Go back to the original proposal by
Brian M. Carlson. It gives implementations a reasonable limit to stick
to, while it allows for larger chunks for special use cases.

Thanks,
Marcus

---

From: Marcus Brinkmann <[email-masked]>
Date: Fri, 29 Mar 2019 12:47:52 +0100

> If we really want to tie a bow around everything, then define some notation or other marker so that an implementation can mark in a key what the max chunk size is.

FWIW, I think that's reasonable. It is very similar to the original
OpenPGP AEAD proposal by Brian M. Carlson, which just had "MUST support
up to 64KB chunks" and "MAY support larger chunks".  That original
proposal is simpler than yours, but given that Protonmail already uses
larger chunk sizes, that's ok.  From the perspective of mobile and
desktop devices, 64KB and 256KB are similarly reasonable.

There is a little bit of tension here, because any increase in the MUST
area may give trouble to constraint devices, who should be able to hold
a single chunk in memory if they are interested in non-malleability.
The difference between 16KB and 256KB can matter here.  So, I guess I
understand why you introduced the SHOULD requirement, although it makes
things more complicated.

Thanks,
Marcus

---

From: Peter Gutmann <[email-masked]>
Date: Fri, 29 Mar 2019 12:53:42 +0000

Neal H. Walfield <[email-masked]> writes:

>I'm having trouble imagining why a larger chunk size would ever be better in
>either of these cases.
>

See my followup messages, for data at rest you probably don't care about
errors at all, and if you really do then having the report at the whole-file
level is fine.  What's the benefit gained from knowing that the 64kB block at
offset xyz is corrupt?

As I mentioned earlier, we really need some data on real-world use cases
rather than hypothesising problematic corner cases that will rarely, if ever,
occur.

Peter.

---

From: "Neal H. Walfield" <[email-masked]>
Date: Fri, 29 Mar 2019 14:20:21 +0100

On Fri, 29 Mar 2019 13:53:42 +0100,
> level is fine.  What's the benefit gained from knowing that the 64kB block at

But what is the cost?  I would say there is basically none.  So it
makes no sense to me to optimize for this case.  It's irrelevant.

So, why make the important case (when you actually do want to stream)
more complicated?

> rather than hypothesising problematic corner cases that will rarely, if ever,

Efail occured.  Why is that not enough?

---

From: Peter Gutmann <[email-masked]>
Date: Fri, 29 Mar 2019 13:43:35 +0000

Neal H. Walfield <[email-masked]> writes:

>But what is the cost?  I would say there is basically none.  So it makes no
>sense to me to optimize for this case.  It's irrelevant.

There is a significant cost in terms of implementing, debugging, and interop-
testing every implementation that wants to do this.  If no-one cares about
auth protection of data at rest, and in the complete absence of real-world
data I'm going to claim no-one does because you can't prove otherwise, using
what we currently have has zero cost because it's already implemented.  Adding
blocked auth protection has a distinctly nonzero cost.

>Efail occured.  Why is that not enough?

That was due to broken email apps.  If I can convince your email app to
forward the plaintext of a decrypted message to me, you lose no matter what
encryption mechanism you use.

Admittedly CBC/CFB made this easier, but it was the email apps that needed
fixing, not PGP.

I'm not saying it's not worth addressing, but before endlessly debating
solutions we need to figure out what problem we're solving.  "We have this
cool AEAD mode lying around and want to apply it to something" isn't a
problem, or at least not a problem that a PGP user cares about solving, it's
something interesting for geeks to play with.

In the last five years or so I've received approximately zero PGP-encrypted
emails, and I'm one of the people who helped write the thing.  OTOH I've
probably encrypted gigabytes of data with it, almost always symmetric-key with
the key communicated out of band.  In none of those cases would blocked auth
protection have been useful.

Peter.

---

From: "Neal H. Walfield" <[email-masked]>
Date: Fri, 29 Mar 2019 15:37:12 +0100

At Fri, 29 Mar 2019 13:43:35 +0000,
> blocked auth protection has a distinctly nonzero cost.

Ok.  Is your position that the working group should remove AEAD from
4880bis until there is an academic study proving people need it?

> Admittedly CBC/CFB made this easier, but it was the email apps that needed

I see it differently.  I would say it was a combination of the email
applications needing fixing and PGP needing fixing.

PGP encourages implementations to support streaming, and most do.
But, using 4880, this means that an application may see plaintext from
unauthenticated ciphertext.  Efail shows how that can be exploited by
***modifying the ciphertext*** (a PGP problem) to create a potential
exfiltration channel.  Using chunked AEAD correctly, this type of
attack is not possible: it is possible to stream, and only release
plaintext from authenticated ciphertext.

Now, applications could have protected themselves from this attack if
they had backed out the message on MDC failure.  But, they didn't.
And, I'd argue that a major reason that they didn't was because this
type of attack is not well understood by application developers.
Application developers understand truncation.  But, ciphertext
modification is something that most have probably never heard of.
Since we can protect application developers from ciphertext
modification, I would argue that not doing so is negligent.

So, if we are distributing blame, and I'd rather not play that game,
then I'd place 90% of the blame on the WG and the PGP implementations,
and only 10% on the mail application developers.

---

From: Jon Callas <[email-masked]>
Date: Fri, 29 Mar 2019 12:59:21 -0700

> On Mar 29, 2019, at 2:07 AM, Neal H. Walfield <[email-masked]> wrote:
> 
> I still think this is a bad idea.  

Noted. It was a position that I threw out because I thought it encompassed a lot of things that people said and also allows you, the implementer to say, “screw it, we’re doing 256K only and to heck with all of you” and it would meet the standard. You could even implement 64K, thus generating a brinksmanship debate with you and Protonmail and I don’t have to listen to it.

I think that my proposal above is flawed because there’s this squishy space between 64K and 256K, and really hoped someone would either say, “Fine. I’ll give up on 256K” or “Fine. 256K is fine.” And then we’d modify the proposal and probably be done. It was, however a position I thought we might reason from.

> advantage to having large chunk sizes.

Well, actually what I was doing there was offering a number lower than 2^whatever for the max max max oh-dear-god-why-do-you-want-to-do-this size at a (to me) very, very generous gigabyte to the other side on the max size that you can then safely ignore.

> complexity, which is a security concern.

This isn’t actionable.

There’s a lot of rationale here, but not an actual proposal.

My proposal (left at the top of this missive for easy referral) was a concrete proposal. The above paragraph has a lot of justification in it and is interesting polemic but it is not an actual proposal.

If you had said:

* Chunk size is 256K. Always. If you have less than 256K, pad it with zeroes.

Then that would be a proposal.

> I'm sorry if you did and I didn't understand.

You don’t understand my position.

My position is that there are a bunch of people (like you, at least I think you) who want to build a general-purpose implementation that you hope will be used all over the place. You have a legitimate need for small chunks.

There are other people who observe that there’s also a legitimate need for huge chunks because they want to do storage encryption on very large things and — well, they have considerations that I’m not paying a lot of attention to.

My proposal above lets them do their thing and lets you ignore them. That’s my position. Making as many people happy as possible.

> Was that not actionable enough?  Should I link to the PR more often?

It would help, but even better would be to send what the proposal is. I really did read it and it’s a patch file. Patch files are hard for humans to read. It’s really nice to have a redline, but it’s also nice to know what the full thing is without having to mentally emulate git.

>  that would profit from different chunk sizes?

I’m not the one proposing the large chunk sizes. And as I understand the people who are doing it, performance isn’t their issue, data integrity is.

There’s a dilemma in here.

One is that there’s a belief that 

>  an error, or release unauthenticated plaintext.

Report an error. I’ve said that many times.

> constraints.  Our primary goal is to write a secure implementation.

Okay, so you’re arguing resources when you don’t really mean it. Gotcha. I thought that was the case that the things you were saying didn’t jibe together.

> which we don't want to do.

That constraint is a long-standing meta-requirement in OpenPGP, so cool.

However, I detect that there is an impossible-to-solve problem here, and I’ll outline it in another note later.

> is what I see the AEAD chunk size doing.

I think I understand where you’re coming from, but this is a standards organization. Standards are concerned with interoperability between implementors. I learned that from Jeff Schiller ages ago. Think of a standard as being like a dictionary and grammar. When you send me a message, I use that grammar/dictionary to determine what it means. Similtharly, if I want to encode a message that I want you to understand, then I encode it according to that grammar as well.

My proposal above says to an encoder that the safest thing to do is to use a chunk size of 64K. Everyone can decode that, and you’re fine. Many, many, many of them are going to be okay with 256K, and odds are that anyone of any import will do fine with it. (That’s what the SHOULD means). It also says there are people out in Lala-land who might be playing with chunks that are out to 2^whatever, and that’s kinda cool, but ignore them. 

That’s why I made it. It gives a sandbox for people who want to go out on the edge that the mainstream can ignore.

> manner.  I sincerely haven't understand your position, and I really

I think that’s because I’m trying to support a community of people who have reasonable, differing needs, and you want to mandate your implementation ideas on the universe. I apologize if that’s harsh, but it’s what I perceive. If I’m mistaken, let me know.

	Jon

---

From: Wyllys Ingersoll <[email-masked]>
Date: Fri, 29 Mar 2019 16:19:07 -0400

As an implementor (iPGMail on iOS), I strongly support Jon's suggestion
(16K "MUST", 256K "SHOULD", everything else "MAY").  I dont want to deal
with lots of optional sizes that just introduce complexity with little
practical benefit beyond some outliers that want to consume huge files in
one big bite.  Mobile devices are somewhat memory limited (though that is
every growing), but 64K or 256K is certainly not a problem.

-Wyllys Ingersoll

On Fri, Mar 29, 2019 at 3:59 PM Jon Callas <joncallas=
[email-masked]> wrote:

> >> Like some interim replies, particularly Bart Butler, I thought we had a
> rough consensus and that it was approximately:
> >>
> Noted. It was a position that I threw out because I thought it encompassed
> a lot of things that people said and also allows you, the implementer to
> say, “screw it, we’re doing 256K only and to heck with all of you” and it
> would meet the standard. You could even implement 64K, thus generating a
> brinksmanship debate with you and Protonmail and I don’t have to listen to
> it.
>
> I think that my proposal above is flawed because there’s this squishy
> space between 64K and 256K, and really hoped someone would either say,
> “Fine. I’ll give up on 256K” or “Fine. 256K is fine.” And then we’d modify
> the proposal and probably be done. It was, however a position I thought we
> might reason from.
>
> >> I could also get behind a hard limit of 2^30 on the grounds that that’s
> what we had for partial body lengths, but I understand the comment that
> there are things like multi-terabyte S3 buckets and out and out forbidding
> those to be single-chunk is a bit churlish, but only a bit.
> >
> Well, actually what I was doing there was offering a number lower than
> 2^whatever for the max max max oh-dear-god-why-do-you-want-to-do-this size
> at a (to me) very, very generous gigabyte to the other side on the max size
> that you can then safely ignore.
>
>
> >
> My proposal (left at the top of this missive for easy referral) was a
> concrete proposal. The above paragraph has a lot of justification in it and
> is interesting polemic but it is not an actual proposal.
>
> If you had said:
>
> My position is that there are a bunch of people (like you, at least I
> think you) who want to build a general-purpose implementation that you hope
> will be used all over the place. You have a legitimate need for small
> chunks.
>
> There are other people who observe that there’s also a legitimate need for
> huge chunks because they want to do storage encryption on very large things
> and — well, they have considerations that I’m not paying a lot of attention
> to.
>
> My proposal above lets them do their thing and lets you ignore them.
> That’s my position. Making as many people happy as possible.
>
> >
> It would help, but even better would be to send what the proposal is. I
> really did read it and it’s a patch file. Patch files are hard for humans
> to read. It’s really nice to have a redline, but it’s also nice to know
> what the full thing is without having to mentally emulate git.
>
> >
> I’m not the one proposing the large chunk sizes. And as I understand the
> people who are doing it, performance isn’t their issue, data integrity is.
>
> Okay, so you’re arguing resources when you don’t really mean it. Gotcha. I
> thought that was the case that the things you were saying didn’t jibe
> together.
>
> I think I understand where you’re coming from, but this is a standards
> organization. Standards are concerned with interoperability between
> implementors. I learned that from Jeff Schiller ages ago. Think of a
> standard as being like a dictionary and grammar. When you send me a
> message, I use that grammar/dictionary to determine what it means.
> Similtharly, if I want to encode a message that I want you to understand,
> then I encode it according to that grammar as well.
>
> My proposal above says to an encoder that the safest thing to do is to use
> a chunk size of 64K. Everyone can decode that, and you’re fine. Many, many,
> many of them are going to be okay with 256K, and odds are that anyone of
> any import will do fine with it. (That’s what the SHOULD means). It also
> says there are people out in Lala-land who might be playing with chunks
> that are out to 2^whatever, and that’s kinda cool, but ignore them.
>
> I think that’s because I’m trying to support a community of people who
> have reasonable, differing needs, and you want to mandate your
> implementation ideas on the universe. I apologize if that’s harsh, but it’s
> what I perceive. If I’m mistaken, let me know.
>
>         Jon
>
>

---

From: Jon Callas <[email-masked]>
Date: Fri, 29 Mar 2019 13:39:13 -0700

> On Mar 29, 2019, at 2:52 AM, Marcus Brinkmann <marcus.brinkmann=[email-masked]> wrote:
> 
> Hi,
> 

Yes, you did, and really, that’s the one that I think we’ve been debating from.

Summarizing, you said max chunk size of 64K.

There has been a lot of quibbling from there, like the Protonmail people saying 256K, and others wanting some other things.

Your proposal was the starting point from the one I kicked out yesterday.

	Jon

---

From: Jon Callas <[email-masked]>
Date: Fri, 29 Mar 2019 13:42:46 -0700

> On Mar 29, 2019, at 6:20 AM, Neal H. Walfield <[email-masked]> wrote:
> 

Many of us think that Efail is orthogonal to this problem and consequently while it’s a real issue, as a rationale, it’s lacking.

You might disagree with us, and I respect that. Nonetheless, I find Efail to be unconvincing. I’d love to debate this further, but do you have *another* rationale, or it only Efail?

	Jon

---

From: Jon Callas <[email-masked]>
Date: Fri, 29 Mar 2019 13:48:59 -0700

> On Mar 29, 2019, at 7:37 AM, Neal H. Walfield <[email-masked]> wrote:
> 
> 

I think that if Peter wanted to remove AEAD, he’d just say that.

But no, the whole reason he and I and others are debating is that we think that AEAD in OpenPGP is a Good Idea.

> applications needing fixing and PGP needing fixing.

Before I go further, it’s OpenPGP. This working group is OpenPGP.

PGP is a software product owned by Symantec. It implements OpenPGP, as well as S/MIME, X.509, and a whole lot of other things.

> and only 10% on the mail application developers.

Then why does it work with S/MIME? Do they get 90% too?

That brings us up to 190% of the blame, which might be called for, given that it is a major cluster, but I think it’s orthogonal to what we’re talking about here.

How about if we just work on AEAD instead of debating Efail? Especially since we agree on AEAD being needed.

	Jon

---

From: "brian m. carlson" <[email-masked]>
Date: Sat, 30 Mar 2019 00:19:49 +0000

> * MAY reject or error out on chunks larger than 16KB, but repeating ourselves, SHOULD support 256K.

I think this is fine. It covers what's requires for interoperability, it
makes prudent suggestions, and it allows people leeway if they know
they're the only one consuming their data.

If people like my original proposal, that's also fine with me (as would
be expected).
-- 
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204

---

From: Bart Butler <[email-masked]>
Date: Sat, 30 Mar 2019 02:17:55 +0000

Hi Jon,

As others have noted, there is a lot of confusion on this thread, some of which you touched in your AEAD Conundrum message, like when we say AEAD should not release unauthenticated plaintext, do we mean the entire message or the chunk?

Another piece of confusion is that Efail isn't a single vulnerability, it was several vulnerabilities related (at best) thematically.

So to be very specific, for the purpose of the following discussion, the advantage of smaller AEAD chunks is specifically to prevent Efail-style ciphertext malleability/gadget attacks, and the prohibition on releasing unauthenticated plaintext is applied to individual chunks, which is sufficient to foil this kind of attack in email.

The kind of attack we are talking about is fundamentally about exfiltration of plaintext data to an attacker-controlled endpoint. Borrowing from your AEAD Conundrum message, if the first chunk passes and is released, and the second chunk fails, that is OK, at least for email, because the part that was modified (the second chunk) is never released, so you get a truncated message and an error, but the truncated message without the modifications isn't going to exfiltrate itself.

Now if releasing ANY authenticated chunk of a message that hasn't been fully authenticated (in an AEAD sense) is a real problem for your application, I'd argue that you're trying to make AEAD do something it's not suited for and you should enforce this in your application if it applies to you, probably by not streaming.

So to recap, small-chunk AEAD provides specific value in preventing ciphertext malleability/gadget attacks, particularly in HTML email, which is a common use case.

What value does large-chunk AEAD actually provide? What I'm getting from the AEAD Conundrum message is that it's a way for the message encrypter to leverage the "don't release unauthenticated chunks" prohibition to force the decrypter to decrypt the whole message before releasing anything. Why do we want to give the message creator this kind of power? Why should the message creator be given the choice to force her recipient to either decrypt the entire message before release or be less safe than she would have been with smaller chunks?

Coming back to Neal's point, it's really hard to see any sort of value in really large AEAD chunks, because the performance overhead is negligible at that point and the only security 'benefit' that I can see is the encrypter trying to use the spec to force the decrypter to not stream, which does not seem like something at all desirable.

-Bart

P.S. ProtonMail doesn't use V5 keys or the new draft yet, but some users of OpenPGP.js have started using them with 256kB chunks, so we are not arguing on behalf of ourselves for the 256kB chunk size. The proposed language seems more or less OK in this regard, as most implementations will presumably keep 256kB support so these early adopters will not lose interoperability with their messages.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Friday, March 29, 2019 1:42 PM, Jon Callas <joncallas=[email-masked]> wrote:

> > On Mar 29, 2019, at 6:20 AM, Neal H. Walfield [email-masked] wrote:
> > 

> You might disagree with us, and I respect that. Nonetheless, I find Efail to be unconvincing. I’d love to debate this further, but do you have another rationale, or it only Efail?
>

---

From: Benjamin Kaduk <[email-masked]>
Date: Sat, 30 Mar 2019 10:04:38 -0500

My apologies for being only an occasional participant in  this thread (and
it will likely take me another week before I can reply again), but there
are a few points I would like to make.

> As others have noted, there is a lot of confusion on this thread, some of which you touched in your AEAD Conundrum message, like when we say AEAD should not release unauthenticated plaintext, do we…

It's really quite something to have gone through a week's worth all in one
go.  There are many people writing out careful descriptions of how they see
things, and yet we still seem to be talking past each other at times.

I propose that we use "plaintext corresponding  to non-modified ciphertex"
for the non-malleability protection that is provided by an AEAD
authentication tag on a single chunk, and "fully authenticated complete
plaintext" for the output after processing an entire message (i.e., all
chunks) with guarantee of non-truncation.  (Are there other cases in
between that we care about?)

> The kind of attack we are talking about is fundamentally about exfiltration of plaintext data to an attacker-controlled endpoint. Borrowing from your AEAD Conundrum message, if the first chunk pass…

One concern that I have (and  is only tangentially related to this quoted
part) is that I want to make it easy for implementations to "do the right
thing" when ciphertext is modified, i.e., return an error, and specifically
to return an error without releasing any plaintext that originates from the
modified ciphertext.  The current openpgp ecosystem does not seem to be
very compliant to that desired behavior, and part of that may be due to a
lack of philosophical support/help from the spec.

> Coming back to Neal's point, it's really hard to see any sort of value in really large AEAD chunks, because the performance overhead is negligible at that point and the only security 'benefit' that…

I'm still not sure I understand the point of very large chunks, since once
they get really  big an implementation is choosing between streaming
plaintext from potentially modified ciphertext or return an error without
even attempting to process the chunk.  I'm not convinced that the second
will win out in implementations if  we alow very large chunks.

Some other notes, not relating to anything specifically quoted from this
message (but derived from other parts of the thread):

TLS allows for arbitrarily variable-length chunks because it is
a synchronous transport for higher-level application streams and the
application may have arbitrary message sizes.  OpenPGP is used in an
asynchronous model, where a message generator can be modelled to make all
its actions before the receiver processes anything, and there is only
one-directional communication within the OpenPGP format.  So there does not
seem to be much demand for "take all the bytes that you have so far and
send them right now", and AFAICT the message generator can just wait until
end of data arrives or enough data to make a complete chunk arrives.  So
from that point of  view, there is not much argument in favor of varying
the chunk size within a single message, and possibly even across messages
(i.e., this line  of reasoning would be okay with a single chunk size fixed
for everyone as a protocol constant).  There are of course other factors
that may come into play, like constrained systems and  such, but we can
treat those separately.

I also have a use case for authentication of large chunks of data at rest:
they allow me to use a cheap bulk storage service that provides
(best-effort) replication and archiving but has poor physical security.  So
I encrypt my data to myself and put it in storage, but when I get it  back
I need to know that it's valid.  I can imagine at least one case where
knowing exactly which chunk was corrupted would save effort; it may be a
toy example but perhaps it is illustrative of a broader case.  Note that
there are algorithms to compute pi to arbitrary precision, and even to
compute the Nth digit thereof without coputing the previous digits.  If I
need to have random-access inquiries into the value of pi, I could
precompute using softare I trust and do this self-encryption thing, and
when a chunk is bad I can recompute only that chunk and still trust that I
only ever use values generated by my trusted implementation.

And finally, there is no openpgp Working Group; all we have here is a bunch
of folks interested in a topic talking amongst each other on a public
mailing list hosted at the IETF.  There are no WG chairs and no expectation
of Area Director supervision (i.e., I don't feel obligated to read the
messages here).  That said, I'm happy to see that we're staying calm and
civil, and AFAICT everyone is honestly trying to understand everyone else's
position and come to a consensus.  Let's try to keep focusing on the
technical details and what use cases we need to cover.

Thanks,

Ben

---

From: Bart Butler <[email-masked]>
Date: Sat, 30 Mar 2019 16:42:19 +0000

Hi Ben,

> lack of philosophical support/help from the spec.

If you mean 'modified ciphertext' to equal 'modified chunk', and are OK with releasing previously unauthenticated chunks, then I completely agree. The alternative is is just no streaming.

> will win out in implementations if we alow very large chunks.

Agreed. Part of the rationale with a smaller chunk limit is not forcing implementations to make this choice. The guidance becomes very simple--never release unauthenticated chunks, full stop.

---

From: Benjamin Kaduk <[email-masked]>
Date: Sat, 30 Mar 2019 11:57:47 -0500

> If you mean 'modified ciphertext' to equal 'modified chunk', and are OK with releasing previously unauthenticated chunks, then I completely agree. The alternative is is just no streaming.

Right.  (I think  I heard some  people argue that they were not okay with
releasing previously unauthenticated chunks, to be clear.  That seems like
something of a philosophical question, though perhaps there is some better
agreement to have.

-Ben

---

From: "Neal H. Walfield" <[email-masked]>
Date: Sat, 30 Mar 2019 22:16:46 +0100

Hi Ben,

Thanks for your note.

At Sat, 30 Mar 2019 10:04:38 -0500,
> only ever use values generated by my trusted implementation.

Just to be clear: when you say "large chunks of data at rest," you're
not arguing that large AEAD chunks are better, are you?  It seems to
me that if you use small chunks, at least in your example, you have
less work to do when you discover a corrupted chunk.

Thanks,

:) Neal

---

From: Jon Callas <[email-masked]>
Date: Sat, 30 Mar 2019 16:09:41 -0700

> On Mar 29, 2019, at 7:17 PM, Bart Butler <bartbutler=[email-masked]> wrote:
> 
> Hi Jon,
> 

That is precisely the question. But the bigger question is whether you care about that. Sometimes it matters, sometimes it does not.

For example, let’s suppose that I have a very large blob on mass storage. Media failures happen. If there’s a bad block on a disk, do you want to lose the whole file because of it? Sometimes you want to throw your hands up in the air. This is most common in an interactive protocol, and in general the answer is yes. For example, an SSL connection to a server, if there’s funny business going on, you want to blow up the connection and try over again. On the other hand, if you had an archival thing (e.g. tar file with some historic documents), you want to recover as much as you can.

OpenPGP is in general the latter case rather than the former. I believe it’s less important to have strict semantics on failures because it’s usually storage.

> Another piece of confusion is that Efail isn't a single vulnerability, it was several vulnerabilities related (at best) thematically.

I understand Efail. Trust me.

> So to be very specific, for the purpose of the following discussion, the advantage of smaller AEAD chunks is specifically to prevent Efail-style ciphertext malleability/gadget attacks, and the proh…

I disagree. If you want to prevent something like Efail, you want larger chunks. Assuming that you believe that early release matters. 

Let’s rewind here, and not talk about Efail, let’s talk about the real issue. If you want the entire blob to have all-or-nothing semantics, then you want the fewest number of chunks as is reasonable. If you have attacker-controlled inputs, then every joint between the chunks is a vulnerability.

> The kind of attack we are talking about is fundamentally about exfiltration of plaintext data to an attacker-controlled endpoint. Borrowing from your AEAD Conundrum message, if the first chunk pass…

I agree with this. 

> So to recap, small-chunk AEAD provides specific value in preventing ciphertext malleability/gadget attacks, particularly in HTML email, which is a common use case.

Also agree 100%.

This is why a long time ago, I said that Efail is orthogonal to the chunking issue. Either way, large or small, you end up in the same place of not being hacked.

> What value does large-chunk AEAD actually provide? What I'm getting from the AEAD Conundrum message is that it's a way for the message encrypter to leverage the "don't release unauthenticated chunk…

Let me summarize the conundrum: If you want strict AEAD no-release semantics, you want a fewer number of chunks. 

> Coming back to Neal's point, it's really hard to see any sort of value in really large AEAD chunks, because the performance overhead is negligible at that point and the only security 'benefit' that…

Okay, here’s another thing that’s a pet peeve of mine. We’re arguing security and you brought up performance. I never mentioned performance, the people who want large chunks haven’t brought it up. They want large chunks because they perceive it to be more secure. 

If you respond to a security request with a performance answer, you literally don’t know what you’re talking about. So let’s toss that aside.

Bigger chunks *are* more secure. Does that security matter? Well, it does to the people who want big chunks.

> P.S. ProtonMail doesn't use V5 keys or the new draft yet, but some users of OpenPGP.js have started using them with 256kB chunks, so we are not arguing on behalf of ourselves for the 256kB chunk si…

I am going to rewind again, back to the proposal I made. Here it is, so we can look at it. 

Elided out of this, and possibly important is that “support” includes chunks smaller than that size. I should have said that, but I wanted it to be as stark as possible. So let me repeat it and abstract it with some variables:

(1) MUST support up to <small-chunk-size> chunks.
(2) SHOULD support up to <larger-chunk-size> chunks, as these are common.
(3) MAY support larger up to the present very large size.
(4) MAY reject or error out on chunks larger than <small-chunk-size>, but repeating ourselves, SHOULD support <larger-chunk-size>.

Clauses (3) and (4) set up a sandbox for the people who want very large chunks. They can do whatever they want, and the rest of us can ignore them. Why get rid of that? It doesn’t add any complexity to the code. It lets the people who want the huge ones do them in their own environment and not bother other people.

My concern is over (1) and (2) and specifically that there’s both <small> and <large> sizes. 

I think that’s an issue. If there are two numbers we are apt to end up with skew before settling on one, so it’s better to agree on just one. That’s the real wart in my proposal.

Neal said he’s fine with 256K. We could do that. 16K has also been suggested (I think Neal said it’s his preference). My intuition is that whatever GnuPG is using for partial body lengths is also a good option as we know that that’s reasonable for both performance and everything else, and my preference is that it be as small as possible to support the smallest footprint possible. I spent about ten minutes looking for the answer and I think it’s 8K, but I’m not sure. Someone correct me if that’s not it. Whatever it is, that’s what I’d do, but I’m not an implementer.

Opinions on what a single number should be? 256? 16? 8? Something else?

	Jon

---

From: Peter Gutmann <[email-masked]>
Date: Sun, 31 Mar 2019 02:58:44 +0000

>I think that if Peter wanted to remove AEAD, he’d just say that.

I'm not saying remove it, just get some data to support making a decision in
some way.  In particular, AEAD is a good thing, but there's no evidence that
chunking with AEAD, which complicates things greatly, is useful or necessary.

Here's three actual real-world data points: 

For the last twenty-eight years, PGP has had functionality equivalent to AEAD
in the form of encrypt+sign.  The only mechanism this supported was chunk-
size = data-size, and in twenty-eight years this never seems to have
caused/been seen as a problem. 

If you don't think encrypt+sign is equivalent functionality then we've also
had pseudo-AEAD in the form of MDC for eighteen years (draft-ietf-openpgp-
rfc2440bis-02) and chunking wasn't an issue then either.

Finally, CMS has had this for more than a decade (Authenticated-Enveloped-
Data) and it's not been a problem there either.

Why is it suddenly a big deal now, and only with OpenPGP, after twenty-eight
or eighteen years (depending on which one you choose) of equivalent mechanisms
not being a problem?  Is this because it really is, or just because AEAD can
do all sorts of cool things and people want to play with them?

Peter.

---

From: Bart Butler <[email-masked]>
Date: Sun, 31 Mar 2019 04:11:25 +0000

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Saturday, March 30, 2019 4:09 PM, Jon Callas <[email-masked]> wrote:

> > On Mar 29, 2019, at 7:17 PM, Bart Butler bartbutler=[email-masked] wrote:
> > Hi Jon,

> OpenPGP is in general the latter case rather than the former. I believe it’s less important to have strict semantics on failures because it’s usually storage.

I agree. I would say my point is that with sufficiently small chunks, the user/decrypter can choose what kind of failure behavior is appropriate. Large chunks robs the decrypter of that.

> I understand Efail. Trust me.

I do. I was bit excessively pedantic defining everything because I felt that this thread has suffered from ambiguity in several terms.

> Let’s rewind here, and not talk about Efail, let’s talk about the real issue. If you want the entire blob to have all-or-nothing semantics, then you want the fewest number of chunks as is reasonabl…

OK, I think this is the part that I don't understand. Why does it matter what chunking scheme is used here? If my app requires all-or-nothing semantics, I would program my app to enforce that all chunks must pass and not release plaintext unless that happened, with no truncation, etc. So why would every joint be a vulnerability?

> Let me summarize the conundrum: If you want strict AEAD no-release semantics, you want a fewer number of chunks.

I guess this is my fundamental question. You can force no-release semantics at the application level for any chunk size scheme, right?

> If you respond to a security request with a performance answer, you literally don’t know what you’re talking about. So let’s toss that aside.

I apologize, I was not trying to create a strawman here, but I am completely at a loss for what the benefit of large chunks is.

> I think that’s an issue. If there are two numbers we are apt to end up with skew before settling on one, so it’s better to agree on just one. That’s the real wart in my proposal.

I'm OK with eliminating (2) and just using the MAY part to take care of any legacy 256K messages OpenPGP.js users might have. As I said, we don't have any of these messages in production yet and I'd err on the side of a cleaner spec. I just really want to understand the benefit of large chunks for security and right now I clearly do not.