Subject: Multiple OpenPGP messages per file: legal or not?

From: "Rachel Willmer" <[email-masked]>
Date: Sun, 7 Oct 2007 15:26:30 +0100

I've hit an interoperability problem with GnuPG, and need guidance on
the legality or otherwise of multiple OpenPGP messages in a single
file where those messages are "Encrypted Messages" (as defined in
Section 11.3 "OpenPGP Messages".

If I have a file which contains 2 Encrypted Messages, e.g. the packet
sequence goes:

ESK Sequence | Encrypted Data | ESK Sequence | Encrypted Data

is this:

a) legal

b) illegal

c) legal but not supported by GnuPG

d) outwith the scope of the ID since that defines only "OpenPGP
Message Format", not what you can do with those messages

I believe the answer is either (c) or (d)....

Rachel

---

From: David Shaw <[email-masked]>
Date: Sun, 7 Oct 2007 12:55:05 -0400

> I believe the answer is either (c) or (d)....

The answer, I would say, is (d): out of scope.

The spec requires that Transferable Public Keys can be concatenated,
but not Encrypted Messages.

David

---

From: "Rachel Willmer" <[email-masked]>
Date: Sun, 7 Oct 2007 23:41:25 +0100

If (d) "out of scope", which spec does define file formats?

If the answer is "there isn't one which does", how do we plan to do
interoperability between applications which use the OpenPGP packet
format? e.g. to transfer Encrypted Messages.

Is there an implicit assumption in the spec that Message = File?

Rachel

---

From: David Shaw <[email-masked]>
Date: Sun, 7 Oct 2007 21:51:08 -0400

> format? e.g. to transfer Encrypted Messages.

I didn't quite mean "out of scope" in that sense.  I meant "the spec
doesn't mandate it or forbid it, so it's up to the implemention to
decide."  As you saw, GPG doesn't do it.

> Is there an implicit assumption in the spec that Message = File?

Not exactly.  The spec just defines bytes in a particular order.
Those bytes could be an array in memory, a file, a stream over a
network, etc, and the spec doesn't really care.  That said, the spec
does pretty much say that a file is one of the possible carriers of
those bytes.

David

---

From: "Rachel Willmer" <[email-masked]>
Date: Mon, 8 Oct 2007 10:35:53 +0100

> It shouldn't be up to the implementation to decide (IMHO). If the goal
> of the WG is "to provide IETF standards for the algorithms and formats
> of PGP processed objects", then surely some standard file formats
> should be defined [*] : e.g. keyring, encrypted file, signed file,
> etc.

... even if that just means documenting what GPG does now (and what
PGP2,etc) did as de-facto standards....

Rachel

---

From: "Rachel Willmer" <[email-masked]>
Date: Mon, 8 Oct 2007 10:34:25 +0100

Rachel

[*] I'm using "file format" in the loose sense of disk file or file stream.

---

From: Ian G <[email-masked]>
Date: Mon, 08 Oct 2007 12:37:52 +0200

> should be defined [*] : e.g. keyring, encrypted file, signed file,

The architectural imperative here is that the definition has 
to stop somewhere.

Stopping before the file / message level is a good idea 
because if you stop after the file level, you also have to 
define what a file is [*].  I'm sure the theoreticians will 
describe this more clearly than I .. but files aren't 
necessarily just a sequence of bytes.  Emails aren't 
necessarily the same as files.  Chat messages are different 
to emails.  Who knows what the next innovation in storage or 
communication will ask of us.

By stopping short of how the sequence of bytes is stored or 
passed, OpenPGP ensures that it is applicable and 
interoperable across a wide range of apps ... just by 
forcing the developers of the apps to agree on a few local 
details.

Also, files like keyrings are not the subject of 
interoperability in normal security practice.  The app owns 
its keyring, and it doesn't want to let anyone touch it, 
it's a bit of a historical accident that the PGP keyring was 
so easy to share.  If there is a need to converse with 
another app, then they should use an export/import format, 
and there are well-defined export sequences in the spec, so 
that the app has a chance to deal with any complications of 
sharing its internal data.

> [*] I'm using "file format" in the loose sense of disk file or file stream.

A spec would not be able to be so conveniently loose ;)

iang

---

From: "Rachel Willmer" <[email-masked]>
Date: Mon, 8 Oct 2007 12:26:18 +0100

> forcing the developers of the apps to agree on a few local

"just" ???

That argument implies 3 things:

a) it's acceptable to require every implemention to support each other
implementation as a separate special case

b) it's acceptable to require each user to specifiy at the time of
encryption or signing which implementations (not algorithms, or keys,
but implementations!) should be able to parse the result

c) it's acceptable for 2 implementations which support the same set of
algorithms and have the appropriate keys for sharing encrypted or
signed data to say they both have an OpenPGP-conformant application -
but which can't actually share the data.

Am I the only one that thinks this is a flawed argument?

Surely this should be a fundamental purpose for the standard: for you
to be able to encrypt a file using your implementation, send it to me,
and for me to be able to decrypt it with a different standard.

Isn't it??

> A spec would not be able to be so conveniently loose ;)

As someone who is trying to write a set of interoperability tests for
my own implementation, I'm not sure I quite understand your use of the
word "convenient" here... :-)

Rachel

P.S. I can take the argument that says a keyring is different, because
it's local to the app. But surely, not encrypted/signed files???

---

From: Ben Laurie <[email-masked]>
Date: Mon, 08 Oct 2007 13:11:13 +0100

Ian G wrote:
> The architectural imperative here is that the definition has to stop
> somewhere.
> 
> Stopping before the file / message level is a good idea because if you
> stop after the file level, you also have to define what a file is [*]. 
> I'm sure the theoreticians will describe this more clearly than I .. but
> files aren't necessarily just a sequence of bytes.  Emails aren't
> necessarily the same as files.  Chat messages are different to emails. 
> Who knows what the next innovation in storage or communication will ask
> of us.
> 
> By stopping short of how the sequence of bytes is stored or passed,
> OpenPGP ensures that it is applicable and interoperable across a wide
> range of apps ... just by forcing the developers of the apps to agree on
> a few local details.

I think this entirely misses the point - the question is not how
messages are stored/transmitted, but what an implementation should do
when some unit of storage/transmission contains more than one message.

It seems entirely reasonable to me that it should be expected to keep
eating messages until there are no messages left to eat, however the
spec does not state that. It seems to me that it could, quite easily.

> 
> Also, files like keyrings are not the subject of interoperability in
> normal security practice.  The app owns its keyring, and it doesn't want
> to let anyone touch it, it's a bit of a historical accident that the PGP
> keyring was so easy to share.  If there is a need to converse with
> another app, then they should use an export/import format, and there are
> well-defined export sequences in the spec, so that the app has a chance
> to deal with any complications of sharing its internal data.
> 
> 
>> Rachel
>>

-- 
http://www.apache-ssl.org/ben.html           http://www.links.org/

"There is no limit to what a man can do or how far he can go if he
doesn't mind who gets the credit." - Robert Woodruff

---

From: Ian G <[email-masked]>
Date: Mon, 08 Oct 2007 14:57:04 +0200

>> forcing the developers of the apps to agree on a few local

Well, like I say, the spec has to stop somewhere.  Consider 
that once you cross the boundary, you enter into the 
argument of MIME v. Ascii Armour, and that's a never-ending 
barrel of worms.  In fact, it even made an appearance in the 
recent message from the IETF where there was a ham-fisted 
and embarrassing attempt to downgrade AA in favour of MIME.

Now, this brings up an interesting direction:  if you feel 
that this area should be more standardised, the MIME 
approach of a separate RFC is a good one.  Start a files 
format RFC.

> implementation as a separate special case

Well, no, the spec stops before the application level. 
Applications will need to do some extra stuff there, and if 
GnuPG creates a format that can't be read by PGP Inc's email 
program, then that's an app issue.  GnuPG is an app that 
happens to include an implementation;  it's not just an 
implementation.

> but implementations!) should be able to parse the result

Applications do have that issue to deal with, yes.
E.g., PGP Inc's email program can't read the messages in 
Jabber's chat implementation of OpenPGP ... but that's 
because it can't read chat, and Jabber clients can't read email!

> but which can't actually share the data.

Yes, that's always acceptable.  There is no law that says 
that apps have to communicate, only specifications that say 
"if you can share these bytes, then we know of no reason you 
can't communicate."

> Am I the only one that thinks this is a flawed argument?

I would say you would find many friends in the developer 
community :)

> and for me to be able to decrypt it with a different standard.

I see it a bit of a pipe-dream.  If I can send you an 
OpenPGP encrypted email that you can decrypt, *guaranteed*, 
then that also means that OpenPGP has to specify enough of 
the mail system that it can survive various munging effects.

Line-slicing, 7 bit transformations, conversion into 
different messaging systems like LotusNotes, ISPs inserting 
adverts in freemail, etc etc ...  Ascii Armour was an 
*attempt* at that, and it works, sort of, but not well 
enough to be proud of.  MIME was another attempt, and it 
suffers its own set of problems.  Neither of these attempts 
guarantee communicability, they just assist if you are 
prepared to work with their assumptions.

> word "convenient" here... :-)

Right.  Literally, your pain is there because OpenPGP threw 
the problem over the wall, and you're on the other side of 
that wall.  What I'm saying is that the wall is actually in 
a good place, so please don't throw it back :)

> it's local to the app. But surely, not encrypted/signed files???

Hmm....  Consider this delicious example of a signed file:

http://webfunds.org/ricardo/contracts/webfunds/BeerVouchers.html

What would it mean if there were *two* such sequences in the 
one file?  Does the order matter?  Is there any meaning to 
what comes before or after that data stream?  Does the file 
creation date mean anything?

In that example the app benefits from OpenPGP's decision to 
concentrate on the byte-stream definition.  The app then 
arranges matters locally to extract out the byte-stream and 
deal with it, knowing that it is dealing with one and only 
one chunk.  In that case, the app certainly ignores any 
follow-on packets, and/or declares an error, and it 
certainly doesn't want OpenPGP telling it to expect an 
endless stream of them.

(We could also talk about signed transactional messages 
being concatonated ... or about Java's J2EE where files are 
considered not part of the system ... or mobile ... or ...)

iang

---

From: Ian G <[email-masked]>
Date: Mon, 08 Oct 2007 15:02:30 +0200

> spec does not state that. It seems to me that it could, quite easily.

Nope.

The decision you are asking for is a semantic one.  I.e., 
what does it mean to have two messages?  In a transactional 
environment, it would be the height of protocol rudeness for 
OpenPGP to impose a meaning on the messages.  That decision 
lives firmly at the app level.

See also that other example in rejoinder to Rachel.

iang

---

From: David Shaw <[email-masked]>
Date: Mon, 8 Oct 2007 09:22:17 -0400

> should be defined [*] : e.g. keyring, encrypted file, signed file,

I think the spec does define this.  It defines all of those items that
you mention in the grammar in section 11.3.  Any OpenPGP-compliant
application should be able to write such a message in such a way that
any other OpenPGP-complaint application can read it, or one or both of
the implementations aren't OpenPGP compliant.

What the spec doesn't define is whether an application must process an
"OpenPGP Message, OpenPGP Message" -- two messages concatenated
together.  There is an assumption in the spec that a single stream
contains a single message, and in fact there are some legal ways to
encode data that actually require a single stream to contain a single
message.

This doesn't mean that an application can't read concatenated messages
(when possible) if it chooses to, of course, though it should be
careful about generating them, as there is no guarantee that the
recipient can read them.

David

---

From: Ben Laurie <[email-masked]>
Date: Mon, 08 Oct 2007 14:31:37 +0100

Ian G wrote:
> What would it mean if there were *two* such sequences in the one file? 

It would mean there were two signed messages in that file.

> Does the order matter?  Is there any meaning to what comes before or
> after that data stream?  Does the file creation date mean anything?

These questions are beyond the scope of OpenPGP.

Also, I might as well ask the same questions about two separate files
with messages in. For example, if I follow this URL
http://webfunds.org/ricardo/contracts/webfunds/, there appear to be four
such messages linked. Does the order matter? blah, blah...

> In that example the app benefits from OpenPGP's decision to concentrate
> on the byte-stream definition.  The app then arranges matters locally to
> extract out the byte-stream and deal with it, knowing that it is dealing
> with one and only one chunk.  In that case, the app certainly ignores
> any follow-on packets, and/or declares an error, and it certainly
> doesn't want OpenPGP telling it to expect an endless stream of them.

The app is, of course, free to say "I expect a single OpenPGP message
per file". That doesn't mean the spec can't permit multiple messages per
file.

---

From: "Rachel Willmer" <[email-masked]>
Date: Mon, 8 Oct 2007 14:27:27 +0100

On 08/10/2007, Ian G <[email-masked]> wrote:
> Well, like I say, the spec has to stop somewhere.

The spec for a single message should only define how a single message
is structured, I agree. No complaints with RFC2440 in that regard.
But...

> program, then that's an app issue.

... if 2 apps supporting the same standard (and I mean OpenPGP in its
wider sense, not just RFC2440) which support the same set of
algorithms and suitable keys, can't actually talk to each other, then
I'm not sure what the point of the standard is.

The analogy would be if we'd defined the format of a SMTP mail
message, but left it up to individual developers as to how 2 mail
servers actually talked to each other. ("oh, that's just an
application issue....")

> because it can't read chat, and Jabber clients can't read email!

By current rules, 2 Jabber clients which both support OpenPGP wouldn't
necessarily be able to talk to each other unless they use the same
implementation....

Rachel

---

From: Derek Atkins <[email-masked]>
Date: Mon, 08 Oct 2007 09:57:04 -0400

> careful about generating them, as there is no guarantee that the

I know for a fact that the PGP implementation of OpenPGP can certainly
handle multiple ascii-armored messages in a single "file".  But I'm
pretty sure that it will NOT handle multiple "binary" messages in a single
"file".

> David

-derek

-- 
       Derek Atkins, SB '93 MIT EE, SM '95 MIT Media Laboratory
       Member, MIT Student Information Processing Board  (SIPB)
       URL: http://web.mit.edu/warlord/    PP-ASEL-IA     N1NWH
       [email-masked]                        PGP key available

---

From: "Rachel Willmer" <[email-masked]>
Date: Mon, 8 Oct 2007 14:57:30 +0100

> contains a single message,

If that is intended, it should be explicitly stated, IMHO.

And if that is the case, then ok, I can get back to writing test
cases, and forget about writing RFCs.

If it is stated somewhere and I've missed it, can you point me at the
reference please?

ta
Rachel

---

From: Derek Atkins <[email-masked]>
Date: Mon, 08 Oct 2007 10:26:43 -0400

> necessarily be able to talk to each other unless they use the same

That's not at all true.  The Jabber/OpenPGP integration rules (which
do exist, by the way!) specify the message boundaries outside of the
OpenPGP work.  In other words the jabber specification talks about how
to break apart the XML stream into multiple OpenPGP messages, and then
you just need to feed each "message" into your OpenPGP processor
separately.

As an implementor (chair hat not on) I really don't think we need a
general draft to specify how to frame multiple OpenPGP messages.
I think that the application using OpenPGP can define how it wants
to frame messages (just like how Jabber specifies the message framing).
RFC2440 and 2440bis already have a framing algorithm, called Ascii Armor,
which can let you store multiple OpenPGP messages into a single file.

So I just don't really see the need.

As the chair, however, I'm amenable to adding the work if there is a consensus
that such work is actually required.

-derek

---

From: Ben Laurie <[email-masked]>
Date: Mon, 08 Oct 2007 15:37:30 +0100

Ian G wrote:
> The decision you are asking for is a semantic one.  I.e., what does it
> mean to have two messages?  In a transactional environment, it would be
> the height of protocol rudeness for OpenPGP to impose a meaning on the
> messages.  That decision lives firmly at the app level.

I am not arguing with that. However, I am not asking for any semantic
interpretation from, say, GPG, which does not really do semantics.

> See also that other example in rejoinder to Rachel.

In which I agree that it is up to the application to interpret the
meaning of having two signed messages in one file. Or, indeed, in having
two signed messages in two files.

---

From: Rodney Thayer <[email-masked]>
Date: Mon, 08 Oct 2007 08:51:58 -0700

>> Ben Laurie wrote:
...and Derek Atkins wrote, and and and...

I have a question.  Back to Rachel's original
point - she seems to have come up with a non-interoperability
case with GnuPG.  Is something wrong?  Is this a
situation where, in a perfect world, there would have
been more GPG documentation or more/more verbose instrumentation?
Is there a section of (the spec or something else) that the
other implementation interpreted "wrong" or "different".

As I recall there's still a body of "genetically separate
implementations" - is there any experience with this
'artifact' happening before?

---

From: Rodney Thayer <[email-masked]>
Date: Mon, 08 Oct 2007 09:47:24 -0700

> The architectural imperative here is that the definition has to stop

I think there are many architectural imperatives here.  "get the
rfc out" is one, "don't spawn non-interoperable implementations" is
another.

"the definition has to stop somewhere" I would have said as
"we wish the definition to be as precise as we can so as to ensure
the definition stops somewhere", but, yeah, that too...

---

From: Ian G <[email-masked]>
Date: Fri, 12 Oct 2007 08:38:44 +0200

> If that is intended, it should be explicitly stated, IMHO.

Just in an effort to get closure on this issue, I see it as 
"undefined."

That is, it is strictly undefined whether messages can be 
concatonated.

This makes sense to me, as how the app packages these 
messages up and ships them is ... undefined, including in 
the trivial case of files containing concatonations.

> If it is stated somewhere and I've missed it, can you point me at the

Now, maybe the spec should say that the issue is undefined, 
I'm not sure.

I'm also not sure whether we can even add a change like that 
at this late stage?

iang

---

From: Derek Atkins <[email-masked]>
Date: Sat, 20 Oct 2007 18:52:32 -0400

Hi,

> If that is intended, it should be explicitly stated, IMHO.

I do not agree.  rfc2440 is about the OpenPGP Message, not about
framing OpenPGP messages or about how you can store multiple
OpenPGP messages.

> If it is stated somewhere and I've missed it, can you point me at the

Basically, 2440bis defines the message formats.  It does not (nor does
it need to) define the process of carrying or storing those messages.
Therefore, the question of "message == file" is out of scope for
2440bis and various implementations may choose to implement it
however they choose.

So, no, I don't think it is stated explicitly, but I do believe that
it is implicit that OpenPGP messages are "atomic", and how your
implementation deals with multiple messages is implementation
dependent.

I'll note for the record (as an implementor, not as chair) that the
parser implementation I wrote (many many MANY moons ago) allows
multiple ASCII Armored messages in a single file, but only a single
non-armored OpenPGP message in a single "file".  If you have a
"stream" then you need an "out of band" mechanism to separate the
OpenPGP messages.

Personally I think there's no additional draft needed unless you have
a specific application in mind and want to define how to frame
multiple OpenPGP messages within your application communication
stream.

Good Luck,

-derek

-- 
       Derek Atkins                 617-623-3745
       [email-masked]             www.ihtfp.com
       Computer and Internet Security Consultant