Subject: Re: [openpgp] keyserver protocol

From: Daniel Kahn Gillmor <[email-masked]>
Date: Tue, 07 May 2013 11:11:45 -0400

On Thu 2013-01-03 17:53:15 -0500, David Shaw wrote:

> I actually wrote this up at one point as an informational draft, but
> for one reason or another didn't finish submitting it.  If there is
> interest, I can clean it up and submit:
>
>   http://tools.ietf.org/id/draft-shaw-openpgp-hkp-00.txt

David, i would like to see this picked back up if possible.  Is there a
way that i can help?

In particular, I would like to see the error signalling and semantics be
more clearly and explicitly defined, so that (for example) when a
keyserver has a problem the user agents (e.g. client tools like gpg
--refresh) have a clear way to distinguish between cases like:

 0) "I have no key material matching this name/keyid at all"

 1) "I have too many keys that match this search to bother you with an
     insanely long list"

 2) "something is broken in my database, and I'm confused"

and so forth.

Any thoughts on what would be reasonable next steps?

Regards

    --dkg

---

From: "Daniel A. Nagy" <[email-masked]>
Date: Tue, 07 May 2013 17:44:40 +0200

>  2) "something is broken in my database, and I'm confused"

Also, I think that it makes sense to explicitly allow for partial
implementations of the protocol. For example, one that only allows for
searching by keyID's or even just long keyID's and fingerprints. I
think, there should be a clear way to communicate that the use of an
unsupported part of the protocol has been attempted.

Bests,

Daniel

---

From: John Clizbe <[email-masked]>
Date: Tue, 07 May 2013 23:02:25 -0500

> 1) "I have too many keys that match this search to bother you with an 

You /must/ mean documenting how those two are already implemented?

X-HKP-Results-Count: number of matching keys
Content-Length: number of bytes in resulting keys

- From the SKS CHANGELOG(+) and Mercurial commit log(+>):

+ 1.1.4
+   - Fix X-HKP-Results-Count so that limit=0 returns no results, but include
+     the header, to let client poll for how many results exist, without
+     retrieving any. Submitted by Phil Pennock. See:
+     http://lists.nongnu.org/archive/html/sks-devel/2010-11/msg00015.html

+> changeset:   115:47835fd59b63
+> parent:      113:73ba20267254
+> user:        Phil Pennock <[email-masked]>
+> date:        Sat Apr 21 18:24:46 2012 -0500
+> files:       dbserver.ml key.ml request.ml wserver.ml
+> description:
+> Limit fix for limit=0
+> Return real status text strings, rather than confusing "500 OK".
+> Handle No_results as an exception type, giving 404 instead of 500.
+> Treat limit of -1 (or <0) as "unlimited".
+> Handle limit=0 so that can ask for number of results without getting results.
+>
+> From email submission:
+> Back when X-HKP-Results-Count: was discussed, David Shaw suggested that
+> limit=0 should return no results, but include the header, to let a
+> client poll for how many results exist, without retrieving any.  See:
+>   http://lists.nongnu.org/archive/html/sks-devel/2010-11/msg00015.html
+>
+> Please find attached a patch. Plus a couple of related cleanups in HTTP error
+> response handling.

+ 1.1.2:
+  - Johan van Selst's patch implementing Phil Pennock's suggestion
+       of an X-HKP-Results-Count: header to returned web server queries
+   - Johan van Selst's patch to add Content-length header to web results

+> changeset:   49:68f88ae59b6a
+> user:        John Clizbe <[email-masked]>
+> date:        Thu Nov 04 02:37:31 2010 -0500
+> files:       dbserver.ml request.ml wserver.ml
+> description:
+> Johan van Selst's patch implementing Phil Pennock's suggestion
+> of an X-KHP-Results-Count: header to returned web server queries
+>
+> http://lists.nongnu.org/archive/html/sks-devel/2010-11/msg00016.html
+>
+> changeset:   48:e6d918ac4c66
+> user:        John Clizbe <[email-masked]>
+> date:        Wed Nov 03 21:58:51 2010 -0500
+> files:       wserver.ml
+> description:
+> Johan van Selst's patch to add Content-length header to web results
+>
+> http://lists.nongnu.org/archive/html/sks-devel/2010-11/msg00005.html

> 2) "something is broken in my database, and I'm confused"

Could you /maybe just possibly/ tie this down to something like a real error
condition instead of something so ambiguous?  Taking a look at lines 245-307
of wserver.ml may be helpful.

- -John

PS: Dan, please DO NOT CC me on replies to the list.

- -- 
John P. Clizbe                      Inet: John (a) Gingerbear DAWT net
SKS/Enigmail/PGP-EKP                  or: John ( @ ) Enigmail DAWT net
FSF Assoc #995 / FSFE Fellow #1797  hkp://keyserver.gingerbear.net  or
     mailto:[email-masked]?subject=HELP

Q:"Just how do the residents of Haiku, Hawai'i hold conversations?"
A:"An odd melody / island voices on the winds / surplus of vowels"

- -- 
John P. Clizbe                   Inet:   JPClizbe(a)comcast DOT nyet
Golden Bear Networks             PGP/GPG KeyID: 0x608D2A10
"Be who you are and say what you feel because those who mind don't matter
and those who matter don't mind." - Dr Seuss, "Oh the Places You'll Go"

---

From: Daniel Kahn Gillmor <[email-masked]>
Date: Wed, 08 May 2013 00:32:36 -0400

Thanks for these details, John.  This is exactly the sort of thing that
i wanted to start getting fleshed out.

> You /must/ mean documenting how those two are already implemented?

well, this is how they're implemented in SKS, which is the defacto
reference implementation, for sure.  so yes, documenting this in the
only public spec of the HKP protocol would be good.

> X-HKP-Results-Count: number of matching keys

This header (i think you're implying that it is an HTTP response header)
doesn't seem to be used at all in GnuPG if i'm searching
git://git.gnupg.org/gnupg.git properly.

I know there are other HKP client implementations but (like sks on the
server side) gnupg is a sort of defacto reference implementation.  If
it's not making use of this header, then it probably needs to be better
documented and patches pushed to gpg.

What should a responsible client do or assume if this header is absent
from an HKP response?

How should a responsible client respond if X-HKP-Results-Count is
greater than the limit query parameter?  for that matter, limit itself
doesn't seem to be documented in draft-shaw-openpgp-hkp-00.  This seems
like another detail worth including in any update.  looking in
dbserver.ml, i don't see any corresponding offset= query parameter.  so
if X-HKP-Results-Count == limit + 2, if an HKP client wants to get the
last two that were beyond the limit, they have to re-fetch the whole
list of keys.  Is that right?

> of wserver.ml may be helpful.

I'm thinking here of the protocol from the perspective of an HKP client;
i don't think a client should need to know the gory details of how any
particular HKP server is implemented in order to be able to interact
with it robustly.  While we may decide that there are certain error
cases that are common enough (or dangerous enough) to call them out
specifically in the protocol spec, it seems simplest to start by
specifying a general error code that we expect to be interpreted as
"Server broken, please try again later or try someone else" (perhaps
this is HTTP's "500 Internal Server Error").  It would also be useful to
provide guidance for how we expect a responsible HKP client to respond
to that error state.

For those of us who run sks or other keyservers behind reverse proxies,
there may also be system errors that come into play at the proxy level.
 So having an explicit understanding of what sorts of errors HKP clients
expect to see and how they should handle them would help to inform best
practices for reverse proxy configuration.

It might turn out that these answers are all very simple.  that's great!
 in that case, let's document them explicitly, and then we can set about
testing the various HKP clients and reverse proxies to ensure that they
behave reasonably in the face of the expected range of responses.

Regards,

	--dkg

---

From: John Clizbe <[email-masked]>
Date: Wed, 08 May 2013 07:25:07 -0500

> only public spec of the HKP protocol would be good.

Casey Marshall is well along in his implementation, Hockeypuck, which if I
understood correctly, is implementing HKP with the behavior of SKS. He is
implementing the reconciliation process of SKS as well. He has implemented
them in Go.

> documented and patches pushed to gpg.

Yes, it's a HTTP header, returned since 1.1.2. See this thread for more
details about why it exists: [Sks-devel] SKS, Content-Length and HEAD requests
[https://lists.nongnu.org/archive/html/sks-devel/2010-11/threads.html#00000 ]

This may be getting a bit SKS-centric for the OpenPGP list.
--