Subject: Re: [openpgp] User ID conventions (it's not really a RFC2822 name-addr)

From: "Neal H. Walfield" <[email-masked]>
Date: Tue, 05 Nov 2019 23:35:11 +0100

I'm considering using the following "grammar".  (I've put grammar in
scare quotes, because it is not a valid grammar according to RFC 5322
due to several ambiguities.  In particular, the production "*WS
[name] *WS" is ambiguous when applied to a string containing a single
whitespace character: the whitespace character could match the first
WS or the second one.  In practice, this ambiguity doesn't matter,
because we only care about what the "name", "comment-content" and
"addr-spec" productions match.)

     WS                 = 0x20 (space character)

     comment-specials   = "<" / ">" /   ; RFC 2822 specials - "(" and ")"
                          "[" / "]" /
                          ":" / ";" /
                          "@" / "\" /
                          "," / "." /
                          DQUOTE

     atext-specials     = "(" / ")" /   ; RFC 2822 specials - "<" and ">".
                          "[" / "]" /
                          ":" / ";" /
                          "@" / "\" /
                          "," / "." /
                          DQUOTE

     atext              = ALPHA / DIGIT /   ; Any character except controls,
                          "!" / "#" /       ;  SP, and specials.
                          "$" / "%" /       ;  Used for atoms
                          "&" / "'" /
                          "*" / "+" /
                          "-" / "/" /
                          "=" / "?" /
                          "^" / "_" /
                          "`" / "{" /
                          "|" / "}" /
                          "~" /
                          \u{80}-\u{10ffff} ; Non-ascii, non-control UTF-8

     name-char-start    = atext / atext-specials

     name-char-rest     = atext / atext-specials / WS

     name               = name-char-start *name-char-rest

     comment-char       = atext / comment-specials / WS

     comment-content    = *comment-char

     comment            = "(" *WS comment-content *WS ")"

     addr-spec          = dot-atom-text "@" dot-atom-text

     pgp-uid-convention = addr-spec /
                          *WS [name] *WS [comment] *WS "<" addr-spec ">" /
                          *WS name *WS [comment] *WS

Beyond being more fleshed out, this grammar is different from the
grammar in dkg's second proposal in a few ways.

First, it matches comments.  dkg made this a non-goal.  Given that
people who add comments intend them as comments and not as part of
their name, it seems reasonable to me to not display comments in
places where only the user's name is desired.  And, since it turns out
that matching non-nested comments is relatively straightforward, why
not?  Note: doing this might actually help deprecate comments, because
they won't be shown as often.

The grammar more carefully handles whitespace.  It ignores whitespace
at the beginning of the User ID (this is what motivates the
name-char-start production) and between the individual components in
the pgp-uid-convention production.  As is, the grammar only ignores
the 0x20 space character.  We may also want to include the tab
character, unicode's NO-BREAK SPACE (U+00A0) character and its
IDEOGRAPHIC SPACE (U+3000) character for thoroughness.  But, since
software will normally concatenate the individual components, just
recognizing the ASCII space character here is probably fine.  Whatever
the case, I think we can safely ignore the rest of unicode's
whitespace characters:

  https://en.wikipedia.org/wiki/Whitespace_character

My pgp-uid-convention production also matches user ids without email
addresses, e.g., "Daniel Kahn Gillmor".  This is convenient.  Instead
of having to figure out why parsing failed (is it not valid UTF-8? is
it just missing an addr-spec?), we explicitly cover this common
pattern in the grammar.  I think this will significantly simplify code
that uses this interface: if there is an error, then the code can just
assume the User ID is trash and can be ignored.

In RFC 2822, "specials" are only allowed in a display name if they are
quoted.  dkg removes this requirements.  I think this is mostly
sensible, but it means that we can have User IDs like:
"<[email-masked]> <[email-masked]>" where the first
<[email-masked]> is the display name and the second is the addr-spec.
I think we should exclude angle brackets from the display name.  In my
grammar, I have an "atext-specials" which is just RFC 2822 specials
without the angle brackets.

I'm a bit concerned about allowing the backslash character: with this
grammar, it is just a normal character, but for an RFC 2822 parser,
it's an escape character.  Since User IDs may be used in contexts
where RFC 2822 things are expected, we should be careful.  But, I fear
that if we reject it, we'll end up gratuitiously rejecting some
emojis.  ¯\_(ツ)_/¯.

:) Neal

---

From: "brian m. carlson" <[email-masked]>
Date: Wed, 6 Nov 2019 00:05:46 +0000

>      comment-specials   = "<" / ">" /   ; RFC 2822 specials - "(" and ")"

dot-atom-text isn't defined here, so it isn't clear to me what it
includes.  Does it permit UTF-8 in addresses according to the SMTPUTF8
RFCs?
-- 
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204

---

From: "Neal H. Walfield" <[email-masked]>
Date: Wed, 06 Nov 2019 08:37:22 +0100

Hi Brian,

On Wed, 06 Nov 2019 01:05:46 +0100,
> includes.  Does it permit UTF-8 in addresses according to the SMTPUTF8

Thanks for catching that.  When turning my code into a grammar, I
somehow forgot that production.

The dot_atom_text is unchanged from e.g. RFC 2822:

   dot_atom_text      = 1*atext *("." *atext)

But since we've extended atext to include non-control UTF-8
characters, this should allow international email addresses.

RFC 6531 (the SMTPUTF8 RFC) extends atext as follows:

  atext   =/  UTF8-non-ascii
    ; extend the implicit definition of atext in
    ; RFC 5321, Section 4.1.2, which ultimately points to
    ; the actual definition in RFC 5322, Section 3.2.3

  https://tools.ietf.org/html/rfc6531#section-3.3

which, I think, is what I did above.

But, I've only skimmed RFC 6531 so I might have missed something else.

Thanks!

:) Neal

---

From: "Neal H. Walfield" <[email-masked]>
Date: Wed, 06 Nov 2019 12:49:08 +0100

On Tue, 05 Nov 2019 23:35:11 +0100,
> that if we reject it, we'll end up gratuitiously rejecting some

The double quote character has a similar problem.

---

From: Daniel Kahn Gillmor <[email-masked]>
Date: Wed, 06 Nov 2019 01:37:14 -0500

Hi Neal--

Thanks for this thoughtful writeup!

> they won't be shown as often.

User IDs are full UTF-8 strings.  The idea that any part of that string
would be hidden from the user is pretty disturbing to me.  Consider the
situation where someone is certifying a user ID on an OpenPGP
certificate.  If the comment is hidden, do they know what identity
assertion they're making?

I'd much rather have comments be deprecated *because they are weird and
show up in places where you'd think your name should go* rather than
have them be some vestigial thing that people don't even notice any
longer.

I would recommend dropping the comment from your grammar and letting the
"name" part subsume it, when you're splitting out e-mail address from
the rest of the user ID.

Furthermore, because you've allowed "(" and ")" in atext-specials, it
looks to me like your proposed grammar is ambiguous:

    bob (joe) <[email-masked]>

is either:

    name: "bob (joe)"
    comment: None
    addr-spec: "[email-masked]"

or:

    name: "bob"
    comment: "joe"
    addr-spec: "[email-masked]"

I don't think this is helpful to anyone.

>   https://en.wikipedia.org/wiki/Whitespace_character

I'm fine with being judicious about selecting whitespace characters.  In
addition to tab (U+0009, ascii "HT"), i note that you've declined to
include U+000A and U+000D (ascii "LF" and "CR") in the grammar at all.

I like that kind of opinionated decision, as unprintable symbols like
this are likely to be problematic in many ways (hard for users to
distinguish at least!)

I also think that whitespace at the beginning of a user ID is asking for
trouble, and would be happy with a grammar that considers that user ID
non-conventional.  Is there a use case for leading whitespace in a user
ID?

> assume the User ID is trash and can be ignored.

I should be clear that i intended my earlier proposal specifically to
match OpenPGP User ID conventions *that have an e-mail address in
them*.  There are indeed other User ID conventions (like "Daniel Kahn
Gillmor", or "ssh://foo.example") that aren't covered by this, and i
thought i would be doing folks a favor by focusing on the e-mail address
side of things specifically.  My thought was that common interfaces
would allow for matching against a User ID that has an e-mail address,
and then they would have other matchers for other common conventions
that they could try applying if this convention didn't match.

This is probably an implementation detail, though.

> without the angle brackets.

I totally agree with this constraint.  If you're doing away with
comments (as i recommend above) then you would have to prohibit angle
brackets in commas too, which seems fine to me.

Even if you decide to go ahead with splitting out comments, I would go
so far as to ban them in comments too.  is there any plausible reason
for including angle brackets in a comment?  Simplify simplify :)

> that if we reject it, we'll end up gratuitiously rejecting some

There are all kinds of things that will break if implementations
casually stick OpenPGP user IDs into an e-mail header, not just
backslashes.  for example, commas are likely to cause a problem.
consider trying to mail two people whose OpenPGP certificates have these
User IDs:

    Lucy Hernandez, MD <[email-masked]>
    Chuck Wilson, Jr. <[email-masked]>

A simple concatenation with commas yields the disastrous:

To: Lucy Hernandez, MD <[email-masked]>, Chuck Wilson, Jr. <[email-masked]>

and DQUOTE is just as bad if not worse :)

So i have no problem with including backslash in the display name area.

    --dkg

---

From: "brian m. carlson" <[email-masked]>
Date: Wed, 6 Nov 2019 23:32:49 +0000

> But, I've only skimmed RFC 6531 so I might have missed something else.

Yup, in that case, I have no objections to your grammar.  It seems fine
to me.
-- 
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204