Subject: #1084 USEFOR 2.1, 3: Names for ABNF productions redefining 822 constructs

From: Harald Tveit Alvestrand <[email-masked]>
Date: Thu, 21 Jul 2005 14:38:29 +0200

I've created the following ticket:
----------------------------------------------------
Subject: USEFOR 2.1, 3: Names for ABNF productions redefining 822 constructs

>From Bruce Lilly July 20:

 The issue persists in usefor-05:

 from = "From:" SP mailbox-list CRLF

 That differs from RFC 2822:

 from = "From:" mailbox-list CRLF

 The rule name "from" appears multiple places in RFC 2822; we have no
authority to alter the meaning of RFC 2822.
If there is a good reason
to define ABNF -- and AFAICT there is not as the SP issue can be
handled via an implementation note along the lines suggested by Mark
Crispin -- then simply choose a different rule name so that it does
not conflict with rule names used in references. E.g. "usefor-from"
or "blurfl".
------------------------------------------------------

I've noted Frank's comment.

Personal comment: I don't find the issue very important, since I don't find 
the redefinitions confusing. Your mileage may differ.

>From a formalistic standpoint (and making the ABNF easier to check as an 
extension to the 2822 grammar), I think there's something to be said for an 
intro saying something like:

  This document defines a number of ABNF productions that are named
  "news-" and the name of an RFC 2822 production (for instance
  "news-from"). This is intended to document a restriction in the Netnews
  article format compared to the RFC 2822 message format; a valid Netnews
  article will be a valid RFC 2822 message, but will also satisfy the
  ABNF specified in the "news-" rules.

And then adding "news" in front of the rules, of course.

Another alternative is just leaving them as-is.

Thoughts?

                          Harald

---

From: Henry Spencer <[email-masked]>
Date: Thu, 21 Jul 2005 11:36:22 -0400 (EDT)

On Thu, 21 Jul 2005, Harald Tveit Alvestrand wrote:
>   "news-" and the name of an RFC 2822 production...

This seems like a reasonable idea, although I don't have strong feelings
about it either way -- leaving them as-is would work for me too.

                                                          Henry Spencer
                                                       [email-masked]

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 22 Jul 2005 16:00:18 GMT

>Thoughts?

It ain't broke. Don't fix it.

More specifically, the syntax of Netnews is different from that of Email,
even though large parts of it are in common. We should not disguise that
fact. There are rules in Email that are absent from Netnews (most of the
obs-syntax). There are rules in Netnews that are absent from Email. And
there are rules that are plain different. But we are still a strict subset
of Email, and that needs to be stated clearly.

And of course, we must draw attention where the rules are different (which
we do, usually using the word "restricted").

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Russ Allbery <[email-masked]>
Date: Fri, 22 Jul 2005 09:20:47 -0700

Harald Tveit Alvestrand <[email-masked]> writes:

> Personal comment: I don't find the issue very important, since I don't
> find the redefinitions confusing. Your mileage may differ.

> extension to the 2822 grammar), I think there's something to be said for
> an intro saying something like:

>   "news-from"). This is intended to document a restriction in the
>   Netnews article format compared to the RFC 2822 message format; a
>   valid Netnews article will be a valid RFC 2822 message, but will also
>   satisfy the ABNF specified in the "news-" rules.

> Thoughts?

I'm okay with leaving them as-is, since like you, I don't find them
confusing, but there's certainly some appeal in the above.  I think it's a
slightly better option, although I don't know whether it's worth the
additional work.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: Bruce Lilly <[email-masked]>
Date: Fri, 22 Jul 2005 14:21:47 -0400

And then went on to say:
 
> fact.

Disguising the differences is precisely what using an established ABNF
rule name with a different definition does.  It is broken, and needs to
be fixed.

> There are rules in Email that are absent from Netnews (most of the 

We're discussing the Internet Message Format, not "Email".  SMTP as a
mail-specific protocol is nearly irrelevant to our work, whereas the
Internet Message Format is a normative reference and is relevant because
messages (news, mail, and other) are handled by various protocols (e.g.
IMAP) which are in widespread use in Usenet.

My preference remains to use implementation notes which clearly explain
*how* things differ, *why* there is a difference, *where* and *when* the
difference matters, and *what* might be done, by *whom*, to minimize the
differences.

RFCs 850 and 1036 (section 2 of both RFCs) very clearly chose to use the
Internet Message Format, replacing an older format, for reasons clearly
explained in those RFCs: to fit in with existing tools.  Each difference
reduces the applicability of existing tools.  The long term goal should
be convergence with the long-standing, widely-used Internet Message
Format which was adopted for use by RFC 850.  Reaching that goal is
facilitated by clearly explaining the how, why, where, when, and what for
each difference, and is impeded by disguising the differences by violating
the methodology which we have stated that we use (usefor sect. 1.6).

A redefinition -- regardless of the name -- is unsatisfactory.  Tools and
protocols deal with messages, and a From field in a message needs to have
precisely one definition.  It's OK to add restrictions on content where
necessary, but having conflicting definitions in ABNF is a problem.
Given a message, it's one thing for a tool or protocol to observe "this
is a From field meeting *the* syntax of RFC 2822, and by the way the field
body of this instance happens to begin with SP" and quite a different matter
to say to IMAP (e.g.) implementers "here are two conflicting definitions of
*the* syntax for a From field; have a nice day".

A redefinition of "from" is the latter and is a problem.  It also has the
unfortunate characteristic of not clearly indicating whether or not the
redefinition is for generation or for parsing or for both; specifically it
is unclear in the redefinition of "from" whether or not parsers are required
to enforce the SP presence, whether they're permitted to reformat hard
drives, etc. if it's absent, etc.  Defining something called "news-gen-from"
that specifies a news-specific generation syntax and something called
"news-parse-from" that specifies a news-specific parse syntax might be one
viable approach, but:
o it does not remove the need for a clear exposition of why, where, when,
  and what
o it requires careful scrutiny to ensure full compatibility
o it is going to require rework when the 2822 successor is published with
  grammar correcting some errors and eliminating [CFWS] [CFWS] ambiguities
  etc.
o it requires additional work for implementers to figure out from ABNF
  precisely what differs and how to deal with the differences
o it's going to require some text to explain how the -gen- and -parse-
  rules are used, and that text is likely to take up as much room as
  an implementation note which would obviate not only that explanatory
  text but dozens of ABNF lines.

It's much simpler, clearer, and overall more concise to say something like:

Implementation note:
  Although there is no requirement regarding a field body first character
  value in RFCs 822 or 2822, widely-deployed software "foo" crashes when
  a field body begins with a character other than SP.  Therefore, for
  backward compatibility with that deployed base, generators of messages
  conforming to this specification MUST begin all field bodies with SP.
  In the interest of compatibility with the Internet Message Format on
  which this specification is based, deployed implementations of foo
  SHOULD be upgraded to accept any legal field bodies permitted by RFC
  2822.

That:
a. explains how, why, where, when, and what, as well as who needs to take
   action
b. avoids a conflicting definition
c. obviates a rewrite when 2822 is updated (except for updating the RFC
   number).

It of course depends on the actual existence of some widely-deployed "foo",
rather than SP-after-colon being somebody's personal fetish.  After Frank's
tests, it's not clear that there is a "foo".

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 22 Jul 2005 23:31:25 +0200

Bruce Lilly wrote:

> It of course depends on the actual existence of some
> widely-deployed "foo", rather than SP-after-colon being
> somebody's personal fetish.  After Frank's tests, it's not
> clear that there is a "foo".

My two tests found two "smart foo", they added the fetish SP
on their own behalf.  A "stupid foo" could reject the article.

With header signatures it's far from clear what's "smart" and
what's "stupid", but we're still firmly at square one, the SP
is a required fetish.

Even if all injection agents of the world are "smart" there's
still a mandatory SP after the colon.  I've now also tested
the last chance for a major step forward towards RfC 2822:

<news://news.spamcop.net/[email-masked]>

That was with HT instead of SP.  No luck, the "smart" agent
replaced it by SP.  We're stuck with it, unfortunately.

I'd really love to meet halfway with RfC 2822, replace [FWS]
by MWS ("mandatory white space") and MWS = 1*WSP / obs-FWS

But that's of course only a dream, the mail-extremists (e.g.
you, but you'd say "message-fan") want their [FWS], and the
news-extremists love their fetish SP.

                           Bye, Frank