Subject: #1053 Relationship to RFC 2822 - suggestion

From: Harald Tveit Alvestrand <[email-masked]>
Date: Fri, 12 Aug 2005 13:23:12 +0200

We've been around the bush of relationship to RFC 2822 a number of times.
This impacts #1052 (changes from RFC 2822), #1084 (Names for ABN 
forpductios redefining 2822 constructs).
Some clarification may be needed - I THINK we have a mostly-shared opinion, 
but we might benefit from being more explicit.

Current text from 2.1:

2.1  Base

   News articles MUST conform to the syntax specified in Section 3 of
   [RFC2822].  Netnews agents MAY also accept the obsolete syntax
   specified in Section 4 of [RFC2822], but they MUST NOT generate
   productions of such syntax.

   Agents MUST accept the obs-phrase construct (use of a phrase like
   "John Q. Public" without the use of quotes, see [RFC2822] section
   4.1) but they MUST NOT generate productions of such syntax.

Suggested, expanded text:

2.1  Base

   An article is said to be conformant to this specification if it
   conforms to the format specified in RFC 2822 section 3 and to the
   additional requirements of this specification.

   An article that uses the obsolete syntax specified in section 4 of
   [RFC2822], except for the two exceptions mentioned below, is NOT
   conformant to this specification.

   This document, and specifications that build upon it, specifies how
   to handle conformant articles. Handling of non-conformant articles
   is outside the scope of this specification; agents may choose
   to accept such articles, reject such articles or modify such articles.

   Agents conforming to this specification MUST generate only conformant
   articles.

   Articles are conformant if they use the obs-phrase construct
   (use of a phrase like "John Q. Public" without the use of quotes,
   see [RFC2822] section 4.1) but they MUST NOT generate productions
   of such syntax.

   Articles are conformant if they use the "GMT" timezone, as specified
   in section 3.1.2.

   The text below uses ABNF to specify restrictions on the syntax
   specified in [RFC2822]; this grammar is intended to be more restrictive
   than the RFC 2822 grammar.
   Articles must conform to both the ABNF specified in RFC 2822 and to the
   ABNF specified here, as applicable.

Does this make sense?

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 12 Aug 2005 22:57:41 +0200

Harald Tveit Alvestrand wrote:

>| Handling of non-conformant articles is outside the scope of
>| this specification; agents may choose to accept such
>| articles, reject such articles

So far it's fine...

>| or modify such articles.

...but that's one of the things I'd not say in a standard, even
if it's true.  Let's say "may" ... "accept such articles", not
more.  Obviously there's an implicit "maybe not" (= reject),
don't talk about it, and there's also a very hidden implicit
"how they accept it is their business" (= modify), but don't
say it, it's out of scope.

> Does this make sense?

Yes, I like it better than the old terse text, but please don't
mention "modify", that's the private horror show of those who
try to pull this stunt (e.g. mail2news gateways, news servers
fixing the magic SP, etc.)
                             Bye, Frank

---

From: Henry Spencer <[email-masked]>
Date: Fri, 12 Aug 2005 17:28:55 -0400 (EDT)

On Fri, 12 Aug 2005, Frank Ellermann wrote:
> if it's true.

Concur.  The standard should say nothing that might appear to endorse
modifying already-injected articles, except for the obvious requirement to
adjust Path and possibly Xref. 

                                                          Henry Spencer
                                                       [email-masked]

---

From: Bruce Lilly <[email-masked]>
Date: Sun, 14 Aug 2005 09:10:07 -0400

> forpductios redefining 2822 constructs).
[...]
>    to accept such articles, reject such articles or modify such articles.

Does USEPRO build upon "this specification" (presumably USEFOR, though you
didn't say so)?  UsePRO is going to have to deal with acceptance/rejection
of proto-articles/articles, and modifications (by gateways).
 
>    Agents conforming to this specification MUST generate only conformant

"generate" is ambiguous; if a non-conforming article is received by a
transport agent which then modifies Path and retransmits the modified
article (but w/o changing whatever caused the article to be non-conformant),
does that constitute conforming behavior?  John says no, the non-conformant
article MUST be rejected, and I tend to agree with his reasoning.

>    see [RFC2822] section 4.1) but they MUST NOT generate productions

I agree with John's comments.  I would add that it's rather silly to make
an exception for something which is hardly necessary *and* which is
expressly forbidden by any article generators.
 
>    Articles are conformant if they use the "GMT" timezone, as specified

That is a silly, inconsistent, and unnecessary difference from 2822.  It is
silly because it singles out one alphabetic zone abbreviation from many,
all of which are in current use in Usenet, and all of which are treated
uniformly by RFCs 561, 724, 733, 822, 850, 1036, and 2822.  The
inconsistency is explained below.  I believe that the lack of necessity is
evident from past discussion as well as the fact that there is no such
exception for any other use of the Internet Message Format.

>    ABNF specified here, as applicable.

That fails to indicate *why* syntax is specified to be "more restrictive"
(in the case of "GMT" it is in fact *less* restrictive, which is
inconsistent with the claim of being "more restrictive").  It also fails
to provide a roadmap to full conformance with the Internet Message Format,
which we are agreed is desirable.  There is also the issue of field and
field component naming conflicts with 2822.  And any remaining differences
are going to have to be dealt with by gateways, so we ought to take time
to carefully review the implications and relevant USEPRO text regarding
gateways before finalizing USEFOR.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Mon, 15 Aug 2005 10:22:12 +0200

> say it, it's out of scope.

I wonder at the value of trying to keep this skeleton in the closet.

There are cases where software DOES modify articles (outside of the Path 
and Xref headers), and where it will continue to do so. We don't want to 
open the pandora's box of specifying which changes are OK and which are not.

The situation I'd like to handle cleanly:

 - Person A: Person B's server mangled my article!
 - Person B: Person A's article was broken - I repaired it!
 - Person A: But the standard says you can't change articles!

I'd like Person B to say "Your article was broken, so it's outside the 
rules", and Person A to say "OK, I agree the standard says that, I'll fix 
my UA and stop whining" (or whatever).

If that's a laudable goal (is it?), are we best served by mentioning the M 
word (modify) in our description of what can happen to nonFrom [email-masked] Mon Aug 15 04:47:44 2005
Date: Mon, 15 Aug 2005 10:22:12 +0200
From: Harald Tveit Alvestrand <[email-masked]>
To: Frank Ellermann <[email-masked]>, [email-masked]
Subject: Re: #1053 Relationship to RFC 2822 - suggestion
Sender: [email-masked]

> say it, it's out of scope.

I wonder at the value of trying to keep this skeleton in the closet.

There are cases where software DOES modify articles (outside of the Path 
and Xref headers), and where it will continue to do so. We don't want to 
open the pandora's box of specifying which changes are OK and which are not.

The situation I'd like to handle cleanly:

 - Person A: Person B's server mangled my article!
 - Person B: Person A's article was broken - I repaired it!
 - Person A: But the standard says you can't change articles!

I'd like Person B to say "Your article was broken, so it's outside the 
rules", and Person A to say "OK, I agree the standard says that, I'll fix 
my UA and stop whining" (or whatever).

If that's a laudable goal (is it?), are we best served by mentioning the M 
word (modify) in our description of what can happen to non-conformant 
articles (because it's beyond the scope of the standard), or are we best 
served by not mentioning it?

                     Harald

---

From: Henry Spencer <[email-masked]>
Date: Mon, 15 Aug 2005 10:57:05 -0400 (EDT)

> open the pandora's box of specifying which changes are OK and which are not.

Correct:  we want to forbid them all.  That software is *broken*.

"Neglect of duty does not cease, by repetition, to be neglect of duty."
(Napoleon)

For one thing, the user doesn't necessarily get feedback that such changes
are happening.  For another, they potentially bollix up the "same message ID
means identical article" principle.  For a third, people who write servers
that mangle articles often have only the vaguest notion of what's conforming
and what's non-conforming.

The best solution really is to outlaw all such (post-injection,
non-Path/Xref) changes, whether the article is conforming or not. 

                                                          Henry Spencer
                                                       [email-masked]

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 15 Aug 2005 17:21:06 +0200

> I wonder at the value of trying to keep this skeleton in

It could be like the CRLF in 2234bis:  Bruce proposed to say
[CR] LF, maybe ( [CR] LF ) / CR would be more accurate.  But
the best solution it to stick to CRLF as the only "line end"
in ABNF.

What the IETF ftp servers really do with I-Ds containing any
ABNF is beside the point for this issue.  This CRLF is a real
fetish, and "message modification" is a real taboo.

Of course some gateway operators will know it better, but for
those who don't know it the more important message is "never
try to 'fix' messages".

> We don't want to open the pandora's box of specifying which
> changes are OK and which are not.

Yes, only the specified modifications are allowed.  All other
modifications are formally illegal - they can have dangerous
side effects.  E.g. destroy header signatures.  Or with our
funny length limit for the <msg-id> and other less obscure
restrictions, whatever a mai2news gateway decides to do with
an invalid <msg-id>, it will be wrong.  The least damage is
probably "reject" depending on the gateway.

> I'd like Person B to say "Your article was broken, so it's
> outside the rules", and Person A to say "OK, I agree the
> standard says that, I'll fix my UA and stop whining" (or
> whatever).

Tricky.  How do you define "repair" for a broken / missing
<msg-id>, or utter dubious MESSFOR-minus-USEFOR <msg-id>s ?

A mail2news gateway will be hit by stuff legal in mail but
not news.  There are many ways to (try to) "fix" problems,
and for most wannabe-solutions you'd find cases where they
fail miserably making it worse.

Person B could say "if I don't try to 'fix' it I'm forced
to reject it".  Person A could insist on rejecting it.  It
is a can of worms, and near to non-technical legal trouble.

 [the M-word]
> are we best served by not mentioning it?

IANAL.  I'd stay away from it, out of scope is out of scope.

                           Bye, Frank

---

From: Bruce Lilly <[email-masked]>
Date: Tue, 16 Aug 2005 19:07:23 -0400

On Mon August 15 2005 11:21, Frank Ellermann wrote:

[off-topic digression]
> the best solution it to stick to CRLF as the only "line end"

As both of the implementations cited in the so-called interoperability
report for ABNF in fact implement what was proposed for possible
corrections to the specification (one does one thing, the other does
the other), *and* as actual drafts and RFCs using ABNF do not contain
CRLF unless modified from their native form in the IETF repository,
the ABNF specification as written really has a problem that ought to
be addressed (several problems actually, that's just one of them).

We now return to on-topic issues...
 
> "message modification" is a real taboo.
> 

If article syntax is fully conformant and compatible with RFC 2822, there
is no need for even a gateway to perform any modifications.  But once you
(yes, you Frank) insist on a difference from 2822, then gateways are
forced into making modifications (or being anal retentive and rejecting
submitted messages).  Wasn't it also you, Frank, who pointed to gmane as
an example of a "gateway" that specifically made msg-id modifications?
Sorry Frank, you can't have your cake and eat it too.
 
> probably "reject" depending on the gateway.

The best thing to do is to minimize the differences from 2822 to the
absolute minimum, with rationale for each and a roadmap to full
compatibility.
 
> <msg-id>, or utter dubious MESSFOR-minus-USEFOR <msg-id>s ?

During transport, only Path should be modified, and only in 
specified ways.  In the given example, I'd expect "B" to say "OK, I
agree that articles shouldn't be modified (other than trace fields) in
transit, and I'll fix my server accordingly; it will silently drop
broken [non-]articles".
 
> fail miserably making it worse.

Again, the best way to minimize that problem is to minimize the
differences from 2822.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 16 Aug 2005 21:36:52 GMT

>gateways before finalizing USEFOR.

On the contrary, we have NOT agreed that a roadmap to full conformance
with the Internet Message Format is desirable. Indeed, there are some
differences which are intended to remain permanently different from RFC
2822, except insofar as some RFC 2822bis may agree to an adjustment (which
many of us will be pressing for at the proper time).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 16 Aug 2005 21:31:44 GMT

>Current text from 2.1:

Yes, I agree the current text is a bit too terse. The new one is better,
but maybe needs a little polishing.

>   additional requirements of this specification.

"An article is conformant ...".

>   conformant to this specification.

Might be simpler to give the two exceptions, bulleted, at this point, to
save people having to backtrack later.

>   to accept such articles, reject such articles or modify such articles.

That "modify" has attracted much comment.

The current situation, within USEPRO, is that articles may only be
modified in three circumstances (variant headers apart):

1. Injecting agents are permitted a heavily circumscribed list of changes
(that list was heavily fought over long long ago, so please do not reopen
it now).

Observe that "fixing" things that are broken is NOT in that list, and the
consensus in the recent discussion seems to be that it should not be. An
injecting agent is near enough to the original poster to be able to get
an error message back to him. So you reject the malformed article and
invite him to fix it.

2. Moderators have a limited authority to change things.

3. Gateways are permitted to change whatever is necessary to make the
article suitable for the new medium (and detailed examples of likely
problems are discussed). Beyond that, they should leave things alone.

Observe that relaying and serving agents have only two options open to
them. Either accept the article as is (warts and alleged brokenness and
all - then it becomes someone else's proplem), or else drop it on the
floor silently.

So whatever you say about modification, make sure that it is only
modifications explicitly permitted by our standard or its extensions.
Saying nothing at all would fulfil that aim.

>   Agents conforming to this specification MUST generate only conformant

Care is needed with the meaning of "generate", as others have mentioned.
Agents involved in transport are not obliged to check everything, and may
thus (inadvertently or by design) pass articles that are dubious (someone
else's problem - see above). So I think "generate" only applies strictly
to posting agents, or other places where modifications (incl. variant
headers) are made.

>   see [RFC2822] section 4.1) but they MUST NOT generate productions

"...but productions of such syntax MUST NOT be generated."
s/obs-phrase/<obs-phrase>/
and maybe mention that it is really an extended-phrase, and RFC 2822bis
will undoubtedly rename it.

>   Articles are conformant if they use the "GMT" timezone, as specified

mention that "GMT" is extracted from <obs-zone>.

>   ABNF specified here, as applicable.

Various people have commented on this.

"Articles must conform to the ABNF specified in RCFC 2822, except insofar
as it is superseded/extended/modified by this specification/standard.
Observe that the forms permitted by this specification/standard
nevertheless remain a strict subset of those permitted by RFC 2822."

Observe that there are two 's's in "superseded" :-) .

---

From: Russ Allbery <[email-masked]>
Date: Tue, 16 Aug 2005 20:09:03 -0700

Bruce Lilly <[email-masked]> writes:

> That fails to indicate *why* syntax is specified to be "more
> restrictive" (in the case of "GMT" it is in fact *less* restrictive,
> which is inconsistent with the claim of being "more restrictive").  It
> also fails to provide a roadmap to full conformance with the Internet
> Message Format, which we are agreed is desirable.

Desireable, maybe.  It's only going to be possible if the Internet Message
Format gets considerably more restrictive about message IDs, to name one
major issue.  Until that happens, we won't ever be completely identical to
the RFC 2822 (or successor) format.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 17 Aug 2005 09:16:16 +0200

Bruce Lilly wrote:

> If article syntax is fully conformant and compatible with
> RFC 2822, there is no need for even a gateway to perform any
> modifications.

That's of course wrong, A <-- gateway --> B.  The gateway has
to transform A-format into B-format (or vice versa), and it
only gets away with no modifications for a proper subset A'
of A which happens to be also a subset of B.

So if it gets a valid 8bit NetNews article and sends it to a
mailing list it might have to modify it for 7bit as needed.

If the news article was in MIME format there's even a serious
chance to get it right.  One of the reasons why we say "MUST
support MIME", and why s-o-1036 pushed for MIME 11 years ago.

Similar problems in the opposite direction, watch the magic SP,
modify In-Reply-To to References if there are none, create a
Message-ID if there is none (but take care with that detail),
the works.

You can even operate Fido <--> RfC gateways, the special case
Internet / mail <--> news / Usenet is relatively simple.  The
most difficult part is to deter idiots from causing havoc, it's
not as trivial as "garbage in garbage out" (GiGo was a broken
gateway product a decade ago).

> once you (yes, you Frank) insist on a difference from 2822,

There _are_ differences, I didn't _invent_ them.  I was a Fido
user when Henry wrote s-o-1036.  And he also didn't _invent_
them, he found them, trying to minimize them.  Same procedure
as now 11 years later.

> gateways are forced into making modifications

That's the definition of a gateway, yes.

> (or being anal retentive and rejecting submitted messages)

Sure, before you cause havoc reject.  That's not "anal", that's
your responsibilty as gateway operator.  Or as an admin of any
public server.

> Wasn't it also you, Frank, who pointed to gmane as an example
> of a "gateway" that specifically made msg-id modifications?

Yes, and it's no problem in this case, GMaNe isn't a mailing
lists <--> UseNet gateway, it's mailing lists <--> News, only
on its own server.  GMaNe "pretends" that mailing lists are
moderated newsgroups using its own gmane.all newsgropus names.

That has some consequences if the "same" Message-ID arrives
at different times in different lists.  Unlike Usenet a mail
Message-ID doesn't guarantee identity, if list A did this
with the mail, and list B did that with the same mail, then
what GMaNe sees are in fact slightly different articles with
the same Message-ID.  Therefore it has to do something about
it in this case.  It's mostly transparent for the users.

> The best thing to do is to minimize the differences from
> 2822 to the absolute minimum, with rationale for each and
> a roadmap to full compatibility.

That roadmap is _ready_ for the <msg-id>, you find (a variant
of) it in the Usefor draft.  2822bis will copy it as is for
its future "chapter 3", moving NO-WS-CTL and other creatures
to its future obs-chapter where they can die.

Maybe it won't adopt the less convincing idea to forbid ">" or
the length limit, but after another iteration we might end up
with one syntactically identical concept for both worlds.  Bye

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 17 Aug 2005 09:48:52 +0200

> Saying nothing at all would fulfil that aim.

ACK so far.

> and maybe mention that it is really an extended-phrase,
> and RFC 2822bis will undoubtedly rename it.

NAK, that's an 2822-oddity, we have more than enough own
oddities, renaming <obs-phrase> isn't our job.

> mention that "GMT" is extracted from <obs-zone>.

The proposal is for the intro, chapter 2.1, the details
+ rationale for GMT are explained in 3.1.2, doing this
twice doesn't help.  The "message" of chapter 2.1 is
"2822 minus obs, but keeping two popular obs-ceneties".

Details would be a destraction at this point (in 2.1).

                   Bye, Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 18 Aug 2005 13:36:04 GMT

>oddities, renaming <obs-phrase> isn't our job.

Sure. I don't want to change any ABNF. I merely want to point out (in
parenthesis or a NOTE) that it is not actually an "obsolete" feature in
RFC 2822 (it was a new invention in 2822, and might even become "MAY
generate" at some future date).

---

From: Bruce Lilly <[email-masked]>
Date: Sun, 21 Aug 2005 12:42:10 -0400

> of A which happens to be also a subset of B.

If the syntax is fully conformant and compatible, there is no need for any
transformation; that and its associated problems only arise when
incompatibilities are introduced.

> Similar problems in the opposite direction, watch the magic SP,

That's an example of an incompatibility.

---

> Message-ID doesn't guarantee identity,

Please reread RFC 2822 section 3.6.4, in particular:

   The "Message-ID:" field provides a unique message identifier that
   refers to a particular version of a particular message.  The
   uniqueness of the message identifier is guaranteed by the host that
   generates it (see below).  This message identifier is intended to be
   machine readable and not necessarily meaningful to humans.  A message
   identifier pertains to exactly one instantiation of a particular
   message; subsequent revisions to the message each receive new message
   identifiers.

> to its future obs-chapter where they can die.

Do you have evidence to support that claim, or is it wishful thinking?

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Sun, 21 Aug 2005 20:21:50 +0200

> nevertheless remain a strict subset of those permitted by RFC 2822."

Since there are no extensions or modifications, just restrictions:

Articles must conform to the ABNF specified in RFC 2822.
Articles must also conform to the restrictions specified here, both those 
that are expressed as text and those that are expressed as ABNF.

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 22 Aug 2005 07:34:40 +0200

> Please reread RFC 2822 section 3.6.4, in particular:

That doesn't talk about mailing lists doing odd and different
things to a mail - they arrive in different forms with the
same Message-ID, and for a gateway there's no reliable way
to determine the "original" form without manual configuration
of the "particular mailing list oddity".  And sometimes these
oddities depend on text/* vs. multipart/*, it's messy.

For a gateway operator the 2822 ideals are pure fantasies :-(

>| subsequent revisions to the message each receive new
>| message identifiers.

Mailing lists mangling the bodies and consequently using new
Message-IDs would be a nightmare.  It's bad enough as it is.

>> That roadmap is _ready_ for the <msg-id>, you find (a
>> variant of) it in the Usefor draft.  2822bis will copy it
>> as is for its future "chapter 3", moving NO-WS-CTL and other
>> creatures to its future obs-chapter where they can die.
 
> Do you have evidence to support that claim, or is it wishful

No, I consulted my crystal ball.  You also try this sometimes,
when you have certain ideas what will or will not (future) pass
"IESG muster".
                            Bye, Frank

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 22 Aug 2005 07:50:17 +0200

Charles Lindsey wrote:

> I merely want to point out (in parenthesis or a NOTE) that
> it is not actually an "obsolete" feature in RFC 2822

There is already a note about this in 2822, I see no reason
to copy this to USEFOR, we're interested in the differences
from 1036 and 2822, not identical stuff.

> might even become "MAY generate" at some future date

Maybe not.  IMHO it's an utter dubious "feature", not much
better than bare CR, bare LF, NUL, or NO-WS-CTL.  Bye, Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 22 Aug 2005 13:35:37 GMT

>Since there are no extensions or modifications, just restrictions:

I think restrictions are a subset of modifications (i.e. things modified
wrt 2822).

>that are expressed as text and those that are expressed as ABNF.

Indeed, but since the rewrite of the present bare text is for the purpose
of user friendliness, then I think it is useful to draw attention to the
fact that our articles are a strict subset of 2822 messages.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 22 Aug 2005 13:35:37 GMT