Subject: Admin: I'm back.....

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 19 Jul 2005 09:50:50 +0200

I see that usefor-05 has arrived in my absence, along with 150 messages, 
none of which seem to talk about -05.

I'm currently catching up on mail, and will try to go back to resolving 
tickets when I'm caught up.

One note to Bruce: Using ABNF to define a restriction on a production 
defined in ABNF elsewhere is *not* creating a conflicting definition, 
providing the surrounding text makes it clear that's what's happening.

If you find a value that is accepted by the "restrictive" ABNF and not 
accepted by the "less restrictive" ABNF, you have reason for an objection. 
But not just because ABNF is used to define the restriction.

                       Harald

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Tue, 19 Jul 2005 09:58:08 -0400

> providing the surrounding text makes it clear that's what's happening.

Good, then someone please propose that surrounding text, and all the places
it should go.  I assume the text will have the effect that implementors
will know they can use RFC2822 for parsing and USEFOR for production of
header fields.

---

From: Bruce Lilly <[email-masked]>
Date: Wed, 20 Jul 2005 08:50:06 -0400

> But not just because ABNF is used to define the restriction.

It would help if you would provide some context for your comments.

I can recall no discussion based on "just because ABNF is used".

There has been discussion related to conflicts caused by cases of
the form where a normative reference defines some ABNF production
rule "foo", and our document gives a different ABNF rule named "foo".
That is a conflict, one which can be easily resolved by choosing a
different rule name.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 20 Jul 2005 15:01:24 +0200

--On onsdag, juli 20, 2005 08:50:06 -0400 Bruce Lilly <[email-masked]> 
wrote:

>> accepted by the "less restrictive" ABNF, you have reason for an
>> objection.  But not just because ABNF is used to define the restriction.
>

Context, your message from July 3 with subject "Required SP" (didn't 
realize it was from that long ago):

> 2822 defines the From field (among others) and its syntax.  If
> you introduce a different syntax for the same thing, you are
> either trying to alter that syntax or you are introducing a
> conflict (in this case with a normative reference).  Best to
> keep the ABNF grammar as defined in 2822, and introduce notes
> restricting usage of the full syntax where necessary or
> recommended.

Given its age, it was not reasonable of me to comment on it as if it was a 
presently active discussion. Apologies.

                           Harald

---

From: Bruce Lilly <[email-masked]>
Date: Wed, 20 Jul 2005 10:52:27 -0400

> presently active discussion. Apologies.

No problem -- just that it's difficult to respond out-of-context.

The issue persists in usefor-05:

   from            =  "From:" SP mailbox-list CRLF

That differs from RFC 2822:

from            =       "From:" mailbox-list CRLF

The rule name "from" appears multiple places in RFC 2822; we have no
authority to alter the meaning of RFC 2822.  If there is a good reason
to define ABNF -- and AFAICT there is not as the SP issue can be
handled via an implementation note along the lines suggested by Mark
Crispin -- then simply choose a different rule name so that it does
not conflict with rule names used in references.  E.g. "usefor-from"
or "blurfl".

As earlier noted, "from" is not the only instance of such a conflict.

In this case, I recommend an implementation note rather than renaming
ABNF rules.  The first bullet point in section 2.2 (usefor-05) could
easily be converted into such a note.

Specifically, OLD text:

   o  All agents MUST generate header fields so that at least one space
      immediately follows the ':' separating the header field name and
      the header field body (for compatibility with deployed software).
      News agents MAY accept header fields which do not contain the
      required space.

Suggested NEW text:

Backward Compatibility Requirements

   Some legacy injection agents reject messages which do not have an SP
   character following the colon delimiting a header field name from
   the field body.

Implementation Recommendations

   It is RECOMMENDED that an SP character be the first character of the
   field body for compatibility with legacy injection agents.  Agents
   SHOULD accept messages which do not have an SP character as the first
   character of the field body.  Note that RFC 2822 conformance REQUIRES
   acceptance of such messages.

As the issue does not warrant BCP 14 imperatives (MUST/MUST NOT), I have
suggested keywords appropriate to the (lack of) gravity of the situation.
See
http://www.imc.org/ietf-usefor/2003/Feb/0554.html
http://www.imc.org/ietf-usefor/2003/Feb/0493.html
http://www.imc.org/ietf-usefor/2003/Feb/0486.html

With the above text, the contradictory ABNF can be removed, avoiding
conflict with a normative reference and with the first paragraph of
section 1.6, and reducing the document length.

Incidentally, usefor-05 also has a self-contradictory definition of "from"
in section 3.1.2:

   from            =  "Date:" SP date-time CRLF

Was the ABNF checked prior to submission of usefor-05 as required by
the ID-Checklist (note that issues with ABNF have previously been noted
and a request was made that ABNF be checked)?  If so, what tool was used
for checking, and how was it used? [if all of the ABNF was input to
checking, a failure to flag conflicting definitions would be a serious
flaw in the tool; if only snippets of ABNF were separately checked, then
the ABNF as a whole wasn't really checked]  See:
http://www.ietf.org/ID-Checklist.html
http://www.imc.org/ietf-usefor/mail-archive/msg00742.html

There remain many other instances of conflicts with section 1.6 and with
RFC 2822, e.g. an attempted redefinition of "unstructured" in section 2.2,
redefinition of "message-id", etc.

At this late stage, we really shouldn't have unchecked ABNF, nor should
we expect dozens of individual WG members to manually have to check ABNF
when there are tools available and ABNF is required to be checked prior
to submission of a draft.  We also shouldn't have ABNF or text which
conflicts with section 1.6 first paragraph, and the document editor should
be (and should have been) pushing back hard against suggestions that
conflict with 1.6.

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 20 Jul 2005 13:41:44 GMT

>will know they can use RFC2822 for parsing and USEFOR for production of

The 1st para of 2.1 seems to give the ground rules.

Then I see, in the various places where changed ABNF is given:

         ...... As a result, this
         document updates the <unstructured> construct from Section
         3.2.6 of [RFC2822] as follows:
[I grant you that would be better said outside of a NOTE]

   The following news header fields extend those defined in section 3.6
   of [RFC2822]:

   ..... this section therefore restricts the syntax
   of <msg-id> as compared to Section 3.6.4 of [RFC2822]. ...

   The References header field is the same as that specified in Section
   3.6.4 of [RFC2822] with the added restrictions detailed in
   Section 2.2 and those listed below:
   ........
   o  Message identifiers MUST be separated with CFWS.

What more do you want?

I have also suggested, to Ken, adding at the end of the new Appendix C:

   NOTE: The effect of all these differences still preserves the property
   that the articles that Usefor permits to be generated form a proper
   subset of the articles that are required to be acceptable to RFC 2822.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 21 Jul 2005 11:40:33 +0200

[PEBKAC or re-post, sorry if that article appears twice]

Bruce Lilly wrote:

> The rule name "from" appears multiple places in RFC 2822; we
> have no authority to alter the meaning of RFC 2822.

We have every authority to do the right thing for NetNews, and
if it pleaes us we could even try "updates 2822".  But in the
case of the mandatory SP that's certainly not necessary.

> E.g. "usefor-from"

It's an idea for the minor cases, and <news-date> is arguably
better than <orig-date>.  OTOH <news-subject> etc. is dubious,
there's nothing special with it, only a mandatory SP, even the
erroneous optional trailing FWS was copied as is from RfC 2822.

> or "blurfl".

Nonsense names like <blurf1>, <id-left>, or <id-right> are a
bad idea.  The name should indicate the semantics where that's
possible.

> "from" is not the only instance of such a conflict.

There are several classes of "conflicts":

1 - minor changes like the mandatory SP, adding "news-" to the
    2822 name could make sense, but it's IMHO unecessary.

2 - major changes completely different from 2822 productions
    like <id-local>, <id-domain>, <id-literal>, and <id-quote>
    in my terminology - using the same names as in RfC 2822 is
    IMNSHO _wrong_ (and the 2822 names were already bad).

    The RfC 2822 "semantical content" for <no-fold-quote> and
    <no-fold-literal> differs from <id-quote> or <id-literal>.

    Abusing the old name fools nobody who knows what's this is
    about, but it could confuse implementors and cause harm.

3 - major changes intended to overload the equivalent concept
    in 2822, the <msg-id>.  The <references> are a border case.

>| legacy injection agents reject messages which do not have an
>| SP character following the colon delimiting a header field
>| name from the field body.

<news://news.spamcop.net/[email-masked]>
Posted without SP in all header fields => no problem

<news://news.gmane.org/[email-masked]>
Dito.  Note that the injection agent _added_ the missing
SPs, for all header fields.  That was a "real" newsgroup,
GMaNe's "junk" is unaffected by its gateway tricks.

Same effect in the first test, missing SP _added_ by SC.

Last but not least (but behind GMaNe's gateway magic) see
<http://article.gmane.org/gmane.test/2412/raw>

What I posted was this:

X-Mozilla-Status: 0001
Message-ID:<[email-masked]>
Date:Thu, 21 Jul 2005 09:04:51 +0200
From:Frank Ellermann <[email-masked]>
Newsgroups:gmane.test
Subject:no SP

no SP (2)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Can we now please return from the deserts of wishful thinking
to the reality of NetNews and USEFOR ?  If you want to add a
blurb about how proto-articles without the mandatory SP might
be fixed by injection agents to usePRO it's okay, but it does
not change the article format in useFOR.

> Note that RFC 2822 conformance REQUIRES acceptance of such
> messages.

Only relevant for gateways (in usePRO) if they intend to post
such crap, and their injection agent doesn't fix a missing SP.

And this _is_ in fact a job for posting / followup agents, for
the complete fun visit the planned MASS WG - if other agents
"fix" header fields it could cause havoc for header signatures.

                             Bye, Frank

---

From: Bruce Lilly <[email-masked]>
Date: Thu, 21 Jul 2005 09:35:10 -0400

> case of the mandatory SP that's certainly not necessary.

See usefor section 1.6 first paragraph.  Either we do what it says
we do or we need to change what it says (however, what it says is in
fact what we're supposed to do, as established ca. two years ago).

> erroneous optional trailing FWS was copied as is from RfC 2822.

The change imposes structure (SP) on an unstructured field.  There
is no "erroneous optional trailing FWS" in RFC 2822.

>     2822 name could make sense, but it's IMHO unecessary.

I'd prefer to see an implementation note and no ABNF (other than
that in our normative references).  As noted in the reference
messages which I supplied yesterday, that clearly indicates *why*
there is a difference, whereas a different ABNF not only doesn't
provide a reason, it adds confusion.
 
>     about, but it could confuse implementors and cause harm.

Again, these should be handled by implementation notes in order to
provide a rationale.
 
>     in 2822, the <msg-id>.  The <references> are a border case.

Ditto.
 
> Posted without SP in all header fields => no problem

IIRC, Russ reported that INN;s injection agent rejects such messages.
 
> such crap, and their injection agent doesn't fix a missing SP.

No. It's relevant for news/mail (i.e. messaging) UAs that wish to
conform to RFC 2822.

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 21 Jul 2005 17:11:33 +0200

Bruce Lilly wrote:

> See usefor section 1.6 first paragraph.  Either we do what it
> says we do or we need to change what it says

We do what it says, not more the complete syntax as in the old
<http://purl.net/xyzzy/usefor07.htm> but only the _differences_
from 2822.  No _subtle_ mandatory SP hidden somewhere in prose
and tons of more or less identical syntax, but as _obvious_ as
possible, as in (to do) orig-date = "Date:" SP date-time CRLF

Everybody who knows the 2822 <orig-date> will immediately see
the added SP.

> The change imposes structure (SP) on an unstructured field.

That's a weak argument, you could also say that the terminating
CRLF or the embedded [FWS] is an "imposed structure".

> There is no "erroneous optional trailing FWS" in RFC 2822.

Of course there is, but fortunately only one in <unstructured>,
compare <http://mid.gmane.org/[email-masked]>

In usefor-05 we have about 2 * 10 = 20 of the same [FWS] bugs -
they should be all replaced by *WSP.  But it won't surprise me
if the final compromise are ten "MUST NOT" in prose:

| Note: the leading and trailing [FWS] MUST NOT result in CRLF,
| it's a compromise between the correct *WSP and a 2822 [CFWS],
| let's see what the IESG thinks about this folly, but do not
| implement it.

After the latest "SHOULD NOT RECOMMENDED" stunt I canot trust
that they'll simply shoot us for this "engineering".  <sigh>

> these should be handled by implementation notes in order to
> provide a rationale.

The rationale for the <msg-id> modifications is quite complete.
Only the hidden agenda of "random garbage as <id-right> will do"
is obscured by "commonly derived from <domain>s".

With s/commonly/sometimes/ this part would be clearer.

> IIRC, Russ reported that INN;s injection agent rejects such

`telnet -p 119 news.spamcop.net` says "INN 2.3.2".
`telnet -p 119 news.gmane.org`   says "INN 2.4.1" (but for that
server it only means that Lars didn't modify the 200 message)

I've not tested other servers, these two added the missing SP.

> It's relevant for news/mail (i.e. messaging) UAs that wish to
> conform to RFC 2822.

The mandatory SP conforms to the message/rfc822 format, the UA
can always generate it.  In addition it must accept all mails
without the SP and some weird cases with WSP before the colon.

                        Bye, Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 21 Jul 2005 11:36:10 GMT

>   acceptance of such messages.

Those two texts are mot equivalent in their effects. The first represents
the consensus view of this WG as established almost from day 1 of its
formation. The second does not.

>suggested keywords appropriate to the (lack of) gravity of the situation.

The issue most certainly _does_ warrant such imperatives. There are severe
interoperability problems (and not just in "legacy injection agents").
Moreover, that SP is a REQUIRED feature of the new NNTP standard-to-be,
and there is no way that this WG can produce a standard that will not
interoperate with the NNTP standard.

Mark Crispin's suggestion of handling this matter by means of an
implementation note was raised during the discussion of the NNTP drafts,
and was roundly rejected.

---

From: Bruce Lilly <[email-masked]>
Date: Fri, 22 Jul 2005 08:10:55 -0400

> Everybody who knows the 2822 <orig-date> will immediately see

But it doesn't say *why* there is an added SP, what is affected by
the difference, or what should be modified to bring the specification
into alignment with the Internet Message Format.

And yet you seem to be happy with _subtle_ modifications to the same
field w.r.t. zone abbreviations and MUST means MAY but MAY means MUST
(moreover where the field has no protocol role).  Inconsistency detected.
 
> CRLF or the embedded [FWS] is an "imposed structure".

No.  The CRLF terminates the field (or line thereof) and is part of the
overall message structure (the terminating CRLF is not part of the field
body). [FWS] is necessary to allow for whitespace and folding as utext
contains no whitespace characters or CR or LF.
 
> Of course there is

We disagree on your narrow interpretation which does not recognize that
CFWS ("comments, line folding, and/or whitespace") includes FWS ("line
folding and/or whitespace").

> if the final compromise are ten "MUST NOT" in prose:

If we have one implementation note that clearly identifies the issue(s)
and indicates the necessary workaround (and ideally direction for
migration to resolve the issue in a manner conforming to the Message
Format), then we can simply cite by reference those fields defined in
2822, which are bound by the 2822 section 3.2.3 restriction.  Whereas
if we produce conflicting syntax for "from" "orig-date", "message-id",
"references", "subject", "supersedes", etc., then it becomes unclear
whether 2822 3.2.3 applies.

> that they'll simply shoot us for this "engineering".  <sigh>

Give it a rest; this isn't MARID or ASRG.
 
> The rationale for the <msg-id> modifications is quite complete.

No, there is no rationale for excluding NO-WS-CTLs.  We can guess that
the reason is "Frank doesn't like them", but that's not a rationale.
There is nothing that indicates what has to be changed in any news
software implementations or protocols to achieve alignment with 2822,
which itself has been restricted (for Usenet-related reasons) from what
has been permitted by 822, 733, and 724.

> I've not tested other servers, these two added the missing SP.

Then we need to reexamine whether the SP is really necessary at all,
and if so, adjust the rationale accordingly.  And if not, simply
drop the difference.
 
> can always generate it.

True but irrelevant.  A UA generating a message cannot readily determine
where that message will go, and in principle altering messages after
generation causes problems (e.g. invalidates security mechanisms).

> In addition it must accept all mails 
s/mails/messages/
> without the SP and some weird cases with WSP before the colon.

Yes, this is the main point; any UA that handles messages in a 2822-
conforming manner must handle fields whether or not the first character
of the field body happens to be SP.  Not solely UAs of course, but any
software handling messages in a 2822-conforming manner.  As Mark has
noted, this MUST/REQUIRED to "protect" such software from dreaded fields
with no SP after the colon is silly; it is also unwarranted unless there
is a real problem meeting the BCP 14 criteria for use of imperative
keywords.  "SHOULD"/"RECOMMENDED" would be another matter.

Your tests seem to indicate that at least some versions of one
implementation cope fine with no SP; it remains to be seen precisely
where, if anywhere, there is a real problem warranting a BCP 14
imperative keyword.  As we have agreed on the general principle of
using 2822 except where necessary to deviate, it's up to those who wish
to impose the additional SP requirement to specify precisely where
the serious issues warranting imperative keywords arise; then we can
craft a suitable implementation note that identifies the issues and where
future changes can be applied to address them.

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 23 Jul 2005 03:41:23 +0200

Bruce Lilly wrote:
 
>> Everybody who knows the 2822 <orig-date> will immediately
>> see the added SP.
 
> But it doesn't say *why* there is an added SP

Maybe propose some text for the general MUST about this issue.
I've just tested the very last chance:  ":" SP HT CRLF SP ...
It was rejected, "missing Message-ID". 

What we have describes the situation, only the [FWS] is wrong,
it should be *WSP for all instances of <name> ":" SP [FWS] ...

For the trailing [FWS] CRLF it's less clear, Charles admitted
that it's a bug in 2822 <unstructured>, therefore it's also a
bug in USEFOR:

~~~ old ~~~
    unstructured    =  1*( [FWS] utext ) [FWS]
~~~ new ~~~
    unstructured    =  *WSP utext *( [FWS] utext ) *WSP
~~~ end ~~~

The "lost" obs-FWS doesn't affect us, because it's not allowed,
or do I miss something here ?

 [orig-date]
> yet you seem to be happy with _subtle_ modifications to the
> same field w.r.t. zone abbreviations and MUST means MAY but
> MAY means MUST

Now "happy" isn't what I'd say, it was a compromise proposed
by Harald after long disussions, refined several times.

We had the restricted syntax allowing GMT / UT, now we have
only GMT without the syntax.  I proposed to add a deprecated
GMT example, just to be sure that all readers get the drift,
see <http://mid.gmane.org/[email-masked]>.

> Inconsistency detected.

Not really.

 [back to <unstructured>]
> We disagree on your narrow interpretation

Apparently.  Your broad interpretation won't work for similar
cases like <word> and <atom> as discussed in...

<http://mid.gmane.org/[email-masked]>

...I doubt that you'd really want all prose rules for <word>
automagically affect any <atom> outside of <word>.  

> No, there is no rationale for excluding NO-WS-CTLs.  We can
> guess that the reason is "Frank doesn't like them"

Good guess, you have to add some RfCs from 1036 up to NNTPbis,
common practice, and my tests with ESC.  In my last count we
are now at 3 : 3 for the <id-domain> vs. <id-right> versions.

You're a special case, you want the RFC 2822 <msg-id> "as is",
ignoring all boring facts like reality or the USEFOR charter.

>> The mandatory SP conforms to the message/rfc822 format, the
>> UA can always generate it.
 
> True but irrelevant.  A UA generating a message cannot
> readily determine where that message will go

Yes, that's why it always generates ": " with SP, that works.

> in principle altering messages after generation causes
> problems (e.g. invalidates security mechanisms).

ACK, therefore we stick to a mandatory SP instead of trusting
that injection agents could "fix" this if it's missing.  It's
bad enough for mail2news gateways, let's not make it worse.

> Your tests seem to indicate that at least some versions of
> one implementation cope fine with no SP

IBTD, it's never "fine" if messages are modified in transit.
You could say that an injection agent is entitled to do this,
like an MSA. 

But don't ask me how this is supposed to work with stuff like
MASS / DKIM / PGP / S/MIME - "fixed" articles can leave news,
e.g. this article posted on GMaNe should reach you as mail.

> up to those who wish to impose the additional SP requirement
> to specify precisely where the serious issues warranting
> imperative keywords arise

As long as injection agents feel obliged to "fix" my articles
the serious issues are clear.  I'm not sure why they do this,
but I trust that it's not for fun (or for your fetish idea ;-)

                          Bye, Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 25 Jul 2005 11:31:12 GMT

>that it's a bug in 2822 <unstructured>, therefore it's also a

Not so. The "Bug" is in 3.2.3 of RFC 2822, but USEFOR contains a different
text to cover that problem, and the USEFOR text has no bug.

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 26 Jul 2005 12:19:21 +0200

Charles Lindsey wrote:

>> For the trailing [FWS] CRLF it's less clear, Charles
>> admitted that it's a bug in 2822 <unstructured>, therefore
>> it's also a bug in USEFOR:

> Not so. The "Bug" is in 3.2.3 of RFC 2822, but USEFOR
> contains a different text to cover that problem, and the
> USEFOR text has no bug.

If the ABNF says [FWS] with some additional prose elsewhere
explaining that sometimes [FWS] actually stands for *WSP,
then that's FUBAR.  It has to be fixed because it's broken.
And it's no feature, therefore it's a bug.

Otherwise we could as well delete all ABNF with the magic SP,
because USEFOR contains text to cover the magic SP.  Or other
parts of the ABNF fully covered by the prose.  That makes no
sense, your argument doesn't convince me.

Bruce's objection that *WSP won't allow <obs-FWS> was better.

OTOH we don't allow the <obs-FWS>, and at least for the [FWS]
after the magic SP that's no error, it really MUST be *WSP.

For the trailing [FWS] before the CRLF we should say what we
mean, e.g. OFWS = *WSP / obs-FWS, where the <obs-FWS> is only
relevant for servers using the "MAY accept obs" clause in 2.1.

For the leading [FWS] after the magic SP we MUST say *WSP, it
affects critical header fields like Control, Supersedes, and
Distribution.   If a "MAY accept obs" server accepts <obs-FWS>
it won't work as expected.  That's no ABNF-aesthetical issue,
that's HAVOC, it's harmful.
                            Bye, Frank

---

From: Bruce Lilly <[email-masked]>
Date: Tue, 26 Jul 2005 09:03:53 -0400

> And it's no feature, therefore it's a bug.

You are in effect proposing writing a complete message syntax specification
down to low-level detail, one which is not based on 2822 and MIME (as our
work is supposed to be, and as it is claimed (at the moment, falsely) to be
in section 1.6), and which could require a separate parser from those used
for the Internet Message Format, invalidating the primary consideration of
RFCs 850 & 1036: "fit in with existing tools as well as possible". 

> Otherwise we could as well delete all ABNF with the magic SP,

And probably should do so, failing any justification for it, esp. w/ BCP 14
imperative keywords.

> OTOH we don't allow the <obs-FWS>,

That's incorrect; usefor states "agents MAY also accept the obsolete syntax
specified in Section 4 of [RFC2822]" whereas you are stating that "we don't
allow" that.

> that's HAVOC, it's harmful.

Not in the absence of a justification for the "MUST contain at least one non-
whitespace character" statement.

---

From: Bruce Lilly <[email-masked]>
Date: Tue, 26 Jul 2005 09:29:38 -0400

> It was rejected, "missing Message-ID". 

I don't follow your argument; why would a Date field be rejected
ostensibly for "missing Message-ID"?

> cases like <word> and <atom> as discussed in...

There's no similarity there; CFWS is an obviously-capitalized, non-word
abbreviation; "word" is not capitalized, is a word, and is not an
abbreviation.

> ignoring all boring facts like reality or the USEFOR charter.

No, *if* there is an implementation consideration, I have no objection
to a suitable implementation note pointing out the specific issue(s),
the workaround(s), and the long-term roadmap to full conformance with
the Internet Message Format.  What I object to is violating our stated
(sect. 1.6) and agreed-upon methodology (cite by reference, not rewrite)
and attempts to rewrite a normative reference which is outside our
jurisdiction.
 
> Yes, that's why it always generates ": " with SP, that works.

No, it doesn't necessarily do so. ":" is the standard message format
field body delimiter (not ": "), and a message using only ":" as the
delimiter is perfectly valid.  Messages may find their way to a gateway
(possibly w/o knowledge of the author or sender), and an artifical
requirement means that a gateway or associated injection agent would have
to:
o modify the message, possibly invalidating security measures
o because the length is increased, take into account any encoded-words
  and re-fold if length changes e.g. from 76 octets to 77 (and in turn
  that presents a problem w.r.t. the other artificial (failing some
  rationale) "MUST contain at least one non-whitespace character" rule)
  or decode and re-encode encoded-words to reduce length (not always
  possible, non-trivial, almost certain to invalidate security measures,
  and may cause minor problems for naive killfile processors).
 
> that injection agents could "fix" this if it's missing.

No, no, no.  You are simply trying to move the problem to gateways. RFC
1925:

   (6)  It is easier to move a problem around (for example, by moving
        the problem to a different part of the overall network
        architecture) than it is to solve it.

The problem (caused by the not-justified SP requirement) remains. A
"fix" (i.e. breaking security mechanisms and MIME compliance) at a
gateway is no better than one at an injection agent.

> It's 

You are making it worse for gateways (see above).
 
> but I trust that it's not for fun (or for your fetish idea ;-)

If you're not sure and cannot provide a precise specification, then it
can't be clear.  At the moment, "fetish" is the theory which best fits
the observable facts.

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 26 Jul 2005 17:59:09 +0200

Bruce Lilly wrote:

> You are in effect proposing writing a complete message syntax

No, I propose to replace all explicitly forbidden (in prose)
[FWS] in the explicitly given syntax by a correct *WSP.  The
syntax is already there, it's only wrong.

> down to low-level detail

That's no low-level detail.  Control, Supersedes, Message-ID,
etc. are vital.  If some servers accept an obs-FWS and others
ignore any folded info for the [FWS] after the magic SP, then
it could result in chaos.

Maybe we should limit this "MAY accept obs" in 2.1 somehow.

First idea "...where it doesn't violate an explicitly stated
MUST or MUST NOT in this memo".
                                Bye, Frank

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 26 Jul 2005 18:58:00 +0200

> I don't follow your argument; why would a Date field be
> rejected ostensibly for "missing Message-ID"?

Because I tested 'colon SP HT CRLF SP info' in all critical
header fields simultaneously, the Message-ID was the first.

> *if* there is an implementation consideration, I have no
> objection to a suitable implementation note pointing out
> the specific issue(s), the workaround(s), and the long-term
> roadmap to full conformance with the Internet Message Format.

The long-term roadmap is to fix the <msg-id> in a 2822bis for
full conformance with what USEFOR will say, because NO-WS-CTL
was plain wrong.

> You are simply trying to move the problem to gateways.

LOL, now that would be the very last I do, it's also the only
aspect of USEFOR where I can claim some rudimentary knowledge
above "read and understood s-o-1036".   ^^^^^^^^^^^

> A "fix" (i.e. breaking security mechanisms and MIME
> compliance) at a gateway is no better than one at an
> injection agent.

Gateway operators are used to all kinds of trouble, mail2news
is a trivial case, with fido2news it starts to get interesting.

Yes, of course attempts to "fix" something are often a really
bad idea.  There are conditions where it could break.  I've
posted one "impossible" case here, see...

<http://mid.gmane.org/[email-masked]>

...that should be a USEPRO ticket if Harald's procedure works,
the subject was "NEW USEPRO 7.9 gateway (was: LTRU questions)".
So far for your idea that I ignore gateway issues.

> You are making it worse for gateways (see above).

I'm just stating the obvious, it IS bad for mail2news gateways.

Where injection-agents silently "fix" a missing SP - compare
my test results incl. s/HT/SP/ - it hits also mail (behind a
news2mail gateway), any signature of header fields created by
the UA could then arrive as trash depending on the algorithm.

Therefore the UA has to generate the magic SP.

>> I'm not sure why they do this, but I trust that it's not for
>> fun (or for your fetish idea ;-)

> If you're not sure and cannot provide a precise
> specification, then it can't be clear.

Hey, I never operated any news server.  I had a Fido node for
some time, and used Crosspoint some time (that's an UA with a
builtin pseudo-gateway, rfc2Z, fido2Z, even Z2Z because it had
its own vision of the Z-format, and v.v.), I read some German
groups about gateways some time.  I have the grizzly book and
an original 1994 copy of s-o-1036, that's all.  I know nothing
about INN, and I couldn't post with telnet without looking in
the RfC for each and every command.  And my DOS UUCP was mail
only.  So if I don't know why this SP is necessary it proves
only that I don't know it, not that it's a "fetish".

> At the moment, "fetish" is the theory which best fits the
> observable facts.

Don't count my ignorance as supporting this theory.  Bye, Frank

---

From: Bruce Lilly <[email-masked]>
Date: Wed, 27 Jul 2005 11:31:44 -0400

> header fields simultaneously, the Message-ID was the first.

So in this case you found some software that doesn't accept that
specific instance of a Message-ID field.  Which does not necessarily
say anything about *all* fields (certainly not ones newly defined
in our documents) or about other implementations.  Nor does it imply
anything about best practice, which is what we're supposed to be
working on.
 
> full conformance with what USEFOR will say, because NO-WS-CTL

That's an unsubstantiated assertion.  RFC 822 permitted CTLS in a
domain literal, RFC 733 permitted them via qtext, and RFC 724
permitted them in quoted-string (in word in phrase in mach-host-phrase).
So there is a long-established history.  You are of course free as an
individual to argue for a further restriction in an RFC 2822 successor,
but it is not within this WG's scope to make such a change.  It is within
the scope of this WG's work to make specific recommendations for software
related to this WG's work (viz. Usenet software) to bring it into
conformance with the underlying normative Internet Message Format
specification, as well as to specify workarounds for "baby programmer
errors" in Usenet software in the interim.
  
> LOL, now that would be the very last I do

Any "restriction" which imposes constraints not in the underlying
normative Internet Message Format specification necessarily creates
problems for gateways (perhaps other software as well).  That is why
we should be working hard to minimize the differences, and where
some difference is necessary to clearly document the reasons and
provide a roadmap for Usenet software to migrate to full conformance.

> is a trivial case,

Not at all trivial, for reasons detailed in an earlier message.

> I'm just stating the obvious, it IS bad for mail2news gateways.

Reducing the magnitude of that problem means removing unnecessary
differences.
 
> Therefore the UA has to generate the magic SP.

There are (rough estimate) thousands of gateways, tens of thousands
of newsgroups, tens of thousands of news servers, tens of millions
of UAs, tens of billions of messages (including archives).  "Change every
gateway" is perhaps feasible (but you agree that it's not only a
gateway issue), "Change every newsgroup name" or "change every server"
is bad, but "change every UA" is unthinkable; the scope of the change
is three orders of magnitude larger than that required to change
servers.  "Change every message" (which is implicit in in "fixing"
messages) is simply not an option.  "Change every gateway such that it
changes every message" is the second-worst scenario (after change
every server such that it changes every message).  By comparison,
"change every server to conform to the IMF" is
a) orders of magnitude simpler
b) works well with existing messaging protocols (e.g IMAP)
c) works well with existing and potential security mechanisms for
   the IMF
d) is consistent with migration towards (rather than away from) IMF
   conformance/compatibility
e) can be done gradually

Instead of the currently unjustified "SP (space) is REQUIRED) [...]" and
"MUST contain at least one non-whitespace character", a more productive
approach would be
o To stick with the normative IMF, where SP (via [FWS] or [CFWS}) is
  optional and the only whitespace/field line restriction is against
  generation of whitespace-only continuation lines
o to document any specific interoperability issues with widely-deployed
  Usenet software related to those parts of the IMF specification
o to affirm that message modification other than trace fields is not
  conformant with the specification (IIRC there is already something
  related to that)
o to recommend (at most with BCP 14 keywords appropriate for recommendations)
  steps to promote interoperability with widely-deployed software which
  has interoperability issues and/or fails to conform w.r.t. message
  modification.

At most, that would affect only the non-interoperable software and software
currently making message modifications, and would do so in a manner
compatible with gradual improvement consistent with the goal of minimizing
differences from the IMF and compatible with security mechanisms.

---

From: Bruce Lilly <[email-masked]>
Date: Wed, 27 Jul 2005 11:37:14 -0400

On Tue July 26 2005 11:59, Frank Ellermann wrote:
> That's no low-level detail.

FWS/WSP are low-level field components; a detail.  Changing things
at that level also affects higher-level use and amounts to writing
a different message syntax.

> If some servers accept an obs-FWS and others
[...] 
> MUST or MUST NOT in this memo".

The goal should be convergence, not divergence.  See separate message
on how to move towards that goal rather than away from it.

---

From: Bruce Lilly <[email-masked]>
Date: Wed, 27 Jul 2005 11:44:53 -0400

> interoperability problems (and not just in "legacy injection agents").

An unsupported assertion.  Specific documentation, please.

> interoperate with the NNTP standard.

Specifics please, including rationale.  If NNTP is supposed to be 8-bit
clean, it's rather silly to require an SP in a particular place in a
header field of a message which is merely transferred -- it would be as
silly as a corresponding requirement in SMTP (and there is not and never
has been one).

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 28 Jul 2005 12:22:09 +0200

Bruce Lilly wrote:

> So in this case you found some software that doesn't accept
> that specific instance of a Message-ID field.

Yes, same order of header fields as in all my other tests.

> Which does not necessarily say anything about *all* fields
> (certainly not ones newly defined in our documents) or about
> other implementations.

Yes, just for fun I now also tested Date before Message-ID,
and 'colon SP HT CRLF SP info' _only_ in the Date:

Date: 	
 Thu, 28 Jul 2005 11:55:52 +0200
Message-ID: <[email-masked]>
From: Frank Ellermann <[email-masked]>
Organization: <URL:http://purl.net/xyzzy>
X-Mailer: Mozilla 3.0 (OS/2; U)
MIME-Version: 1.0
Newsgroups: junk
Subject: SP HT CRLF SP
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

for Bruce / USEFOR

This worked modulo some header field sorting:
<http://article.gmane.org/junk:338:raw>

For obscure reasons my browser crashes whenever I try to
save the raw file on disk - I cannot see the SP HT in the
normal "view source" mode.  <sigh>

dosx rxgeturl article.gmane.org/junk:338:raw
unix

Apparently it's there.  The magic SP survived this test.
Please do NOT expect me to test all permutations for all
header fields.   Start your own tests, everybody can use
news://news.spamcop.net/spamcop.test

> Nor does it imply anything about best practice, which
> is what we're supposed to be working on.

"Sometimes 'colon SP HT CRLF SP body' works" is not what I
what to read in a standard.

> That's an unsubstantiated assertion.

That's a substantiated fact as far as I'm concerned.  I tested
it.  It's in 1036, s-o-1036, 1738, NNTPbis, and common practice.

> "Change every gateway" is perhaps feasible

We're not inventing something new, and they already do what is
necessary.

> "change every UA" is unthinkable

Dito, all newsreaders know how to get it right.  Including all
existing combined mail + news UAs.
                                  Bye, Frank

---

From: Bruce Lilly <[email-masked]>
Date: Fri, 29 Jul 2005 10:12:07 -0400

> what to read in a standard.

What I'd like to read -- given the fact that numerous people have on
multiple occasions said that rewriting syntax is a problem and that
our stated methodology is to cite by reference -- is "use the Internet
Message Format as defined in RFC 2822, supplemented by extension
fields...".  *If* there is some specific justification, supplemented by
implementation notes documenting existing problems in widely-deployed
software with workarounds and a roadmap for conformance to 2822.
Definitely NOT rewritten syntax with various and sundry changes but
no rationale for the changes, especially since the stated methodology
says that won't be done.

> >> The long-term roadmap is to fix the <msg-id> in a 2822bis

a) it's not broken
b) not in our scope of work

> >> for 
> it.

No, you found one mailing-list-to-news-like site that rewrote an ID
(and which was known a priori to do so).

> It's in 1036,

RFC 1036 has (unsurprisingly) no mention of NO-WS-CTL.  While it mentions
"printing ASCII characters", it also strongly discourages '/', which is
not mentioned in the usefor draft.  We therefore have already deviated
from 1036 (which is what is expected from an update and revision), and we
should continue to do so in order to converge with the Internet Message
Format specification except where there is a serious issue (in which case
the issue, workarounds, and a roadmap should be specified).

> s-o-1036,

Not a standard of any type, not an RFC of any type, not even a draft.

> 1738,

Obsolete and irrelevant to our work.

> NNTPbis,

*Perhaps* suitable for documentation in an implementation note.

> and common practice.     

No, "common practice" can't declare something to be "plain wrong".
  
> We're not inventing something new, and they already do what is

A mandatory SP with unjustified RFC 2119 imperative keywords is new.
Changes to msg-id syntax (esp. declaring the RHS not to be a domain;
see RFC 1036) are new.  Etc.
 
> existing combined mail + news UAs.

Imposing a new requirement would require code review of all UAs.  Only
warranted if there's a very good reason.  So far none have been presented.

---

From: Henry Spencer <[email-masked]>
Date: Fri, 29 Jul 2005 11:25:50 -0400 (EDT)

On Fri, 29 Jul 2005, Bruce Lilly wrote:
> > We're not inventing something new, and [gateways] already do what is
> > necessary.
> 

The mandatory SP is not new, and the only reason RFC 1036 (and 850) didn't
use RFC 2119 keywords for it is that the keywords hadn't been invented
yet.  The form of description may be new, but the (mis)feature being
described is not. 

                                                          Henry Spencer
                                                       [email-masked]

---

From: Bruce Lilly <[email-masked]>
Date: Fri, 29 Jul 2005 11:38:30 -0400

> yet.  The form of description may be new, but the (mis)feature being

There's nothing in 1036 that specifies that "blank" means SP, nor that
says that it's mandatory.  Nearly two decades later, if there isn't a very
good reason to require a mandatory SP, we shouldn't do so.

---

From: Henry Spencer <[email-masked]>
Date: Fri, 29 Jul 2005 12:04:47 -0400 (EDT)

On Fri, 29 Jul 2005, Bruce Lilly wrote:
> > The mandatory SP is not new...
> 
> There's nothing in 1036 that specifies that "blank" means SP

SP is also known as "blank", and there is no indication that 1036 is using
the term in a more generic sense.  In particular, when 1036 wishes to say
that either SP or HT is acceptable, it explicitly says "blank or tab". 
The terminology may not quite meet modern standards of rigor, but its
meaning is plain.

> nor that says that it's mandatory.

"Each header line consist of a keyword, a colon, a blank, and some
additional information."  There is no provision for the blank to be
omitted, any more than for the keyword or the colon to be omitted.

> Nearly two decades later, if there isn't a very

All previous news specifications (RFC and non-RFC) have required it, and
there definitely exists news software which insists on it.  The reasons
just don't come much better than that. 

                                                          Henry Spencer
                                                       [email-masked]

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 29 Jul 2005 18:54:49 +0200

> a) it's not broken

Of course it is, you claimed that it "invented" <id-left> and
<id-right> especially because <local-part> and <domain> don't
work in news.  Obviously they failed to get it right, so now
we do it, where's the problem ?

There will be also some bugs and oddities in what we do, fixed
by a future 2822bis.  Get over the idea that 2822 is perfect
and the end of the road, and don't fear that USEFOR will be
perfect and the last word on what works in news.

We've already heard that for huge amounts of news all we do is
irrelevant because they hate MIME and love yenc and think that
NetNews is the perfect way to distribute files.  No wonder if
they are less interested in the finer points of References etc.

> b) not in our scope of work

Print the charter and pin it to your monitor, I'm tired of your
endless rants about the "normative Internet message format".

NetNews always was a proper subset, and it always will be, crap
like length 250 (instead of 320) for the <msg-id> and no ">"
and the magic SP won't make it in 2822bis, it's just too silly.

OTOH if John's plan for I18N local parts makes it nonsense like
quoted-pairs and quoted-strings will die, good riddance.  Our
job is to update 1036 for this millennium _compatible_ with
NNTP and common pratice in news.

Eliminating almost all obs-cruft of 2822 which either never
worked in news or is useless is a huge step forward.  It will
help any future 2822bis WG to delete it (or some of it, bare
CR and similar crap).

  [colon SP CRLF SP msg-id]
>> That's a substantiated fact as far as I'm concerned.  I
>> tested it.

> No, you found one mailing-list-to-news-like site that rewrote
> an ID (and which was known a priori to do so).

I tested this in three or four groups on two or three servers.
Only one of these groups was gmane.test ("moderated NG" working
as a mailing list), and I didn't test anything where I knew the
result before I saw the outcome.  I tested '":" SP CRLF SP' and
'":" HT CRLF SP'.  Roll your own tests if you don't like what I
found.

>> It's in 1036,
> RFC 1036 has (unsurprisingly) no mention of NO-WS-CTL.  While
> it mentions "printing ASCII characters", it also strongly
> discourages '/', which is not mentioned in the usefor draft.

Yes, that has to be listed in the "differences from 10336".
The first time I saw a slash some years ago in a Message-ID I
was quite upset (it was in an admin group).

> We therefore have already deviated from 1036 (which is what
> is expected from an update and revision)

Yeah, 1036 meet IPv6, IPv6 meet 1036.  So let's add the domain
literals, and be silent about BITNET and UUCP (where possible).
And add those crappy slashes while we're at it.

> a roadmap should be specified

The roadmap is clear:  msg-id = "<" id-local "@" id-domain ">"
It updates "<" unique "@" full_domain_name ">" and other old
or incompatible constructs like "<" id-left "@" id-right ">".

> Obsolete and irrelevant to our work.

Two errors in six words.  For the relevance see the charter,
and for the status see <http://purl.net/net/rfc/1738> = PS.

> *Perhaps* suitable for documentation in an implementation

That should be a _normative_ reference, not only informative.
You have a knack to find all bugs even if you don't want it,

> No, "common practice" can't declare something to be "plain

After you have already printed the charter, and later read it,
now comes the final step, guess what it meands.  It explains
why 1036 plus s-o-1036 are a de-facto standard (twice, you
cannot miss it).

"publish a standards-track successor to RFC 1036 that with
 particular attention to backward compatibility, formalizes
 best current practice and best proposed practice."

Anything that doesn't work with NNTP etc. would be FUBAR, not
limited to 2822.

> A mandatory SP with unjustified RFC 2119 imperative keywords

| Each header line consist of a keyword, a colon, a blank, and
| some additional information.

> Changes to msg-id syntax (esp. declaring the RHS not to be a
> domain; see RFC 1036) are new.

They are not "new", they are completely wrong.  Do not use the
erroneous <id-left>"@"<id-right>, use <id-local>"@"<id-domain>.

> Imposing a new requirement would require code review of all
> UAs.

Fortunately the magic SP is several decades old, it's not new,
it's only odd - I'd prefer WSP and go for FWS, but all my tests
with HT failed resulting in s/HT/SP/.  Don't think that I like
the magic SP.

> So far none have been presented.

IIRC I quoted the magic SP line in 1036 several times over the
years.  The given reason "This is a subset of the Internet
standard, simplified to allow simpler software to handle it"
makes sense, but I'm lost why it doesn't allow HT.

Starting with a magic WSP we could recommend to support a FWS.
Starting with a magic SP we can only declare this to be a
hopeless casee, and accept it as is.
                                     Bye, Frank