Subject: Re: Control: cancel ABNF bug

From: Harald Tveit Alvestrand <[email-masked]>
Date: Mon, 02 Jan 2006 17:25:15 +0100

--On søndag, januar 01, 2006 01:44:59 +0100 Frank Ellermann 
<[email-masked]> wrote:

> For section 3.2.5 (changing only its <argument>):
>
>       control-command =  verb *( [FWS] argument )
>       verb            =  token
>       argument        =  1*VCHAR
>
> Or directly  argument = 1*(%x21-7E)  without the VCHAR.

Personal opinion: I like the last version - simple and self-contained.

Q: Is there no case where a control: message needs a space?

>
>                         Bye, Frank
>
> P.S.:  why on earth do they need more than 10 months to
> publish an erratum submitted by one of the 2045 authors ?

I sure don't know - when it was my problem, every time they asked, they 
said that the IETF had told them publishing RFCs was more important.....

---

From: Russ Allbery <[email-masked]>
Date: Mon, 02 Jan 2006 10:07:04 -0800

> Q: Is there no case where a control: message needs a space?

Like:

    Control: newgroup example.moderated moderated

?  Or am I misunderstanding?

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 02 Jan 2006 19:16:15 +0100

>>       control-command =  verb *( [FWS] argument )
[...]
>>              argument = 1*(%x21-7E)
{...]

> Is there no case where a control: message needs a space?

That would be ambiguous with the [FWS].  For new/rm/mvgroup
it's <newsgroup-name> (like FQDN plus underscore, no space).

Dito for checkgroups, adding "!" for excluded scopes, and a
serial number "#" 1*DIGIT

For cancel it's <msg-id> (no space even for 2822 NO-WS-CTL).

For ihave / sendme it's <relayer-name> (= <path-identity> ).
Whatever we do with barewords and #1047, we shouldn't allow
a space within a <path-identity> :-)

The obsolete 1036 commands version and sendsys have no
argument.  The "archaeological" whogets and senduuname are
unspecified, apparently no argument for senduuname and a
<newsgrup-name> for whogets.  No spaces within arguments.

> I sure don't know - when it was my problem, every time they
> asked, they said that the IETF had told them publishing RFCs
> was more important.....

For the obvious 2045 "bug" that's true.  OTOH it's so obvious
that any delay is more work than just publishing the erratum.

Generally I'm not so sure, e.g. I'm unable to reproduce most
DIGEST MD5 examples published in various RFCs starting with
2069 (no problem for 2617 and 2831), so I'd say "stay away
from this stuff and use CRAM-MD5 or OTP" to anybody who asks.
For 2069 it's most probably only a stupid error on my side.

For the strange claim that UTF-8 needs %x00-FF in a fresh RFC
I didn't try anymore, and just posted it on the rfc-interest
list, that's a place where GoogleBot might find it.

More examples where I didn't bother anymore skipped, 2368bis
is one of the potential victims - won't fly without erratum
for 3986 or reverse engineering of a proper <uric>.

Okay, 2368bis has also several other problems, not only this.
And at least John's 3696 erratum made it.

                         Bye, Frank

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 02 Jan 2006 19:38:39 +0100

Russ Allbery wrote:
 
> Or am I misunderstanding?

Yes, not FWS between arguments, WSP within an argument.  BTW,
is this [FWS] as you want it, or should it be 1*WSP ?  Would...

Control: newgroup 
 new.group.example
 moderated

...work anywhere ?  Everywhere ?  Bye, Frank

---

From: Russ Allbery <[email-masked]>
Date: Mon, 02 Jan 2006 10:41:54 -0800

> it's <newsgroup-name> (like FQDN plus underscore, no space).

Oh, I understand now.  Yes, no individual argument to the Control header
may ever contain a space since software does a simple split on spaces to
parse it.

---

From: Russ Allbery <[email-masked]>
Date: Mon, 02 Jan 2006 10:51:04 -0800

> ...work anywhere ?  Everywhere ?  Bye, Frank

I would be surprised if that would work anywhere.  It certainly wouldn't
work with any version of INN.  Sorry, I missed that -- I thought we'd
already discussed this before and agreed on WSP.

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 02 Jan 2006 20:44:57 +0100

Russ Allbery wrote:

> I missed that -- I thought we'd already discussed this before
> and agreed on WSP.

We discussed "[FWS] => *WSP almost everywhere", but IIRC Bruce
said it would remove 2822 obs-FWS, and then we visited any
rathole in sight, I forgot how that general debate ended...

Here it's actually a case of FWS => 1*WSP, not [FWS] => *WSP,
the [FWS] as separator is of course also wrong.  Fixed ABNF:

|       control-command =  verb *( 1*WSP argument )
|       verb            =  token
|       argument        =  1*(%x21-7E)

Maybe there are more obscure [FWS].  In USEPRO it should be
FWS => 1*WSP for all control header field arguments.

The control: checkgroups might have an additional problem:

      Checkgroup-command  = "checkgroups" Checkgroup-arguments
      Checkgroup-arguments= [ chkscope ] [ chksernr ]
      chkscope            = 1*( FWS ["!"] newsgroup-name )
      chksernr            = FWS "#" 1*DIGIT

The <chkscope> can't be optional, or can it ?  Bye, Frank

---

From: Russ Allbery <[email-masked]>
Date: Mon, 02 Jan 2006 11:55:52 -0800

> The <chkscope> can't be optional, or can it ?  Bye, Frank

I dunno, all that chkscope and chksernr stuff is new innovation that no
one's ever implemented, so I suppose it can be just about anything we
want.  :)  I think the assumption was that the leading # in chksernr would
disambiguate even if chkscope wasn't provided.

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 03 Jan 2006 06:16:37 +0100

Russ Allbery wrote:
 
>> The <chkscope> can't be optional, or can it ?

> I dunno, all that chkscope and chksernr stuff is new
> innovation that no one's ever implemented, so I suppose
> it can be just about anything we want.  :)  I think the
> assumption was that the leading # in chksern would

Okay, I didn't know that a <chkscope> is completely new,
I only knew that checkgroups de !de.alt would be new.

If no <chkscope> at all is state of the art, how do
servers know what the checkgroups is about ?  Simply
all mentioned TLHs in the body ?

                      Bye, Frank

---

From: Russ Allbery <[email-masked]>
Date: Mon, 02 Jan 2006 21:59:00 -0800

Frank Ellermann <[email-masked]> writes:

> If no <chkscope> at all is state of the art, how do servers know what
> the checkgroups is about ?  Simply all mentioned TLHs in the body ?

Yup.

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 03 Jan 2006 08:50:30 +0100

Russ Allbery wrote:

>> Simply all mentioned TLHs in the body ?
> Yup.

Okay, then the USEPRO checkgroups ABNF is fine, it could
even stick to FWS as separator if 1*WSP isn't good enough.

But if we intend to allow FWS _anywhere_ for control we
also have to keep it in the general USEFOR control ABNF.

In that case we could enforce 1*WSP in USEPRO for all old
control header fields.

Rationale:  1*WSP is a proper subset of FWS, but not the
other way around.  I'm not completely sure what's better:

1:  FWS in USEFOR, 1*WSP in USEPRO "only where necessary",
    in other words everywhere excl. the new checkgroups
2:  1*WSP in both USEFOR and USEPRO, if somebody manages
    to create a checkgroups with more than 998 char.s in
    the checkgroups header field this can't be serious.

                           Bye, Frank

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 03 Jan 2006 15:04:37 +0100

I've created ticket #1157 for this problem.

--On søndag, januar 01, 2006 01:44:59 +0100 Frank Ellermann 
<[email-masked]> wrote:

>
> Hi, there's apparently a subtle bug in the -06 ABNF.
> We want (USEPRO 6.3):
>
>       control-command     =/ Cancel-command
>       Cancel-command      = "cancel" Cancel-arguments
>       Cancel-arguments    = FWS msg-id [FWS]
>
> That's your  msg-id = "<" id-left "@" id-right ">"
> (same as my  msg-id = "<" id-unique "@" id-domain ">" ).
>
> But USEFOR and 2045 (with Ned's pending erratum) say:
>
>       argument        =  value
>       value := token / quoted-string
>       token := 1*<any (US-ASCII) CHAR except SPACE, CTLs,
>                   or tspecials>
>       tspecials :=  "(" / ")" / "<" / ">" / "@" /
>                     "," / ";" / ":" / "\" / <"> /
>                     "/" / "[" / "]" / "?" / "="
>                     ; Must be in quoted-string,
>                     ; to use within parameter values
>
> In other words Control: cancel <whatever@example> is not
> allowed, and we certainly don't want to quote a <msg-id>:
>
>                Control: cancel "<whatever@example>"
>
> Substituting <word> for <value> has the same problem.
> 1*utext also can't solve it, we don't want NO-WS-CTL.
> Taking 1*VCHAR should do the trick, for section 1.4:
>
>       VCHAR           =  <see RFC 4234 appendix B.1>
>

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 03 Jan 2006 17:46:47 +0100

> I've created ticket #1157 for this problem.

Thanks.  The quoted fixed ABNF did not yet addreess the
[FWS] issue as discussed with Russ, for that it should be:

        control-command =  verb *( 1*WSP argument )
        verb            =  token
        argument        =  1*(%x21-7E)

So that's s/ [FWS] / 1*WSP / plus s/ value / 1*(%x21-7E) /

                     Bye, Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 3 Jan 2006 15:37:31 GMT

>already discussed this before and agreed on WSP.

No, We agreed not to allow <comment>s. But the simple folding rule of RFC
2822 ought really to be accepted everywhere eventually, even if not on day
one. It is just ugly to have such special exceptions lying around for ever.

Now any hierarchy admin who issues the control message above with the
expectation that the group will be created correctly everywhare deserves
all that he gets, but I doubt anyone is likely to try.

Therefore, although to claim 100% compliance implementations will need to
accept it and process it correctly, I fully accept that fixing that
particular bit of INN is likely to be just about at the bottom of Russ's
list of priorities, and likely to remain there for quite a long time. So
be it, but people writing new code had better get it right.

You will rmemember that when we discussed this in connection with the Path
header, it was pointed out that the worst that could happen in
implementations that neglected to notice the folding was that a few
articles might be relayed to one's peers unnecessarily.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 3 Jan 2006 15:25:00 GMT

>      control-command =  verb *( [FWS] argument )

Aargghh! Well spotted. My fault I think, because I persuaded Ken to put
that in, just so that people could parse a <control-command> into its
components before handing over to the code for the specific command, if
they preferred to do it that way. OTOH, it _should_ also make some nention
that further syntax rules will be provided in USEPRO, or else that USEPOR
will provide the requisite verbs and arguments, in the same way as we did
for Injection-Info. Whatever way we write it here in USEFOR, I can make
USEPRO fit in (though I think my preference would be to leave the USEPRO
syntax as-is and simnple write in here that the commands to be added will
be of the form of "verb *( [FWS] argument )".

For <verb>, <token> is fine, or course, but we need some other definition
of <argument>.

>      VCHAR           =  <see RFC 4234 appendix B.1>

Yes, I would be happy with that.

---

From: Russ Allbery <[email-masked]>
Date: Tue, 03 Jan 2006 10:02:02 -0800

Charles Lindsey <[email-masked]> writes:

> No, We agreed not to allow <comment>s. But the simple folding rule of
> RFC 2822 ought really to be accepted everywhere eventually, even if not
> on day one. It is just ugly to have such special exceptions lying around
> for ever.

I think this objection is silly, and strenuously object to FHS in the
Control header.  It is gratuitous backwards incompatibility for no purpose
whatsoever.  Introducing backwards incompatibility that WILL NOT WORK in
practice just for the sake of textual consistency makes the protocol
*more*, not less, complex for implementors.  Now they can't trust the text
or have to worry about exception clauses and the uncertainty over whether
a historical note is actually likely to affect them or is really
historical.

Can we *please* try to spend less time unringing old bells and more time
documenting the protocol so that we can release a specification for what
Usenet actually is?

> You will rmemember that when we discussed this in connection with the
> Path header, it was pointed out that the worst that could happen in

I still don't like it in the Path header, but in the Path header nothing
really cares that much; it's a small optimization.  If you fold your
control header, you may as well just post the control message to
/dev/null.

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 4 Jan 2006 11:53:15 GMT

>Control header. ...

RFC 2822, and ALL its predecessors, established the simple convention that
you could fold wherever whitespace could occur. It is my belief that RFC
1036 bought into that same convention (hence why I have always argued that
Path was foldable under RFC 1036, as opposed to Newsgroups which clearly
was not).

Yes, implementations of 1036 have been lax on many points, and where a
great many implementations have implemented the same laxity (as with the
SP after ':') then we had to go along with it.

But that does not apply here. The sky will not fall in if we correct this
anomaly (or, rather, fail to introduce it).

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 04 Jan 2006 16:06:37 +0100

Charles Lindsey wrote:

> The sky will not fall in if we correct this anomaly (or,
> rather, fail to introduce it).
          ^^^^^^^^^^^^^^^^^^^^
S-o-1036 sections 6.6 and 4:

| Control-content  = verb *( space argument )
                             ^^^^^
| space            = 1*( <HT (ASCII 9)> / <blank (ASCII 32)> )

RFC 1036 has it at the begin of section 3.  It's not straight
forward to find and interpret the convoluted 1036 section 2:

| The USENET News standard is more restrictive than the
| Internet standard, placing additional requirements on each
| message and forbidding use of certain Internet features.
[...]
| Each header line consist of a keyword, a colon, a blank, and
| some additional information.  This is a subset of the Internet
| standard, simplified to allow simpler software to handle it.
[...]
| The Internet convention of continuation header lines
| (beginning with a blank or tab) is allowed.

If the whole world interpreted this as "no FWS in control" and
we say something else in USEFOR or USEPRO, then this won't fly.

Or as Henry put it (and maybe we should copy his statement):

  Several  of  the mechanisms described in this Draft may seem
  somewhat strange or even bizarre at first reading.  As  with
  Internet  mail, there is no reasonable possibility of updat-
  ing the entire installed base of news software promptly,  so
  interoperability  with  old  software  is  crucial  and will
  remain so.  Compatibility with existing practice and robust-
  ness  in  an  imperfect world necessarily take priority over
  elegance.

---

From: Russ Allbery <[email-masked]>
Date: Wed, 04 Jan 2006 10:10:34 -0800

Charles Lindsey <[email-masked]> writes:

> RFC 2822, and ALL its predecessors, established the simple convention
> that you could fold wherever whitespace could occur. It is my belief
> that RFC 1036 bought into that same convention (hence why I have always
> argued that Path was foldable under RFC 1036, as opposed to Newsgroups
> which clearly was not).

RFC 1036 is not the protocol that Usenet currently uses.  Usenet currently
uses a protocol that is a combination of RFC 1036, son-of-1036, and some
things that are in neither.  RFC 1036 was somewhat poorly written and it's
very difficult to determine in some cases exactly what it's specifying,
but it also doesn't really matter.  Backward compatibility at this point
means compatibility with the deployed software, not with what RFC 1036 and
tea leaves indicate the Usenet standard was supposed to be.

My goal here is to publish a *real* standard that people can actually
implement so that hopefully in the future we won't continue to have this
problem.

Anyway, I think Charles and I are probably not going to agree on this
ever, and I don't want to prolong the argument on the list.  Please don't
take future silence on my part on this point to be agreement.

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Thu, 05 Jan 2006 08:03:13 -0500

> take future silence on my part on this point to be agreement.

Ah, confirmation that the fantasy-land cooks have worn out Russ, too.

Strike up the band!  There is no reason that everyone need be troubled
by the reality of a sinking ship.

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 5 Jan 2006 17:00:56 GMT

>| space            = 1*( <HT (ASCII 9)> / <blank (ASCII 32)> )

If you follow that argument, then Son-of-1036 would not be allowing you to
fold anything anywhere (because it uses "space" in that way in the syntax
of all headers). But that was clearly not the intention, as should be
clear from the last paragraph of 4.2.1 and from 4.2.3.

>| (beginning with a blank or tab) is allowed.

RFC 1036 has clearly contradicted itself there, in which case you can
argue that the rule which resolves conflicts in favour of RFC 822 should
be applied.

>If the whole world interpreted this as "no FWS in control" ...

But it is clear that the world HAS interpreted this as allowing FWS in
some places (In Subjects for sure and in References for sure and with
varying degrees of assurance in other places). Would you be upset to see

   Reply-To: First <[email-masked]>,
        Second <[email-masked]>             ?

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 06 Jan 2006 01:29:51 +0100

Charles Lindsey wrote:

 [space in s-o-1036 is 1*WSP in 4234 or FWS in 2822]
> that was clearly not the intention, as should be clear
> from the last paragraph of 4.2.1 and from 4.2.3.

If that would be clear we wouldn't discuss it for ages:

| In MAIL, and arguably in RFC 1036 (although  the wording
| is vague), it is technically legitimate for the white
| space to be part of a continuation line rather than the
| start-line, but not all existing software will accept this.

That's about our "magic SP", and all my attempts to replace
our ":" SP [FWS] by a simple ":" FWS spectacularly failed...

| Posting agents SHOULD not break headers unnecessarily.
| Relayers SHOULD preserve existing header breaks, and SHOULD
| not introduce new breaks.  Breaking headers SHOULD be a last
| resort

...for some of our eight critical "no-CFWS" header fields it's
_never_ necessary.  That "SHOULD not" deteriorated into some
kind of "MUST NOT" for Control, Message-ID, and a few others,
Certainly not my fault, and I don't care _how_ we say it, but
saying only "FWS" where a folding will never work as expected
can't be a good idea.

BTW, what you said elsewhere about WSP in a "Newsgroups" also
affects WSP in "Distribution", s-o-1036 uses the same wording.
IMHO we should keep this stuff consistent, dishonest or not.

> Would you be upset to see
>

Not at all, but Reply-To is none of or eight critical "no CFWS"
cases:  No news server or gateway is supposed to do anything
with it (ignoring the general issue of our "magic SP").

                              Bye, Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 6 Jan 2006 17:10:35 GMT

>Charles Lindsey wrote:

[from Son-of-1036]
>| not introduce new breaks.  Breaking headers SHOULD be a last

Yes, I agree with that, and SHOULD[2] and SHOULD[3] are in fact MUSTs in
USEPRO. And if you want wording that discourages breaking when it serves
no useful purpose, then by all means let us have such wording.

But nevertheless, all software SHOULD/MUST be written to accept such
breaks as occur (whether useful or not). In the case of Control messages,
it is hard to imagine a case where breaking would be useful, and the sky
is unlikely to fall in if Russ leaves implementing that feature until he
has time to do it (which may be a long time off, but not "never"). And the
worst that can happen if some control messages _does_ get folded is that
some sites may not create/remove the group, and if their users care it
will soon get reported and fixed.

>_never_ necessary.

And for some it is, which is why we reduced CFWS to FWS (but no lower),
just to keep to a simple policy throughout.

>IMHO we should keep this stuff consistent, dishonest or not.

Yes. A remark in the bit where it says SHOULD NOT use FWS in Newsgroups to
the effect "this also applies to Distribution" would be in order.

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 07 Jan 2006 11:04:03 +0100

Charles Lindsey wrote:

> In the case of Control messages, it is hard to imagine a case
> where breaking would be useful, and the sky is unlikely to
> fall in if Russ leaves implementing that feature until he
> has time to do it (which may be a long time off, but not
> "never").  And the worst that can happen if some control
> messages _does_ get folded is that some sites may not
> create/remove the group, and if their users care it will soon
> get reported and fixed.

IMHO that misses the point.  As an example, in a popular TLH
the checkgroups guy proposed to change the key from pgp 2.6.x
to a modern gpg and more than 1024 bits.  He was newly elected,
knew his trade, and intended to do it a.s.a.p.   That got him
almost "killed", some Usenet legends in this TLH (who hadn't
posted for ages) were seriously pissed, "did you discuss this
in the relevant international group and above all with Russ ?",
"most news servers will _never_ upgrade, they only run because
 nobody wants the trouble to shut them down at this time" were
the most striking rants.

Needless to say that the key wasn't changed.  So even if Russ
or somebody else implements USEFOR + USEPRO on the same day
when the IESG posts its approval major parts of Usenet won't
use this "soon".  That might be "never" for many news servers.

OTOH there will be folks trying to create simple gateways or
new UAs based on the new standard, after all that's the purpose
of this WG, the old 1036 doesn't reflect reality anymore, and
parts of (2)822 are misleading wrt NetNews, to put it mildly.

The standard is for those poor folks, and if they try to fold
a control for whatever reasons they screwed up.  It would hurt
their product, their users, and they'd never trust a single
word of the standard if they found any lie "FWS" the hard way.

So we have to get the working idea to them.  One way to do this
is to say FWS in the ABNF, then add some "MUSTard" essentially
explaining why FWS actually stands for 1*WSP.  Another way is
to simply say 1*WSP and be done with it.

The first way would never work for me, I won't read the prose
if the ABNF is apparently clear.  We could decree that that's
my fault, and besides it's unlikely that I start to implement
a gateway / UA / server.  But apparently it's not only my
problem, Russ said it's precisely what INN does at the moment.

Wishful thinking won't help, timetravel is unavailable, we're
forced to describe common practice.  Unless there are serious
and important reasons to change something just leave it as it
is, if it ain't broken don't fix it.

If a news2mail gateway manages to export control messages, a
stupid idea in most contexts, and something behind it folds
them, no harm done.  But the important point is "won't work
if folded".

We already discussed this in several hundreds of articles,
doing it again won't help.  In your USEPRO diffs I found:

! Because of its importance to all serving agents, the whitespace
! and folding in Newsgroups header fields newly permitted by
! [USEFOR] SHOULD NOT be generated (though it MUST be accepted);
! this restriction may well be removed in a future version of this
! standard.
+ [That last bit needs discussion. It should probably be moved to
+ USEFOR if it is to be retained.]

Yes, let's move it.  Maybe with a note for "Distribution" that
it's in theory the same issue there.  In USEPRO we could still
explain why a decent UA or Injection-Agent might wish to fix
that for human users, it's too easy to get this wrong.

Or maybe "reject".  The one thing it shouldn't do is "accept"
and then silently ignore anything after "," 1*WSP white space.

> Yes. A remark in the bit where it says SHOULD NOT use FWS in
> Newsgroups to the effect "this also applies to Distribution"
> would be in order.

ACK, let's get a ticket for that, subject adjusted.  Bye, Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 9 Jan 2006 11:08:12 GMT

>parts of (2)822 are misleading wrt NetNews, to put it mildly.

Yes, but control messages are not issued by gateways or UAs (except maybe
for cancels). They are issued by people who know (or ought to know) what
they are doing.

>+ USEFOR if it is to be retained.]

Yes, that "SHOULD NOT" (and the similar "SHOULD NOT" for comments in
References headers) was the wording that we agreed for those cases. If you
want a similar "SHOULD NOT" for control messages, then that would be fine
(and consistent). I just don't want to see a large collection of similar
cases which are all treated slightly differently.

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 09 Jan 2006 16:38:28 +0100

Charles Lindsey wrote:

> If you want a similar "SHOULD NOT" for control messages,
> then that would be fine (and consistent).

No, and I want no crappy [FWS] CRLF anywhere.  It's forbidden
by a MUST NOT in 2822 3.2.3 for CFWS (minus comments), and it
is also forbidden by a MUST in draft -06 2.2 bullet 2.

The very first ABNF specified, same section 2.2 bullet 2 is...

| unstructured    =  1*( [FWS] utext ) [FWS]
                                       ^^^^^
...declaring "red is green":  This [FWS] can never be folded
without violating the preceding MUST, and the similar idea in
RFC 2822 3.2.3 for CFWS (excl. comments).  It should be *WSP.

This infuriates me, especially when folks think that I support
this stuff, which isn't the case, and come to the conclusion
that this must be some kind of sly manipulative dishonesty, or
a similar stunt.  Better no ABNF than intentionally wrong ABNF.

                            Bye, Frank

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Mon, 09 Jan 2006 17:30:29 +0100

Frank,

could you clarify which section of the spec you are talking about, and what 
the issue is?

At the moment, I can't tell whether this is about Control:, Newsgroups:, 
Distribution:, general philosophy of SP [FWS] or all 4.

---

From: Ken Murchison <[email-masked]>
Date: Mon, 09 Jan 2006 16:17:49 -0500

> a similar stunt.  Better no ABNF than intentionally wrong ABNF.

Frank, are you saying that something in -06 is incorrect/inconsistent? 
If so, could you please clarify?

-- 
Kenneth Murchison
Project Cyrus Developer/Maintainer
Carnegie Mellon University

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 10 Jan 2006 14:05:13 GMT

>is also forbidden by a MUST in draft -06 2.2 bullet 2.

Yes, but that is nothing to do with control messages - it applies to all
headers. Do you accept that a "SHOULD NOT" wording (as in Newsgroups and
References) would be an acceptable resolution of the control message
situation (and Distribuition too, while we are at it)?

>| unstructured    =  1*( [FWS] utext ) [FWS]

That [FWS] at the end is already in RFC 2822. But even there it is
meaningless as regards folding because there is verbiage telling you not
to fold there (and we have similar, but slightly stricter, verbiage). We
have enough differences from RFC 2822 that matter, without introducing
further differences that don't.

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 11 Jan 2006 04:05:14 +0100

Harald Tveit Alvestrand wrote:

> could you clarify which section of the spec you are talking
> about, and what the issue is?  At the moment, I can't tell
> whether this is about Control:, Newsgroups:, Distribution:,
> general philosophy of SP [FWS] or all 4.

The latter plus a sudden urge to start some kind of ranting.
I've submitted an erratum for 2822 <unstructured>, see also

 <http://permalink.gmane.org/gmane.ietf.rfc.interest/110>

Somebody CCed the author privately, and I've added a link
to the relevant discussion on the rfc822 list last year.

For obscure reasons I don't find the article with a question
which header fields in USEFOR are affected by the [FWS] CRLF
anomaly anymore.  Maybe it was a PM and I screwed up deleting
it, anyway, here's a list.  This might be incomplete, always
the last [FWS]:

 1: <unstructured>   (also submitted as 2822 erratum)
 2: <message-id>
 3: <newsgroup-list>
 4: <path-list>      (or its #1047 successor)
 5: <poster-text>
 6: <control>
 7: <supersedes>
 8: <dist-list>
 9: <xref>
10: <lines>

Unrelated, I just stumbled over it, Keith has an explicit
[CFWS] before <parameter> and before CRLF in RFC 3834.  We
don't have those two [CFWS] in the case of <archive>.  One
of these two solutions must be incorrect, 3834 or our idea.

Dito <injection-info>, two missing [CFWS] _iff_ 3834 has it
right for its <parameter>.  In our <user-agent> it's only a
<token>, but otherwise that follows Keith's pattern with an
explicit [CFWS] CRLF at the end and before each <token>.

                            Bye, Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 12 Jan 2006 16:56:32 GMT

>I've submitted an erratum for 2822 <unstructured>, see also

I agree to the extent that [FWS] in places where folding cannot possibly
occur in an RFC 2822 problem, and hence RFC 2822 is the place to fix it.
Whether it is so broken that it needs an immediate erratum rather than
waiting for 2822-bis I am not so sure.

>explicit [CFWS] CRLF at the end and before each <token>.

I think Keith is wrong (and in any case he has failed to mention the
possibility of [CFWS] inside the <parameter>). But it is an arguable point,
depending on how you interpret bits of RFC 822 semantics in the midst of
what is otherwise based on RFC 2822.

We have indicated, in a NOTE, how we have interpreted it, so our intention
should be clear. But we could equally well follow Keith's interpretation
and amend the NOTE accordingly. In that case, the proper syntax would be

   archive         =  "Archive:" SP [CFWS] ("no" / "yes")
                      *( [CFWS] ";" [CFWS] archive-param ) [CFWS] CRLF

   injection-info  =  "Injection-Info:" SP [CFWS] path-identity
                      [CFWS] *(';' [CFWS] parameter) [CFWS] CRLF

and the NOTE would have to read

      NOTE: The syntax of <parameter> ([RFC2045] as amended by
      [RFC2231]), taken in conjunction with the folding rules of
      [RFC0822], effectively allows [CFWS] to occur on either side of
      the "=" inside a <parameter>.

On balance, I think we should do that (it reduces the number of places
where you might possibly be unsure whether [CFWS] was allowed).

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 12 Jan 2006 19:48:53 +0100

Charles Lindsey wrote:

> I agree to the extent that [FWS] in places where folding
> cannot possibly occur in an RFC 2822 problem, and hence RFC
> 2822 is the place to fix it.  Whether it is so broken that
> it needs an immediate erratum rather than waiting for
> 2822-bis I am not so sure.

Many errata are only typos like Ned's pending RfC 2045 "/".
What they accepted for BCP 78 was a missing "s" in "RFCs",
ducking a real question about incomprehensible legalese ;-)

Nothing's "immediate" about this procedure, don't hold your
breath.  The 2069 issue (or PEBKAC) is pending for almost a
year now.

 [3834 paramter + USEFOR token vs. USEFOR archive etc.]
> I think Keith is wrong (and in any case he has failed to
> mention the possibility of [CFWS] inside the <parameter>).

He imports the complete 2045 / 2231 <parameter> construct in
RfC 3834, leaving the "gibbous" internal details in RfC 2231,
tricky.  Yes, it's possible, ignoring a SHOULD in 2822 about
comments and syntactical units for a moment.

> But it is an arguable point, depending on how you interpret
> bits of RFC 822 semantics in the midst of what is otherwise
> based on RFC 2822.

2822 wants to update 822, and an old 822 parser should still
work, the main difference is no NUL outside of quoted pair if
you want to accept all old obs-ceneties.

An interpretation of the [CFWS] mess made it into the security
considerations.

> We have indicated, in a NOTE, how we have interpreted it, so
> our intention should be clear.

For <injection-info>, not for <archive>.  It should be okay for
implementors to take the ABNF as is, put it into some magic
scripts, and get a proper parser without reading a note about
another header field.  At the moment it's illegal to say:

        Archive: yes (CC-BY-SA)

> we could equally well follow Keith's interpretation and amend
> the NOTE accordingly. In that case, the proper syntax would be

>            *( [CFWS] ";" [CFWS] archive-param ) [CFWS] CRLF

Yes, with that version "Archive: no (but CC-BY-SA okay)" works,
on a syntactical level.

>                    [CFWS] *(';' [CFWS] parameter) [CFWS] CRLF

I think you want the last [CFWS] next to the <parameter> as in

  injection-info  =  "Injection-Info:" SP [CFWS] path-identity
                     [CFWS] *(';' [CFWS] parameter [CFWS]) CRLF

Otherwise in ... foo=bar ; aaa=bbb white space after foo=bar
before the semicolon would be invalid.  You could also say...

  injection-info  =  "Injection-Info:" SP [CFWS] path-identity
                     *([CFWS] ';' [CFWS] parameter) [CFWS] CRLF

...to get the normal [CFWS] CRLF construct at the end.  I like
a consistent look and feel for ABNF.

                           Bye, Frank

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 12 Jan 2006 21:26:46 +0100

{...]
>                    *([CFWS] ';' [CFWS] parameter) [CFWS] CRLF

Plus Charles' proposed modification of the corresponding Note.

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 13 Jan 2006 14:05:29 GMT

>Plus Charles' proposed modification of the corresponding Note.

I concur. And in both cases it is now entirely consistent with Keith in
RFC 3834.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Fri, 27 Jan 2006 09:12:00 -0800

Ticket created.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 30 Jan 2006 14:22:19 GMT

>> JFTR:
>>
>>>> archive =3D  "Archive:" SP [CFWS] ("no" / "yes")
>>> injection-info  =3D  "Injection-Info:" SP [CFWS] path-identity

Fine by me.