Subject: FWS problem

From: Frank Ellermann <[email-masked]>
Date: Fri, 03 Dec 2004 03:10:12 +0100

Hi, I still don't understand the overall syntax for all header
fields.  The drafts say something like:

         field = "name:" SP body CRLF
         body  = [FWS] content [FWS]

But it also says:

| The header contents of every header line (including the first
| and any that are subsequently folded) MUST contain at least
| one non-whitespace character.

The definition of "header content" is "field body" as in 2822.
Therefore "name:" SP CRLF SP content CRLF is illegal, and the
version   "name:" SP content CRLF SP CRLF is also illegal, in
both cases there would a line without non-whitespace content.

First problem, why this fuzz about a mandatory SP ?  Isn't it
enough to say that there must be "something" between the colon
and the content, and then define "something" to be FWS ?  This
would result in either

         field = "name:" FWS body CRLF
         body  = content *WSP
or
         field = "name:" body CRLF
         body  = FWS content *WSP

Some of the then legal forms are:

(ugly) name: TAB content CRLF
(ugly) name: CRLF SP content TAB CRLF
(????) name: SP CRLF TAB content CRLF
(????) name: SP content CRLF TAB CRLF
(good) name: SP SP content CRLF
(good) name: SP TAB content TAB CRLF

ugly = allowed in 2822, but not by the syntax in usefor-02.
???? = allowed in 2822 and the usefor-02 syntax, but not by
       its additional "MUST contain at least one non-WS" rule
good = allowed in 2822 and in usefor-02.

Still excluded (bad) are name:content CRLF and versions like
name: SP content CRLF SP CRLF or name: SP content CRLF TAB CRLF

This would reflect the CFWS rule in 2822:

| However, where CFWS occurs in this standard, it MUST NOT be
| inserted in such a way that any line of a folded header field
| is made up entirely of WSP characters and nothing else.

For 2822 it was hell with numerous obs-rules, but for the news
drafts it's possible to integrate this rule firmly in the ABNF
of all header fields.  Maybe "they" even copy this idea later
in 2822bis, it's obvious (?).
                              Bye, Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Sat, 4 Dec 2004 20:58:14 GMT

>both cases there would a line without non-whitespace content.

Yes, you are right, and the wording in USEFOR needs to be fixed.

Perhaps something like:

   This specification uses the terms "header", "header name", and
   "header content" which are synonymous with the [RFC2822] terms
   "header field", "field name", and "field body" respectively. Note
   that "header content" includes all whitespace (including the
   initial SP) after the ':'.

and then:

   o  All agents MUST generate headers so that at least one space
      immediately follows the ':' separating the header name and the rest
      of the header content (for compatibility with deployed software).
      News agents MAY accept headers which do not contain the required
      space.
   o  The header content of every header line (including the first and
      any that are subsequently folded) MUST contain at least one
      non-whitespace character.
         NOTE: This means that no header content defined by or referenced
	 by this document can consist of WSP only.  As a result, this
	 document updates the <unstructured> construct from Section 3.2.6
	 of [RFC2822] as follows:

Alternatively, we might define "header content" to exclude any leading
WSP.

Ken, could you please fix this in the next draft?

>First problem, why this fuzz about a mandatory SP ?

Because much deployed software breaks without it. ANd for the same reason
it is also required by the new NNTP draft.

>(????) name: SP content CRLF TAB CRLF

And all of those would cause problems (often severe) for existing
software.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 13 Dec 2004 12:29:57 +0100

Charles Lindsey wrote:
 
> This specification uses the terms "header", "header name",
> and "header content" which are synonymous with the [RFC2822]
> terms "header field", "field name", and "field body" 
> respectively.

For your MUST you either need "header content = field name
plus body", or simply use "header" (if that's name plus body):

> The header content of every header line (including the first
> and any that are subsequently folded) MUST contain at least
> one non-whitespace character.

Yes, that's the erroneous paragraph.  You could replace it by

| The header content of every header line (including any that
| are subsequently folded) MUST contain at least one
| non-whitespace character.

That's for the definition "header content = field name + body".
Or just say this without changing the existing definitions:

| Every header line (including any that are subsequently
| folded) MUST contain at least one non-whitespace character.

It's shorter, it must be correct... ;-)

>> why this fuzz about a mandatory SP ?
 
> Because much deployed software breaks without it. ANd for
> the same reason it is also required by the new NNTP draft.

Please confirm, you're saying that "name: SP CRLF SP body CRLF"
is okay, but "name: CRLF SP body CRLF" breaks some obscure NNTP
servers ?  I have great difficulties to believe this, because
for me it's "obvious" that trailing white space is always evil.

If my server refuses to accept extremely long References: I'm
sometimes forced to edit the file "outbox" manually.  And my
text editor strips all trailing spaces.  I know that I must
watch signatures, but I never considered any folded headers.

Is this NNTP draft something we can kill ?  The 2822 style of
folding is much better than a mandatory trailing white space.

Only the 2822 obs-style "name:body CRLF" is bad, but a simple
"name: CRLF SP body CRLF" should be okay.  And all variants
with TAB instead of SP of course.

Instead of copying excessively bad ideas like NO-WS-CTL in a
Message-ID from RfC 2822 it would be nice to have at least a
_similar_ concept of header and folding in news and mail.

Or do we want mail2news gateways to insert a mandatory trailing
space in the first line of any "name: CRLF SP body CRLF" ?

                        Bye, Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 14 Dec 2004 11:49:25 GMT

>plus body", or simply use "header" (if that's name plus body):

No, the whole point of "header content" is that it does not include the
header name (if you want to refer to the header inclusing its name, then
we already have the term "header" for that). The question at issue is how
much of the FWS (including the obligatory SP) is to be included in the
term.

>Yes, that's the erroneous paragraph.  You could replace it by

No, that is definitely the effect we want, given the intended meaning of
"header content".

>for me it's "obvious" that trailing white space is always evil.

Things that will certainly break existing software are:

name: body CRLF
name: CRLF

It may well be that the case you mention "name: SP CRLF SP body CRLF" will
not actually break anything, but OTOH some "helpful" intermediate site
that thinks trailing white space is evil might well change it to
"name: CRLF SP body CRL", at which point it _will_ break existing
software.

So that is why we forbid empty content on the first line. RFC 2822 also
forbids enpty content on folded lines for essentially the same reason,
namely that the line "SP CRLF" might well be changed to "CRLF" by that
"helpful" intermediate site, whereupon it becomes indistinguishable from
the divede between headers and body.

>folding is much better than a mandatory trailing white space.

No. But the only point about it is that it requires the SP after the
name:. It does not concern itself with folding.

>with TAB instead of SP of course.

I think some existing software also breaks with

name: TAB body CRLF

which is why we insist on SP.

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 14 Dec 2004 18:46:23 +0100

Charles Lindsey wrote:

> the whole point of "header content" is that it does not
> include the header name

Fine, that's what you have in the draft, we're now back to
square one.

>>> The header content of every header line (including the
>>> first and any that are subsequently folded) MUST contain
>>> at least one non-whitespace character.

>> Yes, that's the erroneous paragraph
[...]
> No, that is definitely the effect we want

Now that's also back to the start of this discussion.  In
other words "name: SP CRLF body CRLF" is invalid, because
the first line "name: SP CRLF" has no non-whitespace header
content, it has only a single trailing SP.

JFTR, Bruce said that this is okay.  Your draft says that
this is invalid.  If you are right the syntax in the draft
should be "name: SP *WSP header content CRLF" everywhere.

The syntax "name: SP [FWS] header content CRLF" is wrong,
because "name: SP CRLF SP header content CRLF" is illegal.

That's what I said in my first article about this problem:
replace [FWS] by *WSP before and after any header content,
you want FWS only *within* the header content.

> Things that will certainly break existing software are:

Okay, then "name: FWS body CRLF" is unfortunately dead.

> some "helpful" intermediate site that thinks trailing
> white space is evil might well change it to
> "name: CRLF SP body CRL", at which point it _will_ break
> existing software.

My text editor does this when I forget to SET TRAILING ON.
Is it the duty of injecting agents to fix any "name: CRLF
SP body CRLF", i.e. insert the missing SP after the colon ?

It's a major difference from RfC 2822, at least gateways
should know this problem.  But it's probably better to say
that it's a duty of all injecting agents.

> which is why we insist on SP.

In other words, the syntax of message/news is very different
from message/rfc822.  I have no problem with this consequence,
but please say so in the draft, and don't delete message/news.

                         Bye, Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 15 Dec 2004 11:36:50 GMT

>Fine, that's what you have in the draft, we're now back to

I am not sure how we ever got away from square one.

>>> Yes, that's the erroneous paragraph

You are confusing two issues. Whether that provision is correct, and the
manner in which we document it (and other SP-related matters).

So no, it is not erroneous. We have discussed it many times, including
once in the last 6 months, and we have always concluded it needs to be so.
It has indeed been so since Son-of-1036.

>content, it has only a single trailing SP.

Yes.

>should be "name: SP *WSP header content CRLF" everywhere.

Maybe for our own news headers, but in some cases the syntax for the
[*WSP header] bit comes straight from RFC 2822, and so it is not ours to
change.

>because "name: SP CRLF SP header content CRLF" is illegal.

I don't think this prohibition can be made by syntax. The corresponding
provision in RFC 2822 (the one against empty folded lines) was not made by
syntax, and we are following RFC 2822 wherever possible as a matter of
policy.

>SP body CRLF", i.e. insert the missing SP after the colon ?

It is not just text editors and injecting agents. Some half-baked
relaying/serving agent or some gateway might remove a trailing whitespace
somewhere en route, turning a possibly acceptable article into one that
would break something further on. We could not take that risk, and hence
the prohibition of :name: SP CRLF".

>but please say so in the draft, and don't delete message/news.

The draft makes it perfectly clear what the rule is, and that it is
different from RFC 2822.

What IS wrong with the present USEFOR draft is the definition of the term
"header content", and in particular how much of the whitespace it
includes, and that needs to be fixed.

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 17 Dec 2004 11:50:34 +0100

> I am not sure how we ever got away from square one.

I wanted the ABNF to reflect your "MUST contain at least one
non-whitespace", and Bruce claimed that this is not the case
for "name: SP CRLF SP body CRLF".  But now "we" (you and me ;-)
obviously agree that this is really illegal.
 
> we have always concluded it needs to be so.

Okay, maybe it's a "we" excl. Bruce, and I can live with it,
if "we" (= you ;-) replace all corresponding [FWS] by *WSP in
the ABNF.

> It has indeed been so since Son-of-1036.

You can always convince me with this line.  But s-o-1036 says:

| header          = start-line *continuation
| start-line      = header-name ":" space [ nonblank-text ] eol
[...]
| space           = 1*( <HT (ASCII 9)> / <blank (ASCII 32)> )

I don't see any mandatory SP in this syntax, I only see a WSP.

Are you absolutely sure that we want to go back to the "blank"
in 1036 ?  It's the old problem of "does RfC 2822 update 1036
in some way ?"  IMHO a "name: 1*WSP body CRLF" is better than
your "name: SP *WSP body CRLF".

> Maybe for our own news headers, but in some cases the syntax
> for the [*WSP header] bit comes straight from RFC 2822, and
> so it is not ours to change.

Let's fix it at least for our headers, replace [FWS] by *WSP
before and after any header content of "our" headers.

> I don't think this prohibition can be made by syntax. The
> corresponding provision in RFC 2822 (the one against empty
> folded lines) was not made by syntax, and we are following
> RFC 2822 wherever possible as a matter of policy.

We're following the _intent_ of RfC 2822 whereever possible,
but you're not forced to copy inaccurate syntax.  If there's a
way to get it right in the ABNF, just do it please.  And as a
matter of policy I'm not happy with making a TAB illegal in a
place where it's allowed by both s-o-1036 and RfC 2822.

> The draft makes it perfectly clear what the rule is, and that
> it is different from RFC 2822.

Yes, but if message/news is a proper subset of message/rfc822,
then why remove it from the IANA registry ?  It's a practical
problem at least for gateways and moderated newsgroups, or in
your terminology for all injecting agents:

For "name: HT CRLF HT body CRLF" the injecting agent is forced
to format it as "name: SP body CRLF" somehow, and that's not
always trivial, e.g. what about a field body starting with a
maximal length encoded word ?
                                Bye, Frank

---

From: Henry Spencer <[email-masked]>
Date: Fri, 17 Dec 2004 13:22:04 -0500 (EST)

> I don't see any mandatory SP in this syntax, I only see a WSP.

As in any standard, you have to read the text as well as the BNF; the text
is not just commentary.  Down in 4.2.3, son-of-1036 says: 

  In particular, RFC 1036 appeared to specify that the character
  immediately following the colon after a header name was required to be 
  a blank, and some news software insists on that, so this character MUST 
  be a blank. 

> then why remove it from the IANA registry ?

Because message/rfc822 can be used to carry news without harm to its
readability.  If you want reliable bit-for-bit-correct transmission,
that's what application/news-transmission is for.

                                                          Henry Spencer
                                                       [email-masked]

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 18 Dec 2004 03:02:23 +0100

Henry Spencer wrote:

> As in any standard, you have to read the text as well as the
> BNF; the text is not just commentary.

Not when I'm looking for a very fast shot. ;-)  Okay, I stand
corrected, s-o-1036 had SP in its text, RfC 1036 had it also,
and the new standard will keep it.

But the new standard _should_ use a syntax reflecting the text
where possible, name: SP *WSP ... instead of name: SP [FWS] ...
and ... *WSP CRLF instead of ... [FWS] CRLF

> message/rfc822 can be used to carry news without harm to its
> readability.

If you're really determined to kill message/news, my good old
"Mozilla 3" will survive it (it uses it in forwarded articles).

That's not under the top 10 of all "Mozilla 3" issues, I'm used
to fix its Content-Type: message/news manually if necessary.

                         Bye, Frank

---

From: Henry Spencer <[email-masked]>
Date: Fri, 17 Dec 2004 23:37:34 -0500 (EST)

On Sat, 18 Dec 2004, Frank Ellermann wrote:
> where possible...

Yup, seems like a good idea.

> If you're really determined to kill message/news...

I think it deserves killing.  There was some reason for it at the time,
but it hasn't worked out as well as hoped, and there has been some
favorable evolution in the definition of message/rfc822. 

                                                          Henry Spencer
                                                       [email-masked]

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 20 Dec 2004 14:53:46 GMT

>I don't see any mandatory SP in this syntax, I only see a WSP.

I think somebody reported that there was software out there that barfed on
"name: TAB body CRLF", which is why we currently insist on SP.

>before and after any header content of "our" headers.

No, we can't even do that. Consider the Approved header, for example:

   approved        =  "Approved:" SP mailbox-list CRLF

You cannot reformulate that in your
    Approved: 1*WSP body CRLF
style because <mailbox-list> is defined in RFC 2822, where it is already
defined to start with '[CFWS]'. We can't do anything about that without
rewriting RFC 2822 syntax, and our Chair has forbidden that (and I think
everybody here accepts that - even draft-13 took the syntax of
mailbox-list from RFC 2822 without question).

>but you're not forced to copy inaccurate syntax.

Yes we are, unless the RFC 2822 syntax is irretrievably broken, which it
isn't.

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 21 Dec 2004 02:25:12 +0100

Charles Lindsey wrote:
 
> I think somebody reported that there was software out there
> that barfed on "name: TAB body CRLF"

Okay, Henry already convinced me that s-o-1036 has a mandatory
SP in the text, not reflected in the syntax.

  [erroneous FWS] 

>|         approved = "Approved:" SP mailbox-list CRLF

> You cannot reformulate that in your Approved: 1*WSP body CRLF
> style because <mailbox-list> is defined in RFC 2822, where it
> is already defined to start with '[CFWS]'.

If you're free to replace [CFWS} by [FWS] because we don't want
comments, then you're also free to replace [CFWS] by *WSP where
both 2822 and we don't want a CRLF.  RfC 2822 didn't bother to
get the sytax right, they only mentioned it in the text:

CFWS minus C is FWS, and FWS minus CRLF is 1*WSP, or a case of
obs-FWS.  We don't allow obs-FWS, because that would be again a
line of folded header content without non-whitespace.

In other words, we want *WSP instead of [CFWS] before and after
the header content.  I don't see a problem with this approach.
It reflects 2822, minus obs-FWS, and minus the C in CFWS, these
restrictions are already a MUST in your draft.

> We can't do anything about that without rewriting RFC 2822
> syntax, and our Chair has forbidden that

IIRC Alexej only said that whatever we do, the result must be
a valid message/rfc822 as defined in 2822.  But not all valid
message/rfc822 are also valid news articles.  If we have some
restrictions, and that's the case, then it's perfectly okay to
have a proper syntax reflecting these restrictions.

For those who don't get the idea by looking at the ABNF there
is still the text with explicit MUSTs:  The header content of
all header lines MUST contain a non-witespace (=> obs-FWS RIP).

Back to your example:

| mailbox-list = (mailbox *("," mailbox))

BTW, please add the sytax again, as it was in old drafts, you
probably don't want the obs-mbox-list in RfC 2822.

| mailbox      = name-addr / addr-spec
| name-addr    = [display-name] angle-addr
| display-name = phrase
| phrase       = 1*word

Dito no obs-phrase here, please.

| word         = atom / quoted-string
| atom         = [CFWS] 1*atext [CFWS]

Okay, I see the general problem, to get it right you'd have to
move all [CFWS] from the bottom of the 2822 syntax to the top,
and then replace all [CFWS] before and after anything else by
*WSP, keeping only [CFWS] between something that's not empty.

That should be possible, but manually without a tool the result
would be probably worse than any dubious [CFWS] in 2822... :-(

> even draft-13 took the syntax of mailbox-list from RFC 2822
> without question

No obs-stuff in draft 7, where did you lose this improvement ?

Maybe it's impossible to fix the [CFWS] before and after the
mailbox-list, but there are still many explicit [FWS] in your
draft which can be fixed without touching 2822 terms. e.g. in
newsgroup-list, path-list, and poster-text.

                           Bye, Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 21 Dec 2004 14:08:18 GMT

>get the sytax right, they only mentioned it in the text:

If you start to tinker with the RFC 2822 syntax to that extent, then you
will find that there is a considerable amount of it which would have to be
changed, and I think that is way beyond our remit. Hopefully the RFC 2822
situation can be improved if/when RFC 2822 comes up for revision, but we
have to work with it as it is.

>have a proper syntax reflecting these restrictions.

Well if Alexey were to say we could make these changes, then that would be
a different matter. But as it stands, with these particular cases covered
by verbiage, both in RFC 2822 and in USEFOR, nothing is actually broken
(apart from the definition of "header content", which needs to be worked
on).

>probably don't want the obs-mbox-list in RfC 2822.

No, that is OK. USEFOR already says that we take only the syntax from
section 3 of RFC 2822, not that from section 4.

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 23 Dec 2004 14:32:06 +0100

Charles Lindsey wrote:

> USEFOR already says that we take only the syntax from

Okay, then the FWS problem is reduced to 2*7 occurences
in 7 productions of usefor-02, and about 2*7 [CFWS].  For
[FWS] I get:

1: unstructured    = *WSP 1*utext *([FWS] 1*utext) *WSP

Very dubious, it's essentially any ASCII minus NUL, CR, or LF,
but allowing CRLF.  And again all those funny NO-WS-CTL, sigh.

Is "Subject:" SP ESC CRLF TAB NAK CRLF SP SUB CRLF really what
we want ?  A SUB is an EOF on some systems, and my text editor
and some other tools still handle it as EOF unless I want them
to ignore it.

2: msg-id          = *WSP msg-id-core *WSP
3: newsgroup-list  = *WSP newsgroup-name
                     *( [FWS] "," [FWS] newsgroup-name ) *WSP
4: path-list       = *WSP
                     *( path-identity [FWS] path-delimiter [FWS] )
                     tail-entry *WSP
5: poster-text     = *WSP %d112.111.115.116.101.114 *WSP
6: control         = "Control:" SP control-message CRLF
   control-message = *WSP verb *( [FWS] argument ) *WSP
7: dist-list       = *WSP dist-name *( "," [FWS] distaname ) *WSP

For newsgroup-list you allow [FWS] before and after the comma,
for dist-list it's only after the comma, is this what you want ?

And now the seven [CFWS] in "our" headers, defining "our" by
"already not the same as in RfC 2822 for some other reasons"):

1: references      = "References:" SP msg-id-list CRLF
   msg-id-list     = *WSP msg-id-core *([CFWS] msg-id-core) *WSP

Is that really what we want, no separator or only a comment as
separator is okay ?  s-o-1036 has apparently 1*WSP as separator.

2: supersedes      = "Supersedes:" SP *WSP msg-id-core *WSP CRLF
3: injection-info  = "Injection-Info:" SP *WSP path-identity
                     *( [CFWS] ";" [CFWS] inj-info-para )
                     *WSP CRLF

Dubious.  I've added [CFWS] before and after the semicolon, your
syntax allowed "Injection-Info:" SP CFWS path-identity CFWS CRLF
among other things.

4: lines           = "Lines" ":" SP *WSP 1*DIGIT *WSP CRLF

Who needs a comment in a deprecated header, let alone folding ?

The remaining three [CFWS] cases:  xref, archive, and user-agent.
Especially the latter needs any "CWSP" it can get, but no FWS
before or after its header content.  TBD if you agree so far
with the 7+3 simple cases, and maybe the dubious Injection-Info.

                           Bye. Frank

P.S. and OT:  Sometimes something on your side adds ***SPAM-3***
              to the subject, and I forget to remove it.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 28 Dec 2004 14:13:33 GMT

>but allowing CRLF.  And again all those funny NO-WS-CTL, sigh.

Indeed, but politically impossible to change it unilaterally.

>   control-message = *WSP verb *( [FWS] argument ) *WSP

Move <control> down to your [CFWS] cases. I think that is a bug in USEFOR
which I have aksed Ken to fix.

>for dist-list it's only after the comma, is this what you want ?

No, I think you just spotted another bug in USEFOR (draft-13 allowed the
[FWS] on either side of the ",").

Yes, all that syntax works, but that is not to say that we should change
it.

>"already not the same as in RfC 2822 for some other reasons"):

But your next lot of syntax does not work, because you have forbidden, for
example:

Supersedes:  (a comment)
    <[email-masked]>
    (another comment)

So you would need some syntax of the form:

    WSPC = *WSP [ comment [CFWS] ]
    CWSP = [ [CFWS] comment ] *WSP

>separator is okay ?  s-o-1036 has apparently 1*WSP as separator.

No, you just spotted another bug in USEFOR.

>syntax allowed "Injection-Info:" SP CFWS path-identity CFWS CRLF

Yes, that is right. Optional CFWS is allowed on either side of any MIME
<parameter>, and that needs to be made clear in USEFOR.

>Who needs a comment in a deprecated header, let alone folding ?

It is a fundamental principle of RFC 2822 that comments and folding are
allowed in _all_ headers. We stuck out our necks rather in the case of
Newsgroups, Message-ID, etc where there are performance issues, but there
are no grounds to make an exception for Lines.

But I am still not convinced that your syntax has forbidden folded lines
containing only WSP. All you need is some situation where two CFWS can
appear adjacent, and you will get it. And there are plenty of places in
RFC 2822 where that can happen. Moreover, you have got now to deal with
all those things we take from RFC 2822, such as <mailbox-list> in the From
header. That has [CFWS] at either end, and you can't start rewriting all
the syntax inside RFC 2822. So you still need wording to forbid folded
lines with WSP only, and header content 1st lines with WSP only, so all
your modified syntax has still not solved the problem it set out to solve,
so what have you gained?

>              to the subject, and I forget to remove it.

Yes, that is added as a warning by my Spamassassin filter when the score
is 3 (anything with a score of 4 or more goes straight to devnull).

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 30 Dec 2004 04:19:00 +0100

Charles Lindsey wrote:

 [NO-WS-CTL in utext]
> Indeed, but politically impossible to change it unilaterally.

Sooner or later you get me to believe in this IETF-2822-cabal,
and then I'll start unilaterally a war against it.  You found
a good reason to change the 2822 "unstructured" into a USEFOR
"unstructured".  It's already different, so why stick to the
utter dubious NO-WS-CTL ?  You could still add a "MAY accept"
and "SHOULD NOT generate" in the text, the polite form of "be
liberal".  [[ I'm only ranting, please leave utext as it is ]]

 [general [FWS] => *WSP]
> all that syntax works, but that is not to say that we should
> change it.

Why not ?  It reflects the MUST about non-empty characters, if
you don't like it, then something with this MUST is not okay.

> you have forbidden, for example:
> Supersedes:  (a comment)
>     <[email-masked]>
>     (another comment)

Yes, "Supersedes:" is an odd case, it's no 2822 header, and you
have already restricted it to one msg-id, s-o-1036 had a list.
Here are some alternatives:

 [your version]
| supersedes =  "Supersedes:" SP [CFWS] msg-id-core [CFWS] CRLF

 [simple version based on msg-id]
| supersedes = "Supersedes:" SP msg-id CRLF

 [s-o-1036 version with a msg-id-list also used in references]
| supersedes = "Supersedes:" SP msg-id-list CRLF

But IIRC we have a rough consensus for one and only one msg-id
in supersedes, and that kills any msg-id-list here.  Why do you
want comments in this header ?  After all it's something a news
server has to understand, and you eliminated CFWS in almost all
other headers relevant for servers.  Is that something about
relaying agent vs. injecting agent, or what's the idea ?

> you would need some syntax of the form:

Yes, that's the idea, but it's a syntactical PITA.  Apparently
it's better to leave all [CFWS] before and after the header
content alone, both your text and 2822 have an explicit MUST
about non-empty header lines.  With one minor and very obscure
difference, 2822 allows "name:" SP CRLF WSP stuff CRLF, and
you don't allow it, no matter how stuff begins (incl. comment).

 [msg-id-list in references]
> you just spotted another bug in USEFOR.

Maybe this works:

| references  = "References:" SP [CFWS] msg-id-list [CFWS] CRLF
| msg-id-list = msg-id-core *( WSP [CFWS WSP] msg-id-core )

 [Lines = "Lines:" SP *WSP 1*DIGIT *WSP CRLF]
> It is a fundamental principle of RFC 2822 that comments and
> folding are allowed in _all_ headers.

They do have FWS in their syntax where they needed it, e.g. a
Date: header has no CFWS before or within the date, only at the
end.  That cries for a compromise:

| Lines = "Lines:" SP *WSP 1*DIGIT [WSP CFWS] CRLF

Actually it's not that important if Lines: isn't a part of the
overview.  But if it is, then I'd want a trivial way to spot
very long articles before I download this stuff.  You allow a
"Lines:" SP CRLF WSP "(many" CRLF WSP "lines)9999(!)" CRLF

It's an old header only used in news, it has no comments today
(taking s-o-1036 as state of the art, the news.txt.z fraction
 please reads 1036 ;-), you deprecate it, so why add comments
to its grave ?

> there are no grounds to make an exception for Lines.

Now admit it, the big IETF-2822-cabal is in fact you... <gd&r>

> I am still not convinced that your syntax has forbidden
> folded lines containing only WSP. All you need is some
> situation where two CFWS can appear adjacent

That's then a problem of 2822, any you've convinced me that we
keep [CFWS] as is where necessary (not in Lines and Supersedes
IMHO).  They tried to avoid this CFWS problem in 2822 somehow:

| CFWS = *([FWS] comment) (([FWS] comment) / FWS)

A "comment" is never empty, it starts with "(".  So there's a
"(" after each folding, i.e. *WSP CRLF 1*WSP, and whatever
happens within a comment, it ends with a non-empty ")".  Even
if the syntax allows CFWS CFWS somewhere it works for comments.

It fails if you replace CFWS CFWS by FWS FWS, and then fold
both resulting in an illegal *WSP CRLF 1*WSP *WSP CRLF 1*WSP.

The discussed simple cases in "our" headers had no CFWS CFWS
and no FWS FWS, or I missed it.  If I missed it it's within the
header content, not at the begin or the end.

> there are plenty of places in RFC 2822 where that can happen

None of our business, or are you planning to edit a 2822bis ?

> Moreover, you have got now to deal with all those things we
> take from RFC 2822, such as <mailbox-list> in the From

No, you already said that you won't touch any 2822 constructs,
only if it's necessary, but not for cases like mailbox-list.

> you still need wording to forbid folded lines with WSP only

Sure, I wanted our syntax to _reflect_ our MUST where possible,
not to _replace_ it.  And it's not worth the effort for [CFWS],
if that means that we need a CWSP and a WSPC.  But for [FWS]
it's possible.

> your modified syntax has still not solved the problem it set
> out to solve, so what have you gained?

A correct syntax for all [FWS] cases.  But if you say that we
should also try CWSP and WSPC looking for cases of CFWS CFWS,
then let's do it.
                     Bye, Frank

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 30 Dec 2004 04:49:05 +0100

>| msg-id-list = msg-id-core *( WSP [CFWS WSP] msg-id-core )

Nonsense, that should be:

| references  = "References:" SP msg-id-list CRLF
| msg-id-list = [CFWS WSP] msg-id-core *( WSP [CFWS WSP] msg-id-core ) [WSP CFWS]

-or- (if a comment is good enough as separator)

| msg-id-list = [CFWS] msg-id-core *(CFWS msg-id-core ) [CFWS]

The latter allows (x)<y@z>(a)<b@c>(u)<v@w>(e) where the former
insists on (x) <y@z> (a) <b@c> (u) <v@w> (e).  These comments
almost everywhere are a pain, just like NO-WS-CTL.  Bye, Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 30 Dec 2004 16:12:59 GMT

>and then I'll start unilaterally a war against it.

Yes, please do. When they start to develop RFC 2922bis, please join in and
get them to remove some of their stupidities. But not now, for USEFOR.

>Here are some alternatives:

No, "Supersedes:" is just the example I chose to give. In fact, the
problem arises eberywhere you propose to use *WSP, even in the References
header (even though comments are "SHOULD NOT generate yet" there).

>end.

Yes, that was an explcit exception put into RFC 2822 after discussion.

But everywhere else it was understood that comments would be allowed
wherever WSP was allowed. You can't have them in some headers but not in
others, because no-one can be expected to remember which cases are which.
There is no reason for us to break with that principle (indeed, taking it
on board was one of the first things this WG did). It was only because of
loud protests by the server implementors that we agreed to make exceptions
for the Newsgroups and Path headers, etc, but only on account of the
performance hits. It was _never_ the intention to exclude them for other
News headers, even for ones that we invented.

>if that means that we need a CWSP and a WSPC.  But for [FWS]

If there is something you want to forbid in a standard (such as empty
header lines, etc), then it is an exceedingly Bad Idea to have two
mechanisms for forbidding it, one for some cases and another for others.
It is a recipe for confusion and for the introduction of unintended
errors (of which your failure to allow header lines containing just a
comment was a typical example).

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 31 Dec 2004 19:00:07 +0100

Charles Lindsey wrote:

> When they start to develop RFC 2922bis, please join in and

It's not too bad, we'll fix their msg-id for the real world.

The remaining problems for 2822bis are only NO-WS-CTL and a
mandatory FWS after the colon.  I'm not too happy with our
SP plus "MUST be non-empty" for the body of the first line.

The old 2822 obs-stuff could be moved to an appendix together
with NO-WS-CTL.  The new obs would be a missing FWS after the
colon, ready, submit this 2822bis this year or wait for 2005 ?

> "Supersedes:" is just the example I chose to give.

Yes, my general idea to fix a leading or trailing [CFWS] did
not work as expected.  In the cases of Lines: and Supersedes:
it was intentionally ( [CFWS] => [FWS] ) => *WSP, because I
didn't want the comments there.

> the problem arises eberywhere you propose to use *WSP

Not where it was meant to replace [FWS] (all 7 cases of [FWS]
plus Lines: and Supersedes:).  It did not work for References:
(msg-id-list) and Injection-Info (but at least there was still
a real problem also with your syntax).

 [2822 FWS in Date:]
> that was an explcit exception put into RFC 2822 after
> discussion. But everywhere else it was understood that
> comments would be allowed wherever WSP was allowed. You can't
> have them in some headers but not in others, because no-one
> can be expected to remember which cases are which.

ACK, that's a point.  And References: _is_ a real 2822-header,
you only need a new syntax for it because you modified msg-id
and added msg-id-core.  And maybe because no separator at all
between msg-ids is dubious, s-o-1036 had 1*WSP.

> only because of loud protests by the server implementors that
> we agreed to make exceptions for the Newsgroups and Path
> headers, etc, but only on account of the performance hits.

Supersedes: is relevant for news servers, like Control:, where
you also have only three [FWS] or in the fixed variant two *WSP
and still one [FWS] in control-message.  No [CFWS] in Control:

For Lines: you could say that it's now deprecated and the old
syntax was Lines = "Lines:" SP *WSP 1*DIGIT *WSP CRLF

That's no lie, and if somebody tries new tricks with deprecated
headers, then old software might fail for Lines: (huge)9999(!)

> If there is something you want to forbid in a standard (such
> as empty header lines, etc), then it is an exceedingly Bad
> Idea to have two mechanisms for forbidding it

Then let's use your CWSP and WSPC trick.  It's a worse idea to
say something as a MUST in the text and then something else in
the syntax.  The [FWS] problem is already solved for 7+2 cases,
or 7+1 cases if you insist on Lines: (over)999(lines)

Only 5 (or 6) similar [CFWS] cases, all solved by your idea:

| WSPC = *WSP [ comment [CFWS] ]

Replace all (5 or 6) "leading" (near colon) [CFWS] by WSPC.

| CWSP = [ [CFWS] comment ] *WSP

Dito 5 or 6 "trailing" [CFWS] => CWSP (near CRLF), ready.
For our most critical example of the References: I get

| references  = "References:" SP msg-id-list CRLF
| msg-id-list = CWSP msg-id-core *( CFWS msg-id-core ) WSPC

That's a variant enforcing at least either a comment or a WSP
between msg-ids, it allows (x)<y@z>(a)<b@c>(u)<v@w>(e) and
<y@z> <b@c> <v@w>, but not adjacent <y@z><b@c><v@w>.   The RfC
2822 syntax is very similar:

| references  = "References:" 1*msg-id CRLF
| msg-id      = [CFWS] "<" id-left "@" id-right ">" [CFWS]

You don't use it, because msg-id one of these overloaded terms:

| msg-id      = [FWS] msg-id-core [FWS]

IMHO you should get rid of your special USEFOR msg-id, there's
exactly one line where you use it:

| message-id  = "Message-ID:" SP msg-id CRLF

Why not simply delete the special msg-id, and use msg-id-core ?

  message-id  = "Message-ID:" SP *WSP msg-id-core *WSP CRLF

                              Bye, Frank