Subject: Re: Differences between RFC 2822 and Usefor

From: Frank Ellermann <[email-masked]>
Date: Wed, 01 Jun 2005 02:00:25 +0200

Charles Lindsey wrote:
 
>> As soon as Harald had enough fun with the References we're
>> IMHO ready to tackle the last details.

> No Way! There is a lot to be done yet

Not in USEFOR and from my POV.  The ticket system lists four
open tickets, two about the References.

> The WG has been in continuous existence since it was formed.
> It does not have "previous incarnations".

Here's what I consider as the end of a "previous incarnation":
<http://article.gmane.org/gmane.ietf.usenet.format/21194>

> when mail comes to its senses and decides not to have a line
> length limit in bodies anymore, "they" won't be able to say
> "you can't change that because it will break news".

Yes, the "they" listing Alexey, you, Claus, Dave, Harald, John,
John, Kai, Keith, Ned, Paul, Randy, and many many others. <eg>

Even yEnc 2002 didn't believe in lines longer than 998 char.s:
| <linevalue> :=
|        unsigned integer. (Range: 63-998)
|        // Lines have this size

> And if you look closely, s-o-1036 also had unlimited body
> lines (as a SHOULD).

Okay, the bible trumps a MUST in a proposed standard, and news
might support unencoded binaries <shudder />.

>> At the moment this isn't specified in USEFOR.
[...]
> I would like to see advice in USEFOR to treat the 998 limit
> in bodies with liberality.

We could add it in 2.3

| User agents MUST meet the definition of MIME-conformance in
| [RFC2049] and MUST also support [RFC2231].
................^^^^
Omigod, with a MUST it _is_ "normative" and a security risk :-(

It's the beauty of MIME that user agents not supporting it have
no serious problems.  That's USEAGE, not USEFOR, and a SHOULD,
not a MUST.  

Anyway, we could add unlimited body lines at the end of 2.3:

: Depending on the Content-Type and Content-Transfer-Encoding
: lines in the body may be limited to 998 or less characters,
: compare [RFC2045] 7bit and 8bit data.  Binary parts without
: "lines" MAY in fact allow arbitrary line lengths.  The body
: of a Netnews article MUST NOT be empty, note that one empty
: line is allowed.

> the world is moving, and we should not hold it back.

Depends on the direction.  Periods in the <display-name> could
be still used for a negative score, so it's not that bad.

>> My Subject;<sp><crlf><sp>tests failed miserably with two
>> news servers...

> If the injecting agent of your server rejected it, then that
> is quite proper.
[...]
>> "but SHOULD accept" weakens this simple and clear position.
 
> No, just liberality of acceptance, which is a well-established
> internet principle.

Now what, proper or liberal ?  In standards I'd prefer "clear".

> Read RFC 2822 again:
>|    The more conservative 78 character recommendation
[...]
I'm very happy with the 78 in 2822, let's just don't touch it
in USEFOR.

> It arises because the default width of terminals/windows/etc
> for editors, command interpreters, etc is usually 80
> characters (you can thank IBM for that), but you can't set
> the limit at 80 because too many such terminals/windows/etc
> have a habit of wrapping the line if column 80 is used

Yes, and if they are ported to a CRLF system they do strange
things in column 79, because 79 + CRLF is greater than 80.  No
argument with this, I rarely use columns 73..80, because that
is where I'm planning to punch the serial card number later ;-)

> we now have an IESG AD as our co-chair

IETF Chair emeritus.  But a MUST mentioning a proposed standard
is of course "normative", so we need "Security Considerations"
with a Caveat about the MIME boundary parameter.  What a mess.

>> (2) in Harald's list is perfectly compatible with 2822.
 
> It is compatible with RFC 2822, but imposes a stricter
> REQUIREMENT than RFC 2822.

A mere definition of "followup" is no REQUIREMENT.  We had the
discussion about "green => colour", after that you can't say
"green MUST be a colour".  And you also can't say that this is
a difference from "green => color" because of MUST and the "u".

> The differences from 1036 are currently detailed in USEPRO,
> though they should probably be moved to USEFOR at some
> convenient moment.

It would be convenient before this WG is closed, but if it's
in USEPRO it's fine.  Is there anything else for a usefor-05 ?

                      Bye, Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 1 Jun 2005 11:00:19 GMT

In <[email-masked]> Frank Ellermann <[email-masked]> writes:

>Hiding this in "miscellaneous obsolete tokens" in the chapter
>"obsolete syntax" is more odd than dangerous.  No showstopper
>for USEFOR, we could just redefine the RfC 2822 display-name:

>{2822]
>| display-name    =       phrase
>| phrase          =       1*word / obs-phrase
>| obs-phrase      =       word *(word / "." / CFWS)

>[USEFOR]
>  display-name    =       word *(word / "." / CFWS)

>This syntax would still disallow obscene keywords.

If we are going to put it into USEFOR (and I think we should), then it may
as well go into Keywords too. It is clearly allowed in Keywords in RFC
2822, and I am not aware of any news software that will break if we allow
it too (come to think of it, I am not aware of any regular news software
that looks at Keywords at all).

For good measure, I have included such a Keywords header in this message.
Anyone see any problems with it?

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: "Charles H. Lindsey" <[email-masked]>
Date: Wed, 1 Jun 2005 10:52:24 GMT

In <7720B187EEDCD0754BC8E464@B50854F0A9192E8EC6CDA126> Harald Tveit Alvestrand <[email-masked]> writes:

>--On 30. mai 2005 13:45 +0000 Charles Lindsey <[email-masked]> wrote:

>> It was a newly introduced "feature" of RFC 2822, on a MUST accept but do
>> not generate yet basis. It is widely agreed that calling it "obs-" was a
>> mistake, because it was very much intended for future use. Many agents
>> accept it already. Bruce Lilly asked us to put it in (and for once he was
>> right).

>hey - where did that come from??????

It was hidden away in a section entitled "Obsolete Syntax", and so a lot
of people (myself included) never spotted it.

There was some discussion on the ietf-822 list around 12 months ago,
involving Bruce Lilly, Pete Resnick and others, in which all (Pete
included) agreed that hiding it there was a mistake, and that it ought to
be brought out and called <extended-phrase> if and when a 2822bis
happened.

>the text in RFC 2822 section 4.1 is:

>>    Note: The "period" (or "full stop") character (".") in obs-phrase is
>>    not a form that was allowed in earlier versions of this or any other
>>    standard.  Period (nor any other character from specials) was not
>>    allowed in phrase because it introduced a parsing difficulty
>>    distinguishing between phrases and portions of an addr-spec (see
>>    section 4.4).  It appears here because the period character is
>>    currently used in many messages in the display-name portion of
>>    addresses, especially for initials in names, and therefore must be
>>    interpreted properly.  In the future, period may appear in the
>>    regular syntax of phrase.

>The way I remember the discussions (after 4-10 years, I'm not that sure any 
>more...), people observed that this period was used quite widely, so 
>parsers that refused to handle it were not terribly useful, but the WG 
>refused to recommend it as part of existing syntax.

But clearly recommended it as part of the MUST accept syntax. The wording
above is a classic example of "Here is a mew feature; compliant software
MUST accept it now, but it MUST/SHOULD/whatever NOT be generated yet" (for
some value of "yet", which apparently included regularizing it in a future
standard).

Actually, I doubt any current email software refuses to tramsit and
display such a thing once it get onto the wire. I could well believe that
some software might be confused if asked to reply to a From: header with
it in (I have tried it in this message, as you will see - feel free to try
to reply to it).

Anyway, for News, Bruce Lilly pointed out some years ago that we should
adopt this new "feature", and it duly went into our draft (but entitled
<extended-phrase>). It does not seem to have made it into USEFOR yet, but I
think it should.

>Illustrates the danger of future-looking statements in standards that we're 
>arguing based on this text 4 years later.....

It illustrates the danger of putting important things in places where
nobody will notice them :-( .

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 01 Jun 2005 18:23:46 +0200

> If we are going to put it into USEFOR (and I think we
> should), then it may as well go into Keywords too. It is
> clearly allowed in Keywords in RFC 2822

Only by a very obs-cure rule, hiding itself as obs-olete,
and I'm not sure how it's supposed to work with RfC 2047:

Keywords: =?utf-8?Q?K._Moore?=

That's illegal, see 2047 5.3.  Dito for a <display-name>.

Normally you can split a phrase without quoted-string at
any word boundary and replace one or more words by an QP
encoded-word.

This obsure RfC 2822 rule messses with specials, STD 11,
RfC 2047, and s-o-1036.

And what about the CFWS in obs-phrase ?  If we'd say...

    phrase =       word *(word / "." / CFWS)

...simple RfC 2047 strategies also fail for the comments:

Keywords: =?utf-8?Q?Jonathan_(Jon)_Postel?=

Okay, that's kind of obvious, RfC 2047 only promises that
you can encode word(s) in a phrase, and "(Jon)" is no word.

Nevertheless, I don't like it.  For the same reasons why I
don't like RfC 2231:  MIME is supposed to be simple, robust,
and compatible.  Any screw after 2049, and it's screwed up.

> come to think of it, I am not aware of any regular news
> software that looks at Keywords at all

Certainly not my UA, unless I enable its "spam analysis"
aka "show headers - all".  It's much less important than
X-Face, and we decided against specifying X-Face... ;-)

> Anyone see any problems with it?

Yes, wasting more than one word about it in USEFOR is a
waste of bandwidth and therefore net abuse. <eg>

                       Bye, Frank

---

From: Bruce Lilly <[email-masked]>
Date: Wed, 1 Jun 2005 14:36:55 -0400

> That's illegal, see 2047 5.3.  Dito for a <display-name>.

Leaving aside for the moment whether or not putting Keith's initial
and surname in a Keywords field makes sense, I think you mean
section 5, paragraph marked "(3)"; there is no 5.3.

> any word boundary and replace one or more words by an QP

No, it's not like that.  If you replace an entire SP-delimited word,
you have to consider whether what is on the other side of the SP is
or is not an encoded word and is or is not part of the phrase and is
or is not a special, and you may have to include an encoded SP --
underscore for Q encoding, more complex for B encoding -- depending
on that analysis.  If you did that, the encoded-word length limit
might be exceeded, and you'd therefore have to choose a different or
additional splitting point.  You can't simply replace a word.  And
N.B. Q encoding differs from QP encoding.

In any event, '.' being a special, it cannot appear in an
encoded-word in a phrase.  RFC 2047 uses and is based on RFC 822
syntax rules, not 2822's, and 2822 does not discuss MIME other than
say that it's outside of 2822's scope.  Whether or not 
   Keywords: =?utf-8?Q?K?= . =?utf-8?Q?_Moore?=
might or might not be legal to generate under some hypothetical
successors of RFCs 2047 and 2822 is academic.  Under 2822, unquoted
'.' in a phrase is illegal for generation. And
   Keywords: =?utf-8?Q?K?= ". " =?utf-8?Q?Moore?=
is legal under 2047 + (822 or 2822).  Why one would use encoding
with utf-8 in the absence of language tagging for text which can be
represented in the default us-ascii charset is a complete mystery.
  
> This obsure RfC 2822 rule messses with specials, STD 11,

2822 obsoletes STD 11, and defines specials as the same set of twelve
characters as the obsoleted STD 11.

> RfC 2047,

orthogonal to 2822.

> and s-o-1036. 

Not a standard of any kind.  Not an RFC of any kind.  Predated
RFC 2822 and the observed widespread syntax error which 2822 accounts
for.

> you can encode word(s) in a phrase, and "(Jon)" is no word.

'.' is not a word either.

As for keywords, there might be some legitimate use, e.g. "X.25" or "T.3"
or "H.263" as keywords.  For now, they must be quoted.  The decision
regarding possible future codifcation in generation rules is up to an
RFC 2822 successor. 
----------
The reason for this to be part of USEFOR is that we have agreed that
USEFOR is to be based on 2822, and it is part of 2822.  There also seems
to be support for minimizing differences between the article format and
the Internet Message Format on which it is based wherever possible.

And where there is some operational issue of concern, Mark Crispin's
suggestion of an implementation note seems to be a vastly superior
method for handling the issue than a horde of "like 2822, except..."
clauses, incompatible redefinitions of RFC 2822 terms and grammar
productions, and/or endless debates over whether a difference should be
expressed by SHOULD or MUST:
http://www.imc.org/ietf-usefor/2003/Feb/0486.html
http://www.imc.org/ietf-usefor/2003/Feb/0493.html
http://www.imc.org/ietf-usefor/2003/Feb/0554.html

Had more of us listened to Mark's advice, we might have finished our
work two years ago.

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 02 Jun 2005 03:19:45 +0200

Bruce Lilly wrote:

 [2047]
> I think you mean section 5, paragraph marked "(3)";
> there is no 5.3.

Yes, your assumption is correct.

> If you replace an entire SP-delimited word, you have to
> consider whether what is on the other side of the SP is
> or is not an encoded word and is or is not part of the
> phrase and is or is not a special

Yes, and so far there never were any specials in a phrase,
unless they were already encoded / quoted / in a comment.

In other words a recipe like "grab anything in a phrase up to
the next delimiter (= comma in the case of keywords), DQUOTE,
encoded-word (=?), or comment , and then encode it as you see
fit under the rules of 2047" worked.

> N.B. Q encoding differs from QP encoding.

Yes, they are <quote rfc=2047> similar </quote>.

> encoded-word in a phrase.

Exactly, that's the point (pun intended).  The 2822 obs-phrase
would break the simple recipe to get it automatically right.

> Why one would use encoding with utf-8 in the absence of
> language tagging for text which can be represented in the
> default us-ascii charset is a complete mystery.

utf-8 was shorter than us-ascii for my example.  Think of any
reasonable case, e.g. =?utf-8*en?q?Moore?= combined with voice
output.  =?utf-8*de?q?Moore?= would sound like "more a" and is
the German plural of "moor".

> 2822 obsoletes STD 11

Don't tell me, tell it to legacy applications.  BTW, the IESG
has just approved an RfC explaining these issues of backwards
compatibility:

| There are many subtleties and variations that may come into
| play.   Authors should very carefully consider backwards
| compatibility when devising extensions, and should clearly
| describe all known compatibility issues.

Sounds familiar ?

> orthogonal to 2822.

It _redefines_ phrase.  There might be one redefinition too
many if USEFOR tries it for the third time, and it could...

| Cause a deployed implementation to behave differently from
| how it would behave in the absence of the proposed change,
| in ways that are not intended by the proposal.  That is a
| failure of the proposal to remain backwards compatible with
| the deployed base of implementations.

>> s-o-1036.
> Not a standard of any kind.  Not an RFC of any kind.

The best available standard available for Netnews articles, if
you have difficulties to decide which of the 13 + 4 usefor
drafts might be better than 1036.  Probably it inspired some
developers over the years, and resulted in deployed software.

Unlike msg-id = "<" id-local "@" id-domain ">" and potential
2231-boundary-risks I don't care much about this period-issue.

But I hate anything making usefor-05 longer / more obscure /
less readable / unclear / a waste of time.

                            Bye, Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 2 Jun 2005 11:35:49 GMT

>That's illegal, see 2047 5.3.  Dito for a <display-name>.

It is a MUST accept (but NOT generate) in RFC 2822. But it is not a valid
<encoded-word> according to RFC 2047 (because RFC 2047 has not caught up
with the change introduced by RFC 2822). But you could write

Keywords: =?utf-8?Q?K=2E_Moore?=

and then everyone should be happy.

But the whole point is that people are going to write
   From: K.Moore <keith@wherever>
whether we like it or not. So RFC 2822 said that MUST be accepted, and I
(and Bruce, it seems) are saying that USEFOR should say essentially the
same.

And if it does it for <addr-spec>s, then it may as well do it for Keywords
too, just to avoid yet more exceptions in the grammar (for sure, it won't
break anything to allow that).

---

From: Bruce Lilly <[email-masked]>
Date: Thu, 2 Jun 2005 15:22:34 -0400

> unless they were already encoded / quoted / in a comment.

The veracity of that statement depends on the definition of "in",
e.g. given
  Keywords: =?us?q?foo?= (bar), baz
the comment can be said to be "in" the phrase containing the
encoded-word (both appear before the delimiting comma), and
the left parenthesis is a special but would normally not be
considered to be "in" the comment (it delimits comments in
many structured fields).  This is an ambiguity related to
vagueness of RFC 822, which permits comments but is silent
about whether a comment such as the one above is or is not
"in" the phrase; RFC 2822 tacks on [CFWS] to many productions,
meaning the comment can clearly be said to be "in" the phrase:

atom            =       [CFWS] 1*atext [CFWS]

word            =       atom / quoted-string

phrase          =       1*word / obs-phrase

but unfortunately the 2822 grammar also results sometimes in
adjacent [CFWS] [CFWS], resulting in some remaining ambiguities
(but not in this instance).

> fit under the rules of 2047" worked.

The "under the rules of 2047" entails quite a bit of complexity;
it is definitely not as simple as considering a word without
looking at surrounding context.  Even in the simple case of
encountering a comma (== special), there are rules; an encoded-word
has to be separated from specials by linear whitespace.
 
> would break the simple recipe to get it automatically right.

The "simple recipe" isn't as simple as you initially stated, and
the rules handling specials suffice as well for a period as for a
comma.  Note that 2822's rule only states that if an unquoted
period appears in certain places it might be considered part of a
phrase for parsing; it does not permit it for generation.  Your
concern seems to be about generation ("grab anything [...] then
encode"), and as 2822 does not change the prohibition regarding
unquoted periods for generation, there is no problem.
  
> | describe all known compatibility issues.

Yes, as does "non-binding informational supplement".  In the
specific case under discussion, there is no change to generation
rules.  Lack of backwards compatibility happens when some new
set of generation rules permits something to be foisted upon
deployed implementations which are unprepared for it.  That doesn't
apply here.

> It _redefines_ phrase.

Not significantly (note that -- intentionally -- the important
characters constituting an encoded-word, '=', '?', and in the
case of the RFC 2231 amendment, '*', are all legal in "atom",
as are the B and Q encoded characters, legal charset and language
tag names (at least as currently defined)).  That was done
precisely for backwards compatibility; an encoded-word *is* an
atom as far as pre-MIME RFC 822 parsing is concerned.  The
"redefinition" is for the benefit of MIME-aware parsers that
recognize encoded-words as a special subset of atoms.  While
both =?us?q?foo?= and =??*-???= are atoms, only the former is an
encoded-word.

> There might be one redefinition too 
> many if USEFOR tries it for the third time,

Are you suggesting that there's a proposal on the table that would
do so?

> less readable / unclear / a waste of time.

In this case, a normative reference to RFC 2822 should suffice.  If
there is some operational issue related to it, then the most concise,
clearest, most readable, and simplest way to address the matter would
be along the lines suggested by Mark Crispin (not limited to *this*
issue; it applies to SP-after-colon, length of msg-id, etc. ad nauseum).

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 02 Jun 2005 22:07:22 +0200

Bruce Lilly wrote:

>>>> RfC 2047,
>>> orthogonal to 2822.
>> It _redefines_ phrase.
> Not significantly (note that -- intentionally

Sure, it's clear what it's really doing.  But some subtleties
with specials in Q-encoded words are still fascinating.  Like
attempts to decode without parsing "only to display it", where
"it" was a structured header field.

>> There might be one redefinition too many if USEFOR tries it
>> for the third time,

> Are you suggesting that there's a proposal on the table that
> would do so?

If this table includes "an idea I posted yesterday here", yes,
just replace the 2822 phrase by its obscene incarnation:

 {2822]

 [idea]

> a normative reference to RFC 2822 should suffice

We have a normative "2822 excl. all obs- forms".  The same
strategy as for Date would also work, but that's more than
just a reference, it's two paragraphs of text plus the ANBF.

                       Bye, Frank