Subject: draft-allbery-usefor-usepro-00 errata

From: Russ Allbery <[email-masked]>
Date: Thu, 14 Dec 2006 08:50:27 -0800

As a summary from the recent long threads, here are the changes I believe
should be made to draft-allbery-usefor-usepro-00 if we go forward with
that document.  The first one below has not previously been discussed on
the list.

 * USEFOR was changed (correctly, IMO) to allow only WSP between the
   arguments of control commands, not FWS.  I caught this for newgroup and
   rmgroup but missed it for checkgroups and cancel, both of which need
   FWS to WSP changes.

 * application/news-groupinfo and application/news-checkgroups need
   language limiting the selection of character sets.  I like Harald's
   proposal and would tend towards something like:

     Implementations predating this standard may not understand MIME
     headers and expect newsgroup names to be in ASCII.  Therefore,
     regardless of what charset is used, the result of reading each octet
     of the body and setting bit 8 to zero MUST be a valid message
     specifying the same newsgroup name [or names for checkgroups].  There
     is no requirement that the newsgroup description survive this
     treatment.

 * multipart/related in newgroup control messages should be
   multipart/mixed instead.

 * A separate sub-section of duties, before the individual agent duties
   and possibly as part of the same sub-section as the identity
   discussion, should be added specifying Path header construction.  The
   Path header example should be moved to be a sub-section of that
   section.  That section can lay out all the construction requirements
   for the Path header field and just be referred to by the individual
   duties sections.

 * The possibility of adding multiple path-identities should be
   reintroduced as part of the rework of the Path description.

 * Serving agents also are not required to modify the Path header field if
   processing an article from a relaying agent or injecting agent that's
   part of the same server.  This can be handled more generally in the
   redone Path section.

 * application/news-transmission should explicitly note that, contrary to
   the previous registration, batches are not permitted.

 * The trimming requirement for References should probably go back to
   MUST.

 * The correct description of special routing for newgroup and rmgroup
   control messages is something more like:

     Exceptionally, control messages creating or removing newsgroups
     (newgroup or rmgroup control messages, for example) SHOULD be relayed
     if the affected group appears in its Newsgroups header field and the
     sending agent would have supplied and the receiving agent would have
     received the newsgroup affected by the control message had it
     existed, even if it currently does not.

   The current USEPRO text for construction of the Newsgroups header field
   for those control messages should be restored (and probably as a
   general statement about group control messages that affect only one
   group rather than stated independently in both newgroup and rmgroup).

 * We need some resolution of the different conflicting syntaxes for ihave
   and sendme control messages.  Path of least resistance is probably to
   revert to the current USEPRO tactic of two separate ABNFs.

 * (Still under discussion.)  We may wish to reintroduce the possibility
   of a path-identity that is not a resolvable name in DNS.

 * (Still under discussion.)  We may wish to add a sentence about the
   construction of newsgroups in the to.* hierarchy.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: Harald Alvestrand <[email-masked]>
Date: Thu, 14 Dec 2006 21:48:44 +0100

>    of a path-identity that is not a resolvable name in DNS.

Last time I looked at the content of Path: headers, I think that there were 
a fair number of names in dotted.name form that were not DNS resolvable. So 
banning those names would be a break with existing practice.

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 15 Dec 2006 20:30:16 GMT

>that document.  The first one below has not previously been discussed on

Mostly agreed, but wordings may need tweaking. But there will surely be
lots of other changes needed in due course, so no great rush with these.

>   rmgroup but missed it for checkgroups and cancel, both of which need

Yes, I spotted that one too, and it was on my list of minor typos, etc.

>     is no requirement that the newsgroup description survive this

That seems terribly wordy. Surely all you need is that octets 0-127 (or at
least all those that can occur in newsgroup-names plus TAB) MUST represent
the corresponding ASCII characters.

OTOH, one slight worry. Suppose we later go to newsgroup-names in UTF8
(following the path now beng blazed by the EAI WG). Would we than have to
say that the charset MUST have UTF-8 as a subset? Or merely that the
charset used must be able to represent all the newsgroup-names in that
checkgroups? One of the beauties of newsgroup-names in UTF8 is that (as we
have already established by trying it with the group dk.test.utf8-æøå)
that existing news servers can cope immediately (user agents would have to
change, of course).

But if checkstatus was allowed to show "dk.test.utf8-æøå" in either UTF-8
or in Iso-8859-1, would checkstatus still work on existing news servers
(it probably would in the UTF-8 case)?

>   for the Path header field and just be referred to by the individual

Yes, that might be good.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 15 Dec 2006 20:32:59 GMT

>banning those names would be a break with existing practice.

Or, alternatively, do we think existing practice is flawed (especially if
we want to encourage the use of MISMATCH et al)? Whatever we say, it will
not change overnight, of course.

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 16 Dec 2006 15:13:06 +0100

Charles Lindsey wrote:

> That seems terribly wordy. Surely all you need is that octets
> 0-127 (or at least all those that can occur in newsgroup-names
> plus TAB) MUST represent the corresponding ASCII characters.

IIRC we've ruled that out as not good enough, an example is
http://source.icu-project.org/repos/icu/data/trunk/charset/data/xml/glibc-SJIS-2.1.2.xml

[...]
 <validity>
  <state type="FIRST" next="VALID" s="0" e="7F" max="FFFF"/>
  <state type="FIRST" next="SECOND" s="81" e="9F" max="FFFF"/>
  <state type="FIRST" next="VALID" s="A0" e="DF" max="FFFF"/>
  <state type="FIRST" next="SECOND" s="E0" e="FC" max="FFFF"/>
  <state type="SECOND" next="VALID" s="40" e="7E" max="FFFF"/>
  <state type="SECOND" next="VALID" s="80" e="FC" max="FFFF"/>
 </validity>
[...]

The s="40" e="7E" in the state type="SECOND" means that the
octets 0x40..0x7E can be the second byte (after a FIRST byte
0x81..0x9F or 0xE0..0xFC) of double-byte characters.

Apparently Ned's example "euc-jp" doesn't have this problem.

Russ' new wording also works for charsets like UTF-7, UTF-1,
or the "SJIS" variant shown above - the online version of ICU
makes it almost impossible to determine what it really is.

---
> OTOH, one slight worry. Suppose we later go to newsgroup-names
> in UTF8 (following the path now beng blazed by the EAI WG).

And 3977.

> Would we than have to say that the charset MUST have UTF-8
> as a subset?

Russ' text doesn't use the term "subset".  There are only two
charsets with UTF-8 as "subset" I'm aware of, the old UTF-8 and
CESU.  Software expecting UTF-8 would panic as soon as it sees
0xFE, 0xFF, or similar crap.

Without explicit charset= the ASCII rule works for ASCII names,
for future UTF-8 names an explicit charset= would be required.

IMO it's obvious, but that future draft could say that using
any charset that's not UTF-8, like Latin-1, is NOT RECOMMENDED
if non-ASCII group names are affected.

Frank

---

From: Harald Alvestrand <[email-masked]>
Date: Sun, 17 Dec 2006 19:13:34 +0100

> the corresponding ASCII characters.

MUST or MUST ALWAYS? The set of useful character sets is quite different in 
the two cases.
The language Russ suggested allows one to evaluate a specific message in a 
specific charset against the criterion - for instance, an ISO-2022-JP 
message MAY satisfy the criterion, IF the shift to Japanese characters 
always occurs in the description, and never in the newsgroup name.

> that existing news servers can cope immediately (user agents would have to

If we ever want to standardize such a thing, I think the logical thing to 
do would be to say that the charset MUST be UTF-8.

But that is not part of any proposal under discussion, so it's pure 
speculation at this stage.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 18 Dec 2006 19:50:26 GMT

>always occurs in the description, and never in the newsgroup name.

OK, I see what the problem is now. Essentially, the property I suggested
needs to apply up to the TAB that introduces the description part. After
that, we don't care.

But anything with a bit8 set in the <newsgroup-name> part will break
existing software, because existing software will not be setting that bit
to zero before trying to interpret it.

So what you say must be more like:

      The charset that is specified MUST have the property that any valid
      <newsgroup-name> appearing at the start of each line, and before the
      TAB character, is represented by the same sequence of octets both in
      that charset and in ASCII (this does not preclude that, beyond the
      TAB, there may be some shift character that changes the
      interpretation of subsequent octets).

That also is wordy and might be improved, but at least it makes clear what
the issue is.

>do would be to say that the charset MUST be UTF-8.

Yes, but you then lose backward compatibility if you have earlier allowed
something else. The wording I suggested above, with s/ASCII/UTF8/ would
still work, except I doubt there exists any charset with the necessary
property.

>speculation at this stage.

True, but we should not paint ourselves into a corner, since I regard that
development as quite likely, and not so far away.

---

From: Russ Allbery <[email-masked]>
Date: Mon, 18 Dec 2006 14:03:06 -0800

Charles Lindsey <[email-masked]> writes:
> Harald Alvestrand <[email-masked]> writes:

[ Proposed new text ]
>>>>     regardless of what charset is used, the result of reading each
>>>>     octet of the body and setting bit 8 to zero MUST be a valid
>>>>     message specifying the same newsgroup name [or names for
>>>>     checkgroups].  There is no requirement that the newsgroup
>>>>     description survive this treatment.

>> The language Russ suggested allows one to evaluate a specific message
>> in a specific charset against the criterion - for instance, an
>> ISO-2022-JP message MAY satisfy the criterion, IF the shift to Japanese
>> characters always occurs in the description, and never in the newsgroup
>> name.

> needs to apply up to the TAB that introduces the description part. After

Right.

> existing software, because existing software will not be setting that
> bit to zero before trying to interpret it.

Correct.  Which is why the above wording declares such messages invalid
(since such a message would specify a different newsgroup name after the
treatment described was applied).

>       The charset that is specified MUST have the property that any
>       valid <newsgroup-name> appearing at the start of each line, and
>       before the TAB character, is represented by the same sequence of
>       octets both in that charset and in ASCII (this does not preclude
>       that, beyond the TAB, there may be some shift character that
>       changes the interpretation of subsequent octets).

I think that also describes the same situation, but I prefer the wording
above.  I think it's clearer and more explicit.

> That also is wordy and might be improved, but at least it makes clear
> what the issue is.

The first sentence makes that clear, I think.

>> If we ever want to standardize such a thing, I think the logical thing
>> to do would be to say that the charset MUST be UTF-8.

> Yes, but you then lose backward compatibility if you have earlier
> allowed something else.

You only have to require a charset of UTF-8 if you have non-ASCII
newsgroups; otherwise, anything can be used that satisfies the above
requirement.  So I don't agree that you would lose backward compatibility.

---

From: Russ Allbery <[email-masked]>
Date: Mon, 18 Dec 2006 14:21:35 -0800

> if non-ASCII group names are affected.

Exactly.

Currently non-ASCII group names are banned entirely by USEFOR, so it
doesn't make much sense to say this now.

We could RECOMMEND use of either US-ASCII or UTF-8 for the charset in the
name of making future transitions easier, though.  I can see some merit in
going that route, particularly since I think we're fairly certain at this
point that newsgroup names will either be UTF-8 or some ASCII encoding.  I
don't see any other non-ASCII character set cropping up in the IETF.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 19 Dec 2006 14:59:34 GMT

>above.  I think it's clearer and more explicit.

I think we are agreed about the effect to be achieved. But the wording
needs to be such that readers not familiar with the background to this
issue will easily be able to see what it implies. Moreover, if ever we do
go for UTF-8 newsgroup names, then it would be nice to have a wording that
can be adapted, rather than being a complete rewrite.

But I don't think either wordings suggested really works as it stands.

>requirement.  So I don't agree that you would lose backward compatibility.

Ah! So when that happens, existing ASCII newsgroups could continue to use,
say, Iso8859-1 in checkgroups, but any new Utf-8 groups would be
restricted to using UTF-8. Seems slightly weird, but it would work.

BTW, I agree with the suggestion, in another thread, that RECCOMMENDING
(or recommending) UTF-8 at this stage would be a good idea.

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 20 Dec 2006 01:17:12 +0100

> Exactly.

Harald proposed MUSTard, he's probably right, and my RECOMMENDED
is too weak.  Buth that's an issue for a future USEPRO-bis in 2010.

> We could RECOMMEND use of either US-ASCII or UTF-8 for the charset
> in the name of making future transitions easier, though.

> I don't see any other non-ASCII character set cropping up in
> the IETF.

BOCU-1 isn't user friendly, UTF-4 is no alternative, SCSU has no
unique encoding, UTF-7 makes no sense in an 8bits environment,
and the s-o-1036 concept of 2047-encoded words was apparently
stillborn,

Frank

---

From: Russ Allbery <[email-masked]>
Date: Fri, 29 Dec 2006 10:06:01 -0800

>> IMO it's obvious, but that future draft could say that using any
>> charset that's not UTF-8, like Latin-1, is NOT RECOMMENDED if non-ASCII
>> group names are affected.

> We could RECOMMEND use of either US-ASCII or UTF-8 for the charset in
> the name of making future transitions easier, though.  I can see some
> merit in going that route, particularly since I think we're fairly
> certain at this point that newsgroup names will either be UTF-8 or some
> ASCII encoding.  I don't see any other non-ASCII character set cropping
> up in the IETF.

While implementing this wording change, I realized that most of the
problem is already handled by the ABNF for these media types.  It already
specifies specific octets for things such as HTAB and the allowable
characters for a newsgroup name.  Because of that, it looked to me like we
could simplify this language considerably and use a sentence closer to
what Charles suggested.

The ABNF was restricting the newsgroup description to 7bit characters, so
that also needed to be changed.

Here's what I currently have:

   The content of the application/news-groupinfo body part is defined
   as:

         groupinfo-body      = [ newsgroups-tag CRLF ]
                                  newsgroups-line CRLF
         newsgroups-tag      = %x46.6F.72 SP %x79.6F.75.72 SP
                                  %x6E.65.77.73.67.72.6F.75.70.73 SP
                                  %x66.69.6C.65.3A
                                  ; case sensitive
                                  ; "For your newsgroups file:"
         newsgroups-line     = newsgroup-name
                                  [ 1*HTAB newsgroup-description ]
                                  [ 1*WSP moderation-flag ]
         newsgroup-description
                             = eightbit-utext *( *WSP eightbit-utext )
         moderation-flag     = %x28.4D.6F.64.65.72.61.74.65.64.29
                                  ; case sensitive "(Moderated)"
         eightbit-utext      = utext / %d127-255

   This unusual format is backward-compatible with the scanning of the
   body of newgroup control messages for descriptions done by Netnews
   implementations that predate this specification.  Although optional,
   the <newsgroups-tag> SHOULD be included for backward compatibility.

   The <newsgroup-description> MUST NOT contain any occurrence of the
   string "(Moderated)" within it.  Moderated newsgroups MUST be marked
   by appending the case sensitive text " (Moderated)" at the end.

   While a charset parameter is defined for this media type, most
   existing software does not understand MIME header fields or correctly
   handle descriptions in a variety of charsets.  Using a charset of US-
   ASCII where possible is therefore RECOMMENDED; if not possible, UTF-8
   [RFC3629] SHOULD be used.  Regardless of the charset used, the
   constraints of the above grammar MUST be met and the <newsgroup-name>
   MUST be represented using the same octets as would be used with a
   charset of US-ASCII.

and under application/news-checkgroups:

   The same restrictions on charset, <newsgroup-name>, and <newsgroup-
   description> apply for this media type as for application/
   news-groupinfo.

Let me know if I'm missing a subtlety that would have been expressed by
Harald's language.

---

From: Russ Allbery <[email-masked]>
Date: Fri, 29 Dec 2006 11:00:14 -0800

>    group rather than stated independently in both newgroup and rmgroup).

I now have:

   Exceptionally, control messages creating or removing newsgroups
   (newgroup or rmgroup control messages, for example) SHOULD be relayed
   if the affected group appears in its Newsgroups header field and the
   sending agent would have supplied and the receiving agent would have
   received the newsgroup affected by the control message had it existed,
   even if it currently does not.

for the propagation rule, the following under group control messages:

   Group control messages affecting specific groups (newgroup and
   rmgroup control messages, for example) SHOULD include the <newsgroup-
   name> for the group or groups affected in their Newsgroups header
   field.  Other newsgroups MAY be included in the Newsgroups header
   field so that the control message will reach more news servers, but
   due to the special relaying rules for group control messages (see
   Section 3.5) this is normally unnecessary and may be excessive.

the following under cancel control messages:

   A cancel control message SHOULD have the same Newsgroups header field
   as the message it is cancelling so that it will attempt to reach the
   same news servers as the original message.

and the following in Security Considerations:

   Cancel control messages are not required to have the same Newsgroups
   header field as the messages they are cancelling and, since they are
   sometimes processed before the original message is received, it may
   not be possible to check that they do.  This allows a malicious
   poster to inject a cancel control message for an article in a
   moderated newsgroup without adding an Approved header field to the
   control message, and to hide malicious cancel control messages from
   some reading agents by using a different Newsgroups header field so
   that the cancel control message is not accepted by all news servers
   that accepted the original message.

---

From: Russ Allbery <[email-masked]>
Date: Fri, 29 Dec 2006 11:21:49 -0800

>    revert to the current USEPRO tactic of two separate ABNFs.

I've since reviewed RFC 1036 and Son-of-1036.  The former makes the
relayer name optional and requires that the message IDs be in the control
header field.  The latter makes the relayer name mandatory and RECOMMENDS
that the message IDs be put in the body.  I think the following is the
most reasonable compromise; it makes the relayer-name mandatory if there
are no message IDs (the form never allowed in RFC 1036 but according to
Son-of-1036 the most widely used in practice), and if there are message
IDs, permits the same syntax as RFC 1036.

Let me know if anything looks wrong.

   ihave and sendme control messages share similar syntax for their
   Control header fields and bodies:

       control-command     =/ Ihave-command
       Ihave-command       = "ihave" Ihave-arguments
       Ihave-arguments     = 1*WSP relayer-name
                             / 1*( 1*WSP msg-id ) [ 1*WSP relayer-name ]
       control-command     =/ Sendme-command
       Sendme-command      = "sendme" Sendme-arguments
       Sendme-arguments    = Ihave-arguments
       relayer-name        = path-identity  ; see 3.1.5 of [USEFOR]
       ihave-body          = *( msg-id CRLF )
       sendme-body         = ihave-body

   The body of the article consists of a list of <msg-id>s, one per
   line.  The message identifiers SHOULD be put in the body of the
   article, not in the Control header field, but news servers MAY
   recognize and process message identifiers in the Control header field
   for backward compatibility.  Message identifiers MUST NOT be put in
   the Control header field if they are present in the body of the
   control message.

---

From: Russ Allbery <[email-masked]>
Date: Fri, 29 Dec 2006 11:36:11 -0800

>    construction of newsgroups in the to.* hierarchy.

I'm convinced.  I now have:

   These control messages are normally sent as point-to-point articles
   between two sites and not then sent on to other sites.  Newsgroups
   beginning with "to." are reserved for such point-to-point
   communications and are formed by prepending "to." to a <relayer-name>
   to form a <newsgroup-name>.  Articles with such a group in their
   Newsgroups header fields SHOULD NOT be sent to any news server other
   than the one identified by <relayer-name>.

---

From: Russ Allbery <[email-masked]>
Date: Fri, 29 Dec 2006 12:31:59 -0800

>    part of the same server.  This can be handled more generally in the

As promised, here is the new section.  I also made the corresponding
updates to the other sections (removing the description and replacing it
with a reference to this section) and moved the Path header field example.

3.2.1.  Constructing the Path Header Field

   If a relaying or serving agent receives an article from an injecting
   or serving agent that is part of the same news server, it MAY leave
   the Path header field of the article unchanged.  Otherwise, every
   injecting, relaying, or serving agent that accepts an article MUST
   update the Path header field as follows.  Note that the Path header
   field content is constructed from right to left by prepending
   elements.

   1.  The agent MUST prepend "!" to the Path header field content.

   2.  An injecting agent SHOULD prepend the <path-diagnostic>
       "!.POSTED", optionally followed by "." and the FQDN or IP address
       of the source, to the Path header field content.

   3.  A relaying or serving agent SHOULD prepend a <path-diagnostic> to
       the Path header field content, where the <path-diagnostic> is
       chosen as follows:

       *  If the expected <path-identity> of the source of the article
          matches the leftmost <path-identity> of the Path header
          field's content, use "!" (<diag-match>).

       *  If the expected <path-identity> of the source of the article
          does not match, use "!.MISMATCH." followed by the expected
          <path-identity> of the source or its IP address.

       *  If the relaying or serving agent is not willing or able to
          check the <path-identity>, use "!.SEEN." followed by the FQDN,
          IP address, or expected <path-identity> of the source.

   4.  The agent MUST then prepend its own <path-identity> to the Path
       header field content.

   5.  The agent MAY then prepend additional <path-identity>s for itself
       to the Path header field content, following each <path-identity>
       so added with either "!!" or "!".  This is permitted for agents
       that have multiple <path-identity>s (such as during a transition
       from one to another).  Each of these <path-identity>s MUST meet
       the requirements set out in Section 3.2.

   Any agent which modifies the Path header field MAY fold it by
   inserting FWS immediately after any <path-identity> or <diag-other>
   it added (see section 3.1.5 of [USEFOR] for allowable locations for
   FWS).

>    of a path-identity that is not a resolvable name in DNS.

This was already in my draft.  I have:

   The <path-identity> used by an agent may be chosen via one of the
   following methods (in decreasing order of preference):

   1.  The fully-qualified domain name (FQDN) of the system on which the
       agent is running.

   2.  A fully-qualified domain name (FQDN) within a domain affiliated
       with the administrators of the agent and guaranteed to be unique
       by the administrators of that domain.  For example, the
       uniqueness of server.example.org could be guaranteed by the
       administrator of example.org even if there is no DNS record for
       server.example.org itself.

   3.  Some other (arbitrary) name in the form <path-nodot> believed to
       be unique and registered at least with all the other news servers
       to which that relaying agent or injecting agent sends articles.
       This option SHOULD NOT be used unless the earlier options are
       unavailable or unless the name is of longstanding usage.

This is the same as what's in the current USEPRO draft except that my 1.
replaces:

   1. A fully qualified domain name (FQDN) that SHOULD be resolvable in
      the DNS (whether via an A, AAAA or MX record or an equivalent
      CNAME), thus guaranteeing a unique identity. Ideally, it will also
      provide a means to contact the administrators by email (according
      to [RFC 2142], the forms "usenet@server" and "news@server" are
      common addresses for a news server administrator).

or

   1. A fully qualified domain name (FQDN) that can be resolved to an
      email server via an MX, A or AAAA record according to the
      procedures of [RFC 2821]; this guarantees that the name is unique,
      and makes it easy to contact the administrators if needed.

The most common current practice is to use the FQDN of the news server,
not a mail server, as a path identity.  If one wants to use a mail server,
2. still allows for that; it doesn't require that the name *not* exist in
DNS.

In my original draft, I intentionally removed any protocol-required
connection between a <path-identity> and a contact address.  After lots of
previous discussion of this point, I don't think the potential gains of
discussing such a connection in this specification are worth the
complexity and divergence from current practice.

---

From: Russ Allbery <[email-masked]>
Date: Fri, 29 Dec 2006 12:33:16 -0800

Russ Allbery <[email-masked]> writes:

> As a summary from the recent long threads, here are the changes I
> believe should be made to draft-allbery-usefor-usepro-00 if we go
> forward with that document.  The first one below has not previously been
> discussed on the list.

These errata have now been applied to my working copy.  Where the text was
substantial or I felt it was possibly controversial, I sent the new text
to the list; for the rest of these errata, I made the obvious changes.

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 30 Dec 2006 15:54:01 +0100

Russ Allbery wrote:

>    (newgroup or rmgroup control messages, for example)

Ignoring "mvgroup" the example is a complete enumeration, isn't it ?
The rest of the text doesn't make sense for a "checkgroups".  Maybe
remove the parentheses.

>    rmgroup control messages, for example)

Same here, a "checkgroups" listing _all_ newsgroups in the Newsgroups
header field would be odd.  Get rid of "for example", the qualifier is
confusing without "mvgroup".

##########################################################################

>    that accepted the original message.

So far it's clear.  Is that good enough for a MUST instead of a SHOULD ?
Or an explicit OPTION to ignore cancels violating the SHOULD ?

Frank

---

From: Russ Allbery <[email-masked]>
Date: Sat, 30 Dec 2006 09:13:04 -0800

> The rest of the text doesn't make sense for a "checkgroups".  Maybe

*Now*, but others may be introduced later and this should still hold, so
it felt right to me to keep it general.  The standard makes it clear that
anyone who wishes can invent a new control message and start sending them.

> Or an explicit OPTION to ignore cancels violating the SHOULD ?

Two reasons not to make it a MUST:

 * Posting a cancel message to a test newsgroup will result in autoreplies
   from some sites, so it's conventional to drop test newsgroups from
   cancel messages.

 * Some hierarchies post all spam cancels to a designated newsgroup to
   make it easier for people who don't want them to opt out by not
   accepting the group.

---

From: Frank Ellermann <[email-masked]>
Date: Sun, 31 Dec 2006 15:11:31 +0100

Russ Allbery wrote:

> The standard makes it clear that anyone who wishes can invent a new
> control message and start sending them.

That's apparently a derivation from usepro-06 in your text, you have
"MAY send or accept whatever" with a "MUST ignore or reject unknown".

IMO that can't fly if relaying agents reject unknown stuff.  We could
create a IANA registry of control <verb>s, reserving x-whatever for
experimental / unofficial <verb>s, and anything else to be defined in
an RFC.  With a general "MUST ignore unknown", limiting "MAY reject
unknown" to injecting agents.  Or better removing the "reject" option.

The IANA registry stuff is fairly simple, Harald has written a kind of
"howto" (2434bis), and "defined in an RFC" is straight forward.

A "MUST ignore or reject unknown" is IMHO pointless, what else should
they do ?

#######################################################################

  [A cancel SHOULD match the newsgroups of the original article]
>> Is that good enough for a MUST instead of a SHOULD ?  Or an explicit
>> OPTION to ignore cancels violating the SHOULD ?

>  * Posting a cancel message to a test newsgroup will result in
>    autoreplies from some sites, so it's conventional to drop test
>    newsgroups from cancel messages.

The first thing I did on any server I've used (all 5 of them :-) was to
post an article in the local test group and cancel it, checking out if
if that works as expected, and where I'd find the cancel.  I also did
that in de.test and de.alt.test (for experiments with X-Posts and with
Google's "purge" procedure).  No trouble with checkbots anywhere, and I
think we can ignore checkbot oddities while discussing cancel
standards.

> * Some hierarchies post all spam cancels to a designated newsgroup
>   to make it easier for people who don't want them to opt out by not
>   accepting the group.

Spam cancels intentionally violate official rules, following their own
standards.  I'd like an informative reference to Skirvin's cancel FAQ,
but for USEPRO we need some minimal "normal" procedure.

Tons of SHOULD not explaining why and when they're violated don't help.

If your justification for a SHOULD instead of a MUST is "spam cancel"
please say so explicitly, with an informative reference.   I think we
have four cases for an article X-posted in unmoderated groups A and B:

1: cancel in A + B: sane
2: cancel in A:     bad but acceptable
3: cancel in C + B: odd variant of (2)
4: cancel in C:     bad, unacceptable

With that I get "at least one of the newsgroups of the cancel MUST
match a newsgroup of the cancelled article" in addition to your SHOULD.

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Sun, 31 Dec 2006 16:41:20 +0100

> 4: cancel in C:     bad, unacceptable

More about that for "Supersedes", if an article was posted in group X,
and the poster a.s.a.p. decided that it should go to group Y instead:

a: NG: X,Y; Fup2: Y   : that's clear for folks only seeing group X
b: NG: Y              : bad, old article mysteriously vanishes from X
c: cancel in X
   new article in Y   : obviously okay

The same rule (Russ' SHOULD match all NGs + my MUST match at least one)
apparently also works for "Supersedes".  Actually "Supersedes" is worse
than "cancel", should we say "MUST match all NGs" for a "Supersedes" ?

Frank

---

From: Russ Allbery <[email-masked]>
Date: Sun, 31 Dec 2006 09:36:18 -0800

Frank Ellermann <[email-masked]> writes:

> an RFC.

I don't think this is really necessary and it adds bulk and procedure to
the document and will mean a fair bit more discussion to figure out review
and registration procedures.  I can see the merit, but personally I'd
prefer not to.  I'm willing to be convinced otherwise, though.

> With a general "MUST ignore unknown", limiting "MAY reject unknown" to
> injecting agents.  Or better removing the "reject" option.

Since relaying agents may reject messages for any reasons they chose,
listing reject as an option doesn't change anything about the requirements
for a relaying agent and makes it clearer in the text what the options are
for an implementor.  I want to make it clear that many agents *will*
reject unknown control messages (since that's what happens in practice).

> A "MUST ignore or reject unknown" is IMHO pointless, what else should

Attempt to guess at the meaning or bother the news administrator.  It's
standard to include language of this sort for extension mechanisms;
something like this has been in every other RFC that I've worked with that
has optional extensions.

> think we can ignore checkbot oddities while discussing cancel standards.

I don't agree.  The plural of anecdote is not data and this has been a
known problem for many years.  Autoresponders come and go, different test
newsgroups behave differently, and it's very easy to cause this problem
when writing a naive autoresponder.

> Spam cancels intentionally violate official rules, following their own

Spam cancels and other sorts of administrative cancels are compliant with
my protocol spec and I want to keep it that way.  They're an authorization
problem, not a protocol problem.

I think the current text is fine.  It reflects what news servers currently
do when they honor cancels, and I don't believe that tweaking the language
will provide sufficient practical benefit for it to be useful.  The
Security Considerations section will have to remain anyway given the
number of existing servers that work that way.

---

From: Frank Ellermann <[email-masked]>
Date: Sun, 31 Dec 2006 21:24:45 +0100

> I don't think this is really necessary and it adds bulk and procedure
> to the document and will mean a fair bit more discussion to figure out
> review and registration procedures.

Not really, it just needs a numbered list in 2434bis, it's the 7th point
in chapter 4.1 of this I-D:

|  RFC Required - RFC publication (either as IETF Submission or as an
|                 RFC Editor submission [RFC3932]) suffices.

Reserving x-verbs would cover points 1 / 2 (private / experimental use).
We don't need "expert review" or anything more restrictive than "RFC".

It was your idea to add the OPTION of new control verbs, with the odd
effect of talking about newgroup + rmgroup "for example", as if there
are others.   If you really want this we can also get it right, we've
all ingredients for a proper IANA registry, some obsolete verbs, some
"real" verbs, and one hypothetical mvgroup to be specified elsewhere.

If you don't really want this let's please get rid of the MAY and "for
example" cruft, it's halfhearted and confusing.

>> With a general "MUST ignore unknown", limiting "MAY reject unknown"
>> to injecting agents.  Or better removing the "reject" option.

> Since relaying agents may reject messages for any reasons they chose,

This is not the case, compare usepro-06 chapter 6.3 points 1..8.  They
must / should / may reject articles for the reasons specified in 1..8.

Your points 1..10 in 3.5 are similar, just add what's missing - maybe
relaying agents need to be free to enforce an UDP no matter what their
peers do, to reject obvious spam, the works.  Don't say "any reason",
that's censorship and evil.

Articles MUST NOT disappear for unknown "any" reasons.  It needs good
reasons, and USEPRO is supposed to provide a list of good reasons, and
ways to check this for observers.

> I want to make it clear that many agents *will* reject unknown control
> messages (since that's what happens in practice).

That's fine, but why write so much about these hypothetical new control
messages if you don't believe in it ?  And it wasn't clear for me, some
lines above I still thought that you might really want to support new
control messages.  So I am (or was) confused.  "Reject crap" is a clear
directive - unlike "MAY send / accept crap + MUST ignore / reject crap".

> It's standard to include language of this sort for extension mechanisms;
> something like this has been in every other RFC that I've worked with
> that has optional extensions.

I don't recall any "MUST ignore or reject unknown" in RfCs I've read, do
you have an example ?  If the idea is to protect the network from unknown
and potentially dangerous crap "reject" is good, if the idea is to allow
unknown future extensions "ignore" is good, but any party doing whatever
it likes better is dubious.  For me it sounds like "MUST do X or not X".

Frank

---

From: Russ Allbery <[email-masked]>
Date: Sun, 31 Dec 2006 15:30:42 -0800

> We don't need "expert review" or anything more restrictive than "RFC".

I think requiring an RFC is too restrictive and think x-verbs are a bad
idea that cause more problems than they solve in this area, so it would
require more discussion than that.

> are others.

No, it was not my idea.  It's how Netnews actually works and has for
longer than I've been involved in it.  From the first version of INN, you
could drop in a handler for a brand new control message by doing nothing
more than putting a shell script in a directory, and people did so to
various ends.  Adding a new group control message would also require
changing one line of code in innd that classifies control messages as
"newgroup-like" for propagation purposes, but that's trivial.  That new
control verbs weren't allowed was a pure artificial creation of previous
documents (assuming one even read them that way) which I have attempted to
bring in line with reality.

Now we can, if the working group wants, discuss whether we want to try to
change reality, but what I did was fix an inconsistency between what was
on paper and how Netnews software actually works.

> must / should / may reject articles for the reasons specified in 1..8.

I didn't realize that anyone in the working group didn't consider this
principle implicit and assumed.  I'm glad, in this case, that I made it
explicit in my draft.  Section 3.1 of my draft says:

   No Netnews agent is ever required to accept any article.  It is
   common for injecting, relaying, and serving agents to reject well-
   formed articles for reasons of local policy (such as not wishing to
   carry a particular newsgroup or attempting to filter out unwanted
   articles).  This document specifies how articles are to be treated if
   they are accepted and specifies some cases where they must be
   rejected, but an agent MAY always reject any article for other
   reasons than those stated here.

I consider this principle to be at the core of how Netnews works and will
strenuously object to any attempt to fail to include it in a protocol
specification for Netnews (not to mention that an assumption that agents
will only reject messages for well-defined reasons specified in the
protocol document is catastrophically misleading if the goal if this is to
produce a document that defines how Netnews works in the real world).

> messages if you don't believe in it ?

If I don't believe in what?  I certainly believe that it's possible to
introduce new control messages; it's been done before.  Many agents will
reject them and many won't.  It will take some time and general agreement
with the new control message before they will have the propagation that
one might desire, so in the interim people should have backup procedures
available.  None of this is horribly difficult.  It just takes time and an
idea that's worth the effort.

>> It's standard to include language of this sort for extension
>> mechanisms; something like this has been in every other RFC that I've
>> worked with that has optional extensions.

> you have an example ?

By "of this sort" I meant stating explicitly what should be done with
unrecognized extensions.  The common language, as you point out, is "MUST
ignore", which is present in, for example, RFCs 2060, 2251, 2449, 2616,
2919, 3501, 3546, 3977, and 4121, just to pick out a few RFCs that I have
lying around.

Because a Netnews agent MAY reject any message it choses, it MAY reject an
unknown control message rather than ignoring it, so saying in this case
that such messages "MUST be ignored or rejected" is more complete and
avoids even the possible perception of self-contradiction.