Subject: draft-ietf-usefor-usepro-01

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 21 Sep 2004 11:33:26 GMT

There was some discussion a while back as to whether various bits or
protocol material specific to particular headers should remain with those
headers after the Usefor/Usepro split, or be extracted and placed in
Usepro. Our Chair has now asked that they be moved to Usepro, and this
draft implements that change.

The draft is now (or soon will be) on
www.landfield.com/usenet-format/drafts/draft-ietf-usefor-usepro-01.txt
www.landfield.com/usenet-format/drafts/draft-ietf-usefor-usepro-01.unpaged
and is also on its way to the drafts editor.

It makes no changes wrt the earlier drafts apart from the items listed
below, though the wording is revised to suit the new environment. Full
diffs are at the end of this message.

The (possible) changes are as follows:

1. In 7.3, I have demoted the MUST in the first para of draft-13:5.5 which
seemed a bit limiting on those relaying agents which tend to fire off
articles whether on not the receiving site wanted them. Surely SHOULD is
sufficient, but I could be persuaded I have gone too far.

2. In 7.3 Steps 3 and 4 it said that Relaying agents MUST check the
legality of all mandatory headers and SHOULD check the legality of the
optional ones as well. I doubt that all relaying agents do such full
checks (surely the injecting and serving sites are the ones that should
check everything), and I suspect something nearer to SHOULD and MAY would
be better. Comments?

3. I have added a section on "Duties of a Reading Agent". The prime reason
for this was to find a home for the bit in draft-13:6.6 that "... reading
agents MAY be configured so that unwanted distributions do not get
displayed" - this was part of our grand attempt to make the
Distribution-header more effective by encouraging it to be checked by both
senders and receivers. I have padded this out with other obvious duties
for the reading agent.

4. In draft-13:5.6.1 (the Path-header) it said:

   A relaying agent SHOULD NOT pass an article to another relaying agent
   whose path-identity (or some known alias thereof) already appears in
   the Path-content. ...

   A relaying agent MAY decline to accept an article if its own path-
   identity is already present in the Path-content or if the Path-
   content contains some path-identity whose articles the relaying agent
   does not want, as a matter of local policy.

        NOTE: This last facility is sometimes used to detect and decline
        control messages (notably cancel messages) which have been
        deliberately seeded with a path-identity to be "aliased out" by
        sites not wishing to act upon them.

The first para reflects current practice AIUI, but is the 2nd para
correct? Surely it would be the serving agent that might do that? For
example, if someone "aliases out" some site (e.g. the $alz or cybercancel
conventions used by spam cancellers, is it the final serving agent that
notices that the pseudo-site 'cybercancel' is present (supposing it does
not want to honour those cancels), or is it the final relaying site that
does not send the article to it? I suspect the former, and have moved the
2nd para and the NOTE to serving agents (with copious pseudo comments to
draw attention to the matter). Please advise me what the correct current
practice is.

So finally, here are the diffs:

*** draft-ietf-usefor-usepro-00.unpaged	Thu Sep  2 18:26:27 2004
--- draft-ietf-usefor-usepro-01.unpaged	Mon Sep 20 17:18:31 2004
***************
*** 1,10 ****
  
  INTERNET-DRAFT                               Charles H. Lindsey
  Usenet Format Working Group                  University of Manchester
!                                              August 2004
  
                  News Article Architecture and Protocols
!                    <draft-ietf-usefor-usepro-00.txt>
  
  Status of this Memo
  
--- 1,10 ----
  
  INTERNET-DRAFT                               Charles H. Lindsey
  Usenet Format Working Group                  University of Manchester
!                                              September 2004
  
                  News Article Architecture and Protocols
!                    <draft-ietf-usefor-usepro-01.txt>
  
  Status of this Memo
  
***************
*** 30,36 ****
     The list of Internet-Draft Shadow Directories can be accessed at
     http://www.ietf.org/shadow.html.
  
!    This Internet-Draft will expire in February 2005.
  
  Abstract
  
--- 30,36 ----
     The list of Internet-Draft Shadow Directories can be accessed at
     http://www.ietf.org/shadow.html.
  
!    This Internet-Draft will expire in March 2005.
  
  Abstract
  
***************
*** 112,123 ****
    7.2.  Duties of an Injecting Agent ..............................    0
      7.2.1.  Proto-articles ........................................    0
      7.2.2.  Procedure to be followed by Injecting Agents ..........    0
!   7.3.  Procedure for Forwarding to a Moderator ...................    0
!   7.4.  Duties of a Relaying Agent ................................    0
!     7.4.1.  Example ...............................................    0
!   7.5.  Duties of a Serving Agent .................................    0
!   7.6.  Duties of a Posting Agent .................................    0
!   7.7.  Duties of a Followup Agent ................................    0
    7.8.  Duties of a Moderator .....................................    0
    7.9.  Duties of a Gateway .......................................    0
      7.9.1.  Duties of an Outgoing Gateway .........................    0
--- 112,124 ----
    7.2.  Duties of an Injecting Agent ..............................    0
      7.2.1.  Proto-articles ........................................    0
      7.2.2.  Procedure to be followed by Injecting Agents ..........    0
!     7.2.3.  Procedure for Forwarding to a Moderator ...............    0
!   7.3.  Duties of a Relaying Agent ................................    0
!     7.3.1.  Path-Header Example ...................................    0
!   7.4.  Duties of a Serving Agent .................................    0
!   7.5.  Duties of a Posting Agent .................................    0
!   7.6.  Duties of a Followup Agent ................................    0
!   7.7.  Duties of a Reading Agent .................................    0
    7.8.  Duties of a Moderator .....................................    0
    7.9.  Duties of a Gateway .......................................    0
      7.9.1.  Duties of an Outgoing Gateway .........................    0
***************
*** 1301,1308 ****
     matter of policy to be determined by the administrators of each
     injecting agent, who have the responsibility to ensure that no harm
     arises. In all other circumstances, unintented reinjection is to be
!    avoided (see 7.8).  Nevertheless, in order to preserve the integrity
!    of the network in these special cases, this standard sets out the
     correct way to reinject.
  
     It is usual for an injecting agent to be closely associated with a
--- 1301,1308 ----
     matter of policy to be determined by the administrators of each
     injecting agent, who have the responsibility to ensure that no harm
     arises. In all other circumstances, unintented reinjection is to be
!    avoided (see 7.9).  Nevertheless, in order to preserve the integrity
!    of the network in these special cases, this standard does set out the
     correct way to reinject.
  
     It is usual for an injecting agent to be closely associated with a
***************
*** 1372,1379 ****
        the mandatory headers other than Injection-Date), or which
        contains any header that does not have legal contents.  It SHOULD
        reject any article which contains any header deprecated for
!       Netnews (a-4.2.1).
  
     5. If the article is rejected (for reasons given above, or for other
        formatting errors or matters of site policy) the posting agent
        SHOULD be informed (such as via an NNTP 44x response code) that
--- 1372,1391 ----
        the mandatory headers other than Injection-Date), or which
        contains any header that does not have legal contents.  It SHOULD
        reject any article which contains any header deprecated for
!       Netnews (a-4.2.1).  It SHOULD reject any article whose
!       Newsgroups-header does not contain at least one newsgroup-name for
!       an existing group (as listed by its associated serving agent) and
!       it MAY reject any newsgroup-name which, although syntactically
!       correct, violates a policy restriction established, for some
!       (sub-)hierarchy, by an agency with the appropriate authority
!       (1.2).  Observe that crossposting to unknown newsgroups is not
!       precluded provided at least one of those in the Newsgroups-header
!       is listed.
  
+         NOTE: This ability to reject newsgroup-names in breach of
+         established policy does not extend to relaying agents, though it
+         might be reasonable for posting agents to do it.
+ 
     5. If the article is rejected (for reasons given above, or for other
        formatting errors or matters of site policy) the posting agent
        SHOULD be informed (such as via an NNTP 44x response code) that
***************
*** 1382,1395 ****
  
     6. The Message-ID, Date and From-headers (and their contents) MUST be
        added when not already present (but that situation could not arise
!       during reinjection).
  
     7. The injecting agent MUST NOT alter the body of the article in any
!       way. It MAY, except when reinjecting, add other headers not
!       already provided by the poster, but SHOULD NOT alter, delete, or
!       reorder any existing header, with the specific exception of
!       "tracing" headers such as Injection-Info and Complaints-To, which
!       are to be removed as already mentioned.
  
     8. If the Newsgroups-header contains one or more moderated groups and
        the article does NOT contain an Approved-header, the injecting
--- 1394,1414 ----
  
     6. The Message-ID, Date and From-headers (and their contents) MUST be
        added when not already present (but that situation could not arise
!       during reinjection).  A User-Agent-header MAY be added (or an
!       already present User-Agent-header MAY be augmented) so as to
!       identify the software (e.g. "INN/1.7.2") used by the injecting
!       agent.
  
     7. The injecting agent MUST NOT alter the body of the article in any
!       way (including any change of Content-Transfer-Encoding). It MAY,
!       except when reinjecting, add other headers not already provided by
!       the poster, but SHOULD NOT alter, delete, or reorder any existing
!       header, with the specific exception of "tracing" headers such as
!       Injection-Info and Complaints-To, which are to be removed as
!       already mentioned.  It MAY also, as an interim measure pending
!       widespread adoption of the newly introduced (a-5.5) folding
!       whitespace, reformat the Newsgroups- and any Followup-To-header by
!       removing any such whitespace inserted by the posting agent.
  
     8. If the Newsgroups-header contains one or more moderated groups and
        the article does NOT contain an Approved-header, the injecting
***************
*** 1403,1431 ****
     10.It MUST then prepend the path-identity of the injecting agent and
        a '%' path-delimiter (which serves to separate the pre-injection
        and post-injection regions of the Path-content) to the Path-
!       content; moreover, that path-identity MUST be an FQDN mailable
!       address. This could result in more that one '%' path-delimiter in
        the case of reinjection. See a-5.6.4 for the significance of the
        various path-delimiters.
  
     11.An Injection-Info-header (a-6.19) SHOULD be added, identifying the
        trusted source of the article, and a suitable Complaints-To-header
!       (a-6.20) MAY be added.
  
!    12.The injecting agent MUST then add an An Injection-Date-header (a-
!       5.7) if one is not already present, but it MUST NOT alter, or
!       remove, an already present Injection-Date-header (and likewise
!       SHOULD NOT alter, or remove, an already present NNTP-Posting-
!       Date-header).  Finally, it forwards the article to one or more
!       relaying or serving agents, and the injection process is to be
!       considered complete.
  
! 7.3.  Procedure for Forwarding to a Moderator
  
     An injecting agent forwards ar article to a moderator as follows:
  
     1. It MUST forward it to the moderator of the first (leftmost)
!       moderated group listed in the Newsgroups-header via email (see 7.7
        for how that moderator may forward it to further moderators).
        There are two possibilities for doing this:
  
--- 1423,1467 ----
     10.It MUST then prepend the path-identity of the injecting agent and
        a '%' path-delimiter (which serves to separate the pre-injection
        and post-injection regions of the Path-content) to the Path-
!       content; this SHOULD then be followed by CRLF and WSP if it would
!       otherwise result in a line longer than 79 characters.  The
!       prepended path-identity MUST be an FQDN mailable address (a-
!       5.6.2).  This could result in more that one '%' path-delimiter in
        the case of reinjection. See a-5.6.4 for the significance of the
        various path-delimiters.
  
     11.An Injection-Info-header (a-6.19) SHOULD be added, identifying the
        trusted source of the article, and a suitable Complaints-To-header
!       (a-6.20) MAY be added.  Each injecting agent SHOULD use a
!       consistent form of the Injection-Info-header for all articles
!       emanating from the same or similar origins.
  
!         NOTE: The step above is the only place in which an Injection-
!         Info- or Complaints-To-header is to be created. It follows that
!         these headers MUST NOT be created, replaced, changed or deleted
!         by any other agent (except during reinjection, in which case
!         they will always relate to the latest injection and can, to that
!         extent, be regarded as variant headers).
  
!    12.The injecting agent MUST then add an Injection-Date-header (a-5.7)
!       if one is not already present, but it MUST NOT alter, or delete,
!       an already present Injection-Date-header (and likewise SHOULD NOT
!       alter, or delete, an already present NNTP-Posting-Date-header).
!       Finally, it forwards the article to one or more relaying or
!       serving agents, and the injection process is to be considered
!       complete.
  
+         NOTE: The step above is the only place where an Injection-Date-
+         header is to be created It follows that it MUST NOT subsequently
+         be replaced, changed or deleted by any other agent, even during
+         reinjection.
+ 
+ 7.2.3.  Procedure for Forwarding to a Moderator
+ 
     An injecting agent forwards ar article to a moderator as follows:
  
     1. It MUST forward it to the moderator of the first (leftmost)
!       moderated group listed in the Newsgroups-header via email (see 7.8
        for how that moderator may forward it to further moderators).
        There are two possibilities for doing this:
  
***************
*** 1440,1446 ****
  
        (b)  The article is sent as an email as it stands, with the
             addition of such extra headers (e.g. a To-header) as are
!            necessary for an email.
  
        Although both of these methods have seen use in the past, the
        preponderance of current usage on Usenet has been for method (b)
--- 1476,1483 ----
  
        (b)  The article is sent as an email as it stands, with the
             addition of such extra headers (e.g. a To-header) as are
!            necessary for an email. The existing Message-ID-header SHOULD
!            be retained.
  
        Although both of these methods have seen use in the past, the
        preponderance of current usage on Usenet has been for method (b)
***************
*** 1462,1468 ****
             "news.announce.important" would be emailed to "news-
             [email-masked]".
  
! 7.4.  Duties of a Relaying Agent
  
     A Relaying Agent accepts injected articles from injecting and other
     relaying agents and passes them on to relaying or serving agents
--- 1499,1505 ----
             "news.announce.important" would be emailed to "news-
             [email-masked]".
  
! 7.3.  Duties of a Relaying Agent
  
     A Relaying Agent accepts injected articles from injecting and other
     relaying agents and passes them on to relaying or serving agents
***************
*** 1469,1487 ****
     according to mutually agreed policy. Relaying agents SHOULD accept
     articles ONLY from trusted agents.
  
     A relaying agent processes articles as follows:
  
     1. It MUST establish the trusted identity of the source of the
        article and compare it with the leftmost path-identity of the
        Path-content. If it matches it MUST then prepend its own path-
!       identity and a '/' path-delimiter to the Path-header.  If it does
!       not match then it prepends instead two entries to the Path-
!       content; firstly the true established path-identity of the source
!       followed by a '?'  path-delimiter, and then, to the left of that,
!       its own path-identity followed by a '/' path-delimiter as usual.
!       This prepending of two entries SHOULD NOT be done if the provided
!       and established identities match.  See a-5.6.4 for the
!       significance of the various path-delimiters.
  
          NOTE: In order to prevent overloading, relaying agents should
          not routinely query an external entity (such as a DNS-server) in
--- 1506,1564 ----
     according to mutually agreed policy. Relaying agents SHOULD accept
     articles ONLY from trusted agents.
  
+    An article SHOULD NOT be relayed unless the sending agent has been
+    configured to supply and the receiving agent to receive at least one
+    of the newsgroup-names in its Newsgroups-header and at least one of
+    the distributions in its Distribution-header, if any.  Exceptionally,
+    ALL relaying agents are deemed willing to supply or accept the
+    distribution "world", and NO relaying agent should supply or accept
+    the distribution "local".
+ [That SHOULD has been demoted from a MUST in draft-13. Any objections?]
+ 
+         NOTE: Although it would seem redundant to filter out unwanted
+         distributions at both ends of a relaying link (and it is clearly
+         more efficient to do so at the sending end), many sending sites
+         have been reluctant, historically speaking, to apply such
+         filters (except to ensure that distributions local to their own
+         site or cooperating subnet did not escape); moreover they tended
+         to configure their filters on an "all but those listed" basis,
+         so that new and hitherto unheard of distributions would not be
+         caught. Indeed many "hub" sites actually wanted to receive all
+         possible distributions so that they could feed on to their
+         clients in all possible geographical (or organizational)
+         regions.
+ 
+         Therefore, it is desirable to provide facilities for rejecting
+         unwanted distributions at the receiving end. Indeed, it may be
+         simpler to do so locally than to inform each sending site of
+         what is required, especially in the case of specialized
+         distributions (for example for control messages, such as cancels
+         from certain issuers) which might need to be added at short
+         notice.  The possibility for reading agents to filter
+         distributions has been provided (7.7) for the same reason.
+ 
+    An article SHOULD NOT be relayed if the path-identity of the
+    receiving agent (or some known alias thereof) appears in its Path-
+    header, and the receiving agent MAY detect whether its own path-
+    identity is already present in the Path-content so as to avoid
+    further unnecessary relaying.
+ [See related remarks under serving agents.]
+ 
     A relaying agent processes articles as follows:
  
     1. It MUST establish the trusted identity of the source of the
        article and compare it with the leftmost path-identity of the
        Path-content. If it matches it MUST then prepend its own path-
!       identity and a '/' path-delimiter to the Path-content; this SHOULD
!       then be followed by CRLF and WSP if it would otherwise result in a
!       line longer than 79 characters.  If it does not match then it
!       prepends instead two entries to the Path-content; firstly the true
!       established path-identity of the source followed by a '?'  path-
!       delimiter, and then, to the left of that, its own path-identity
!       followed by a '/' path-delimiter as usual. This prepending of two
!       entries SHOULD NOT be done if the provided and established
!       identities match.  See a-5.6.4 for the significance of the various
!       path-delimiters.
  
          NOTE: In order to prevent overloading, relaying agents should
          not routinely query an external entity (such as a DNS-server) in
***************
*** 1499,1504 ****
--- 1576,1584 ----
  
     4. It SHOULD reject any article whose optional headers (section a-6)
        do not have legal contents.
+ [Is that too strong? Are relaying agents really expected to check
+ headers in that detail? I suggest s/SHOULD/MAY/. Even the MUST in Step 4
+ for mandatory headers might be demoted to SHOULD.]
  
     5. It SHOULD reject any article that has already been sent to it (a
        database of message identifiers of recent messages is usually kept
***************
*** 1508,1526 ****
        cancel message (or an equivalent Supersedes-header) issued by its
        poster or by some other trusted entity.
  
     7. It MAY reject any article without an Approved-header posted to
        newsgroups known to be moderated (this practice is strongly
        recommended, but the information necessary to do so may not be
        available to all agents).
  
!    8. Finally, it passes articles which match mutually agreed criteria
!       on to neighbouring relaying and serving agents. However, it SHOULD
!       NOT forward articles to sites whose path-identity is already in
!       the Path-header.
  
!         NOTE: It is usual for relaying and serving agents to restrict
!         the Newsgroups, Distributions, age and size of articles that
!         they wish to receive.
  
     If the article is rejected as being invalid, unwanted or unacceptable
     due to site policy, the agent that passed the article to the relaying
--- 1588,1604 ----
        cancel message (or an equivalent Supersedes-header) issued by its
        poster or by some other trusted entity.
  
     7. It MAY reject any article without an Approved-header posted to
        newsgroups known to be moderated (this practice is strongly
        recommended, but the information necessary to do so may not be
        available to all agents).
  
!    8. It MAY delete any Xref-header that is present.
  
!    9. Finally, it passes the articles on to neighbouring relaying and
!       serving agents.
  
     If the article is rejected as being invalid, unwanted or unacceptable
     due to site policy, the agent that passed the article to the relaying
***************
*** 1530,1542 ****
     external entity that an article was not relayed UNLESS that external
     entity has explicitly requested that it be informed of such errors.
  
     Relaying agents MUST NOT alter, delete or rearrange any part of an
!    article expect for headers designated as variant (a-4.2.5.3).
  
! 7.4.1.  Example
  
        Path: foo.isp.example/
           foo-server/bar.isp.example?10.123.12.2/old.site.example!
           barbaz/baz.isp.example%dialup123.baz.isp.example!not-for-mail
--- 1608,1633 ----
     external entity that an article was not relayed UNLESS that external
     entity has explicitly requested that it be informed of such errors.
  
     Relaying agents MUST NOT alter, delete or rearrange any part of an
!    article expect for headers designated as variant (a-4.2.5.3).  In
!    particular
  
!      o they MUST NOT create or augment a User-Agent-header in order to
!        identify themselves;
!      o they MUST NOT rewrite the Newsgroups-header in any way, even if
!        some supposedly non-existent newsgroup is included;
!      o they MUST NOT refold any header (i.e. they must pass on the
!        folding as received), even to remove FWS from a Newsgroups-
!        header;
!      o they MUST NOT alter the Date-header or the Injection-Date-header;
!      o they MUST NOT delete any unrecognized header whose header-name is
!        syntactically correct (whether or not it is registered with IANA
!        [RFC 3864]);
!      o they MUST NOT change the Content-Transfer-Encoding of the body or
!        any body part.
  
+ 7.3.1.  Path-Header Example
+ 
        Path: foo.isp.example/
           foo-server/bar.isp.example?10.123.12.2/old.site.example!
           barbaz/baz.isp.example%dialup123.baz.isp.example!not-for-mail
***************
*** 1576,1582 ****
          fold the line, on the grounds that it seemed to be getting a
          little too long.
  
! 7.5.  Duties of a Serving Agent
  
     A Serving Agent takes an article from a relaying or injecting agent
     and files it in a "news database". It also provides an interface for
--- 1667,1673 ----
          fold the line, on the grounds that it seemed to be getting a
          little too long.
  
! 7.4.  Duties of a Serving Agent
  
     A Serving Agent takes an article from a relaying or injecting agent
     and files it in a "news database". It also provides an interface for
***************
*** 1585,1594 ****
     article-locator (usually in the form of a decimal number - see a-
     6.16).
  
!    A serving agent MUST maintain a list showing the moderation status
!    (see 6.2.1) of the newsgroups it stores in the news database, and
!    SHOULD include in that list all groups likely to be crossposted to
!    from those groups (e.g. all other groups in the same hierarchy(ies)).
  
          NOTE: Since control messages are often of interest, but should
          not be displayed as normal articles in regular newsgroups, it is
--- 1676,1686 ----
     article-locator (usually in the form of a decimal number - see a-
     6.16).
  
!    A serving agent MUST maintain a list of the newsgroups it stores in
!    its news database showing the moderation status of each one (see
!    6.2.1), and SHOULD include in that list all groups likely to be
!    crossposted to from those groups (e.g. all other groups in the same
!    hierarchy(ies)).
  
          NOTE: Since control messages are often of interest, but should
          not be displayed as normal articles in regular newsgroups, it is
***************
*** 1596,1604 ****
          newsgroup named "control" or in a pseudo-newsgroup in a sub-
          hierarchy under "control." (e.g. "control.cancel").
  
     A serving agent processes articles as follows:
  
!    1.  It MUST establish the trusted identity of the source of the
        article and modify the Path-header as for a relaying agent (7.3).
  
     2. It MUST examine the Injection-Date-header (or, if that is absent,
--- 1688,1715 ----
          newsgroup named "control" or in a pseudo-newsgroup in a sub-
          hierarchy under "control." (e.g. "control.cancel").
  
+    A serving agent MAY decline to accept an article if its own path-
+    identity is already present in the Path-content or if the Path-
+    content contains some path-identity whose articles the serving agent
+    does not want, as a matter of local policy.
+ [That has been changed from a-5.6.1 in previous drafts, where it said "A
+ relaying agent MAY decline...". That seemed plain wrong to me. It is
+ fine for a relaying agent to ignore articles which have apparently
+ already passed through it (and I still say that), but surely it is for
+ serving agents to "reject" for policy reasons, and to which the NOTE
+ below would apply.  Comments?]
+ 
+         NOTE: This last facility is sometimes used to detect and decline
+         control messages (notably cancel messages) which have been
+         deliberately seeded with a path-identity to be "aliased out" by
+         sites not wishing to act upon them.
+ [Again, is this "aliasing out" usually detected by the serving agent, or
+ does it more usually work because the previous relaying agent will never
+ have sent it in the first place?]
+ 
     A serving agent processes articles as follows:
  
!    1. It MUST establish the trusted identity of the source of the
        article and modify the Path-header as for a relaying agent (7.3).
  
     2. It MUST examine the Injection-Date-header (or, if that is absent,
***************
*** 1612,1618 ****
        header that does not have legal contents.
  
     4. It SHOULD reject any article that has already been sent to it (a
!       database of message identifiers of recent messages is usually kept
        and matched against).
  
     5. It SHOULD reject any article that matches an already received
--- 1723,1729 ----
        header that does not have legal contents.
  
     4. It SHOULD reject any article that has already been sent to it (a
!       database of message identifiers of recent articles is usually kept
        and matched against).
  
     5. It SHOULD reject any article that matches an already received
***************
*** 1627,1634 ****
  
     8. Finally, it stores the article in its news database.
  
! 7.6.  Duties of a Posting Agent
  
     A Posting Agent is used to assist the poster in creating a valid
     proto-article and forwarding it to an injecting agent.
  
--- 1738,1753 ----
  
     8. Finally, it stores the article in its news database.
  
!    Serving agents MUST NOT create new newsgroups simply because an
!    unrecognized newsgroup-name occurs in a Newsgroups-header (see a-
!    7.2.1 for the correct method of newsgroup creation).
  
+    Serving agents MUST NOT alter, delete or rearrange any part of an
+    article in any other way. The list of particular cases given for
+    relaying agents (7.3) applies here also.
+ 
+ 7.5.  Duties of a Posting Agent
+ 
     A Posting Agent is used to assist the poster in creating a valid
     proto-article and forwarding it to an injecting agent.
  
***************
*** 1641,1648 ****
     attempt to post an article which cancels or Supersedes another
     article of which the poster is not the author.
  
- 7.7.  Duties of a Followup Agent
  
     A Followup Agent is a special case of a posting agent, and as such is
     bound by all the posting agent's requirements. Followup agents MUST
     create valid followups and are subject to special requirements
--- 1760,1771 ----
     attempt to post an article which cancels or Supersedes another
     article of which the poster is not the author.

+ 7.6.  Duties of a Followup Agent
+ 
     A Followup Agent is a special case of a posting agent, and as such is
     bound by all the posting agent's requirements. Followup agents MUST
     create valid followups and are subject to special requirements
***************
*** 1709,1714 ****
--- 1832,1856 ----
        Followup agents SHOULD NOT attempt to send email to any address
        ending in ".invalid".
  
+ 7.7.  Duties of a Reading Agent
+ 
+    A reading agent downloads articles from a serving agent, as directed
+    by the reader, and displays them (or processes them in some other
+    manner).  The article as displayed MUST be identical to the article
+    as originally posted, subject only to limitations of the display
+    device (such as availability of charsets, etc.). It MUST provide
+    facilities for decoding any Content-Transfer-Encodings, encoded-
+    words, etc., but SHOULD also have the capability to show the article
+    exactly as received.
+ 
+    It MAY present lists of articles available for display, and MAY
+    structure those lists so as to show the relationships between the
+    articles, as determined by the References-, Subject-, Date- and
+    other-headers (see [USEAGE] for some usual methods of doing this). It
+    MAY also be configured so that unwanted distributions (a-6.6) are
+    ignored.
+ 
+ 
  7.8.  Duties of a Moderator
  
     A Moderator receives news articles by email, decides whether to
***************
*** 1898,1903 ****
--- 2039,2049 ----
     4. Netnews control messages should not be gated to another medium
        unless they would somehow be meaningful in that medium.
  
+    5. Changes MAY be made to the Content-Transfer-Encoding of some or
+       all parts of the body, and even to the charsets specified in
+       encoded-words or in Content-Type-headers, but such changes SHOULD
+       NOT be made unless absolutely necessary.
+ 
  7.9.2.  Duties of an Incoming Gateway
  
     The incoming gateway has the serious responsibility of ensuring that
***************
*** 2297,2302 ****
--- 2447,2455 ----
     [RFC 2822] P. Resnick, "Internet Message Format", RFC 2822, April
          2001.
  
+    [RFC 3864] G. Klyne, M. Nottingham, and J. Mogul, "Registration
+         procedures for message header fields", RFC 3864.
+ 
     [RFC 850] Mark R. Horton, "Standard for interchange of Usenet
          messages", RFC 850, June 1983.
  
***************
*** 2341,2348 ****
     Comments on this draft should preferably be sent to the mailing list
     of the Usenet Format Working Group at
  
-         [email-masked]
-         shortly to be replaced by
          [email-masked].
  
  Appendix A.1 - A-News Article Format
--- 2494,2499 ----
***************
*** 2494,2496 ****
--- 2644,2673 ----
  Appendix C - Change Log
  
  [This Appendix is to be removed prior to final publication.]
+ 
+    For version 01
+ 
+    1    Numerous texts describing protocol features related to
+         particular headers in parts of [ARTICLE] which were destined to
+         become part of [USEFOR] have been moved to appropriate locations
+         within section 7 of this document. Such revised texts will be
+         found in sections
+         7.2.2 Steps 4, 6, 7, 10, 11, 12;
+         7.2.3 Step 1(b);
+         7.3 introductory paragraphs, Steps 1, 4, 8, 9, and some final
+         paragraphs;
+         7.4 introductory and final paragraphs;
+         7.9.1 Step 5.
+ 
+    2    A section on "Duties of a Reading Agent" (7.8) has been added.
+ 
+    3    Some demotions MUST -> SHOULD -> MAY, as noted in pseudo-
+         comments, have been made or proposed in sections
+         7.3
+         7.3 Step 4.
+ 
+    4    Part of the procedure for examining Path-headers by relaying
+         agents has been moved to serving agents, as explained in
+         pseudo-comments in section 7.4.
+ 
+    5    Some renumbering of sections and minor textual clarifications.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Russ Allbery <[email-masked]>
Date: Sun, 26 Sep 2004 17:12:33 -0700

Charles Lindsey <[email-masked]> writes:

> check everything), and I suspect something nearer to SHOULD and MAY
> would be better. Comments?

A relaying agent MUST check the legality of all headers that it needs to
use to do its job, namely at least Path, Date, Message-ID, Control if
present, Injection-Date if present, and probably also Newsgroups.
Demoting the rest to a SHOULD seems reasonable to me.  Thinking about it,
I see no reason to distinguish between mandatory and optional headers
here, although perhaps I'm missing something.

> 3. I have added a section on "Duties of a Reading Agent". The prime
> reason for this was to find a home for the bit in draft-13:6.6 that
> "... reading agents MAY be configured so that unwanted distributions do
> not get displayed" - this was part of our grand attempt to make the
> Distribution-header more effective by encouraging it to be checked by
> both senders and receivers. I have padded this out with other obvious
> duties for the reading agent.

I think both that comment and really this entire section should simply be
striken in its entirety.  It seems completely unnecessary to me.

> correct?

The second paragraph and the NOTE are pointless; relaying agents can
reject anything they want based on local policy, and there's no reason to
single this out in particular since it doesn't do anything particularly
useful.  I'd drop both entirely.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 27 Sep 2004 11:58:20 GMT

>here, although perhaps I'm missing something.

OK, an explicit list makes sense (leaves From and Subject out of it). I am
not so sure about Control; surely that is for serving agents, not relaying
agents (and the serving agent will probably just hand it over to whatever
subsystem deals with control messages and let that check as it sees fit)?
Also, one could omit Date if Injection-Date is present.

As for the rest, why not demote to MAY rather than SHOULD? Do current
relaying agents routinely check all the lesser known headers, especially
if they are just transit agents?

>striken in its entirety.  It seems completely unnecessary to me.

Well the bit about distributions was in all our earlier drafts, so I can't
just throw it away without some WG discussion. I could be persuaded that
it belongs in USEAGE.

However, I have now found another bit in draft-13 which really belongs in
USEPRO. In 6.21.2 it said:

   Reading agents SHOULD NOT, unless explicitly configured otherwise,
   act automatically on Application types which could change the state
   of that agent (e.g. by writing or modifying files), except in the
   case of those prescribed for use in control messages (7.2.1.2 and
   7.2.4.1).

So that reads like a duty of a reading agent, though maybe it is a
security consideration.

Anyway, I will leave that section in the draft for now until I hear more
opinions from this list.

>useful.  I'd drop both entirely.

I could easily be persuaded that the second paragraph and NOTE are
supefluous, but I would still be interested to hear whether relaying
agents actually do it or not.

But the 1st paragraph seems to be an important part of the protocol, and
part of the machinery for avoiding loops.

---

From: Henry Spencer <[email-masked]>
Date: Mon, 27 Sep 2004 12:36:13 -0400 (EDT)

On Mon, 27 Sep 2004, Charles Lindsey wrote:
> >present, Injection-Date if present, and probably also Newsgroups...
> 
> OK, an explicit list makes sense (leaves From and Subject out of it).

In the C News days, persistent pressure eventually persuaded us that our
relaying stuff really should check for at least the presence of *all*
mandatory headers -- that it served nobody's interests for blatantly
malformed articles to propagate.  If memory serves, we did not actually do
any checking on the From content beyond verifying its presence. 

Moreover, we did not really do a syntax check on the Path content.  Date
and Message-ID had to be parsable, as did Newsgroups (although our parsing
of that was very simple and would not catch some subtle goofs).

> if they are just transit agents?

Certainly C News didn't.

> >>    identity is already present in the Path-content...
> 
> ...I would still be interested to hear whether relaying

If I recall correctly, C News examined Path only on transmission, not on
reception.

> part of the machinery for avoiding loops.

Indeed, the first paragraph remains necessary.  It's primarily an efficiency
hack rather than loop prevention -- the message ID check is what really
breaks loops -- but it's an important one.

                                                          Henry Spencer
                                                       [email-masked]

---

From: Russ Allbery <[email-masked]>
Date: Mon, 27 Sep 2004 13:08:07 -0700

Charles Lindsey <[email-masked]> writes:
> Russ Allbery <[email-masked]> writes:

>> A relaying agent MUST check the legality of all headers that it needs
>> to use to do its job, namely at least Path, Date, Message-ID, Control
>> if present, Injection-Date if present, and probably also Newsgroups.
>> Demoting the rest to a SHOULD seems reasonable to me.  Thinking about
>> it, I see no reason to distinguish between mandatory and optional
>> headers here, although perhaps I'm missing something.

> OK, an explicit list makes sense (leaves From and Subject out of it).

Right.  Those are the two mandatory headers that have no real impact on
relaying.  Henry's point is well-taken in that it might be worthwhile
doing some sort of cursory check just to keep from propagating trash, but
on the other hand, major servers don't do this now and Usenet hasn't
broken.  To me, that argues against a MUST and towards a SHOULD.

> I am not so sure about Control; surely that is for serving agents, not
> relaying agents (and the serving agent will probably just hand it over
> to whatever subsystem deals with control messages and let that check as
> it sees fit)?

No, see the special propagation rules for newgroup and rmgroup control
messages.  Relaying agents have to know about that.

> Also, one could omit Date if Injection-Date is present.

Yes, but beware of backward compatibility here; not everyone will use
Injection-Date by preference.  Plus, it's more complicated to say that;
it's easier to just check both of them.

> if they are just transit agents?

INN does, yes.  Remember that for a relaying agent, if it accepts the
message, it also passes it on to other sites.  I think that one SHOULD
avoid propagating known-broken messages.

However, I don't feel strongly enough about this to argue it at length and
will happily go along with whatever the group consensus is.

> However, I have now found another bit in draft-13 which really belongs
> in USEPRO. In 6.21.2 it said:

> So that reads like a duty of a reading agent, though maybe it is a

Yes, I think this is a security consideration and would rather see it
there than in a separate duties section.  This isn't a statement about
what a reading agent should do, but rather what a reading agent shouldn't
do.

I really think the Distribution thing is pointless.  Is anyone aware of a
reading agent that actually does that?  Would anyone reading this list
actually use that feature, as opposed to just reading the relevant
regional group?

> agents actually do it or not.

INN doesn't.

> part of the machinery for avoiding loops.

Oh, certainly, I have no objections at all to the first paragraph and
indeed believe it should stay.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 28 Sep 2004 11:24:15 GMT

>broken.  To me, that argues against a MUST and towards a SHOULD.

Two points, arising from Henry:
1. MUST/SHOULD check for presence of mandatory headers.
2. MUST/SHOULD check for legality of listed headers, but only to the
extent needed to ensure they work. For example, you told us recently that
you only parse the Date header until you have seen enough to construct a
valid date; any trailing rubbish is not looked at. Again, for newsgroups,
you are only concerned to see whether anything in your active file is
included; you don't care if some malformed names of non-existent groups
are present, so long as your parser can manage to skip over them.

Is that OK, assuming I can invent a wording for "parse to the extent
necessary for proper operation". And in that case, please choose between
MUST and SHOULD.

>messages.  Relaying agents have to know about that.

Yes, but that does not impose any requirement to check the Control header
for legality, which is what we are discussing here. At most, it means
recognizing that a Control header is present, and I think the existing
wording (now in USEPRO section 6) says what needs to be said.

However, upon looking at that wording, it does seem to say more than I
intended. In draft-13 it said (and the new USEFOR needs to say) that
the Newsgroups header of a control message needs to include the
newsgroup-name affected, and anything else that will help to steer it to
where it is wanted. But if I only take the big-8 plus uk.* on my system, I
don't want to be flooded out with newgroup messages for jp.*.

Now the other paragraph from draft-13, which now appears in USEPRO section
6, says:

   Relaying Agents MUST propagate all control messages regardless of
   whether or not they are recognized or processed locally.

That was intended to say that, even if your local policy is to ignore all
newgroup message and all cancels, you are still supposed to propagate
control messages normally for the benefit of neighbouring sites who are
not so fussy. It certainly was not meant to say that you were to propagate
them to peers who did not take the affected hierarchies at all.

If it is agreed that what I have just said is what we intend, then I need
to review that wording, especially as the two paragraphs which were once
together are now going to be in different documents.

>it's easier to just check both of them.

OK, I shall not bother if it uses too much verbiage.

>avoid propagating known-broken messages.

Yes, but if you don't look, it isn't "known". My inclination is to take
Henry's lead and demote to MAY. I am pretty sure that transit relayers
will not be making such checks.

>what a reading agent should do, but rather what a reading agent shouldn't

Agreed.

>actually use that feature, as opposed to just reading the relevant

We agreed long ago to attempt to make the Distribution header more
effective by making sure it got looked at at the proper places. So that
action by reading agents was something we wanted to encourage to happen,
rather than something that was happening currently (hence the MAY). But I
am happy to move that particular bit to USEAGE.

>INN doesn't.

And Henry says CNews doesn't, and I have just tried it on my own CNews and
it didn't.

So am I right in thinking that if I don't want to see cancels from all the
regular spam cancellers, I have got to instruct my upstream peers to
register 'cybercancel' as an alias for my site in their sys files?

---

From: Russ Allbery <[email-masked]>
Date: Tue, 28 Sep 2004 11:31:17 -0700

>> doing some sort of cursory check just to keep from propagating trash,
>> but on the other hand, major servers don't do this now and Usenet
>> hasn't broken.  To me, that argues against a MUST and towards a SHOULD.

> necessary for proper operation". And in that case, please choose between

I am very leery of statements like "parse to the extent necessary for
proper operation" in Usenet standards, since that concept cannot be
tightly specified.

I don't believe there's any reasonable middle point.  Either we're going
to require that relay agents detect and discard some set of invalid
messages, or we're not going to do that and leave it up to each individual
relay agent.  Currently, existing practice is the latter -- relay agents
accept and propagate all sorts of syntactically broken messages.  If
that's acceptable to people, then I think we should just say SHOULD check
or MAY check headers and leave it at that.

That attitude is incompatible with what Henry was saying about making sure
that the article is at least basically formed, I think.  I don't see how
it makes any sense to be sure mandatory headers are present without making
sure that they contain usable data (and the only definition of usable data
that makes any sense to me is compliant and parsable data).

> Yes, but that does not impose any requirement to check the Control
> header for legality, which is what we are discussing here.

It requires that the Control message be parsed to the degree that the type
of control message and its parameters be recognized, which inherently
requires validating its syntax to some degree.  Of course, validating the
syntax doesn't mean that one has to reject invalid messages; one can just
treat them as if they don't have a Control header at all.  Is that what
you're saying?

> At most, it means recognizing that a Control header is present,

You have to at least know whether it's a newgroup or rmgroup control
message.

> intended. In draft-13 it said (and the new USEFOR needs to say) that the
> Newsgroups header of a control message needs to include the
> where it is wanted.

Currently, INN can optionally ignore the Newsgroups header is ignored in
newgroup and rmgroup control messages by INN; such messages are treated as
if they were posted only to the affected newsgroup (and then propagated
using the special propagation rules for control messages, which ignore
newsgroup existence).  I'm trying to remember previous discussions we had
on this topic; did we decide that was wrong?  It doesn't make a lot of
sense to me right now that it's an option in INN, so it would be nice to
standardize one type of behavior or another.

RFC 1036 is silent on the subject and Son-of-1036 addresses it only in a
note:

    Third, the relayer MUST determine whether any of the article's
    newsgroups are "subscribed to" by the host, i.e. fit a description of
    what hierarchies or newsgroups the site wants to receive.

         NOTE:  This check is significant because information on what
         newsgroups a relayer wishes to receive is often stored at its
         neighbors, who may not have up-to-date information or may
         simplify the rules for implementation reasons.  As a hedge
         against the possibility of missed or delayed newgroup control
         messages, relayers may wish to observe a notion of a newsgroup
         subscription that is independent of the list of newsgroups
         actually known to the relayer.  This would permit reception and
         relaying of articles in newsgroups that the relayer is not (yet)
         aware of, subject to more general criteria indicating that they
         are likely to be of interest.

which is not specific to control messages.  (This is interesting to me,
since I thought that special propagation of newgroup and rmgroup control
messages was long-standing established practice, but apparently it isn't
as much as I thought it was.)

> Now the other paragraph from draft-13, which now appears in USEPRO
> section 6, says:

> That was intended to say that, even if your local policy is to ignore
> all newgroup message and all cancels, you are still supposed to
> propagate control messages normally for the benefit of neighbouring
> sites who are not so fussy. It certainly was not meant to say that you
> were to propagate them to peers who did not take the affected
> hierarchies at all.

I don't know where this sentence came from and it's never made any sense
to me.  It's on my list of language that I believe should be striken from
the USEPRO document.

> If it is agreed that what I have just said is what we intend,

Certainly not.  Relay agents are never required to propagate anything
whatsoever.  Instead, the special propagation rules for newgroup and
rmgroup control messages need to be specifically described.

> OK, I shall not bother if it uses too much verbiage.

It will add unnecessary complexity even if there's a short way of saying
it.  Please, don't bother.  :)

> will not be making such checks.

INN does, as I've said several times, although in a somewhat limited
capacity (in that it's somewhat particular about what it cares about and
what it doesn't care about).  I'm hoping to clean that up over time.

> effective by making sure it got looked at at the proper places.

Note that I do not consider myself to be part of this "we" at this time
and do not agree that we have consensus to do this.

> So am I right in thinking that if I don't want to see cancels from all
> the regular spam cancellers, I have got to instruct my upstream peers to

I have no idea how CNews works in this area, but INN allows you to list
Path identities that, if found in incoming articles, should cause those
articles to be rejected.  This configuration option is entirely
independent of your own Path identity and is intended specifically for
aliasing out sites or pseudosites.  This facility has been in INN since
1.0, I believe.

---

From: Henry Spencer <[email-masked]>
Date: Tue, 28 Sep 2004 14:58:19 -0400 (EDT)

> that makes any sense to me is compliant and parsable data).

The feedback we got, way back when, was that this distinction *was*
valuable -- that merely demanding the presence of the mandatory headers
eliminated a class of problems.  Since I never knew the details of what
those problems were, I can't say whether they're still relevant.  On the
whole, I rather doubt it.

My feeling is that we should *encourage* relaying agents to do as much
checking -- beyond the minimum they need to function -- as they think they
can manage, but given that they historically have serious performance
constraints, we should refrain from *demanding* extra work from them
unless it is clearly vital. 

If for no other reason, I think this is preferable because it avoids the
whole question of exactly which things need to be checked and how
thoroughly. 

                                                          Henry Spencer
                                                       [email-masked]

---

From: Russ Allbery <[email-masked]>
Date: Tue, 28 Sep 2004 12:04:06 -0700

> whole, I rather doubt it.

Oh, okay.

> checking -- beyond the minimum they need to function -- as they think
> they can manage, but given that they historically have serious
> performance constraints, we should refrain from *demanding* extra work
> from them unless it is clearly vital.

> whole question of exactly which things need to be checked and how

That sounds like an argument for SHOULD (that's the encouragement) check
the syntactical validity of the article, without distinctions between
mandatory and optional headers.  Does that match what you're thinking?

---

From: Henry Spencer <[email-masked]>
Date: Tue, 28 Sep 2004 15:31:45 -0400 (EDT)

On Tue, 28 Sep 2004, Russ Allbery wrote:
> > from them unless it is clearly vital...
> 

I'm a little reluctant to use SHOULD for any of this, because that keyword
is generally understood to mean that the action in question is nearly
mandatory, that almost everyone is expected to do it but there are special
situations where it is impractical.  This doesn't seem right for something
where incomplete implementations are common and expected to remain so.

I think the right choice is either to say MAY with some accompanying words
urging people to do it, or to say SHOULD but accompany it with an escape
hatch like "to the extent that performance requirements permit".

                                                          Henry Spencer
                                                       [email-masked]

---

From: Russ Allbery <[email-masked]>
Date: Tue, 28 Sep 2004 13:16:16 -0700

Henry Spencer <[email-masked]> writes:

> I think the right choice is either to say MAY with some accompanying
> words urging people to do it, or to say SHOULD but accompany it with an
> escape hatch like "to the extent that performance requirements permit".

Hm.  Okay, yeah, that works for me.

---

From: Bill Davidsen <[email-masked]>
Date: Tue, 28 Sep 2004 17:01:30 -0400

> striken in its entirety.  It seems completely unnecessary to me.

Clearly some agent is responsible for generating the required headers, 
beyond the enumeration of mandatory headers need anything else be mentioned?

-- 
    -bill davidsen ([email-masked])
"The secret to procrastination is to put things off until the
  last possible moment - but no longer"  -me

---

From: Bill Davidsen <[email-masked]>
Date: Tue, 28 Sep 2004 17:01:30 -0400

---

From: Russ Allbery <[email-masked]>
Date: Tue, 28 Sep 2004 14:07:20 -0700

> beyond the enumeration of mandatory headers need anything else be

It's probably worthwhile to say something about the breakdown of duties
between the posting agent and the injecting agent, but we do already do
that (in those sections).

---

From: Bill Davidsen <[email-masked]>
Date: Tue, 28 Sep 2004 17:28:15 -0400

> hatch like "to the extent that performance requirements permit".

Another case where SHOULD is too strong and MAY is too weak. Perhaps we 
can split this hair and say SHOULD check that mandatory headers are 
present and MAY check the validity of any recognized header?

---

From: Bill Davidsen <[email-masked]>
Date: Tue, 28 Sep 2004 17:21:21 -0400

>>here, although perhaps I'm missing something.

Depending on the relaying agent, I would expect the newsgroups and path 
headers to be checked, since those are needed for delivery. Beyond that 
anything is optional, control means nothing since group names need not 
be validated beyond seeing if we accept them and some peer wants them. 
And date? Not my exchange servers, I don't know your policy on age and 
it takes less time to send something old than make a decision on it. The 
correct age of most posts today is measured in ms, anything older is 
down in the noise.

> agents actually do it or not.

I could be convinced that checking that required headers are present, 
but the number of cases where that is not the case is vanishingly small.
> part of the machinery for avoiding loops.

In many ways an exchange server is like a router, doing no more checking 
that is required. Some do much more, some are not just doing exchange. I 
would rather not see a lot of mandated checking. Clearly a relaying 
agent may be more than an exchange server, and may do many thing beyond 
routing posts, like handling readers, spam filtering, etc. But it need 
not be, and I don't see any reason the standard should be changed to 
suggest otherwise.

---

From: Henry Spencer <[email-masked]>
Date: Tue, 28 Sep 2004 17:38:54 -0400 (EDT)

On Tue, 28 Sep 2004, Bill Davidsen wrote:
> it takes less time to send something old than make a decision on it...

Note that history-based loop detection does not work if you don't verify
that the date of the article is within your history list.

Not doing loop detection is acceptable only in tightly controlled
situations. 

                                                          Henry Spencer
                                                       [email-masked]

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 29 Sep 2004 14:19:03 GMT

>proper operation" in Usenet standards, since that concept cannot be

Yes, I would be too. It would need careful wording. But it may not be
necessary, since the protocol requires you to do various things which
obviously involve looking at the Path and Newsgroups headers, and if you
find that those headers are unintelligible as regards what you need to do
with them, then you are going to barf anyway without being told to.

>         are likely to be of interest.

That NOTE has never made its way into our drafts, and it is not clear
quite what it is suggesting. I think everyone accepts that a message
posted to a group two minutes after that group was newgrouped is going to
fail to appear in many places, though the flooding algorithm should get it
to most of them eventually.

>as much as I thought it was.)

No, that has never been raised in this group before, to my knowledge.

We did discuss the text that says control messages need to have a
Newsgroups-header that will steer them to the proper places, and the text
you see contains some tweaks arising from that.

Now if INN has implemented some optimizations that will further improve the
propagation of control messages, then good luck to it, but it has never
been suggested that they should form part of this standard.

>to me.  It's on my list of language that I believe should be striken from

I wrote it, and it was intended to say what I said in the previous
paragraph. But its meaning is not clear, so I have rewritten it as follows
(I have also included the preceding paragraph so you get the full
context).

   The descriptions below set out REQUIREMENTS to be followed by sites
   that receive control messages and choose to honour them. However,
   nothing in these descriptions should be taken as overriding the right
   of any such site, in accordance with its local policy, to refuse to
   honour any particular control message, or to refer it to an
   administrator for approval (either as a class or on a case-by-case
   basis).

   Even if relaying agents do not, as a matter of local policy, intend
   to honour certain control messages, they MUST still propagate them in
   the normal manner.
[The inclusion of that paragraph still remains under discussion.]

Hopefully, it is now clearer what it means, so we can proceed to discuss
whether we need it at all. Comments?

>whatsoever.

Sure, but there is a general Caveat somewhere covering that, so we do not
need to repeat it every time we say "Relaying agents MUST ...".

>and do not agree that we have consensus to do this.

It was a long way back, but there was certainly a consensus at the time to
try and make Distributions work.

>aliasing out sites or pseudosites.  This facility has been in INN since

Ah! I had always understood, since the matter was first raised on the
news.admin.net-abuse groups, that it came out of the normal processing of
the Path header rather than on special code in transport agents.

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 29 Sep 2004 14:26:38 GMT

>present and MAY check the validity of any recognized header?

Yes, that is more or less the position I had come to following this
exchange. My draft text now reads as follows:

   3.   It SHOULD reject any article that does not have the correct
        mandatory headers (section a-5) present.

   4.   It MAY reject any article whose headers do not have legal
        contents.

Any advance on that?

---

From: Henry Spencer <[email-masked]>
Date: Wed, 29 Sep 2004 14:37:49 -0400 (EDT)

>         mandatory headers (section a-5) present.

Why "correct"?  That seems redundant, and it is bound to make people
wonder whether it has some special significance.  I'd take that word out.

Otherwise, this seems okay.

                                                          Henry Spencer
                                                       [email-masked]

---

From: Russ Allbery <[email-masked]>
Date: Wed, 29 Sep 2004 11:32:30 -0700

>> since I thought that special propagation of newgroup and rmgroup
>> control messages was long-standing established practice, but apparently
>> it isn't as much as I thought it was.)

> No, that has never been raised in this group before, to my knowledge.

I've raised it in this group in the past, but generally as part of other
discussions of control message handling.

> Newsgroups-header that will steer them to the proper places, and the
> text you see contains some tweaks arising from that.

Control messages should not, in general, be crossposted to other
newsgroups than they one that they affect.  David Lawrence was the one who
first taught me the problems caused by doing so.  For example, when people
form a habit of crossposting control messages to alt.config, someone
limiting what hierarchies in alt.* they receive, they still ends up
getting control messages for the hierarchies they don't want.

Unfortunately, this bad habit is sufficiently established that I doubt we
can really get people to stop (although relay agents could be configured
to ignore the Newsgroups header entirely).  I don't think it's a good idea
to encourage it, though (but I remember the previous discussion and
recognize that documenting it is to a degree documenting existing
practice).

However, relay agents that check the Newsgroups header against a list of
valid groups *do* need to propagate control messages as if the group they
were affecting existed.  *That* behavior does need to be specified in the
standard.  This is independent of the question of whether one should
ignore the Newsgroups header on newgroup and rmgroup control messages
entirely.  I'm surprised that it's not in son-of-1036 and doesn't appear
to be in CNews, but perhaps both predate the understanding that this was
the right thing to do.  This has been implemented in INN since the 1.0
release in 1991.

            if (IsNewgroup && Approved) {
                /* Newgroup being sent to a group that doesn't exist.
                 * Assume it is being sent to the group being created,
                 * and send the group to all sites that would get the
                 * group if it were created. */
                ARTsendnewgroup(*groups);
                Accepted = TRUE;
            }

(By INN 1.4 in 1993, and possibly earlier, this logic had been corrected
to include rmgroups as well.)

If relay agents don't implement this behavior, booster rmgroups are
futile because they won't propagate past the first host that already
removed the group and newgroups won't propagate through relay agents that
don't act on them immediately and require that newsgroups exist before
propagating messages.  This is clearly wrong.

> Now if INN has implemented some optimizations that will further improve
> the propagation of control messages, then good luck to it, but it has
> never been suggested that they should form part of this standard.

Well, I don't believe that's correct, but there's no point in arguing
about past history.  I'm making that statement now.  This should be part
of the standard.

> paragraph. But its meaning is not clear, so I have rewritten it as
> follows (I have also included the preceding paragraph so you get the
> full context).

> [The inclusion of that paragraph still remains under discussion.]

This is still not correct.  This doesn't help with the issue addressed
above in propagation of newgroup and rmgroup control messages and simply
contradicts other portions of the document in a fashion that leaves one
wondering what it's supposed to mean.  Please, remove this language and
just explain that newgroup and rmgroup control messages should always be
propagated as if the group they're affecting exists, even if it does not.

> Sure, but there is a general Caveat somewhere covering that, so we do
> not need to repeat it every time we say "Relaying agents MUST ...".

Given that relaying agents clearly do *not* have such a restriction, I
think that we should view anywhere that the document states that relaying
agents MUST propagate some article with a great deal of suspicion.  At the
least, every occurance of such language is an internal contradiction.

If I recall correctly from my reading, there are rather few places where
this contradiction occurs.  This may be one of the only ones.

> It was a long way back, but there was certainly a consensus at the time
> to try and make Distributions work.

Well, again, it's futile to argue about what happened years ago, but I
don't believe a consensus continues to exist in this area.

> news.admin.net-abuse groups, that it came out of the normal processing
> of the Path header rather than on special code in transport agents.

Nope.  It was possible because of a pre-existing feature in INN (the path
aliasing concept predates pseudo-sites by quite some time), but path
aliasing is not really part of the normal processing of the Path header.

---

From: Russ Allbery <[email-masked]>
Date: Wed, 29 Sep 2004 11:35:12 -0700

> Any advance on that?

I thought Henry and I just agreed on language, and I'm curious as to why
you decided to go with this language instead, from an article that
predates the conclusion that Henry and I reached.  Maybe it would be
better to see if Bill agrees with Henry and I and if so, go with that
language since it has more of a consensus?

I don't see a lot of purpose served in distinguishing between existence
and syntactic validity of headers.  I think that makes the logic of the
standard slightly more complex for no particularly clear reason.

---

From: Henry Spencer <[email-masked]>
Date: Wed, 29 Sep 2004 14:54:36 -0400 (EDT)

On Wed, 29 Sep 2004, Russ Allbery wrote:
> you decided to go with this language instead...

To clarify my own position on this...

+ I see a small benefit from insisting on presence of mandatory headers,
but only a small one.  It's unobjectionable but not very important. 

+ I'm reluctant to demand full checking by relay agents which otherwise
don't have reason to do it.  Even SHOULD seems too strong a word here. 

+ I would *like* to see relay agents not only permitted, but explicitly
encouraged, to do as much checking as they feel they can. 

+ I don't care deeply about the issue, and could live with any wording
that permits full checking but doesn't require it. 

                                                          Henry Spencer
                                                       [email-masked]

---

From: Henry Spencer <[email-masked]>
Date: Wed, 29 Sep 2004 15:00:28 -0400 (EDT)

On Wed, 29 Sep 2004, Russ Allbery wrote:
> ...I'm surprised that it's not in son-of-1036 and doesn't appear
> the right thing to do [1991]...

C News is mostly a late-80s creation and was pretty much in maintenance
mode by 1991, so if this wasn't prominently brought to our attention it
could easily have been missed.  And son-of-1036 reflected C News
experience more than anything else.  I don't recall this issue being
raised when son-of-1036 came out, but there are probably some son-of-1036
responses in my files that never got dealt with properly, as I was most
definitely in news burnout by that time. 

                                                          Henry Spencer
                                                       [email-masked]

---

From: Bill Davidsen <[email-masked]>
Date: Thu, 30 Sep 2004 09:53:57 -0400

> Not doing loop detection is acceptable only in tightly controlled

Unless you assume some evil entity editing the Path header loop 
detection will work anyway, I would assume. I certainly won't send 
anything to a site in the Path, so the worst that could happen is that I 
would get something so old it expired out of history. And if I've seen 
it I don't expect to have my peers offer it again.

---

From: Henry Spencer <[email-masked]>
Date: Thu, 30 Sep 2004 18:51:53 -0400 (EDT)

On Thu, 30 Sep 2004, Bill Davidsen wrote:
> detection will work anyway, I would assume.

The Path-header check is just an optimization; loop detection is done by
message ID and history list.  Yes, mangled or reinitialized Path headers
have been known to happen, especially when gateways get involved. 

> ...the worst that could happen is that I 
> would get something so old it expired out of history.

Correct... and it is important that you recognize that case and discard
the article, because loop detection is not reliable for it.  Yes, loops
with propagation delays longer than history-list retention times have been
known to happen, especially when gateways or server outages get involved. 

> And if I've seen 

This isn't a safe assumption unless local feed topology and relayer
software are sufficiently tightly controlled that you can be sure that
either (a) there are no feed loops, or (b) all loops contain at least one
host which does full history/date-based loop detection. 

                                                          Henry Spencer
                                                       [email-masked]