Subject: #1416: Reinjection and Injection-Date proposed text

From: Russ Allbery <[email-masked]>
Date: Sat, 21 Apr 2007 22:13:38 -0700

Okay, here we go.  Some time away from this, while unfortunate for
progress in the working group, definitely cleared my head and gave me a
new perspective.

Here's a concrete wording proposal.  This implements the IR proposal
rather than the IC proposal in Charles's writeup.  If we decide to go with
IC, the modifications should be slight.  This will hopefully give us
something more concrete to discuss, and I think we'll be able to quickly
reach consensus on some parts of it.

After some time away and reconsidering this, I favor Harold's (and, I
think, Charles's) proposal of simply not worrying about the case that was
giving me so much heartburn and simply ruling out serial multiple
injection in cases where you have to drop the Injection-Date header field
in the main language of the standard.  If someone wants to handle that
separately as a gateway, I think the out is there, and we don't need to
draw a bright arrow to it.

Please pay special attention to the new Article History and Duplicate
Suppression section and the Multiple Injection of Articles section that
replaces the whole Reinjection section in the current draft.  The former
should hopefully be uncontroversial and address many of Frank's concerns.

Hopefully a unified diff is reasonably readable.  If folks would like me
to send out the complete revised text, let me know.  I've omitted the
obvious diffs to section numbers and the like, so this diff probably won't
apply entirely cleanly.

--- draft-ietf-usefor-usepro-07.txt	2007-02-19 22:20:36.000000000 -0800
+++ usepro-1416.txt	2007-04-21 22:08:45.000000000 -0700
@@ -309,8 +309,8 @@
 
    "Injecting" an article is the processing of a proto-article by an
    injecting agent.  Normally this action is done once and only once for
-   a given article.  "Reinjecting" an article is passing an already-
-   injected article to an injection agent.
+   a given article.  "Multiple injection" is passing the same article to
+   multiple injecting agents, either serially or in parallel.
 
    A "gateway" is software which receives news articles and converts
    them to messages of some other kind (such as [RFC2822] mail
@@ -631,7 +631,72 @@
    baz.isp.example, bar.isp.example, and foo-news folded the Path header
    field.
 
-3.3.  Duties of a Posting Agent
+3.3.  Article History and Duplicate Suppression
+
+   Netnews normally uses a flood-fill algorithm for propagation of
+   articles in which each news server offers articles it accepts to
+   multiple peers and each news server may be offered the same article
+   from multiple other news servers.  Accordingly, duplicate suppression
+   is key; if a news server accepted every article it was offered, it
+   may needlessly accept (and then potentially retransmit) dozens of
+   copies of every article.
+
+   Relaying and serving agents therefore MUST keep a record of articles
+   they have already seen and use that record to reject additional
+   offers of the same article.  This record is called the history file
+   or database.
+
+   Each article is uniquely identified by its message identifier, so a
+   relaying or serving agent could satisfy this requirement by storing a
+   record of every message identifier that agent has ever seen.  Such a
+   history database would grow without bound, however, so it is common
+   and permitted to optimize based on the Injection-Date or Date header
+   field of an article as follows.  (In the following discussion, the
+   "date" of an article is defined to be the date represented by its
+   Injection-Date header field if present, otherwise its Date header
+   field.)
+
+   o  Agents MAY select a cutoff interval and reject any article with a
+      date farther in the past than that cutoff interval.  If this
+      interval is shorter than the time it takes for an article to
+      propagate through the network, the agent may reject an article it
+      had not yet seen, so it ought not be aggressively short.  For
+      Usenet, for example, a cutoff interval of no less than seven days
+      is conventional.
+
+
+
+
+
+Allbery & Lindsey         Expires July 5, 2007                 [Page 12]
+
+Internet-Draft     Netnews Architecture and Protocols       January 2007
+
+
+   o  Agents that enforce such a cutoff MAY then drop records of
+      articles that had dates older than the cutoff from their history
+      databases.  If such an article were offered to the agent again, it
+      would be rejected due to the cutoff date, so the history record is
+      no longer required to suppress the duplicate.
+
+   o  As an optimization for easier history database manipulation,
+      agents MAY instead drop history records written longer ago than
+      the cutoff interval plus one day.  If this retention mechanism is
+      used, the history retention period MUST be longer than the cutoff
+      interval to allow for articles dated in the future unless the
+      agent rejects all articles dated in the future.  One day is the
+      maximum allowed error into the future for article dates, so it is
+      a convenient and safe extension for the retention interval.
+
+   This is just one implementation strategy for article history, albeit
+   the most common one.  Relaying and serving agents are not required to
+   use this strategy, only to meet the requirement of not accepting an
+   article more than once.  However, implementors of general-purpose
+   Netnews relaying and serving agents who do not have extensive
+   experience with Netnews and the subtle effects of its flood-fill
+   algorithm are encouraged to use the above algorithm by default.
+
+3.4.  Duties of a Posting Agent
 
    A posting agent is the component of a user agent that assists a
    poster in creating a valid proto-article and forwarding it to an
@@ -639,8 +704,33 @@
 
    Posting agents SHOULD ensure that proto-articles they create are
    valid according to [USEFOR] and any other applicable policies.  They
-   MUST NOT create any Injection-Date or Injection-Info header fields;
-   these headers will be added by the injecting agent.
+   MUST NOT create an Injection-Info header field; this headers will be
+   added by the injecting agent.
+
+   The Injection-Date header field is new in this revision of the
+   Netnews protocol and is designed to permit the Date header field to
+   hold the composition date, even if the proto-article is not injected
+   for some time after its composition.  However, note that all
+   implementations predating this specification ignore the Injection-
+   Date header field and use the Date header field in its stead for
+   rejecting articles older than their cutoff (see Section 3.3), and
+   injecting agents predating this specification do not add an
+   Injection-Date header.  Posting agents MAY add an Injection-Date
+   header field and SHOULD do so if the Date header field (representing
+   the composition time of the proto-article) is more than a day in the
+   past in case the injecting agent predates this specification.  Note,
+   though, that articles with a Date header field substantially in the
+   past will still be rejected by implementations predating this
+
+
+
+Allbery & Lindsey         Expires July 5, 2007                 [Page 13]
+
+Internet-Draft     Netnews Architecture and Protocols       January 2007
+
+
+   specification, regardless of the Injection-Date header field, and may
+   suffer poor propagation.
 
    Contrary to [RFC2822], which implies that the mailbox or mailboxes in
    the From header field should be that of the poster or posters, a
@@ -652,7 +742,7 @@
    attempt to post an article which cancels or Supersedes another
    article of which the poster is not the author or sender.
 
-3.3.1.  Proto-articles
+3.4.1.  Proto-articles
 
    A proto-article is an article in the format used by a posting agent
    for offering to an injecting agent.  It may omit certain header
@@ -662,54 +752,75 @@
    SHOULD NOT be transmitted to any other agent.
 
    A proto-article has the same format as a normal article except that
-   the Injection-Date, Injection-Info, and Xref header fields MUST NOT
-   be present; the Path header field MUST NOT contain a "POSTED" <diag-
-   keyword>; and any of the following mandatory header fields MAY be
-
+   the Injection-Info and Xref header fields MUST NOT be present; the
+   Path header field MUST NOT contain a "POSTED" <diag-keyword>; and any
+   of the following mandatory header fields MAY be omitted: Message-ID,
+   Date, and Path.  In all other respects, a proto-article MUST be a
+   valid Netnews article.  In particular, the header fields which may be
+   omitted MUST NOT be present with invalid content.
 
+   If a posting agent intends to offer the same proto-article to
+   multiple injecting agents, the header fields Message-ID and Date MUST
+   be present and identical in all copies of the proto-article, and the
+   header field Injection-Date SHOULD be present and identical in all
+   copies of the proto-article.  See Section 3.4.2.
+
+3.4.2.  Multiple Injection of Articles
+
+   Under some circumstances (posting to multiple disjoint networks,
+   injecting agents with spotty connectivity, or redundancy, for
+   example), a posting agent may wish to offer the same article to
+   multiple injecting agents.  In this unusual case, the goal is to not
+   create multiple independent articles but rather to inject the same
+   article at multiple points and use the normal duplicate suppression
+   facility of Netnews (see Section 3.3) ensure that any given agent
+   only accepts the article once.
 
-Allbery & Lindsey         Expires July 5, 2007                 [Page 12]
-
-Internet-Draft     Netnews Architecture and Protocols       January 2007
+   Whenever possible, multiple injection SHOULD be done by offering the

-   omitted: Message-ID, Date, and Path.  In all other respects, a proto-
-   article MUST be a valid Netnews article.  In particular, the header
-   fields which may be omitted MUST NOT be present with invalid content.
 
-   If a posting agent intends to offer the same proto-article to
-   multiple injecting agents, the header fields Message-ID and Date MUST
-   be present and identical in all copies of the proto-article.
+Allbery & Lindsey         Expires July 5, 2007                 [Page 14]
+
+Internet-Draft     Netnews Architecture and Protocols       January 2007
 
-3.3.2.  Reinjection of Articles
 
-   A given article SHOULD be processed by an injecting agent once and
-   only once.  The Injection-Date or Injection-Info header fields are
-   added by an injecting agent and are not permitted in a proto-article.
-   Their presence (or the presence of other unstandardized or obsolete
-   trace headers such as NNTP-Posting-Host, NNTP-Posting-Date, or
-   X-Trace) indicates that the proto-article is instead an article and
-   has already been processed by an injecting agent.  A posting agent
-   SHOULD normally reject such articles.
-
-   In the exceptional case that an article needs to be reinjected for
-   some reason (such as transferring an article from one Netnews to
-   another where those networks have no relaying agreement), the posting
-   agent doing the reinjection MUST convert the article back into a
-   proto-article before passing it to an injecting agent (such as by
-   renaming the Injection-Info and Injection-Date header fields and
-   removing any Xref header field) and MUST perform the date checks on
-   the existing Injection-Date or Date header fields that would
-   otherwise be done by the injecting agent.
-
-   Reinjecting articles may cause loops, loss of trace information, and
-   other problems and should only be done with care and when there is no
-   available alternative.  A posting agent that does reinjection is a
-   limited type of gateway and as such is subject to all of the
-   requirements of an incoming gateway in addition to the requirements
-   of a posting agent.
+   same proto-article to multiple injecting agents.  The posting agent
+   MUST supply the Message-ID, Date, and Injection-Date header fields,
+   and the proto-article offered to each injecting agent MUST be
+   identical.
+
+   In some cases, offering the same proto-article to all injecting
+   agents may not be possible (such as when transferring, after the
+   fact, articles found on one Netnews network to another, unconnected
+   one).  In this case, the posting agent MUST convert the article back
+   into a proto-article before passing it to another injecting agent,
+   but it MUST retain unmodified the Message-ID, Date, and Injection-
+   Date header fields.  It MUST NOT add an Injection-Date header field
+   if it is missing from the existing article.  It MUST remove any Xref
+   header field and either rename or remove any Injection-Info header
+   field and other trace fields.
+
+   Multiple injection inherently risks duplicating articles.  Multiple
+   injection after the fact, by converting an article back to a proto-
+   article and injecting it again, additionally risks loops, loss of
+   trace information, and other problems and should be done with care
+   and only when there is no alternative.  The requirement to retain
+   Message-ID, Date, and Injection-Date header fields minimizes the
+   possibility of a loop and ensures that the newly injected article is
+   not treated as a new, separate article.
+
+   Multiple injection of an article listing one or more moderated
+   newsgroups in its Newsgroups header field SHOULD only be done by a
+   moderator and SHOULD only be done after the proto-article is approved
+   for all moderated groups to which it is posted and has an Approved
+   header field.  Multiple injection of an unapproved article intended
+   for moderated newsgroups will normally only result in the moderator
+   receiving multiple copies, and if the newsgroup status is not
+   consistent across all injecting agents, may result in duplication of
+   the article or other problems.
 
-3.3.3.  Followups
+3.4.3.  Followups
 
    A followup is an article that contains a response to the contents of
    an earlier article, its precursor.  In addition to its normal duties,
@@ -831,30 +942,31 @@
 
    2.   It MUST reject any proto-article that does not have the proper
         mandatory header fields for a proto-article; that has Injection-
-        Date, Injection-Info, or Xref header fields; that has a Path
-        header field containing the "POSTED" <diag-keyword>; or that is
+        Info or Xref header fields; that has a Path header field
+        containing the "POSTED" <diag-keyword>; or that is not
+        syntactically valid as defined by [USEFOR].  It SHOULD reject

-Allbery & Lindsey         Expires July 5, 2007                 [Page 15]
+Allbery & Lindsey         Expires July 5, 2007                 [Page 17]
 
 Internet-Draft     Netnews Architecture and Protocols       January 2007

-        not syntactically valid as defined by [USEFOR].  It SHOULD
-        reject any proto-article which contains a header field
-        deprecated for Netnews.  It MAY reject any proto-article that
-        contains trace header fields indicating that it was already
-        injected by an injecting agent that did not add Injection-Info
-        or Injection-Date.
+        any proto-article which contains a header field deprecated for
+        Netnews.  It MAY reject any proto-article that contains trace
+        header fields indicating that it was already injected by an
+        injecting agent that did not add Injection-Info or Injection-
+        Date.
 
    3.   It SHOULD reject any article whose Date header field is more
         than 24 hours into the future (and MAY use a margin less than 24
-        hours).  It SHOULD reject any article whose Date header appears
-        to be stale (more than 72 hours into the past, for example, or
-        too old to still be recorded in the database of a relaying agent
-        the injecting agent will be using) since not all news servers
-        support Injection-Date.
+        hours).  It SHOULD reject any article whose Date header is too
+        far into the past (older than the cutoff interval of a relaying
+        agent the injecting agent is using, for example), since not all
+        news servers support Injection-Date and only the injecting agent
+        can provide a useful error message to the posting agent.  This
+        interval SHOULD NOT be any shorter than 72 hours into the past.
 
    4.   It SHOULD reject any proto-article whose Newsgroups header field
         does not contain at least one <newsgroup-name> for a valid
@@ -904,13 +1016,19 @@
         source of the article and possibly other trace information as
         described in Section 3.2.8 of [USEFOR].
 
-   11.  The injecting agent MUST then add an Injection-Date header field
-        containing the current date and time.
+   11.  If the proto-article already had an Injection-Date header field,
+        it MUST NOT be modified or replaced.  If the proto-article had
+        both a Message-ID header field and a Date header field, an
+        Injection-Date header field MUST NOT be added, since the proto-
+        article may have been multiply injected by a posting agent that
+        predates this standard.  Otherwise, the injecting agent MUST add
+        an Injection-Date header field containing the current date and
+        time.
 
    12.  Finally, the injecting agent forwards the article to one or more
         relaying agents, and the injection process is complete.
 
-3.4.1.  Forwarding Messages to a Moderator
+3.5.1.  Forwarding Messages to a Moderator
 
    An injecting agent MUST forward the proto-article to the moderator of
    the leftmost moderated group listed in the Newsgroups header field,
@@ -964,7 +1084,7 @@
    that database.  Such methods may be more appropriate for smaller
    Netnews networks.
 
-3.5.  Duties of a Relaying Agent
+3.6.  Duties of a Relaying Agent
 
    A relaying agent accepts injected articles from injecting and other
    relaying agents and passes them on to relaying or serving agents.  To
@@ -993,28 +1113,24 @@
 
    1.  It MUST reject any article without a Newsgroups header field or
        Message-ID header field, or without either an Injection-Date or
-       Date header field.
-
-   2.  It MUST reject any article that has already been successfully
-       sent to it, based on the Message-ID header field of the article.
-       To satisfy this requirement, a relaying agent normally keeps a
-       database of message identifiers it has already accepted.

-
-
-Allbery & Lindsey         Expires July 5, 2007                 [Page 18]
+Allbery & Lindsey         Expires July 5, 2007                 [Page 20]
 
 Internet-Draft     Netnews Architecture and Protocols       January 2007

-   3.  It MUST examine the Injection-Date header field or, if absent,
-       the Date header field, and reject the article if that date
-       predates the earliest articles of which it keeps record or if
-       that date is more than 24 hours into the future.  It MAY reject
-       articles with dates in the future with a smaller margin than 24
-       hours.
+       Date header field.
+
+   2.  It MUST examine the Injection-Date header field or, if absent,
+       the Date header field, and reject the article if that date is
+       more than 24 hours into the future.  It MAY reject articles with
+       dates in the future with a smaller margin than 24 hours.
+
+   3.  It MUST reject any article that has already been successfully
+       sent to it or that is dated older than its cutoff date, as
+       described in Section 3.3.
 
    4.  It SHOULD reject any article that does not include all the
        mandatory header fields.  It MAY reject any article that contains
@@ -1052,19 +1168,20 @@
    modify the body of articles in any way.  If an article is not
    acceptable as-is, the article MUST be rejected rather than modified.
 
-3.6.  Duties of a Serving Agent
 
-   A serving agent accepts articles from a relaying agent or injecting
-   agent, stores it, and makes it available to reading agents.  Articles
-   are normally indexed by newsgroup and <article-locator> (Section

-Allbery & Lindsey         Expires July 5, 2007                 [Page 19]
+Allbery & Lindsey         Expires July 5, 2007                 [Page 21]
 
 Internet-Draft     Netnews Architecture and Protocols       January 2007

+3.7.  Duties of a Serving Agent
+
+   A serving agent accepts articles from a relaying agent or injecting
+   agent, stores it, and makes it available to reading agents.  Articles
+   are normally indexed by newsgroup and <article-locator> (Section
    3.2.14 of [USEFOR]), usually in the form of a decimal number.
 
    If the serving agent stores articles by newsgroup, control messages
@@ -1088,20 +1205,17 @@
        fields that do not have valid contents.
 
    2.  It MUST examine the Injection-Date header field or, if absent,
-       the Date header field, and reject the article if that date
-       predates the earliest articles of which it keeps record or if
-       that date is more than 24 hours into the future.  It MAY reject
-       articles with dates in the future with a smaller margin than 24
-       hours.
+       the Date header field, and reject the article if that date is
+       more than 24 hours into the future.  It MAY reject articles with
+       dates in the future with a smaller margin than 24 hours.
 
    3.  It MUST reject any article that has already been successfully
-       sent to it, based on the Message-ID header field of the article.
-       To satisfy this requirement, a relaying agent normally keeps a
-       database of message identifiers it has already accepted.
+       sent to it or that is dated older than its cutoff date, as
+       described in Section 3.3.
 
    4.  It SHOULD reject any article that matches an already-received and
        honored cancel message or Supersedes header field following the
-       same rules as a relaying agent (Section 3.5).
+       same rules as a relaying agent (Section 3.6).
 
    5.  It MUST reject any article without an Approved header field
        posted to any newsgroup listed as moderated.
@@ -1212,7 +1327,7 @@
        understood by the moderator (such as delays in the moderation
        process) in which case they MAY substitute the current date.  Any
        Injection-Date, Injection-Info, or Xref header fields already
-       present (though there should be none) MUST be removed.
+       present MUST be removed.
 
    5.  Any Path header field MUST either be removed or truncated to only
        those entries following its "POSTED" <diag-keyword>, if any.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: Frank Ellermann <[email-masked]>
Date: Sun, 22 Apr 2007 19:12:45 +0200

> +3.3.  Article History and Duplicate Suppression
[...]

Very clear, thanks.

>+ it ought not be aggressively short.  For For Usenet, for example,
>+ a cutoff interval of no less than seven days is conventional.

Yes, no MUSTard needed here, the consequences are clear.  Commercial
servers with shorter cutoff intervals for "binary" newsgroups may
need the support staff to answer complaints by their angry customers.

>- MUST NOT create any Injection-Date or Injection-Info header
>- fields; these headers will be added by the injecting agent.
>+ MUST NOT create an Injection-Info header field; this headers will
>+ be added by the injecting agent.

Nit: s/headers/header field/

>+ for some time after its composition.

Please add a reference to RFC 2822 here, it's likely not only you who
hates this, it needs a compelling justification.

>+ Posting agents MAY add an Injection-Date header field and SHOULD do
>+ so if the Date header field (representing the composition time of
>+ the proto-article) is more than a day in the past in case the
>+ injecting agent predates this specification.

I think you're mixing unrelated concepts here.  The SHOULD is clear,
it doesn't need the qualifier starting with "in case...".  The SHOULD
addresses the "stale 2822 date" issue among other problems.

The MAY is something more subtle, and from the text it's not clear if
this is limited to "more than a day ago".  I _guess_ that the MAY is
about multiple injections, and you want to allow that posting agents
create their own unique Message-ID + Injection-Date combination.  And
with some reading between the lines such smart posting agents might
even decide to ignore RFC 2822 and force a corresponding unique Date.

>+ Note, though, that articles with a Date header field substantially
>+ in the past will still be rejected by implementations predating
>+ this specification, regardless of the Injection-Date header field,
>+ and may suffer poor propagation.

Okay, it's not only "between the lines", and I like that, too.

>+ the header field Injection-Date SHOULD be present and identical in
>+ all copies of the proto-article.  See Section 3.4.2.

Yes, and the MAY mentioned above might be also related to 3.4.2,  If
that's the case you've just overwritten it here by a SHOULD, or is
the MAY about a 3rd case ?  Like a posting agent always creating its
own Injection-Date even if it's not one of the two SHOULDs, that
could be a 3rd case.  In that case I hope that it gets "identical"
right under all foreseeable circumstances wrt the "multi-SHOULD".

>+ the goal is to not create multiple independent articles but
>+ rather to inject the same article at multiple points and use the
>+ normal duplicate suppression facility of Netnews (see Section 3.3)
>+ ensure that any given agent only accepts the article once.

Nit: s/use/let/

The move from "reinjection" to "multiple injection" resulted in a
far better readable story.

>-  A posting agent SHOULD normally reject such articles.

Odd, was that really a "posting agent" rejecting articles ?

>+ The posting agent MUST supply the Message-ID, Date, and
>+ Injection-Date header fields, and the proto-article offered to
>+ each injecting agent MUST be identical.

Some lines above it was a "multi-SHOULD", now you have a "multi-MUST"
for the Injection-Date.  How about this:

| The posting agent MUST supply the Message-ID and the Date, it
| SHOULD also supply an Injection-Date, and the proto-article
| offered to each injecting agent MUST be identical.

Arguably you can also use MUST here as you have it, and replace
the "multi-SHOULD" at the end of 3.4.1 by MUST.  But that would
render all existing posting agents with no idea about the new
Injection-Date as "non-conforming" when they try multi-Injection.

>+ It MUST remove any Xref header field and either rename or remove
>+ any Injection-Info header field and other trace fields.

I see why they MUST remove Injection-Info, an old injecting agent
would let it pass causing havoc.  But why MUST a posting agent
remove Xref, this job could be handled by the injecting agent ?

Is it "a simple MUST is better than more convoluted statements",
or do I miss a serious Xref-problem here ?

>+ SHOULD only be done after the proto-article is approved for
>+ all moderated groups to which it is posted and has an Approved
>+ header field.

What's a good excuse to violate this SHOULD ?  Is it a moderator
removing moderated and unapproved newsgroups, and injecting the
article at least for the approved or unmoderated newsgroups ?
If there's no good excuse I prefer a MUST.

>+ This interval SHOULD NOT be any shorter than 72 hours into the
>+ past.

That's a rather short minimum, barely enough for a normal weekend.
For this radical approach, how about a MUST NOT ?  Disclaimer, I've
no idea about what's usual for "binary" newsgroups.  But I doubt
that they have any use for the 2822-Date concept like "text" groups
or, more important, mail2news gateways.

>+ If the proto-article had both a Message-ID header field and a
>+ Date header field, an Injection-Date header field MUST NOT be
>+ added, since the proto-article may have been multiply injected
>+ by a posting agent that predates this standard.  Otherwise, the
>+ injecting agent MUST add an Injection-Date header field
>+ containing the current date and time.

We really need to check that you use MUST and SHOULD consistently.

How about a 2 * 2 table, two for legacy vs. new posting agent, and
two for legacy vs. new injecting agent, stating that all new agents
MUST get it right, resulting in an overall SHOULD for "any" agent.

Your MUST NOT + MUST above talks about new injecting agents.  For
the MUST NOT it's obvious, a legacy agent doesn't know what the
Injection-Date is and has no problem with not adding it.  But the
MUST is only for new agents.  Overall it's a SHOULD, and legacy
agents have a good excuse to violate the overall SHOULD.

>+ It MAY reject articles with dates in the future with a smaller
>+ margin than 24 hours.

Does that really help ?  Otherwise strike it, we have a precision
of "days" elsewhere, not "hours", and "days" allow us to ignore
all plausible timezone issues.

>+ dates in the future with a smaller margin than 24 hours.

Same here, let's stick to one day, and get rid of less than one day.

Impression without checking the details:  the quoted parts are now
clearer than in I-D.ietf-usefor-usepro-06.

Frank

---

From: Russ Allbery <[email-masked]>
Date: Sat, 28 Apr 2007 19:24:41 -0700

> need the support staff to answer complaints by their angry customers.

Just for clarity in the discussion (I don't think a wording change is
needed), note that the cutoff date for history retention is independent of
the period of article retention in the spool.  Usually the latter is
longer than the former, but it may also be shorter.  They don't need to be
related.

Normally, the cutoff date is set uniformly across the server regardless of
the group, but the article retention period may be set much shorter for
binary groups.  The requirement is then that the server keep a record of
those received and expired binary articles in the history file even after
deleting the article off disk until they meet the cutoff requirements.

> Nit: s/headers/header field/

Thanks, fixed.

> hates this, it needs a compelling justification.

Yes, good idea.  Done.

> addresses the "stale 2822 date" issue among other problems.

Yes, I think you're right.

> with some reading between the lines such smart posting agents might even
> decide to ignore RFC 2822 and force a corresponding unique Date.

The intention was to allow posting agents to add an Injection-Date header
field any time they feel like it.  The other point that's missing here is
that the Injection-Date header shouldn't be added at the same point the
Date header is.  I now have:

   If a proto-article contains both Message-ID and Date header fields,
   posting agents MAY add an Injection-Date header field to that proto-
   article immediately before passing that proto-article to an injection
   agent.  They SHOULD do so if the Date header field (representing the
   composition time of the proto-article) is more than a day in the past
   at the time of injection.  If the proto-article is being sent to more
   than one injecting agent, see Section 3.4.2.

   The Injection-Date header field is new in this revision of the
   Netnews protocol and is designed to permit the Date header field to
   hold the composition date (as defined in section 3.6.1 of [RFC2822]),
   even if the proto-article is not injected for some time after its
   composition.  However, note that all implementations predating this
   specification ignore the Injection-Date header field and use the Date
   header field in its stead for rejecting articles older than their
   cutoff (see Section 3.3), and injecting agents predating this
   specification do not add an Injection-Date header.  Articles with a
   Date header field substantially in the past will still be rejected by
   implementations predating this specification, regardless of the
   Injection-Date header field, and may suffer poor propagation.

on the grounds of putting the requirement first and then the explanation.

> that's the case you've just overwritten it here by a SHOULD, or is the
> MAY about a 3rd case ?  Like a posting agent always creating its own
> Injection-Date even if it's not one of the two SHOULDs, that could be a
> 3rd case.  In that case I hope that it gets "identical" right under all
> foreseeable circumstances wrt the "multi-SHOULD".

This was a bit of a mess.  In my current draft, I've simplified this whole
situation by saying that posting agents doing multiple injection MUST add
the Injection-Date header field.  This makes all existing multiple
injection posting agents non-compliant, but we also did the same thing
with injecting agents and their Injection-Date requirement, so this seems
symmetric.

I would prefer that the whole Injection-Date feature be either a SHOULD or
a MUST, rather than using differing levels of requirement for different
agents.  Thoughts?  And does anyone want to argue that the whole feature
should be a SHOULD?

> Nit: s/use/let/

Fixed.

> far better readable story.

Yes, I was rather happy with that.  Thanks to Harald for the suggestion.
I think that casting of the idea makes everything much simpler.

> Odd, was that really a "posting agent" rejecting articles ?

Yeah.  Dumb idea on my part, in retrospect.  It was discussed in some of
the side-threads of this overall discussion, and I decided to drop it as
part of these changes since the bits that it was referring to have now
substantially changed anyway.

> Injection-Date as "non-conforming" when they try multi-Injection.

As mentioned above, I went the MUST route, but I think it's a debatable
point and I'd love to hear other comments.

> I see why they MUST remove Injection-Info, an old injecting agent would
> let it pass causing havoc.  But why MUST a posting agent remove Xref,
> this job could be handled by the injecting agent ?

> or do I miss a serious Xref-problem here ?

It's more the former, plus backwards compatibility.  Existing injecting
agents will reject articles with Xref rather than remove the header.  It
seems simple enough given the other context to have posting agents remove
it, and existing posting agents remove it, so I just said MUST instead of
adding the explanation that existing injecting agents will reject and then
go through and try to change the language to allow injecting agents to
cope with it.

> removing moderated and unapproved newsgroups, and injecting the article
> at least for the approved or unmoderated newsgroups ?  If there's no
> good excuse I prefer a MUST.

If the moderator changed anything about the Newsgroups header field, it
would no longer be multiple injection as defined in this document.  I
can't think of any reason to ever violate that SHOULD, so I've
provisionally changed it to MUST.  (Well, I suppose there's the case where
one has access to several injecting agents, each of which are unreliable,
and therefore one is using multiple injection to try to increase the
chances that at least one copy of the article gets to the moderator, but
this is enough of a special case that I don't think we need to call that
multiple injection as opposed to the poster intentionally posting several
articles.)

> or, more important, mail2news gateways.

On Usenet, I think a MUST NOT would probably be warranted, but I wasn't
comfortable enough saying that there would be no reason to ever use a
shorter interval in any Netnews implementation.  I don't really feel
strongly about this, but I mildly lean towards SHOULD NOT here just on the
grounds of permitting an out for some reason that we didn't anticipate.

Of course, the presence of this provision for injecting agents in the
draft at all is still being debated.  I know Charles would prefer that it
be removed, and I would like to get more feedback on that.

> How about a 2 * 2 table, two for legacy vs. new posting agent, and two
> for legacy vs. new injecting agent, stating that all new agents MUST get
> it right, resulting in an overall SHOULD for "any" agent.

> Your MUST NOT + MUST above talks about new injecting agents.  For the
> MUST NOT it's obvious, a legacy agent doesn't know what the
> Injection-Date is and has no problem with not adding it.  But the MUST
> is only for new agents.  Overall it's a SHOULD, and legacy agents have a
> good excuse to violate the overall SHOULD.

Here's what it looked like before I started writing this section of my
reply:

Legacy posting agent, legacy injecting agent:
    No one will add Injection-Date since neither agent knows about it,
    although as a generic unknown header, either agent is permitted to
    do so.

Legacy posting agent, compliant injecting agent:
    Posting agent won't add Injection-Date.  Injecting agent MUST add
    Injection-Date if one or both of Date and Message-ID are not
    specified, otherwise MUST NOT (since multiple injection may be
    happening).

Compliant posting agent, legacy injecting agent:
    Posting agent MAY add Injection-Date any time, SHOULD if the Date
    is more than a day in the past at the time of injection, MUST if
    doing multiple injection.  The injecting agent doesn't know about
    the header and therefore will do nothing with it.

Compliant posting agent, compliant injecting agent:
    Posting agent MAY add Injection-Date any time, SHOULD if the Date
    is more than a day in the past at the time of injection, MUST if
    doing multiple injection.  Injecting agent generally MUST add the
    header if the posting agent doesn't.  The one exception is if the
    posting agent provides both Message-ID and Date but not
    Injection-Date, which it is always allowed to do and which doesn't
    even violate a SHOULD if the Date is fresh.  In this case, the
    injecting agent MUST NOT add it either.

Looking at that table, and given that we have spelled out elsewhere how to
handle articles with no Injection-Date, I think I agree with your point
and don't see a reason for the injection agent requirement to be MUST.  We
already have a scenario in which a post only handled by compliant agents
gets no Injection-Date and it doesn't break anything, so it seems like
addition of Injection-Date by an injecting agent need only be a SHOULD.

Opinions?

This situation might change if we went with Charles's approach where
injecting agents are permitted to add Injection-Date to posts that already
have both Message-ID and Date.  In that case, there is no gap.

> Does that really help ?

It's there to make it clear that some agents do not give you 24 hours of
leeway and this is permitted in the protocol.  It's redundant with the
general rule that anyone can reject any article for any reason they
please, but I felt like it was worth drawing special attention to this so
that people don't think they can rely on having 24 hours of slop in the
date checking.

> Same here, let's stick to one day, and get rid of less than one day.

This wording should be identical to the wording called out above, so I'll
change both if we decide to change it.