Subject: Re: #1416 Injection-Date: current wording proposal (corrected)

From: Frank Ellermann <[email-masked]>
Date: Tue, 01 May 2007 09:14:40 +0200

Russ Allbery wrote:

> I think it may be worthwhile to reach consensus on several parts of
> this separately.

Yes, e.g. I've no clue what precisely you adopted from what I wrote,
or what I wrote in the first place.  Sometimes there are situations
where I think B is better than A, and as soon as I see B I think A
is better than B, etc. until somebody says enough.

> For example, if we decide that the whole current history section is
> fine, I can incorporate that into the draft and then stop including
> it in further diffs for discussion.

Checking what I recall, the "one day in the future" business is clear.

Apparently the definition of "retention" is based on the article date,
not the arrival date, is that as it should be ?  [[In the bullet with
the "optimization for easier history manipulation"]]

Maybe add a qualifier "if using the article date and not the arrival
date" to the "history retention" in this paragraph.

For the "MUST NOT contain POSTED", in theory this would be fine for
an injecting-agent not creating its own POSTED because it doesn't
know the new diagnostics.  But when it doesn't know them it can't
identify them, so I guess the MUST NOT is unnecesary.  Or I forgot
why we wanted this.

You still have 72 hours, can we use 3 days, getting rid of "hours"
everywhere ?  And MUST NOT instead of SHOULD NOT about the minimum,
I don't see how it can work as expected for a seriously shorter
periods, or what's a good excuse for shorter periods.

Similarly, s/24 hours/one day/ for the definition of "future".  The
"MAY reject" about less than 24 hours could run into timezone issues,
please delete it.  It MAY reject depending on the phase of the moon,
or anything from odd IPs on Tuesdays ((c) JCK), "less than 24 hours
in the future" needs no special MAY.

The XML diff listed the 24 hours in two overlapping contexts (just
an observation).

Frank

---

From: Russ Allbery <[email-masked]>
Date: Wed, 02 May 2007 20:05:25 -0700

> date" to the "history retention" in this paragraph.

Is this part clear now?

> For the "MUST NOT contain POSTED", in theory this would be fine for an
> injecting-agent not creating its own POSTED because it doesn't know the
> new diagnostics.  But when it doesn't know them it can't identify them,
> so I guess the MUST NOT is unnecesary.  Or I forgot why we wanted this.

I'm not sure that it was a matter of a lot of on-list discussion.  It feel
out of a tighter definition of just what a proto-article is, and I may
well have made it too tight.  It's a trace diagnostic, so having it
multiple times doesn't hurt anything.

I think we either say that an injecting agent MUST reject articles that
already have POSTED or we drop that wording entirely; I don't think a
SHOULD would make sense.

> everywhere ?  And MUST NOT instead of SHOULD NOT about the minimum, I
> don't see how it can work as expected for a seriously shorter periods,
> or what's a good excuse for shorter periods.

I don't mind changing this to three days.

I'm still uncomfortable about making this a MUST NOT in general.  The
interoperability problems of using a shorter interval don't feel like
MUST-level problems to me.  But I'd certainly welcome other opinions and
can be convinced.

> Similarly, s/24 hours/one day/ for the definition of "future".  The "MAY
> reject" about less than 24 hours could run into timezone issues, please
> delete it.  It MAY reject depending on the phase of the moon, or
> anything from odd IPs on Tuesdays ((c) JCK), "less than 24 hours in the
> future" needs no special MAY.

Here, I do want to use 24 hours.  The definition of a "day" may be 25
hours or 23 hours depending on daylight saving time transitions, and I
don't think we want the ambiguity.  It doesn't really matter once we're
talking about longer periods, but it does matter at the interval of a day.

Any other feelings on whether or not to keep the MAY about shorter
rejection periods for articles dated in the future?

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 2 May 2007 18:02:06 GMT

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>Gr.  Ignore the previous version of this message.  Saving the file before
>regenerating the diff is useful.  Please ignore the previous version and
>use this one instead.

OK, I have finally got around to reading the whole of this. Many of my
points are minor textual niggles, but some are quite substantive.

>--- usepro.xml	2007-02-19 22:20:06.000000000 -0800
>+++ usepro-1416.xml	2007-04-28 19:30:11.000000000 -0700
>@@ -446,6 +447,68 @@
> 
>+      <section anchor="history"
>+               title="Article History and Duplicate Suppression">
>+
>+        <t>Relaying and serving agents therefore MUST keep a record of
>+        articles they have already seen and use that record to reject
>+        additional offers of the same article.  This record is called the
>+        history file or database.</t>
          ^^^^^^^
         "history"
{since you have jsut defined a new term}
>+
>+            <t>Agents that enforce such a cutoff MAY then drop records of
>+            articles that had dates older than the cutoff from their
>+            history databases.  If such an article were offered to the
>+            agent again, it would be rejected due to the cutoff date, so
>+            the history record is no longer required to suppress the
>+            duplicate.</t>
>+
>+            <t>As an optimization for easier history database
>+            manipulation, agents MAY instead drop history records written
>+            longer ago than the cutoff interval plus one day.  If this
>+            retention mechanism is used, the history retention period MUST
>+            be longer than the cutoff interval to allow for articles dated
>+            in the future unless the agent rejects all articles dated in
>+            the future.  One day is the maximum allowed error into the
>+            future for article dates, so it is a convenient and safe
>+            extension for the retention interval.</t>

It takes a lot of VERY careful reading to spot the curicial diference
between those two cases (and Frank seems to have the same problem). I
therefore suggest the following alternative for the 2nd one:

              Alternatively, agents MAY drop history records according to
	      the date when the article was received, in which case they
	      MUST retain them for 24 hours longer than the cutoff
	      interval, since the date that accompanies the article is
	      permitted to be up to 24 hours later than its actual time of
	      injection (see <xref duties-injecting>).

{BTW, I prefer using 24 hours rather than 1 day for such short time
intervals, especially since the discussion under duties-injecting speaks
of even shorter intervals}

>+        <t>This is just one implementation strategy for article history,
             ^^^^^^^^^^^^^^^^                ^^^^^^^^
             These are just two             strategies
>+        albeit the most common one.  Relaying and serving agents are not
                                 ^^^
                                 ones
>+        required to use this strategy, only to meet the requirement of not
>+        accepting an article more than once. ....

However, the next bit of the sermon is a bit overdone, and the use of
"default" seems not quite right:

>+        ......... However, implementors of
>+        general-purpose Netnews relaying and serving agents who do not
>+        have extensive experience with Netnews and the subtle effects of
>+        its flood-fill algorithm are encouraged to use the above algorithm
>+        by default.</t>

So I would suggest:

          ......... However, implementors are advised normally to use
	  these strategies, especially if they do not have extensive
	  experience with Netnews and the subtle effects of its flood-fill
	  algorithm.

>@@ -453,9 +516,33 @@
> 
>+        applicable policies.  They MUST NOT create an Injection-Info
                                                     ^^
                                                     any
>+        header field; this header field will be added by the injecting
>+        agent.</t>
>+
>+        <t>If a proto-article contains both Message-ID and Date header
                ^              ^
               the           already
>+        fields, posting agents MAY add an Injection-Date header field to
>+        that proto-article immediately before passing that proto-article
>+        to an injection agent.  They SHOULD do so if the Date header field
>+        (representing the composition time of the proto-article) is more
>+        than a day in the past at the time of injection.  If the
>+        proto-article is being sent to more than one injecting agent, see
                                 ^^^^
                               submitted
>+        <xref target="multi-injection" />.</t>

{Note that this paragraph might change if we adopt option IC rather than
IR}
>+
>+        <t>The Injection-Date header field is new in this revision of the
>+        Netnews protocol and is designed to permit the Date header field
                                              ^^^^^^
                                              allow
>+        to hold the composition date (as defined in section 3.6.1 of <xref
                                           ^^^^^^^
                                         recommended
>+        target="RFC2822" />), even if the proto-article is not injected
>+        for some time after its composition.  However, note that all
>+        implementations predating this specification ignore the
>+        Injection-Date header field and use the Date header field in its
>+        stead for rejecting articles older than their cutoff (see <xref
>+        target="history" />), and injecting agents predating this
>+        specification do not add an Injection-Date header.  Articles with
>+        a Date header field substantially in the past will still be
>+        rejected by implementations predating this specification,
>+        regardless of the Injection-Date header field, and may suffer poor
                                                            ^           ^^^^
                                                          hence         poorer
>+        propagation.</t>
> 
>@@ -478,48 +565,70 @@
> 
>           <t>A proto-article has the same format as a normal article
>+          except that the Injection-Info and Xref header fields MUST NOT
>+          be present; the Path header field MUST NOT contain a "POSTED"
>+          &lt;diag-keyword>; ........

I disagree with forbidding POSTED (already raised in a separate thread),
and am dubious about Xref ("SHOULD" would be quite strong enough).

> 
>           <t>If a posting agent intends to offer the same proto-article to
>+          multiple injecting agents, the header fields Message-ID, Date,
>+          and Injection-Date MUST be present and identical in all copies
>+          of the proto-article.  See <xref target="multi-injection" />.</t>
               ^^^^^^^^^^^^^^^^^
                  it
> 
>+        <section anchor="multi-injection"
>+                 title="Multiple Injection of Articles">
>+          <t>Under some circumstances (posting to multiple disjoint
>+          networks, injecting agents with spotty connectivity, or
                                                                    ^
                                                                   for
>+          redundancy, for example), a posting agent may wish to offer the
>+          same article to multiple injecting agents.  In this unusual
>+          case, the goal is to not create multiple independent articles
>+          but rather to inject the same article at multiple points and let
>+          the normal duplicate suppression facility of Netnews (see
>+          <xref target="history" />) ensure that any given agent only
                                                                   ^^^^
>+          accepts the article once.</t>
                               ^
                              only
>+
>+          <t>Whenever possible, multiple injection SHOULD be done by
>+          offering the same proto-article to multiple injecting agents.
>+          The posting agent MUST supply the Message-ID, Date, and
>+          Injection-Date header fields, and the proto-article offered to
                                                  ^^^^^^^^^^^^^
                                               proto-articles as
>+          each injecting agent MUST be identical.</t>
>+
>+          <t>In some cases, offering the same proto-article to all
>+          injecting agents may not be possible (such as when transferring,
                                                               ^^^^^^^^^^^^
                                                               gatewaying
>+          after the fact, articles found on one Netnews network to
>+          another, unconnected one). .........
            ^^^^^^^^^^^^^^^^^^^^^^^^
            another, supposedly unconnected, one

However, this does not really convey the scenario as it is anticipated to
occur. In essence, the first network is typically small and private and
the second network is usually large, and probably Usenet. And one does not
usually just "find" articles there - one usually placed them there
oneself.

So how about:

            ....... (such as when gatewaying, after the fact, articles
            from a small private network, supposedly with no other
	    connections, to a larger network, such as Usenet). ......

>+          ...   In this case, the posting agent MUST
>+          convert the article back into a proto-article before passing it
>+          to another injecting agent, but it MUST retain unmodified the
>+          Message-ID, Date, and Injection-Date header fields.  It MUST NOT
>+          add an Injection-Date header field if it is missing from the
>+          existing article.  It MUST remove any Xref header field and
                                  ^^^^                              ^^^
                               MAY/SHOULD                           but
>+          either rename or remove any Injection-Info header field and
            ^
           MUST
>+          other trace fields.</t>
                              ^
                   (such as NNTP-Posting-Host and XTrace)

There is also the question of what to do about Path. My preference would
be for "MAY/SHOULD retain"

I think the next paragraph should be a NOTE. There is nothing normative
in it.

>+          <t>Multiple injection inherently risks duplicating articles.
>+          Multiple injection after the fact, by converting an article back
>+          to a proto-article and injecting it again, additionally risks
>+          loops, loss of trace information, and other problems and should
                                             ^
                      unintended reinjection into the same network,
>+          be done with care and only when there is no alternative. ......

In fact, it is usually "unintended reinjection" that is the cause of
"loops".

>+          <t>Multiple injection of an article listing one or more
>+          moderated newsgroups in its Newsgroups header field SHOULD only
>+          be done by a moderator and MUST only be done after the
>+          proto-article is approved for all moderated groups to which it
>+          is posted and has an Approved header field.  .......
              ^                                       ^
            to be                       (see <xref duties-moderator>)

>@@ -644,23 +753,24 @@
> 
>             <t>It MUST reject any proto-article that does not have the
>             proper mandatory header fields for a proto-article; that has
>+            Injection-Info or Xref header fields; that has a Path header
              ^              ^^^^^^^^^^^^^^^^^^^^^  ^^^^^^^^^^^^^^^^^^^^^^
             any                   field
>+            field containing the "POSTED" &lt;diag-keyword>; or that is
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

>+            not syntactically valid as defined by <xref target="USEFOR"
>+            />.  It SHOULD reject any proto-article which contains a
>+            header field deprecated for Netnews.  It MAY reject any
                                                 ^
                                      (see e.g. <xref RFC2298>)
>+            proto-article that contains trace header fields indicating
                                                             ^
                                        (e.g. NNTP-Posting-Host or Xtrace)
>+            that it was already injected by an injecting agent that did
>+            not add Injection-Info or Injection-Date.</t>
> 
>             <t>It SHOULD reject any article whose Date header field is
>             more than 24 hours into the future (and MAY use a margin less
>             than 24 hours).  It SHOULD reject any article whose Date
>+            header is too far into the past (older than the cutoff
                    ^
                  field
>+            interval of a relaying agent the injecting agent is using, for
>+            example), .......

Would that apply to Injection-Date, if any? Or would you allow a stale
Date if there was a plausible Injection-Date?

>+            ...... since not all news servers support Injection-Date
>+            and only the injecting agent can provide a useful error
>+            message to the posting agent.  This interval SHOULD NOT be any
>+            shorter than 72 hours into the past.</t>

Yes, I can accept the 72 hours in that last sentence. I could also accept
omitting the sentence entirely.
> 
>@@ -704,8 +814,14 @@
> 
>+            <t>If the proto-article already had an Injection-Date header
>+            field, it MUST NOT be modified or replaced.  If the
>+            proto-article had both a Message-ID header field and a Date
>+            header field, an Injection-Date header field MUST NOT be
>+            added, since the proto-article may have been multiply injected
>+            by a posting agent that predates this standard.  Otherwise,
>+            the injecting agent MUST add an Injection-Date header field
>+            containing the current date and time.</t>

That paragrpaph is one of the ones that would change if we adopt IC rather
than IR. ALternaitvely, one might consider s/MUST NOT be added/SHOULD NOT
be added/. My concern with the IR option is that it leaves no room for
some future day when everybody routinely implements Injection-Date and it
would be sensible for all injecting agents to add it regardless (unless
already present, of course).
> 
>@@ -801,18 +917,15 @@
>+
>+            <t>It MUST reject any article that has already been
>+            successfully sent to it or that is dated older than its cutoff
>+            date, as described in <xref target="history" />.</t>

No, that won't work. You have not actually defined the term "cutoff date".
But, in any case even providing a "cutoff interval" is not a REQUIRTEMENT.

How about:

>+            <t>It MUST reject any article that has already been
               successfully sent to it, or alternatively where its date
               falls outwith any cutoff interval as described in <xref
               history>.

>@@ -886,16 +999,13 @@
> 
>             <t>It MUST reject any article that has already been
>+            successfully sent to it or that is dated older than its cutoff
>+            date, as described in <xref target="history" />.</t>

Same problem - same cure.

I think some mention of Injection-Date also needs to be made in section
3.9.2 (incoming Gateways), though I am not quite sure what.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Harald Alvestrand <[email-masked]>
Date: Thu, 03 May 2007 09:59:48 +0200

>
> @@ -704,8 +814,14 @@
>              the source of the article and possibly other trace information
>              as described in Section 3.2.8 of <xref target="USEFOR" />.</t>
>  
> -            <t>The injecting agent MUST then add an Injection-Date header
> -            field containing the current date and time.</t>
>              <t>Finally, the injecting agent forwards the article to one or
>              more relaying agents, and the injection process is
>   
I (still) am not convinced of the need for not adding an Injection-Date: 
header when a message-ID and Date is present, but Injection-Date is not.

There are 3 cases:

- Message is singly injected. Injection-Date: does no harm.
- Message is multiply injected, Injection-Date: on all copies are quite 
close to each other, because they all got injected in rapid succession. 
Injection-Date: does no harm.
- Message is multiply injected, and some of the injections got 
significantly delayed.

I think most user agents will add Date: and Message-ID: by default. 
(Anyone have data on this, rather than supposition?)

So the net result of catering for the minority third case is that *most* 
messages on the Net will be sent without an Injection-Date.

If so, what's the point of claiming that we're "mandating" it?

                        Harald

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 03 May 2007 17:28:33 +0200

Russ Allbery wrote:
 
 [a definition of future]
> talking about longer periods, but it does matter at the interval of a

If your implementations get 24 hours right for the two interesting days
in a year I withdraw my objection... :-)  I didn't know that I'd need
the complete timezone stuff to implement a news server.

Frank

---

From: Russ Allbery <[email-masked]>
Date: Thu, 03 May 2007 10:18:22 -0700

> in a year I withdraw my objection... :-) I didn't know that I'd need the
> complete timezone stuff to implement a news server.

For INN, I wrote my own RFC 2822 date parser that converts to seconds
since epoch and deals with various non-RFC-2822 dates that are seen in the
wild, including ones that give a time zone without an offset.

/* Additional non-numeric time zones supported because the old parsedate
   parser supported them.  These aren't legal in RFC 2822, but are supported
   in lax mode. */
static const struct {
    const char name[5];
    long offset;
} OBS_ZONE_OFFSET[] = {
    { "UTC",    0 },                 /* Universal Coordinated */
    { "CUT",    0 },                 /* Coordinated Universal */
    { "WET",    0 },                 /* Western European */
    { "BST",    1 * 60 * 60 },       /* British Summer */
    { "NDT",  (-2 * 60 + 30) * 60 }, /* Newfoundland Daylight */
    { "NST",  (-3 * 60 + 30) * 60 }, /* Newfoundland Standard */
    { "ADT",   -3 * 60 * 60 },       /* Atlantic Daylight */
    { "AST",   -4 * 60 * 60 },       /* Atlantic Standard */
    { "YDT",   -8 * 60 * 60 },       /* Yukon Daylight */
    { "YST",   -9 * 60 * 60 },       /* Yukon Standard */
    { "AKDT",  -8 * 60 * 60 },       /* Alaska Daylight */
    { "AKST",  -9 * 60 * 60 },       /* Alaska Standard */
    { "HADT",  -9 * 60 * 60 },       /* Hawaii-Aleutian Daylight */
    { "HAST", -10 * 60 * 60 },       /* Hawaii-Aleutian Standard */
    { "HST",  -10 * 60 * 60 },       /* Hawaii Standard */
    { "CES",    2 * 60 * 60 },       /* Central European Summer */
    { "CEST",   2 * 60 * 60 },       /* Central European Summer */
    { "MEZ",    1 * 60 * 60 },       /* Middle European */
    { "MEZT",   2 * 60 * 60 },       /* Middle European Summer */
    { "CET",    1 * 60 * 60 },       /* Central European */
    { "MET",    1 * 60 * 60 },       /* Middle European */
    { "EET",    2 * 60 * 60 },       /* Eastern European */
    { "MSK",    3 * 60 * 60 },       /* Moscow Winter */
    { "MSD",    4 * 60 * 60 },       /* Moscow Summer */
    { "WAST",   8 * 60 * 60 },       /* Western Australian Standard */
    { "WADT",   9 * 60 * 60 },       /* Western Australian Daylight */
    { "HKT",    8 * 60 * 60 },       /* Hong Kong */
    { "CCT",    8 * 60 * 60 },       /* China Coast */
    { "JST",    9 * 60 * 60 },       /* Japan Standard */
    { "KST",    9 * 60 * 60 },       /* Korean Standard */
    { "KDT",    9 * 60 * 60 },       /* Korean Daylight (no change?) */
    { "CAST",  (9 * 60 + 30) * 60 }, /* Central Australian Standard */
    { "CADT", (10 * 60 + 30) * 60 }, /* Central Australian Daylight */
    { "EAST",  10 * 60 * 60 },       /* Eastern Australian Standard */
    { "EADT",  11 * 60 * 60 },       /* Eastern Australian Daylight */
    { "NZST",  12 * 60 * 60 },       /* New Zealand Standard */
    { "NZST",  13 * 60 * 60 },       /* New Zealand Daylight */
};

Whee.

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 4 May 2007 11:50:35 GMT

>header when a message-ID and Date is present, but Injection-Date is not.

Then you are essentially supporting option IC rather than option IR.

>There are 3 cases:

..........

>- Message is multiply injected, and some of the injections got 

That, in combination with legacy user agents which do not insert an
Injection-Date when multiply injecting, is what may cause a later relaying
agent to re-accept an article it has already accepted and expired once
before (which would not have happened under the present arrangements).

That is Russ's argument for Option IR. I am not convinced, on the grounds
that it will be a very rare occurrence and causes little harm (someone
finds himself reading an article he had already read a week earlier).

>(Anyone have data on this, rather than supposition?)

I think most will add Date, but fewer will add Message-ID.

>messages on the Net will be sent without an Injection-Date.

Indeed, which will mean a slower adoption of it, but also an anomaly which
will remain long after it eventually is in universal use.

---

From: Russ Allbery <[email-masked]>
Date: Fri, 18 May 2007 18:06:54 -0700

> points are minor textual niggles, but some are quite substantive.

Changes snipped without remark have been accepted and made in my draft
text.

> of even shorter intervals}

I couldn't resist wordsmithing this a bit, but I think I retained the
intent.  I now have:

   o  Alternatively, agents MAY drop history records according to the
      date when the article was first seen by that agent rather than the
      date of the article.  In this case, the history retention interval
      MUST be at least 24 hours longer than the cutoff interval to allow
      for articles dated in the future.  This interval matches the
      allowable error in the date of the article (see Section 3.5).

> 	  experience with Netnews and the subtle effects of its flood-fill

I now have:

   These are just two implementation strategies for article history,
   albeit the most common ones.  Relaying and serving agents are not
   required to use these strategies, only to meet the requirement of not
   accepting an article more than once.  However, these strategies are
   safe and widely deployed and implementors are encouraged to use one
   of them, especially if they not have extensive experience with
   Netnews and the subtle effects of its flood-fill algorithm.

going with explaining why this is a good idea rather than just emphasizing
it.

> and am dubious about Xref ("SHOULD" would be quite strong enough).

I'm not sure we've reached a conclusion on those other threads yet.  So
far, I've not made any changes here.

>> +          of the proto-article.  See <xref target="multi-injection" />.</t>

This was intentional.  It reads awkwardly, but it's clearer.  "It" is
somewhat ambiguous and I kept stumbling over thinking it referred to the
posting agent for a moment.

>> +          Injection-Date header fields, and the proto-article offered to

I'm intentionally not pluralizing proto-article, since the whole point is
that there's only one of them, sent to multiple locations.  Adding "as"
does make the sentence read better to me, though, so I did that.

>> +          injecting agents may not be possible (such as when transferring,

Okay.

> 	    connections, to a larger network, such as Usenet). ......

Well, that's one of the cases where offering the same proto-article to all
injecting agents *is* possible, normally, so I'm not sure it's a great
example.  That's the use case where I really wish people would just use
normal multiple injection.

> be for "MAY/SHOULD retain"

As discussed above, I haven't made any changes here.

Likewise here.

>                                       (see e.g. <xref RFC2298>)

RFC 2298 is obsolete, so I don't really want to generate even an
informative reference to it, and the superseding RFC (RFC 3798) makes no
mention of Netnews, so I'm not sure this is a great example or
particularly helpful for the reader.

>                                         (e.g. NNTP-Posting-Host or Xtrace)

Added just NNTP-Posting-Host, since USEFOR already mentions and explicitly
obsoletes it.

> Date if there was a plausible Injection-Date?

Covered in another thread.

>                falls outwith any cutoff interval as described in <xref

I now have:

   3.  It MUST reject any article that has already been successfully
       sent to it, which may include rejecting articles whose dates fall
       outside a cutoff interval as described in Section 3.3.

---

From: Russ Allbery <[email-masked]>
Date: Fri, 18 May 2007 18:09:36 -0700

> close to each other, because they all got injected in rapid
> succession. Injection-Date: does no harm.

Right, the problems only potentially happen in the third case, but that
third case includes many serial injection cases.

> I think most user agents will add Date: and Message-ID: by
> default. (Anyone have data on this, rather than supposition?)

Charles had some data, I believe.  If I remember it correctly, "most" is
iffy, but many do.

> messages on the Net will be sent without an Injection-Date.

Correct, until posting agents are updated to generate it.  This is the way
that I think it should work, since all the benefits of Injection-Date are
born by the posting agents, not by the injecting agents (which don't care
at all).

> If so, what's the point of claiming that we're "mandating" it?

We're mandating checking it instead of Date so that reading agents can
start using the feature.

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Fri, 18 May 2007 23:08:08 -0400

>        outside a cutoff interval as described in Section 3.3.

This changes the meaning significantly.  Is that right?

---

From: Russ Allbery <[email-masked]>
Date: Fri, 18 May 2007 20:47:17 -0700

> This changes the meaning significantly.  Is that right?

Probably not.  I'm failing to find the right wording here.

News servers MUST reject articles that have already been successfully sent
to them.  They're given an allowance to reject messages that have not
already been successfully sent to them as part of this if that's needed
for their algorithm (which doesn't change the protocol, since they can
always reject more articles for any reason at all).  *If* the server uses
the schemes in 3.3, *then* they must reject articles whose dates lie
outside the cutoff interval.

Suggestions for wording that captures this?

---

From: Frank Ellermann <[email-masked]>
Date: Sun, 20 May 2007 00:51:21 +0200

Russ Allbery wrote:

>>> It MUST reject any article that has already been successfully
 
> Suggestions for wording that captures this?

s/which may include/and this typically requires/ or similar.

Frank

---

From: Russ Allbery <[email-masked]>
Date: Sat, 19 May 2007 17:03:08 -0700

> s/which may include/and this typically requires/ or similar.

That sounds pretty good to me.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 21 May 2007 16:30:12 GMT

>This changes the meaning significantly.  Is that right?

Not really. The second half of the sentence only applies IF the agent
maintains a "cutoff interval" (in practice, they all do, but if they don't
then neither wording places any REQUIREMENT on them).

But I think Frank's suggested wording covers it well.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 21 May 2007 16:27:29 GMT

>Correct, until posting agents are updated to generate it.  ...

No, it's worse than that. As currently worded, that situation will persist
FOR EVER, even after every agent on the Net implements Injection-Date
fully.

This is essentially the difference between options IR and IC, where IC
allows that things may temporarily get slightly worse in order that the
final steady-state situation will be cleaner.

We need to resolve the IR vs IC issue soon, but I think it best to get all
the other aspects of Injection-Date sorted out, which we now seem to be
well on the way to doing.

> ...This is the way

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 21 May 2007 16:12:47 GMT

>far, I've not made any changes here.

OK, that is an issue still waiting resolution. At least all the places
affected are now documented.

>normal multiple injection.

Yes, but the case arises where the small private network contains several
sites some of which, by misunderstanding, misconfiguration, or the result
of some anticipated cross-posting also inject the article (so the
"official" injecting channel does not get the option of multi-injecting it
properly, even if he wanted to).

So I think such examples need mentioning, but maybe in different words.
The problem with "small private networks" is that they tend to grow
without the knowledge of the network administrator :-( .

>particularly helpful for the reader.

No, RFC 3798 still contains the relevant wording:

   Messages posted to newsgroups SHOULD NOT have a Disposition-
   Notification-To header.

and we need to draw attention to that (or, alternatively, explicitly
prohibit that header ourselves). The problem is that use of that header in
Netnews can give rise to the Mother of all Mailbombs. This is also a
Security Consideration, and I think it is also mentioned as such (or
should be).

I am not aware of any other headers that are explicitly deprecated for
Netnews by other standards, but there might be.

---

From: Russ Allbery <[email-masked]>
Date: Mon, 21 May 2007 21:55:23 -0700

Charles Lindsey <[email-masked]> writes:

> No, it's worse than that. As currently worded, that situation will
> persist FOR EVER, even after every agent on the Net implements
> Injection-Date fully.

I don't see how this would be the case.  Either posting agents add
Injection-Date or they don't add Date and Message-ID and the injecting
agent does.

---

From: Russ Allbery <[email-masked]>
Date: Mon, 21 May 2007 21:59:57 -0700

Charles Lindsey <[email-masked]> writes:
> Russ Allbery <[email-masked]> writes:

>    Messages posted to newsgroups SHOULD NOT have a Disposition-

Ah, newsgroups.  That's one of the keywords I didn't search for.

Added to my draft text.

> The problem is that use of that header in Netnews can give rise to the
> Mother of all Mailbombs. This is also a Security Consideration, and I
> think it is also mentioned as such (or should be).

I think it's overkill to be that worried about an obscure header that no
one has ever implemented for Netnews.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 22 May 2007 05:41:31 +0200

> Suggestions for wording that captures this?

Going to 2 sentences would help me, I think.

  It MUST reject any article that has already been successfully sent to it.
  If it implements the mechanism described in <xref history>, this means 
that it
  will have to reject any article where its date falls outside a cutoff 
interval,
  since it doesn't know whether it has been successfully sent to it or not.

If shorter text is desired, drop the part after the comma.

                Harald

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Tue, 22 May 2007 08:01:21 -0400

> If shorter text is desired, drop the part after the comma.

I really like that this expresses the idea that if you can't know if you 
accepted it or not, you must reject.

I would wordsmith:

	successfully sent to it -> accepted.

(There have been some propagation algorithms which trust articles from certain 
peers, but not others.)

And I don't have a strong opinion, but should this part also change:
	will have to reject -> MUST reject.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 22 May 2007 15:22:23 GMT

>one has ever implemented for Netnews.

I think the problem is combined mail/news user agents, or which there are
a lot around. The naive user is likely to configure his agent to always
include this header ("because it seems like a Good Idea"), and for the
badly written user agent (or which, again, there are many) not to notice
that the option is not supposed to be activated for News.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 22 May 2007 15:18:31 GMT

>Injection-Date or they don't add Date and Message-ID and the injecting

Yes, but I am not expecting posting agents to be routinely adding
Injection-Date (except in the special case where they are
multi-injecting). My view is that it is really the injecting agent's job,
and that injecting agents are more likely to do it right than posting
agents in the hands of unskilled users.

---

From: Russ Allbery <[email-masked]>
Date: Tue, 22 May 2007 09:42:22 -0700

> multi-injecting).

Agents that add Date and Message-ID should also add Injection-Date.  I'd
be happy to add wording to that effect.

> My view is that it is really the injecting agent's job, and that
> injecting agents are more likely to do it right than posting agents in
> the hands of unskilled users.

The current protocol lets posting agents add Date, which has the same
protocol role.  Having the posting agent add Injection-Date doesn't make
matters any worse than it is now.

---

From: Russ Allbery <[email-masked]>
Date: Tue, 22 May 2007 09:41:03 -0700

Charles Lindsey <[email-masked]> writes:

> I think the problem is combined mail/news user agents, or which there
> are a lot around. The naive user is likely to configure his agent to
> always include this header ("because it seems like a Good Idea"), and
> for the badly written user agent (or which, again, there are many) not
> to notice that the option is not supposed to be activated for News.

This header has been around for years.  Has this ever happened?  If it
has, I have no objections to mentioning it in Security Considerations, but
if it hasn't, it seems like wasted verbiage to me.

Including it as an example of a deprecated header is certainly fine.

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 23 May 2007 10:07:32 GMT

>be happy to add wording to that effect.

That would certainly help if Option IR is to be retained, but it all gets
rather messy. Option IC is much cleaner, even though it may occasionally
make things worse in the short term.

>matters any worse than it is now.

My worry is that poor implementations will just create Injection-Date as a
copy of Date, which defeats the whole purpose. Your wording would need to
make it very clear that that is not good enough (and why).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLEFrom [email-masked] Wed May 23 07:15:25 2007
To: [email-masked]
Xref: clerew local.usefor:24698
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: #1416 Injection-Date: current wording proposal (corrected)
Date: Wed, 23 May 2007 10:07:32 GMT
Lines: 34
Sender: [email-masked]

>be happy to add wording to that effect.

That would certainly help if Option IR is to be retained, but it all gets
rather messy. Option IC is much cleaner, even though it may occasionally
make things worse in the short term.

>matters any worse than it is now.

My worry is that poor implementations will just create Injection-Date as a
copy of Date, which defeats the whole purpose. Your wording would need to
make it very clear that that is not good enough (and why).

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 23 May 2007 10:02:01 GMT

>if it hasn't, it seems like wasted verbiage to me.

Not that I know of. But the inventors of that header wisely included a
SHOULD NOT use it in News, and we should either invoke that by reference,
or specify it explicitly. And it is a good idea for injecting agents to
check for its absence, and for sure it needs a mention in the Security
Considerations.

, SK8 3JU, U.K.

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 23 May 2007 10:02:01 GMT

---

From: Russ Allbery <[email-masked]>
Date: Wed, 23 May 2007 20:05:23 -0700

> (There have been some propagation algorithms which trust articles from
> certain peers, but not others.)

I now have:

            <t>It MUST reject any article that has already been accepted.
            If it implements the mechanism described in <xref
            target="history" />, this means that it MUST reject any
            article whose date falls outside a cutoff interval since it
            won't know whether such articles had been accepted or not.</t>

since indeed I think that rejecting such articles is a MUST *if* the
server is using that algorithm.

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 24 May 2007 08:25:23 GMT

>            article whose date falls outside a cutoff interval since it
                                              ^
                                             the
>            won't know whether such articles had been accepted or not.</t>
                                                               ^
                                                         earlier/previously