Subject: #1416 Reinjection - an attempted summary, and a suggested resolution

From: Harald Alvestrand <[email-masked]>
Date: Wed, 14 Feb 2007 09:02:36 +0100

So far, my buffer shows:

42 postings on the "#1416" thread, Jan 19 to Feb 7, including Charles' 
initial issue mail
Active participants:
- Charles Lindsey
- Russ Allbery
- Forrest J. Cavalier
- Frank Ellermann
- Harald Tveit Alvestrand
- Richard Clayton

Relevant technical information from the thread:

- Injecting a message (with an unique Message-ID:) several times into a 
single Usenet network is supported, and it's important that it remains 
supported ("multipoint injection").
- Current USENET servers (when acting as relaying agents) check Date: 
for staleness (rejecting articles that have a Date: that is so old that 
their history files won't cover the message-ID).
- Injection of articles into USENET where the client posting agent sends 
to someone acting as an injecting agent, and the article then ending up 
with someone acting as a posting agent towards another injecting agent 
("reinjection from an isolated network"), is quite common.
- No technical means exists today for verifyng whether a network is 
truly isolated.
- No existing mail-to-news gateways modify Date:

There was some report of an experiment on propagation with different 
Date: headers, but I've not been able to find that in my archive. Bad 
choice of keywords to search for, I assume.

Suggested hacks include:
- Changing Date: at injection to be within a short distance from the 
current time
- Allowing posting agent to set Injection-Date:, to ensure it's 
consistent between injected copies
- Drop Injection-Date: from the spec

I've probably dropped some important pieces of information, but I've 
seen the above.

--------------------------------------------------------------------------------------------------------------------------
PROPOSED RESOLUTION:

This is very much on the "Keep It Simple" approach to protocol design.

- Date: identifies the date of composition. Don't change it, ever.
- Injection-Date: identifies the date of injection, by the last agent 
that identifies itself as an injecting agent. Proto-articles don't have 
it. Relaying and serving agents never, ever change it.
- "Reinjection" doesn't exist. There is only multipoint injection. The 
means by which a proto-article (including message-ID, but NOT including 
Injection-Date:) gets from wherever it started out to the posting agent 
is not defined by the protocol; if the protocol happens to be NNTP, 
that's not an issue for this specification. (we can add warnings...)
- When doing multipoint injection, Message-ID: MUST be consistent in all 
copies. Date: SHOULD be consistent, if it's not, Bad Things Can Happen.

On the stale article problem:

- We state that existing implementations use Date: to check for 
staleness, and that will cause discarding of articles that are too old. 
Period, end of story, accepted.
- We recommend (require?) that injecting agents refuse posting of "too 
old" articles by Date: - the exact acceptable Date: range should be a 
local decision.
- We recommend that the staleness check use Injection-Date: if present, 
otherwise Date:
- We state that if stale article rejection is an issue for people, they 
should:
  - Ensure that their injecting agent adds Injection-Date:
  - Put pressure on the discarding agent to check Injection-Date: if present

Case analysis:
If a message has Date: on day 0, and is injected on day 7, into a 
network where anything older than 3 days is considered stale, but the 
injecting agents accept 14-day-old proto-articles, this will happen:

- Servers that check Injection-Date: will relay the message. More 
propagation than today.
- Servers that check Date: will reject the message. Unchanged from today.
- Servers that feed from a server that checks Date: will not get the 
message in the normal feed; they may get it offered from feeds that go 
via Injection-Date: checking servers.

If a message has Date: on day 0, gets injected on day 1 into the 
network, and gets injected again on day 7 into the same network, the 
following will happen:

- Servers that check Date: will see the first copy.
- Servers that check Injection-Date: will relay the message twice, 
possibly causing users to see it twice.

The last looks like a loop, but can be solved by sysadmins beating up on 
the injecting agent to enforce stricter policing of posting of old articles.
Is the latter a Big Deal?

                            Harald

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 19 Feb 2007 15:45:58 GMT

>choice of keywords to search for, I assume.

Essentially, I established that some injecting agents impose an arbitrary
cutoff (e.g. 4 days) as a matter of site policy, and some will accept any
degree os staleness (but whether they then succeed in relaying/propagating
them is another matter).

Also, it seems to be the case that articles up to 8 days stale, once
accepted by an injecting agent, seemed to propagate, and to do so without
much degradation of speed or by using noticeably different routes. But I
have insufficient data to give a definitive answer as to whether these
observations are typical (now, if Google reported arrival times, or
retained the Path, such experiments would be much easier :-( ).

>consistent between injected copies

Yes, Russ suggested that, and I agree - modulo various niggles about what
constitutes a 'posting agent' in this particular context.

>- Date: identifies the date of composition. Don't change it, ever.

Yes.

>it.

Almost. It might have been put there by a posting agent (see 'suggested
hacks above) at the time of posting (Note: that does not mean that the
proto-article has it).

> Relaying and serving agents never, ever change it.

Yes. And likewise gateways (if the outgoing medium will allow it).

>- "Reinjection" doesn't exist.

You can say that, but it will not prevent it from happening :-( . And
sometimes it is even sensible. And safe if the Injection-Date is retained
(there _may_ be some special cases where not retaining it might be OK, but
those need careful thinking about).

> There is only multipoint injection. The 

Eh? Surely the prime duty of a posting agent is to generate
proto-articles. What the protocol does not define is whether a
posting-agent maintains a queue of stuff waiting to be injected (lots of
them do, and those are the ones which might be allowed to insert
Injection-Date). OTOH, "normal" posting agents will not be doing
multipoint injection; that is more likely to arise from specially written
scripts.

>copies. Date: SHOULD be consistent, if it's not, Bad Things Can Happen.

And likewise Injection-Date.

>Period, end of story, accepted.

Yes.

>old" articles by Date: - the exact acceptable Date: range should be a 

Disagree. That is a matter of site policy. It is the relayer that should
make that choice, and hopefully it will be using Injection-Date for that
putpose. OK, if the injecting agent knows that its immediately following
relaying agent is going to reject, then it would be couteous to reject it
itself (but equally well, if could look at Injection-Date itself). As I
said, this is all a matter of site policy.

>- We recommend that the staleness check use Injection-Date: if present, 

Yes. REQUIRE.

>  - Ensure that their injecting agent adds Injection-Date:

Or add it themselves (e.g the man who carries his server around on his
laptop).

>  - Put pressure on the discarding agent to check Injection-Date: if present

Yes.

However, I think we should wait until Russ has produced the revised
proposal that he promised a couple of days back.

>via Injection-Date: checking servers.

All correct. Injecting that late is currently a bit dodgy. It will improve
as more servers are upgraded to use Injection-Date.

>possibly causing users to see it twice.

But not if whoever reinjected on day 7 kept the original Injection-Date
(which is what I claim SHOULD/MUST be the normal practice - though a few
exceptions might be worth looking into).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 19 Feb 2007 20:05:28 +0100

I see that you did not understand what I proposed, so I'll make the 
essential points more clear:

- Multipoint injection consists of the same article being posted 
multiple times on the same network.
There is no presumption that those injections happen at the same time, 
or are being done by the same agent.

- In my proposal, injection-date: identifies the time of injection. 
Period, end of story. It is, in my opinion, harmful and of no benefit to 
let the posting-agent specify this. The injecting agent knows its own 
current time, and there is no later agent that is more trustworthy to 
leave the decision to.

- What you are calling "reinjection" is what I call "multipoint 
injection, where one of the points listens to a Netnews network to get 
hold of the proto-article".

I have yet to understand any tangible benefit from your proposal to 
allow the posting-agent to specify injection-date.

                  Harald

---

From: Russ Allbery <[email-masked]>
Date: Mon, 19 Feb 2007 23:29:10 -0800

> - Current USENET servers (when acting as relaying agents) check Date: for
> staleness (rejecting articles that have a Date: that is so old that their
> history files won't cover the message-ID).
> - No technical means exists today for verifyng whether a network is truly
> isolated.

Yup, all of those technical points look correct to me.

> - Allowing posting agent to set Injection-Date:, to ensure it's consistent
> between injected copies

Also there's the current draft, which sets Injection-Date at injection to
the current time but leaves Date alone.

> - Injection-Date: identifies the date of injection, by the last agent that
> identifies itself as an injecting agent. Proto-articles don't have

So far, this is what's in the current spec.

> Injection-Date:) gets from wherever it started out to the posting agent is
> not defined by the protocol; if the protocol happens to be NNTP, that's
> not an issue for this specification. (we can add warnings...)

As long as Injection-Date is in the specification, keeping Date consistent
is mostly meaningless, since our specification says that the header field
that goes into the identity of the article isn't Date, it's
Injection-Date.  Any reason you have for keeping Date consistent, in the
current draft, applies to Injection-Date instead, since we're replacing
the protocol use of Date with Injection-Date.

So this proposal as stated doesn't make sense to me, since it talks about
keeping Date consistent when the only reason to keep Date consistent
(beyond generally not wanting to modify articles and keeping the human
information component) is something that we're phasing out.

Anyway, your summary is basically the current draft.  It doesn't address
the identified problem in the current draft that different de facto
identities are given to the different copies of multiply injected
articles, which can potentially result in duplicates (with the chances
increasing as the time difference between the two injections increases).
I can't tell from your note whether that was intentional and you just
don't consider this much of a problem, or whether you'd missed that aspect
of the problem.

For the rest, annotating te proposal to make it clearer what the
differences are from the current draft.

> staleness, and that will cause discarding of articles that are too
> old. Period, end of story, accepted.

Currently we don't say this for injecting and serving agents.

> old" articles by Date: - the exact acceptable Date: range should be a

This is what the draft currently has.

> - We recommend that the staleness check use Injection-Date: if present,

This is what the draft currently has, except that it's mandatory, not just
recommended.

>  - Put pressure on the discarding agent to check Injection-Date: if present

We don't currently say anything about this, but it's to a degree implicit
in the Injection-Date support.

> Case analysis:
> If a message has Date: on day 0, and is injected on day 7, into a network
> where anything older than 3 days is considered stale, but the injecting
> agents accept 14-day-old proto-articles, this will happen:

> message in the normal feed; they may get it offered from feeds that go via
> Injection-Date: checking servers.

Agreed.

One of the reasons why I believe injecting agents SHOULD continue to check
Date for staleness is that I believe no propagation plus a clear error
message is better than spotty propagation and no diagnostics.  I believe,
from other messages, that Charles disagrees and would prefer spotty
propagation improving as (hopefully) more hosts support Injection-Date.
The disagreement may to some extent depend on one's beliefs about how fast
Injection-Date adoption will happen.

> If a message has Date: on day 0, gets injected on day 1 into the network,
> and gets injected again on day 7 into the same network, the following will
> happen:

> Is the latter a Big Deal?

Well, it's a Big Deal in the sense that it's happened in the past with
agents that discard Date, it's far more likely to happen when ignoring
Date becomes the standard way the servers are supposed to behave, and
every time it's happened that I've seen it caused a big stink and people
got rather upset that the software let it happen.

It's not the sort of loop that melts down a newsgroup, so it's not as big
of a deal in that sense.  The loop has to be time-delayed.  But it does
feel like a fairly fundamental failure of the protocol to me.

I think the main alternative (assuming we're keeping Injection-Date) looks
like this:

 * Servers are required to check Injection-Date in preference to Date for
   determining staleness.

 * Posting agents may provide Injection-Date in proto-articles.  They MUST
   do so if doing multiple injection.  If Injection-Date is provided, the
   injecting agent MUST NOT modify it.

 * If the proto-article contains neither a Date nor an Injection-Date
   header field, the injecting agent MUST add both.

 * If the proto-article contains only a Date header but no Injection-Date
   header, it's using the old article identity model.  If multiple
   injection is done, Date is provided, and no Injection-Date header is
   ever added, everything works as it does now.  If multiple injection is
   done, Injection-Date is provided, and all agents check Injection-Date,
   everything works as it does now.  We have to keep things into one of
   those two cases as much as possible to maintain the existing guarantees
   of the protocol, so I think an injecting agent, when presented with a
   proto-article that contains a Date but no Injection-Date, must assume
   the old identity model is being used and MUST NOT add Injection-Date.
   This means that people who want to send time-delayed messages will need
   to update their posting agents to add Injection-Date.  I think that's
   reasonable.

Remaining issues are to what degree we want to standardize serial multiple
injection -- I think it may be worthwhile describing the specific steps
one takes to turn an article into a proto-article somewhere -- and whether
we need to deal with stale pre-existing Date/Injection-Date headers or
just ignore the issue and let people make it up as they go if they end up
in a situation where stale Date/Injection-Date headers are preventing them
from doing some gatewaying they want to do.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: Russ Allbery <[email-masked]>
Date: Mon, 19 Feb 2007 23:33:50 -0800

Harald Alvestrand <[email-masked]> writes:

> - In my proposal, injection-date: identifies the time of
> injection. Period, end of story. It is, in my opinion, harmful and of no
> benefit to let the posting-agent specify this. The injecting agent knows
> its own current time, and there is no later agent that is more
> trustworthy to leave the decision to.

Well, I don't think you can say that it's of no benefit.  It's clearly of
benefit in establishing a consistent identity for the article, which will
then prevent loops or duplicates from appearing.  Whether that benefit is
worth the other costs is another question, but it's not non-existent.

> hold of the proto-article".

I think that's a reasonable simplification of terms.

> allow the posting-agent to specify injection-date.

I tried to address this (at some length, unfortunately) in my message
about the identity model of an article.  The advantage is that it lets the
posting agent specify the complete identity of the article, and therefore
multiply injected copies of the same article can all have the same
identity when that injection is done by a well-behaved posting agent.

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Tue, 20 Feb 2007 08:16:46 -0500

> Currently we don't say this for injecting and serving agents.

I think it is important to note this. It is important to document how it works 
today. The discarding happens at relays independently.  Some sites see the 
article, some do not.  That's worse than refusing it at the point of
injection.

We should not suggest a specific number of days cutoff, since the typical value 
has changed, and is not uniform across sites.

Would it be appropriate to write this as "In the case of delayed injection, 
existing relaying and serving agents' article staleness checks on Date and not 
Injection-Date will lead to non-uniform propagation. If injection is delayed, 
the posting agent MAY [also] place the time of injection in the Date: header field."

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 23 Feb 2007 16:45:28 GMT

>article, some do not.  That's worse than refusing it at the point of

It's not as bad as that, since if Site A refuses to relay it, it will
likely flood around A, and still arrive (but not as rapidly) at sites
beyind A. So it will be A's direct clients who will not be able to read
the article.

But yes, it would be useful to note this present practice. But when
Injection-Date is widely implemented, things will be better (not that any
vast number of articles actually get lost this way, even at present).

Agreed.

>the posting agent MAY [also] place the time of injection in the Date: header field."

I would not want to encourage this. A few current agents may be doing
this, and let them continue. But I would rather then moved to using
Injection-Date for such situations.