Subject: ISSUE: Reinjection and Injection-Date

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 16 Jan 2007 12:22:37 GMT

ISSUE Number requested.

This ISSUE concerns whether injecting agents should remove or preserve an
already present Injection-Date header (it is agreed that they should
replace any pre-existing Injection-Info).

There is a separate issue concerning whether injecting agents should
reject articles that are more than 72 hours stale according to the Date
header, but I will raise a separate ISSUE on that one.

There are essentially two options:

Option 1: Injection-Dates are ALWAYS rewritten by (re-)injecting agents.

Option 2: Injection-Date headers, once written, are NEVER altered.

I prefer Option 2, on the grounds that Option 1 is unsafe in the presence of
unanticipated leaks.

Further details follow:

----------------------------------------------------------------------------

The purpose of the Injection-Date header is to allow relaying and serving
agents to detect articles that may have passed through them previously,
but which may have expired from their history file (some relaying agents
expire their history file in as little as 3 days). This check is important
because, if not made, there are scenarios where the same article might
then be propagated with further delays and then return to the same agent
again and again.

In the absence of an Injection-Date header, the Date-header is to be used
(for interoperability with existing software). However, the purpose of
introducing Injection-Date was to allow Date to be used for the time of
composition of the article, which might predate its injection by many days.

It is to be noted that Usenet propagation is nowadays so rapid that only
exceptionally will this check be called into play. Nevertheless, unusual
setups with delayed paths can exist in a variety of ways (gatewaying,
private networks, etc). If the examples shown below appear improbable and
far fetched, remember that in a network of hundreds of thousands of
servers and millions of end nodes even the most outrageous scenario is
bound to occur sooner or later.

Option 1: Injection-Dates are ALWAYS rewritten by (re-)injecting agents.

We are concerned with an article that is being reinjected. Let us refer to
its two incarnartions as FIRST and SECOND and the person/entity doing the
reinjection as the REINJECTOR. For this option to be safe, it is ESSENTIAL
that FIRST and SECOND were injected into totally disjoint networks, with
absolutely no possibility of leaks between them, and it is the absolute
responsibility of the REINJECTOR to ensure that this condition is met.

Where the REINJECTOR is a conventional incoming gateway (mail2news,
fido2news, etc) it is argued (see 3.9.2) that they already MUST be taking
separate precautions (e.g. tagging) to prevent loops, and hence loss of
the Injection-Date when reinjected into Usenet can be tolerated (though,
oddly, 3.9.1 still recommends preservation of Injection-Date - wherever it
is possible - for outgoing gateways). However, it is also arguable that
the consequences of looping are so severe that having more than one
precaution against it is always a Good Idea.

But the real problems arise when FIRST is injected in a small private
network (even a private news server maintained by an individual poster).
Although it is argued that such a network should be regarded as gatewaying
into the network where SECOND is to be propagated, it is far from certain
that the owner of such a network or such an individual poster will regard
himself as a "gatewayer", or that he will believe that section 3.9 applies
in any way to him. Such small private neworks are notorious for "leaking"
(and it is not necessarily the "owner" of the network who will be the
REINJECTOR that perpetrates the leak).

The only really safe situation is where the REINJECTOR is an individual
poster who has total control of the one and only machine on which the
private network exists. As soon as you escalate to a small organization
with several users, and then to a loose confederation of cooperating
hosts, the possibility of the owner or the REINJECTOR being in a position
to give the required absolute guarantee rapidly recedes.

Consider the following scenario. Suppose the Anytown Squash Club sets up a
private network with a newsgroup anytown.squash, for discussing meetings,
matches, court bookings, and general chat. It is totally informal, uses
dialup copnnections and UUCP, and is barely reliable (propagation can be
slow, especially when some machine is down, or its owner forgets to dialup
as often as he is supposed to). So propagation times of three days are not
uncommon. Who cares?  Everybody knows it is a private network, and that
the 'anytown' hierarchy does not exist anywhere else, so nobody bothers
about security or leaks.  Maybe one or two of the members are also users
of Usenet (how else did they get the idea of using Netnews, and they only
used UUCP because nobody had access to a permanently connected machine to
run an NNTP server). Amateurish and shambolic.

And then, one day, one of the members wanted to raise an issue that went
beyond the mere politics of squash in Anytown, and so he had a bright
idea. "I will crosspost this article to rec.sport.squash". And three days
later his article reaches another member who also had a usenet connection
and was subscribed to rec.sport.squash. Need I say any more?

Option 2: Injection-Date headers, once written, are NEVER altered.

This clearly removes the risks inherent in Option 1. It is inherently much
safer, and it is 'fail-safe' in the sense that the only thing that can go
wrong is that an article whose propagation was genuinely delayed at some
point might fail to propagate beyond a relaying agent with a short
history file expiry. But better to lose a few articles than run the risk
of loops.

Moreover, we MIGHT allow exceptions to this rule in carefully defined
circumstances (whether we define them explicitly in the draft, or leave it
to those who really REALLY know what they are doing to break the rules
when they know it is safe). The only situation I can think of is where
there is a private server run on a single machine whose owner writes an
article and then fails to (re)inject it for several days (like the server
was actually on his laptop). So he is indeed a REINJECTOR who can provide
that absolute guarantee. Moreover, it will always be the case that the
owner of such a private server can in any case make it appear to his
injecting agent that it is really just a user agent (even though it is
internally organized otherwise).

The only scenario that has been suggested which causes problems with this
option is where some (small and local) server/network B, which takes
articles from a well-connected server A, is offline for a week following
some power failure and then seeks to catch up by reading all the missed
articles from A. Note that it only requires these articles for local use -
any role it might normally have had in relaying articles onwards to other
Usenet sites is irrelevant, since these articles would already have
propagated to those other sites by other means.

So it sucks the missing articles and reinjects them into itself,
preserving their injection dates as is required by this option, and hence
possibly triggering the local staleness check.

But since B wants the articles for its own use (not for relaying back into
Usenet), it is presumably a serving agent, in which case its expiry time
is likely to be more than a week, in which case the staleness check is
unlikely to be triggered. But even if that is not so, this is a case where
all B's admin needs to do is to temporarily increase the expiry time of
his history file. This is always safe; even if some site on his local
network tries to relay an article back into Usenet, it is still protected
by its _original_ Injection-Date. The worst that can happen is that a few
articles that had already been received just before his power failure
might be received again (some small error in the time given in his NEWNEWS
command, perhaps); that seems like an acceptably small blip in what is
likely to be a pretty traumatic recovery process.

So what we have to consider is whether the problems encountered in
recovering B's server (which will be localized to his own local users) are
worse or better than the problems caused by the poor design of the Anytown
Squash Club's setup (which may be evident throughout Usenet).

And also whether a reasonable service can be provided to a user who
carries an article around on his laptop for a week before injecting it
into Usenet, and whether it is reasonable for him to omit or "fiddle" his
Injection-Date in that sort of situation (and also whether we would
explicitly mention such small "cheats" in out draft - for sure, that user
has to bear the responsibility if he gets it wrong).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Russ Allbery <[email-masked]>
Date: Thu, 18 Jan 2007 21:27:01 -0800

> Option 2: Injection-Date headers, once written, are NEVER altered.

Just to try to be painfully clear here, Injection-Date using the method
laid out in the current USEPRO draft is *not* rewritten by injecting
agents.  The whole point of that option is that only the gateways who are
doing the reinjection make this transformation.

The current draft doesn't define a "reinjecting agent" exactly, but such
an agent would be a gateway and posting agent, *not* any sort of injecting
agent.

> We are concerned with an article that is being reinjected. Let us refer
> to its two incarnartions as FIRST and SECOND and the person/entity doing
> the reinjection as the REINJECTOR. For this option to be safe, it is
> ESSENTIAL that FIRST and SECOND were injected into totally disjoint
> networks, with absolutely no possibility of leaks between them, and it
> is the absolute responsibility of the REINJECTOR to ensure that this
> condition is met.

Lack of safety can only occur if the networks are not disjoint *and* there
is a delay in gatewaying longer than the history retention of whatever
path causes the networks to not be disjoint (whether a relaying agent or
another gateway) but shorter than the staleness check in the reinjecting
gateway (*and* no staleness checks are applied to the Date header field,
but we can all agree on that as a long-term goal).

Observe that even in this situation, the problem can only happen once per
article.  In other words, the worse case scenario is for a host to see the
same article twice and twice only.  At that point, the requirements of a
gateway cut off further reinjection of the same article.

> Although it is argued that such a network should be regarded as
> gatewaying into the network where SECOND is to be propagated, it is far
> from certain that the owner of such a network or such an individual
> poster will regard himself as a "gatewayer", or that he will believe
> that section 3.9 applies in any way to him. Such small private neworks
> are notorious for "leaking" (and it is not necessarily the "owner" of
> the network who will be the REINJECTOR that perpetrates the leak).

Note that *relaying* leaks from such private networks are quite rare
(although not non-existent).  Leaks are almost always via reinjection
(suck/rpost feeds).

> This clearly removes the risks inherent in Option 1. It is inherently
> much safer, and it is 'fail-safe' in the sense that the only thing that
> can go wrong is that an article whose propagation was genuinely delayed
> at some point might fail to propagate beyond a relaying agent with a
> short history file expiry. But better to lose a few articles than run
> the risk of loops.

As you note, this approach makes it impossible for a gateway that is down
for longer than the retention period of its most immediate upstream
relaying agent to ever go back and gate the articles that it missed during
its period of downtime.

This is exactly the sort of problem that caused us to add Injection-Date
in the first place.  If this is acceptable, I move that we drop
Injection-Date entirely and go back to the well-understood and
already-implemented staleness checking on Date and make people who hold
posts for a while work around this in their posting agent.  It would save
a ton of complexity.

> But since B wants the articles for its own use (not for relaying back
> into Usenet), it is presumably a serving agent, in which case its expiry
> time is likely to be more than a week, in which case the staleness check
> is unlikely to be triggered. But even if that is not so, this is a case
> where all B's admin needs to do is to temporarily increase the expiry
> time of his history file. This is always safe; even if some site on his
> local network tries to relay an article back into Usenet, it is still
> protected by its _original_ Injection-Date.

No, this is *unsafe* if that host is part of a Netnews network because it
means that articles that the relaying agent had already accepted it may
accept again because it now thinks that its staleness cutoff is longer
than it really was and will therefore think that articles that it had
expired out of its history would be present in history had it already seen
it.

Extending the staleness cutoff is *not* safe unless it's done at least
that length of time after increasing the expiration time on the history
file so that the history file really does contain that many days of
history.

Please remember that one of the very common cases of suck/rpost feeds is
*from* a standalone server *to* Usenet, so assuming that the destination
of such reinjected articles is always a private stand-alone server is
clearly incorrect.  Please also remember that you cannot assume that the
person running the gateway and the person running its first upstream
relaying agent are the same person or that the latter cares about the
concerns of the former.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 19 Jan 2007 21:56:15 GMT

>doing the reinjection make this transformation.

Indeed, whether it is the injecting agent that rewrites the header, or
whether it simply creates a new header because the
poster/gayewayer/reinjector had removed the old one, makes little
difference, since the observed effect is the same.

>> We are concerned with an article that is being reinjected. ...

>but we can all agree on that as a long-term goal).

Agreed, which is why you need something rather odd, like the Anytown
Squash Club, that creates a longish delay for things to go wrong. And to
cause an actual loop you need two separate longish delays somewhere
(though that is not impossible, just requires something odder than that
Squash Club).

>> .... Such small private neworks

>(although not non-existent).  Leaks are almost always via reinjection

And usually caused by someone far removed from the first injector, so the
two of them are unknown to each other, and neither of them believes he is
doing anything out of the ordinary. The problem, in essence, is that I do
not believe in this concept of "disjoint networks", since it is almost
impossible to prove that two networks are truly disjoint.

>relaying agent to ever go back and gate the articles that it missed during

Right. And that is the problem we need to fix. It is only really fixable
if the gateway that is down can be Absolutely Sure that he is leak-proof
(e.g. he is a stand-alone server). In that case, it is perfectly
reasonable for him to rewrite the Injection-Date when he come back online,
or to remove it and let the injecting agent create a new one. The only
question is whether and how to write that into the draft as a "truly
exceptional" situation.

>in the first place.

Yes, it is the man who carries his prepared article, or even a complete
private server, around on his laptop. But I still want the Date header to
reflect the time of composition, since that is what is clearly stated by
RFC 2822, which we are following in this regard (that was also part of the
reason to add Injection-Date).

> If this is acceptable, I move that we drop

No, because whatever scenario we come up with that causes problems with
Injection-Date, you can write a similar scenarion that breaks with Date.

>expired out of its history would be present in history had it already seen

Not so. This is not a relaying agent we are talking about (if a relaying
agent goes down for some length of time, then articles will still flood
around it, albeit perhaps more slowly).

We are talking about a serving agent that has been down for some period,
and has therefore received no articles during that time. Therefore, the
only articles that will get accepted twice are those that were in transit
at or around the time of the breakdown, such that there is some
uncertainty about exactly which articles were missed, and for safety it
might have asked for a few more than it actually needed (e.g. by a
slightly pessimistic date-time in a NEWNEWS).

And it is only his direct clients who will see those few messages again.
If, having accepted articles that he has already seen before, he then
tries to relay them out to the rest of Usenet, they will still be
protected by their original Injection-Date. So any damage/inconvenience
will be restricted to himself and his immediate clients. That seems a
reasonable price to pay after such a major disruption.

>clearly incorrect.

Eh? The cases I am concerned about are when the destination of some
articles is the wider Usenet (and possibly via several routes, and
possibly unintentionally so). So I am not at all assuming or concerned
with articles sent to a stand-alone server. So could you please explain
your point there more clearly?

---

From: Russ Allbery <[email-masked]>
Date: Sat, 20 Jan 2007 14:35:54 -0800

I'm going to bow out of any further discussion of Charles's proposal for
how reinjection should be dealt with unless someone else weighs in and
supports it in a way that I can make head or tails of.  I think we're now
at the point where posting agents are sending messages directly to serving
agents, which is so confused that I can't even work out what model of
Netnews is being applied.

So far, other than the changes and clarifications from Forrest's proposal
(which I want to continue discussing and which I think is roughly in the
same spirit as my text but more rigorous), I haven't seen anything
proposed which in my opinion is technically better than what's currently
in the draft.