Subject: Re: #1416 Reinjection - an attempted summary, and a suggested resolution

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 20 Feb 2007 16:43:36 GMT

In <[email-masked]> Russ Allbery <rra@xxxxxxxxxxxx> writes:

>Harald Alvestrand <harald@xxxxxxxxxxxxx> writes:

>> - What you are calling "reinjection" is what I call "multipoint
>> injection, where one of the points listens to a Netnews network to get
>> hold of the proto-article".

>I think that's a reasonable simplification of terms.

Not so sure. As used hitherto, "multipoint injection" has always implied
one site (posting agent) that injected in several places at essentially
the same time. "Reinjection" implied that an article had wandered around
through assorted sites/networks/gateways/whatever and then got injected
again (whether deliberately or accidentaiiy). As such, it is inherently
more dangerous than the simpler multipoint case. So I suggest we keep both
terms going until we understand exactly what we are going to do

>> I have yet to understand any tangible benefit from your proposal to
>> allow the posting-agent to specify injection-date.

>I tried to address this (at some length, unfortunately) in my message
>about the identity model of an article.

Agreed. The posting agent is in a better position to know exactly what is
going on in the more unusual cases, and if he has available an earlier
Injection-Date than the one the injecting agent is likely to apply, then
it is safer for the injecting agent to accept his version (and, after all,
if the posting agent make the wrong call, then it(he) has only
it(him)self to blame if propagation suffers). This feature it _not_ for
normal straightforward posters to use.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Russ Allbery <[email-masked]>
Date: Tue, 20 Feb 2007 09:21:33 -0800

Charles Lindsey <[email-masked]> writes:

> Agreed. The posting agent is in a better position to know exactly what
> is going on in the more unusual cases, and if he has available an
> earlier Injection-Date than the one the injecting agent is likely to
> apply, then it is safer for the injecting agent to accept his version
> (and, after all, if the posting agent make the wrong call, then it(he)
> has only it(him)self to blame if propagation suffers). This feature it
> _not_ for normal straightforward posters to use.

I don't agree with the last sentence.  I don't think there's any reason
for a posting agent not to just always supply an Injection-Date similar to
how many posting agents already always supply Date.

The original goal, after all, is to move all the protocol weight of Date
onto a new header without fundamentally changing anything, which means
that anything that's fair game today with Date should probably also be
fair game with Injection-Date.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 20 Feb 2007 17:17:58 GMT

In <[email-masked]> Russ Allbery <rra@xxxxxxxxxxxx> writes:

>So this proposal as stated doesn't make sense to me, since it talks about
>keeping Date consistent when the only reason to keep Date consistent
>(beyond generally not wanting to modify articles and keeping the human
>information component) is something that we're phasing out.

Well "not wanting to modify articles" is an important principle of Usenet,
and so one should not be tinkering with the Date for the same reason that
one should not be tinkering with the Subject, or any other header, and for
that matter the body.

>One of the reasons why I believe injecting agents SHOULD continue to check
>Date for staleness is that I believe no propagation plus a clear error
>message is better than spotty propagation and no diagnostics.  I believe,
>from other messages, that Charles disagrees and would prefer spotty
>propagation improving as (hopefully) more hosts support Injection-Date.
>The disagreement may to some extent depend on one's beliefs about how fast
>Injection-Date adoption will happen.

Yes, that's a fair summary of my position. For sure, a 3-day limit is far
too short. Current practice appears to be much longer than that, and I can
easily imagine situations where injection might be delayed that long after
composition of the article. And currently (so far as I can tell)
propagation does not deteriorate until something closer to 8 days.

>It's not the sort of loop that melts down a newsgroup, so it's not as big
>of a deal in that sense.  The loop has to be time-delayed.  But it does
>feel like a fairly fundamental failure of the protocol to me.

Indeed.

>I think the main alternative (assuming we're keeping Injection-Date) looks
>like this:

> * Servers are required to check Injection-Date in preference to Date for
>   determining staleness.

> * Posting agents may provide Injection-Date in proto-articles.  They MUST
>   do so if doing multiple injection.  If Injection-Date is provided, the
>   injecting agent MUST NOT modify it.

Yes, I agree entirely so far (though I might have SHOULD for that first
MUST - but not for the second).

But I am not so sure abut the next bit.

> * If the proto-article contains neither a Date nor an Injection-Date
>   header field, the injecting agent MUST add both.

> * If the proto-article contains only a Date header but no Injection-Date
>   header, it's using the old article identity model.  If multiple
>   injection is done, Date is provided, and no Injection-Date header is
>   ever added, everything works as it does now.  If multiple injection is
>   done, Injection-Date is provided, and all agents check Injection-Date,
>   everything works as it does now.  We have to keep things into one of
>   those two cases as much as possible to maintain the existing guarantees
>   of the protocol, so I think an injecting agent, when presented with a
>   proto-article that contains a Date but no Injection-Date, must assume
>   the old identity model is being used and MUST NOT add Injection-Date.
>   This means that people who want to send time-delayed messages will need
>   to update their posting agents to add Injection-Date.  I think that's
>   reasonable.

I disagree there. Posting agents have no concept of which identity model
thet are using. Current practice has often been that they do not set the
Date, and leave it to the injecting agent to set it. But that preactice is
already changing, since mail user agents tend to write the Date at
composition time, and lots of mail user agents are also news user agents
(or think they are :-( ), and hence do the same thing.

So what I am hoping to see in the future is that posting agents will
routinely add a Date at composition time (following the RFC 2822 practice,
which we are now adopting) but that they will still leave it to the injecting
date to add the Injection-Date. So that will not imply an "old article
identity model", but rather the Best Current Practice. But that then means
that an injecting agent MUST supply an Injection-Date if none was present
in the article as submitted.

>Remaining issues are to what degree we want to standardize serial multiple
>injection -- I think it may be worthwhile describing the specific steps
>one takes to turn an article into a proto-article somewhere -- and whether
>we need to deal with stale pre-existing Date/Injection-Date headers or
>just ignore the issue and let people make it up as they go if they end up
>in a situation where stale Date/Injection-Date headers are preventing them
>from doing some gatewaying they want to do.

People who know what they are doing (e.g. they are running their own
private servers, or are arranging for multipoint injection) are the ones
who are likely to provide their own Injection-Dates, or even to remove
existing ones if they really REALLY know that the article cannot already
have leaked out onto Usenet. How much we say/recommend/warn-against (or
don't say/etc) is a matter for discussion after some further examination
of the risks. But I don't think that need affect the broad picture, which
I think should be as outlined above

---

From: Russ Allbery <[email-masked]>
Date: Tue, 20 Feb 2007 21:27:10 -0800

Charles Lindsey <[email-masked]> writes:

> Yes, that's a fair summary of my position. For sure, a 3-day limit is
> far too short. Current practice appears to be much longer than that, and
> I can easily imagine situations where injection might be delayed that
> long after composition of the article. And currently (so far as I can
> tell) propagation does not deteriorate until something closer to 8 days.

3-day limit?  Oh, hm:

   3.   It SHOULD reject any article whose Date header field is more
        than 24 hours into the future (and MAY use a margin less than 24
        hours).  It SHOULD reject any article whose Date header appears
        to be stale (more than 72 hours into the past, for example, or
        too old to still be recorded in the database of a relaying agent
        the injecting agent will be using) since not all news servers
        support Injection-Date.

That's really not saying what I wanted to say.  I think I borrowed that
language from something else (probably the staleness checks for
Injection-Date).

What I really wanted to say there was something more like:

    It SHOULD reject any article whose Date header is too far into the
    past (too old to still be recorded in the database of a relaying agent
    the injecting agent is using, for example).  This cutoff SHOULD NOT be
    any shorter than 72 hours into the past.

In other words, the 72 hours was intended to be a minimum, not a suggested
value.

I know you probably still disagree, but I didn't intend to be that harsh.

> thet are using.

I don't agree with this at all.  I think that you're selling posting
agents short.  Many posting agent authors know very well what identity
model they're using and know that Date plus Message-ID make up the
message's identity, and provide header fields appropriately.  We will
cause problems if we break this.

> Current practice has often been that they do not set the Date, and leave
> it to the injecting agent to set it.

This will continue to work as expected in my proposal (just to be clear)
since in that case the injecting agent can add an Injection-Date as well
(and indeed MUST).

> But that preactice is already changing, since mail user agents tend to
> write the Date at composition time, and lots of mail user agents are
> also news user agents (or think they are :-( ), and hence do the same
> thing.

I think the point of disagreement here is my belief that maintaining the
current identity model and not breaking posting agents that are using it
intentionally is more important than making use of Date for composition
date quietly work without changing posting agents.

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 23 Feb 2007 16:54:13 GMT

>how many posting agents already always supply Date.

That is fine for well-written agents, but there are an awful lot of people
out there writing posting agents wo think that news is just a minor
variant of email, and I would not trust them to do it right. They need to
ensure that the Injection-Date header is not added until they actually have
a connection to an NNTP server open and ready to accept the injection.

As far as USEPRO is concerned, we have to allow either party to add the
Injection-Date. As to what Best Practice might be, that may be more or
USEAGE.

>fair game with Injection-Date.

Except that "fair game" for Date today covers a variety of practices, not
all of which are appropriate for Injection-Date (though 99% of the time
injectin follows composition so closely that it hardly matters - it is the
other 1% we are trying to make better).

---

From: Russ Allbery <[email-masked]>
Date: Fri, 23 Feb 2007 09:31:03 -0800

Charles Lindsey <[email-masked]> writes:

> That is fine for well-written agents, but there are an awful lot of
> people out there writing posting agents wo think that news is just a
> minor variant of email, and I would not trust them to do it right. They
> need to ensure that the Injection-Date header is not added until they
> actually have a connection to an NNTP server open and ready to accept
> the injection.

Well, I think you've already proven by experiment that if they miss by a
day, it's not a big deal.  It's only a problem if they miss by three or
four, at which point we're mostly talking about off-line readers, which
are more sophisticated clients to start with.  I'm more confident in the
ability of people who write off-line readers to get this right than people
who write a simple normal client.

> Injection-Date. As to what Best Practice might be, that may be more or

In the absence of any stronger arguments than "some clients are stupid"
(and right now people don't seem to be presenting any), I'm opposed to
breaking the existing identity model for existing clients written before
the introduction of Injection-Date.  I don't think that's the sort of
non-backward-compatible change that's okay for us to make just because
some clients get it wrong.  I think we can provide mechanisms for clients
that want to do the right thing to do it right, but potentially breaking
the identity model of currently well-behaved clients to work around
problems with others doesn't make me comfortable.

So I only agree with this statement if there is no Date header in the
proto-article.

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 23 Feb 2007 17:35:48 GMT

>In other words, the 72 hours was intended to be a minimum, not a suggested

Well that is certainly better.

>I know you probably still disagree, but I didn't intend to be that harsh.

But if an injecting agent is aware of the cutoff of its immediately
following relaying agent (which is a reasonable assumption, because they
are often two aspects of the same agent), then it will also be aware
whether said relaying agent understands Injection-Date. In which case that
SHOULD should not apply. IOW, what you are proposing should be regarded as
only a temporary measure for use when Injection-Date is not observed (i.e.
a continuation of what some agents are currently doing anyway).

>cause problems if we break this.

If you are saying that some posting agents deliberately don't add the Date
header until they are actually about to inject, then that is fine, and in
future they will be doing the same with Injection-Date. But I suspect the
majority of current agents either create the Date at composition time, or
else don't create it at all.

>since in that case the injecting agent can add an Injection-Date as well

Yes, but what I want is MUST when Injection-Date is absent, irrespective
of whether Date is present or not. I still do not see what can go wrong
in that situation. Can you construct an example?

---

From: Russ Allbery <[email-masked]>
Date: Fri, 23 Feb 2007 21:53:13 -0800

Charles Lindsey <[email-masked]> writes:

> whether said relaying agent understands Injection-Date. In which case
> that SHOULD should not apply. IOW, what you are proposing should be
> regarded as only a temporary measure for use when Injection-Date is not
> observed (i.e.  a continuation of what some agents are currently doing
> anyway).

Yes, exactly.  I think that the root of our disagreement is again that I
believe said transition period until Injection-Date is widely enough
observed that an article with a very stale Date will propagate usefully is
on the order of five to ten years.

> If you are saying that some posting agents deliberately don't add the
> Date header until they are actually about to inject, then that is fine,
> and in future they will be doing the same with Injection-Date. But I
> suspect the majority of current agents either create the Date at
> composition time, or else don't create it at all.

I believe that some posting agents deliberately add the Message-ID and
Date header so that they can inject the article with the same identity to
multiple servers simultaneously.  I know that at least some proxies have
that capability.  If each injecting agent then adds their own
Injection-Date, we break the identity model.

>> This will continue to work as expected in my proposal (just to be
>> clear) since in that case the injecting agent can add an Injection-Date
>> as well (and indeed MUST).

> in that situation. Can you construct an example?

Sure.  Suppose that I'm using a local news proxy which talks to three
different remote servers for me and presents me with a combined view of
newsgroups from those servers.  (This is an increasingly common substitute
for running a full-fledged local news server, and in some ways works much
nicer.)  Suppose that for maximum propagation (for exactly the same sorts
of reasons that you do something similar with C News), I have it
configured to post my messages to all of those remote servers at the same
time.

If one of those servers is down at the time of the posting, the safest and
most reliable behavior, and the one that I believe is currently
implemented in at least some software of this sort, is to queue the post
and keep trying.

So you potentially get multiple injection separated by time, which if
Injection-Date is added separately by each injection agent creates the
possibility of duplicate articles via the means that we've just been
discussing in this thread.

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 24 Feb 2007 16:04:35 +0100

Russ Allbery wrote:

> I believe said transition period until Injection-Date is widely enough
> observed that an article with a very stale Date will propagate usefully
> is on the order of five to ten years.

Or more.  But at least it will be _possible_ to get it right for those
who care about it.  You need a rationale pointing out that RFC 2822 has
its own concept for Date now also adopted for NetNews.  Injection-Date
is a simple workaround for this issue because histories are limited.

> Date header so that they can inject the article with the same identity
> to multiple servers simultaneously.

I don't believe in this "identity = ID + timestamp" theory.  The ID is
the identity, the timestamp is a kludge because histories are limited,
and because accepting stale articles isn't in the interest of the poster.

> If each injecting agent then adds their own Injection-Date, we break
> the identity model.

Either the ID is still in the history, or it isn't.  It's already odd to
post articles days after they were written.  If somebody manages to post
the same article again elsewhere after a significant delay that's just a
very bad idea, add a MUST NOT or something.

How about "posting agent MUST generate an Injection-Date at the time of
the 1st injection if it intends to inject this article later elsewhere" ?
Multiple injection is gateway magic, "above all avoid loops" etc,

Frank

---

From: Russ Allbery <[email-masked]>
Date: Sat, 24 Feb 2007 11:33:37 -0800

> and because accepting stale articles isn't in the interest of the

I think that's a wording distinction without any semantic difference.
Kludge or identity, history files are limited and therefore the timestamp
is a component of the effective identity of the article.

> very bad idea, add a MUST NOT or something.

We just had a long thread discussing the reasons why this is useful to do
when going between Netnews networks that may or may not be disjoint.
Those uses don't go away just because you think they're a bad idea.

> the 1st injection if it intends to inject this article later elsewhere"?

I think everyone's already agreed to this and we've moved on to discussing
the backward-compatibility impact on agents that have not updated to
support this draft.

> Multiple injection is gateway magic, "above all avoid loops" etc,

That's where we started this whole discussion at.  I think several cogent
arguments have been presented since for why just leaving all of that to
the section on gateways isn't a good idea.  Again here it would be more
useful if you would present an argument rather than making an assertion.

---

From: Frank Ellermann <[email-masked]>
Date: Sun, 25 Feb 2007 10:09:29 +0100

Russ Allbery wrote:

> Again here it would be more useful if you would present an argument
> rather than making an assertion.

The draft has to:

(1) explain why Date can't be used anymore in the sense of s-o-1036
    without getting into conflicts with the RFC 2822 usage
(2) state that Injection-Date is introduced to fill that gap
(3) explain what a history is, and why a reasonable minimum like N=7
    in s-o-1036 is required for its purpose

If the principles of operation are clear the consequences are also
clear.  Small differences of the Injection-Date of multiple injections
are IMO no problem.  If you think it's a problem explain what that is.

Implementors need to know that "listed in the history" isn't the same
as "we still have that article".  They also need to know when they can
get rid of a history entry (based on arrival date or Injection-Date).

If all else fails just copy the s-o-1036 text replacing its "Date".

Frank

---

From: Russ Allbery <[email-masked]>
Date: Sun, 25 Feb 2007 10:19:16 -0800

> (2) state that Injection-Date is introduced to fill that gap

This seems like a reasonable addition to the point on Injection-Date in
appendix A where changes to the protocol are discussed.  Please feel free
to propose text.

>     in s-o-1036 is required for its purpose

Currently, we have:

   2.  It MUST reject any article that has already been successfully
       sent to it, based on the Message-ID header field of the article.
       To satisfy this requirement, a relaying agent normally keeps a
       database of message identifiers it has already accepted.

   3.  It MUST examine the Injection-Date header field or, if absent,
       the Date header field, and reject the article if that date
       predates the earliest articles of which it keeps record or if
       that date is more than 24 hours into the future.  It MAY reject
       articles with dates in the future with a smaller margin than 24
       hours.

I agree that an explicit link between staleness and the lack of a
requirement to keep an unbounded database isn't drawn there as clearly as
it could be.  Specific text recommendations are welcome.

As for the reasonable minimum on history retention, you have repeatedly
raised this and repeatedly lost this argument every time that you have.  I
have no interest in opening the discussion again.

> are IMO no problem.  If you think it's a problem explain what that is.

I already did.  I don't have anything further to add.

> Implementors need to know that "listed in the history" isn't the same as
> "we still have that article".

Why?

> They also need to know when they can get rid of a history entry (based
> on arrival date or Injection-Date).

I think this is a reasonable point.

---

From: Bill McQuillan <[email-masked]>
Date: Sun, 25 Feb 2007 15:25:48 -0800

>        articles with dates in the future with a smaller margin than 24

To me this reads like:

     2 - Keep all message identifiers FOREVER!
and
     3 - Injection-Date helps reject articles that were injected before
         the agent was ever run for the first time!

Since this is clearly a mistaken notion, I think that #2 should be modified
to include a note that the history may be pruned after a "decent interval".
Perhaps something like:

   2.  It MUST reject any article that has already been successfully
       sent to it, based on the Message-ID header field of the article.
       To satisfy this requirement, a relaying agent normally keeps a
       database of message identifiers it has already accepted. Since
       this database may grow without limit, message identifiers that
       have been stored for more than a certain amount of time
       (often 7 days) may be purged.

-- 
Bill McQuillan <[email-masked]>

---

From: Russ Allbery <[email-masked]>
Date: Sun, 25 Feb 2007 17:19:41 -0800

>        (often 7 days) may be purged.

The retention time is specifically tied to the stale article time and
shouldn't be set independently of it, so we need to capture that somehow.

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 26 Feb 2007 09:26:39 +0100

> shouldn't be set independently of it, so we need to capture that somehow.
Should we add an appendix with a list of "recommended timers"?

There, we could say

   Stale article time (section x.x): RECOMMENDED to be larger than 7 days
        MUST (SHOULD?) be less than Message-ID retention time.

   Message-ID retention time (section x.y): RECOMMENDED to be larger 
than 7 days

I seem to remember seeing such lists in other specs. "When in doubt, be 
explicit".

                   Harald

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 27 Feb 2007 11:06:16 +0100

> is a component of the effective identity of the article.

I still don't believe that this "effective identity" concept is a Good 
Thing.

The Netnews environment includes, among other things:
- Current spools (with different expiry times)
- History databases (with different expiry times)
- Long-term archive (which don't expire at all)

Introducing a concept of "effective identity" that involves Date: clouds 
the issue for me.

I feel that we're better off sticking to the words in USEFOR that two 
messages with the same message-ID *are* the same message (albeit one or 
both may have been damaged in transit, such as by having its entire 
contents replaced :-)

An alternative formulation that I think results in the same semantics:

- News servers want to serve messages only once.
- The primary means of ensuring this is to keep a database of message-IDs 
they have already served. This database usually contains message-ID + 
arrival time.
- This creates three different classes of message:
  - Those that are in the database
  - Those that are not in the database, and never have been
  - Those that have been removed from the database because it's been a long 
time since they entered it.

- Servers will try to discard messages of the first and third classes
- To see the difference between the second and third class, they look at 
the age of the message, to see if they would have already removed it from 
the database if they had seen it before.
- They use two hints to the age of the message: Date and Injection-Date
- Of these, Date: is (usually) older than Injection-Date
- Since most messages arrive shortly after injection, it is reasonable to 
assume that a message "has been here and gone" if Injection-Date is such 
that if it arrived here shortly after the injection-date, it would have 
been discarded.
- If Injection-date: isn't there, use Date:
- Since some servers will use Date: anyway, injecting-agents should police 
injection based on Date:

This reformulation (which is hardly English...) positions Date and 
Injection-Date as a *hint* that allows one to heuristically guess at 
whether the article has been here or not, without introducing a new 
identity concept. I'm not sure if that's a better way to attack it, but I'd 
like to try.

(parenthesis: has anyone tried using Bloom filters for history databases? 
They're constant size, so they don't need to be trimmed.... and can have a 
fairly low false-positive hitrate...)

               Harald

---

From: Russ Allbery <[email-masked]>
Date: Tue, 27 Feb 2007 08:43:30 -0800

> Introducing a concept of "effective identity" that involves Date: clouds

Note that it's just a terminology for discussion purposes and not
something that I wanted to put into the wording of the draft.

> - News servers want to serve messages only once.

They want to accept messages only once.  Usually messages are served many
times.

> time since they entered it.

Right.

> the database if they had seen it before.

Right, although that's a rather weak way of stating it.  I would rather
say that servers MUST discard messages that, had they seen the message
before, they would have already removed it from their database.

> - Of these, Date: is (usually) older than Injection-Date

True, although not guaranteed by anything, and there are cases where this
won't be true (a client with a bad clock, for instance).

> that if it arrived here shortly after the injection-date, it would have

The assumpion is that if the article is not mentioned in history and the
Injection-Date is older than the cut-off date, the server has no way of
being sure that it's a new article and therefore MUST reject the article
to prevent duplicates.  It's not so much an assumption about the article
as it is a correctness guarantee in the news server.

Minor digression: Usually, a news server implements a cut-off date that
matches the history retention time.  History entries for unwanted or
expired articles are maintained until the retention time has passed since
the history entry was written.  Articles are rejected if Injection-Date is
older than the cut-off date.  The cut-off date MUST be more recent than or
the same as the retention date.

So, if the cut-off date is seven days in the past, and an article arrives
on the server with an Injection-Date that's six days in the past, the
article will be accepted and, normally, the history entry will be retained
for *another* seven days past that.  That configuration isn't mandatory
and a configuration that does article and history expiration based on the
Injection-Date is also possible and probably conforming.

> - Since some servers will use Date: anyway, injecting-agents should police

Yup.

> identity concept. I'm not sure if that's a better way to attack it, but
> I'd like to try.

As long as we're strong enough about a news server having to reject a
message that it cannot prove it's not seen before, I'm okay with that.

> (parenthesis: has anyone tried using Bloom filters for history
> databases? They're constant size, so they don't need to be
> trimmed.... and can have a fairly low false-positive hitrate...)

Interesting.  I haven't, in that form, although I've used various
variations on hash tables that reduce to k = 1 Bloom filters.  A Bloom
filter can't serve some of the ancillary purpose of the history file, such
as being the message ID to article mapping database, but those are only
needed on a serving agent and can be handled other ways.  You'd probably
have to rebuild the Bloom filter periodically from a simple list of
message IDs once the false positive rate became worrisome, but this might
still be faster.

You're potentially doing a lot of random-access writes for each incoming
message, which is the place I'd worry about performance.  The traditional
history database algorithm requires a ton more space and a very
time-consuming expiration run, but storing a new message ID is two pwrites
and a buffered write.  I wonder how many disk pages you'd make dirty when
writing a message ID to a memory-mapped Bloom filter with an ideal k.

INN's current history algorithm uses a Bloom filter with k = 1 as an
initial check for unknown articles, but it does a full hash-table lookup
if the Bloom filter lookup succeeds to eliminate the false positives.

This would be an interesting experiment.  INN does have a pluggable
history mechanism precisely so that people can write new ones and see how
they work.

---

From: Russ Allbery <[email-masked]>
Date: Tue, 27 Feb 2007 08:45:13 -0800

>   Message-ID retention time (section x.y): RECOMMENDED to be larger than 7
> days

> I seem to remember seeing such lists in other specs. "When in doubt, be

We could, but are there any other timers that would go there?

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 27 Feb 2007 17:04:17 GMT

>        MUST (SHOULD?) be less than Message-ID retention time.

I am not sure that "stale article time" is a well-defined concept. A news
server that is both a relaying agent and a serving agent (and that is
nearly all of them, except for the dedicated
relay-only-as-fast-as-possible sites which act as a Usenet backbone) will
keep an article in its history file as long as it stores the article and
is prepared to deliver it to reading agents, and that will usually by much
longer than 7 days. It is only those dedicated fast relayers that are
likely to purge their history files early (even after 3 days) and who are
at risk of accepting a delayed article a second time.

OTOH, one would hope that it will be those dedicated relayers who will be
the early adopters of Injection-Date.

---

From: Russ Allbery <[email-masked]>
Date: Tue, 27 Feb 2007 09:23:46 -0800

Charles Lindsey <[email-masked]> writes:

> OTOH, one would hope that it will be those dedicated relayers who will
> be the early adopters of Injection-Date.

Given that development of NNTPRelay has essentially stopped, the Highwinds
support forums appear to be down, and Highwinds hasn't issued a press
release on their site for nearly four years, I wouldn't hold my breath.
Highwinds did just release new builds last November for the first time in
two years, though, so they don't appear to have disappeared entirely.

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 27 Feb 2007 19:32:57 +0100

>        articles with dates in the future with a smaller margin than 24

I'd split that - future timestamps can get their own (small) paragraph:

3b: It MAY reject articles with dates more than 24 hours in the future.

Normally magic numbers like "24 hours" need a justification, but here
it should be obvious that this is about a plausible timezone confusion.
(If that's not the real reason for "24 hours" better say what it is.)

The "articles of which it keeps records" isn't clear.  It's not about
the oldest available article (that could be months or more), it's the
oldest unavailable article (only recorded in the history).  Maybe it's
better if you define and use a term like "retention time" or "cut off
time" as you've done it elsewhere.  I'm not sure what the difference
is, "cut off => reject as too old, retention => could be removed from
the history" (?)

What I have in mind is a server keeping articles in some groups for
long periods, e.g. for "answers" groups, and articles in other groups
only shortly, e.g. in binary groups.

> raised this and repeatedly lost this argument every time that you have.

IMO the argument in s-o-1036 is compelling.  Its UUCP considerations are
not more as important as in 1994.  On the other hand we have this "Date
to Injection-Date transition" now, that's not the moment to lower the
s-o-1036 minimum.   If a news server hasn't the disk space to keep a 
reasonable history it carries too many groups or something else is odd.

>> as "we still have that article".
 
> Why?

For the example mentioned above:  groups with very short expirations, an
extreme case would be a server only relaying some groups without keeping
the articles at all.  It would still note the Message-IDs as "seen" in
its history for some days.  Or rather that's what I think how it works
after long discussions more than a decade ago where some folks finally
convinced me that NetNews doesn't rely on the Path for this purpose 
(unlike Fido echomail).

Frank

---

From: Russ Allbery <[email-masked]>
Date: Tue, 27 Feb 2007 11:21:07 -0800

> Normally magic numbers like "24 hours" need a justification, but here it
> should be obvious that this is about a plausible timezone confusion.

That's exactly the reason for 24 hours (and I'm not adverse to adding a
comment saying so).

> oldest unavailable article (only recorded in the history).

Yes.  Exactly.

> Maybe it's better if you define and use a term like "retention time" or
> "cut off time" as you've done it elsewhere.  I'm not sure what the
> difference is, "cut off => reject as too old, retention => could be
> removed from the history" (?)

Right.

I think we're converging on a desire to rephrase things that way and
introduce those terms.

> convinced me that NetNews doesn't rely on the Path for this purpose

You're entirely correct in your understanding.  I think I understand your
point now (and agree with it).

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 27 Feb 2007 21:23:04 GMT

>on the order of five to ten years.

It only needs to be implemented by those news servers with exceptionally
short history retention times for its benefits to be felt, and the number
of such servers is relatively small.

>Injection-Date, we break the identity model.

OK, so the issue only arises with agents that multi-inject, and harm will
only arise if those injections are significantly separated in time
(and by a period measured in hours rather than minutes).

>implemented in at least some software of this sort, is to queue the post

And that would seem to be the only situation in which a significant time
separation could occur during multi-injection.

And that has to be set against my concern about the man who prepares
articles offline and then injects them much later.

So let us examine the two competing strategies.

I think we are agreed that

. Injecting agents MUST insert Date and Message-ID when absent.
. Injecting agents MUST insert Injection-Date when Injection-Date is
absent AND (either Date OR Message-ID is absent).
. Injecting agents MUST NOT insert Injection-Date if it is already
present.

We disagree on the following two possibilities:

C. Injecting agents MUST insert Injection-Date when it is absent.

R. Injecting agents MUST NOT insert Injection-Date if both (Date AND
   Message-ID are already present).

So now we can examine the consequences of those two rules.

Consequences of Rule C:

Posting agents (or proxies etc) which multi-inject AND where one of the
injections sits around in a queue because a server is down, and which
inserts an Injection-Date when it eventually wakes up, may cause relaying
agents further on which understand Injection-Date to re-accept articles
that they have already accepted before.

But note that this problem is not permanent - it will go away as and when
those multi-injecting agents start inserting Injection-Date themselves
(which they would have to do to claim compliance with our new standard).

OTOH, posting agents which prepare articles offline, adding the Date at
composition time, and then failing to inject until hours/days later, and
who then inject to a single injecting agent (either adding Injection-Date
themselves or letting the injecting agent do it) will find that their
articles propagate well amongst servers that understand Injection-Date. So
their situation will improve as Injection-Date becomes more widely
adopted.

Consequences of Rule R:

Posting agents (or proxies etc) which multi-inject AND where one of the
injections sits around in a queue because a server is down will not cause
articles to be re-accepted.

OTOH, posting agents which prepare articles offline, adding the Date at
composition time, and then failing to inject until hours/days later, and
who then inject to a single injecting agent and do not add Injection-Date
themselves will find that their articles propagate as badly as they do at
present.

Note that this problem is permanent (or at least until those posting
agents learn to insert Injection-Date themselves). Moreover, the problem
will get worse as more and more posting agents accept our recommendation
to add the Date header at composition time.

So now we have to consider which of those scenarios is the more
troublesome. For this, we have to make some guesses:

1. How many multi-injecting agents are there around, and how often will
there be disruption because one of their injectees is down; and how does
this compare with the number of offline posting agents around and how
often their injecting get delayed?

2. How will the deployment of Injection-Date go? My expectation is that
it will be taken up first by those relaying agents which currently have
short expiries on their history files (because they have the most to
gain). It will then be taken up by smaller sites with clueful
administrators, and that probably includes people running proxies and
doing multi-injecting. And finally, it will be taken up by ordinary user
agents, and indeed that will be a _very_ slow uptake.

3. How rapidly will the practice of constructing the Date at composition
time spread? Probably faster than the spread of posting agents that know
how to insert Injection-Date themselves (which I regard as normally
unnecessary, although we seem to disagree there). But this is a guess
where we have some evidence to go on, because this practice is already
widespread.

I performed an experiment on 350 articles in the group uk.misc (where I
would expect to find a pretty average bunch of posters). I compared the
NNTP-Posting-Date with the Date header. I would expect these to be
identical (to the same second) where they were both created on the
injecting agent. Where they were different, they were often surprisingly
close (indicating that lots of people were actually online while they
composed articles, though differences between the two dates were both
positive and negative, indicating differences between the clocks on the
two systems). But the interesting fact to emerge was that nearly 70% of
posting agents were apparently already constructing their own date
headers. OK, the sample was small, and I will happily perform a larger
experiment if people wish. But that 70% was considerably higher than I
would have expected.

So my conclusion is that it will soon be commonplace for Date to be already
present at injection time (indeed we are almost there), and so to make
assumptions on the basis that this indicates a multi-injecting proxy is
unwise. On balance, it sees that rule C is going to provide fewer
problems, especially in the longer term.