Subject: Re: #1416 Injection-Date - Summary of options

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 17 Apr 2007 16:09:26 GMT

In <[email-masked]> "Charles Lindsey" <[email-masked]> writes:

>I think we have now said everything that is to be said on this, so I have
>produced a summary of where I think we have got to.

>It lists rules for what various agents are supposed to do with
>Injection-Date in various situations. Most of these are, I think, agreed
>(and they differ quite a bit from our starting point in the two versions
>of USEPRO and in assumptions made at the start of this thread, so that at
>least indicates some progress).

I had hoped for some response to this, especially from Russ, particularly
as to whether the points I claimed to be "agreed" were in fact agreed.
That would then enable us to proceed to the main issue of disagreement:

>There still remains the two conflicting possibilities (labelled IC and IR
>in this summary) that we have to choose between, though I think we
>understand the consequences of each better than we did.

I understand that Ruus has been tied up on other matters since then, but
that he hopes to return to active participation shortly.

In the meantime, however, I have encountered one feature that we need to
look at further. In addition to delays caused by a poster who prepares his
article offline and then delays for some days before injecting it, there
is also the case where the poster injects his article promptly but,
because it is posted to a moderated group, the moderator may then delay
for several days before approving it.

>Posting agents:
>---------------

.....

>P4. MAY include Injection-Date indicating time of actual injection (as
>opposed to composition).

...........

>P6. When injecting at multiple injecting agents, MUST include Injection-Date
>which MUST then indicate time of earliest injection at any such agent.

But of course all those multiple injections will now end up being sent to
the same moderator, and the Injection-Date provided by the posting agent
(which usually would not be aware whether a particular group was moderated
or not) is largely meaningless.

I am assuming, of course, that if the 'moderator' is actually a team of
moderators with some bot to distribute articles amongst them, they will
have set up arrangements to ensure that a given article only gets approved
once.

>Injecting Agents:
>-----------------

......

>I4. MUST NOT insert (or alter or delete) Injection-Date if it is already
>present.

So I think, in this case, the moderator MUST now discard any
Injection-Date already provided, and then proceed to inject the Approved
article in the normal manner, at which point he MAY then add an
Injection-Date, and MUST add one if he is injecting at multiple sites.

That will give the best chance of the article propagating normally, even
if the moderator delayed before Approving it.

There is a separate but related question of whether, for the benefit of
legacy injecting and relaying agents, he might need to do something about
a Date header that had been rendered unusually stale because of hiw own
delay. Usepro-06 made some special provision for this, and I think
Usepro-07 does also. Normally, of course, Date should be left strictly
alone as being a matter to be determined by the Poster, but one can hardly
blame the Poster for subsequent delays caused by the moderator.

Apart from this one case, I think all the other rules listed in my
original message are unaffected.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Russ Allbery <[email-masked]>
Date: Tue, 17 Apr 2007 11:17:39 -0700

> that he hopes to return to active participation shortly.

Yes, sorry.  That's still my plan, and responding to that message is near
the top of my priorities when I can find a bit of time again.

It's clear to me from the discussion that we need a couple new sections to
the document that provide clearer guidance and descriptions of how the
history mechanism works.

> Injection-Date, and MUST add one if he is injecting at multiple sites.

Yup, that sounds right to me.  Good catch.  I think it's reasonable to let
the article possibly end up duplicated in the bizarre case where the
poster is multiply injecting and some sites have the group marked
moderated and some marked unmoderated.  Articles are occasionally
duplicated in similar cirumstances today.

> blame the Poster for subsequent delays caused by the moderator.

Yup.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 17 Apr 2007 23:22:56 +0200

Russ Allbery wrote:

> It's clear to me from the discussion that we need a couple new sections
> to the document that provide clearer guidance and descriptions of how
> the history mechanism works.

That's good news from my POV as somebody who wants to know "how
stuff works" without cheating like "get hold of a s-o-1036 copy".

> I think it's reasonable to let the article possibly end up duplicated
> in the bizarre case where the poster is multiply injecting and some
> sites have the group marked moderated and some marked unmoderated.

If that case is considered as "bizarre" please say so in the draft.
Maybe not literally, but the tendency should be clear for readers.

Frank

---

From: Russ Allbery <[email-masked]>
Date: Sat, 21 Apr 2007 19:44:46 -0700

Charles Lindsey <[email-masked]> writes:

> I think we have now said everything that is to be said on this, so I
> have produced a summary of where I think we have got to.

Thank you very much for writing this up.

It's been a long time since the last message, so I'm going to quote
extensively to save people from having to refer back to the mailing list
archives.

> The following rules are mostly agreed (and indeed some are already in
> the present USEPRO). They are all included to give a complete
> picture. Note that these are indications of what is supposed to happen,
> and not wordings ready for use in the document.

> The chief issue is to choose between Rules IC and IR, for which the
> several "Good/Bad News" paragraphs need to be studied. Neither Rule is
> perfect. It is a question of whether a little pain should be endured for
> the benefit of an ultimate gain. In either case, the Sky will not Fall
> In - the worst is that an occasional article might be seen twice at some
> server.

> P1. MAY omit various headers from proto-articles (specifically Message-ID,
> Date, Path).

> P2. But should (with a small 's') not omit Date (because we are following
> the RFC 2822 idea that Date reflects time of composition).

>    NOTE P2: This practice is already widespread, and growing, but it works
>    against the man whose injections are significantly delayed after
>    composition time; hence P5 below.

> P3. MUST include Message-ID and Date if article is to be injected at
> multiple injecting agents (and the same at each injecting agent, of course).

> P5. SHOULD/should include Injection-Date if injection time is significantly
> later than composition time.

>    Good News P5: A certain cure for the man with delayed injection problems;

>    Bad News P5: Posting agents tend not to get upgraded with such
>    new-fangled ideas until long after servers have adopted them. But at
>    least this man has a strong incentive to upgrade (once he is aware of the
>    possibility).

> which MUST then indicate time of earliest injection at any such agent.

I agree with all of those statements.

> I1. MUST insert Date and Message-ID when absent and insert/extend PATH.

Agreed.

> I2. MUST/SHOULD remove/rename Injection-Info (plus NNTP-Posting-*,
> X-Trace, etc.) and SHOULD provide fresh Injection-Info. MAY reject
> (reinjected) articles already containing those headers.

I believe that an injecting agent MUST reject an article that already has
an Injection-Info header and MAY reject an article that contains other
trace headers that it happens to recognize.  Removing headers or renaming
headers provided by a posting agent I believe is a SHOULD NOT.

So some disagreement here.

> I3. MUST/SHOULD insert Injection-Date when Injection-Date is absent AND
> (either Date OR Message-ID is absent).

>    NOTE I3: But see IC and IR below for further details.

>    NOTE I4: That is to prevent re-acceptance problems arising from
>    unanticipated (and unpreventable) "leaks" between small private
>    networks and major networks such as Usenet. But see E1 for how a
>    gateway might do such tricks.

Agreed on these.

> We disagree on the following two possibilities:
 
> IC. Injecting agents MUST insert Injection-Date when it is absent
> (unless it is apparent that this is a reinjection).

>    Good News IC: This provides immediate beneffit to the posting agent
>    which has delayed injection, even if P5 is not implemented.

>    Bad News IC: Until multi-injecting sites can be assumed to have
>    implemented P6 this can cause re-acceptance of articles at later
>    relaying agents if one of the multi-injected versions suffers
>    substantial delay en route. So the existing behaviour is changed to
>    that extent.
 
> IR. Injecting agents MUST NOT insert Injection-Date if both (Date AND
>    Message-ID are already present).

>    Good News IR: Inverse of the Bad News IC.

>    Bad News IR: Posting agents with delayed injection see no benefit
>    unless/until they implement P5.

I think that's a fair summary of the options.

> Relaying and Serving Agents:
> ----------------------------

> R1. MUST discard incoming articles whose Injection-Date (or Date if
> Injection-Date is absent) predates the earliest articles of which it
> keeps record (or if more than 24 hours into the future, etc.).

> That "earliest articles of which it keeps record" might be reworded (see
> my reply to Frank on March 1st). The essential question is "if this
> article had arrived X days ago, would I still be recording that fact?" 
> (which allows for different retentions for different groups, etc). And
> maybe it records the time of arrival, or the original Date header, or
> whatever (about which I don't think we care).

Right.  I'm going to propose a new section that discusses how the history
file works and which should make all of this much clearer and more
straightforward.  There are a couple of standard ways of handling history
file entries, and we should probably just spell them out.

> Gateways:
> ---------

> And especially Gateways from one Netnews system A to another one B, in
> particular from small private one-node networks into Usenet:

> G1. MUST retain all headers as received (including in particular Date
> and Injection-Date), unless some EXCEPTION applies, as listed below.

> G2. MUST hand article to an injecting agent, or else perform all the
> duties on an injecting agent (except perhaps insertion of
> Injection-Date) themselves and then relay.

I would just say that they hand the article to an injecting agent, since
the second option there is just a special case of the first where the
injecting agent they use is built in to the gateway.

However, I think the best solution to this is, rather than tackling the
gateways section, to write a separate section about multiple injection to
replace the section about reinjection.  I'm going to take a crack at that
and see if I can come up with something that's fairly clear.

> EXCEPTIONs:
> -----------

> These are the tricky bits

> E1. An existing Injection-Date MAY be removed, but ONLY IF

> a) There is no possibility that another Gateway (direct or indirect)
> between those networks exists (which includes all manner of "leaks" by
> other nodes in A over which the gatewayer has no control).

> b) The article had not been previously gatewayed from B into A (i.e. it
> had been originally posted by a user of A).

>    NOTE: That covers the man who carries his private server around on a
>    laptop, and other users of suck/rpost, but if covers very little else.

>    NOTE: It is safer to allow gateways (which hopefully understand the
>    circumstances of the particular article) to do any tinkering with
>    Injection-Date than for injecting agents to do it; Hence Rule I4.

Yup, this seems right.

> E2. Other evidence of injection within network A (e.g. Injection-Info,
> NNTP-Posting-*, etc) MAY be removed, but ONLY IF

> a) The gatewaying is to use an injecting agent which would not otherwise
> allow reinjection.

>    NOTE: That screams of "cheating", "horrible hacking", etc. So be it.
>    Perhaps it is better not mentioned.

I don't believe injecting agents should ever allow reinjection, so I would
remove the conditional.  Otherwise, I agree.

---

From: Russ Allbery <[email-masked]>
Date: Sat, 21 Apr 2007 21:39:04 -0700

>> Injection-Date already provided, and then proceed to inject the
>> Approved article in the normal manner, at which point he MAY then add
>> an Injection-Date, and MUST add one if he is injecting at multiple
>> sites.

> Yup, that sounds right to me.  Good catch.

Turns out we already had this:

   4.  Moderators are encouraged to retain the Message-ID header field
       if it is valid, and also retain the Date header field unless it
       appears to be stale (72 hours or more in the past) for reasons
       understood by the moderator (such as delays in the moderation
       process) in which case they MAY substitute the current date.  Any
       Injection-Date, Injection-Info, or Xref header fields already
       present (though there should be none) MUST be removed.

I'm dropping the "(though there should be none)" when proposing wording
for this change, since it's now more likely.

>> legacy injecting and relaying agents, he might need to do something
>> about a Date header that had been rendered unusually stale because of
>> hiw own delay. Usepro-06 made some special provision for this, and I
>> think Usepro-07 does also. Normally, of course, Date should be left
>> strictly alone as being a matter to be determined by the Poster, but
>> one can hardly blame the Poster for subsequent delays caused by the
>> moderator.

> Yup.

And the current provision is also quoted above.

---

From: Russ Allbery <[email-masked]>
Date: Sat, 21 Apr 2007 21:41:03 -0700

> Maybe not literally, but the tendency should be clear for readers.

   Multiple injection of an article listing one or more moderated
   newsgroups in its Newsgroups header field SHOULD only be done by a
   moderator and SHOULD only be done after the proto-article is approved
   for all moderated groups to which it is posted and has an Approved
   header field.  Multiple injection of an unapproved article intended
   for moderated newsgroups will normally only result in the moderator
   receiving multiple copies, and if the newsgroup status is not
   consistent across all injecting agents, may result in duplication of
   the article or other problems.

Look good?

---

From: Frank Ellermann <[email-masked]>
Date: Sun, 22 Apr 2007 19:26:36 +0200

> Look good?

Yes, I read and answered in reverse chronological order, see
my longer reply to your long diff with a question about the
second moderator multi-SHOULD.  I tried to figure out why
it's no MUSTard.

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Sun, 22 Apr 2007 19:40:15 +0200

Russ Allbery wrote:

> I believe that an injecting agent MUST reject an article that
> already has an Injection-Info header and MAY reject an article
> that contains other trace headers that it happens to recognize.
> Removing headers or renaming headers provided by a posting 
> agent I believe is a SHOULD NOT.
 
> So some disagreement here.

The idea is important, rejecting is clearer than most attempts
to automatically "fix" articles.  It needs human intervention.

Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 23 Apr 2007 14:45:13 GMT

>Thank you very much for writing this up.

OK, so it seems my summary is largely accepted, modulo any additions
arising from my more recent message regarding moderators. So it should now
be just a straight question of choosing IC or IR.

>So some disagreement here.

This arises when an article has been "reinjected" for some strange reason.
Essentially, you are saying that it is the responsibility of the
reinjector to remove any existing Injection-Info, otherwise the injecting
agent will throw it back at him.

I could almost be persuaded of this. My only doubt arises from the case
where an article prepared on a small private (one-node) network is
then (for redundancy or other reasons) is to be forwarded to two places,
one of which accepts relaying and the other insists it is injecting (even
though IHAVE has been used, as now permitted by INN). In that case,
different versions of the article have to be sent to the two sites.

That is not impossible, but it is a nuisance, especially if the site
actually believes (though misunderstanding) that it is relaying in both
cases (in which case you would then argue, I suppose, that it needs to be
"educated").

However, even if Injection-Info is removed by a reinjector, I would still
prefer any original Path to be retained (even if that results in two
"POSTED"s within it). A Path is meant to record an article's history, and
destroying a part of that history may hinder subsequent troubleshooting.
Better the evidence of reinjection should be there for all to see.
Moreover, if by some complicated means the article ever propagates back
towards one of the "pre-historic" sites, it would still be in order not to
send it there on the grounds that is had already been there once.

>I think that's a fair summary of the options.

OK, so are we ready to count heads on that?

Essentially, it is a choice between accepting an occasional duplicate
article at un-upgraded sites, versus not getting the full benefit of
Injection-Date with current posting agents as soon as injecting/relaying
agents start to recognize it.

>replace the section about reinjection.

However, reinjection is not always associated with multiple injection, so
it still needs some mention. For sure, Gateways have to do the right thing
in the ways I have outlined, whether we use the word "reinjection" or not.
The problem is that such reinjectors may not think to regard themselves as
gatweways unless we point it out to them.

I have read your revised wordings to implement all these things, but I
need to examine them more carefully before responding. I notice that Frank
has made some comments, not all of which I agree with.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 23 Apr 2007 14:52:26 GMT

>for this change, since it's now more likely.

OK.

>And the current provision is also quoted above.

However, I am a bit dubious about mentioning exact figures like "72
hours", especially as you now speak elsewhere of "1 week" as a normal
cutoff time (I had always understood that a 3-day cutoff was common
amongst the large relay-only sites). That "72" may be justified here, but
there was another "72" elsewhere in your draft that is much more doubtful.

---

From: Russ Allbery <[email-masked]>
Date: Mon, 23 Apr 2007 10:43:31 -0700

Charles Lindsey <[email-masked]> writes:

> amongst the large relay-only sites). That "72" may be justified here,
> but there was another "72" elsewhere in your draft that is much more
> doubtful.

I'm trying to find a balance between being conservative in what one sends
and liberal in what one accepts, and three days and seven days are the
most common lower cutoff numbers that I've seen.  It's not clear to me the
best way to handle this.

---

From: Russ Allbery <[email-masked]>
Date: Mon, 23 Apr 2007 10:48:11 -0700

>> I believe that an injecting agent MUST reject an article that already
>> has an Injection-Info header and MAY reject an article that contains
>> other trace headers that it happens to recognize.  Removing headers or
>> renaming headers provided by a posting agent I believe is a SHOULD NOT.

> This arises when an article has been "reinjected" for some strange
> reason.  Essentially, you are saying that it is the responsibility of
> the reinjector to remove any existing Injection-Info, otherwise the
> injecting agent will throw it back at him.

Correct.

> where an article prepared on a small private (one-node) network is then
> (for redundancy or other reasons) is to be forwarded to two places, one
> of which accepts relaying and the other insists it is injecting (even
> though IHAVE has been used, as now permitted by INN).

INN doesn't permit IHAVE in a default configuration, to note.

> In that case, different versions of the article have to be sent to the
> two sites.

Correct.  The existing software for doing these sorts of things does
handle this.  In INN, for instance, you would use a conventional nntpsend
feed for the relaying site and an rpost feed for the injecting site.

> cases (in which case you would then argue, I suppose, that it needs to
> be "educated").

Well, yes, because the site that requires injection is *not* relaying, by
definition.  But the point of rpost is to convert a relaying attempt into
an injecting attempt.

> However, even if Injection-Info is removed by a reinjector, I would
> still prefer any original Path to be retained (even if that results in
> two "POSTED"s within it). A Path is meant to record an article's
> history, and destroying a part of that history may hinder subsequent
> troubleshooting.

I can be convinced that injecting agents shouldn't reject a Path
containing two POSTEDs.  Anyone else have an opinion on that?  Is there
any drawback to not allowing that?

>> gateways section, to write a separate section about multiple injection
>> to replace the section about reinjection.

> However, reinjection is not always associated with multiple injection,
> so it still needs some mention.

It is if you define multiple injection that way.  :)  See my proposed new
section and see what you think.

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 24 Apr 2007 14:47:32 +0200

My technical opinion:

I think it's reasonable for a posting agent (under the control of the 
poster) to "fix" articles.
I think it's unreasonable for an injecting agent (under the control of 
the news admin) to do so.

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 24 Apr 2007 14:49:40 +0200

> be just a straight question of choosing IC or IR.
Don't bet on it. I haven't bothered commenting on your text in detail, 
since I find it more constructive to comment on Russ' proposed texts.

I still think that having the posting agent specify injection-date is a 
Bad Idea, but I could be willing to be part of the "rough" in "rough 
consensus".

---

From: Russ Allbery <[email-masked]>
Date: Tue, 24 Apr 2007 10:14:11 -0700

> Bad Idea, but I could be willing to be part of the "rough" in "rough

After writing the text, I think it's weird, but it's not quite as bad as I
thought it was going to be.  It does have the merit of closely mirroring
how Date works today, and if we're going to change something in this area,
sticking as close to what we know works as possible seems like a good idea
to me.

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 27 Apr 2007 20:23:28 GMT

>> of which accepts relaying and the other insists it is injecting...

>definition.  But the point of rpost is to convert a relaying attempt into

OK, I think you have persuaded me on that one.

>any drawback to not allowing that?

AFAIR, the WG always accepted that could happen. In cases where things
have gone wrong, or malpractice is suspected, not throwing away what might
be useful evidence is never a good idea. Some sites copy original Paths
into an X-header, but I see no gain in doing that as opposed to just
leaving the Path as received and prepending new stuff in the usual way.

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 27 Apr 2007 20:26:48 GMT

>sticking as close to what we know works as possible seems like a good idea

+1

It may well be that most 'normal' posting agents leave it to the injecting
agent, but for the guy who is doing something slightly out of the ordinary
(multiple injection, delayed injection, etc) it does have some advantages,
and I can't see any disadvantage.

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 27 Apr 2007 20:16:36 GMT

>most common lower cutoff numbers that I've seen.  It's not clear to me the

I would be inclined to mention in in the Moderators bit (as indeed was
done in Usepro-06), but not in the duties of an injecting agent.
Essentially, if a guy is slow at injecting an article he wrote some days
earlier, then it is his problem if it gets lost. But you can hardly blame
that guy in the case where it was the Moderator who mucked it up.