Subject: Re: #1416: Reinjection and Injection-Date proposed text

From: Frank Ellermann <[email-masked]>
Date: Tue, 01 May 2007 10:40:04 +0200

Russ Allbery wrote:

> Just for clarity in the discussion (I don't think a wording change is
> needed), note that the cutoff date for history retention is independent
> of the period of article retention in the spool.  Usually the latter is
> longer than the former, but it may also be shorter.  They don't need to
> be related.

Oops, now I've read and answered the corrected diff before I read this
article, some of my remarks there might be obsolete...

Yes, I thought that there's only one meaning of "retention", so please
ignore the remark / question about that.

>>| The posting agent MUST supply the Message-ID and the Date, it
>>| SHOULD also supply an Injection-Date, and the proto-article
>>| offered to each injecting agent MUST be identical.
 
>> Arguably you can also use MUST here as you have it, and replace
>> the "multi-SHOULD" at the end of 3.4.1 by MUST.  But that would
>> render all existing posting agents with no idea about the new
>> Injection-Date as "non-conforming" when they try multi-Injection.
 
> As mentioned above, I went the MUST route, but I think it's a debatable
> point and I'd love to hear other comments.

Okay, if it's consistent I can certainly live with both, and the
MUST route results in clearer text.  

>> why MUST a posting agent remove Xref, this job could be handled
>> by the injecting agent ?  Is it "a simple MUST is better than
>> more convoluted statements", or do I miss a serious Xref-problem
>> here ?
 
> It's more the former, plus backwards compatibility.  Existing
> injecting agents will reject articles with Xref rather than remove
> the header.

Okay, "reject" is serious enough...  Reminds me of some cmsg: cancel
threads, we're not going to discuss Xref removal for a year or so :-)

>>> + This interval SHOULD NOT be any shorter than 72 hours into the
>>> + past.
 
>> That's a rather short minimum, barely enough for a normal weekend.
>> For this radical approach, how about a MUST NOT ?  Disclaimer, I've
>> no idea about what's usual for "binary" newsgroups.  But I doubt
>> that they have any use for the 2822-Date concept like "text" groups
>> or, more important, mail2news gateways.
 
> On Usenet, I think a MUST NOT would probably be warranted, but I wasn't
> comfortable enough saying that there would be no reason to ever use a
> shorter interval in any Netnews implementation.  I don't really feel
> strongly about this, but I mildly lean towards SHOULD NOT here just on
> the grounds of permitting an out for some reason that we didn't
> anticipate.

If Usenet needs a MUST NOT I'm tempted to ignore others where shorter
might also work.  The text makes it clear that implementors can't
hardwire "three days", and as soon as it's configurable admins can 
also try shorter periods, and get what they asked for.  

> Legacy posting agent, legacy injecting agent:
>  No one will add Injection-Date since neither agent knows about it,
>  although as a generic unknown header, either agent is permitted to
>  do so.
 
> Legacy posting agent, compliant injecting agent:
>  Posting agent won't add Injection-Date.  Injecting agent MUST add
>  Injection-Date if one or both of Date and Message-ID are not
>  specified, otherwise MUST NOT (since multiple injection may be
>  happening).
 
> Compliant posting agent, legacy injecting agent:
>  Posting agent MAY add Injection-Date any time, SHOULD if the Date
>  is more than a day in the past at the time of injection, MUST if
>  doing multiple injection.  The injecting agent doesn't know about
>  the header and therefore will do nothing with it.

> Compliant posting agent, compliant injecting agent:
>  Posting agent MAY add Injection-Date any time, SHOULD if the Date
>  is more than a day in the past at the time of injection, MUST if
>  doing multiple injection.  Injecting agent generally MUST add the
>  header if the posting agent doesn't.  The one exception is if the
>  posting agent provides both Message-ID and Date but not
>  Injection-Date, which it is always allowed to do and which doesn't
>  even violate a SHOULD if the Date is fresh.  In this case, the
>  injecting agent MUST NOT add it either.

Thanks.
 
>>> + It MAY reject articles with dates in the future with a smaller
>>> + margin than 24 hours.
 
>> Does that really help ?
 
> It's there to make it clear that some agents do not give you 24 hours
> of leeway and this is permitted in the protocol.  It's redundant with
> the general rule that anyone can reject any article for any reason
> they please, but I felt like it was worth drawing special attention
> to this so that people don't think they can rely on having 24 hours
> of slop in the date checking.

Yes, but we want to give them that leeway for timezone issues.  The
warning is okay, but an explicit MAY sounds like a *permission* to
be more restrictive, not like a *warning* "some servers out there
might be more restrictive".  

Keep the MAY if you like it, it's no important point.

Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 30 Apr 2007 16:37:55 GMT

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>Frank Ellermann <[email-masked]> writes:
>> Russ Allbery wrote:

>>> + it ought not be aggressively short.  For For Usenet, for example,
>>> + a cutoff interval of no less than seven days is conventional.

>> Yes, no MUSTard needed here, the consequences are clear.  Commercial
>> servers with shorter cutoff intervals for "binary" newsgroups may
>> need the support staff to answer complaints by their angry customers.

>needed), note that the cutoff date for history retention is independent of
>the period of article retention in the spool.  Usually the latter is
>longer than the former, but it may also be shorter.  They don't need to be
>related.

You surprise me. I would have expected the article period retention to be
used if longer than the cutoff, on the grounds that, if you still have the
article stored, then it is easy to detect that anotherone that arrives
should be ignored. OK, I suppose the site might keep two history files one
of which (in a more abbreviated form) was just for cutoff.

>Normally, the cutoff date is set uniformly across the server regardless of
>the group, but the article retention period may be set much shorter for
>binary groups.  The requirement is then that the server keep a record of
>those received and expired binary articles in the history file even after
>deleting the article off disk until they meet the cutoff requirements.

Yes, that is what CNews does. If the actual retention is less than the
cutoff, the article is removed, but the history file remembers it to
prevent duplicates. But if the actual retention is longer, then that
longer period is used for both purposes.

>This was a bit of a mess.  In my current draft, I've simplified this whole
>situation by saying that posting agents doing multiple injection MUST add
>the Injection-Date header field.  This makes all existing multiple
>injection posting agents non-compliant, but we also did the same thing
>with injecting agents and their Injection-Date requirement, so this seems
>symmetric.

>I would prefer that the whole Injection-Date feature be either a SHOULD or
>a MUST, rather than using differing levels of requirement for different
>agents.  Thoughts?  And does anyone want to argue that the whole feature
>should be a SHOULD?

In the case of a simple singly-injecting agent that does not delay
injection, then I don't think we care whether it adds Injection-Date or
not, so MUST would be too strong there. We say that if it is >1day late,
then he SHOULD add it (but it's his own fault if he doesn't).

>>> + the goal is to not create multiple independent articles but
>>> + rather to inject the same article at multiple points and use the
>>> + normal duplicate suppression facility of Netnews (see Section 3.3)
>>> + ensure that any given agent only accepts the article once.

>> Nit: s/use/let/

>Fixed.

>> The move from "reinjection" to "multiple injection" resulted in a
>> far better readable story.

>Yes, I was rather happy with that.  Thanks to Harald for the suggestion.
>I think that casting of the idea makes everything much simpler.

>>> -  A posting agent SHOULD normally reject such articles.

>> Odd, was that really a "posting agent" rejecting articles ?

>Yeah.  Dumb idea on my part, in retrospect.  It was discussed in some of
>the side-threads of this overall discussion, and I decided to drop it as
>part of these changes since the bits that it was referring to have now
>substantially changed anyway.

>>> + The posting agent MUST supply the Message-ID, Date, and
>>> + Injection-Date header fields, and the proto-article offered to
>>> + each injecting agent MUST be identical.

>> Some lines above it was a "multi-SHOULD", now you have a "multi-MUST"
>> for the Injection-Date.  How about this:

>point and I'd love to hear other comments.

I think I have lost the context here, but if this id multiple-injection,
then MUST is needed.

I have no problem with existing agents that move from "conforming" to
"legacy" when our standard comes out, provided we arrange (as we have)
that they continue to interoperate.

>>> + It MUST remove any Xref header field and either rename or remove
>>> + any Injection-Info header field and other trace fields.

>> I see why they MUST remove Injection-Info, an old injecting agent would
>> let it pass causing havoc.  But why MUST a posting agent remove Xref,
>> this job could be handled by the injecting agent ?

>> Is it "a simple MUST is better than more convoluted statements",
>> or do I miss a serious Xref-problem here ?

>It's more the former, plus backwards compatibility.  Existing injecting
>agents will reject articles with Xref rather than remove the header.  It
>seems simple enough given the other context to have posting agents remove
>it, and existing posting agents remove it, so I just said MUST instead of
>adding the explanation that existing injecting agents will reject and then
>go through and try to change the language to allow injecting agents to
>cope with it.

Xrefs only become of interest to clients of the serving agent that
generated them. Removal of redundant Xrefs by earlier agents is just a
matter of tidiness and possible removal of confusion, so I don't think you
have any grounds for MUST there.

>Here's what it looked like before I started writing this section of my
>reply:

>Looking at that table, and given that we have spelled out elsewhere how to
>handle articles with no Injection-Date, I think I agree with your point
>and don't see a reason for the injection agent requirement to be MUST.  We
>already have a scenario in which a post only handled by compliant agents
>gets no Injection-Date and it doesn't break anything, so it seems like
>addition of Injection-Date by an injecting agent need only be a SHOULD.

>Opinions?

If we are trying to move to a position where all articles routinely carry
an Injection-Date, then MUST is appropriate. Under option IR, which is
what you describe above, things are complicated by that MUST NOT, so under
IR a SHOULD might be enough.

But under IC, it is a simple MUST add it regardless (and put up with a
little pain from some hopefully rare legacy situations).

>This situation might change if we went with Charles's approach where
>injecting agents are permitted to add Injection-Date to posts that already
>have both Message-ID and Date.  In that case, there is no gap.

Indeed. I think we need to resolve the IR/IC issue before we can finalize
any wording here.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Russ Allbery <[email-masked]>
Date: Wed, 02 May 2007 19:59:17 -0700

Charles Lindsey <[email-masked]> writes:
> Russ Allbery <[email-masked]> writes:

> You surprise me. I would have expected the article period retention to
> be used if longer than the cutoff, on the grounds that, if you still
> have the article stored, then it is easy to detect that anotherone that
> arrives should be ignored.

Since articles are not usually stored by message ID, most implementations
will not find it any easier to verify that they already have an article
when the article is still in the spool.

However, related to that, INN and CNews both use the history file for
various protocol operations, such as retrieving articles by message ID.
There's no inherent need to use the history file for this purpose; it just
happened to be convenient to extend it with a few additional fields since
it was already keyed by message ID.  This is just an implementation
detail, and it's fair game to have the history entirely independent of the
local spool.

That wasn't really what I was driving at, though.  What I was saying was
that the cutoff period for history entries (how long they're retained
regardless of whether the article is still stored) is often shorter than
and independent of the article retention time for articles that were
received.  It so happens that entries for articles still in the spool are
also retained in history past the cutoff, but that's irrelevant for the
section of the protocol that we're talking about.  Since not *all* seen
articles are retained for that longer period, the existence of a few
longer-term entries doesn't mean the cutoff is any longer.

> longer period is used for both purposes.

INN works the same way.

>> This was a bit of a mess.  In my current draft, I've simplified this
>> whole situation by saying that posting agents doing multiple injection
>> MUST add the Injection-Date header field.  This makes all existing
>> multiple injection posting agents non-compliant, but we also did the
>> same thing with injecting agents and their Injection-Date requirement,
>> so this seems symmetric.

>> I would prefer that the whole Injection-Date feature be either a SHOULD
>> or a MUST, rather than using differing levels of requirement for
>> different agents.  Thoughts?  And does anyone want to argue that the
>> whole feature should be a SHOULD?

> then he SHOULD add it (but it's his own fault if he doesn't).

I didn't mean that I would ever say a posting agent MUST add
Injection-Date unconditionally.  What I mean is more like what we have
with <path-diagnostic>.  Right now, the draft has the feature as a whole
at the SHOULD level.  It's possible to ignore <path-diagnostic> completely
and implement the protocol without violating any MUSTs, only SHOULDs.

I'd prefer to make a similar decision one way or the other for
Injection-Date.  Either the entire feature is only a SHOULD, in which case
it should be possible to implement any agent completely ignoring
Injection-Date and never violate a MUST, or we consider the feature
mandatory, in which case both posting agents and injecting agents MUST use
it when appropriate.  I don't think it makes as much sense to end up in a
mixed situation where Injection-Date is a MUST for injecting agents but
only a should for multi-injecting posting agents, or vice versa.

> I think I have lost the context here, but if this id multiple-injection,

Yeah, that's the context.  That's the direction that I went, since my
feeling was that we wanted Injection-Date to be a MUST implement feature.

>> agents will reject articles with Xref rather than remove the header.
>> It seems simple enough given the other context to have posting agents
>> remove it, and existing posting agents remove it, so I just said MUST
>> instead of adding the explanation that existing injecting agents will
>> reject and then go through and try to change the language to allow
>> injecting agents to cope with it.

> matter of tidiness and possible removal of confusion, so I don't think
> you have any grounds for MUST there.

I think interoperability is fine grounds for a MUST.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 3 May 2007 10:31:14 GMT

>only a should for multi-injecting posting agents, or vice versa.

In that case, I think I prefer MUSTs in the relevant places (essentially
what is in your proposed text).

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Thu, 03 May 2007 08:04:50 -0400

Russ wrote:
 >  I don't think it makes as much sense to end up in a

I don't understand this.

There are lots of cases where SHOULD/MUST are conditional (apply to
certain situations.)

Our effort is to agree on language to optimize what happens with delayed 
injections, which happens rarely.

So a "mixed situation" seems OK, especially if the burden is shouldered
by those who intend to reap the benefit.

Ha!  I see a large irony with that statement, because the underlying
principle of fairness is violated within this WG all the time.

It's nice that Russ is spending so much time hammering out language
and details.  But this really should be someone else's burden to get
a working solution.  It is obvious that whoever conceived Injection-Date,
did not understand the implications, and got it wrong, and is not
working here to get it right.

USEFOR/USEPRO has been delayed countless months/years because someone
suggested a half-baked idea, and then expects the WG to figure out
a fully-baked solution.  I don't see how the charter indicates that
was supposed to happen.  But here we are again.

Is this feature really so important that we need all this discussion,
adding paragraphs and paragraphs about the internal working mechanisms
of history files?

Based on all the discussion, isn't it obvious that trying to make 
"Injection-Date" work is a big problem?  Doesn't that indicate a
half-baked idea, not ready for standard track?

I'd still recommend:
   Contrary to RFC-2822, the contents of the Date header field in an
   article MUST be close to the time of injection, and MUST be identical
   in all copies of an article.  A "Composition-Date:" header field MAY
   be used to indicate the time an article was composed.  Servers MAY
   choose to act on the contents of Composition-Date.

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 4 May 2007 11:34:10 GMT

>There are lots of cases where SHOULD/MUST are conditional (apply to

Yes. All Russ is asking for here is that two comparable situations should
bother say either MUST or both not say it. There are still other cases
where SHOULD would be strong enough.

>working here to get it right.

The people who conceived injection date are the same people who are
discussing it now. All that has changed is that we have seen some benefit
(especially if option IR wins) in allowing Posting Agents to insert it,
and are working out the details.

>   choose to act on the contents of Composition-Date.

But Canute cannot stop the tide. User agents are already routinely
inserting the Date at composition time, and you will have a hard struggle
to stop them.