Subject: #1416: USEPRO 3.9: Reinjection and Injection-Date

From: Russ Allbery <[email-masked]>
Date: Sat, 20 Jan 2007 16:57:15 -0800

Okay, I took a long walk to think about this (which I should have done
before sending the last message -- I'm sorry, Charles, I'm confused by all
the specific examples but after thinking about this on a walk, I can see
the general drive of what you're getting at).  Let's take a step back and
look at the theory.  I think it's the specific use cases that are mixing
things up.

First, let's go back to the Netnews protocol as it exists today and look
at how injection and reinjection work without Injection-Date.

Every article has a unique identity in its message identifier, but we
don't expect agents to retain a database of every message identifier.
Therefore, in practice, articles are uniquely identified by the
combination of message identifier and Date header field.  If the Date
header field is recent, we can rely on a database lookup for the message
identifier to know if we've seen this article before.  If the Date header
field is old, we may not know whether we've seen this article before or
not, so we assume we have.  Therefore, an article's functional identity is
the Date and message identifier pair.  Uniquely identifying an article is
how we prevent duplicating that article on the network.

Now, in the current Netnews protocol, an injecting agent is responsible
for ensuring that every article has a unique identity by adding
Message-ID and Date header fields if they're not present in a
proto-article.  However, the posting agent MAY assert the identity of the
article by filling out the Message-ID and Date header fields in advance,
in which case the injecting agent MUST use the existing identity of the
article.

In any case in which multiple injection happens, either in parallel
(submitting the same proto-article to multiple injecting agents) or in
serial (reinjection), the identity of the article must be preserved to
avoid the possibility of duplicating it on the network.  So long as the
identity of the article is preserved, every agent on the network can
independently be assured that it won't accept a duplicate of an article.
This independent verification is important because it allows for multiple
independent paths of propagation.

Note that for proto-articles, this guarantee only holds if the posting
agent provides the identity of the article.  Therefore, in the existing
Netnews protocol, it's best practice for a posting agent injecting an
article at multiple injection agents to provide the complete identity of
the article (both Message-ID and Date header fields) so that every copy of
the article has the same identity and no agent will accept multiple copies
of it.  Reinjection similarly can be assured of not creating loops simply
by preserving the identity of the article (both Message-ID and Date header
fields) when reinjecting.

Note, however, that this means that the full propagation of a message is
constrained to only those sites that it can reach prior to the expiration
of its Date header field.  So currently, reinjection breaks down if the
reinjection cannot happen within the staleness period of the Date header
(and it breaks down differently at different sites, so some sites in a
network may accept a delayed reinjected article and others may reject it).
This is the same as for relaying.  The only difference is that reinjecting
agents are generally the sole path to reach a particular site, so if the
reinjection is delayed, articles go missing.  Because of this, some
reinjecting agents change the identity of the message by regenerating the
Date header.  More on the risks of that in a moment.

Now, the problem with this is that the Date header field carries two
meanings.  It's part of the identity of the article, and it's also
human-readable information about when the message was composed.  Some
posting agents always treat it as the former and either let the injecting
agent generate it or only generate it at injection time.  Some posting
agents treat it as the latter and generate it at message composition time.
This means that proto-articles that are held for a time before injection
will either lose information about when the message was composed or will
run the risk of being rejected by servers because their Date header field
is stale.

The working group therefore introduced a new header, Injection-Date, which
serves *only* the protocol function and separates the protocol function
from the user presentation function.  Date is now of interest only to
humans and contains the composition time, which serves no protocol
function.  Injection-Date is part of the identity of the message.

However, as part of this change, we also changed who was responsible for
establishing the identity of a message.  Now, the injecting agent *always*
establishes the identity of a proto-article, and the posting agent isn't
permitted to do so.  Only when a proto-article is injected does it acquire
a unique identity in practice; prior to that, it will only have a message
identifier, which is not sufficient alone to prevent duplication of the
message.

This isn't a problem only for reinjection.  It's a problem for every case
where an article is touched by multiple injecting agents.  In particular,
since the introduction of Injection-Date, a proto-article that's injected
at multiple injecting agents will be assigned a slightly different
identity by each one.  In the normal case, these identities will only vary
by seconds or at most minutes, which in practice is highly unlikely to
cause problems, but theoretically we still broke the identity model.
Since the introduction of Injection-Date, time-delayed multiple injection
is no longer safe against duplication.  If a posting agent injects an
proto-article at one injecting agent immediately and then at another
injecting agent some days later, it may introduce a duplicate article into
the network, and there's nothing the posting agent can do except not do
time-delayed multiple injection to prevent this.  Prior to Injection-Date,
it could make sure that all copies had the same identity and the later
injection would never duplicate, just possibly suffer from poor
propagation because it's stale.

Of course, serial multiple injection, reinjection, suffers the most, since
it is the most likely to happen some time later.  So in usepro-06,
reinjection was distinguished from injection and only in the reinjection
case (as determined by the injecting agent) was the article permitted to
retain its original Injection-Date.  In other words, normal posting agents
were still not permitted to assert the complete article identity, but a
special type of posting agent was if an injecting agent chose to let it.

So, we have some competing goals:

 * We want every copy of an article to have the same identity so that
   duplicates can be effectively suppressed at any node of the network
   without requiring reliance on prior nodes to do verification in any
   particular way.  This is robust against multiple propagation paths.

 * We can't rely on the Date provided by a posting agent to be part of
   the article's identity because then we'll give some articles too old
   of a date and fail to propagate them.  In other words, we have to
   treat a Date header field provided by a posting agent as a comment
   rather than a protocol element.

 * We want to support passing the same proto-article to multiple injecting
   agents and reprocessing an article through an injecting agent because
   both are useful in practice in some specific situations and both have
   historically been supported by Netnews.

So far, we have a couple of different solutions.

 * The current draft, my approach, basically takes the stance of "well, we
   asked for it, so we take the consequences."  Multiply injected articles
   simply have different identities -- that's the consequence of how we
   defined Injection-Date.  Each time an article passes through an
   injecting agent, its identity changes, and in the case where it's most
   likely that identity change will be serious (reinjection), the agent
   doing the reinjection is required to verify as well as possible that it
   won't create the chance of duplicates.  In practice, the drawbacks are
   hopefully limited.  There's no way for that check to be perfect, but
   hopefully it's no worse than the other case of multiple injection that
   we're already dealing with.  As a nice side effect, this also means
   that delays in reinjection don't cause articles to be lost.

 * usepro-06 makes reinjection a special case with a different set of
   rules that require that the identity of the article be preserved.  In
   other to do this, negotiation between the posting agent and the
   injecting agent is required and software has to know whether it's doing
   injection or reinjection.  There are some follow-on effects for trace
   headers, but this could be separated out and handled differently.  The
   core point is that reinjection preserves article identity, but in the
   process maintains the current drawback that delays in reinjection may
   cause articles to be lost.  There may be a bit of an attractive
   nuisance here, the same one that's present in the existing protocol,
   for reinjecting agents to drop the Injection-Date header anyway so as
   to not lose articles, thereby reintroducing the problems the current
   draft causes.

Forrest has proposed only allowing reinjection between disjoint networks.
If the reinjecting agent is the only possible path between two otherwise
disjoint networks, then the reinjecting agent can do article identity
verification for the second network and it doesn't matter if the article
acquires a new identity when crossing that network boundary.  The
difficulty here, as Charles correctly points out, is that there's no
general way of establishing that two networks are disjoint.  One can
sometimes determine for individual articles that they passed through a
host on network A and on network B and therefore the networks are not
disjoint, but to be sure would require full knowledge of at least one of
the networks.  There are some common cases where this is possible, but
it's not possible in general, and it's possible to think that you know the
networks are disjoint and be wrong.

There is another solution that require modifying USEFOR:

 * Drop Injection-Date entirely and go back to the current protocol.  This
   solves this problem and reintroduces the problem that we were trying to
   fix originally.

There is another solution that arguably requires modifying USEFOR:

 * Change the definition of the Injection-Date header field so that the
   posting agent can provide it, and indeed MUST provide it if they wish
   Date to be treated as a comment rather than a protocol element.
   Require posting agents doing multiple injection to include either a
   Date or (preferrably) a Posting-Date header.  Don't allow injecting
   agents to set Injection-Date at all if Date was provided.  This has the
   advantage of solving the multiple injection problem (present in both my
   draft and in usepro-06), and allows reinjecting agents to assert the
   same article identity in the same way as is possible with the current
   protocol.  Basically, this would make Injection-Date *entirely* the
   same as the protocol purpose of the current Date header, rather than
   mostly the same.  It has the same difficulties with delayed
   reinjection.

One could make the argument that we could do this now, that nothing in the
current USEFOR definition says that the posting agent can't provide this
header.  The current text implies it, but doesn't say so outright.  The
header name is confusing if the posting agent is providing it, but we
could live with that.

The drawback of this approach, of course, is that it still doesn't deal
with the issue of delayed reinjection.  Some reinjecting agents are going
to want to change the identity of a message so that it will still be
accepted even though it's old.  Doing so inherently runs the risk of
creating duplicates, but doing so is going to be a common desire --
gateways go down or off-line for a while, networks go out, people want to
pull down older news to bootstrap or seed a new Netnews network, etc.  If
we just outlaw it, we have no control over how people do it if they choose
to break the protocol.  If we can come up with something useful to say
about *how* to change the identity of the message, maybe we can reduce the
risk.

That's the summary of the current situation as I see it.  I think it's the
most complex problem we have left to resolve before we have a publishable
draft.  Every approach to this problem that I can see introduces risk
somewhere, including sticking with the current protocol and thereby
tacitly encouraging people to drop the Date header when they need to.  My
approach in the current draft is a mess, but so is every other approach
that I can think of.  It doesn't, however, deal with multiple injection,
which would be valuable.

If we want to pursue the last option mentioned above, namely requiring the
posting agent to provide Injection-Date if it wants Date to be ignored
(ideally renaming and slightly redefining the header in the process), I
can propose some text, but we still need to decide what, if anything, we
want to say about delayed reinjection.  I think this route is cleaner than
my current draft, and it takes an important step back towards Charles's
draft in that we would then require Injection-Date not be changed by the
injecting agent if already present (modulo whatever we wanted to say about
intentionally changing article identity).

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Sat, 20 Jan 2007 23:42:42 -0500

Russ Allbery wrote:
>   I think it's the specific use cases that are mixing
> things up.

I think that indicates that no matter what we do, someone is going
to screw it up sometime, somewhere.

Russ, I think your lengthy analysis is accurate, except for two points.

1. The article identity (for preventing duplicates) is not actually a 2-tuple
of Message-ID, Date.  It is Message-ID + fuzzy arrival date.  Every date within
the past 3 days is probably as good as any other date in that range.  The
header fields do not always set the roll-off from message-id history.  It is 
arrival order.  That changes your conclusions a bit too, I think.

Overconstraining the solutions to guaranteeing (Message, Date) is not
strictly necessary, and I think you throw out some potentially rewarding
arrangements.

2. I think you present all branches of your fault tree as if they were all
equally likely in practice.  I disagree with that approach when we
admit we don't have a solution for all cases for all people.

I prefer to take the most likely of situations, standardize something that 
works, and then ignore the least likely branches.  They fall outside our
standard, or they can go look at Duties of a Gateway and figure out something.

If we disallow "late (re)injection", don't most of the problems go away?

Sure, I'd like late reinjection to work too, but it doesn't happen enough to 
work to standardize.  Let some future standard tackle it.

So, after all this time, no one seems to have proposed something that we have 
consensus that works.  Too bad.  It isn't our problem that the people who 
proposed Injection-Date as a solution to this problem didn't get it right.  If 
they want it, they can propose something that gets consensus.  I don't care to
fix it for them.

My opinion: Standardize reinjection from private leaf nodes (it's common).
Standardize multi-point injection.

Drop late injection.  It almost requires a flag day, not because we get
duplicates (we would), but because the contents of a newsgroup is going to be 
different at different sites, depending on whether they implemented 
expiry/incoming relay on Injection Date, or Date.  And isn't that is the problem 
that it was intending to prevent?  If an article is re-injected late, we have
the amusing situation of an article appearing, not appearing, and appearing
as a duplicate on various servers.

And all this complexity about late injections for what?  A handful of posters 
per day on Usenet?  Hours later is not going to matter.    How many posters per 
day on Usenet will be posting days later than authoring?  Does anyone have an 
estimate?

Aren't we knocking ourselves out over mere annoyance?  Just tell those posters 
that their message had better have a current Date: header if they want it to 
propagate, (that's reality, and telling them different is not helpful) and if a 
late Date header seems like deception, they can note it in the body of their 
message. If they want a sooner expiry, add a header field.  Voila!  Backwards
compatible too.

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Sun, 21 Jan 2007 00:00:15 -0500

> Forrest has proposed only allowing reinjection between disjoint networks.

Well, no.  I was coerced.  My original text said little.  Then I upped it
to handle leaf nodes.  Then I was coerced to consider the disjoint network 
multi-injection.  This WG is a broadening experience, doncha know.

> networks are disjoint and be wrong.

I don't think we can solve the general case.  Are we going to try to standardize 
"Daves the resurrectors"?  Yikes!

I fully admit I provided text that applied for the narrow case, but I think it
is the extremely most common case.

If you are a private leaf node, you know you want to reinject articles posted 
locally and no others.  In this case, you KNOW articles did not pass through A 
nor B, because they originated at your node and you haven't relayed them yet,
since you don't relay.  QED.

You can implement this many different ways.  I don't think we need to 
standardize that implementation.

Sure, I understand there are people who will want to reinject who are not
private leaf nodes.  They are not my concern unless they can fit the description 
in the text we provide.

Let those with other cases read Duties of Gateways.  They don't get
a standard formula, because this WG doesn't have a consensus about
their needs and prevalance, and the Injection-Date scheme that was
proposed is now consensed to be faulty (although different WG members
think it is flawed for different reasons.)

---

From: Russ Allbery <[email-masked]>
Date: Sat, 20 Jan 2007 21:36:57 -0800

> 1. The article identity (for preventing duplicates) is not actually a
> 2-tuple of Message-ID, Date.  It is Message-ID + fuzzy arrival date.
> Every date within the past 3 days is probably as good as any other date
> in that range.  The header fields do not always set the roll-off from
> message-id history.  It is arrival order.  That changes your conclusions
> a bit too, I think.

> strictly necessary, and I think you throw out some potentially rewarding

This is, of course, entirely correct, and as soon as I read this, I went
"doh, yes."  However, after poking at this idea for a few minutes, I
couldn't come up with an effect of the distinction that would change the
conclusions.  Could you elaborate a bit more?  Having history expiration
be by arrival rather than by header means that in practice servers will
keep a longer history than strictly necessary and could check some older
articles, but if the Date header (by whatever name) is stale, the article
still loses.

(Plus, I think it's actually conforming to our current specification to
write a news server that expires history entries based on the
Injection-Date header of the article rather than by arrival time.  INN
optionally can do this, and I think such an implementation still fulfills
the requirements of the standard.  That, of course, doesn't change your
point, but does mean that if we want to use the normal arrival-based
expiration, we need to require it somewhere.)

> 2. I think you present all branches of your fault tree as if they were
> all equally likely in practice.  I disagree with that approach when we

Yeah, that was partly a side effect of trying to look at this at a
theoretical level rather than analyzing practical examples.  I'm worried
that we're not agreeing on the overall theoretical analysis structure
before discussing what we can do for individual cases.

It's true that not all of these problems are equal.  For example, we're
theoretically breaking our uniqueness model with multiple injection a
minute apart.  This opens the door to duplicates.  However, to get a
duplicate in practice, the later copy of the article would have to arrive
less than a minute after the earlier copy had expired, something that in
practice is hideously unlikely.

> I prefer to take the most likely of situations, standardize something
> that works, and then ignore the least likely branches.  They fall
> outside our standard, or they can go look at Duties of a Gateway and
> figure out something.

> If we disallow "late (re)injection", don't most of the problems go away?

If we disallow late injection in general, both injection and reinjection,
then we don't need Injection-Date.  If we can drop Injection-Date, we get
back the current Netnews protocol, which we know works and which we can
make statements about based on current practice.  In this case, we can add
a requirement that posting agents doing multiple injection always provide
the same Message-ID and Date header fields in all copies and that
reinjection always preserves both of those headers, and everything works
reliably and without duplicates except for late injection and reinjection.

We may or may not want to say something about how to work around late
reinjection, or we could just leave it to the gateways section for someone
who really knows what they're doing.

This, however, reverses a decision made in USEFOR, which we were treating
as sacrosanct, which is why I was trying not to push that route.  (It's
been my preference for a while.  I think Injection-Date introduces more
problems than it solves and the complexity isn't worth it.)

> duplicates (we would), but because the contents of a newsgroup is going
> to be different at different sites, depending on whether they
> implemented expiry/incoming relay on Injection Date, or Date.  And isn't
> that is the problem that it was intending to prevent?  If an article is
> re-injected late, we have the amusing situation of an article appearing,
> not appearing, and appearing as a duplicate on various servers.

> And all this complexity about late injections for what?  A handful of
> posters per day on Usenet?  Hours later is not going to matter.    How
> many posters per day on Usenet will be posting days later than authoring?
> Does anyone have an estimate?

> Aren't we knocking ourselves out over mere annoyance?  Just tell those
> posters that their message had better have a current Date: header if
> they want it to propagate, (that's reality, and telling them different
> is not helpful) and if a late Date header seems like deception, they can
> note it in the body of their message. If they want a sooner expiry, add
> a header field.  Voila!  Backwards compatible too.

This is basically the line of reasoning under why I put Date staleness
checking back in for injecting agents.  I think that if we're introducing
Injection-Date, we need to provide a transition during which it's clear
that Injection-Date still isn't going to be a panacea, and in that
transition period, I don't think we're doing posters any favors accepting
articles that most people will keep ignoring.  It may be that checking at
the injecting agent so that we can return an error message isn't the best
transitional plan, but it feels better to me than not giving people any
warning at all.

One point on which I think Charles is definitely right, though, is that
Injection-Date is a different thing than Injection-Info and the other
non-proto-article headers like Xref.

---

From: Russ Allbery <[email-masked]>
Date: Sat, 20 Jan 2007 21:41:42 -0800

> Well, no.  I was coerced.  My original text said little.  Then I upped
> it to handle leaf nodes.  Then I was coerced to consider the disjoint
> network multi-injection.  This WG is a broadening experience, doncha
> know.

*heh*.  Sorry if I mis-stated there!  :)

> If you are a private leaf node, you know you want to reinject articles
> posted locally and no others.  In this case, you KNOW articles did not
> pass through A nor B, because they originated at your node and you
> haven't relayed them yet, since you don't relay.  QED.

> standardize that implementation.

That makes sense.  Basically, if we say that you're only allowed to do
reinjection that changes the identity of the article (dropping
Injection-Date) when you *know* that the networks are disjoint, and leave
the problem of knowing that to the implementation, it handles the simple
leaf-node case while leaving the question of how to determine this in more
complicated cases to the individual.

We can still get problems if that knowledge is wrong, of course.

> Let those with other cases read Duties of Gateways.  They don't get a
> standard formula, because this WG doesn't have a consensus about their
> needs and prevalance, and the Injection-Date scheme that was proposed is
> now consensed to be faulty (although different WG members think it is
> flawed for different reasons.)

There is that -- the Gateways section allows for Netnews-to-Netnews
gateways with more general descriptions of what sorts of care one has to
use, and is available as a fallback description of what has to be done for
weird and difficult cases.

---

From: Frank Ellermann <[email-masked]>
Date: Sun, 21 Jan 2007 09:57:31 +0100

Russ Allbery wrote:

> if we want to use the normal arrival-based expiration, we need to
> require it somewhere.

Or allow it somewhere.  I don't get the difference if articles older
than "X" determined by a header field are anyway rejected as too old.

> For example, we're theoretically breaking our uniqueness model with
> multiple injection a minute apart.  This opens the door to duplicates.
> However, to get a duplicate in practice, the later copy of the article
> would have to arrive less than a minute after the earlier copy had
> expired, something that in practice is hideously unlikely.

Hitting one critical minute of 4321 for 4320=3*24*60 isn't impossible,
but finding a route to this server needing 3 days might be difficult.

> then we don't need Injection-Date.

Yes, but that's not realistic.  Users can poll their server, and then go
offline for some time creating articles with different days, waiting in
their outbound for better times.  Then they go online again and try to
post their articles.  That's one scenario where Injection-Date could be
better than Date.

Another scenario is a xyz2news gateway where the xyz comes with a Date,
but may take some time (fido2news, mail2news).  It's then desirable to 
preserve the original Date.  Unless there's another xyz2news gateway
injecting the same article with a very different earlier Injection-Date.

Either I'm too clumsy to get this right, or both USEPRO drafts don't
discuss the function of a history file yet.  S-o-1036 9.2 explains how
that's supposed to work, proposing a minimum of N = 7 days for "seen",
with a Date older than now - N considered as "stale".

If it's really not yet there USEPRO needs a similar section with the
same RECOMMENDED N = 7 days minimum.

> I think Injection-Date introduces more problems than it solves and
> the complexity isn't worth it.

The reinjection part was somewhat odd, I never got it.  Based on your
explanation I finally begin to understand the USEPRO-06 idea.  Your
proposal to treat reinjection as special case of gatewaying offered a
way to ignore these oddities.

But the underlying concepts (history + dupes + late injection) have
to be clear.  Going back to no Injection-Date doesn't help for late
injections, and that's no irrelevant corner case like reinjection.

>> How many posters per day on Usenet will be posting days later than
>> authoring?  

Poll Friday, write Saturday, and inject Monday could be a plausible
scenario.  The USEFOR decision to stick to RFC 2822 where that's at
all possible was intentional.  2822 defines:

 The origination date specifies the date and time at which the creator
 of the message indicated that the message was complete and ready to
 enter the mail delivery system.  For instance, this might be the time
 that a user pushes the "send" or "submit" button in an application
 program.  In any case, it is specifically not intended to convey the
 time that the message is actually transported, but rather the time at
 which the human or other creator of the message has put the message
 into its final form, ready for transport.  (For example, a portable
 computer user who is not connected to a network might queue a message
 for delivery.  The origination date is intended to contain the date
 and time that the user queued the message, not the time when the user
 connected to the network to send the message.)

However s-o-1036 states:

 The Date header contains the date and time when the article was 
 submitted for transmission

That's obviously incompatible, and the reason why USEFOR adds the new
beast Injection-Date for the function related to history files.  Just
removing Injection-Date would also cause havoc for combined UAs (mail
or news), implementors can't get incompatible Date semantics right...
or maybe they could, but they won't bother, and that's the problem.
 
> I think that if we're introducing Injection-Date, we need to provide
> a transition during which it's clear that Injection-Date still isn't
> going to be a panacea, and in that transition period, I don't think
> we're doing posters any favors accepting articles that most people
> will keep ignoring.

If injecting agents behave like relaying agents, and use the Date as
surrogate for the missing Injection-Date, then they'll reject a Date
older than N = 7 days.  That should work to some degree.  Or there's
a smaller window, less than N days, for this purpose.  Declaring when
the transition period is finished is the job for another draft in the
late 40s of this century.  Maybe 2048+, then it's in sync with BCP 18.

Frank

---

From: Russ Allbery <[email-masked]>
Date: Sun, 21 Jan 2007 09:25:39 -0800

>> If we disallow late injection in general, both injection and
>> reinjection, then we don't need Injection-Date.

> Yes, but that's not realistic.

It should be a warning sign for a train of logic if the thing that one
declares not realistic is how all of Netnews currently works and what RFC
1036 requires.

It's realistic; Netnews has worked that way for decades.  The question is
rather whether the benefits from changing it are worth the costs, and
proceduraly, whether it's worth reopening the topic rather than working
with what we've got now in USEFOR.

> Users can poll their server, and then go offline for some time creating
> articles with different days, waiting in their outbound for better
> times.  Then they go online again and try to post their articles.
> That's one scenario where Injection-Date could be better than Date.

Or said off-line posting agent can rewrite Date when the message is
actually injected.  That's what they have to do now.  You lose information
for humans, but the protocol works.

> same RECOMMENDED N = 7 days minimum.

We already discussed this years ago and reached a working group consensus
to not document any specific retention time, but instead to document the
restraint on retention time (namely that you have to reject any article
with a Date older than your retention time).  I'd prefer not to revisit
that decision again.

In practice, people use all sorts of different retention times, ranging
from three days to a year (and in some specialized cases, even shorter
than three days -- I've used a retention time as short as a day before).

---

From: Frank Ellermann <[email-masked]>
Date: Sun, 21 Jan 2007 22:37:26 +0100

Russ Allbery wrote:
 
> declares not realistic is how all of Netnews currently works and what 
> RFC 1036 requires.

The incompability of the 2822 Date and the s-o-1036 Date is obvious, is
that the reason why you invented the new Injection-Date here ?

  [s-o-1036 explanation of "history"]
 
> We already discussed this years ago and reached a working group 
> consensus to not document any specific retention time, but instead to
> document the restraint on retention time

The reason for N = 7 given in s-o-1036 is convincing, that's also long 
enough to tolerate plausible "late injections" by implementations of an
RFC 2822 Date.  Maybe it's unnecessary after a "transition period" for
the introduction of Injection-Date, but that's years away.

> people use all sorts of different retention times, ranging from three
> days to a year (and in some specialized cases, even shorter than three
> days -- I've used a retention time as short as a day before).

news.clara.net also tried shorter times for their binary groups, with
many unhappy users wanting their money back, because they didn't get all
pieces of the binaries.  I don't care much about attempts to distribute
complete CD images with NNTP.  But combined mail+news UAs implementing
the RFC 2822 Date instead of a "posting date" make sense.

Frank

---

From: Russ Allbery <[email-masked]>
Date: Sun, 21 Jan 2007 13:53:36 -0800

> that the reason why you invented the new Injection-Date here ?

No.  (I thought you were here for the previous discussions?)  We invented
Injection-Date to solve a specific problem with off-line posts.  Avoiding
having to redefine an RFC 2822 header field was an additional advantage,
but wasn't the initial motivation.

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Mon, 22 Jan 2007 08:37:00 -0500

> with what we've got now in USEFOR.

Why should we treat USEFOR as sacrosanct?  I mean we don't want to
descend into any more tarpits, but why ignore hindsight?

> for humans, but the protocol works.

Here's a thought....

USEFOR should have defined "Authoring-Date:", or some such thing, not 
"Injection-Date:"  This would be completely backwards compatible, and put the 
work to resolve it on reading agents.

Let's go back and fix USEFOR.

---

From: Russ Allbery <[email-masked]>
Date: Mon, 22 Jan 2007 09:18:52 -0800

> "Injection-Date:"  This would be completely backwards compatible, and
> put the work to resolve it on reading agents.

Yeah, that was my fourth proposal.  It's possible to argue that it doesn't
require any change to USEFOR.

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Mon, 22 Jan 2007 13:09:26 -0500

> require any change to USEFOR.

I didn't understand that as your fourth proposal.  I am saying that we
don't modify servers at all.  UA can generate and look at Authoring-Date,
and servers look at Date, just as they do now.

I still want to see statistics about how often this late posting business
happens.  People shouldn't be posting stale articles to active discussions.
Most discussions would have moved on between when they authored and when they 
injected days later.  It is rude to think everything you write is timeless.

And if you read all the intervening messages before you late posted, just
to ensure that what you wrote days ago is still applicable, why should the
date header be a record of when you typed it, and not when you affirm the 
message to contain what you want to send?  It's splitting hairs.

---

From: Russ Allbery <[email-masked]>
Date: Mon, 22 Jan 2007 12:18:36 -0800

> and servers look at Date, just as they do now.

Oh!  Sorry, yes, that's something different.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 22 Jan 2007 23:32:48 GMT

>look at the theory.  I think it's the specific use cases that are mixing

OK, there's a lot of good stuff in there, but it is late at night, and
needs a lot of thinking about. So I have printed it out, and shall study
it carefully before coming back, which may not be for a day or so.

But just one immediate reaction. On reading it through, I kept saying to
myself "Ah But! There is more to be said on that". And then a few further
paragraphs down you indeed "said more on that". Which means, perhaps, that
there is some convergence afoot.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 22 Jan 2007 23:35:53 GMT

>but wasn't the initial motivation.

No, I think Frank is largely right. The wording in RFC 2822 was a
significant part of the thinking that led to Injection-Date (and, after
all, that RFC 2822 wording explicitly talks about off-line preparation of
emails).

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 25 Jan 2007 17:09:55 GMT

>at how injection and reinjection work without Injection-Date.

OK, so we stop firing obscure (but possible) scenarios back and forth
while we examine these possibilities.

>Every article has a unique identity in its message identifier, but
>...........  Therefore, an article's functional identity is
>the Date and message identifier pair.

Yes, Modulo some "fuzziness" of the role of the Date, as Forrest has
pointed out.

Agreed paragraphs snipped ......

>Note that for proto-articles ........ it's best practice for a posting
> agent injecting an article at multiple injection agents to provide the
> complete identity of the article (both Message-ID and Date header fields)
.....

More than that, both Uespro-06 and -07 REQUIRE the posting agent to do
that before multiple injection.

>  Reinjection similarly can be assured of not creating loops simply

Hence it follows that if we were to make the rules for Injection-Date
exactly the same as the current rules for Date, it would result in a
system no different from the present situation, except for an increase in
the staleness margin for late-injected articles.

>....  So currently, reinjection breaks down if the
>....  The only difference is that reinjecting
>reinjection is delayed, articles go missing.

Eh? ITYM "reinjecting agents are generally the sole path for injected
articles from a particular site to reach the wider Usenet, so if ...".

>  Because of this, some
>Date header.

OK, let us coin the term "Magic Exception" for that practice. Clearly it
is an "Exception" to the laid-down rules, but why "Magic"? Because:

. It is hard to define exactly when the practice is safe, beyond "you
would know one of these situations when you see it".
. It is certainly only for those who "really REALLY" know what they are
doing, and certainly "Kiddies, don't try this at home" applies.
. It may not be documented in any standard, but people "in the know" will
recognize and condone it so long as it is wisely used - a bit like we
discussed for the $alz convention.

>  More on the risks of that in a moment.

>agents treat it as the latter and generate it at message composition time.

And we want to encourage the latter, for conformity with RFC 2822, and
contrary to the advice in s-o-1036, so explicit words of encouragement
would be in order.

>function.  Injection-Date is part of the identity of the message.

Right!

>permitted to do so.

Yes, Usepro-06 did not permit that but, in the light of this discussion, I
am thinking it went too far. I have no problem with letting the posting
agent do it in appropriate circumstances.

>  Only when a proto-article is injected does it acquire

Look at it this way. It is the "act of injection" which requires an
Injection-Date to be created. Whether it is created by the injector (the
posting agent) or by the injectee (the injecting agent) is a secondary
matter, so long as one of them does it. Of course, there are good reasons
why normal practice should be for the injecting agent to do it, but
special cases such as multiple- or re-injection could be different.

But, from the posting agent's POV, the "act of injection" comes later than
the "act of authorship". Typical current posting agents work as follows.
The poster writes an article and presses "send". At that point, a Date
header is created and the posting agent attempts to make an NNTP
connection to a server. If that fails (because the server cannot be
reached), it puts the article into an "Outbox" where it sits until a
server can be reached. Maybe it tries every 5 minutes, or maybe there is a
mechanism for it to be activated when next the user dials in. Either way,
only then should the Injection-Date be added (assuming the posting agent
is gong to add it, which would probably not be the usual case anyway).

>cause problems, but theoretically we still broke the identity model.

So if posting agents are permitted to add Injection-Date, you might
recommend them to do so when multi-injecting, though it is hardly
necessary with those short delays (delays of several days are a different
matter of course, meriting at least a warning in the document). But
typical current posting agents don't do multi-injecting, so this is only
likely to arise with savy users who have written their own injecting
scripts (or, more likely, are running their own private news servers).

>is no longer safe against duplication. ... Prior to Injection-Date,

Unless it took advantage of the Magic Exception, which you say some of
them did.

>retain its original Injection-Date.

Actually, it did say that an existing Injection-Date (however arising)
MUST NOT be altered. But it was never my intention that reinjection would
require prior agreement from the injecting agent (which is not to say that
an injecting agent has not the usual right to refuse articles which it
detects to be reinjections).

>So, we have some competing goals:

Though actually the areas of competition competition are not all that
great, leading us to hope that they can be bridged.

........

>   asked for it, so we take the consequences."........ the agent
>   hopefully limited.

The chief drawback is the difficulty for the reinjecting agent to do that
verification.

>  There's no way for that check to be perfect, but

>   injecting agent is required .....

No, there was no necessity for negotiation.

>  ..........  The

Which would be a Magic Exception, neither better nor worse than that
practised by current Magicians.

>Forrest has proposed only allowing reinjection between disjoint networks.

Apparently he did not intend to propose that. Anyway, it would be
dangerous and unenforceable.

>...........  The

Exactly! That is an excellent summary of my position. It is even possible
not to be aware that you are reinjecting.

>   solves this problem and reintroduces the problem that we were trying to

No. If, as a minimum, Injection-Date follows exactly the same rules as the
existing Date, then you are never any worse than the present situation.
OTOH, you are sometimes better because you can gain some extra time before
the staleness check hits you.

>There is another solution that arguably requires modifying USEFOR:

Actually, I think the existing USEFOR works here just fine. What it says
is:

   "This header field MUST be inserted whenever an article is injected."

No mention of whether it is the injector or the injectee that is
responsible for that, so we are free to specify the mechanism as seems
best [1]. So the first sentence in what follows is not needed.

>   Date or (preferrably) a Posting-Date header.

ITYM s/Posting-Date/Injection-Date/. Note that we already require posting
agents to provide Date and Message-ID when multi-injecting. We might
choose to add Injection-Date to that.

>                                                Don't allow injecting
>   agents to set Injection-Date at all if Date was provided.

No No! "Don't allow injecting agents to set Injection-Date at all if
Injection-Date was already provided". If Date alone was already provided,
the injecting agent should presume it was an authorship date and add a
proper Injection-Date (even if it was only a few seconds later). That
satisfies the MUST in USEFOR.

>  This has the

Which brings us back to the Magic Exception.

Yes, with the above provisos, I would support that proposal.

>header.  The current text implies it, but doesn't say so outright.

Nor could it, because the terms "posting agent" and "injecting agent" are
not even defined in USEFOR.

>  The

Not confusing at all if you regard "injection" as an interaction between
an injector and an injectee.

>with the issue of delayed reinjection.

Right! So now we need to take a serious look at the "Magic Exception".
Firstly, we need to understand in which circumstances it would be safe.
For sure, it is safe in the case of a user who runs a private news server
on his own machine and never interacts with users outside that machine
except via proper Usenet injecting agents. Fortunately, that is the
commonest case in which reinjection is likely to arise (next most common
will be exotic gatewaying situations, and after that I am not sure beyond
people who are reinjecting without realising it).

Are there any other situation where it is guaranteed safe? Perhaps a small
private network which is a 'peninsula' to the rest of Usenet, with the
reinjecting server on the 'isthmus'. But can you be sure it is really a
'peninsula'? Perhaps you can if it is all behind a firewall that blocks
all unauthorized outgoing NNTP. But the further you go beyond that, the
more "iffy" it becomes.

So, if/when we feel we understand that, do we then write some form of
Magic Exception into the draft, and if so how? Or do we leave it as a
piece of Magic only to be practised by Magicians who "really REALLY" know
what they are up to? Because, for sure, such magicians will indeed exist
and do it whatever we say (merely claiming, if asked, that their "private
server" is really just an exotic user agent from the POV of the rest of
Usenet).

>  Some reinjecting agents are going

Yes, I think that is a good summary of what we would need to look at. But
all those issues already arise, in much the same form, on the existing
network, and the predicted "Death of Usenet" still has not happened.

[1] OK, there is a hint that Injection-Date is done by news servers in an
example in the definition of "generate", but it would be a minor tweak to
fix that.

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 25 Jan 2007 17:21:58 GMT

>injected days later.  It is rude to think everything you write is timeless.

The late injection may or may not be for reasons beyond your control. But
the advantage of having Date tied to the moment of authorship (as clearly
set out in RFC 2822) means that if some reader thinks "what is this guy
talking about - where has he been these last 3 days" he can then look at
the Date header and realise "Ah! he wrote it before all the momentous
events of the last 3 days happened".

>message to contain what you want to send?  It's splitting hairs.

No, the moment when you pressed 'send' is the moment when you considered
your article complete (see RFC 2822 again). It is not the moment of
injection.

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Thu, 25 Jan 2007 12:38:44 -0500

> the staleness margin for late-injected articles.

Sure, after the flag day.

Since that won't happen, re-injectors would rely on the false statements in 
USEPRO that tell them Injection-Date is honored, when the non-upgraded servers 
still look at Date.

Don't you think that is a problem?

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 29 Jan 2007 04:46:08 +0100

Russ Allbery wrote:

 [why was Injection-Date invented]
> (I thought you were here for the previous discussions?)

No.  I found the odd "TLD invalid" invention in a draft, and
that inspired me to check what the hell this WG is planning.

That was near the last-last-last usefor-07 WGLC in 2002, and
after a few articles and Ralph's "no consensus" poll I forgot
about it, expecting that it will be ready soon (minus UTF-8).

Later I sometimes looked, but for some time I refrained from
getting involved into the debates about "Re:" and references,
or the "OUGHT-ification" phase.

In May 2004 that changed, one ASRG Chair resigned, Caller-ID
was added to the MARID agenda, and I wrote my first mail to
anything@ietf, in that case ietf-ipr@ietf, in only three days.
It convinced me that something is decisively rotten with the
standards process, and that waiting for good RFCs is no plan.

Back in 2002 there was no "Injection-Date", trying GMaNe's
search I found it, apparently you invented this 2004-02-01:

http://article.gmane.org/gmane.ietf.usenet.format/24867/

Frank

---

From: Russ Allbery <[email-masked]>
Date: Sun, 28 Jan 2007 20:01:45 -0800

> http://article.gmane.org/gmane.ietf.usenet.format/24867/

Ah, yes.  Looks like you and Charles are correct and it was originally
related to compatibility with RFC 2822's Date header.  I'd completely
forgotten about that part of the discussion.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 30 Jan 2007 14:01:38 -0800

Question:

While email gateways are outside of the scope of the WG, it might be 
relevant to ask...

what's current practice for email gateways on incoming messages? Do they:

- keep the Date: header and accept rejection for old articles?
- modify the Date: header if the article is rejected on submission?
- modify the Date: header in all cases?

If they're all currently keeping the Date: header and accept rejection, 
then I don't see any reason to bend over backwards to do something 
different; if they all mangle Date: before submission, that's a clear 
indication that they've learned to do so by bad experiences.

                Harald

---

From: Russ Allbery <[email-masked]>
Date: Tue, 30 Jan 2007 14:27:35 -0800

> indication that they've learned to do so by bad experiences.

I think it's hard to get a read on what all gateways do, since there are
so many different implementations and they're not really discussed in a
common place.  All of mine keep the Date header, though.

I'm not sure what Mailman does.