Subject: Meta-level thought: Expectations of interworking?

From: Harald Tveit Alvestrand <[email-masked]>
Date: Sun, 21 Aug 2005 20:57:35 +0200

The debate about ticket #1088 (whether or not injection-info should be 
optional) leads me to think that we could benefit from a better definition 
of what we expect to happen when this standard is fielded.....

I *think* (based on the discussion here) that people expect that 
USEFOR-conformant UAs and servers will be plugged into the existing USENET 
without any great disruption - the conformant UAs will see most articles in 
the existing USENET, the conformant servers will pass most existing 
articles, and the "old" (non-conformant, but reasonably implemented) 
servers will pass most or all articles coming from the "new" UAs and 
servers.

We certainly envision some non-interworking happening; we have explicitly 
decided that it's permitted for a server to not accept an article that uses 
EST in its "timezone" field, for instance.

But I wonder if we should make it explicit, for instance by adding to 
section 1.2 (or a new section below it it) something like the following:

  In developing this standard for the format of messages, backwards
  compatibility was given a high priority. We expect that most
  reasonably-correct messages generated by 2004-vintage Netnews
  clients and servers will be accepted by servers checking messages
  against this syntax, and we expect that most messages generated
  according to this syntax will be accepted by 2004-vintage Netnews
  clients and servers.

Based on this, it's an obvious conclusion that Injection-info and 
Injection-date HAVE to be optional in the format (since otherwise a 
conformant server would still be conformant if it chose to reject all 
articles that did not have them).

But the WG may want something else....

                     Harald

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 22 Aug 2005 08:35:04 +0200

> But the WG may want something else....

...not me, I like your explicit statement about the design
goals.  "2004-vintage" sounds odd, and "vintage 2004" can't
be much better, how about "legacy" ?  And maybe a pointer
to appendix B (differences from 1036).

                           Bye, Frank

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Mon, 22 Aug 2005 07:26:14 -0400

> articles that did not have them).

USEPRO says that an injecting agent MUST REMOVE the Injection-Info header field
for re-injection, (which is a really bad idea to bless.)  It means that a reader
cannot look at the header field and determine anything about the origin of an article
with certainty.  That makes Injection-Info easy to construct, but basically
unreliable information at the Reader end.

I think this is a part of USEPRO that needs major work.  Since we can't discuss
USEPRO right now, we can't know the final purpose of Injection-Info.  Without knowing
the purpose, how can we have any opinions on syntax, except legalistic ones about
whether encoded words are allowed in comments?  (Not that the legalistic concerns
are unfruitful.  They can be used to eliminate a lot of poll choices.)

Since Complaints-To is an injection info parameter since USEFOR-03 (which
was a bad idea I think) it will have some kind of complaints-to indication
to enable automatic generating of complaints (which is a another BAD thing,
in my opinion, so I would recommend to any admin to avoid its use.)  Since
UAs don't support Injection-Info initially, it should be human-parseable so
someone can dig through it to find the complaints contact information that
they used to easily find in an X- header, (in case they decide that emailing
[email-masked] and [email-masked] and abuse@tld for every path-identity
wasn't good enough.)

As for the other parts of Injection-Info, it is my understanding that the
sub-fields are going to be used as inputs to automatic spam detectors.

I'd say spam detection is hardly interoperability.  There are way too many
"non-spam" (I would not say "legitimate", since it mostly is copyrighted
content) multi-part posters on fast connections that exceed any
reasonable posts-per-unit-time threshold triggers.  So I doubt that it
gets used much.  When people were on dialup and not posting gigabytes of
CDs and DVDs, it might have made some sense.  That isn't now.

So, all in all, there are no interworking concerns on Injection-Info.
It might as well be unstructured.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Mon, 22 Aug 2005 06:00:42 -0700

> to appendix B (differences from 1036).

I started writing "current" rather than "2004-vintage", but then thought of 
how that statement will look to a reader in 2010.....

A pointer to appendix B is a good idea.

---

From: Henry Spencer <[email-masked]>
Date: Mon, 22 Aug 2005 12:04:52 -0400 (EDT)

On Mon, 22 Aug 2005, Frank Ellermann wrote:
> ...not me, I like your explicit statement about the design goals...

I go along with Frank on this one:  the wording may perhaps need some
touchup, but having an explicit statement about this is a good idea. 

                                                          Henry Spencer
                                                       [email-masked]

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 22 Aug 2005 14:28:01 GMT

>servers will pass most or all articles coming from the "new" UAs and 

Yes. We seek an orderly transition.

>section 1.2 (or a new section below it it) something like the following:

I think what needs to be said is broadly covered in section 3.2
(Transitional Arrangements) of USEPRO. I think we agreed to leave that in
USEPRO, but that is not to say that its wording might not be improved.

>articles that did not have them).

But there I disagree with you. I do not see a problem if much existing
software becomes technically non-compliant on the day our standard appears
- just provided that care is taken to ensure that the new standard will
still interwork with articles adhering to the previous practice (which we
have been careful to do). That, surely, is what is meant by "backwards
compatibility"?

Yes, I can see that some "MUST accept articles with missing mandatory
Injection-* header fields" wording should perhaps be written, but what I
do NOT want to say is that compliant servers can continue to omit such
headers indefinitely far into the future.

BTW, Injection-Info is currently only "SHOULD generate" as things stand,
and there is no proposal on the table to change that. Injection-Date is
"MUST generate" in USEPRO because there exist circumstances (possibly
rare) where omitting it can cause looping.

[Now you are going to ask why the present network does not exhibit that
problem. The answer is that the meaning of the Date header has changed -
according to RFC 2822, it is now the time of compositon of the article,
not the time of injection).]

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Henry Spencer <[email-masked]>
Date: Mon, 22 Aug 2005 15:16:02 -0400 (EDT)

On Mon, 22 Aug 2005, Charles Lindsey wrote:
> ...Yes, I can see that some "MUST accept articles with missing mandatory
> Injection-* header fields" wording should perhaps be written...

No "perhaps" about it, if those headers are officially mandatory.  Given
that existing software does not generate them, and will continue that way
for a while yet, interoperability absolutely requires either (a) making
them optional (whether or not they are strongly recommended), or (b) saying
*both* "MUST generate" and "MUST NOT demand". 

> ...Injection-Date is

It would seem reasonable to omit Injection-Date if composition time and
injection time are the same, which is sometimes exactly true and often
approximately true.  (The major case where it's not even approximately
true is gatewaying.)

                                                          Henry Spencer
                                                       [email-masked]

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Mon, 22 Aug 2005 11:42:13 -0700

> not the time of injection).]

I don't understand this.
By that definition, Date: would always be earlier than Injection-Date (if 
it  is not erroneous), and would cause the article to stop propagating 
earlier.

How can limiting an article's propagation cause loops?

(Not that I want to reopen the "MUST generate" debate. I just want to 
understnad.)

             Harald

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 23 Aug 2005 12:06:38 GMT

>unreliable information at the Reader end.

Yes, but there is only supposed to be one Injection-Info header in any
article, and that needs to be the latest one if re-injection has occurred.
And you carefully omitted to mention the advice given (right next to that
"MUST REMOVE") to rename the old header with something like
X-Injection-Info.

>sub-fields are going to be used as inputs to automatic spam detectors.

They can be used for whatever ad-hoc scripts people want to use them for.
For that reason, they need to have a reasonably well-defined syntax. But
mostly humans will just read them and extract whatever information they
need manually.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 23 Aug 2005 12:12:00 GMT

>approximately true.  (The major case where it's not even approximately

Yes we might have said that, but that would have meant defining some
official meaning for "approximately". And since we really want to
encourage Date as meaning "when I wrote this" (it is always useful to see
the time of composition to get the context of a message in a rapidly
evolving discussion), having a separate and visible Injection-Date header
will help towards that.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 23 Aug 2005 13:37:50 GMT

>*both* "MUST generate" and "MUST NOT demand". 

Yes I agree. There are a couple of places in USEPRO where I would have to
say that, and if Injection-Date is reverted to being mandatory I would
make the necessary change.

My concern is that we seem to agrree that this header MUST be inserted
(but not in proto articles of course) and yet we baulk at saying it is
mandatory.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 23 Aug 2005 13:27:20 GMT

>it  is not erroneous), and would cause the article to stop propagating 

Indeed, Date will always be earlier than Injection-Date. But it is the
Injection-Date (when present) that is used to stop propagation.

Consider a man who writes an article (Date: = day-1) on his laptop and
then carries it around for three days before finding an internet
connection where he can inject it. If the injecting site does not add an
Injection-Date: (= day-3), then subsequent relaying agents might reject it
as 'stale' (3 days is quite a common delay after which some relaying
agents erase all trace of an article, so they regard anything >3 days old
as stale). So his article fails to propagate. Which is why we want to
REQUIRE Injection-Date.

>How can limiting an article's propagation cause loops?

Actually, there can be no permanent loops (if the protocol for
Injection-Date is followed correctly), but there are circumstances where
an article might be accepted a second time by a server that has already
seen it once, even if Injection-Date is used. See
www.imc.org.ietf-usefor/2004/May/0092.html for a possible (if improbable)
example.

---

From: Henry Spencer <[email-masked]>
Date: Tue, 23 Aug 2005 11:09:04 -0400 (EDT)

On Tue, 23 Aug 2005, Charles Lindsey wrote:
> >approximately true...
> official meaning for "approximately"...

No, I don't think that's necessary.  My thought was that the spec would
simply say that Injection-Date can be omitted if its content would be
*identical* to that of Date.  (As it very likely would be, for articles
whose composition and injection happen on the same machine.)

A lazy implementer might decide that being within a few seconds is close
enough, but that's his decision, not ours.

                                                          Henry Spencer
                                                       [email-masked]

---

From: Henry Spencer <[email-masked]>
Date: Tue, 23 Aug 2005 11:14:55 -0400 (EDT)

On Tue, 23 Aug 2005, Charles Lindsey wrote:
> as stale). So his article fails to propagate...

Mind you, for the near future that must be assumed anyway, because old
servers will look only at Date for such decisions. 

                                                          Henry Spencer
                                                       [email-masked]

---

From: Henry Spencer <[email-masked]>
Date: Tue, 23 Aug 2005 11:19:33 -0400 (EDT)

On Tue, 23 Aug 2005, Charles Lindsey wrote:
> ...My concern is that we seem to agrree that this header MUST be inserted

Yes, that does seem inconsistent.

                                                          Henry Spencer
                                                       [email-masked]

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 23 Aug 2005 10:58:06 -0700

> Yes, that does seem inconsistent.

actually "MUST generate / MUST NOT require" is not much more inconsistent 
than "MUST accept / MUST NOT generate", which we have quite a few examples 
of.

                Harald

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 23 Aug 2005 23:07:38 -0700

--On 23. august 2005 13:27 +0000 Charles Lindsey <[email-masked]> 
wrote:

> as stale).

can someone who runs news services confirm this - that they not only erase 
articles from the spool after 3 days, but expunge the message-ID from the 
database of seen messages, and/or refuse to relay articles with a Date: 
header more than 3 days in the past?

It seems like a quite short window to me.... in a world of servers run on a 
more-or-less volunteer basis, I can easily see a news server go belly-up on 
Friday afternoon and only get back online mid-Monday, thereafter happily 
announcing to the world all the "new" articles that it got on Friday 
morning, 3 days before....

                       Harald

---

From: Russ Allbery <[email-masked]>
Date: Tue, 23 Aug 2005 23:58:48 -0700

Harald Tveit Alvestrand <[email-masked]> writes:
> Charles Lindsey <[email-masked]> wrote:

>> connection where he can inject it. If the injecting site does not add
>> an Injection-Date: (= day-3), then subsequent relaying agents might
>> reject it as 'stale' (3 days is quite a common delay after which some
>> relaying agents erase all trace of an article, so they regard anything
>> >3 days old as stale).

> can someone who runs news services confirm this - that they not only
> erase articles from the spool after 3 days, but expunge the message-ID
> from the database of seen messages, and/or refuse to relay articles with
> a Date: header more than 3 days in the past?

People have done it in the past, but it's not best practice and people who
use an expiration window that short tend to not get a lot of sympathy in
news.software.nntp.  INN has defaulted to 10 days for a long time, and if
I were writing best-practice documentation, I would recommend no less than
7 days.  I haven't done a comprehensive survey, but my guess is that most
people run INN with the default, and among those who change the default,
increasing it is more common than decreasing it.

One can certainly expire the articles off the server faster than that (one
of my transit servers expires articles in about a day), but history should
be retained for longer than that.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 24 Aug 2005 10:09:24 GMT

>servers will look only at Date for such decisions. 

Indeed. That is the main reason why Injection-Date was invented, and the
"MUST" is there to prevent the "harm" of losing the laptop user's
articles.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 24 Aug 2005 07:57:38 -0700

> increasing it is more common than decreasing it.

Thanks, Russ!

Seems that if we ever get around to USEAGE, we should recommend keeping 
history for at least 10 days... and correspondingly, if we mention this 
problem, we should use 10 days in the examples....

                      Harald

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 25 Aug 2005 10:34:53 GMT

>increasing it is more common than decreasing it.

This issue came up when we were inventing the Injection-Date header about
18 months ago. I recall pointing out that servers typically kept at least
the Message-IDs in their history files for 14 days (that being the default
in CNews, which I use). But I was told be someone (and I think it was you,
Russ) that modern transit servers worked to a figure more like three days,
and so that is the figure I have been quoting ever since. (I have not
delved into the archives to find the message, but I could do so if
necessary.)

But bear in mind that the man who keeps an article on his laptop for 3
days before injecting it is not quite so badly off as I may have implied.
His article may well be rejected by the fastest relayers on the net, but
the flooding algorithm may well find a route through smaller servers that
keep things for 10 days or so, and so most of the net will still get a
chance to see it, but the propagation will be slow.

---

From: Russ Allbery <[email-masked]>
Date: Thu, 25 Aug 2005 10:48:28 -0700

Charles Lindsey <[email-masked]> writes:

> This issue came up when we were inventing the Injection-Date header
> about 18 months ago. I recall pointing out that servers typically kept
> at least the Message-IDs in their history files for 14 days (that being
> the default in CNews, which I use). But I was told be someone (and I
> think it was you, Russ) that modern transit servers worked to a figure
> more like three days, and so that is the figure I have been quoting ever
> since. (I have not delved into the archives to find the message, but I
> could do so if necessary.)

I think you're half-remembering something I said and mixing up a few of
the details.  Let's try to put this issue firmly to bed.

In the original thread, you had made the statement that history entries
were kept for at least 14 days.  I explained that it was configurable and
that some servers kept history for a very short period of time, as low as
a couple of days.  In fact, here's what I actually said:

| I've run reader systems with considerably reduced history expiration
| times for articles that aren't in the spool for years, and my
| understanding is that this is not uncommon.  news.stanford.edu expires
| old history entries after two days, for example.
| 
| The only operational requirement is that one's history retention time
| matches one's cutoff limit for Date headers.  As long as that's true,
| people can and do run history with cutoff times as low as two days or
| lower.

Note that there are various special cases that one might want to tune for.
news.stanford.edu, as mentioned above, is a leaf system that is only fed
by my transit server.  My transit server has a retention of 7 days and
takes care of suppressing duplicates so that my reader server doesn't have
to.  In that situation, there's very little need for the records of
expired or rejected articles on my reader server and I retain only a
couple of days worth, just in case.

I'm not at all sure where the three day figure came from.  My guess is
that you got it from the following message from me:

| Charles Lindsey <[email-masked]> writes:
|
| > Anyway, the purpose of this discussion is to decide what number to
| > write in for how stale an article could be allowed to be at the
| > injector. John Stanley said 24 hours was too short. I suggested 72
| > hours as a possible answer. There has to be _some_ number there. I am
| > waiting for further suggestions or agreements.
| 
| How about we guarantee that 24 hours is safe (thus requiring sites to
| keep at least 24 hours of history, which I think is reasonable and proof
| against at least the forseeable future), and then explain why setting it
| to longer than that may cause articles not to propagate but leave it up
| to sites to make their own decisions?  We can't anticipate what this
| setting is going to be in another ten years.
| 
| Right now, my guess is that three days is almost certainly safe, five
| days is mostly okay but may cause problems, and ten days is going to
| almost definitely hit significant issues.
| 
| There is an interoperability issue here, so I agree that we have to say
| something, but there isn't actually a hard upper limit that one can
| justify.  Really, all the numbers we might come up with are magic and
| completely arbitrary.

Reading the above carefully, what I'm saying is that I don't believe there
are any significant transit servers with less than three day retention,
very few with less than five day retention, but quite a few with less than
ten day retention.  This is very different from saying that three days is
a "quite common" configuration for transit server history retention.

What I *am* saying is that it's very hard to anticipate how this
configuration may change in the future, particularly since the trend is
towards a more tightly connected Usenet with fewer propagation delays and
considerably more traffic.  CNews used 14 days, after all, which is
probably fairly rare now.  We therefore didn't want to write any tight
assumptions into the *standard* (but they're certainly fair game for best
practices, since it's more reasonable for best practices to change over
time).

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Thu, 25 Aug 2005 12:05:33 -0700

--On 23. august 2005 13:37 +0000 Charles Lindsey <[email-masked]> 
wrote:

>> them optional (whether or not they are strongly recommended), or (b)
>> saying *both* "MUST generate" and "MUST NOT demand".
>

I balk at it because:

1) we MUST handle messages from old servers
2) we DO NOT specify handling of messages that do not conform to the syntax

So if it's mandatory, we have said that we do not specify handling of ANY 
old messages. And I think that is a mistake.

                             Harald

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 25 Aug 2005 21:31:56 +0200

> So if it's mandatory, we have said that we do not specify
> handling of ANY old messages. And I think that is a mistake.

Oops, point taken.  RECOMMENDED is better.  Bye, Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 26 Aug 2005 14:16:45 GMT

>old messages. And I think that is a mistake.

Right! Now I see where you are coming from. In that case, I suggest a NOTE
in USEFOR under the Injection-Date something like the following:

     NOTE: [USEPRO] REQUIRES this header field to be inserted whenever as
     article in injected into the network. However, since it is newly
     introduced in this standard, agents MUST continue to accept articles
     generated by earlier systems which do not include it. For this
     reason, it has not formally been designated as "mandatory".

BTW, I am on holiday all next week.

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 26 Aug 2005 14:08:56 GMT

>couple of days worth, just in case.

Yes, that is the article I remembered, and I was assuming that the
"news.stanford.edu" that you mentioned was engaged mainly in relaying
(since 2 days expiry seemed extraordinarily short for a serving agent).

So, let me see if I understand:

The laptop man writes his article on day-1, and injects it on day-3, at a
site which does not yet do "Injection-Date". It arrives at your transit
server well within the 7 days, so it is accepted (it might well expire off
there a day later, but the history file - or something - will still remember
its message-id for 7 days in case it belatedly comes around again).

In the meantime, it is passed to your leaf system which does not perform
any staleness check of its own (which might have failed, given that
ancient Date header), because it has been instructed to accept everything
sent from your transit server without question.

>that you got it from the following message from me:

No, it was probably a mis-remembering of the "two days".

>| > waiting for further suggestions or agreements.

In USEPRO, the only mention of 24 hours is how far into the future
articles are allowed to be before you reject them as bogus (one needs some
allowance, since clocks on the various agents can be in error - that 24
hours is an upper limit).

It also mentions 72 hours (into the past) as a lower limit at which you
MAY reject articles when _reinjecting_ and with no Injection-Date to help
you.

There is no figure given for staleness in other circumstances, except that
it MUST be no later than the amount of history you keep (e.g. 7 days for
your transit server). We probably ought to put a figure for this in
USEAGE, and it looks like 7 or 10 days would be about right.

---

From: Russ Allbery <[email-masked]>
Date: Fri, 26 Aug 2005 09:48:59 -0700

> (since 2 days expiry seemed extraordinarily short for a serving agent).

Ah.  Yes, the key is that news.stanford.edu is only ever fed articles in
groups that it carries, so the memory of old articles isn't particularly
interesting.  I made that change years ago when I was having disk space
trouble with the partition containing the history file.

> The laptop man writes his article on day-1, and injects it on day-3, at
> a site which does not yet do "Injection-Date". It arrives at your
> transit server well within the 7 days, so it is accepted (it might well
> expire off there a day later, but the history file - or something - will
> still remember its message-id for 7 days in case it belatedly comes
> around again).

> ancient Date header), because it has been instructed to accept
> everything sent from your transit server without question.

Right.

This is definitely not a best practices configuration; it's a "I know what
I'm doing" configuration.  It's also not a very good example in retrospect
(critiquing myself of a year and a half ago), since it violates the
equality between history retention and old Date rejection, which probably
should be a MUST.  (I can prove that it doesn't matter in this case, but
this is not something I would recommend other people try at home.)

A better example would be a site with multiple transit servers, one with a
ton of peers and a lot of load and extremely restrictive rules about Date
headers that's designed to carry the majority of traffic as fast as
possible, and a second transit server with fewer peers and a much longer
history retention that serves as a backup to catch slow articles.  People
do things like this from time to time.

But, for the normal case, I would recommend either 7 or 10 days as good
time intervals to keep previously rejected or expired articles and to use
for Date staleness checks.

> There is no figure given for staleness in other circumstances, except
> that it MUST be no later than the amount of history you keep (e.g. 7
> days for your transit server). We probably ought to put a figure for
> this in USEAGE, and it looks like 7 or 10 days would be about right.

Yup.