Subject: Re: draft-ietf-usefor-usepro-01

From: [email-masked] (Kai Henningsen)
Date: 02 Oct 2004 12:05:00 +0200

[email-masked] (Russ Allbery)  wrote on 29.09.04 in <[email-masked]>:

> Control messages should not, in general, be crossposted to other
> newsgroups than they one that they affect.

That never seemed right to me. Surely group-list changing ones should be  
crossposted to a designated control message group for the hierarchy in  
question.

>David Lawrence was the one who
> first taught me the problems caused by doing so.  For example, when people
> form a habit of crossposting control messages to alt.config, someone
> limiting what hierarchies in alt.* they receive, they still ends up
> getting control messages for the hierarchies they don't want.

Frankly, I have trouble coming up with a reason why that's bad. From where  
I sit, that's an intended result.

MfG Kai

---

From: Russ Allbery <[email-masked]>
Date: Sat, 02 Oct 2004 12:26:07 -0700

Kai Henningsen <[email-masked]> writes:
> [email-masked] (Russ Allbery) wrote:

> crossposted to a designated control message group for the hierarchy in

Why?  What purpose does this serve?

>> David Lawrence was the one who first taught me the problems caused by
>> doing so.  For example, when people form a habit of crossposting
>> control messages to alt.config, someone limiting what hierarchies in
>> alt.* they receive, they still ends up getting control messages for the
>> hierarchies they don't want.

> Frankly, I have trouble coming up with a reason why that's bad. From
> where I sit, that's an intended result.

Why?  If I have all of my peers sending me alt.*,@alt.binaries.*, I don't
want to get control messages for alt.binaries.* either.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 6 Oct 2004 22:20:41 GMT

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>Charles Lindsey <[email-masked]> writes:

>> We did discuss the text that says control messages need to have a
>> Newsgroups-header that will steer them to the proper places, and the
>> text you see contains some tweaks arising from that.

>newsgroups than they one that they affect.  David Lawrence was the one who

Indeed so, which is why our draft says you SHOULD include the affected
group in the Newsgroups-header, and you MAY include others, but it then
points out that this may cause other problems. However, it may sometimes
be useful, and others have said as much in response to your message.

>However, relay agents that check the Newsgroups header against a list of
>valid groups *do* need to propagate control messages as if the group they
>were affecting existed.  *That* behavior does need to be specified in the
>standard.  This is independent of the question of whether one should
>ignore the Newsgroups header on newgroup and rmgroup control messages
>entirely.

But if control messages are to be propagated regardless of what the
Newsgroups-header says, then how is it that I do not see newgroup messages
for jp.* and other hierarchies to which I do not subscribe? For sure
someone upstream of me is filtering them out, so I can only suppose that
some 'sys' files are being consulted somewhere in the process, but where?

>  I'm surprised that it's not in son-of-1036 and doesn't appear
>to be in CNews, but perhaps both predate the understanding that this was
>the right thing to do.  This has been implemented in INN since the 1.0
>release in 1991.

>If relay agents don't implement this behavior, booster rmgroups are
>futile because they won't propagate past the first host that already
>removed the group and newgroups won't propagate through relay agents that
>don't act on them immediately and require that newsgroups exist before
>propagating messages.  This is clearly wrong.

Sure, but different relay agents attack this problem in all sorts of
different ways.

For example, in CNews, it checks for the presence of a Control-header at
an early stage, and immediately decides to file it in the 'control'
newsgroup (which is always in the active file). So any control message
received (i.e. which the upstream peer was somehow configured to send to
it) will get stored regardless.

Then, whenever a new article is stored it consults its 'sys' file to see
which of its downstream peers it should send it to, and that of course is
determined by what is in the article's Newsgroups-header. Note that
'sys' files are usually broadly written, disposing of complete
(sub-)hierarchies using wildmat notation or similar.

So, back when Tale first created 'humanities.misc', I think you will find
that his Newsgroups-header will have contained at least token groups from
sci, comp, news, talk, etc. Otherwise, it would never have propagated very
far.

>Well, I don't believe that's correct, but there's no point in arguing
>about past history.  I'm making that statement now.  This should be part
>of the standard.

I think it is clear that we have to put _something_ in the standard about
this, but I think we need to understand the range of practices that
current relaying agents actually DO before writing any text.

>>    Even if relaying agents do not, as a matter of local policy, intend
>>    to honour certain control messages, they MUST still propagate them in
>>    the normal manner.
>> [The inclusion of that paragraph still remains under discussion.]

>This is still not correct.  This doesn't help with the issue addressed
>above in propagation of newgroup and rmgroup control messages and simply
>contradicts other portions of the document in a fashion that leaves one
>wondering what it's supposed to mean.

Sure, but it's not supposed to be anything to do with the issues discussed
above. It is concerned with agents that try to say "I am not going to
create this group, so I am not going to tell you about it".

>  Please, remove this language and
>just explain that newgroup and rmgroup control messages should always be
>propagated as if the group they're affecting exists, even if it does not.

I could easily be persuaded that the wording should come out, but at least
let us remove it for the right reasons (e.g. that no current agent is so
inconsiderate and mean as to do such a thing).

>> Ah! I had always understood, since the matter was first raised on the
>> news.admin.net-abuse groups, that it came out of the normal processing
>> of the Path header rather than on special code in transport agents.

>Nope.  It was possible because of a pre-existing feature in INN (the path
>aliasing concept predates pseudo-sites by quite some time), but path
>aliasing is not really part of the normal processing of the Path header.

OK. Thanks for that, though it implies that the trick would never have
worked on CNews.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Russ Allbery <[email-masked]>
Date: Wed, 06 Oct 2004 19:28:16 -0700

Charles Lindsey <[email-masked]> writes:
> Russ Allbery <[email-masked]> writes:

>> However, relay agents that check the Newsgroups header against a list
>> of valid groups *do* need to propagate control messages as if the group
>> they were affecting existed.  *That* behavior does need to be specified
>> in the standard.  This is independent of the question of whether one
>> should ignore the Newsgroups header on newgroup and rmgroup control
>> messages entirely.

> Newsgroups-header says, then how is it that I do not see newgroup
> messages for jp.* and other hierarchies to which I do not subscribe?

I am completely baffled by this reply.  You seem to have not read what I
have said.

You don't get newgroup messages for jp.* because you are not being fed the
jp.* hierarchy, and the only mention of "control messages are to be
propagated regardless of what the Newsgroups-header says" is the text
that's currently in USEPRO that I'm trying to get you to take *out*.

Control messages MUST be propagated as if the newsgroup that they affect
existed, even if it doesn't.  It is occasionally additionally a good idea
to behave as if the Newsgroups header contains only the newsgroup being
affected, rather than whatever other random junk the poster has chosen to
put in there, but that's a different topic.  The important part is that if
I post a newgroup message for news.misc, that newgroup message should
propagate to any of my peers that would receive news.misc given their
subscription patterns, even if that group doesn't actually exist.

This is significant since normally, for speed, implementations like INN
turn feed subscriptions into a hash table keyed on newsgroup name and
listing all the sites that want to retrieve that newsgroup.  This is an
effective technique for nearly all traffic if you're maintaining an active
file, but control messages cannot be handled this way, and INN pulls them
out of normal feed processing and performs a special match of the affected
group against the raw feed patterns instead.

> So, back when Tale first created 'humanities.misc', I think you will
> find that his Newsgroups-header will have contained at least token
> groups from sci, comp, news, talk, etc. Otherwise, it would never have
> propagated very far.

How much money would you like to bet me?

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 7 Oct 2004 14:26:57 GMT

>subscription patterns, even if that group doesn't actually exist.

Right! That last sentence is the bit that was missing from your previous
explanations of what INN actually did. There clearly had to be a 'sys'
file (however called) somewhere in the chain that controlled what actually
went to your downstream peers.

>group against the raw feed patterns instead.

OK, it is useful to know that, but I can still see variants on that
technique that would work equally well.

Anyway, for the final question, what about transit-only systems that keep
no active file, nor any permenent store of articles? Do they need to take
any special note of Control-headers, or do they just propagate them like
ordinary articles, as determined by the Newsgroups-headers plus their
peers' subscription information?

---

From: Russ Allbery <[email-masked]>
Date: Thu, 07 Oct 2004 13:50:52 -0700

Charles Lindsey <[email-masked]> writes:

> Anyway, for the final question, what about transit-only systems that
> keep no active file, nor any permenent store of articles? Do they need
> to take any special note of Control-headers, or do they just propagate
> them like ordinary articles, as determined by the Newsgroups-headers
> plus their peers' subscription information?

I think either approach works.

---

From: Bruce Lilly <[email-masked]>
Date: Sun, 17 Oct 2004 11:49:04 -0400

[Incidentally, the original was apparently sent only to the old mailing list
rather than the "new" (as of a couple of months ago) mailing list]

On Sat October 9 2004 14:36, you wrote:
> > That sounds to me like a very special case. It seems to me it's much
> > more likely that you have your peers send you, say, a certain subset of
> > the Big8, and you want to get all the Big8 control messages, not just
> > that subset, so you can decide what other groups you might be interested
> > in.
> 
> Note that this has never been possible for the Big Eight, and I've never
> heard a complaint about it.  You're the first person who I recall
> indicating that you'd prefer things be done that way, rather than only
> sending control messages according to normal newsgroup subscription
> patterns.
[...]
> (This is probably more detailed than we really need to cover in our
> documents, but it is a good argument in favor of not banning crossposting
> control messages.)

We probably need to cover more detail; let's start with examining
the pros and cons for the different types of control messages:

So far you folks have been discussing newgroup messages, but there
are other types.  The considerations vary based on the control
message type.

Cancels could safely be limited to the newsgroup subscription patterns
*assuming* that newsgroups are specified exactly the same on the
cancel as in the original, *and* further assuming that there have
been no changes to newsgroup hierarchies in the time interval
between posting of the original and cancel messages, *and*
still further assuming that no site has changed newsgroup feed
patterns in such a way (e.g. dropping newsgroups) that would
prevent a cancel message from reaching a system that had
received the original.  I suspect that in a small but significant
portion of cases, one or more of those assumptions would fail
to conform to reality.  I believe that the specific mechanism
specified in RFC 1036 is reasonable; viz. propagate widely
initially, stopping propagation to downstream sites unless the
original message had been received.

Rmgroup messages, the newfangled mvgroup messages, newgroup
and checkgroups messages should probably propagate everywhere.
Otherwise a site that gets its initial list of active newsgroups e.g.
via ftp from some source would not be kept up to date w.r.t.
changes in hierarchies (additions, renaming, removals, changes in
suggested moderation status) if upstream sites fail to propagate
relevant control messages when newsgroups directly affected
are not fed.  That in turn could affect downstrean sites that
subsequently get the out-of-date active newsgroup list from the
site in question, administrators and users of that site may be
unaware of changes to newsgroups of potential interest (e.g. for
finding an alternate upstream feed for new groups of interest),
and sites with out-of-date lists may unintentionally become
sources of problems (e.g. crossposts to "old" newsgroups that
have been removed or renamed).

Ihave and sendme control messages should only propagate one
hop regardless of newsgroup subscriptions.  Generally, pseudo-
newsgroups are set up by administrators to implement that, but
the rule should be specified; implementation details are just
that -- implementation details.

Sendsys control messages are intended to gather statistics
about connectivity and newsgroup distribution.  If limited to
newsgroup hierarchies, then gathered connectivity statistics
will obviously only represent a subset of sites to which the
control message propagates. That is probably A Bad Thing for
connectivity (as distinct from newsgroup propagation)
statistics. Causing those control messages to propagate widely
in order to gather more complete statistics requires crossposting
to a huge number of newsgroups, which would likely cause
some breakage (I suspect that some software would be unable
to handle a Newsgroups field listing 30,000 newsgrousp).
On the other hand, collecting newsgroup propagation
statistics could be achieved even if propagation of the control
message is limited; conversely if it is not limited the only
downside is the volume of the responses sent to the requestor.
I believe sendsys control messages should therefore
propagate everywhere, unless superseded by two distinct
types of control messages for the two specific purposes (viz.
connectivity in general, and specific newsgroup hierarchy
propagation).

Version control messages are intended to gather statistics on
transport/spooling software, and is likely only to be of use to
a requestor if reasonably complete (e.g. to definitively answer
the question "is anybody still running B news 2.11?").  It
should therefore propagate widely (i.e. independently of
newsgroups).

Other considerations:

Since different types of control messages are handled differently
and affect different agents in different ways, we should probably
consider whether a long term split of the catch-all Control field
would be worth planning for (i.e. while keeping the catch-all
Control field name tag with a separate "verb" in the field for
the moment, advising implementors that at some future date
there may well be separate fields to facilitate the different
types of propagation and the different handling by different
agents).