Subject: Re: Message IDs and moderators

From: [email-masked]
Date: Tue, 19 Dec 2006 11:41:37 -0800 (PST)

Russ Allbery <rra@xxxxxxxxxxxx>:

>If we go with the definitions in my draft, a proto-article is not an
>article.  There is an intentionally sharp distinction between a
>proto-article and an article.  A proto-article only becomes an article
>once it has been injected.

I would say that any group of bytes that is being produced with the intent 
of becoming a news article, and which follows the standards for formatting 
and definitions, should be subject to those standards.

Under that "philosophy", even a proto-article's header fields have the 
meanings that RFC1036 and eventually our standards define for them. That 
includes From and Message-ID fields.

But where I'm taking this is not specifically involving the idea that the 
message id, once applied, cannot be changed by a moderator. It will 
eventually be a discussion of the "moderators may edit articles in any way 
they see fit" concept -- once we've settled on which draft we use.

>I think RISKS Digest is an excellent example of a corner case.

We agree to disagree. comp.risks is a moderated newsgroup, yes. That's 
where the relevance ends. Each "article" that appears in the newsgroup is 
posted by the editor, not individuals, and are posted under the editor's 
name. That editor uses a system of attributing material to the authors he 
quotes in his digest, but at no time does he claim that the digest itself 
was originated by anyone other than himself.

Were Peter to start posting individual articles under other people's names 
to comp.risks, with the same kinds of editing he does in the digest, then 
it would be a corner case. It would then be open to debate whether his 
editing is acceptable for a moderated group and having the original 
author's name applied to a different message than he sent.

Perhaps a better "corner case" was comp.dcom.telecom. Enough people 
objected to the editing by the moderator of that group that they started 
another one -- comp.dcom.telecom.tech. So, we have at least one precedent 
where moderator editing was considered unacceptable by enough people to 
form a new group.

>We need to
>make sure that our standard is not incompatible with what people do in
>practice, including the corner cases, where those cases are accepted.

Yes, and that is why I expect the attempt to get the codified right to 
edit articles in any way the moderator wishes removed will be a losing 
battle. It would be impossible to prevent, it isn't a matter for the 
protocol, and many people don't care. I cannot accept it on ethical 
grounds, however, and that's why I have to object to it being made
explicit.

Even if you say "moderators may edit according to group standards", since 
the moderator is free to set the standards, you're saying the same thing.

As a side issue -- how does one differentiate between a proto-article that
has all the mandatory headers and a real article? What should be the
result of differentiating between them? I.e., if you have a proto-article
that contains all the mandatory headers, how do you treat it differently
than a real article?

Yes, an injecting agent is supposed to throw away certain things, but
that's a specific case based on an action by a user. In general, what's 
the difference?

---

From: Russ Allbery <[email-masked]>
Date: Tue, 19 Dec 2006 12:10:49 -0800

> I would say that any group of bytes that is being produced with the
> intent of becoming a news article, and which follows the standards for
> formatting and definitions, should be subject to those standards.

I do agree with that.

[ snipping things that I don't disagree with ]

> Were Peter to start posting individual articles under other people's
> names to comp.risks, with the same kinds of editing he does in the
> digest, then it would be a corner case. It would then be open to debate
> whether his editing is acceptable for a moderated group and having the
> original author's name applied to a different message than he sent.

> another one -- comp.dcom.telecom.tech. So, we have at least one
> precedent where moderator editing was considered unacceptable by enough
> people to form a new group.

Good point.  comp.dcom.telecom is a better example.

In that case, we had some people who were comfortable with the editorial
policies of the moderator and some who weren't, and as a result the group
split.  I think that's a good encapsulation of the situation, actually.
Extensive moderator editing is generally frowned upon and it makes a lot
of people unhappy when done on Usenet, but it's also sometimes done
successfully and sometimes particular groups are comfortable with it.
It's that dynamic that, to me, moves it from standard to best practice.

>> We need to make sure that our standard is not incompatible with what
>> people do in practice, including the corner cases, where those cases
>> are accepted.

> grounds, however, and that's why I have to object to it being made

Okay, that makes sense to me.  I'm also happy to try to reword what the
protocol document says so that it specifies that it doesn't constrain
rather than giving explicit blessing to behavior that is controversial.
That can be tricky to do and still be clear, but the purpose, as far as
I'm concerned, is to draw a clear boundary around the protocol and say
that beyond that boundary the protocol doesn't place requirements, not to
say that the protocol is implicitly blessing some particular course of
action.

My currently language doesn't do this anywhere near as well as it could.

> Even if you say "moderators may edit according to group standards",
> since the moderator is free to set the standards, you're saying the same
> thing.

Agreed.

> As a side issue -- how does one differentiate between a proto-article
> that has all the mandatory headers and a real article? What should be
> the result of differentiating between them? I.e., if you have a
> proto-article that contains all the mandatory headers, how do you treat
> it differently than a real article?

My document defines it as follows:

   A proto-article has the same format as a normal article except that
   the Injection-Date, Injection-Info, and Xref header fields MUST NOT
   be present; the Path header field MUST NOT contain a "POSTED" <diag-
   keyword>; and any of the following mandatory header fields MAY be
   omitted: Message-ID, Date, and Path.

Since if one is following the new protocol an injecting agent MUST add an
Injection-Date header, that's the clearest point of distinction.  If it
has an Injection-Date header, it's an article, not a proto-article.

However, since Injection-Date is a new invention of this document, there
is no clear way of determining that a message is a proto-article when you
take into account existing practice.  There are various markers that tell
you that it's an article (presence of any of those three headers or the
POSTED Path diagnostic), and of course if it's missing one of those
mandatory headers it's a proto-article, but there are messages that will
be ambiguous.

Probably the best way of determining whether something is an article or
proto-article currently is to look at whether it has an Xref header field.
That header field is optional, but nearly all servers add it and posting
agents do not, so it's a clear sign that the article has been injected.

Note that inclusion of Xref in this set is new in my document and is a
change from the existing USEPRO draft (but reflects current practice in,
for example, INN, which will reject attempts to post messages with Xref
headers already present).  This is precisely one of the reasons why I
added it to this definition.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 21 Dec 2006 19:47:43 GMT

>say that the protocol is implicitly blessing some particular course of

Yes, but the trouble is that "clear boundaries" do not exist in the Real
World (TM). Whatever we say, moderators will continue to do whatever they
do, and offended readers will send flames and LARTs to ISPs, the B8MB,
their lawyers, NANU, and anywhere else they can think of. And eventually
the fuss will die down and peace will resume. That is the way that Usenet
works. But what we don't want is for any of the parties in the flame war
to be able to point to the standard and say "look, it says that moderators
can make any alterations they choose".

The fact is that Usenet has 'customs', 'conventions' and 'rules'
established by consensus amongst its users and/or the servers that carry
it, and you cannot describe how it works without at least admitting that
they exist (though just how they are established is another matter).
USEPRO at least admitted that they existed, and hinted at various places
that they needed to be followed. Maybe USEPRO said too much, but RUSSPRO
has gone to the opposite extreme, and that does not work either.

Essentially, the customary expectation is that moderators may fiddle with
"boilerplate" but no more, but how can you say that in precise terms? You
can't, but you can at least point to "established custom", in a NOTE if
needs be. The same is true for group creation and other such things.
RUSSPRO uses the word "authorization" in various places, and such a term
is definitely needed. But nowhere does it define that term, nor say who
authorizes the authorized. USEPRO at least admitted that the problem
existed, and carefully said that it did not attempt to provide an explicit
answer, though it did say that the answer might vary from hierarchy to
hierarchy.

>   omitted: Message-ID, Date, and Path.

I think it is not so much a matter of looking at the article as asking
where it was located when you looked at it. If it was still on your
machine waiting to go to the injector, or if it is in some moderator's
input queue, then it is a proto-article. If it is already propagating
around Usenet, then it an article.

But then there are still some corner cases. mostly tied up with
reinjection (which I have written at length about in another thread, and
which I am still hoping for a reply to). But here is an example of a corner
case.

Some guy runs a news server on his home machine (quite a lot of people do
that, because they find it more convenient that fiddling with one of the
big newsreaders such as Turnpike or Agent).

So he posts articles which appear at first in his local serving agent, and
then get sent to his ISP's news spool. His ISP will expect him to use
POST, and will add an Injection-Info, with complaints directed to the
abuse dept. OK so far, and the guy will have to arrange things are done as
his ISP requests. And, Shock Horror! maybe it even has an Xref when the
ISP gets it.

But now his neighbour hears that he has some newsgroups (mayble even some
'local' groups for his neighbourhood) and says "Hey! can you let me have a
feed of those, and BTW can you feed me comp.widgets at the same time as I
am interested in those".

Now what does he do? He really ought to put an Injection-Date on what he
sends to his neighbour, because he cannot be sure whether his neighbour
has other connections (or more likely might establish other connections in
the future without telling him), and the articles he supplies might leak
out onto Usenet that way. So they had better be clean decent articles,
traceable back to him.

So now does he arrange to send different versions of an article to his
neighbour and to his ISP, one already injected and the other not? Or does
he send the same to both? Ah! Now we are into reinjection! Now is it a
proto-article or a real article. Who knows? More importantly, who cares?

Maybe his ISP absolutely forbids it. Fair enough, but if the ISP decides
to reinject it, why not? He just replaces the Injection-Info, if present,
with his own. BUT any Injection-Date MUST then be left as it is.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Fri, 22 Dec 2006 07:45:43 +0100

> can make any alterations they choose".

Poppycock. We do that all the time (define clear boundaries).

And we have tended to put LESS requirements on operators' behaviour as time 
goes by, exactly because our well-meant requirements, stated in an earlier 
time, was used as a LART against people in inappropriate ways.

That's why whe had to revise WHOIS to make it clear that there's no 
PROTOCOL requirement to put the telephone number into a WHOIS record, for 
instance. The "rfc-ignorant" folks were LARTing.

(in case some people haven't seen this term recently: I think LART = Loser 
Attitude Readjustment Tool, also called a "clue-by-4"....)

> has gone to the opposite extreme, and that does not work either.

In the cases cited so far, it works for me.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Fri, 22 Dec 2006 07:48:08 +0100

--On torsdag, desember 21, 2006 19:47:43 +0000 Charles Lindsey 
<[email-masked]> wrote:

> which I am still hoping for a reply to). But here is an example of a
> corner case.
>

can someone who actually runs one of those beasts explain how they handle 
posting of articles - in particular, whether they really make an article 
available to their local serving agent before sending it to the ISP's news 
server?

                      Harald

---

From: Russ Allbery <[email-masked]>
Date: Thu, 21 Dec 2006 23:19:49 -0800

Harald Tveit Alvestrand <[email-masked]> writes:

> can someone who actually runs one of those beasts explain how they
> handle posting of articles - in particular, whether they really make an
> article available to their local serving agent before sending it to the
> ISP's news server?

In actual practice, they work the way that I describe them in my document,
namely as two independent networks and a gateway with all gatewaying work
done by the stand-alone server.

Take, for instance, the popular news package "suck."  It pulls down
messages from a remote system using reader commands and then injects them
into a local news server using IHAVE (a relaying command).  Then, when
someone posts locally to the server, that posting is accepted by the local
server, immediately made available to the local server's readers, and
queued in what, to the server, looks like a regular outgoing feed, a text
file that contains a queue of message IDs to send to a remote server.
However, rather than running a typical relayer command to do that sending,
one runs rpost instead.

rpost takes the message from the local system, converts it back into a
proto-article by stripping or renaming NNTP-Posting-Host, X-Trace, and
other unstandardized trace headers plus headers such as Xref, and then
offers it to the remote injection agent as a new article.

In practice, this works fine.

Currently, the Date header is conventionally preserved through this
transaction, which helps protect against a badly misconfigured rpost
client posting old messages.  The one place where my document would change
this, which is a side effect of the introduction of Injection-Date, is
that rpost would gain an additional requirement of doing date checks
itself to be sure that it wasn't given a bad batch of ancient articles
that shouldn't be gatewayed.  Currently, injecting agents are still also
required to check the Date header.  Should that requirement ever be
dropped, the onus on preventing regurgitation of ancient articles would be
entirely on rpost.

This has advantages and drawbacks.  In some respects, this scheme is a
serious drawback to introducing Injection-Date; in the process of solving
a not horribly significant problem with delayed postings from off-line
readers that can be worked around in other ways, we're messing with one of
the most basic parts of loop detection and introducing other complexities.
However, this scenario also gains from the exact same enhancements as any
other off-line post does: the actual posting date of the message can be
retained even if the stand-alone server from which the posts are being
gatewayed is off-line for several days.  Most importantly, it puts the
onus of the work on the agent that's introducing the risk and that is in
the best position to know the correct action to take, rather than making
every injection agent deal with this unusual case.

What's in USEPRO contradicts existing practice.  What's in my document is
a translation of existing practice that introduces Injection-Date.  My
*preferred* solution after trying to sort out this whole area would be to
get rid of Injection-Date as a nice idea that's too late to introduce, at
which point we're back to what we have right now and what we know works,
and which we therefore can make some guarantees about safety, but that
requires revisiting USEFOR and it's not the time to do that.  Failing
that, modelling this as a gateway is substantially cleaner than trying to
standardize reinjection and simplifies the picture considerably.  The cost
is one missed opportunity to catch a regurgitation of ancient articles at
the injection agent level (and, in practice, not a missed opportunity in
*this* standard since my document still requires that injection agents
check the Date header, but a potential missed opportunity later should we
ever drop that requirement).  I think from a protocol design perspective,
that tradeoff is worth it.

The closer the protocol design is to what is being done today and is
working today, the more comfortable I am that it doesn't have hidden
gotchas that we're going to regret later.

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 22 Dec 2006 14:00:16 +0100

Harald Tveit Alvestrand wrote:

Some potential cases of TIN[WU] (= "There Is No We or Us"):

>> But what we don't want is for any of the parties in the flame war
>> to be able to point to the standard and say "look, it says that
>> moderators can make any alterations they choose".

> And we have tended to put LESS requirements on operators' behaviour
> as time goes by, exactly because our well-meant requirements,
> stated in an earlier time, was used as a LART against people in
> inappropriate ways.

> That's why whe had to revise WHOIS to make it clear that there's
> no PROTOCOL requirement to put the telephone number into a WHOIS
> record, for instance. The "rfc-ignorant" folks were LARTing.

They still do this for _wrong_ phone numbers, same idea as in the
WDPRS (ICANN's whois data problem report system).  Killing RFC 954
didn't help "you", I've found RFC 1032 to replace 954 by something
"better" than 3912.  Of course RFC 1591 would be the best, but it's
"only" informational.  John said that's bogus, and RFC 1591 should
be a BCP.  With that I think that I could convince the RFCI folks
to replace RFC 1032 by 1591.  But Brian didn't like John's proposal
to fix the category of 1591, so that's at a stalemate for now.

>> Maybe USEPRO said too much, but RUSSPRO has gone to the opposite
>> extreme, and that does not work either.

> In the cases cited so far, it works for me.

Not for me, like RFC 3912 was a case of "Do Not Publish".  At that
time I still thought that the RFCI folks would know how to stop it
in its "Last Call".  It turned out that they didn't know it... :-(

Frank

---

From: Dan Schlitt <[email-masked]>
Date: Fri, 22 Dec 2006 11:39:26 -0500

Customs, traditions, etc. are not protocol issues. They are useage
issues if anything at all is to be said about them.

Defining the boundry as Russ has done seems quite reasonable.

/dan

-- 

Dan Schlitt
[email-masked]

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 22 Dec 2006 15:34:33 GMT

>available to their local serving agent before sending it to the ISP's news 

Yes, my own system runs that way (I run a CNews server, as I am sure you
all know). I download mostly from NIN (news.individual.net), but for some
obscure groups from other servers, and I gateway in this list and some
others.

Anything I post from my newsreader gets injected into my local server
(but, since Injection-Info, Injection-Date and .POSTED are not yet
implemented, the only evidence inside an article is the presence of an
Xref header and a short Path).

For sending stuff out to the wide world, I again use various servers
according to the group. CNews is configurable to use any script you care
to provide for each outgoing feed. So I can arrange for each feed whether
it looks like a relay (using IHAVE) or like an injection (using POST, or
the NNRP "IHAVE" which is really injection) or some mail2news, or a
gateway to a list such as this.

So, for most normal groups I relay to Manchester University and, 30
seconds later, inject to NIN. The University facility is useful because it
is guaranteed not to munge anything that might break a PGPVERIFY, but its
administration is a complete shambles (it has been totally down for the
last couple of weeks). OTOH NIN is extremely reliable, but fussy about
what it accepts (e.g. it insists that all From mail addresses are genuine
and checks that an MX record exists). So when wearing my part-time
uk.net.news.announce moderator's hat and having to post a Call For Votes
from a votetaker who insists on munging his From address to defeat spam
harvesters, I have to rely on the Manchester server to get it out. All
that is, of course, quite untypical of the average Usenet user, but
illustrates some of the stranger corner cases that can arise.

So, technically speaking, when an article is injected at NIN, it is
"reinjection". So when USEFOR/USEPRO are in force I shall have to consider
what Injection-Info etc headers to generate, and whether to include them
in all my outgoing feeds or not, and that will depend I suppose on what
NIN are prepared to accept.

It so happens that I never send out an article that was not generated
locally, otherwise I would need to be much more careful.

---

From: Russ Allbery <[email-masked]>
Date: Fri, 22 Dec 2006 09:36:12 -0800

Russ Allbery <[email-masked]> writes:

> In actual practice, they work the way that I describe them in my
> document, namely as two independent networks and a gateway with all
> gatewaying work done by the stand-alone server.

Someone pointed out to me in private e-mail that there's another
frequently used way of doing this which avoids most of the problems
(including the need to do gatewaying), namely immediately proxying any
POST to a remote server.  Some of the end-point single-host news servers
just keep track of where different groups go and, when receiving a POST,
never store the article locally and instead immediately proxy the POST to
the appropriate server.  Then, when the article shows up on that server,
the normal mechanism pulls it down to the local article store.

This system is much more robust than having to do gatewaying in any
fashion, but has the drawback that it can't function off-line and that it
requires a specialized news server that knows to do unusual things with
POST (as opposed to allowing one to plug a gateway into a generic
all-purpose news server).  If I remember correctly, leafnode works this
way.

This is really the best way to handle this, and avoids having to deal with
the issues in either my draft or in the current USEPRO draft.  There's no
reinjection or gatewaying, just the proxying of the conversation between
the posting agent and the injecting agent.  It just has a few drawbacks
that makes something like INN + suck better in a few situations, mostly
ones where you want to maintain a set of local newsgroups in addition to
gating newsgroups from another Netnews network.