Subject: Re: More Path notes (was: Protocol changes in draft-allbery-usefor-usepro-00)

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 2 Jan 2007 16:29:54 GMT

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>Charles Lindsey <[email-masked]> writes:
>> Russ Allbery <[email-masked]> writes:
>>> Charles Lindsey <[email-masked]> writes:

>>>> 1. [-1] (2.2) <path-identity> - Possibility to use a FQDN not
>>>> resolvable in the DNS.

>>>> (This was a suggestion by Harald, & 1 of 2 alternatives in USEPRO)

OK, I responded to this in an earlier thread.

>>>> 2. [00] (3.2) <path-identity>s are case-sensitive.

>> OK, I will buy that.

And now you aren't selling it :-( .

>The more I research this, though, the more I think that while the above is
>true, it's not ideal.  Poking around a bit, it looks like a *lot* of
>existing implementations are case-insensitive, and I'm not sure I want to
>declare that incorrect.  Accordingly, I now have the following text, which
>I think is an improvement.  Please, everyone, let me know if you disagree:

>   Some existing implementations treat <path-identity> as case-
>   sensitive, some case-insensitive.  The <path-identity> therefore
>   SHOULD be all lowercase and implementations SHOULD compare identities
>   case-insensitively.

First SHOULD is OK, but I am no sure why you then choose
case-insensitivity, since that is more work in an intensively used piece
of code. I would have chosen case-sensitively which should be OK if they
obey the first SHOULD, and lots of them do it anyway. Or let them do it
anyway they wish, provided they understand the risk (which is no worse
then being sent some articles you have already seen).

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Russ Allbery <[email-masked]>
Date: Tue, 02 Jan 2007 10:46:26 -0800

>> The more I research this, though, the more I think that while the above
>> is true, it's not ideal.  Poking around a bit, it looks like a *lot* of
>> existing implementations are case-insensitive, and I'm not sure I want
>> to declare that incorrect.  Accordingly, I now have the following text,
>> which I think is an improvement.  Please, everyone, let me know if you
>> disagree:

> of code.

Because existing software looks mostly case-insensitive, indicating that
the performance issue has not been an issue in practice and raising the
possibility that something is relying on that behavior given how
widespread it is.  So this seems safer to me, plus given that it's not
been a problem in practice, your argument that hostnames are inherently
case-insensitive seems fairly well-founded.

> I would have chosen case-sensitively which should be OK if they obey the
> first SHOULD, and lots of them do it anyway.

Yeah, that was my original argument, but when I realized that it differs
from common practice, I got nervous.  Any place we differ from common
practice we should have very solid reasons for doing so.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: Harald Alvestrand <[email-masked]>
Date: Wed, 03 Jan 2007 02:36:08 +0100

I thought we'd been here before, but the context differed:
USEFOR 3.1.3:

   Some software will try to match the <id-right> of a <msg-id> in a
   case-insensitive fashion; some will match it in a case-sensitive
   fashion.  Implementations MUST NOT generate a Message-ID where the
   only difference from another Message-ID is the case of characters in
   the <id-right> part.

USEFOR 3.1.4:

  A <component> SHOULD NOT consist solely of digits and SHOULD NOT
  contain uppercase letters.  Such <component>s MAY be used only to
  refer to existing groups that do not conform to this naming scheme,
  but MUST NOT be used otherwise.

     NOTE: All-digit <component>s conflict with one widely used storage
     scheme for articles.  Mixed-case groups cause confusion between
     systems with case-sensitive matching and systems with case-
     insensitive matching of <newsgroup-name>s.

So it seems that we have established a tradition where two different kinds 
of matching are out there in the wild that we note the fact, and specify 
something that works in all cases.

In this case, I think case-insensitive works in all cases - and since this 
is a procedures document, it's appropriate to specify the matching rule to 
use when executing the procedure, while such a statement wouldn't be 
appropriate in USEFOR.

But - matching the language in USEFOR 3.1.4 - I think "SHOULD" in both 
cases is appropriate. I don't see a big argument for declaring 
case-sensitive matchers unconditionally non-conformant. (But I could easily 
live with MUST too)

                    Harald

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 2 Jan 2007 19:08:23 GMT

>I now have:

(Presumably in section 3.6)

>   3.  It MUST reject any article that has already been successfully
>       sent to it, based on the Message-ID header field of the article.
>       To satisfy this requirement, a relaying agent normally keeps a
>       database of message identifiers it has already accepted.

>   4.  It SHOULD reject any article that matches an already-received and
>       honored cancel message or Supersedes header field following the
>       same rules as a relaying agent (Section 3.5).

OK. That is better. Though in fact it might wait until it sees the actual
article before deciding whether to honor the cancel.

>which is basically the same text as for relaying agents.

>> OK, but people will think it odd that you mention the obscure case of
>> cancels which arise first here, but make to mention of the common case.
>> Perhaps an extra step to say that it SHOULD then process any control
>> messages (including cancels) in acccordance with section 5.

>This would need to be added to relaying agents as well.

ITYM serving agents. Relaying agents should always just pass the cancel
message on and let serving agents downstream decide for themselves whether
to honour it.

But for serving agents, in addition to the wording above, you need at the
least a NOTE or a remark in parentheses to the effect that

"Cancel messages and Supersedes header fields arriving later then the
affected article are to be processed as described in section 5.3."

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 2 Jan 2007 20:06:40 GMT

>>>> 31. [-1] (5.2.1) Newgroup message SHOULD include application/group-info.

>Its absence may mean that no newsgroups file entry is created at all.  The
>newsgroups file is an optional feature of NNTP and need not be
>implemented, nor need the newsgroups file be accurate or complete.  See
>section 7.6.6 of RFC 3977:

>   The list MAY omit newsgroups for which the information is unavailable
>   and MAY include groups not available on the server.  The client MUST
>   NOT assume that the list is complete or that it matches the list
>   returned by LIST ACTIVE.

Hmmm! With hindsight I might have said that, where a Newsgroups file
existed, the Active file SHOULD be a subset of it. Oh well!

>Newsgroups are not required to have descriptions unless you want to send a
>checkgroups for them.  Some Netnews networks (such as the servers on which
>microsoft.* is hosted) simply don't use them.

>> Likewise for checkgroups.

In that case we need to look into checkgroups. In a hierarchy that does
not customarily bother with <newsgroup-description>s, I would expect a
checkgroups message to be of the form:

example.unmoderated.group<HTAB>
example.moderated.group<HTAB> (Moderated)

And many NNTP servers would probably keep a Newsgroups file containing
exactly that. Or at least the (Moderated) ones.

In which case I would expect a newgroup message for a moderated group to
contain an application/group-info with

example.moderated.group<HTAB> (Moderated)

so that such NNTP servers would get their Newsgroups files set up
accordingly.

>> So you need something like:

>>     The application/group-info MUST be included if the group is moderated,
>>     in which case it MUST include a <moderation-flag>. For other groups, it
>>     MAY be omitted if there is no <newsgroup-description> to be provided.

>Why do we need something like that?  There's a perfectly usable moderation
>flag in the Control header field.

OK, point taken there. The question is whether it would be normal for such
an application/group-info to be present for at least all moderated groups
(answer probably 'yes'), in which case should that be reinforced with at
least a 'SHOULD' (answer uncertain, but the 'SHOULD' you already have
covers it anyway)?

So I seem to have accepted your wording, but maybe there is still a need to
indicate that checkgroups are still appropriate even in hierarchies where
there are no <newsgroup-description>s.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 2 Jan 2007 17:45:34 GMT

>>>> 3. [-1] (3.3.1,3.4,3.9.2) From not omittable in proto-article

>>>> There is existing usage where the injecting agent fills in From header
>>>> (not possible in NNTP, of course)

>>> Intentional change.

>>> What injecting agent supports this?

>> INN apparently (see below).

>> Newsreaders such as rn, trn, nn, and others of that generation had (and
>> still have) the capability to interact directly with the newspool
>> (either on the same host, or more likely NFS mounted from some server)
>> rather than going via NNTP (which did not exist when they were first
>> written). They injected articles by calling a program 'inews' which, in
>> the absence of an explicit From:, assumed the poster was the user who
>> had called 'inews'.

>inews as distributed with INN or the common news readers is a (part of a)
>posting agent, not an injecting agent.

Ah! So it boils down to a matter off semantics. Surely, when the first
newsreaders were written (RN and earlier, before NNTP came on the scene),
their final act when posting an article was to hand it off to 'inews',
which came with the software of the server (which would be BNews in those
days). What does the 'i' in 'inews' stand for, if not for 'inject'. For
sure, people using UNIX never configured individual pieces of software
with their own name and id, because they naturally assumed that the system
would pick it up from /etc/passwd and the fact that they were logged in as
their own id.

>  You can see this by observing what
>actions it takes and what agent it talks to.  It sends messages to an
>injecting agent via POST and expects that agent to do the injection;

Not so. 'inews' existed long before the POST command was invented. AFAICT
from the O'Reilly Book, which was the definitive tutorial on administering
Usenet from 1986 onwards, the principal interfaces to BNews were 'inews'
and 'rnews'. Inews had an amazing collection of parameters (you could
even create groups locally with it or issue control messages).

>Our agent distinctions are largely based on how INN and similar news
>servers work.  C News, being a much earlier implementation, may not have
>the same distinctions, or they may be much less clear.  Some
>implementations written prior to our standard will combine different
>agents into one program, so it's possible that in C News inews combines
>functions of a posting agent and an injecting agent.

In CNews, 'inews' IS the injecting agent, which performs all the relevant
checks, sends it to the moderator if needed, and otherwise puts it into
the input queue alongside stuff received via 'rnews', whence it eventually
gets stored locally and relayed onwards. If stuff arrives via the NNTP
POST command, then it merely generates a call on 'inews'. It is certainly
not regarded as a posting agent (a crude posting agent 'readnews' is
provided - not really suited to serious newsreading) and there is a script
'postnews' which prompts you for Newsgroup and Subject, lets you use your
favourite editor to construct the article (adding other headers, even
From:, as you wish), and finally calls 'inews'. So 'inews' is not regarded
in any sense as a posting agent, although it would likely be called
directly by scripts which generated articles automatically (such as
modbots).

For sure, fashions have changed since then. But I am sure there are still
people using updated versions of those early newsreaders (such as nn and
trn) which still call inews, taking advantage of the automatic fillng in
of From, and those will all work happily with Bnews, Cnews, INN, and for
all I know any other newsservers which provide the 'inews' interface.

So there is no reason not to recognise that, even if the usage is not so
common as it once was. The wording I put into USEPRO (when discussing
headers that could be omitted from proto-articles) was:

"...(and even From if the particular injecting agent can derive that
information from other sources.)"

which seems to me an accurate description of the situation I have
described. I see no reason why a similar wording should not continue to be
used.

>INN's inews used to redundantly perform a few (although not all by any
>means) of the functions of an injecting agent, such as mailing posts to
>the moderators of moderated newsgroups.

Sure, 'inews' used to have all sorts of Bells and Whistles, and even CNews
provides a cutdown version 'injnews' which it recommends for routine use.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 2 Jan 2007 18:07:19 GMT

>>>> 20. [00] (3.5) Relaying agent MAY add a new Xref header.

>>> Because the same piece of software also implements a serving agent,
>>> which has to add the Xref header, and the same code path is used for
>>> all articles received by the server whether they are for further
>>> relaying or local serving.

>> Well if that is the case it was nominally the serving function that
>> added the Xref (section 3.6).

>No, because changes made by serving agents would only be seen by posting
>agents.  Serving agents do not sent messages to any other news server.
>Only relaying agents do that.

Ah! I see! We carefully said that the order of the steps in the various
"Duties" sections could be varied so long as the final effect was the
same, but we omitted to say the same for the order in which serving and
relaying were done.

You could cover it in the definitions section 1.4 by saying:

'A "relaying agent" accepts articles from injecting agents, serving
agents, or other relaying agents and distributes them ...".'

possibly with corresponding tweaks in the varius "Duties" sections.

My problem was with your wording under Duties of a Relaying Agent where
you said:

9. It MAY delete any Xref header field present and MAY add a new Xref
header field with any valid content. ...

That will be confusing to a reader who has not spotted the subtlety of the
particular order of happenings that we have described, and he will ask
himself "of what possible use is an Xref header added by a _relaying_
agent?". And of course the answer is "none" because it was not really the
relaying agent that you had in mind, but just an artefact of the way
things were described.

I think the simplest thing is to admit that relaying agents may obtain
their input from serving agents, as I have suggested above, and then that
Step 9 for relaying agents can be reduced to a simple "MAY delete XREFs".

>> Now whether such an agent first stores the article and then relays what
>> it has stored, or whether it relays first and then stores after is none
>> of our business.

And so our document should reflect that.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 2 Jan 2007 19:42:21 GMT

>>>> 30. [-1] (5.2) Nothing said about content of Approved header.

>>>> Surely, it SHOULD identify the person/identity/role of the issuer, ...

>>> The content of the Approved header serves no protocol purpose, and
>>> USEFOR already adequately covers the definition of its content.
>>> Control message authorization is done on the basis of the Sender or
>>> From header (preferrably in combination with a digital signature).

>The description in USEFOR is:

>   The Approved header field indicates the mailing addresses (and
>   possibly the full names) of the persons or entities approving the
>   article for posting.  Its principal uses are in moderated articles
>   and in group control messages; see [I-D.ietf-usefor-usepro].

Which certainly implies that an Approved header needs to contain an email
address identifying the person responsible for
posting/authorizing/whatever (which may well be the same as what is in the
From/Sender, or it may be the 'role' which the From/Sender person claims
to be fulfilling).

But, either way, USEPRO needs to ensure that the Approved header contains
what USEFOR says it is supposed to contain. You have covered this properly
in section 3.8 in the case of moderators. All I am asking is that you
cover it with similar wording in the case of group control messages.

>The Netnews protocol currently does not deal with authorization at all (an
>obvious flaw noted in Security Considerations).  Any authorization
>information you want to use has to be derived from the underlying
>transport protocol or from unstandardized extensions such as digital
>signatures.

No, that is 'authentication'. But I have written about 'authorization' in
another thread.

>If you're referring to group control mesages, said identity is checked
>against the *From or Sender* header field, not the Approved header, at
>least in INN.  INN ignores the contents of the Approved header.  I don't
>know if C News uses the contents of the Approved header field for control
>message permissions, but my impression from having maintained control.ctl
>for some years is that most everyone uses From/Sender.

Yes, I have looked at CNews and I was wrong. It looks at the From (not
even Sender AFAICS) and compares that with what it is configured to
honour.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 2 Jan 2007 19:28:55 GMT

>>>> 26. [-1] (4.2,4.3) Removed requirement that
>>>> application/news-[groupinfo,checkgroups] MUST NOT be used except with
>>>> relevant control messages.

>> Application types are intended to cause applications to be invoked, with
>> possible side effects.

>Could you provide a reference for this?  I don't see this statement in RFC
>2046.  The closest that it has is:

>   The "application" media type is to be used for discrete data which do
>   not fit in any of the other categories, and particularly for data to
>   be processed by some type of application program.  This is
>   information which must be processed by an application before it is
>   viewable or usable by a user.

OK. Perhaps I was carried away by the propensity of Microsoftware to
execute anything that is remotely executable even if the user just blinks
:-( .

And it is certainly the case that application types are mostly intended to
be processed by application programs that may have side effects beyond
fancy forms of display. And RFC 2046 goes to considerable lengths to warn
agains possible side effects of obeying an application/postscript.

>> In this case, the side efect is that a line gets added/changed in the
>> Newsgroups file.

>I don't agree with this interpretation, and I don't see anything in the
>text of the application/news-groupinfo registration that reflects this.

I doubt if you would be sending an application/group-info in an
article/message unless you thought that the recipient might want to use it
for exactly that purpose.

>It sounds to me like you're confusing the newgroup control message with
>the application/news-groupinfo MIME media type.  They're not the same
>thing.

But are clearly related.

>> OK, in that case what you need to say is:

>>     This media type is intended to be used in conjunction with the
>>     newgroup control message as described in section 5.2.1. It MUST NOT be
>>     automatically invoked in any other context without explicit human
>>     intervention.

>> and similarly for checkgroups.

>I disagree with adding anything like this to our document.

OK. In which case the existing sentences at the start of 4.2 and 4.3 will
suffice.

---

From: Russ Allbery <[email-masked]>
Date: Tue, 02 Jan 2007 23:34:22 -0800

>> inews as distributed with INN or the common news readers is a (part of
>> a) posting agent, not an injecting agent.

[...]

>> You can see this by observing what actions it takes and what agent it
>> talks to.  It sends messages to an injecting agent via POST and expects
>> that agent to do the injection;

> Not so. 'inews' existed long before the POST command was invented.

inews *as distributed with INN or the common news readers* does indeed
behave exactly as I describe.  As I mentioned explictly later, the version
that comes with C News is much older and may well do something different.

> For sure, fashions have changed since then. But I am sure there are
> still people using updated versions of those early newsreaders (such as
> nn and trn) which still call inews, taking advantage of the automatic
> fillng in of From, and those will all work happily with Bnews, Cnews,
> INN, and for all I know any other newsservers which provide the 'inews'
> interface.

Sure, they can continue to delegate some of their posting agent duties to
inews if they have an inews on hand that does them.  If they are used with
C News, that inews will apparently also do injection; if they are used
with INN, that inews will hand the post off to a separate injecting
agent.  The newsreader doesn't have to care.  Neither, so far as I can
tell, does the language in our draft.

[...]

> I see no reason why a similar wording should not continue to be used.

It draws the line between posting agent and injecting agent in the wrong
place for the most common case of NNTP, and is therefore unnecessarily
confusing.  It's a semantic distinction that I think is important in
trying to understand which piece of software does what.  C News used
without NNTP will be an unusual case.

---

From: Russ Allbery <[email-masked]>
Date: Tue, 02 Jan 2007 23:40:31 -0800

> That will be confusing to a reader who has not spotted the subtlety of
> the particular order of happenings that we have described, and he will
> ask himself "of what possible use is an Xref header added by a
> _relaying_ agent?". And of course the answer is "none" because it was
> not really the relaying agent that you had in mind, but just an artefact
> of the way things were described.

I think that approach is more confusing than mine and would prefer to keep
the approach that I used in my draft.  The agents are a conceptual model,
and conceptual models are more useful the simpler we can keep them.  We're
always going to have to do some mapping of conceptual agents to real
software and the boundaries are always going to be messy; the best we can
manage is an approximation.  But the purpose of the conceptual model is to
provide a clear view of the overall flow so that one has the theory to
then apply to a particular implementation, and the simpler we can keep the
model while still expressing how Netnews works, the easier that will be.

> their input from serving agents, as I have suggested above, and then
> that Step 9 for relaying agents can be reduced to a simple "MAY delete
> XREFs".

Except they don't obtain their input from serving agents when you look at
the architecture of news servers that implement those roles using separate
programs.  This is confusing primarily because such servers are unusual
(and necessary for only very high reader volume) and most servers instead
implement a simpler combined relaying and serving agent.

---

From: Russ Allbery <[email-masked]>
Date: Tue, 02 Jan 2007 23:51:11 -0800

> example.moderated.group<HTAB> (Moderated)

Yeah, I was wondering about that.  Currently, we don't allow that content
in a checkgroups message, but I've seen this in the wild (without the
trailing HTAB, in fact) and I believe that some servers honor checkgroups
of this form.  I'd have to check to be sure, though.

I don't think it's unreasonable to say that if you want to issue a
checkgroups, you have to provide a description, even if it's something
lame like "?".  But it's probably worth discussing whether we want to
permit the empty description explicitly.

>>>     The application/group-info MUST be included if the group is
>>>     moderated, in which case it MUST include a <moderation-flag>. For
>>>     other groups, it MAY be omitted if there is no
>>>     <newsgroup-description> to be provided.

> OK, point taken there. The question is whether it would be normal for
> such an application/group-info to be present for at least all moderated
> groups (answer probably 'yes'), in which case should that be reinforced
> with at least a 'SHOULD' (answer uncertain, but the 'SHOULD' you already
> have covers it anyway)?

I don't see why a moderated group should be different from an unmoderated
one when it comes to newgroup control messages.  If there's no
description, there's no description; for newgroup, unlike checkgroups,
nothing relies on the description for any protocol function beyond
updating the optional newsgroups file.  (Up to and including reading agent
recognition of moderated newsgroups; that's done via the LIST ACTIVE
flag.)

---

From: Russ Allbery <[email-masked]>
Date: Tue, 02 Jan 2007 23:43:27 -0800

> article/message unless you thought that the recipient might want to use
> it for exactly that purpose.

In my role as control message issuer for the Big Eight hierarchies, I
would exchange such mail messages routinely.  I wouldn't be processing
such parts to change a newsgroups file; I'd use it as input into the
database used to send subsequent control messages.

> But are clearly related.

They're related only insofar as we need application/news-groupinfo to
clearly express the complete action of a newgroup message, but the former
has no inherent connection to the latter.  It's a one-way link.

---

From: Harald Alvestrand <[email-masked]>
Date: Wed, 03 Jan 2007 09:24:18 +0100

>possibly with corresponding tweaks in the varius "Duties" sections.
No.

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 3 Jan 2007 16:04:51 GMT

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>First, one issue that's *not* directly part of this.  Even if we keep the
>current USEPRO method, we need to have a staleness check on the Date
>header field at injection.

Well there we have the first major disagreement. The whole point of
introducing Injection-Date was to remove the need for staleness checks
based on Date (except in cases where older software had not provided an
Injection-Date).

The Date header now indicates when the article was composed (following RFC
2822), so it can arise that the poster carries it around on his laptop for
several days before it actually gets injected. That could reasonably be a
week later (whereas Russ proposes to declare it stale after 72 hours,
which is far too short). Maybe even a fortnight if he writes the article
and then goes off on holiday for two weeks and omits to dialin to his
server before he goes. Clearly, leaving it for several months would be
stupid, but such timing issues are a matter of sensible practice, and not
a matter for the protocol.

>  We can't make a single leap to allowing stale
>Date header fields from not having any Injection-Date header field since
>most existing software is going to ignore the Injection-Date header field
>entirely and impose staleness checks on Date after injection.  Therefore,
>an article with a stale Date is going to propagate very poorly, and
>accepting such an article just to let it be dropped silently later by most
>of the Netnews network would be a poor interoperability choice.

OK, so let it propagate poorly. In fact, I don't think its propagation
will be all that bad. For sure, the fast transit backbone may decline it,
but it will still flood its way through the system via serving agents with
normal expiry times. I would have expected it to have reached most sites
within one day. And as Injection-Date becomes more widely implemented,
things will of course get better.

But this is a thing that is easily testable. I have just posted a series
of articles "1 day stale", "2 days stale" up to "9 days stale" to
misc.test, and I will see what the autoresponders tell me (and it would be
useful for members of this list to look out for them also). I expect to
see some long and unusual Paths for the staler ones, but I expect them
all to arrive eventually. We shall see.

>Now, let's look at the conditions under which this situation arises.

>....  The INN
>code to permit people to inject posts via IHAVE is something I committed
>only under protest, and I'm entirely content to have its behavior declared
>non-comformant by a standard.  It is a *very* specialized configuration
>for strange cases where the people involved know just what they're doing.

Not really. I regard it as a method of avoiding unnecessary wastage of
server resources in cases where articles are injected at more than one
server. If one of the servers has already received it, then you save the
trouble of transmitting the whole thing again only for it then to be
thrown away.

>That being said, there are disjoint Netnews networks that people want to
>connect, and multiple injection or proxying injection isn't always the
>best approach (for one, it requires a real-time connection with the remote
>injecting agent).  My definition of two disjoint Netnews networks is two
>Netnews networks between which there is no relaying agent to relaying
>agent connection.

Not so. You also have to ensure that there is at most only one *-agent to
injecting agent route.

>  Significant examples are:

> * The many people who do not have any relaying agreement with a large
>   Netnews network such as Usenet, only a reading agent relationship, but
>   still want the features of a full-fledged news server (longer article
>   retention, multiple users sharing the same spool, faster service to
>   reading agents, local newsgroups without requiring clients use multiple
>   servers, off-line operation).  This has become increasingly common over
>   the last five years for both personal and small organization use.

That is probably safe for personal use, but in the case of small
organizations, and increasingly as those organizations get larger, there
is the possibility that some well-meaning user in the organization will
create some surreptitious path/leak to the outer world, even if only for
some supposedly safe group that suddenly becomes unsafe because of an
unanticipated crosspost. It is a sad fact of life that such leaks DO
happen.

> * Ad hoc connections to a stand-alone news server that serves only a
>   particular hierarchy. ...

And again these can lead to unanticipated leaks.

>The goal is to handle these cases with a solution that has the
>following properties:

> * Netnews loop detection and prevention is preserved.

> * Articles are stamped with trustworthy injection information.  Since,
>   particularly in the first and third cases, the reason why no relaying
>   agreement is in place is precisely because the original injection
>   information can't be trusted, this requires replacing injection
>   information when the message is gated from one Netnews network to
>   another.

This is certainly true in the case of Injection-Info. The case in dispute
is Injection-Date which, where it already exists, can probably be trusted
to have an accurate date, or at worst a date too far into the past which
is a safe error so far as loop avoidance is concerned.

>My realization when reviewing the draft was that all or nearly all of the
>cases where this arises and is not simply a bug (such as treating relaying
>and injecting as interchangeable) can be meaningfully treated as disjoint
>Netnews networks.

Yes, I see now where you are coming from as treating local networks at the
edges of Usenet as "disjoint", but my concern is that such disjointedness
is too hard to police.

>  The case which Charles raises of a news server that
>acts as a relaying agent to one server and a posting agent to another
>server on the same network is an interesting possible exception; it's
>rather pathological, but I can see the argument that the end results, if
>done properly, are not distinguishable from multiple injection.

Yes, I accept "pathological" as a valid description of what I do, but my
view is that most cases of reinjection will turn out to be "pathological"
to some degree, and that it is impossible to foresee all the strange
circumstances that might be involved.

>  But let's
>put that aside for a moment and look at the alternatives before us in the
>more typical case.

There are essentially two possibilities:

1. Injection-Dates are ALWAYS rewritten by injecting agents (or
alternatively articles containing them are ALWAYS dropped).

2. Once an article has acquired an Injection-Date (whatever network put it
there), it ALWAYS retains that Injection-Date wherever it subsequently
goes.

#2 (which I advocate) is evidently safe against causing loops, but might
cause articles to be lost. So one MIGHT allow an exception for admins who
really REALLY knew what they were up to.

#1 is not safe against loops unless the admin who allows the rewriting
really REALLY knows what he is up to.

>Option 1 (my current draft)

>    Proto-articles may not contain Injection-Date.  Injecting agents must
>    reject any message that contains Injection-Date.  If an agent needs to
>    reinject a message, it must think of itself as a gateway, remove the
>    injection header fields (including Injection-Date), perform whatever
>    checks it needs to ensure that it's not creating a loop, and then
>    reinject the message, at which point it gets a new Injection-Date
>    header field with the current date.  This gateway is fully responsible
>    for not creating loops.

>Option 2 (current USEPRO draft)

>    For a given post, an injecting agent may either be doing injection or
>    reinjection based on whether injection headers fields are already
>    present.  It may decline to do reinjection.  If it doesn't, it may
>    rename an existing Injection-Info header to some other name or remove
>    it and must retain any existing Injection-Date header field.  It may
>    perform a staleness check against the Date header field if no
>    Injection-Date header field is present.  (The current USEPRO draft
>    doesn't say an injecting agent can perform staleness checks against an
>    existing Injection-Date header, but I presume that's simply an
>    oversight.)

Not an oversight. It was presumed that the immediately following serving
or relaying agent would do that check.

>    The injecting agent is responsible for not creating
>    loops, not the agent that offers the article for reinjection.

If the injecting agent retains the Injection-Date (maybe it should check
it is not into the future) then it has fulfilled its loop-avoiding
responsibility. If the offering agent has removed the Injection-Date, then
that is another matter.

>I've already talked about why I think the first option is cleaner and
>clearer from a protocol description and implementation standpoint.  Let me
>instead focus on failure modes.

>The primary risk of the first option is that it could create a slow loop
>in the presence of bidirectional gatewaying as follows:

Which is why USEPRO required an outgoing gateway to retain the
Injection-Date if possible, i.e if the other medium had some place to
record it. If the other medium cannot record it anyway, then the two
Options are equivalent, and both must rely on the circularity checking by
the gateways (which SHOULD be provided anyway).

>The second option does not have this same failure mode, and indeed it
>opens the door to fewer additional problems as long as we still have to
>enforce fresh Date header fields.  The failure mode with the second option
>comes when we want to drop the staleness check on a Date header field.
>Suppose that a stand-alone news server B wants to pull articles from a
>Netnews network A for local consumption.  B has a power failure and is
>off-line for an extended period of time (a week, say).  When it comes back
>up, it wants to catch up on all of the news from A that it missed while it
>was down.  Now, as long as we have to enforce Date staleness, it can only
>do this by ignoring a SHOULD at reinjection, since those articles will
>likely now have stale Dates, but it can be configured to do that.
>However, under option 2, catching up on gatewaying has now become
>impossible because it is REQUIRED to retain the Injection-Date header of
>the original message but that Injection-Date header is now stale and will
>therefore be rejected by any relaying or serving agent in B.

Well that is hardly a common scenario, and for sure B is going to have to
bypass some checks somewhere whichever Option we use. For sure, it no
longer cares about relaying stuff onwards (the rest of the net will have
seen those articles long since), so it only cares about its own serving
agent(s). Now for serving agents, staleness is determined by "the earliest
articles of which it keeps record" (in its history file). Since the expiry
time of serving agents is typically longer than a week, it may not even
need to bypass the staleness check at all, but if not then a temporary
change to that "earliest articles of which it keeps record" is all that is
needed. That might indeed let in a few articles it had actually seen
before the breakdown but which got sent again from A as part of the mixup,
but that is a small price to pay for a breakdown and will not lead to any
loops so long as the original Injection-Date is still retained, and
provided it does not promptly break down again for a further week. Indeed,
it is that magic "72 hours" which seems to be built into Option 1 that is
likely to be more of a problem than Injection-Date staleness.

>Now, my basic argument, apart from the simplicity of option 1, is that the
>failure scenario for option 2 is worse than the failure scenario for
>option 1.

Which doesn't seem to be born out by my analysis.

>Okay, phew.  Now back to Charles's case.

>Suppose we have a new server with a relaying agreement to another
>unreliable news server and a posting agreement with a much more reliable
>news server, and that news server wants to send outgoing articles via both
>paths.  Under option 2, the injecting agent that this server talks to must
>support reinjection (it's an optional feature in the current USEPRO
>draft), must enable it for that client, and then takes responsibility for
>doing the appropriate transformation, but Injection-Date is retained.
>Under option 1, this news server must either support multiple injection
>(far and away the best option) and inject at the remote server at the same
>time as the message is injected locally, or it must both relay the message
>and gateway it to the remote injecting agent.  The gatewaying is more
>complex, but can be done entirely locally and the article will then look
>to the remote server like a regular proto-article.  No special support on
>the remote server is therefore required; all of the work is born by the
>agent doing something unusual.  In either case, two slightly different
>copies of the message will exist with different trace headers, but this is
>acceptable; the same is true for multiple injection.

I agree that the only cases in which Option 2 could be problematical is
with a privately operated local server such as my own. In all other cases,
as I have tried to show, it works at least as well as, and often better
than, Option 1.

Now for a local server there are two problems:

1. It may operate offline, so there will be a delay between when it is
injected locally and when it is relayed (whether by proper relaying or by
posting), though it would be unusual for that delay to be long enough to
cause premature staleness. Hence the reason I mentioned the possibility of
allowing exceptions for admins who "really REALLY" knew what they were
about. In this case, "really REALLY" means being quite sure that there is
no possibility of unofficial leaks to the outer world (easy if he is the
sole user of the system, harder in a small organization where other users
are not so easily controlled). But in that case, a reasonable "cheat"
would be to delay adding the Injection-Date until the time came to make
the outgoing connection.

2. If he only has posting privileges at his outgoing feed (or for some of
his outgoing feeds) then he has to be sure that they will accept what is
technically a reinjection. Again, the obvious "cheat" is to omit the
Injection-Date, and for that he has to be just as "really REALLY" sure as
in the first case.

Yes, the burden of preventing nasty things happening rests with the agent
wanting to do the unusual things.

But note also that the alternative scenario you have suggested for
injecting only to the outgoing feed, or to the outgoing feed and the local
system simultaneously, only works if you have an "always on" connection to
the outside.

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 3 Jan 2007 19:41:42 GMT

>practice we should have very solid reasons for doing so.

I think you differ from common practice whichever you say. And if you say
one or the other then people may tend to assume they can rely on it, in
spite of the first SHOULD. So if you say "case-insensitive" they will tell
their peers that their site is called "FOO" and then put "foo" in their
Path. And if you say case-sensitive, then they will call their site "FOO"
even though a site "foo" already exists.

I think the second case is marginally more harmful, so it seems you have
talked me into agreeing with you :-) . Harald seems to agree.

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 06 Jan 2007 14:39:01 +0100

Charles Lindsey wrote:

> this is a thing that is easily testable. I have just posted a series
> misc.test, and I will see what the autoresponders tell me (and it
> would be useful for members of this list to look out for them also).

So far I don't see 4..9 days.  This group is horrible, thousands of
"NNTP monitor" messages.

The 3 days article is <news:[email-masked]>
Path: nnrp3.clara.net!monkeydust.news.clara.net!news.clara.net!
 wagner.news.clara.net!usenet-fr.net!nerim.net!oleane.net!oleane!
 keepthis.news.telefonica.de!telefonica.de!newsfeed.r-kom.de!
 newsfeed.freenet.de!fu-berlin.de!uni-berlin.de!individual.net!not-for-mail

Please post the Message-IDs of your other tests if I should check
them directly.  The 3 days Path might be already unusual, the last
time I looked at it stuff from German servers reached clara.net via
Arcor, not via Oleane.

Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 8 Jan 2007 19:00:44 GMT

>>which is basically the same text as for relaying agents.

Ah! I did not spot the significance of that ...

>message on and let serving agents downstream decide for themselves whether

But having read the latest draft, I see that this is not so. Relaying
agents are invited to honour cancel messages, even though they have no
articles actually stored which they might want to hide/destroy. And there
is nothing in 5.3 to suggest that cancel messages require action from
other than serving agents.

So why this requirement in relaying agents?

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 8 Jan 2007 19:32:37 GMT

>>possibly with corresponding tweaks in the varius "Duties" sections.

Why not? It reflects exactly the ways in which implementations actually do
things, and removes possible confusions (like the one under discussion)
when the implementation does things in a different order to the one in our
model, necessitating wording which is even more confusing if you do not
appreciate the subtlety behind it.

If the model is left unchanged, then the confusing wording under relaying
needs to be clarified. I have already suggested:

   Any Xref header, whether present on input or added by an associated
   local serving agent, MAY be deleted before relaying.

which is an acceptable alternative to my suggested deffinition change. But
one or the other needs doing.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 8 Jan 2007 21:28:34 GMT

The following list summarizes the outcomes of the long list of protocol
differences that I raised. Each begins with [-1], [00] or [+1] to indicate my
own assessment of it (possibly modified since the original list). I regard the
{+1} and [00] as resolved (unless someone else want to take up one of the
[00] ones).

Of the original 44 items (numbered 1-42 because of some duplication), there
remain 18 [-1]s, some of which are the subject of continuing discussions. The
rest may well become ISSUEs in the next few days. A list of them follows:

1
3
4
6
9
11
12
16
18
20
23
30
34
37
38
39
40
42

1. [-1] (2.2) <path-identity> - Possibility to use a FQDN not resolvable
in the DNS.

2. [+1] was [00] (3.2) Russ[-1] now case-insensitive
<path-identity>s are case-sensitive.

3. [-1] (3.3.1,3.4,3.9.2) From not omittable in proto-article ISSUE

4. [-1] (3.2.2,3.4,3.8) Injection-Date can be rewritten.
<[email-masked]> Wed, 3 Jan 2007 16:04:51 GMT Re: Injection-Date and reinjection

5. [+1] was [-1] (3.3.4) Now fixed
SHOULD trim References to keep <= 998: reduced from MUST.

6. [-1] was [00] (3.4)
Injecting agents reporting rejection to user agent; was
SHOULD, now merely "encouraged".

7. [+1] (3.4) Injecting agent SHOULD have access to list of newsgroups.

8. [00] was [-1] (3.4) with revised wording (accepted)
Requirement for injecting agent to forward articles to
moderator groups reduced from MUST to SHOULD.

9. [-1] (3.4,3.5) under current discussion
Prepending of <path-identity> to Path by injecting agent
reduced from MUST to SHOULD.

10. [00] (3.4) Folding of Path header when length becomes excessive
reduced from SHOULD to MAY.

11. [-1] (3.4) Recommendation that each injecting agent SHOULD use a
consistent format for Injection-Info removed.

12. [-1] (3.5) Explanatory text suggested referring to USEPRO 3.2.4
The significance of "world" and "local" in the Distribution
header are no longer mentioned.

13. [+1] (3.5,3.6) Removed possibility of "aliassing out" a site where you
didn't want an article to go by including it in the Path to the right of
POSTED.

14. [+1] (3.5) Relaying agent MUST reject articles without Newgroups or
Message-ID or (injection-Date or Date).

15. {+1} was [-1] (3.5) Relaying agent MUST reject any article already already sent 
to it.

16. [-1] (3.5) under current discussion
Relaying agent SHOULD deal with cancel message (if it
chooses to honour it)

17. [00] (3.5) Relaying agents SHOULD determine verified/expected source
of article and construct <path-diagnostics> accordingly.

18. [-1] (3.5) Russ[-1] Incorrectly fixed
No provision for >1 <path-identity> per relaying agent.
<[email-masked]> Mon, 1 Jan 2007 20:52:37 GMT Re: Path header field

19. [00] (3.5) Russ[+1] Partially fixed
No provision to omit <path-identity> for intermediate
servers on same site.

20. {-1} was [00] CHL[+1] with revised wording (but wording not accepted)
(3.5) Relaying agent MAY add a new Xref header.

Alert! Duplicated issue number #21!

21A. [00] (3.6) Storage of control messages in the (pseudo) hierarchy
control(.*) is a SHOULD.

21B. [00] was [-1] with revised wording (accepted)
(3.6) No mention of processing cancel messages by serving/storage
agents.

22. [00] Frank[-1] Forrest[+1]
(3.7) Removed recommendation that reading agent SHOULD have the
capability to display the raw article 'as received'.

23. [-1] Frank[-1]
(3.8) Moderators merely "encouraged" (was SHOULD) to retain
existing <msg-id>.

24. [00] (3.2.1) Removed provision (MAY) for outgoing gateway to change
Content-Transfer-Encoding.

25. [+1] (4.1) Application/news-transmission no longer handles batches.

26. {+1} was [-1]
(4.2,4.3) Removed requirement that
application/news-[groupinfo,checkgroups] MUST NOT be used except with
relevant control messages.

Alert! Duplicated issue number #27!

27A. [+1] (4.3) Application/news-[groupinfo,checkgroups] have a charset
parameter.

27B. [+1] was [-1] Russ[-1] Revised wording agreed.
(5) Nothing stated regarding Newsgroups header of Control
messages.

28. [00] Frank[-1]
(5) Subjects starting with 'cmsg' not accompanied by a Control
header MAY be rejected.

29. [+1] was [00]
(5.2) Newsgroup-names MUST be checked against disallowed names before
group control message is honoured.

30. [-1] (5.2) Nothing said about content of Approved header.
<[email-masked]> Tue, 2 Jan 2007 19:42:21 GMT Re: Approved header field content (was: Protocol changes in draft-allbery-usefor-usepro-00)

31. [+1] was [-1] with revised wording (not accepted, but still CHL +1)
(5.2.1) Newgroup message SHOULD include application/group-info.

32. [+1] was [-1] Russ[-1] Reworded
(5.2.1) Newgroup message containing other attachments in addition to
news-groupinfo to be multipart/related (was multipart/mixed).

33. [+1] Russ[+1](just)
(5.2.3) Checkgroups message still contains chkscope and chksernr
arguments.

34. [-1] was [00] with revised wording not accepted
(5.2.3) Checkgroups SHOULD contain sub-hierarchies excluded by
chkscope, for backwards compatibility.

35. [+1] (5.2.3) Chksernr MUST increase with successive checkgroups message.

36. [+1] was [00] (5.2.3) CHL[+1]
Body of checkgroups SHOULD be application/checkgroups, and
otherwise SHOULD treat as if it were.

37. [-1] was [00] Frank[-1]
(5.3) Body of cancel message MAY contain ...explanatory text.

38. [-1] revised wording not accepted
(5.3) No mention of who should be in From/Sender of cancel message.

39. [-1] Russ[-1] still under discussion
(5.5) The <relayer-name> in ihave and sendme control messages is now
optional.

40. [-1] (5.5) Format of batched news in response to sendme removed.

41. [+1] was [-1] (5.5) Russ[-1] new text written
The protocol for using the 'to.' newsgroup-name is omitted.

42. [-1] (5.6) Obsolete control messages SHOULD NOT be sent/honoured/

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 8 Jan 2007 17:13:13 GMT

>behave exactly as I describe.

Which is true, but does not answer the point. Are trn and nn not "common
newsreaders" any more?

>  As I mentioned explictly later, the version

>confusing.

I disagree. Historically (as in the days of BNews and Cnews), as I think I
have shown, 'inews' was the mechanism which caused injection to take
place, and filling in the From header was part of what it did. As I said,
the 'i' in 'inews' stands for 'inject' (just as the 'r' in 'rnews' stood
for 'relay').

When NNTP became the common mode of intereaction with newsreaders,
automatic filling in of the From could not then be done, but that did not
alter the situation wrt to existing usage - 'inews' did not suddenly
cease to become a means of injecting; rather in NNTP contexts it lost some
of its functionality.

>  It's a semantic distinction that I think is important in

USEPRO should not be concerned with which piece of software does what. The
fact remains that some injection mechanisms did (and still do) support
automatic deduction of From, even if those mechanisms are now relatively
uncommon, and USEPRO should recognise this.

The wording in USEPRO-06 did recognize it, but also indicated that it was
unusual, which seems to reflect the situation as it stands today. The
actualy wording used, in connection with headers omittable in a
proto-article, was "... (and even From if the particular injecting agent
can derive that information from other sources)". I see no reason why we
should not continue to say something of that nature.

ISSUE raised.

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 09 Jan 2007 14:05:11 +0100

> cease to become a means of injecting; rather in NNTP contexts it lost some
To put it another way:
In the days of BNews and CNews, the "inews" program formed part of the 
posting agent as well as part of the injecting agent.

In an NNTP context, it is only a part of the posting agent.

PLEASE don't attempt to require exact match between pieces of the model 
and pieces of implementations that predate the model.

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 09 Jan 2007 14:08:38 +0100

> So why this requirement in relaying agents?
Because if it is not there, a relaying agent that stops messages for 
which it has seen a cancel would be nonconformant with the 
specification. My guess is that combined relaying/serving agents would 
indeed do this.

If a relaying agent were not allowed to honor cancels, it would be 
nonconformant if it did.
That would be stupid.

                  Harald

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 09 Jan 2007 14:11:32 +0100

> one or the other needs doing.
You have then abandoned the model that relaying agents talk to each 
other, and that serving agents only talk to their local relaying agents 
and reading agents.

You have increased the amount of coupling in the system model, for no 
benefit except saving a special-case handling of the Xref: header.

If you want to question why increasing the amount of coupling in the 
system model is bad, I can get you some references on system modelling 
and object orientation.

                       Harald

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 09 Jan 2007 16:10:17 +0100

> system model is bad, I can get you some references on system modelling

Small interfaces are nice.  I'm not exactly sure what this Xref business
is about, therefore I compare it with a Return-Path in mail:  There MUST
be one and only one after delivery, and it has to reflect the MAIL FROM
as seen by the MDA - and any other Return-Paths inserted earlier have to
be removed by the MDA.

For an Xref it's similar wrt the serving agent, any old Xrefs have to be
removed, a new is inserted.  Ideally MUST + MUST, but SHOULD + SHOULD is
good enough if some serving agents don't support the concept at all (?)
Or no 2119 keywords if it's more complex.

Frank

---

From: Russ Allbery <[email-masked]>
Date: Tue, 09 Jan 2007 07:26:26 -0800

Harald Alvestrand <[email-masked]> writes:

> Because if it is not there, a relaying agent that stops messages for which
> it has seen a cancel would be nonconformant with the specification. My
> guess is that combined relaying/serving agents would indeed do this.

> If a relaying agent were not allowed to honor cancels, it would be

Also, in the days of spam cancels, people went out of their way to receive
spam cancels on relaying agents prior to the original message so that they
could avoid even relaying the spam.  This is less of an issue these days,
but it may still come up in some situations.

Note too that relaying agents don't relay immediately in all situations.
Cancelling a message may still have an effect for slow peers, causing the
message to disappear before it can be relayed to them (particularly for
peers that were *intentionally* slowed by five minutes or so, another
strategy commonly used in the days of spam cancels).

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 10 Jan 2007 16:33:54 GMT

>posting agent as well as part of the injecting agent.

No, I would put it as "inews" is/was the interface to the injecting function
of the newsserver, as used by those useragents which do not make a direct
TCP connection to port 119 themselves (as modern user agents mostly do). But
such user agents do still exist, and are used.

The interface provided by inews is fairly well documented (see the
O'Reilly book, for example). Where an NNTP connection on port 119 is to be
made regardless, 'inews' is simply a wrapper that makes the TCP
connection, but it still supports the traditional interface.

>and pieces of implementations that predate the model.

I don't think the basic model has changed since those early days. There
have always been user/injecting/relaying/serving agents connected together
according to out present model (though not necessarily known by that
terminology).

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 10 Jan 2007 18:14:36 GMT

>other, and that serving agents only talk to their local relaying agents 

There are essentially two ways in which things may be implemented:

        -------------------                -------------------
       | outgoing relayed  |              | outgoing relayed  |
       |     articles      |              |     articles      |
        -------------------                -------------------
                 ^                             ^
                 |                            /
        -------------------                  /
       |   relaying agent  |                /
        -------------------                /
                 ^              -------------------    -------------------
                 |             |   relaying agent  |  |   serving agent   |
        -------------------     -------------------    -------------------
       |   serving agent   |            ^                      ^
        -------------------              \                    /
                 ^                        \                  /
                 |                         \                /
        -------------------                ------------------- 
       | incoming relayed  |              | incoming relayed  |
       |     articles      |              |     articles      |
        -------------------                -------------------

I suspect implementations mostly follow the LEFT scheme, which is likely
to give rise to Xref headers created by the serving agent which the
relaying agent MAY then choose to remove.

But some no doubt follow the RIGHT scheme in which there are unlikely to
be any Xref headers for the relaying agent to remove. The Model in USEPRO
essentially sets out the RIGHT scheme, which then necessitates some
mention of relayers removing Xrefs which, according to the model, should
never have been there.

My objection is to the wording in the draft which says, for relayers:

   8.  It MAY delete any Xref header field present and MAY add a new
       Xref header field with any valid content.  The Xref header field
       is not used by relaying agents, but relaying agents that are also
       serving agents may generate Xref header fields for their own
       internal purposes.

and particularly to the bit "and MAY add a new Xref header field", since
is is clear that that never happens in either of the two schemes, and it
is only put there as an artefact to explain why there might be an Xref to
remove.

This is evident from the words "relaying agents that are also serving
agents" which, according to our model, is actually a contradiction in
terms. There are relaying agents and there are serving agents, and some
*news-servers* may include both (but that does not cause the relaying
agent to also "be" a serving agent).

Hence my suggeted wording:

which is shorter and avoids that contortion of the model by speaking of
an "associated" serving agent (i.e. one contained in the same
news-server).

>benefit except saving a special-case handling of the Xref: header.

Yes I agree that is undesirable, though not harmful in this case. I would
prefer my suggested rewording.

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 10 Jan 2007 17:30:03 GMT

>> guess is that combined relaying/serving agents would indeed do this.

Looks like a case for a "MAY". Yes, I agree that a combined
relaying/serving agent might not forward the cancelled message since it
might no longer have a copy available for that purpose.

>> If a relaying agent were not allowed to honor cancels, it would be

Ah! I had always taken the view that serving agents would be expected to
act upon cancels (subject to site policy) but that relaying agents ought
to forward all articles received on the grounds that to do otherwise was
interfering with the right of downstream sites (notably downstream serving
agents) to decide for themselves, according to their site policy, whether
to honour them. And I still think that is the proper behaviour.

>but it may still come up in some situations.

So I see now what you are getting at. I think I would therefore want to
take the view that serving agents SHOULD (modulo site policy) act on
cancels, but that for relaying agents it should be, at most, "MAY".
Perhaps USEAGE could then discuss the ethics of doing so. I really don't
like the idea of doing it for "ordinary" cancels, though I can see the
benefit in the case of spam.

But, either way, the wording you have written for relaying agents only
applies to cancels arriving before the actual message, which is not the
common case (except for sites which deliberately try to make it the common
case, as you have said). So it is odd that 5.3 does not mention relaying
agents - but I suppose that, if the cancel arrives after the message, it
is too late for a relaying agent to act. That is an asymmetry which should
perhaps be mentioned.

---

From: Russ Allbery <[email-masked]>
Date: Wed, 10 Jan 2007 11:45:12 -0800

> cancels, but that for relaying agents it should be, at most, "MAY".

Given the current security problems with honoring cancels, honoring
cancels should not be anything more than a MAY.  I prefer the approach
currently taken by the draft, where that MAY is even implicit, as it is
for all control messages.  We simply say "here's what the control message
means and here are the problems with it; whether or not you want to act on
it is up to you."

The language that started this discussion is the language in duties that
explains what the agent needs to do *if the cancel was honored*.  So I
think you've accidentally changed topics in the middle of this thread, and
we should get back to the original topic.

> common case (except for sites which deliberately try to make it the
> common case, as you have said).

Correct.  As mentioned previously, I think that's the only case that
requires any special handling in the duties section.

> So it is odd that 5.3 does not mention relaying agents - but I suppose
> that, if the cancel arrives after the message, it is too late for a
> relaying agent to act.

Not true, as I think should be apparent from my previous message on this
thread.

No specific agents are singled out in 5.3 because the meaning of the
cancel message does not change based on what agent is processing it.

---

From: Russ Allbery <[email-masked]>
Date: Wed, 10 Jan 2007 11:59:59 -0800

> There are essentially two ways in which things may be implemented:

I think that you're considering anything that has article storage to be a
serving agent, which is causing you to talk at cross-purposes to what the
draft actually says.  That's not the definition of a serving agent.  A
serving agent is an agent that makes articles available to reading
agents.  Full stop.

Relaying agents also have article storage.  That does *not* mean that
they're combined with a serving agent.

Stanford's news system doesn't look anything like either of your models.
It looks like this:

 <other news peers> <-+-----> relaying agent <--+-> serving/injecting agent
                       \                       /
                        ----> relaying agent <-

Those two relaying agents both have article storage so that they can hold
articles for a while if a peer is slow, or if the internal serving agent
is slow.  They both generate Xref headers because INN just always
generates Xref headers no matter what -- not necessarily ideal, but in
practice completely harmless.  Neither of those relaying agents accept any
connections from reading agents and could not serve articles to reading
agents because they both have overview and nnrpd disabled.

> be any Xref headers for the relaying agent to remove. The Model in
> USEPRO essentially sets out the RIGHT scheme, which then necessitates
> some mention of relayers removing Xrefs which, according to the model,
> should never have been there.

You're focusing way too much on the removal part.  I know of no agent that
removes Xref headers except when adding its own Xref header.  The draft
allows removal without adding a new one, just because there's no reason
not to, but what happens in practice is that relaying agents remove the
existing Xref header and add a new one.

This is a purposeless modification to the article, and back in the mists
of time in the working group, Brad Templeton used to argue for disallowing
it.  It turns out, however, to be much easier to write a multi-function
news server if one just always generates Xref headers, and Netnews has
functioned in that environment for over ten years.  Everything is now used
to that possibility, and I don't see any reason to pick a fight and try to
get software to change over something that's quite minor and unimportant.

> agent to also "be" a serving agent).

I'm happy to consider a minor change wording proposal that rewords that in
some fashion to avoid the apparent contradiction.  I think the existing
wording is clear, but I don't mind tweaking it if other people don't feel
the same.

>    local serving agent, MAY be deleted before relaying.

This, however, is too broken to be used.

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 11 Jan 2007 11:56:13 GMT

>we should get back to the original topic.

But I do not want to confuse the question of "whether a cancel sould be
honoured", which is a political/policy decision, with the purely technical
question of where and how you actually act upon it once the policy is
decided.

For sure, the primary intent of an honoured cancel is that readers no
longer get to see it, which means that serving agents need to take
appropriate action (so we say "SHOULD", both in 5.3 and 3.6).

But for relaying agents, it is really an optional extra. Some relaying
agents have found it useful to take action so as to reduce the amount of
spam clogging the system, but that action is not _necessary_ from the POV
of achieving that primary intent. I think this is a useful distinction to
make, hence my suggestion to use "MAY" in the case of relaying agents.

---

From: Russ Allbery <[email-masked]>
Date: Thu, 11 Jan 2007 06:23:05 -0800

> But for relaying agents, it is really an optional extra.

It's really an optional extra for relaying agents to honor cancels at all,
since as you point out, it may or may not be that useful depending on the
configuration of the relaying agent and how fast it propagates messages to
its peers.

*If* that relaying agent chooses to honor cancels, which could be
considered an optional extra for relaying agents, *then* it SHOULD act on
them just like any other agent honoring cancels.

I see no reason to increase the complexity of effect of cancel messages by
making it different for different agents when cancels semantically mean
the same thing to all agents and we already provide for agents choosing
not to honor them.  We already (IMO unecessarily) single out serving
agents, and *ideally* I'd rather not even do that, but I can see where
serving agents are the most common and significant case and could stand
explicit mention.  But let's not go farther down that path.  Any agent
(except possibly a posting agent) can usefully choose to honor a cancel
and it means the same thing to any of them, namely "please withdraw the
named article from circulation and access, including proactively if you
haven't seen it yet."

So, opposed, and I think it's unlikely I'm going to change my mind.

If you still think a change is necessary, you should ask to open an issue
and see if other members of the working group support you.

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 11 Jan 2007 12:21:21 GMT

>agents because they both have overview and nnrpd disabled.

Ah! I had not supposed that an Xref header would ever be constructed in
that situation (where does it get its article number from?). It is clearly
unnecessary, and indeed harmless, but anyway justifies a "MAY create it".

>existing Xref header and add a new one.

Sure, Xrefs put there by another system are harmless, even though
meaningless to other systems. Deleting them is probably a good idea, to
remove clutter and possible confusion, but MAY is quite strong enough for
that.

>wording is clear, but I don't mind tweaking it if other people don't feel

If the Xrefs in question are not being added by genuine serving agents,
then it is probably better to omit all mention of serving agents at this
point.

   8.  It MAY delete any Xref header field already present and MAY add a new
       Xref header field for its own use.

s/use/convenience/ if you like, to forestall people who might ask "what
possible use might there be?"

That takes care of the argument "but according to the protocol there is no
possible way for there to be an already present Xref to be deleted",
because the wording has now provided one ("an earlier relaying agent did
it" - and who cares if it was actually an earlier serving agent by some
quirk of an actual implementation).

---

From: Russ Allbery <[email-masked]>
Date: Thu, 11 Jan 2007 13:15:31 -0800

> possible use might there be?"

I think I added the other bit based on WG discussion, but I don't remember
for sure.  I'm happy with taking it out again (or maybe for the first time
if my memory is failing me), but marking this as a Minor change so that
people have a chance to object.

If there are no objections, I'll make this change on Sunday.

---

From: Seth Breidbart <[email-masked]>
Date: Thu, 11 Jan 2007 16:52:23 -0500 (EST)

> people have a chance to object.

Why not just

    8. It MAY delete any Xref header field already present and MAY add
       a new Xref header field.

There's no reason to specify further with a MAY.

Seth

---

From: Russ Allbery <[email-masked]>
Date: Thu, 11 Jan 2007 14:37:19 -0800

> There's no reason to specify further with a MAY.

I'm fine with that too.  Charles, what do you think?

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 12 Jan 2007 00:51:03 +0100

>     8. It MAY delete any Xref header field already present and MAY add

At some point we've to make sure that there's at most one Xref header
field.  How about s/and MAY/and then/ ?  USEFOR states in chapter 3:

Each of these header fields may occur at most once in a news article.

Frank

---

From: Seth Breidbart <[email-masked]>
Date: Thu, 11 Jan 2007 19:11:36 -0500 (EST)

> Each of these header fields may occur at most once in a news article.

That wouldn't allow creating one if there isn't already one to be
removed (unless you define "delete any Xref header field already
present" as "do nothing" when there's no such header).

Simplest would be just to add that it can't add one if one is already
and still present.

Seth

---

From: Russ Allbery <[email-masked]>
Date: Thu, 11 Jan 2007 16:58:14 -0800

> Simplest would be just to add that it can't add one if one is already

How about:

    8. It MAY delete any Xref header field already present and MAY add a
       new Xref header field provided that the resulting article has at
       most one Xref header field.

Wordy, but all the other wordings I came up with were even more wordy.

Currently, we have:

   7.  It MUST (except when specially configured to preserve the
       <article-locator>s set by the sending site) remove any Xref
       header field from each article.  It then MAY (and usually will)
       generate a fresh Xref header field.

for serving agents.  To clear up the same issue and provide consistent
wording, how about:

   7.  It MUST (except when specially configured to preserve the
       <article-locator>s set by the sending site) remove any Xref
       header field from each article.  It then MAY (and usually will)
       add a new Xref header field provided that the resulting article
       has at most one Xref header field.

Since we allow specially-configured serving agents to not remove the Xref
header field, we have to say the same thing here if we want to make sure
this is clear.

(I would say that it's safe to just defer to USEFOR here on the general
principle that of course agents aren't allowed to create invalid articles,
except that exactly the problem of generating multiple Xref headers has
been seen with poorly written news servers in the wild and is therefore
probably worth a specific mention.)

---

From: Seth Breidbart <[email-masked]>
Date: Thu, 11 Jan 2007 22:15:03 -0500 (EST)

Russ Allbery <[email-masked]> wrote:

>       <article-locator>s set by the sending site)

Isn't that again the definition of SHOULD?

7. It SHOULD remove any Xref header field from each article.  It MAY
   add a new Xref header field provided that the resulting article has
   at most on Xref header field.

I don't know about "(and usually will)"; non-crossposted articles
don't need them (which doesn't mean that some software doesn't find it
easier to generate them for all articles anyway).  I suspect that some
(most?) servers always will, and some servers usually won't.

Seth

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 12 Jan 2007 04:21:49 +0100

>> present" as "do nothing" when there's no such header).

Yes, of course they can't remove something that's not there... :-)
But Seth's right, my simple "and then" proposal also wasn't clear.

> Wordy, but all the other wordings I came up with were even more wordy.

Yes, that will do.

> Since we allow specially-configured serving agents to not remove the 
> Xref header field, we have to say the same thing here if we want to
> make sure this is clear.

Okay - took me some time to get it, but now I think it's really better.

Frank

---

From: Russ Allbery <[email-masked]>
Date: Thu, 11 Jan 2007 19:31:36 -0800

> Isn't that again the definition of SHOULD?

No.  There is only one specific exception allowed, not any exception that
the implementor feels is warranted.  I don't believe that should be
weakened.

> I don't know about "(and usually will)"; non-crossposted articles don't
> need them (which doesn't mean that some software doesn't find it easier
> to generate them for all articles anyway).  I suspect that some (most?) 
> servers always will, and some servers usually won't.

Does any modern server not always generate them?

---

From: Seth Breidbart <[email-masked]>
Date: Thu, 11 Jan 2007 22:37:56 -0500 (EST)

> the implementor feels is warranted.  I don't believe that should be

"It MUST delete it unless it's configured to keep it."  I don't see
any strength difference.  (Does requiring "special" configuring mean
anything, compared with, say, "general" configuring?)

Besides, nothing says it can't delete it and replace it with an
identical copy.  The difference between that and not deleting it at
all is rather hard to detect.

Seth

---

From: Russ Allbery <[email-masked]>
Date: Thu, 11 Jan 2007 19:47:20 -0800

>> No.  There is only one specific exception allowed, not any exception
>> that the implementor feels is warranted.  I don't believe that should
>> be weakened.

> "It MUST delete it unless it's configured to keep it."

It's saying more than that.  It's talking about preserving the article
locators from a particular sending site.  The one case where keeping the
Xref header is permitted is when the server is configured to do Xref
slaving to another site.  We're trying to avoid getting into the details
of Xref slaving, but we're trying to capture the idea that this has to be
an explicit configuration choice and should not be the default behavior of
a server because it requires a specific setup to work properly.  It will,
for instance, break if the serving agent receives articles in one group
from more than one source, or doesn't carry exactly the same newsgroups as
the sending server.

> I don't see any strength difference.  (Does requiring "special"
> configuring mean anything, compared with, say, "general" configuring?)

I think so.

> identical copy.

It would violate a SHOULD in Section 3.2, end of first paragraph, but more
than that, that's a bizarre way of describing an action that is a commonly
supported feature of considerable utility in very specific situations.  I
don't want to go through those sorts of unnatural contortions to avoid
explicitly mentioning such an option.

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 12 Jan 2007 21:56:51 GMT

>       most one Xref header field.

     8. It MAY delete any Xref header field already present. It MAY add a
	new Xref header field [for its own use/convenience] (but recall
        that [USEFOR] permits at most one such field).

{observe I still prefer to include "for its own ...", as exaplained to
Seth.}

>       has at most one Xref header field.

You could fix this the same way as above.

I agree that the first part of that is correct. The "(except..." is a
special case for sites which need to keep article numbers synchronized
between several serving agents (it might be reworded to make the reason
clearer, but we don't want to enter into a long discussion here).

And I agree that normal behaviour should be for serving agents to generate
an Xref, even where there is only one article in the Newsgroups header.
But this is really a private matter between the serving agents and its
clients, and I think the "MAY (and uaually will)" covers all that pretty
well.

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 12 Jan 2007 21:32:57 GMT

>There's no reason to specify further with a MAY.

The point is that for a relaying agent to add an Xref serves no useful
purpose for the protocol, and it only happens (as Russ has now pointed
out) because INN finds it less bother to add it that to spot the
situations where it didn't need to. It is, of course, harmless, but of no
use whatsoever to subsequent agents.

Since I was confused as to why this unnecessary provision had been made,
it follows that other readers may be confused too; hence the suggestion of
some wording such as "for its own use/convenience" just to de-confuse
them, by indicating that it is just a private matter for that relaying
agent.

Of course, for serving agents, things are very different, because their
clients want to see Xrefs to prevent showing the same article in different
groups, and maybe for other reasons.

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 12 Jan 2007 21:13:59 GMT

>configuration of the relaying agent and how fast it propagates messages to

Well we agree at least to the extent that relaying agents are less likely
to honour cancels than serving agents. I think that point is worth
mentioning, so would you accept a NOTE, perhaps somethng like this,
probably in section 5.3?

NOTE: It is less usual for cancels to be honored by relaying agents, since
it is serving agents that must ultimately decide whether their clients
should see the affected articles. It may, however, sometimes be useful to
reduce the load and expedite propagation of other articles.

That wording could probably be improved.

---

From: Russ Allbery <[email-masked]>
Date: Fri, 12 Jan 2007 14:25:35 -0800

Charles Lindsey <[email-masked]> writes:

> Well we agree at least to the extent that relaying agents are less
> likely to honour cancels than serving agents. I think that point is
> worth mentioning, so would you accept a NOTE, perhaps somethng like
> this, probably in section 5.3?

> NOTE: It is less usual for cancels to be honored by relaying agents,
> since it is serving agents that must ultimately decide whether their
> clients should see the affected articles. It may, however, sometimes be
> useful to reduce the load and expedite propagation of other articles.

> That wording could probably be improved.

This feels like an implementation decision rather than a protocol issue to
me.  I can see the appeal of trying to distill these various interesting
points we've discussed on the mailing list into a document so that people
can understand more how Netnews software works, but the standards-track
protocol specification doesn't feel to me to be the right place to do
that.

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Fri, 12 Jan 2007 23:35:23 -0500

> protocol specification doesn't feel to me to be the right place to do
<METOO>
I oppose adding any note about this to USEPRO.  I do not expect USEPRO to 
anticipate and explain all the ways Netnews software might work.  Maybe
USEAGE could be the place for such a note.
</METOO>

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 13 Jan 2007 18:26:04 +0100

Seth Breidbart wrote:

>> It MUST (except when specially configured to preserve the
 
> Isn't that again the definition of SHOULD?

No, it's better.  With a SHOULD you need a good excuse to
violate it, and ideally there's a list of common excuses
(like "old app" etc.).  In the end you decide what's good.

With "MUST except" only the listed exceptions are allowed,
a proper subset of the SHOULD definition.

> I don't know about "(and usually will)"; non-crossposted
> articles don't need them

Still nice if you want to create nntp-URLs.  On GMane it's
good to know the article number for various services like
spam reporting, get "raw" article with HTTP, and so on.

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 13 Jan 2007 18:36:03 +0100

Forrest J. Cavalier III wrote:

> I oppose adding any note about this to USEPRO.  I do not expect
> USEPRO to anticipate and explain all the ways Netnews software
> might work.  Maybe USEAGE could be the place for such a note.

USEAGE is for users and UAs, not for server admins and implementors.

For USEPRO it must be clear that not honouring a Cancel is one
thing, and refusing to relay it another mostly unrelated thing.

Frank

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Sat, 13 Jan 2007 14:56:19 -0500

> USEAGE is for users and UAs, not for server admins and implementors.

You have repeated this recently, but I think it is incorrect.  I
read the draft abstract of USEAGE quite differently than you do.
Do you have a basis for your statement?

Some sections already in the USEAGE draft cover the implementation
and operation of servers.  How is it not for server admins and
implementors?

It seems the particpants of the working group disagree on how the
documents are to be split.

Can the chair(s) please state something?  It has been quite a while.

---

From: Russ Allbery <[email-masked]>
Date: Sat, 13 Jan 2007 13:07:26 -0800

> {observe I still prefer to include "for its own ...", as exaplained to

Works for me.  Anyone have any objections?

   7.  It MUST (except when specially configured to preserve the
       <article-locator>s set by the sending site) remove any Xref
       header field from each article.  It then MAY (and usually will)
       add a new Xref header field (but recall that [USEFOR] permits at
       most one such field).

I'll commit this tomorrow unless anyone has objections.

---

From: Frank Ellermann <[email-masked]>
Date: Sun, 14 Jan 2007 00:52:18 +0100

> You have repeated this recently, but I think it is incorrect.

Then it's time to update the WG Charter.  Sam wrote that he'd clear
his [DISCUSS], apparently he forgot to actually do this.  If -12 is
also good enough for Russ USEFOR should get its approval really soon
now, and we can tackle the WG Charter based on an approved RFC   

> Do you have a basis for your statement?

http://article.gmane.org/gmane.ietf.usenet.format/24423/match=useage
http://article.gmane.org/gmane.ietf.usenet.format/26531/match=useage
http://article.gmane.org/gmane.ietf.usenet.format/26684/match=useage

Stuff related to body conventions, GNKSA, netiquette, attribution
lines, and what else.  Not very technical, not standards track.

> Can the chair(s) please state something?

Checking some old articles posted by Alexey he apparently preferred
to move all "policy" and "user interface" issues to USEAGE, and he
also wanted the WG to decide what's eventually going where.

Whatever we decide, we shouldn't abuse USEAGE as dumping ground for
controversial USEPRO issues.

Frank

---

From: Harald Alvestrand <[email-masked]>
Date: Sun, 14 Jan 2007 11:57:21 +0100

> That wording could probably be improved.
We can live without that wording.

As Russ says, it adds complexity, and makes no difference in what is 
permitted or required.

I say (technical contributor hat) "drop it".

---

From: Harald Alvestrand <[email-masked]>
Date: Sun, 14 Jan 2007 12:09:04 +0100

> http://article.gmane.org/gmane.ietf.usenet.format/24423/match=useage
This was Charles' first attempt at split, 2003.
> http://article.gmane.org/gmane.ietf.usenet.format/26531/match=useage
This is Alexey talking about the difference between Informational and 
BCP, 2004. He was wrong on some properties of Informational (a LOT of 
info is WG product), but I think our current goal is BCP.
> http://article.gmane.org/gmane.ietf.usenet.format/26684/match=useage
Alexey doesn't say what goes where here.
> also wanted the WG to decide what's eventually going where.
This doesn't mean that USEAGE has nothing outside these 2 groups.
> controversial USEPRO issues.
I think (chair hat on) that if things can be done 2 ways, both of which 
are valid according to the protocol, and neither of which harms 
interoperability, but the WG thinks that one of the ways is better for 
the users in many common cases than the other, and the world needs to be 
told that, that thing belongs in USEAGE if it is to be stated by the WG 
at all.

Of recent discussions, I suspect that most words about mangling of 
messages to moderated newsgroups by the moderator is the clearest 
example of such an issue.

                          Harald

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 15 Jan 2007 00:59:32 +0100

> I say (technical contributor hat) "drop it".

Please post a picture with that hat :-)  How about this note:

| If server admins decide that they don't wish to honour a
| Cancel locally (in the role as serving agent) they would
| typically still relay the Cancel to their peers.

Are "don't relay => reject" and "reject => don't honour" 
obvious, or does this need MUSTard ?  It could be confusing
if they honour the Cancel, but don't relay it - unless it's
the effect of a corresponding Distribution header field.

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 15 Jan 2007 01:19:10 +0100

Harald Alvestrand wrote:

> if things can be done 2 ways, both of which are valid according to the
> protocol, and neither of which harms interoperability, but the WG thinks
> that one of the ways is better for the users in many common cases than
> the other, and the world needs to be told that, that thing belongs in
> USEAGE if it is to be stated by the WG at all.

Okay, that would also fall under the "policy" category.  But for some
cases it's not absolutely clear what "valid according to the protocol"
is.  If some server admins think that the status of a WG is moderated,
and others think it's unmoderated, maybe because one group missed the
status change, or even intentionally doesn't support it, it's messy.

The protocol should at least offer a default way (= news@) to fix the
simple case "missed the status change".  It took the moderators of a
German NG months to convince Google that the NG is now moderated.  This
is a really bad situation - in that case for the users trying to post
via Google - and I'm talking about a "newusers questions" group... :-(

> example of such an issue.

If the "moderated" concept is too unclear to work in some predictable
way getting rid of it completely would simplify USEPRO and USEAGE.

Frank

---

From: Russ Allbery <[email-masked]>
Date: Sun, 14 Jan 2007 16:47:55 -0800

Frank Ellermann <[email-masked]> writes:

> German NG months to convince Google that the NG is now moderated.

The protocol does.  It's called a newgroup control message, which can be
sent multiple times and which clearly states the moderation status at a
protocol level.  If a given site declines to honor such control messages,
I don't see what standardizing a contact address will do to help.
(Particularly since Google *has* such a contact address, and it never
helps with getting them to fix such things.)

---

From: Russ Allbery <[email-masked]>
Date: Sun, 14 Jan 2007 16:49:45 -0800

> I'll commit this tomorrow unless anyone has objections.

Applied.  I changed field to header field in the parenthetical for
consistency with the wording used elsewhere.

---

From: Richard Clayton <[email-masked]>
Date: Mon, 15 Jan 2007 01:01:41 +0000

In message <[email-masked]>, Frank Ellermann
<[email-masked]> writes

>If some server admins think that the status of a WG is moderated,
>status change, 

it doesn't matter very much until they accept a posted article

>or even intentionally doesn't support it, it's messy.

if it's intentional, then all the email in the world won't help

>simple case "missed the status change". 

Why not expect to use the email address in the Injection-Info ? rather
than overloading the content of path fields -- which are there for very
different reasons  (phx.gbl is rather common, for example) and don't
otherwise need to work for email at all

> It took the moderators of a

I don't think putting requirements to support the address
[email-masked]  into the document will make a lot
of difference to that :(

- -- 
richard                                              Richard Clayton

They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety.         Benjamin Franklin

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 15 Jan 2007 02:49:03 +0100

Russ Allbery wrote:

> It's called a newgroup control message, which can be sent multiple times
> and which clearly states the moderation status at a protocol level.

If a site ignores the newgroup updating the status for unknown reasons
sending it again and again is no plan.  Something has to be fixed manually.

> I don't see what standardizing a contact address will do to help.

It's the same idea as for DNS (hostmaster), SMTP (postmaster), ftp (ftp),
http (webmaster), and more.  It could help.  Especially for underspecified
features like moderated newsgroups.

If USEPRO doesn't update RFC 2142 then 2142 is still "officially" state of
the art wrt NNTP, that can't be what TINW want.

Frank

---

From: Russ Allbery <[email-masked]>
Date: Sun, 14 Jan 2007 18:42:38 -0800

> sending it again and again is no plan.  Something has to be fixed

Which is, hence, outside the scope of the *protocol* and in the arena of
individual choices about configuration.  Marking a group as unmoderated is
not a protocol error.  It is a free choice of the local administrator.
Others are free to argue with them if they believe that choice to be
erroneous.

> It's the same idea as for DNS (hostmaster), SMTP (postmaster), ftp
> (ftp), http (webmaster), and more.  It could help.  Especially for
> underspecified features like moderated newsgroups.

I don't believe that moderated newsgroups are underspecified.  There's
enough information in our current document to implement moderated
newsgroups in an interoperable fashion.  They're just not specified the
way that you would like them to work, with strict limitations on what the
moderator is allowed to do.  Such limitations aren't specified for mailing
lists either.

We could have a long argument about what moderated newsgroups should be
like in an ideal world (and indeed keep having that argument even though
it's pointless), but we have essentially *no* enforcement ability here.
We can specify how moderated newsgroups actually behave (which is largely
at the discretion of the moderator), or we can make something up about how
you would like them to work and have the resulting standard be useless for
implementors because, in the real world, they wouldn't work that way.

> If USEPRO doesn't update RFC 2142 then 2142 is still "officially" state
> of the art wrt NNTP, that can't be what TINW want.

I don't think that's how standards work.  Nothing in NNTP refers to RFC
2142 and compliance with it is not a requirement for NNTP.  It looks to me
like sites can choose whether they wish to comply with RFC 2142 or not
independent of whether they choose to implement NNTP, and I don't see any
serious problems with the current situation that we need to address.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 15 Jan 2007 17:48:15 GMT

>spam reporting, get "raw" article with HTTP, and so on.

And another reason why all articles should have an Xref, whether
crossposted or not, is that reading agents usually maintain a ".newsrc"
file, in a widely used format that relies heavily on article numbers.

So reading agents will expect to have access to these numbers (whether
from the article itself or from the overview).

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 15 Jan 2007 17:44:08 GMT

>protocol specification doesn't feel to me to be the right place to do

The general point at issue here is the extent to which a standards-track
protocol specification should include NOTEs (or other parenthetic remarks)
to explain the reasoning behind requirements (or omission thereof) whose
reasons might not be immediately apparent to the reader (to the "less than
omniscient reader" as Henry Spencer so eloquently put it).

It is sadly true that implementors tend to ignore requirements whose
purpose they do not fully comprehend (and that accounts for a lot or
broken implementations of all sorts of protocols). Standards writers need
to be aware of this and to write documents that not only give the correct
requirements, but are also written with an understanding of how people
might misread them (lawmakers could also learn that lesson too :-( ).

How do you know what is going to be midunderstood? Difficult, but if some
member of this WG misreads/misunderstands some part of the spec, then that
is prima facie evidence that other future readers might well do the same.

For sure, many such cases can be covered by directing attention to USEAGE.
It depends how serious the consequences of the misunderstanding might be.
Whether this particular matter is a case in point is a matter for debate,
but you cannot, in general, turn down all such requests for clarifications
on the grounds that "this document just has to describe the protocol".

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 15 Jan 2007 19:20:46 GMT

>otherwise need to work for email at all

Not really. That "complaints-to" address is intended to reach the 'abuse'
desk, and abuse-desks are notorious for being clueless ignorebots, and are
certainly not the place to complain about configuration issues.

So yes, the "news@" (or "usenet@") address is the proper place to draw
attention to such things, and should hopefully lead to the admins who
maintain the site (who are also often noted for cluelessness, but not to
the same extent as "abuse@").

But the last time the WG discussed this, we agreed not to make any hard
connection between <path-identity>s and "news@". RFC 2142 tried to do
that, and failed. USEAGE might try to fix it (it is still a Good Idea for
those addresses to exist, and it would be in order to USEPRO to NOTE the
need and direct the reader to USEAGE).

>[email-masked]  into the document will make a lot

The Ultimate Sanction, of course, is the UDP :-) .

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 15 Jan 2007 17:25:30 GMT

>thing, and refusing to relay it another mostly unrelated thing.

Hey! I think you're attacking the wrong target. Nobody is suggesting
refusing to relay a cancel message. The protocol clearly intends that
cancel messages should progagate just like other articles, so that
everybody gets to see them and can decide for themselves whether to honour
them.

What is being discussed is whether is is intended/sensible/whatever for
relayers to refuse to relay the cancelled article itself, on the grounds
that they have seen and chosen to honour the cancel message.

My view us that they should not be doing that (let serving agents decide
for themselves whether to honour the cancel) and I was not aware that it
was an allowable practice, and hence it never got mentioned in Usepro-06.

Then Russ pointed out that a few relaying agents _did_ do that in order to
avoid the load of propagating vast quantities of spam (of which there
seems to be less around these days, probably because the spammers find it
more profitable to send spam by email). It is agreed that the practice is
not common (and with current spam levels probably unnecessary).

So all that we are discussing now is whether to include a NOTE pointing
this out.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 15 Jan 2007 19:11:09 GMT

>status change, or even intentionally doesn't support it, it's messy.

I think it is clear from USEPRO that a group is evidently MEANT to be
moderated everywhere, or nowhere. But the USEPRO provides no mechanism to
enforce this (though Usepro-06 did at least allude to the existence of
hierarchy administrators to be in charge of such things). But I think we
are agreed that enforcement is not for this document, and that there are
going to be individual maverick sites who deliberately choose to 'go it
alone'. Even USEAGE can do no more than point out that the problem exists,
and that its only solution lies in achieving a concensus amongst admins
and users. At least USEPRO encourages other relaying sites not to
propagate articles in moderated groups without an Approved header.

>German NG months to convince Google that the NG is now moderated.

Indeed. The larger the organization, the larger the cluelessness :-( . But
"news@" is no magic solution, and the WG has already decided not to
standardize that. So it's no use complaining here - you have to persuade
the chairs to allow the issue to be reopened.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 15 Jan 2007 18:04:55 GMT

>Then it's time to update the WG Charter.

Please! No new charter! We have enough to keep us busy without getting
bogged down in Charter Wars.

But Forrest is right here. USEAGE is largely based on GNKSA, and the GNKSA
was very much concerned with improving the quality of news servers.

Essentially, USEAGE is concerned with establishing "Best Practice" in
areas where the protocol standard permits things which, whilst not causing
things to break, could severely disrupt the smooth operation of Usenet.

>lines, and what else.  Not very technical, not standards track.

And a lot of stuff relating to cancels (e.g. 1st, 2nd and 3rd party) and
other stuff related to news servers. What is expected of hierarchy admins.
Moderation. And lots more.

>also wanted the WG to decide what's eventually going where.

Yes, but that does not mean that things not related to "policy" and "user
interface" don't belong there.

>controversial USEPRO issues.

I agree with that, and I do discern a slight tendency to push too much
into USEAGE. Each case on its merits. USEPRO must work as a standalone
document (you cannot force implementors to read USEAGE). If a Martian
landed in his spaceship and all he had available was the set of
standards-track RFCs (and a Babefish in his ear so he could understand
English), then he ought to be able to write an implementation that would
interoperate successfully with the rest of Usenet.

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 16 Jan 2007 19:40:16 +0100

Charles Lindsey wrote:
 
> Forrest is right here. USEAGE is largely based on GNKSA, and the GNKSA

IIRC the N in GNKSA was for "newsreader" (UA), not for "news server".
All I ever did with it was to evaluate my UA.  Based on that I consider
USEAGE as a kind of Emily Postnews for geeks.

Nobody's forced to follow RFCs, Russ' "we can't enforce this" line of
arguments isn't compelling.  He can say "I can't implement requirement
xy, the xy part of the protocol is typically handled by a human and not
by a piece of software".  That's not the same as "necessarily no part
of the protocol" (if the system can't work as expected without xy).

> If a Martian landed in his spaceship and all he had available was the
> set of standards-track RFCs (and a Babefish in his ear so he could
> understand English), then he ought to be able to write an 
> implementation that would interoperate successfully with the rest of
> Usenet.

Including simple modbots as far as I'm concerned, supporting policies
considered as good way to approve articles automatically by Martians.

Frank

---

From: Russ Allbery <[email-masked]>
Date: Tue, 16 Jan 2007 10:51:56 -0800

Frank Ellermann <[email-masked]> writes:

> by a piece of software".  That's not the same as "necessarily no part of
> the protocol" (if the system can't work as expected without xy).

You're misportraying my argument.  My point is that we need to not say
things that are directly in contradiction with reality unless we can
reasonably expect reality to change.  Otherwise, our document is not
useful for writing an interoperable Netnews implementation, thus defeating
the entire point of this exercise.

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 16 Jan 2007 19:59:40 +0100

> that they have seen and chosen to honour the cancel message.

s-o-1036 has a long explanation how to handle cancels, and it proposes
"reject and discrad" after "not arrived yet".  But it's apparently based
on a "MUST honour cancel" rule.

> So all that we are discussing now is whether to include a NOTE pointing

Of course differences from s-o-1036 need explicit justifications.  It
has
an obscure newsmaster@, so now it's three together with usenet@ and
news@.
Is that finally bad enough to get rid of this chaos once and for all ?

Frank

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Tue, 16 Jan 2007 14:10:38 -0500

There is no concise way to describe a "proper" re-injection.  There
are too many special cases and exceptions.  The role of such software
is not clear.  Is it a relay?  Posting agent?  Gateway?

I find 3.3.2 poorly written and redundant.

I consider the mention of "special exceptions" an attractive nuisance.
An implementor looking for justification will always consider themselves
"special."   That's just psychology.

I think mentioning reinjection will entice novices to try something that only 
experts can be expected to do without causing trouble.

I have annotated the draft 3.3.2 below, but the end result is that would
replace 3.3.2 with....

3.3.2 Reinjection of Articles

    Converting an already injected article into a proto-article and forwarding
    it to an injecting agent, "reinjecting", causes problems such as loops,
    loss of trace information, and re-appearance of expired articles, unless
    handled very carefully.

    A posting agent that does reinjection is a limited type of gateway and is
    also subject to all of the requirements of an incoming gateway.  If the
    header must be altered before forwarding to an injecting agent,
    renaming header fields is preferred over removing.  Checks using those
    header fields (that would otherwise be done by the injecting agent) MUST be
    accomplished before the alteration.

Annotating the draft, here is how and why I would make changes.

3.3.2.  Reinjection of Articles

    A given article SHOULD be processed by an injecting agent once and
    only once.

[Recommend: moving to Duties of an injecting agent.]

    The Injection-Date or Injection-Info header fields are
    added by an injecting agent and are not permitted in a proto-article.
[Repeats something from 3.3.1, which repeats something in 3.3 para 2.
Strike it, and remove the repeat in 3.3.1.]

    Their presence (or the presence of other unstandardized or obsolete
    trace headers such as NNTP-Posting-Host, NNTP-Posting-Date, or
    X-Trace) indicates that the proto-article is instead an article and
    has already been processed by an injecting agent.

[Delete it or Move it to Duties of an injecting agent.]

   A posting agent
    SHOULD normally reject such articles.

[How does a posting agent get to reject an article?  The Posting
agent is what is creating a valid proto-article, is it not?  An
injecting agent already MUST reject such articles, as I read the draft.
Strike it.]

    In the exceptional case that an article needs to be reinjected for
    some reason (such as transferring an article from one Netnews to
    another where those networks have no relaying agreement), the posting
    agent doing the reinjection MUST convert the article back into a
    proto-article before passing it to an injecting agent (such as by
    renaming the Injection-Info and Injection-Date header fields and
    removing any Xref header field) and MUST perform the date checks on
    the existing Injection-Date or Date header fields that would
    otherwise be done by the injecting agent.

[I would say strike it all, but the advice to do the date checks and
rename is possibly useful, if it appears at the end of the next paragraph...]

    Reinjecting articles may cause loops, loss of trace information, and
    other problems and should only be done with care and when there is no
    available alternative.  A posting agent that does reinjection is a
    limited type of gateway and as such is subject to all of the
    requirements of an incoming gateway in addition to the requirements
    of a posting agent.

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 17 Jan 2007 11:07:06 GMT

>USEAGE as a kind of Emily Postnews for geeks.

No, the "GN" stood for "Good-Netkeeping", and was a play on the "Good
Housekeeping Seal of Approval" published (maybe it still is) by Good
Housekeeping magazine.

But yes, now I have re-read it, it was primarily concerned with user
agents.

But USEAGE was always intended to deal with all agents, and much of it
consists of texts pulled from the earlier 'article' series of drafts after
we had decided to make that split.

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 17 Jan 2007 11:18:23 GMT

>I find 3.3.2 poorly written and redundant.

I tend to agree, but am not sure your rewrite is correct either. I think
we really need to resolve the Injection-Date Issue (#1416), and then come
back to exactly what we say here.

>"special."   That's just psychology.

Which is why our drafts sometimes needs to say "Don't try to do this at
home, kiddies", but of course the problem is to find a polite way of
saying that. Essentially, the draft needs to be written so that not only
does it specify the correct tecnical details, but also so that it gives the
correct _impression_ of how things are _intended_ to work

>experts can be expected to do without causing trouble.

Indeed. Reinjection needs to happen in a few cases which are hard to
specify precisely in advance. But the impression to be given is definitely
that "you would recognize those cases when you see them, but it takes
great knowledge and skill to recognize them", though of course you can't
word it like that. But saying less rather than more is probably a move in
that direction.

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 19 Jan 2007 01:51:04 +0100

Russ Allbery wrote:
 
> My point is that we need to not say things that are directly in
> contradiction with reality unless we can reasonably expect reality
> to change.

If the reality is just messy we should try to get it right.  That can
be very harmless, like specifying ONE admin address, or outline ONE
typical procedure for moderators, where we trust that it shouldn't
cause havoc if folks actually adopt it.

"Reasonably expect reality to change" is far stronger, and for the
path diagnosis I wouldn't bet that it flies.  At least it's better
than conflicting inventions for this fifth wheel.

In an unrelated article Forest wrote: 
| Feelings are a fine place to start, but we aren't writing a novel.

If my gut feeling is that moderated articles should always work as
expected if moderators stay away from modifying the body and the
Message-ID it's not meant as novel.  My maybe naive concept of how
I'd write a mod-cancel-bot was really "note approved Message-IDs".

Frank

---

From: Russ Allbery <[email-masked]>
Date: Thu, 18 Jan 2007 17:09:02 -0800

>> contradiction with reality unless we can reasonably expect reality to
>> change.

> If the reality is just messy we should try to get it right.  That can be
> very harmless, like specifying ONE admin address, or outline ONE typical
> procedure for moderators, where we trust that it shouldn't cause havoc
> if folks actually adopt it.

I believe that we have already done this for moderated groups, and that
you are now asking for us to take a step farther beyond that into saying
that all other procedures are wrong.

> "Reasonably expect reality to change" is far stronger, and for the path
> diagnosis I wouldn't bet that it flies.

Which is, as has been previously noted, a strong argument for removing the
path diagnosis.  However, that argument is closed, since it's in USEFOR
and USEFOR is closed.  That doesn't mean it's not a valid argument, or
that it won't apply to other discussions in the future.

---

From: Russ Allbery <[email-masked]>
Date: Thu, 18 Jan 2007 21:01:18 -0800

Forrest J Cavalier <[email-masked]> writes:

> There is no concise way to describe a "proper" re-injection.  There are
> too many special cases and exceptions.  The role of such software is not
> clear.  Is it a relay?  Posting agent?  Gateway?

This is actually exactly the question that I think we can answer, by
saying that it's a gateway.  That's the closest analogy, since a gateway
is already a sort of posting agent.  It's definitely not a relaying agent
(if so, there's no need for reinjection) or an injecting agent (since an
injecting agent could talk to a relaying agent directly and hence preserve
the existing injection fields).  If it were in a position to be either of
those things, the message could just be relayed instead.

> I find 3.3.2 poorly written and redundant.

Quite possible.  :)

> I think mentioning reinjection will entice novices to try something that
> only experts can be expected to do without causing trouble.

A Netnews to Netnews gateway is probably an easier piece of software to
write than a full serving agent.  suck/rpost and newsx aren't horribly
complex.

I generally agree with what you're saying, but this is common enough that
I don't want to remain silent about it.  Plus, there are various ways in
which people do this incorrectly, so we can improve the situation by
explaining clearly how to do it correctly and thus prevent harm.

The goal with the "exceptional" language is to try to point out to people
that there is usually a different, better solution to the problem.  I'd
like to try to say this somehow.  If it's possible to set up a relaying
relationship or to do multiple injection, that should be done instead.
Reinjection should not be the first choice, since it adds complications.

>    Converting an already injected article into a proto-article and
>    forwarding it to an injecting agent, "reinjecting", causes problems
>    such as loops, loss of trace information, and re-appearance of
>    expired articles, unless handled very carefully.

Note that we have a definition of reinjecting in 1.5:

   "Reinjecting" an article is passing an already-injected article to an
   injection agent.

We may want to reconsider this definition as well as part of this
discussion.

I'm worried about this start of the section from an editorial concept,
since it leaps into converting an injected article into a proto-article
without saying how that idea would even come up.  I think it's worthwhile
to say something like:

    Normally, articles are transferred between news servers by relaying
    agents.  To reach news servers not reachable by relaying agents with
    an article, the best approach is to give the identical proto-article
    to injecting agents in each disconnected Netnews network.  If,
    however, neither of these approaches are possible, it may be necessary
    to convert an existing article back to a proto-article so that it can
    be injected into another Netnews network.  This action, called
    reinjection, may cause problems such as loops, loss of trace
    information, and reappearance of expired articles unless handled
    carefully and SHOULD be avoided when possible.

That's a bit of an expansion over what you say and what's already in the
section, but makes the alternatives clearer.  What do people think of that
sort of language?

>    A posting agent that does reinjection is a limited type of gateway
>    and is also subject to all of the requirements of an incoming
>    gateway.  If the header must be altered before forwarding to an
>    injecting agent, renaming header fields is preferred over removing.
>    Checks using those header fields (that would otherwise be done by the
>    injecting agent) MUST be accomplished before the alteration.

I want to explicitly say how you convert an article back to a
proto-article so that it's clear that we're not talking about removing or
changing Message-ID.  So:

    A posting agent that does reinjection is a limited type of gateway and
    is also subject to all of the requirements of an incoming gateway.  It
    MUST also perform the same checks against the existing Injection-Date
    or Date header fields as would be performed by a relaying agent.  It
    then MUST convert the article to a proto-article by renaming or
    removing any existing Injection-Info, Injection-Date, and Xref header
    fields.  Renaming is preferrable in most cases for Injection-Info and
    Injection-Date to preserve trace information.  Such a posting agent
    may need to rename or remove other, non-standard trace headers (such
    as NNTP-Posting-Host or X-Trace) to make the article acceptable to an
    injecting agent.

Note that it's actually the date checks of a *relaying agent* that we
should be performing; the checks of an injecting agent don't take
Injection-Date into account.  I had that wrong in the current text.

I'd agree with a rewritten section including just the two paragraphs I
propose above.  That would contain all the important parts of the current
section, I think.

> [Recommend: moving to Duties of an injecting agent.]

This is only true in the presence of multiple injection if you squint at
it properly.  I'm not sure that this sentence is really adding anything
useful once we explain the normal article propagation rules.

> Strike it, and remove the repeat in 3.3.1.]

This is part of the definition of a proto-article, so the one place it
should be stated is in the proto-article section.  Instead of removing it
from there, I'd change the text in 3.3 para 2 to:

   Posting agents MUST ensure that proto-articles they create are valid
   according to Section 3.3.1, [USEFOR] and any other applicable policies.

I'm not sure why I said SHOULD here originally.  Surely, complying with
the basic requirements of the standard is a MUST.  I guess I said SHOULD
because the primary responsibility lies with an injecting agent, but we
already say that explicitly elsewhere.

---

From: Russ Allbery <[email-masked]>
Date: Thu, 18 Jan 2007 21:27:53 -0800

Sorry, I messed up the Subject header here.  The parent of this article
is also part of this ticket.

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Fri, 19 Jan 2007 02:07:55 -0500

How about standardizing reinjection ONLY for disjoint networks, letting
other uses of reinjection be out of scope?

Taking a shot at that.....

Normally, articles are transferred between news servers by relaying agents.  In 
the case of disjoint Netnews networks, such as a private server with no relaying 
peers, a posting agent SHOULD cause the identical proto-article with Message-Id 
header field to be transmitted to multiple injecting agents, one in each 
disjoint network.

When it cannot be avoided, a posting agent which converts existing articles back 
to proto-articles for reinjection to a disjoint network MUST NOT reinject 
articles already accepted on that network.  It MUST perform the same staleness 
tests of Injection-Date or Date header fields as would be performed by a 
relaying agent.  (This helps prevent reappearance of expired articles.) It MUST 
NOT alter header fields permitted in proto-articles, especially Message-ID. 
(This helps prevent loops.)  To make the article acceptable to an injecting 
agent, it SHOULD rename other header fields to preserve information, but MAY 
remove them.

---

From: Russ Allbery <[email-masked]>
Date: Thu, 18 Jan 2007 23:20:43 -0800

Forrest J Cavalier <[email-masked]> writes:

> agents.  In the case of disjoint Netnews networks, such as a private
> server with no relaying peers, a posting agent SHOULD cause the
> identical proto-article with Message-Id header field to be transmitted
> to multiple injecting agents, one in each disjoint network.

"If one article is to be injected into disjoint Netnews networks, such as
both Usenet and a private server with no relaying peers, a posting agent
..."

Since the normal case for disjoint Netnews networks is that they'll have
disjoint sets of articles as well.  (Also, one doesn't need to limit it to
one in each network necessarily.  I was trying to avoid making that
limitation in my language.)

> When it cannot be avoided, a posting agent which converts existing
> articles back to proto-articles for reinjection to a disjoint network
> MUST NOT reinject articles already accepted on that network.  It MUST
> perform the same staleness tests of Injection-Date or Date header fields
> as would be performed by a relaying agent.  (This helps prevent
> reappearance of expired articles.)  It MUST NOT alter header fields
> permitted in proto-articles, especially Message-ID.  (This helps prevent
> loops.)  To make the article acceptable to an injecting agent, it SHOULD
> rename other header fields to preserve information, but MAY remove them.

Charles has a valid point that it's really beyond the ability of a posting
agent to determine whether two Netnews networks are truly disjoint.

Beyond that, the bit about the Message-ID is a good addition that I want
to keep, regardless of how else we word this.  The gateway stuff already
says this sort of, but we should emphasize it.

We need to say *what* other header fields it will rename.  I think we also
have to allow for drop as well as rename; for example, if I'm posting to a
disconnected INN server and then gatewaying those articles to Usenet, my
local trace information is entirely irrelevant and there's no reason to
retain it.

If this is the complete section, we're not saying that the gatewaying
rules apply anywhere, which I think we still need to do since there may be
bidirectional gatewaying at work (among other things).

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Fri, 19 Jan 2007 09:50:07 -0500

> limitation in my language.)

Simple changes.  Make it conditional... "To cause an article to appear on
disjoint networks a posting agent SHOULD..."

And "s/one/at least one/"

> agent to determine whether two Netnews networks are truly disjoint.

Yes, this point bothered me as I was going to bed.  Relays ought to check Path
and would rewording indicate that?  Would this be adequate.

> local trace information is entirely irrelevant and there's no reason to

Isn't that permitted by the last SHOULD?  Why is the trace information
entirely irrelevant?  It isn't any more irrelevant than opaque trace
information from all the other servers.

> bidirectional gatewaying at work (among other things).

Unnecessary complexity.  Why make people at private leaf nodes consider
the requirements of bidirectional gatewaying, discarding half of those
requirements?

Can't bidirectional gatewaying be discussed under gatewaying, or let
implementors assume it is just two unidirectional reinjections?  The bit about
using the Path header may be important for this case.  What concerns
do you have?

---

From: Russ Allbery <[email-masked]>
Date: Fri, 19 Jan 2007 07:12:10 -0800

>> Since the normal case for disjoint Netnews networks is that they'll
>> have disjoint sets of articles as well.  (Also, one doesn't need to
>> limit it to one in each network necessarily.  I was trying to avoid
>> making that limitation in my language.)

> Simple changes.  Make it conditional... "To cause an article to appear
> on disjoint networks a posting agent SHOULD..."

> And "s/one/at least one/"

Yeah, that fixes my concerns there.

>> Charles has a valid point that it's really beyond the ability of a
>> posting agent to determine whether two Netnews networks are truly
>> disjoint.

> Yes, this point bothered me as I was going to bed.  Relays ought to
> check Path and would rewording indicate that?  Would this be adequate.

What sort of checking of Path do you have in mind?  I can't figure out
exactly what that would entail.

>> We need to say *what* other header fields it will rename.  I think we
>> also have to allow for drop as well as rename; for example, if I'm
>> posting to a disconnected INN server and then gatewaying those articles
>> to Usenet, my local trace information is entirely irrelevant and
>> there's no reason to retain it.

> information from all the other servers.

It often is.  The hostname is localhost, the IP address is 127.0.0.1, etc.
It's allowed by SHOULD, but I don't think we need a SHOULD-level decision
to decide to throw away useless tracking information.  Although I guess I
don't feel strongly about it.

>> rules apply anywhere, which I think we still need to do since there may
>> be bidirectional gatewaying at work (among other things).

> the requirements of bidirectional gatewaying, discarding half of those

Er, I don't understand.  I'm not proposing that one-directional gateways
be subject to the constraints of bidirectional gateways, of course.  I'm
saying that this *is* a gateway and therefore needs to follow the rules of
gateways, including such things as not gatewaying a message that it had
already gatewayed, marking messages it has gatewayed, etc.  This is needed
particularly in the case of bidirectional gatewaying, which you may not
know is happening.  All that is spelled out in the gateway section, and I
don't want to copy it all into this section again.

All those cautions and caveats are applicable here, so I'd like to just
cite that section and be done with it.

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Fri, 19 Jan 2007 12:00:50 -0500

> exactly what that would entail.

 From 3.5 Duties of a Relaying agent.
    In order to avoid unnecessary relaying attempts, an article SHOULD
    NOT be relayed if the <path-identity> of the receiving agent (or some
    known alias thereof) appears as a <path-identity> (excluding within
    the <tail-entry> or following a "POSTED" <diag-keyword>) in its Path
    header field.

(Need to modify it for reinjection.)  An article posted locally on a
private leaf node can easily be distinguished from others received
via suck by checking the Path.

Also I don't know that we need RFC2119 imperatives for the reinjection
case.  There is more than one way of determining that an article was
posted locally.

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Fri, 19 Jan 2007 12:30:02 -0500

> don't feel strongly about it.

Yes, it is often 127.0.0.1, but it is not necessarily 127.0.0.1.  It may
include other information that is not useless. It's opaque information.

A corporate private leaf node site or mom-and-pop-and-mickey-mouse ISP (if there 
are any left) may have enough users internally that they have use for
preservation.

I'm don't feel strongly about this either.

> cite that section and be done with it.

I'm almost OK with that.  The trouble I see is that 3.9 Duties of
Gateways is almost all about mailing lists.  (And most of the trouble is
there too.)  Despite the 3.9 paragraph that says "reinjection is handled
separately below", "proto-article" is mentioned exactly once thereafter,
(just to say it must be valid.)

More seriously The outgoing gateway description in 3.9.1 seems to even 
contradict the RFC2119 language I wrote for the special case (handling local 
posts from a private leaf node.)  I'm not thrilled with the awkwardness and lack 
of clarity in 3.9. I can live with it because it is probably wordy enough that 
clueful people do the right thing when they read it.  I don't count the typical 
  private leaf node operators in that group....not a slur, just a statement of 
skill level.

My idea was to standardize the special case, and then provide enough detail that 
it works.  If people fall outside that special case, then they are gateways.  If
people can fit themselves into the special case, it is better to have the 
RFC2119 imperatives rather than the wishy washy 3.9.1.

If the Path checking and idea is sound, do we really gain anything by having
them keep lists of previously gatewayed articles? Tagging them?  Are those
really PROTOCOL requirements, necessary to get the end result?

I appreciate your concerns that there is lots to consider in the general case. 
Can we "have it both ways" and say, under reinjection....

    "Requirements and precautions to avoid problems are described in detail
    at 3.9 Duties of Gateways."

Does that work?

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 19 Jan 2007 20:52:14 GMT

>is already a sort of posting agent.

But a posting agent is not so obviously a sort of gateway.

I think the answer is that it may be a Posting Agent and it may be a
Gatway. but in fact it will often be a confused situation where the four
parties involved (first injector, second injector and the two injecting
agents) all have a different view of what role they are playing, whether
by accident, by misunderstanding, or by deliberate intent.

>  It's definitely not a relaying agent

True, though one of the parties might believe he was relaying.

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 19 Jan 2007 22:00:21 GMT

>other uses of reinjection be out of scope?

But truly disjoint networks are not the problem.

The poblem is netowrks that are _not_ disjoint (even though somebody
thought they were). You cannot just declare them "out of scope" - you have
to define a protocol that works in spite of them.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 22 Jan 2007 23:27:26 GMT

>information from all the other servers.

Yes, my preference is to preserve all the Path header, even early parts of
it that recorded its progress through some private system before getting
injected into Usenet. It will do litle harm, and may help unravel some
complicated cases where things have gone wrong. That is why Usepro-06 said
SHOULD preserve the old Path during reinjection. Then you get to see the
full story of exactly where it has been.

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Tue, 23 Jan 2007 10:25:01 -0500

> full story of exactly where it has been.

You are comingling two points of argument, I think.  I do not consider
the Path header as trace information.

I don't see any way to preserve Path except by renaming before reinjection. 
Reinjection is a POST through an injecting agent, after all.

Either injecting agents are responsible for adding trustable headers to the 
proto articles (after removing any that are present) or they are not.

That question is already answered: we live with what existing injecting agents do.

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 25 Jan 2007 12:18:34 GMT

>the Path header as trace information.

Eh? I regard Path very much as trace information, our equivalent to the
email Received header (but much less verbose). It is certinaly widely used
for that purpose, for trying to find out where spam and other malpractice
originated - which requires some estimate of which parts of the Path are
bogus, which is the primary reason we have added diagnostic information to
it.

>Reinjection is a POST through an injecting agent, after all.

So, injection inserts a "POSTED" and reinjection inserts a second "POSTED"
which, if the article then appears "iffy" or otherwise spamish, directs
your attention immediately to the point to which your suspicions should be
directed.

>proto articles (after removing any that are present) or they are not.

Destruction of evidence can only hinder forensics.

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Thu, 25 Jan 2007 12:35:57 -0500

> Destruction of evidence can only hinder forensics.

That's not the only goal, especially since forensics is not the primary purpose 
of Path.  (Unlike "Received" headers.)

Do you have any position on the mischief that can be achieved with Path pre-loading?