Subject: Protocol changes in draft-allbery-usefor-usepro-00

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 13 Dec 2006 20:41:13 GMT

This list summarizes the normative changes made between USEPRO and Russ's
document (may we call it "RUSSPRO" for the moment?).

Some are substantial and some trivial; some are obviously right and some
(IMO) obviously wrong; and some may be unintended consequences of the new
wording; many just involve changes between MUST/SHOULD/MAY.

Nevertheless, if the WG decides to go with RUSSPRO, it needs to be aware
of what changes it has made, and to discuss any that are in doubt.

I have indicated my own opinion of each one with a score ([+1], [-1] or
[00]).  Except for item #4, which is important enought for a separate
thread (which I shall hopefully set in motion tomorrow).

1. [-1] (2.2) <path-identity> - Possibility to use a FQDN not resolvable
in the DNS.

(This was a suggestion by Harald, & 1 of 2 alternatives in USEPRO)

2. [00] (3.2) <path-identity>s are case-sensitive.

I was always told that some servers treated them one way and some the
other, so you could not rely on either. If anything, I would have expected
case-insensitive (that is what domain-names are, and what USEFOR declares
<diag-keyword>s are).

3. [-1] (3.3.1,3.4,3.9.2) From not omittable in proto-article

There is existing usage where the injecting agent fills in From header
(not possible in NNTP, of course)

4. [--] (3.2.2,3.4,3.8) Injection-Date can be rewritten.

A major change, which could potentially lead to loops, and has
necessitated reverting to reliance on Date header in many situations. The
whole point of introuducing Injection-Date was to let Date reflect
poster's composition time, even if not injected until weeks later. This is
now lost. I shall raise a separate thread for this issue.

5. [-1] (3.3.4) SHOULD trim References to keep <= 998: reduced from MUST.

This could allow header lines longer than 998 octets, which is known to
severely impact interoperability. It is not disputed that trimming when
within the 998 limit is a MAY.

6. [00] (3.4) Injecting agents reporting rejection to user agent; was
SHOULD, now merely "encouraged".

All it needs is an NNTP 44x response or similar.

7. [+1] (3.4) Injecting agent SHOULD have access to list of newsgroups.

So it can reject if not at least one group in Newgroups header exists This
is an addition to agreed MUST keep list of moderated groups.

8. [-1] (3.4) Requirement for injecting agent to forward articles to
moderator groups reduced from MUST to SHOULD.

If some small latitude is needed for when the moderated group is in a
crosspost to some remote and unknown hierarchy, then this should be stated
explicitly.

9. [-1] (3.4,3.5) Prepending of <path-identity> to Path by injecting agent
reduced from MUST to SHOULD.

I would accept that prepending the <path-diagnostic> POSTED is a SHOULD.

10. [00] (3.4) Folding of Path header when length becomes excessive
reduced from SHOULD to MAY.

11. [-1] (3.4) Recommendation that each injecting agent SHOULD use a
consistent format for Injection-Info removed.

Otherwise, the task of chasing the bad guys becomes harder.

12. [-1] (3.5) The significance of "world" and "local" in the Distribution
header are no longer mentioned.

We recently took care to reserve these in USEFOR, but that requires backup
here to ensure the protocol fulfils that promise.

13. [+1] (3.5,3.6) Removed possibility of "aliassing out" a site where you
didn't want an article to go by including it in the Path to the right of
POSTED.

This is probably an improvement, but it also needs to be considered
alongside other conventions for "aliassing out", such as the 'cybercancel'
convention which is no longer mentioned at all.

14. [+1] (3.5) Relaying agent MUST reject articles without Newgroups or
Message-ID or (injection-Date or Date).

Before, it was SHOULD reject for any missing mandatory header.

15. [-1] (3.5) Relaying agent MUST reject any article already already sent
to it.

This is impossible unless relayer keeps a history file (many don't).
Nothing breaks if you don't do this, and the usual check on the Path
header is normally considered sufficient. I might buy a SHOULD here, with
suitable wording.

16. [-1] (3.5) Relaying agent SHOULD deal with cancel message (if it
chooses to honour it)

Yes, USEPRO says the same, but I suspect both are wrong, because it is
serving/storage agents that should be doing this (all a pure relaying
agent can do is then not to propagate it further).

17. [00] (3.5) Relaying agents SHOULD determine verified/expected source
of article and construct <path-diagnostics> accordingly.

USEPRO offers various MUST/SHOULD choices here for discussion. Earlier
USEPROs had MUST. (It is agreed that the <path-identity> itself is MUST,
in contrast to the case of injecting agents mentioned above.)

18. [-1] (3.5) No provision for >1 <path-identity> per relaying agent.

I think we wrote PATH in USEFOR on the assumption that would be allowed,
and Richard Clayton had an example where he needed a <path-nodot> in
addition to a domain-name. I see you also removed this from the example in
3.5.1.

19. [00] (3.5) No provision to omit <path-identity> for intermediate
servers on same site.

Where a site consists of a large farm of servers (for incoming, outgoing
and whatever else articles) there should be no need to clutter the Path
header with them all (unless the site finds that helpful). USEPRO
therefore relaxed the requirement, provided the outgoing one could be
verified by the next site.

20. [00] (3.5) Relaying agent MAY add a new Xref header.

Why might it want to do that?

21. [00] (3.6) Storage of control messages in the (pseudo) hierarchy
control(.*) is a SHOULD.

Is this any of our business?

21. [-1] (3.6) No mention of processing cancel messages by serving/storage
agents.

This is odd, given that only a serving agent can actually delete an
article or otherwise deactivate it. Also, you need to deal here with the
case where a cancel arrives before the article (it is covered in 5.3, but
it is the serving agent's duty to implement it).

22. [00] (3.7) Removed recommendation that reading agent SHOULD have the
capability to display the raw article 'as received'.

23. [-1] (3.8) Moderators merely "encouraged" (was SHOULD) to retain
existing <msg-id>.

That could cause problems if the original poster later wants to cancel it,
or if he had forwarded it from some other source.

24. [00] (3.2.1) Removed provision (MAY) for outgoing gateway to change
Content-Transfer-Encoding.

Given that all other agents, from injecting agents on, are forbidden to
change the Content-Transfer-Encoding (which will normally be 8bit or 7bit
in Netnews except for genuine binary objects), it seems desirable to
mention the exception here, where it may be necessary before the article
can be passed through some email agents.

25. [+1] (4.1) Application/news-transmission no longer handles batches.

That needs to be mentioned explicitly, as it is a change to an existing
application type.

26. [-1] (4.2,4.3) Removed requirement that
application/news-[groupinfo,checkgroups] MUST NOT be used except with
relevant control messages.

Application types are inherently dangerous if they are executed when they
should not be. Is there any situation where these two could properly be
used outside of their intended control messages? Sure, for informational
purposes, or for passing by private arrangement from one newsadmin to
another, but best leave it to their common sense if they want to break the
rules. In any case, news-checkgroups does not make sense without its
accompanying checkscope parameter.

27. [+1] (4.3) Application/news-[groupinfo,checkgroups] have a charset
parameter.

I think we are now agreed abut this, but with a restriction to having
ASCII as a subset.

27. [-1] (5) Nothing stated regarding Newsgroups header of Control
messages.

Was SHOULD include the groups affected (& possibly others). Without that,
control messages won't propagate to the places they are likely to be
needed.

28. [00] (5) Subjects starting with 'cmsg' not accompanied by a Control
header MAY be rejected.

We discussed this at some stage, but I don't think we ever agreed it one way
or the other.

29. [00] (5.2) Newsgroup-names MUST be checked against disallowed names before
group control message is honoured.

Was SHOULD. That MUST seems a bit excessive for rmgroup and checkgroups.
Shouldn't it apply just to newgroup?

30. [-1] (5.2) Nothing said about content of Approved header.

Surely, it SHOULD identify the person/identity/role of the issuer, and be a
working email address. Even in alt.* "Approved: foo" is not really good
enough. News servers are usually configured with what to expect there for each
hierarchy.

31. [-1] (5.2.1) Newgroup message SHOULD include application/group-info.

Was MUST (though I wouldn't take that to exclude a "For your newsgroups file:"
to be scanned for - if allowing that was the intent of the SHOULD, then some
rewording could cover it). That scanning is currently "MAY" which is possbly
too weak. Or is it trying to allow the case where no description exists for
the group (for which I would still have expected a Newgroups File entry with
just the newsgroup-name, which is how a checkgroups would show it).

32. [-1] (5.2.1) Newgroup message containing other attachments in addition to
news-groupinfo to be multipart/related (was multipart/mixed).

I don't see what the 'relationship' is.

33. [+1] (5.2.3) Checkgroups message still contains chkscope and chksernr
arguments.

No change from USEPRO. I think I finally persuaded Russ this should remain so.

34. [00] (5.2.3) Checkgroups SHOULD contain sub-hierarchies excluded by
chkscope, for backwards compatibility.

Fine in principle, but maybe not implementable. If de.alt.* were as chaotic as
alt.* (is it?) there would be no way the admins of de.* could include a
defnitive list of de.alt.*. So some qualification is needed.

35. [+1] (5.2.3) Chksernr MUST increase with successive checkgroups message.

Was SHOULD.

36. [00] (5.2.3) Body of checkgroups SHOULD be application/checkgroups, and
otherwise SHOULD treat as if it were.

Why does that SHOULD differ from the corresponding MUST in newgroups? I would
have expected "SHOULD ... and otherwise MUST ...".

37. [00] (5.3) Body of cancel message MAY contain ...explanatory text.

Was SHOULD.

38. [-1] (5.3) No mention of who should be in From/Sender of cancel message.

Agreed it is not the person who posted the original article. USEPRO says it
"should" be the person who issued the cancel. That should really be "SHOULD".

39. [-1] (5.5) The <relayer-name> in ihave and sendme control messages is now
optional.

It is required to be present in USEPRO and in Son-of-1036.

40. [-1] (5.5) Format of batched news in response to sendme removed.

Note that this was a format 'on the wire' and was intended to be sent on the
same 'wire' as the ihave and sendme messages (which would usually be a UUCP
connection). Hence it is a legitimate item to standardize. Probably needs some
brief mention of its compressed versions.

41. [-1] (5.5) The protocol for using the 'to.' newsgroup-name is omitted.

Insofar as this protocol may still occasionally be used, it needs to be
documented (as with the batch format).

42. [-1] (5.6) Obsolete control messages SHOULD NOT be sent/honoured/

Was MUST NOT. Cf the MUST NOT for obsolete headers in USEFOR.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Russ Allbery <[email-masked]>
Date: Wed, 13 Dec 2006 13:15:00 -0800

Charles Lindsey <[email-masked]> writes:

> This list summarizes the normative changes made between USEPRO and
> Russ's document (may we call it "RUSSPRO" for the moment?).

> (IMO) obviously wrong; and some may be unintended consequences of the
> new wording; many just involve changes between MUST/SHOULD/MAY.

Thank you for the thorough review; you have caught several places that
were simple errors on my part (multipart/related for newgroup mesages, for
instance; I don't know where I got that from, as it had been my intention
to mostly copy USEPRO there).  I'm embarassed that there were a couple of
those.  I suppose that's the inherent peril of rewriting.

I will try to respond to this message in depth tonight noting which of
these were simple errors in my draft that I would correct as you describe
and which of these changes were intentional.

In several cases, you noted a change that I didn't intend to make and that
I thought I *hadn't* made, and I believe it may just be that the wording
moved from one place to another.  I need to study those in more detail
though before I can say for sure what happened.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: Russ Allbery <[email-masked]>
Date: Wed, 13 Dec 2006 13:24:53 -0800

Responding to this one specifically quickly, since it may save you some
time opening a broader discussion.  I think you may have misunderstood the
impact of my changes to the definition of proto-articles.

> necessitated reverting to reliance on Date header in many
> situations. The whole point of introuducing Injection-Date was to let
> Date reflect poster's composition time, even if not injected until weeks
> later. This is now lost. I shall raise a separate thread for this issue.

Injection-Date may not be rewritten (except by gateways; see below).  The
intention of my draft is stronger than the existing USEPRO in that not
only does it prohibit injecting agents from rewriting Injection-Date, it
prohibits them from accepting articles that already contain an
Injection-Date header at all.  See section 3.4, point 2:

   2.   It MUST reject any proto-article that does not have the proper
        mandatory header fields for a proto-article; that has Injection-
        Date, Injection-Info, or Xref header fields; that has a Path
        header field containing the "POSTED" <diag-keyword>; or that is
        not syntactically valid as defined by [USEFOR].  It SHOULD
        reject any proto-article which contains a header field
        deprecated for Netnews.  It MAY reject any proto-article that
        contains trace header fields indicating that it was already
        injected by an injecting agent that did not add Injection-Info
        or Injection-Date.

supported by section 3.3.1, paragraph 2:

   A proto-article has the same format as a normal article except that
   the Injection-Date, Injection-Info, and Xref header fields MUST NOT
   be present; the Path header field MUST NOT contain a "POSTED" <diag-
   keyword>; and any of the following mandatory header fields MAY be
   omitted: Message-ID, Date, and Path.  In all other respects, a proto-
   article MUST be a valid Netnews article.

The only place where there is a provision for removing an Injection-Date
header is when gatewaying a message from one Netnews network to another,
since such gatewaying involves transforming a message from an article back
to a proto-article and then reinjecting it.  The agent making such a
transformation is, in this case, specifically required to perform the
checks on the Injection-Date header that the injecting agent would
otherwise be performing:

   In the exceptional case that an article needs to be reinjected for
   some reason (such as transferring an article from one Netnews to
   another where those networks have no relaying agreement), the posting
   agent doing the reinjection MUST convert the article back into a
   proto-article before passing it to an injecting agent (such as by
   renaming the Injection-Info and Injection-Date header fields and
   removing any Xref header field) and MUST perform the date checks on
   the existing Injection-Date or Date header fields that would
   otherwise be done by the injecting agent.

   Reinjecting articles may cause loops, loss of trace information, and
   other problems and should only be done with care and when there is no
   available alternative.  A posting agent that does reinjection is a
   limited type of gateway and as such is subject to all of the
   requirements of an incoming gateway in addition to the requirements
   of a posting agent.

---

From: Russ Allbery <[email-masked]>
Date: Wed, 13 Dec 2006 23:43:45 -0800

> (This was a suggestion by Harald, & 1 of 2 alternatives in USEPRO)

Intentional, but I don't object to changing it.

Yeah, I took the first listed option in USEPRO which didn't include this
case.  I have no major problem with including it and no strong opinion
either way.  This isn't a change from USEPRO so much as my picking one of
the two listed options that made the most sense to me, and it's not a
place where I have a strong opinion.

The intentional change here is that I added a different preferred first
option, namely the FQDN of the news server itself.

> other, so you could not rely on either. If anything, I would have
> expected case-insensitive (that is what domain-names are, and what
> USEFOR declares <diag-keyword>s are).

Intentional change.

Declaring them case-sensitive is the conservative choice.  If you treat
them as case-sensitive and they're actually compared case-insensitively,
everything works fine.  The opposite is not true; if your Path identity
varies in case, servers that compare case-sensitively may add incorrect
MISMATCH entries or send you articles that you sent them.

The one exception is if you use a Path header that varies only in case
from another site and assume that it will be compared case-sensitively.  I
thought about adding language to specifically say not to do that, but it's
hard to tell a site how to guarantee that without some way of psychically
determining what <path-nodot> entries might be used by other sites.
Anyone doing this is already violating the SHOULD that says path
identities should be in all lowercase, so adding an additional restriction
that's difficult to phrase didn't seem to me to add anything worthwhile to
the protocol description.

> (not possible in NNTP, of course)

Intentional change.

What injecting agent supports this?  Telling posting agents that this is
allowed when very few injecting agents support it doesn't strike me as a
good idea.  This doesn't allow for injecting agents that replace the From
header with one containing the authenticated identity of the user, and I
do know of some injecting agents which do this, but I don't think that's
something that we want to support.  That's not the purpose of the From
header; it's the purpose of the Sender header or (far better) the
Injection-Info header.

> later. This is now lost. I shall raise a separate thread for this issue.

Addressed in an earlier message.

> within the 998 limit is a MAY.

Intentional change, but I'm dithering.

There is an INN bug in older versions that will cause nnrpd to reject
messages with a single header field in excess of 998 octets, folded.  This
is clearly a bug in INN (and was fixed).

I don't mind changing this to MUST, but I'm not sure there's sufficient
justification for that strength of requirement.  On the other hand, I'm
pretty sure I've argued both sides of this one at different times, so
maybe it's best to just leave good enough alone and not change this.

> All it needs is an NNTP 44x response or similar.

Intentional change.

What this means is that the Netnews protocol would be placing a
requirement on the transport layer that it have a means of reporting
errors back to a client.  NNTP has such a means, but I don't think this is
important enough at the protocol level to require of all transport layers.
I don't see a compelling need for a protocol requirement here.  Not
returning specific rejection errors to a posting agent does not break the
Netnews protocol; it's a quality of implementation issue, and there's
enough rationale still present in the document to make implementors aware
of it.

> So it can reject if not at least one group in Newgroups header exists
> This is an addition to agreed MUST keep list of moderated groups.

Intentional change.

> crosspost to some remote and unknown hierarchy, then this should be
> stated explicitly.

Intentional change.

This section says more than just that, and I think the change makes more
sense with more context:

   7.   If the Newsgroups header contains one or more moderated groups
        and the proto-article does not contain an Approved header field,
        the injecting agent SHOULD forward it to a moderator as
        specified in Section 3.4.1.  If the article cannot be forwarded
        to a moderator for some reason, it MUST be rejected.

In other words, in my language, the standard does not REQUIRE that an
injecting agent be capable of forwarding posts to moderators, but does
REQUIRE that it either forward posts to moderators or reject them and
supporting moderated groups is RECOMMENDED.  I think this strikes a
balance that makes more sense from an implementation perspective than the
current USEFOR language.

> 9. [-1] (3.4,3.5) Prepending of <path-identity> to Path by injecting
> agent reduced from MUST to SHOULD.

> I would accept that prepending the <path-diagnostic> POSTED is a SHOULD.

And that's exactly what my language says.  Here's the text:

   9.   The injecting agent SHOULD then prepend the <path-identity> of
        the injecting agent followed by "!.POSTED", optionally "." and
        the FQDN or IP address of the source, and a further "!" to the
        content of the Path header field.  If the injecting agent does
        not support the use of <diag-keyword>, it MUST instead prepend
        its <path-identity> followed by "!"; one or the other of these
        mechanisms MUST be used.

Adding the <path-identity> with the <path-diagnostic> is a SHOULD.  If it
doesn't add the <path-diagnostic>, it MUST add its own <path-identity>
following the current rules for constructing Path headers.

The language is phrased similarly for relaing and serving agents.

Changing the <path-diagnostic> from a MUST to a SHOULD is an intentional
change.

> reduced from SHOULD to MAY.

Intentional change.

If Path header folding turns out to be a serious problem in practice, I'd
prefer to leave open to agents the alternative of not doing it.

> Otherwise, the task of chasing the bad guys becomes harder.

Intentional change.

I don't see much purpose served by this requirement and the meaning is
very murky to me.  I think I understand the general intention, but the
rule is not very specific and I'd have a hard time saying whether a
particular behavior complied with this rule or not.  If the format allows
so much flexibility that we have to make a rule like this, I think that's
a problem with the specification of the header field.

> 12. [-1] (3.5) The significance of "world" and "local" in the
> Distribution header are no longer mentioned.

> We recently took care to reserve these in USEFOR, but that requires
> backup here to ensure the protocol fulfils that promise.

Intentional change.

Most everything useful to be said about "world" and "local" was already
said in USEFOR, and "world" in particular SHOULD NOT be used anyway (as
already stated in USEFOR).  Distribution is generally a waste of effort,
and I'd rather keep the rules related to it to a minimum since in practice
few sites are ever going to care.

If we really have to, we can say something like "If the Distribution
header field contains a <dist-name> of 'world', the article SHOULD be
treated as if the Distribution header field were absent," but note that
this is *not* how existing software behaves.  INN, for instance, doesn't
treat "world" any differently from any other distribution except in its
default configuration files.

I wouldn't argue strongly against that change.

I don't see anything else that should be said about local.  The language
in USEFOR now is better and more accurate, IMO, than the language
currently in USEPRO.

> 13. [+1] (3.5,3.6) Removed possibility of "aliassing out" a site where
> you didn't want an article to go by including it in the Path to the
> right of POSTED.

> alongside other conventions for "aliassing out", such as the
> 'cybercancel' convention which is no longer mentioned at all.

Intentional change.

I think this whole area is best deferred to USEAGE or some other best
practice document.  It's thankfully becoming less important, and the whole
pseudo-site convention is complex and strange enough (and enough of a
corner case) that it really belongs in a separate document about the
bizarre world of third-party cancels.

> Before, it was SHOULD reject for any missing mandatory header.

Intentional change.

This is already done in practice by relaying agents that perform even the
most basic validity checks on articles, and *everyone* rejects (or at
least should be rejecting) articles without Message-ID or
Date/Injection-Date headers since the protocol simply doesn't work without
them.  I can see removing Newsgroups here, but I'd rather keep it.

> 15. [-1] (3.5) Relaying agent MUST reject any article already already
> sent to it.

> header is normally considered sufficient. I might buy a SHOULD here,
> with suitable wording.

Intentional change.

Every relayer I've ever heard of keeps a history file and has to to be
usable in the real world.  The protocol breaks down badly if you have
multiple peers and don't do this.  This is a core part of the flood-fill
protocol.

> agent can do is then not to propagate it further).

Not actually a change, but I think this should stay.  Here's what it says:

   5.   It SHOULD reject any article that matches an already-received
        cancel control message or the contents of the Supersedes header
        field of an accepted article, provided that the relaying agent
        chose (on the basis of local site policy) to honor that cancel
        control message or Supersedes header field.

This isn't about acting on cancel messages for messages one has already
accepted.  It's about accepting cancels before the message arrives, which
is sufficiently counter-intuitive that it needs to be explicitly
mentioned.  I've seen news server implementations that didn't think about
this, and at least in a spam cancel world it used to be important.

Relaying agents of course always have the option of simply not acting on
cancel messages if they don't want to deal.

> in contrast to the case of injecting agents mentioned above.)

Intentional change.

As above, I made this SHOULD everwhere.  And as mentioned above, I think
you misread the part about injecting agents; adding the <path-identity> is
still a MUST in my language.

> addition to a domain-name. I see you also removed this from the example
> in 3.5.1.

Intentional change, but I dithered about it and agree with changing it
back.

The main reason not to add this is just complexity of description, not
anything internal to the protocol.  The protocol itself doesn't really
care how many <path-identity>s you add, so I just need to find a better
way of describing the Path header field manipulation so that adding this
didn't strain the language.

I think moving the Path header field manipulation into a separate section
that's just referred to by the separate agent duties sections will give us
the right layout to clearly add this without making one of these numbered
steps even more complex.

> verified by the next site.

Unintentional.  Sorry about that.

I handled this case for relaying agents and just missed serving agents.
Relaying agents say:

   7.   If the relaying agent is processing an article from an injecting
        agent that is part of the same news server, it MAY leave the
        Path header field unmodified.

Serving agents similarly need to say:

   7.   If the serving agent is processing an article from an injecting
        or relaying agent that is part of the same news server, it MAY
        leave the Path header field unmodified.

Having a separate section describing Path header construction will make it
easier to catch inconsistencies like this.

> Why might it want to do that?

Intentional change.

Because the same piece of software also implements a serving agent, which
has to add the Xref header, and the same code path is used for all
articles received by the server whether they are for further relaying or
local serving.  Everything already copes with relaying agents adding Xref
or changing Xref header fields and it's common for combined news servers
to do this, so it's easiest to just explicitly allow it.

> Is this any of our business?

Intentional change.  See previous discussion.

> 21. [-1] (3.6) No mention of processing cancel messages by
> serving/storage agents.

> case where a cancel arrives before the article (it is covered in 5.3,
> but it is the serving agent's duty to implement it).

The same language is here as for relaying agents, but you may have missed
it because it's combined with another similar point:

   3.  It SHOULD reject any article that has already been successfully
       sent to it or that matches an already-received and honored cancel
       message or Supersedes header field following the same rules as a
       relaying agent (Section 3.5).

Yes, there is no specific mention in the duties section of acting on
control messages for already received messages.  This is for the same
reason that there's no specific mention in the duties section of acting on
newgroup, rmgroup, or checkgroups control messages.  My presumption is
that the description of the cancel control message later in its own
section is sufficient.  The case of a control message arriving before an
article is only mentioned because it's a surprising corner case.

> capability to display the raw article 'as received'.

Intentional change.

For the reasons spelled out in the section for reading agents, I don't
think makes sense for us to put protocol requirements on the operation of
reading agents.  Nothing about the operation of a reading agent affects
the rest of the network.

I certainly agree with this recommendation as part of the best practices
in USEAGE.

> That could cause problems if the original poster later wants to cancel
> it, or if he had forwarded it from some other source.

Intentional change.

As mentioned in one of my earlier messages about general philosophy, I
removed all requirements placed on agents that are not limited in practice
by the protocol.  Moderators are given the proto-article and are free to
do anything they wish with it; there is no effective protocol limits on
their modifications to messages, including whether or not they retain the
message ID.  This doesn't feel to me to be in scope for a protocol
specification.  It's certainly in scope for a best practices document
about moderation.

It's not at all clear to me that posters should have the ability to cancel
messages that they sent to a moderated group (think RISK digest, for
instance), and I think this is going too far into the realm of network
policy and moderation policy to address in this document.

> change the Content-Transfer-Encoding (which will normally be 8bit or
> 7bit in Netnews except for genuine binary objects), it seems desirable
> to mention the exception here, where it may be necessary before the
> article can be passed through some email agents.

I don't consider this to be a change.  There's nothing that says that it
*can't* change the Content-Transfer-Encoding, and the only explicit
statement in my document about CTE is a parenthetical comment for the
injecting agent that says it's not allowed to change the CTE as a
component of the general prohibition on changing the body of the message.
There isn't a general statement about CTE that has to be overridden here.

An outgoing gateway, by its very nature, is not subject to any protocol
restrictions from a Netnews perspective, only recommendations.  It may
change the CTE or do anything else it needs to do.

> That needs to be mentioned explicitly, as it is a change to an existing

Intentional change.

Agreed that this difference from the registration needs to be mentioned
explicitly.

> Application types are inherently dangerous if they are executed when
> they should not be. Is there any situation where these two could
> properly be used outside of their intended control messages? Sure, for
> informational purposes, or for passing by private arrangement from one
> newsadmin to another, but best leave it to their common sense if they
> want to break the rules. In any case, news-checkgroups does not make
> sense without its accompanying checkscope parameter.

Intentional change.

I can't think of an example of how application/news-groupinfo could be
dangerous.  It's nothing but a bit of text giving a newsgroup name and a
description.  In and of itself, it has no force, power, or action
whatsoever.

Similarly, application/news-checkgroups is simple text; the only
difference is that it inherently represents a claim that it constitutes a
complete list of newsgroups in a particular hierarchy.  I think that the
security implications of this claim are adequately dealt with by the
security considerations section of the registration:

   Security considerations:  This media type provides only a means for
                             conveying a list of newsgroups and does not
                             provide any information indicating whether
                             the sender is authorized to state which
                             newsgroups should exist within a hierarchy.
                             Such authorization must be accomplished by
                             other means.

Prohibiting use of these media types in all other situations is overly
broad and seems unnecessary to me.  If I mail a checkgroups for a
hierarchy to another newsadmin, I don't see any reason why I should be
prohibited from using application/news-checkgroups for that mail message.
Similarly, if a uk.* hierarchy administrator wanted to convey to uk.*
Control the correct information for a new group, I don't see any reason to
prohibit doing so as an application/news-groupinfo part in a mail message.

> I think we are now agreed abut this, but with a restriction to having

Intentional change.

Agreed that we need additional text specifying ASCII as a subset.

> Was SHOULD include the groups affected (& possibly others). Without
> that, control messages won't propagate to the places they are likely to
> be needed.

Intentional change, but after looking at the INN code, I was wrong.

See the duties of a relaying agent, which I modified to reflect what I
thought was the existing practice for newgroup propagation:

   Exceptionally, control messages creating new newsgroups (newgroup
   control messages) SHOULD be relayed if the sending agent has been
   configured to supply and the receiving agent to receive the newsgroup
   affected by the control message, even if that newsgroup does not
   currently exist and even if the control message does not contain that
   group in its Newsgroups header field.

Looking at INN's code, though, that's not exactly how it behaves.  It is
indeed still useful to post the messages to the correct newsgroups.

So, I think we should go back to Charles's previous language for
constructing the Newsgroups header field for newgroup and rmgroup
messages, and what the above should really say is:

   Exceptionally, control messages creating or removing newsgroups
   (newgroup or rmgroup control messages, for example) SHOULD be relayed
   if the affected group appears in its Newsgroups header field and the
   sending agent would have supplied and the receiving agent would have
   received the newsgroup affected by the control message had it existed,
   even if it currently does not.

without that bit that I added at the end, which was based on an incorrect
understanding of the code.

In other words, mostly I messed this up; sorry about that.  The remaining
effective change over the existing USEPRO document is that I made the
special relaying rules a SHOULD rather than a MUST and described them in a
different way.

I'm not horribly happy with my language above; I just think it's a minor
improvement.  I'd welcome an even better improvement.

> We discussed this at some stage, but I don't think we ever agreed it one
> way or the other.

Intentional change.

> 29. [00] (5.2) Newsgroup-names MUST be checked against disallowed names
> before group control message is honoured.

> Shouldn't it apply just to newgroup?

Intentional change.

It's just as important for checkgroups as it is for newgroup.  Given that,
I included it for rmgroup as well since I can't think of any reason why it
would hurt and it improves the protocol description to make it a general
statement for all group control messages.  (Which, note, includes any
introduced by other documents, such as mvgroup.)

> Surely, it SHOULD identify the person/identity/role of the issuer, and
> be a working email address. Even in alt.* "Approved: foo" is not really
> good enough. News servers are usually configured with what to expect
> there for each hierarchy.

Intentional change.

The content of the Approved header serves no protocol purpose, and USEFOR
already adequately covers the definition of its content.  Control message
authorization is done on the basis of the Sender or From header
(preferrably in combination with a digital signature).

> Was MUST (though I wouldn't take that to exclude a "For your newsgroups
> file:" to be scanned for - if allowing that was the intent of the
> SHOULD, then some rewording could cover it). That scanning is currently
> "MAY" which is possbly too weak. Or is it trying to allow the case where
> no description exists for the group (for which I would still have
> expected a Newgroups File entry with just the newsgroup-name, which is
> how a checkgroups would show it).

Intentional change.

Newsgroup descriptions are optional in the protocol and newgroup messages
without descriptions work fine with existing software.  Some hierarchies
do not use newsgroup descriptions at all.  They would have to come up with
something to put there to issue a checkgroups, but not all hierarchies
issue checkgroups control messages.  I think SHOULD strikes the right
balance; MUST is too strong because nothing breaks if it's omitted.

> 32. [-1] (5.2.1) Newgroup message containing other attachments in
> addition to news-groupinfo to be multipart/related (was
> multipart/mixed).

> I don't see what the 'relationship' is.

Unintentional change.

Agreed, this should be multipart/mixed.

> 33. [+1] (5.2.3) Checkgroups message still contains chkscope and
> chksernr arguments.

> No change from USEPRO. I think I finally persuaded Russ this should

I'm not horribly happy about it, and in particular the chksernr parameter
adds a new requirement for a local data store for a news server to keep
track of the previous serial number applied.  I'm not sure if anyone's
going to really implement this.  But the argument in favor is compelling,
so....

I guess I'm reluctantly willing to see if anyone picks it up.

> Fine in principle, but maybe not implementable. If de.alt.* were as
> chaotic as alt.* (is it?) there would be no way the admins of de.* could
> include a defnitive list of de.alt.*. So some qualification is needed.

Intentional change.  See previous discussion.

> Was SHOULD.

Intentional change.

This was added as a security feature, and it's worthless if there aren't
strong guarantees about how it's used.  I also added language not present
in the USEPRO draft RECOMMENDING that servers implement chksernr
processing, since if we're going to specify it, we need to say it has to
be used, or it's a pointless addition.

> 36. [00] (5.2.3) Body of checkgroups SHOULD be application/checkgroups,
> and otherwise SHOULD treat as if it were.

> Why does that SHOULD differ from the corresponding MUST in newgroups? I
> would have expected "SHOULD ... and otherwise MUST ...".

Intentional change.

What MUST in newgroup?  Scanning the body in newgroup messages without a
MIME part is a MAY.  It's raised to SHOULD in checkgroups because unlike
for newgroups, this is actually necessary to honor the checkgroups.

It's SHOULD and not MUST because I don't see the justification for the
stronger requirement.  If a news server wants to apply a different set of
heuristics to examining untagged checkgroups bodies for some reason, I
don't see any obvious cause to prohibit that.  Treating it the same as
application/news-checkgroups is just the RECOMMENDED approach that will
generally work fine.

> Was SHOULD.

Intentional change.

The body text serves no protocol purpose and therefore shouldn't be
subject to a protocol constraint.

> Agreed it is not the person who posted the original article. USEPRO says
> it "should" be the person who issued the cancel. That should really be
> "SHOULD".

Intentional change.

The From/Sender of a cancel message is no different in definition than the
From/Sender of any other Netnews message, and as such I think USEFOR
covers the situation sufficiently.  I don't see a reason to call special
attention to the fields for cancel messages when cancel messages are no
different than normal articles in this regard.

> 39. [-1] (5.5) The <relayer-name> in ihave and sendme control messages
> is now optional.

> It is required to be present in USEPRO and in Son-of-1036.

Unintentional change.

USEPRO offers two different syntaxes, one of which is marked SHOULD NOT,
and I found that confusing.  I tried to combine them and missed this
distinction.  (So, actually, it's not required to be present in USEPRO;
it's only a SHOULD NOT to omit it.)

I'm not sure what to do with this.  I find having two separate ABNFs for
the same field to be very confusing.  I thought I could just use the
varient form as the more general one and mention in the description that
the message IDs SHOULD be in the body and not the header, but apparently
whether the relayer name is optional is also different between the forms,
which makes them not compatible.

This area is annoying.  I hate spending time on these control messages in
the working group when essentially no one uses them.  To make it worse,
I'm almost certainly retreading a discussion that we already had.  Maybe
having two varient ABNFs is the best that we can do.

> Note that this was a format 'on the wire' and was intended to be sent on
> the same 'wire' as the ihave and sendme messages (which would usually be
> a UUCP connection). Hence it is a legitimate item to
> standardize. Probably needs some brief mention of its compressed
> versions.

Intentional change.

   Upon receipt of the sendme message (and a decision to honor it), the
   receiving agent sends the article or articles requested.  The
   mechanism for this transmission is unspecified by this document and
   is arranged between the sites involved.

Specifying the behavior of the transport layer is out of the scope of this
document.  Discussion of the UUCP batch format no more belongs here than
does a discussion of dot-escaping in NNTP.

> documented (as with the batch format).

I have:

   These control messages are normally sent as point-to-point articles
   between two sites and not then sent on to other sites.  Newsgroups
   beginning with "to." are reserved for such point-to-point
   communications.

Son-of has:

          It is conventional to reserve newsgroup names beginning with
          "to." for test messages sent  on  an  essentially  point-to-
          point basis (see also the ihave/sendme protocol described in
          section 7.2); newsgroup names beginning  with  "to."  SHOULD
          not be used for any other purpose.  The second (and possibly
          later) components of such a name should, together,  comprise
          the  relayer name (see section 5.6) of a relayer.  The news-
          group exists only at the named relayer  and  its  neighbors.
          The  neighbors all pass that newsgroup to the named relayer,
          while the named relayer does not pass it to anyone.

So the question, unless someone has some additional documentation more
thorough than Son-of available, is whether we want to get into how
newsgroups in the to.* hierarchy are named.  The problem with doing so is
that we then have to get into defining the relayer name (to use the Son-of
term), which for the use of to.* groups in practice is a UUCP host name.
That feels to me like wandering rather far into the weeds.

I suppose we can throw in something here like "the components after 'to.'
are taken to be the <path-identity> of the agent to which an article
posted to that to.* newsgroup should be sent."

> Was MUST NOT. Cf the MUST NOT for obsolete headers in USEFOR.

Intentional change.

MUST NOT is unwarranted.  Sending them doesn't break anything.

---

From: Harald Alvestrand <[email-masked]>
Date: Thu, 14 Dec 2006 12:08:06 +0100

I read the changes of language around injection-date as resulting from 
Russ' removal of the idea of "reinjection".

Given that reinjection does not happen, I don't see any change in 
injection-date allowed.

                 Harald

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 14 Dec 2006 21:23:14 GMT

>> 4. [--] (3.2.2,3.4,3.8) Injection-Date can be rewritten.

It is clear that the topics of Gatewaying, Reinjection and the role of the
Injection-Date header are closely intertwined, and need to be considered
together. The WG last considered this in any detail in May of 2004
(archives available on www.imc.org/ietf-usefor).

The essential reason for introducing Injection-Date was that we now follow
the RFC 2822 principle that the Date header indicates when the article was
composed, and hence is unsuited to its role of preventing loops when an
article propagates so slowly that it reaches some server after it has
already been seen and expired there. So a man who composes and Dates an
article and then carries it around on his laptop for 3 weeks before
injecting is not disadvantaged (it may be a stupid thing to do, but that
is no concern of ours).

In the old days, servers had to allow for articles arriving up to 14 days
late (transatlantic propagation delays were regularly over a week then).
Nowadays, the cutoff will be nearer 3 days. Injection-Date MUST ONLY be
created by an injecting agent, but once created it must NEVER be removed
so long as there is any possibility that it may still be seen propagating
on the same network where it was injected (and that includes all bizarre
scenarios of gatewaying it to some other medium/network and later
gatewaying it back again).

So gatewaying and reinjection are the critical issues. We always
understood that reinjection was unnecessary on a perfectly regulated
network, but on a real network it would occasionally happen for a whole
variety of bizarre yet plausible reasons. Here is a list of examples we
produced at that time:

   . a relaying agent unable to relay an article to its usual peers due to
   unanticipated communication problems (for example, an agent on a laptop
   that has been physically disconnected from its usual environment) might
   attempt instead to reinject it at a site which did not recognize it as
   a peer;

   . a relaying agent receiving an attempted relay from a source with
   which it did not have a peering agreement might choose to reinject it
   instead, in order to fulfil its obligation to the rest of the network
   (8.2) to be responsible ...

   . there might be a genuine misunderstanding between two sites as to
   whether the relationship between them was as peers for relaying, or as
   client/server for injecting.

   . an article might be gatewayed out of Netnews into some other medium,
   and then back again into the same, or a different, Netnews network.

But we carefully did not write that list into the draft, because we did
not want to imply that such practices were normal, nor that the list was
exhaustive. For sure, there will be other bizarre but plausible reasons
why it makes sense that we cannot possibly foresee now.

Now the only case for reinjection that Russ seems to recognize is where an
article is gatweayed between two totally disjoint Netnews networks, and
that is just one half of one of the scenarios I gave above.

My belief is that it will most commonly arise where it is being reinjected
into the _same_ network; usually because of some misunderstanding between
a server and a client as to whether it is peering or injecting (especially
likely now that NNRP can accept an IHAVE command and treat it as a POST if
the client does not have peering privileges), or maybe as part of a
deliberate decision to inject an article to two servers (for additional
robustness, or other plausible reasons), where one of them allows peering
and the other doesn't.

As regards truly disjoint Netnews networks (which we indeed claim to cater
for in our document), my belief is that they are extremely rare. Most
private networks of which I am aware share the same servers as the regular
Usenet, and hence have a tendency to leak. Preventing such leaks, whilst
not impossible, is a truly difficult task. Even if one site carefully
injects its two categories of netnews to different servers, one can never
be sure that some other site on those networks does not regard them as two
parts of the same network. And then there can be privately and secretly
constructed unofficial paths between them.

Therefore, fixing the problem by saying reinjection MUST convert the
article back into a proto article (by removing the Injectio-Info and the
Injection-Date and cleaning out the Path) just will not work. It *might*
work if you removed the Message-ID at the same time (note that a proto
article can, and frequently does, already contain a Message-ID). But that
would cause all sorts of other problems, especially in the case of the man
who was injecting the same article to two sites, for it is a fundamental
principle of Usenet that articles with the same(different) message-id ARE
the same(different) article.

Moreover, once having allowed an Injection-Date to be removed, you have
had to reinstate the provision that Dates over 72 hours old SHOULD be
rejected (3.4 Step 3 and again in 3.8 Step 4), thus preventing the case of
the man who wants to carry his already written article around for more
than 72 hours on his laptop, which was one of the reasons for introducing
Injection-Date in the first place.

As for what injecting agents should do if offered an article they
recognize as already injected, that is really a matter of site policy
(they clearly have the right to drop it). But to REQUIRE them to drop it
may cut off many perfectly reasonable cases, and is certainly not current
practice. Moreover, allowing Injection-Date to be removed at a Gateway, as
Russ proposes, is only safe if you can be sure you are gatewaying in a
manner that will NEVER allow it back into the original network. Much safer
to leave it there (assuming there is some way to express it in the other
medium).

When we first discussed all this, back in 2004, we tossed the ideas around
for a few weeks, and when it seemed we has reached a rough consensus I
wrote down and posted what I understood we had agreed (because I did not
want to start cutting new text in the draft until it was quite clear what
was to be done).

I reproduce below what I wrote then. On re-reading it now, I still think
it describes correctly what needs to be done.

---------------------------------------------------------------------------

1. There will be a distinction between the process of "injection" (which will
   be the normal situation) and "reinjection", which should be a rare event
   brought about by exceptional circumstances. We discussed many of these
   circumstances (servers on visiting laptops, use of IHAVE by unauthorized
   users, strange gatewaying). I shall write text to indicate that it is for
   such exceptional situations, and that injecting sites should only allow it
   as a result of a clear policy decision to do so.

2. There shall be a header Injection-Date. Syntax as for Date. 
   MUST ONLY be inserted by injecting agents. This header is intended to 
   replace the currently-used but nowhere-documented header 
   "NNTP-Posting-Date" (cf. NOTE in the present 6.19 regarding 
   NNTP-Posting-Host and X-Trace). 
  
3. The existing Injector-Info header to be renamed as Injection-Info (for 
   consistency of naming), and the posting-date parameter to be removed from 
   it. This header SHOULD be rewritten on reinjection (because the reinjecting
   site takes responsibility for it).
  
4. An injecting agent: 
  
   SHOULD reject any article whose Date-header is more than 24 hours into the
   future.

   MUST reject any article which already has an Injection-Date (unless it is
   convinced that it really ought to be reinjecting).

   MUST NOT remove any Injection-Date or NNTP-Posting-Date header that is 
   already present. [Is that too strong for NNTP-Posting-Date?]
  
   MUST[1] create an Injection-Date-header (except when forwarding to a
   moderator).

   SHOULD create an Injection-Info-header (removing any already present);
   likewise Complaints-To header.

5. A reinjecting agent[3]:

   SHOULD reject any article whose Date-header is more than 24 hours into the
   future.

   MUST NOT remove any Injection-Date or NNTP-Posting-Date header that is 
   already present. 
  
   IF there is no Injection-Date-header already present:
      MUST/SHOULD/MAY[4] reject if Date is more than 72 hours into the past,
      and then MUST create an Injection-Date-header.

   SHOULD create an Injection-Info-header (removing any already present);
   likewise Complaints-To header.

6. A posting agent MUST NOT create an Injection-Date-header.
  
7. A relaying or serving agent: 
  
   MUST look at the Injection-Date (or, if missing, at the Date) and 
   refuse the article if that predates the earliest articles of which it 
   normally keeps record, or if it is more than 24 hours into the future 
   (though they MAY use a margin less than that 24 hours). 
  
5. A moderator: 
  
   MUST remove any Injection-Date header already present (though there
   should be none).
  
   SHOULD[5] always retain the incoming Date-header (it is up to his injector 
   to do the 'right thing' as regards staleness checks and inserting 
   Injection-Date). This actually simplifies the moderator's duties 
   compared to the present text. 

Notes: 

[1] If you want a MUST create Injection-Date-header on injecting, then
   that makes it a Required header, and it goes in Chapter 5. So if anyone
   thinks it should be a SHOULD, then please say so quickly, because I
   don't want to be moving it to Chapter 6 once I have started updating
   the text.

[2] NOTE that other agents cannot rely on the presence of Injection-Date,
   since it will be a new feature of our standard.

[3] I don't actually intend to define the term "reinjecting agent"; that
   was just for this explanation.

[4] I put that staleness check in because, in the case of reinjection, the
   thing might have been wandering around the network for days without any
   Injection-Date. Advice needed regarding the MUST/SHOULD/MAY.

[5] I could be persuaded that should be MUST. 

---------------------------------------------------------------------------

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 15 Dec 2006 01:30:36 +0100

> Charles Lindsey <[email-masked]> writes:

Thanks for the interesting diff list.

> reading agents.  Nothing about the operation of a reading agent affects

IMO mail and news desperately need manual intervention in the case of
errors, they're not designed to run "unattended" in the case of errors.

Only real human users can spot some of those issues, and for that job
they need the complete technical info.  It's a part of the design, it
affects the rest of the network if users have no chance to diagnose
errors.

> removed all requirements placed on agents that are not limited in
> practice by the protocol.  Moderators are given the proto-article and
> are free to do anything they wish with it

They're free to approve and inject / forward it, or to reject it.  But
they are NOT free to manipulate the Message-ID.  They can add one if
it's missing, your "proto-article" analogy is fine.  They are not free
to do whatever they wish.

> there is no effective protocol limits on their modifications to
> messages, including whether or not they retain the message ID.

Of course there is, users will see if their article was forged or
manipulated, and report that.  They won't post in the NG of an
abusive moderator.  Admins can remove the NGs from their server.

This is an essential point.  You also wouldn't argue that an
injecting agent is free to do whatever it wishes.  In practice it
is, and the standard says what's good - bad - ugly.

> This doesn't feel to me to be in scope for a protocol specification.

Alternative:  Move the complete _concept_ of moderated NGs to USEAGe,
anything, not only 3.8.

> It's not at all clear to me that posters should have the ability to
> cancel messages that they sent to a moderated group

Of course they can cancel their own articles, or any forged articles
claiming to be "from" them.  It's their article.  If moderators don't
like the content of an article they're free to post their own ideas.

They never need to munge / manipulate / forge articles by somebody
else.  If it's too bad they can reject submitted articles, that's
their job.  But not manipulate it in any way they like.

Adding a note (like the "IESG notes" in RFCs) is already at the border.
Truncating articles (and the note explaining how that was done) is also
at the border (and IMHO at the wrong side of this border).  But without
some clear published rules for the relevant NG warning posters how their
articles might be mutilated the default ought to be SHOULD NOT.

> Intentional change.

USEFOR is now tuned to "don't do odd things with Subject: cmsg", after
years of discussions, this MAY has to be a MUST NOT.  It has to be noted
as new requirement, a radical change from 1036.  That was a WG consensus,
I didn't like it, but the alternative "structured subject" was admittedly
worse.  Please stick to it, news servers have no business to look into
the subject at all, end of story.

If that collides with reality it should be at least clear that it's the
fault of the standard or the server, not the fault of the user or UA.

> subject to a protocol constraint.

It MUST contain an explanation if it's not sent by the original poster,
for admin cancels and spam cancels.  MAY is not good enough, unless you
intend to discuss such details in USEPRO.  Maybe we should do this (?)

Frank

---

From: Russ Allbery <[email-masked]>
Date: Thu, 14 Dec 2006 17:30:20 -0800

> to do whatever they wish.

This clearly isn't true, as moderators have been changing the message ID
routinely for years and continue to do so as I write this.  They are
entirely free to do so; nothing stops them and Usenet does not break as a
result.

I believe that what my draft says is the correct statement for the
protocol to make.

> manipulated, and report that.  They won't post in the NG of an abusive
> moderator.  Admins can remove the NGs from their server.

This behavior is not considered abusive and only rarely draws any sort of
complaint.

> Alternative:  Move the complete _concept_ of moderated NGs to USEAGe,

Extremely strongly opposed.  That would be a show-stopper for me.

> articles might be mutilated the default ought to be SHOULD NOT.

This is a fundamental philosophical difference about what belongs in the
protocol specification.  If the working group feels that these sorts of
restrictions on the actions of a moderator belong in USEPRO rather than in
a best practices document, you should probably chose to move forward with
a different draft than mine.

>>> 28. [00] (5) Subjects starting with 'cmsg' not accompanied by a
>>> Control header MAY be rejected.

> as new requirement, a radical change from 1036.  That was a WG
> consensus, I didn't like it, but the alternative "structured subject"
> was admittedly worse.  Please stick to it, news servers have no business
> to look into the subject at all, end of story.

I don't agree that this was the working group consensus.  I stand by what
my draft says, which is also what existing software is already doing, and
I don't think it contradicts anything in USEFOR.  I don't believe that it
assumes the Subject is structured; I believe, rather, that it is a
reasonable backwards compatibility precaution.

> intend to discuss such details in USEPRO.  Maybe we should do this (?)

Philosophical difference.  If the working group wishes to take this
approach to a protocol specification, you should not use my draft.

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 15 Dec 2006 02:58:51 +0100

> approach to a protocol specification, you should not use my draft.

Okay, apparently I'm better off with USEPRO.  I dislike arbitrariness
of software or moderators hurting the weakest part of the system, the
normal posters and users.  

Frank

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Thu, 14 Dec 2006 23:12:33 -0500

Wow, a pile of things in one message is really hard to discuss.

Should all be split to separate topics.  Sigh.

I think Russ gets it right extremely often.  RFC 2119 language
is what MUST and MUST NOT be done avoid things breaking, what SHOULD
be done unless you have a really good reason, and to let people know
how they MAY embellish.

I know raw wishes and opinion have carried the WG in the past, but
there is a reason USEAGE exists: to collect the rest of the opinions.

Inline responses below.   (Liberal cutting with the intent to provide
context, not modify your comments...)

>>>capability to display the raw article 'as received'.

Display of articles is totally outside the scope of the article transfer 
protocol.  USEPRO scope is end-to-end from agent to agent, not end-to-end from
keyboard to eyeball.

> errors, they're not designed to run "unattended" in the case of errors.

I'm with you 100% here, Frank, but it isn't protocol.  Put it in USEAGE.

>>>it, or if he had forwarded it from some other source.

I'm on the fence here.  The problems with cancels on USENET are not solvable by 
message ID magic.  Cancel locks are a better way.  (Better is a relative term.)

The protocol issue is clear:  is a message with the added approved header the 
same as the original message?  I say it isn't and retaining the message ID is a 
hack that sort of makes cancels sort of maybe work, sometimes.

> articles might be mutilated the default ought to be SHOULD NOT.

And the underlying principle is that if you alter the body the message is
different and the ID should change.

Now, what if you alter the From and the Subject header fields?   Different 
message.

Add or Alter the Approved header field?  Not clear.  What is clear is that the 
message is going to be treated totally differently when received.  It is treated 
as a different message.  (This distinguishes the case from just adding another 
Received header, or Path identity.)

I also wonder if the post was to more than one moderated group, are there 
situations where retaining the message ID fails, because some gateway sees it in 
history and assumes a loop?  Email message IDs are not unique, but....

>>>We discussed this at some stage, but I don't think we ever agreed it one

I recall the WG spent a lot of time on this, and "Re:."

Let's be clear on the reason that is a MAY.  Subject is unstructured, and admins 
should be aware they have the authority to /dev/null any article for any reason, 
especially ones that they think could cause problems.

That is not why I think this deserves a MAY.

That MAY is there to warn User-Agent implementors that putting cmsg starting
a subject line could lead to poor propagation.  Don't do it.

Why not write it as "User Agents SHOULD NOT" then?  Because then you need
a SHOULD NOT and a MAY in order to be clear.  Stick with the MAY, and the
SHOULD NOT is implied.  Nice and concise.

> intend to discuss such details in USEPRO.  Maybe we should do this (?)

Frank, do you have any way to justify your opinion of MUSTard here?
What breaks?

---

From: Russ Allbery <[email-masked]>
Date: Thu, 14 Dec 2006 20:41:57 -0800

> Wow, a pile of things in one message is really hard to discuss.

Yeah, sorry about that.  I thought about breaking up my 700+ line response
into separate messages and just couldn't muster the effort.

>>>> That could cause problems if the original poster later wants to
>>>> cancel it, or if he had forwarded it from some other source.

> I'm on the fence here.  The problems with cancels on USENET are not
> solvable by message ID magic.  Cancel locks are a better way.  (Better
> is a relative term.)

> The protocol issue is clear:  is a message with the added approved
> header the same as the original message?  I say it isn't and retaining
> the message ID is a hack that sort of makes cancels sort of maybe work,
> sometimes.

This isn't actually the direction at which I'm coming to this.  The issue
that you're concerned with (keeping the message ID the same for the same
message) is one that I think only applies after the message becomes a real
article and is present on the network.  When sending proto-articles around
to moderators, no Netnews article has yet been created.  An unknown amount
of interaction between the moderator and the poster will occur before the
message is actually sent, and indeed the poster's message may never be
posted in that form.  Again, consider the RISKS digest, which is a very
illustrative corner case of the sort of thing that can happen to a
submission to a moderated group.  (The protocol needs to cover *every* use
of Netnews, not just the common case.)

So, I think the message ID can be kept the same or changed from a protocol
level and neither decision violates the protocol.  In essence, the article
is still being prepared up until the point where the moderator injects it.

I made this change instead as part of a number of similar changes aimed at
providing a sharp distinction between the Netnews *protocol* and Usenet
*best practice*, basically flushing the remaining bits of the latter out
of the protocol specification and leaving them for USEAGE.

From the perspective of the protocol, nothing in the protocol breaks if
the moderator changes the message ID.  I know this from a practical basis:
many moderators do this now and Usenet still works.  I know this from a
theoretical basis: there is no guarantee in the protocol specification
which is violated by this.  Until the moderator injects the message, only
one copy of that proto-article exists (well, unless the user injected it
multiple times, but if the moderation status is consistent that just means
the moderator gets multiple copies and if it isn't, behavior is not going
to be specifiable anyway).  It's not public until the moderator injects
it, so it doesn't matter what message ID it has from a Netnews
perspective.

As a matter of best practice, absolutely, the moderator should retain the
message ID, even if they edit the post, add notes, or do anything else
they're allowed to do by the protocol.  They should do this because it
enables a variety of other best-practice behavior, such as allowing users
to score up their own articles or replies to their own articles easily
based on message ID patterns.  However, all of the reasons why they should
do this are best practice questions, and therefore so is the
recommendation.  There is nothing in the *protocol* that makes doing this
important, and hence it's not a protocol issue.

I'll be right there with you arguing for it in USEAGE.

To require it in the protocol, it has to matter for the protocol.  Since
there's nothing in the protocol that restricts cancel messages to the
message IDs that one generates locally, or that requires a poster know or
be able to predict the message ID of their article for any reason, I
cannot justify a protocol requirement.

I'm being a bit hard-nosed here about saying "if you don't like it, maybe
my draft isn't for you" because this is typical of multiple changes I made
to remove remaining best-practice requirements from the protocol draft and
that was one of the significant philosophical differences I have with the
existing USEPRO.  I just wanted to make sure everyone realized that
removing these sorts of requirements was one of the core points of my
rewrite, not just a side issue, so if you disagree with this you may not
agree with the philosophy that I used to make other changes and wording
decisions throughout the draft.

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Thu, 14 Dec 2006 23:50:49 -0500

> normal posters and users.  

If a User-agent developer can't see how to help normal posters and
users, then an RFC isn't going to help.  And if your moderator is
a sadist, then an RFC isn't going to help.

RFCs are an invitation to cooperate.  Period.  The MUST and MUST NOT wording
looks stern, but remains nothing more than an invitation to cooperate.  RFC MUST 
and MUST NOT are not commands, they are statements of specification.  It
is IETF shorthand for "non-compliance breaks something."

Maybe the world needs an RFC like USEAGE.  I don't really care.  (IMHO, the 
GNKSA approach is probably better to address your concerns.)

The world suffers from the lack of a specification of how to get
articles from place to place correctly today.  That's the need for RUSSPRO.

We don't want GNKSA-type stuff in RUSSPRO, because bulky specs are not
nearly as useful as concise ones.  And even if some of that is there, you
can't use what you like to call RFC2119 MUSTard to get what you want.

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Fri, 15 Dec 2006 00:21:33 -0500

> of Netnews, not just the common case.)

Russ and I are agreeing for different reasons. It is hard to know you have 
covered all the cases for retaining or changing it.

I understand it isn't a netnews article "yet".  Even if it isn't a netnews 
article, it is an RFC2822 message. Message-ID means something....
    The "Message-ID:" field provides a unique message identifier that
    refers to a particular version of a particular message.  The
    uniqueness of the message identifier is guaranteed by the host that
    generates it (see below).  This message identifier is intended to be
    machine readable and not necessarily meaningful to humans.  A message
    identifier pertains to exactly one instantiation of a particular
    message; subsequent revisions to the message each receive new message
    identifiers.

    Note: There are many instances when messages are "changed", but those
    changes do not constitute a new instantiation of that message, and
    therefore the message would not get a new message identifier.  For
    example, when messages are introduced into the transport system, they
    are often prepended with additional header fields such as trace
    fields (described in section 3.6.7) and resent fields (described in
    section 3.6.6).  The addition of such header fields does not change
    the identity of the message and therefore the original "Message-ID:"
    field is retained.  In all cases, it is the meaning that the sender
    of the message wishes to convey (i.e., whether this is the same
    message or a different message) that determines whether or not the
    "Message-ID:" field changes, not any particular syntactic difference
    that appears (or does not appear) in the message.

I'm not an email RFC expert, by any means.  So there may be more relevant
text.  But is a proto-article and the version with an Approved header field the 
"same message?"  Certainly a server is going to treat those two differently.

If a server gets the version without the Approved header, should it cache the 
message-ID in History as a "reject"?  No.  But why?  Only because we can reason 
that would be detrimental.  (Is there advice on this in any of the RFCs?  I find 
that caching rejects is more art than science.)

---

From: Russ Allbery <[email-masked]>
Date: Thu, 14 Dec 2006 21:31:43 -0800

Forrest J Cavalier <[email-masked]> writes:

> Russ and I are agreeing for different reasons. It is hard to know you
> have covered all the cases for retaining or changing it.

> I understand it isn't a netnews article "yet".  Even if it isn't a
> netnews article, it is an RFC2822 message. Message-ID means
> something....

I don't have much to add here (I think you raise good points) except to
note that if one uses method (a) of forwarding messages to a moderator (or
if one just puts the message into a local database that the moderator then
has access to, as is explicitly permitted in my document), the
proto-article is encapsulated and therefore the mail message identifier
(if any) is irrelevant from a Netnews perspective.

We only run into potential issues with the interaction between the mail
message identifier and the Netnews message identifier when moderation
forwarding is done by pretending the proto-article is an e-mail message
and dropping it into the mail with a minimum of gatewaying, forcing the
moderator to then convert it back to a proto-article before dealing with
it.  I think we're all agreed that this is a wart on the protocol and it
would be great to get rid of it if we can finesse the transition issues.

I'm personally inclined to try to pretend that this ugly bit of gatewaying
involved in the current common method of forwarding messages to moderators
doesn't exist when it comes to considering the disposition of the message
identifier, since otherwise we end up with different rules depending on
how the message is forwarded to the moderator and that feels even uglier
and even harder to explain.

---

From: Harald Alvestrand <[email-masked]>
Date: Fri, 15 Dec 2006 09:46:58 +0100

> normal posters and users.  
I think (technical opinion, not chair hat) that just like USEFOR deals 
with the format and ignores the protocol, USEPRO should deal with the 
protocol and ignore the parts that can only be specified as conventions.

Frank, you may be best served by jumping onto the long-languishing draft 
for USEAGE.

                      Harald

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 15 Dec 2006 12:31:47 +0100

Forrest J. Cavalier III wrote:
 
> And if your moderator is a sadist, then an RFC isn't going to help.

The four times I tried to post articles in Usenet groups with human
moderators all failed.  Based on that somewhat limited experience the
one moderator I trust - after reading the FAQ several times and testing
it - is the modbot protecting nanas.  A modbot is a piece of software,
an RFC can state what it's expected to do.  

> RFCs are an invitation to cooperate.  Period.  The MUST and MUST NOT
> wording looks stern, but remains nothing more than an invitation to
> cooperate.

They're often used as reference for decisions like "observed behaviour
A is right, not A is wrong, so fix it".  Of course you can ignore some
MUSTs, but...

> "non-compliance breaks something."

> IMHO, the GNKSA approach is probably better to address your concerns.

It's limited to newsreaders, otherwise I liked it.  Maybe not exactly
state of the art, it's some years old, but a good start for USEAGE.
That's what Charles did for useage-00.  I'm less impressed by texts
like RFC 1855, and weird body conventions like tear lines.

> articles from place to place correctly today.

For moderated NGs the modbot or moderator is one of these places.  It
has duties, and some are critical.  Some UAs allow to "post + mail",
it could have strange effects if something in that procedure makes up
its own Message-IDs.  

Articles can be x-posted in more than one moderated NG.  It won't do
if the 1st moderator approves Message-ID <x1>, and the 2nd moderator
approves the article replacing the Message-ID by <x2>, the cancelbot
of the 1st moderator could kill the unknown <x2>.

The Message-ID is essential for Netnews, also in moderated groups.

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 15 Dec 2006 12:51:27 +0100

[explanation of cancel]

> Frank, do you have any way to justify your opinion of MUSTard here?

It's hard to decide what's a tolerated spam-cancel or a good admin-cancel
without some hint in the body.  It "breaks" in a certain sense as soon as
admins decide to ignore cancels, when figuring out good vs. rogue takes
too much of their time after complaints.

For some 1st party cancels a hint can be also important, if the article
was posted by somebody else forging the "From".  Not enough for a MUST,
but all in all more than only an option, recommending a hint makes sense.

Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 15 Dec 2006 19:34:16 GMT

>entirely free to do so; nothing stops them and Usenet does not break as a

AFAIR, that bit in USEPRO was put in after some discussion about the habit
of moderators changing Message-IDs (apparently Kent Landfield's FAQ
encourages it :-( ); we decided we didn't like the practice and wanted it
to stop. Hence the wording that appeared in USEPRO. Of course, there may
be some bizarre situations (including when he has significantly changed
the text) where the moderator really has to change it, hence it was SHOULD
rather than MUST.

As to whether things break, they can certainly become highly inconvenient.
If the OP has also submitted it to some mailing list, for example, there
is the possibility that the two versions may come together again at some
point, and the same message-id is an immediate indication that they are
the same message. Also, as someone has mentioned, users like to keep track
of, or be alerted to, followups of their own articles.

>Extremely strongly opposed.  That would be a show-stopper for me.

+1 to that.

>a different draft than mine.

Yes, I think the philosophical difference is the real issue we need to
address. Views on what IETF standards may legitimately say cover a wide
range, and Russ's position is fairly far towards one end of a this
continuum. Essentially, he implies that if nothing breaks, then it doesn't
belong in the standard. OK, that is clearly a gross over-simplification,
so don't take it too literally.

I tried to take a more relaxed view in USEPRO (and the WG was often
pressing me in that direction). Essentially, if some feature, whilst not
actually causing stuff to get lost, makes life terribly difficult for
everyone, then it is better for the standard to disallow it. Security
issues are sometimes an example of this. So are things like Trace Headers.
If Received headers were entirely omitted from RFC 282[12], the mail would
still get through. But sorting out all the awkward things that inevitbly
go wrong would be hellishly difficult without them.

However, it is not all as Black and White as the above might suggest, and
I am not sure we should be addressing this particular issues just yet. I
have lots of differences between USEPRO and RUSPRO that I want to raise
sooner of later, but raising them all at once would just give this group a
huge information overload. So I chose to raise the items related to
protocol changes first, and I think we should mainly stick to those for
the moment.

Ultimately, the document we finally produce will have lots of input from
USEPRO and lots off input from RUSSPRO. Whether it is produced by starting
from RUSSPRO and putting some things back, or by starting with USEPRO and
taking things out, is less important than ensuring that we converge on the
correct document eventually.

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 15 Dec 2006 19:58:44 GMT

>Wow, a pile of things in one message is really hard to discuss.

Sure it was a big list. But we need to see which items on it are going to
cause disagreement, and then we can start separate threads/issues on those
ones. A lot of them will turn out to be minor misunderstandings, or
matters of wording, etc.

>message ID magic.  Cancel locks are a better way.  (Better is a relative term.)

The WG looked at Cancel Locks once, and decided they would not work
without corresponding changes in other places. So it needs that "Security
Extension" which we have often mentioned and could, in theory, even get
arund to one day :-) .

>hack that sort of makes cancels sort of maybe work, sometimes.

I think it is essentially the same message. You do not need to worry about
the precise identity of a message until it is finally let loose on Usenet.
Generally speaking, small additions or changes to headers do not create a
'different' message; even adding stupid warnings about the viruses you
didn't find to the end of the body doesn't make it different. RFC 2822 has
some useful words on this and, modulo a slightly different set of headers,
works also for news.

>different and the ID should change.

Only if you cause the essential "meaning" of the message to change. Which
"most" body changes would do, and even changes of "Subject".

>history and assumes a loop?  Email message IDs are not unique, but....

There will always be some bizarre borderline cases, which is why it is
SHOULD rather than MUST.

>I recall the WG spent a lot of time on this, and "Re:."

The WG certainly spent a lot of time on "Re:", and a very delicate
compromise was reached. I am relieved to see that Russ has left it
strictly alone.

But "cmsg" is now pretty well dead on the real Usenet, which is why simply
confirming its death is probably enough. I am with Frank on this one, that
the principle of "don't tinker with Subject" should be regarded as more
important here.

>a subject line could lead to poor propagation.  Don't do it.

Eh? How can rejecting an article cause it to propagate worse?

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 18 Dec 2006 19:54:30 GMT

>like RFC 1855, and weird body conventions like tear lines.

During the short time that we did discuss USEAGE, we did agree that we
should adopt everything in GNKSA, except where it was obviously outmoded,
or where we expclictly decided to be different (which we did for some
cases and may yet do for more). The current USEAGE draft reflects that.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 18 Dec 2006 20:04:53 GMT

>Charles Lindsey <[email-masked]> writes:

BTW, you spent a lot of time justifying cases which I had already marked
with [+1], indicating my agreement (though other might want to disagree,
so needed to be made aware of them).

>Intentional, but I don't object to changing it.

OK. My vote is still [-1].

>option, namely the FQDN of the news server itself.

Which I did not quite understand. Did you mean the A or AAAA record? In
which case please be explicit. But I would prefer to allow CNAMEs and MXes
as well.

>MISMATCH entries or send you articles that you sent them.

OK, I will buy that.

>hard to tell a site how to guarantee that ...

That's the site's problem :-( .

>identities should be in all lowercase...

Yes, but please add "(since some current usage treats them
case-insensitively)".

>What injecting agent supports this?

INN apparently (see below).

Newsreaders such as rn, trn, nn, and others of that generation had (and
still have) the capability to interact directly with the newspool (either on
the same host, or more likely NFS mounted from some server) rather than
going via NNTP (which did not exist when they were first written). They
injected articles by calling a program 'inews' which, in the absence of an
explicit From:, assumed the poster was the user who had called 'inews'.

I have been reading and posting to Usenet for around 20 years now, and I
have never yet configured my newseader with what I wanted in my From: line
(except when doing tests with upstart agents such as netscape :-) ). I
don't suppose I am the only one. I suspect that 'inews' still exists on a
lot of systems.

It certainly comes as part of CNews. 'nn' comes with a version of 'inews'
that invokes NNTP with a POST command, but it still checks to see whether
a From: header has been provided, and goes looking in /etc/passwd if not.
And Stan Barber's model implementation of NNTP, when it comes to
implementing the POST command, looks for a program 'inews' on the host it
is running on (though of course it doesn't expect automatic deduction of
From: in that situation).

And now I find that INN seems to have an 'inews'. I just found that the
University offers an 'inews', and I just posted an article with no From:
header using it, and read it back:

kiss% telnet wapping 119
Trying 130.88.198.1...
Connected to wapping.
Escape character is '^]'.
200 cs.man.ac.uk InterNetNews NNRP server INN 2.3.3 ready (posting ok).
group man.test
211 1 1680 1681 man.test
article 1681
220 1681 <[email-masked]> article
Path: cs.man.ac.uk!chl
From: [email-masked] (Charles Lindsey)
Subject: testing inews
Date: Mon, 18 Dec 2006 14:08:25 +0000 (UTC)
Organization: School of Computer Science, University of Manchester, U.K.
Lines: 2
Message-ID: <[email-masked]>
NNTP-Posting-Host: kiss.cs.man.ac.uk
X-Trace: wapping.cs.man.ac.uk 1166450905 2390 130.88.192.64 (18 Dec 2006
14:08:25 GMT)
X-Complaints-To: [email-masked]
NNTP-Posting-Date: Mon, 18 Dec 2006 14:08:25 +0000 (UTC)
Xref: cs.man.ac.uk man.test:1681

called /opt/cs/bin/inews from kiss.
No From line, but it was from Charles Lindsey <[email-masked]>

You will see that it has constructed a suitable From header (that is my
correct email address when I am inside the Manchester firewall).

The wording I wrote in USEPRO was

   "... (and even From if the particular injecting agent can derive that
information from other sources)".

That indicates pretty clearly that it would be an unusual possibility,
but fine if you happen to have it. It ain't broke ...

>....  This doesn't allow for injecting agents that replace the From
>something that we want to support.

Sure, and the wording in USEPRO gives no support whatsoever for that
practice.

>Intentional change, but I'm dithering.

One might solve the problem by just saying "fold it so that no header line
is more than 998". However, that is not robust enough.

There was a case on this list a few years back (yes, that was email, but
the problem is the same for both). Some thread got so long as to run into
the 998 limit. This is essentially a problem for user agents, not servers,
and some people's MUAs were clearly folding the huge References headers
in a sensible manner. But it was apparent that other MUAs were first
unfolding them, adding the extra entry, and then hopefully refolding them
again. And some agent(s) out there was evidently trying to do all that in
a buffer of 998, and when it overflowed it just truncated things at that
point (or somesuch - I forget the precise details). So you ended up with
an invalid References header which was then sent forth to the list. And
when that reached my 'nn' (I gateway this list into news), it positively
refused to display the article at all (so I could not even see what was
wrong). I had to go to my newspool and hand-edit the References header
into somethng legal (and chop a lot out of it at the same time), and then
it was fine.

>errors back to a client.

I would regard an injecting agent that was so detached from its clients as
to be unable to report back to its clients as irretrievably broken, hence
making the system hopelessly inadequate for its users (see my remarks in
response to your "philosophy" claims). Surely that is just the sort of
situation that "SHOULD" is designed for (I agree that if you are
communicating with your injecting agent via carrier pigeon, there is no
way to get the bird to fly back, so "MUST" would be too strong).

Count me as [-1] on this.

>        to a moderator for some reason, it MUST be rejected.

OK, the wording could be better. How about:

   7.  If the Newsgroups header contains one or more moderated groups and
	the proto-article does not contain an Approved header field, the
	injecting agent MUST either forward it to a moderator as specified
	in Section 3.4.1 or, if that is not possible, reject it.

>        its <path-identity> followed by "!"; one or the other of these

OK, so it's a matter of wording. I think something based on my USEPRO
would be better, such as

   9.  The injecting agent MUST then prepend to the content of the Path
	header field the its own <path-identity> followed by a "!", which
	SHOULD then be followed by a ".", the <diag-keyword> "POSTED" and
	a further "!".

I just noticed there that you also had an optional FQDN or IP address
after the "POSTED". I don't think that was ever the intention of what we
discussed during USEFOR, but as a matter of wording it could easily be
added to the revised text.

>The language is phrased similarly for relaing and serving agents.

Yes, similar changes are needed there, and I would suggest looking again
at my USEPRO version. The essential difference in those cases is that I
described the stuff to be added from left-to-right (which is how I think
people will tend to read it), so that you do the obligatory
<path-identity> first, and then the various possibilities for the
diagnostics.

>Changing the <path-diagnostic> from a MUST to a SHOULD is an intentional

Yes, I am happy with that (the latest USEPRO left it open for discussion).

>Intentional change.

I am neutral on this one. Let others decide.

>very murky to me.

USEFOR allows a lot of ways for filling in the Injection-Info. I just
don't want to see a single injecting agent, on successive articles,
hopping around at random between various forms of posting-host,
posting-account and logging-data just in order to force its way past my
killfile.

>few sites are ever going to care.

I was more concerned about 'local' than I was about 'world' (which is
unnecessary, though it was seen at one time). If you want to rely on the
wording in USEFOR, then please add something like:

    "See [USEPRO] section 3.2.4 for the significance of the reserved
<dist-name>s "world" and "local".

>multiple peers and don't do this.  This is a core part of the flood-fill

OK, you have persuaded me. And it is also needed too to check for stale
Injection-Dates, which I _did_ have in USEPRO.

>        control message or Supersedes header field.

Agreed, but my concern was that this was a matter for serving agents
rather than for relaying agents. If you aren't actially storing the
article, then there is little point in trying to implement cancels. Just
relay it on, and let the agents that _do_ store it decide whether to
honour the cancel or not. Do current relay-only agents currently bother to
do this?

>local serving.

Well if that is the case it was nominally the serving function that added
the Xref (section 3.6). Now whether such an agent first stores the article
and then relays what it has stored, or whether it relays first and then
stores after is none of our business. So if you were just trying to cover
the first of those cases, where the new Xref might then escape by relaying,
then I think it could be worded better. How about:

   Any Xref header, whether present on input or added by an associated
   local serving agent, MAY be deleted before relaying.

>Intentional change.  See previous discussion.

I remain neutral [00].

Alert! Duplicated issue number #21!

>       relaying agent (Section 3.5).

OK, point taken, but it might be better split out. Moreover, the first bit
corresponds to step 2 of relaying which is a MUST, as is the matter of
stale articles (which is also a history file matter).

>newgroup, rmgroup, or checkgroups control messages.

OK, but people will think it odd that you mention the obscure case of
cancels which arise first here, but make to mention of the common case.
Perhaps an extra step to say that it SHOULD then process any control
messages (including cancels) in acccordance with section 5.

>Intentional change.

Frank and Forrest have commented on this. Seems controversial. I remain
neutral for now.

>Intentional change.

Seems controversial.

>*can't* change the Content-Transfer-Encoding,...

Sure, but it may be helpful to draw attention to it. I remain [00].

>> Application types are inherently dangerous ...

>description.  In and of itself, it has no force, power, or action

Application types are intended to cause applications to be invoked, with
possible side effects. In this case, the side efect is that a line gets
added/changed in the Newsgroups file. Fine when it is part of a newgroup
message that you intend to honour, but not anywhere else without explicit
human intervention. Likewise for application/checkgroups.

>prohibited from using application/news-checkgroups for that mail message.

OK, in that case what you need to say is:

    This media type is intended to be used in conjunction with the
    newgroup control message as described in section 5.2.1. It MUST NOT be
    automatically invoked in any other context without explicit human
    intervention.

and similarly for checkgroups.

Alert! Duplicated issue number #27!

>Intentional change, but after looking at the INN code, I was wrong.

OK.

>Intentional change.

Others have commented, so controversial. I remain [00] for now.

>It's just as important for checkgroups as it is for newgroup.

Point taken.

>> Surely, it SHOULD identify the person/identity/role of the issuer, ...

>(preferrably in combination with a digital signature).

It asserts that this is an "authorized" message (just what "authorized"
means is a separate issue which I shall raise later). As such, it is
pretty pointless unless it identifies the entity that is authorized, and
all news servers that I know of can be configured with the identity that
is to be recognized for each hierarchy. Of course, it really needs to be
digitally signed to make it effective, but that again is a separate issue.

I think the WG early on took the view that the Approved header would
simply fall into disrepute if one could always get away with
   Approved: kibo
and that it therefore needed some firmer language (digitally signed in due
course). I remain of that view.

>> Was MUST ...

>without descriptions work fine with existing software.

OK, if absence means that it just creates a Newsgroups file entry with
just the name of the newsgroup, and maybe the TAB, and certainly the
" (Moderated)" if needed. Likewise for checkgroups. So you need something
like:

    The application/group-info MUST be included if the group is moderated,
    in which case it MUST include a <moderation-flag>. For other groups, it
    MAY be omitted if there is no <newsgroup-description> to be provided.

>Unintentional change.

OK.

>Intentional change.  See previous discussion.

Then it needs to say:
    and SHOULD contain ... unless no definitive list for such
    sub-hierarchies is available.
which, for a sufficuently disorganized alt-like hierarchy might well be
the case.

>for newgroups, this is actually necessary to honor the checkgroups.

OK, I seem to have misread a MUST somewhere. Objection withdrawn.

>Intentional change.

Frank seems to disagree. So do I, so controversial I think.

>different than normal articles in this regard.

I think the point is that we have made a change from RFC 1036 here. If you
reworded it as:

    Contrary to [RFC1036], the From and Sender fields SHOULD contain the
    address of the issuer of the cancel, as opposed to that of the poster
    of the cancelled article. That former requirement only ...

then I would be happy. s/SHOULD/should/ if you like.

>it's only a SHOULD NOT to omit it.)

I think the syntax you need, if you don't want to do it my way, is

    Ihave-command    = "ihave" Ihave-argument
    Ihave-argument   = 1*WSP *( msg-id 1*WSP ) relayer-name

and then say somewhere that , if there is a list of <msg-id>s in the body,
there MUST NOT be any <msg-id>s in the Ihave-argument (ditto for sendme).

The reason I did it the way I did was because it seemed, even at the time
Henry was writing, the practice of putting them in the Ihave-argument was
already long dead, and there was little reason even to continue supporting
it.

>> the same 'wire' as the ihave and sendme messages ...

>does a discussion of dot-escaping in NNTP.

If we are allowed to specify the format of news articles 'on the wire'
(which is what USEFOR is all about), then why on earth cannot we specify
the format of the only other thing that is ever put on the same wire? It
is not the method of transport we are specifying, but the format of the
data - otherwise there will be no interoperability.

>          while the named relayer does not pass it to anyone.

And USEPRO has much the same.

I think the minimum that should be said is that, syntactically, this funny
newsgroup-name consists of "to." followed by a <relayer-name> (we have
already said what <relayer-name>s are), and that control messages using
this notation MUST NOT be propagated to any site other than the sending
site and the <relayer-name> specified. So we leave it to the implementor
to devise how he causes it to propagate to the one site it is allowed to
propagate to (but we have already offered a strong hint that this is all
designed for use with UUCP).

>MUST NOT is unwarranted.  Sending them doesn't break anything.

Except cause mailbombs, which seems like severe breakage to me.

We have forbidden the obsolete headers in USEOR, so why not here?

---

From: Russ Allbery <[email-masked]>
Date: Fri, 29 Dec 2006 13:17:17 -0800

I'm going to start trying to break this message up into separate subjects
per topic.  I'll also try to only respond to places where there seems to
be more to say or some text change that's needed.

Charles Lindsey <[email-masked]> writes:
> Russ Allbery <[email-masked]> writes:

>>> 1. [-1] (2.2) <path-identity> - Possibility to use a FQDN not
>>> resolvable in the DNS.

>> Intentional, but I don't object to changing it.

Actually, I didn't make this change, but both of us misread my text.  :)
This case is still allowed (it's my 2nd point).  What changed was the
wording of the 1st point.

> which case please be explicit. But I would prefer to allow CNAMEs and
> MXes as well.

Those are both allowed by the second option.

FQDN is used without further qualification in USEFOR as well (section
3.2.8).  I think it's a technical term that readers of standards should be
expected to know and don't think it needs further elaboration.  If it
does, USEFOR also needs to be modified.  One of the reasons why I'd rather
leave it as-is is that it's one of those terms where attempting a formal
definition can quickly get off into the weeds dealing with multihomed
hosts, different DNS RRs, and so forth.  But I can see why not clearly
defining a term in a standard could be a problem, so I can be convinced we
need to say something.  (Ideally, maybe there's a definition somewhere
else we can just reference?)

>> them as case-sensitive and they're actually compared
>> case-insensitively, everything works fine.  The opposite is not true;
>> if your Path identity varies in case, servers that compare
>> case-sensitively may add incorrect MISMATCH entries or send you
>> articles that you sent them.

> OK, I will buy that.

The more I research this, though, the more I think that while the above is
true, it's not ideal.  Poking around a bit, it looks like a *lot* of
existing implementations are case-insensitive, and I'm not sure I want to
declare that incorrect.  Accordingly, I now have the following text, which
I think is an improvement.  Please, everyone, let me know if you disagree:

   Some existing implementations treat <path-identity> as case-
   sensitive, some case-insensitive.  The <path-identity> therefore
   SHOULD be all lowercase and implementations SHOULD compare identities
   case-insensitively.

---

From: Russ Allbery <[email-masked]>
Date: Fri, 29 Dec 2006 13:27:36 -0800

> still have) the capability to interact directly with the newspool
> (either on the same host, or more likely NFS mounted from some server)
> rather than going via NNTP (which did not exist when they were first
> written). They injected articles by calling a program 'inews' which, in
> the absence of an explicit From:, assumed the poster was the user who
> had called 'inews'.

inews as distributed with INN or the common news readers is a (part of a)
posting agent, not an injecting agent.  You can see this by observing what
actions it takes and what agent it talks to.  It sends messages to an
injecting agent via POST and expects that agent to do the injection; it
doesn't perform those actions itself and then send messages directly to
relaying and serving agents the way that an injecting agent does.

Our agent distinctions are largely based on how INN and similar news
servers work.  C News, being a much earlier implementation, may not have
the same distinctions, or they may be much less clear.  Some
implementations written prior to our standard will combine different
agents into one program, so it's possible that in C News inews combines
functions of a posting agent and an injecting agent.  However, I believe
you'll find that treating creation of the From header field as a function
of the posting agent, which is possibly combined with an injecting agent,
will work correctly and not contradict the work flow of any existing
implementation.

INN's inews used to redundantly perform a few (although not all by any
means) of the functions of an injecting agent, such as mailing posts to
the moderators of moderated newsgroups.  This behavior was a bad idea,
causing various problems in practice, and has been removed from current
versions.

---

From: Russ Allbery <[email-masked]>
Date: Fri, 29 Dec 2006 13:31:17 -0800

>> This section says more than just that, and I think the change makes
>> more sense with more context:

> 	in Section 3.4.1 or, if that is not possible, reject it.

Yup, that looks better to me as well and it's even shorter.  Changed in my
version.

---

From: Russ Allbery <[email-masked]>
Date: Fri, 29 Dec 2006 13:42:26 -0800

>> Because the same piece of software also implements a serving agent,
>> which has to add the Xref header, and the same code path is used for
>> all articles received by the server whether they are for further
>> relaying or local serving.

> Well if that is the case it was nominally the serving function that
> added the Xref (section 3.6).

No, because changes made by serving agents would only be seen by posting
agents.  Serving agents do not sent messages to any other news server.
Only relaying agents do that.

> Now whether such an agent first stores the article and then relays what
> it has stored, or whether it relays first and then stores after is none
> of our business.

Our agent description describes the flow, and doesn't allow for this,
conceptually.  This is true even in the current USEPRO draft.

   An "injecting agent" takes the finished article from the posting
   agent (often via the NNTP "POST" command), performs some final checks
   and passes it on to a "relaying agent" for general distribution.

   A "relaying agent" is software which receives allegedly compliant
   articles from injecting agents and/or other relaying agents, and
   possibly passes copies on to other relaying agents and "storage
   agents".

   A "storage agent" receives an article from a relaying agent and files
   it in a "news database". It also provides an interface for reading
   agents to access the news database.

The progression of an article is:

    posting -> injecting -> 1* relaying -> serving -> reading

Articles do not go to serving agents and then back to relaying agents.
This is, admittedly, the only place where it really matters from a
protocol standpoint, but it affects the wording in subtle ways in various
other places and I'd rather keep the clarity.

Because of this, the protocol description has to say that the relaying
agent may add an Xref header, since otherwise nothing in the protocol
permits the Xref header to appear in the article until it reaches a
serving agent.

Few news implementations have a true serving agent as defined in our
document.  INN, for example, splits that function between innd and nnrpd.
Diablo is the only implementation I'm familiar with that has a pure
serving agent as a separate service, although I think the Highwind
commercial servers may also have an equivalent.

---

From: Russ Allbery <[email-masked]>
Date: Fri, 29 Dec 2006 13:51:15 -0800

> OK, point taken, but it might be better split out. Moreover, the first
> bit corresponds to step 2 of relaying which is a MUST, as is the matter
> of stale articles (which is also a history file matter).

Good point.  So much for trying to save a bit of length.  :)

I now have:

   3.  It MUST reject any article that has already been successfully
       sent to it, based on the Message-ID header field of the article.
       To satisfy this requirement, a relaying agent normally keeps a
       database of message identifiers it has already accepted.

   4.  It SHOULD reject any article that matches an already-received and
       honored cancel message or Supersedes header field following the
       same rules as a relaying agent (Section 3.5).

which is basically the same text as for relaying agents.

> messages (including cancels) in acccordance with section 5.

This would need to be added to relaying agents as well.  I don't feel like
it's really necessary, but I don't object if others in the WG would like
it to be added.  (I think that's MAY, not SHOULD; many news servers don't
process control messages at all and unless they want to honor cancels,
there's no need for them to do so.  There are numerous other ways to
maintain an active newsgroup list.)

---

From: Russ Allbery <[email-masked]>
Date: Fri, 29 Dec 2006 14:03:02 -0800

> possible side effects.

Could you provide a reference for this?  I don't see this statement in RFC
2046.  The closest that it has is:

   The "application" media type is to be used for discrete data which do
   not fit in any of the other categories, and particularly for data to
   be processed by some type of application program.  This is
   information which must be processed by an application before it is
   viewable or usable by a user.

"To be processed" is far from saying that reception of an application type
will cause an application to be invoked.  application/octet-stream is an
obvious counter-example to this interpretation.

Our case falls into the general scope ("discrete data which do not fit in
any of the other categories") but not fully into the specific one ("data
to be processed by some type of application program").  Our media types
are actually text/* types; the only reason why they're not declared as
such is because text/* comes with baggage involving CTEs that we can't
accept.  Declaring them as application/* is in essence a hack around the
MIME object structure, although unfortunately a necessary one.

> In this case, the side efect is that a line gets added/changed in the
> Newsgroups file.

I don't agree with this interpretation, and I don't see anything in the
text of the application/news-groupinfo registration that reflects this.
It sounds to me like you're confusing the newgroup control message with
the application/news-groupinfo MIME media type.  They're not the same
thing.

> and similarly for checkgroups.

I disagree with adding anything like this to our document.  When we just
defined the media type to contain simple text and nothing in the
description of the media type says anything about treating that text as
instructions to a computer program, this seems to me to just be confusing.
There's nothing here to invoke; it's just an association between newsgroup
names and descriptions.

---

From: Russ Allbery <[email-masked]>
Date: Fri, 29 Dec 2006 14:14:00 -0800

>> The content of the Approved header serves no protocol purpose, and
>> USEFOR already adequately covers the definition of its content.
>> Control message authorization is done on the basis of the Sender or
>> From header (preferrably in combination with a digital signature).

> means is a separate issue which I shall raise later).

The description in USEFOR is:

   The Approved header field indicates the mailing addresses (and
   possibly the full names) of the persons or entities approving the
   article for posting.  Its principal uses are in moderated articles
   and in group control messages; see [I-D.ietf-usefor-usepro].

The Netnews protocol currently does not deal with authorization at all (an
obvious flaw noted in Security Considerations).  Any authorization
information you want to use has to be derived from the underlying
transport protocol or from unstandardized extensions such as digital
signatures.

> As such, it is pretty pointless unless it identifies the entity that is
> authorized, and all news servers that I know of can be configured with
> the identity that is to be recognized for each hierarchy.

If you're referring to group control mesages, said identity is checked
against the *From or Sender* header field, not the Approved header, at
least in INN.  INN ignores the contents of the Approved header.  I don't
know if C News uses the contents of the Approved header field for control
message permissions, but my impression from having maintained control.ctl
for some years is that most everyone uses From/Sender.

> Of course, it really needs to be digitally signed to make it effective,
> but that again is a separate issue.

> course). I remain of that view.

We can add firmer language when there's some meaning to the contents, such
as when digital signatures are added.  Right now, placing any stricter
requirements would contradict existing practice and be pointless since
there's no possible protocol check on the contents of Approved.

---

From: Russ Allbery <[email-masked]>
Date: Fri, 29 Dec 2006 14:20:06 -0800

>> Newsgroup descriptions are optional in the protocol and newgroup
>> messages without descriptions work fine with existing software.

> " (Moderated)" if needed.

Its absence may mean that no newsgroups file entry is created at all.  The
newsgroups file is an optional feature of NNTP and need not be
implemented, nor need the newsgroups file be accurate or complete.  See
section 7.6.6 of RFC 3977:

   The list MAY omit newsgroups for which the information is unavailable
   and MAY include groups not available on the server.  The client MUST
   NOT assume that the list is complete or that it matches the list
   returned by LIST ACTIVE.

Newsgroups are not required to have descriptions unless you want to send a
checkgroups for them.  Some Netnews networks (such as the servers on which
microsoft.* is hosted) simply don't use them.

> Likewise for checkgroups.

I don't see what change you are indicating for checkgroups.  The body of a
checkgroups message can't meaningfully be empty.

> So you need something like:

>     MAY be omitted if there is no <newsgroup-description> to be provided.

Why do we need something like that?  There's a perfectly usable moderation
flag in the Control header field.

---

From: Ned Freed <[email-masked]>
Date: Fri, 29 Dec 2006 15:54:05 -0800 (PST)

> 2046.

That's likely because the invokation of applications is an implementation
artefact rather than a core feature of MIME. In particular, some (but by no
means all) handle different media types through an application dispatch
mechanism. RFC 1343 even defines a specific format to use for such information
- the so-called mailcap format. This is, however, an informational RFC, not a
standard, and not only is there no requirement that such a mechanism be used,
lots of implementations don't work this way.

> The closest that it has is:

> obvious counter-example to this interpretation.

Right. Furthermore, there's nothing that says video, image, audio, message or
model types are exempt from processing and the generation of side effects. For
that matter, it may be reasonable in some cases to handle text types by
dispatching to an external application, e.g., in a voice mail system.

> MIME object structure, although unfortunately a necessary one.

Yes, and while there have been various proposals to define an additional
top-level type or types to deal with the baggage associated with text, none of
them have received much support so far.

				Ned

---

From: Russ Allbery <[email-masked]>
Date: Fri, 29 Dec 2006 18:01:59 -0800

I'm not sure the most effective way to respond to this analysis.  I
disagree with many points of Charles's message in one way or another, and
I'm afraid that trying to discuss it line-by-line will just result in a
muddle that will be confusing and too hard to follow.  I think there are
multiple differences of philosophy, definitions, analysis, and use cases
here.

So, what I'm going to try is to write up a description of how I see
reinjection in general and the issues that we need to deal with (and *not*
deal with) here as a coherent document.  Then, hopefully, we can have a
working group discussion comparing and contrasting my view of this with
the one Charles presented and find some solution that takes into account
all the concerns.

First, one issue that's *not* directly part of this.  Even if we keep the
current USEPRO method, we need to have a staleness check on the Date
header field at injection.  We can't make a single leap to allowing stale
Date header fields from not having any Injection-Date header field since
most existing software is going to ignore the Injection-Date header field
entirely and impose staleness checks on Date after injection.  Therefore,
an article with a stale Date is going to propagate very poorly, and
accepting such an article just to let it be dropped silently later by most
of the Netnews network would be a poor interoperability choice.  Far
better to reject the message at the injection point and make the user
replace the Date header field.  We can lower the requirement to SHOULD
from MUST because of Injection-Date, but not drop it entirely.

Now, let's look at the conditions under which this situation arises.

First, note that multiple injection (sending the same proto-article to
multiple injecting agents) is explicitly allowed for and described in our
drafts.  This is the best way to deal with most issues that would
otherwise give rise to reinjection.  For example, most small home servers
are better implemented by proxying local posts to (potentially multiple)
injecting agents rather than injecting the article into one's own local
Netnews server and then gatewaying it later.  A server that has
ineffective relaying relationships and more effective posting access can
inject posts both locally and at remote servers simultaneously and the
result would be much cleaner.

In other words, in general, confusing injecting and relaying or using them
as interchangeable substitutes for each other is a bug.  They do different
things and confusing and strange things can happen when they're not kept
distinct.  Declaring such behavior non-comformant is a feature.  The INN
code to permit people to inject posts via IHAVE is something I committed
only under protest, and I'm entirely content to have its behavior declared
non-comformant by a standard.  It is a *very* specialized configuration
for strange cases where the people involved know just what they're doing.

That being said, there are disjoint Netnews networks that people want to
connect, and multiple injection or proxying injection isn't always the
best approach (for one, it requires a real-time connection with the remote
injecting agent).  My definition of two disjoint Netnews networks is two
Netnews networks between which there is no relaying agent to relaying
agent connection.  Significant examples are:

 * The many people who do not have any relaying agreement with a large
   Netnews network such as Usenet, only a reading agent relationship, but
   still want the features of a full-fledged news server (longer article
   retention, multiple users sharing the same spool, faster service to
   reading agents, local newsgroups without requiring clients use multiple
   servers, off-line operation).  This has become increasingly common over
   the last five years for both personal and small organization use.

 * Ad hoc connections to a stand-alone news server that serves only a
   particular hierarchy.  Relaying is always better for setting up clean
   connections between news servers with the same hierarchy, but sometimes
   people don't want to be bothered to set up a relaying connection or
   (more ethically questionably) people decide they want to propagate in a
   larger Netnews network (like Usenet) groups that the providers don't
   want to propagate.

 * Broken software that only supports relaying and not injecting but which
   can't be trusted with a true relaying agreement.  This is the case that
   sparked the IHAVE support in nnrpd.

The goal is to handle these cases with a solution that has the
following properties:

 * Netnews loop detection and prevention is preserved.

 * Articles are stamped with trustworthy injection information.  Since,
   particularly in the first and third cases, the reason why no relaying
   agreement is in place is precisely because the original injection
   information can't be trusted, this requires replacing injection
   information when the message is gated from one Netnews network to
   another.

 * The protocol is clear about who does what and who is responsible for
   what in these situations.

 * This edge case does not unnecessarily complicate the regular protocol.
   In particular, I want to avoid making agents do complex things to allow
   for possible reinjection when reinjection is a rare case that most
   agents don't have to be concerned with.

My realization when reviewing the draft was that all or nearly all of the
cases where this arises and is not simply a bug (such as treating relaying
and injecting as interchangeable) can be meaningfully treated as disjoint
Netnews networks.  The case which Charles raises of a news server that
acts as a relaying agent to one server and a posting agent to another
server on the same network is an interesting possible exception; it's
rather pathological, but I can see the argument that the end results, if
done properly, are not distinguishable from multiple injection.  But let's
put that aside for a moment and look at the alternatives before us in the
more typical case.

Option 1 (my current draft)

    Proto-articles may not contain Injection-Date.  Injecting agents must
    reject any message that contains Injection-Date.  If an agent needs to
    reinject a message, it must think of itself as a gateway, remove the
    injection header fields (including Injection-Date), perform whatever
    checks it needs to ensure that it's not creating a loop, and then
    reinject the message, at which point it gets a new Injection-Date
    header field with the current date.  This gateway is fully responsible
    for not creating loops.

Option 2 (current USEPRO draft)

    For a given post, an injecting agent may either be doing injection or
    reinjection based on whether injection headers fields are already
    present.  It may decline to do reinjection.  If it doesn't, it may
    rename an existing Injection-Info header to some other name or remove
    it and must retain any existing Injection-Date header field.  It may
    perform a staleness check against the Date header field if no
    Injection-Date header field is present.  (The current USEPRO draft
    doesn't say an injecting agent can perform staleness checks against an
    existing Injection-Date header, but I presume that's simply an
    oversight.)  The injecting agent is responsible for not creating
    loops, not the agent that offers the article for reinjection.

I've already talked about why I think the first option is cleaner and
clearer from a protocol description and implementation standpoint.  Let me
instead focus on failure modes.

The primary risk of the first option is that it could create a slow loop
in the presence of bidirectional gatewaying as follows:  An article exists
in network A.  It is pulled by a reading agent and reinjected into network
B by a gateway which removes the Injection-Date header field.  Later,
another gateway reads the article from network B and reinjects it into
network A, removing the Injection-Date header field again.  In order for
this to occur, enough time will have to have passed between the reading of
the article from network A and the injection back into network A for the
record of the article to have expired, the two gateways would (contrary to
the SHOULD in 3.9) have to have no effective checks against circular
gatewaying, and the Injection-Date header must at each point still be
fresh enough not to trigger any staleness restrictions in either gateway.
Note that this loop cannot at this point continue unless one or the other
gateway also violates the MUST NOT in 3.9.2 against gatewaying the same
message twice; it stops with one duplicate.  (And, currently, the
injecting agent for A would have to not do a staleness check on the
preserved Date header, contrary to a SHOULD, but we're hoping to eliminate
that SHOULD eventually.)

The second option does not have this same failure mode, and indeed it
opens the door to fewer additional problems as long as we still have to
enforce fresh Date header fields.  The failure mode with the second option
comes when we want to drop the staleness check on a Date header field.
Suppose that a stand-alone news server B wants to pull articles from a
Netnews network A for local consumption.  B has a power failure and is
off-line for an extended period of time (a week, say).  When it comes back
up, it wants to catch up on all of the news from A that it missed while it
was down.  Now, as long as we have to enforce Date staleness, it can only
do this by ignoring a SHOULD at reinjection, since those articles will
likely now have stale Dates, but it can be configured to do that.
However, under option 2, catching up on gatewaying has now become
impossible because it is REQUIRED to retain the Injection-Date header of
the original message but that Injection-Date header is now stale and will
therefore be rejected by any relaying or serving agent in B.  Hence,
option 2 makes impossible during reinjection a scenario similar to why we
invented Injection-Date in the first place.

Now, my basic argument, apart from the simplicity of option 1, is that the
failure scenario for option 2 is worse than the failure scenario for
option 1.  I don't consider either to be particularly significant, but
option 2 does fail in a situation that I consider very plausible (if
unusual).  The circumstances for option 1's failure, on the other hand,
are not particularly plausible: the rarer case of bidirectional gatewaying
has to be in effect, we have to already be losing due to lack of
protection from circular gatewaying, *and* this whole thing has to happen
in slow motion.  In any normal circumstances, these gateways are going to
fire within minutes, hours at most, and when the reinjection to A is
attempted, a record of the article will still be present and the article
will be rejected as a duplicate.

Okay, phew.  Now back to Charles's case.

Suppose we have a new server with a relaying agreement to another
unreliable news server and a posting agreement with a much more reliable
news server, and that news server wants to send outgoing articles via both
paths.  Under option 2, the injecting agent that this server talks to must
support reinjection (it's an optional feature in the current USEPRO
draft), must enable it for that client, and then takes responsibility for
doing the appropriate transformation, but Injection-Date is retained.
Under option 1, this news server must either support multiple injection
(far and away the best option) and inject at the remote server at the same
time as the message is injected locally, or it must both relay the message
and gateway it to the remote injecting agent.  The gatewaying is more
complex, but can be done entirely locally and the article will then look
to the remote server like a regular proto-article.  No special support on
the remote server is therefore required; all of the work is born by the
agent doing something unusual.  In either case, two slightly different
copies of the message will exist with different trace headers, but this is
acceptable; the same is true for multiple injection.

I would argue that option 1 is still better for this case.  It's slightly
more work, but it puts the work in the right place and the burden is born
by the agent wanting to do the unusual work.  Furthermore, it's more
likely in practice that option 1 will be *possible*; if the remote
injecting agent isn't willing to set up a relaying agreement with this
news server, it seems unlikely to me that it would be willing to set up an
even more rare reinjection agreement.

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 30 Dec 2006 15:27:41 +0100

Russ Allbery wrote:
 
> FQDN is used without further qualification in USEFOR as well
[...]
> maybe there's a definition somewhere else we can just reference?

Check out RFC 3696 chapter 2.  It doesn't discuss DNS details,
but I don't think we need that for USEPRO.

>    SHOULD be all lowercase and implementations SHOULD compare 
>    identities case-insensitively.

Messy.  No chance for s/SHOULD/MUST/ ?

Frank

---

From: Russ Allbery <[email-masked]>
Date: Sat, 30 Dec 2006 09:10:21 -0800

> Messy.  No chance for s/SHOULD/MUST/ ?

I could see a MUST for the second one.  I'm leery of telling people they
have to change their path identity.  What do others think?