Subject: NEW: USEFOR -04 3.2.11 Xref vs. transport

From: Bruce Lilly <[email-masked]>
Date: Sat, 25 Jun 2005 11:14:20 -0400

RFC 1036 says, regarding the Xref field:

    This is only of value to the local system, so it should not be
    transmitted.

There is no corresponding text in the usefor draft version -04.  Best
current practice seems to be to keep track of local data in a separate
database rather than in a non-transmitted addition to the article
header.  I therefore propose that the following text (subject to discussion
of course) be added to the usefor draft Xref text:

  The optional Xref field has been used as a place to store
  cross-reference information of use to a local system.  Other
  methods of handling such metadata include separate databases.

That is to clarify the semantics of the field.  When we get to the usepro
document, text similar to the following could appear there:

  Because the Xref field is only of use to a particular system, it
  SHOULD NOT be transmitted to other systems.

---

From: Russ Allbery <[email-masked]>
Date: Sat, 25 Jun 2005 09:20:04 -0700

> There is no corresponding text in the usefor draft version -04.

That's because that sentence describes behavior that very few people
actually follow.  It's not a bad idea, but patching the Xref header out of
outgoing articles seems to be considered too much pain for not nearly
enough gain (that's certainly my opinion).

> Best current practice seems to be to keep track of local data in a
> separate database rather than in a non-transmitted addition to the
> article header.

No, Xref is very widely used in practice.  My guess would also be that
some news readers want it in the article proper in order to mark
crossposts already read appropriatley.

> I therefore propose that the following text (subject to discussion of
> course) be added to the usefor draft Xref text:

>   methods of handling such metadata include separate databases.

s/has been/may be/.  Other than that, I don't particularly mind this.

>   SHOULD NOT be transmitted to other systems.

I do object to this.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: Bruce Lilly <[email-masked]>
Date: Sat, 25 Jun 2005 13:29:30 -0400

> No, Xref is very widely used in practice.

Perhaps I was unclear.  I didn't imply that Xref was not used, perhaps
even widely so.  Simply that *best* practice is to put local data in a
local database.

> My guess would also be that 

I have no problem with communicating such information (however it is
locally stored) via an appropriate mechanism (e.g. overview data).

> s/has been/may be/.  Other than that, I don't particularly mind this.

I'd prefer not to confuse readers between the several meanings of
"may".  How do you feel about "might"?
 
> >   SHOULD NOT be transmitted to other systems.

Because...?  (N.B. the BCP 14 meaning of SHOULD NOT is a recommendation,
not an imperative).

---

From: Russ Allbery <[email-masked]>
Date: Sat, 25 Jun 2005 10:36:29 -0700

> even widely so.  Simply that *best* practice is to put local data in a

This depends on what definition you're using for "best."  This is
certainly not the most interoperable practice.

> Because...?  (N.B. the BCP 14 meaning of SHOULD NOT is a recommendation,

Because putting an Xref header in the message is how much Usenet software
works, and I don't see a compelling reason to change that.

---

From: Bruce Lilly <[email-masked]>
Date: Sat, 25 Jun 2005 14:17:59 -0400

On Sat June 25 2005 13:36, Russ Allbery wrote:
> This depends on what definition you're using for "best."

In terms of the principle of keeping strictly local (and therefore variable
between systems) data separate from invariant data (the article content).

> This is 

Since this is local data, how does interoperability apply?

> works, and I don't see a compelling reason to change that.

Security is one issue.  Sending local data exposes that local data,
constituting unauthorized data disclosure.  The threat consequences
may include a threat action of inference, specifically traffic
analysis.  See the security glossary, BCP 36 (a.k.a. RFC 2828).  In
common terms, it's a violation of the security principle of restricting
disclosure of data to those with a "need to know".

Although security is not mentioned (at all!) in RFC 1036, the principle
of not exposing local data espoused in the RFC 1036 text is sound.

---

From: Russ Allbery <[email-masked]>
Date: Sat, 25 Jun 2005 11:21:53 -0700

> Since this is local data, how does interoperability apply?

I already answered that.  I'm fairly certain that at least some news
readers use this data.  There's also Xref-slaving, of course.

I think I've said everything I have to say.  I object to requiring Xref be
removed from outgoing articles for the reasons previously stated, mostly
because it's not worth the trouble and the justifications are not
compelling enough to change existing software, and do not find Bruce's
counter-arguments convincing.

---

From: Bruce Lilly <[email-masked]>
Date: Sat, 25 Jun 2005 14:35:50 -0400

On Sat June 25 2005 14:21, Russ Allbery wrote:
> readers use this data.

That does not affect transmission between servers.  What happens between
a server and a client is a separate matter.

> I object to requiring Xref be
> removed from outgoing articles

Fine.  No proposal for a *requiring* removal has been made, as far as I know.

---

From: Russ Allbery <[email-masked]>
Date: Sat, 25 Jun 2005 11:52:38 -0700

> a server and a client is a separate matter.

I already covered this too.  We're really just repeating ourselves at this
point.

>> I object to requiring Xref be removed from outgoing articles

> Fine.  No proposal for a *requiring* removal has been made, as far as I

I object to saying that removal of Xref is a SHOULD for servers.

(Was it really necessary for you to make me say that?  Was not your
message and this message really a waste of WG attention and bandwidth?
I'm going to now assume that people understand my position on this
proposal without these sorts of small clarifications.)

---

From: Henry Spencer <[email-masked]>
Date: Sat, 25 Jun 2005 15:29:55 -0400 (EDT)

> >   SHOULD NOT be transmitted to other systems.

Concur.  It's a nice idea in abstract, but the real world has never worked
that way, and it's time to stop pretending that it does.  (Even B News,
despite its alleged close relationship to RFC 1036, never did that.)

                                                          Henry Spencer
                                                       [email-masked]

---

From: Henry Spencer <[email-masked]>
Date: Sat, 25 Jun 2005 15:38:03 -0400 (EDT)

> Because...?  (N.B. the BCP 14 meaning of SHOULD NOT is a recommendation,

By author's intent and wide community consensus, SHOULD NOT is an
*emphatic* recommendation, not to be disregarded without specific and
important reasons.  BCP 14 itself calls for using its keywords sparingly,
adding:  "they must not be used to try to impose a particular method on
implementors where the method is not required for interoperability". 

The correct way to make a mild recommendation is not to use a keyword at
all.  (And as noted in previous mail, this particular recommendation would
be commanding the tide not to come in, so it seems pointless.)

                                                          Henry Spencer
                                                       [email-masked]

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Sat, 25 Jun 2005 16:06:51 -0400

>   SHOULD NOT be transmitted to other systems.

If a client obtains an article by Msg-Id instead of newsgroup:number,
it needs the Xref information to properly mark seen/unseen articles,
even if the message is in the CURRENT newsgroup.

So the wording "of use to a local system" is not accurate.  Consider
what you are proposing: clients cannot be expected to grab a message by
ID, and then only have a newsgroups header to go on.  Getting the message
numbers would require getting the entire XOVERs for all the articles
in all the groups listed in Newsgroups.  So servers really MUST serve
out Xref to clients.  It isn't a local system issue.

If the servers have to serve it out to clients, they have
to store it on spool, for all the same reasons I just mentioned.  It
would be very difficult for a server to generate Xref by cross-
referencing a message ID to a database, and then getting all the
newsgroup:number pairings and inserting the data.  Or do you think
that there should be a way to go from message-ID to overview record?
Keeping history is hard enough!

I also don't agree with your suggestion that it is BCP which would
make any improvement in complexity of implementation or CPU load.

I expect that most server software accepting a feed does Path header
slicing and Xref header replacement in the same section of code, using
similar code.  INN works this way.  The work to cut information out of
the header is about the same in both cases, and we certainly can't drop
Path splicing.

Software complexity is not going to be reduced by this SHOULD NOT either.
A SHOULD NOT added to USEFOR is not going to change the necessity
of having Xref stripping capability in the software.  There are going
to be legacy peers around for a long time, and a SHOULD NOT permits
persistence in the behavior even for peers updated to conform.

And as discussed above, the change isn't going to save that many CPU cycles,
and could actually waste them depending on how you decided to store and
strip Xref on the way out to your N peers, and insert it on the way out
to M clients.

A note about how Xref leaks information might be something for USEAGE.
But peers can gather a lot more information about the internals of
your systems in other ways.

---

From: Bruce Lilly <[email-masked]>
Date: Sat, 25 Jun 2005 16:25:28 -0400

On Sat June 25 2005 15:29, Henry Spencer wrote:
> Even B News,
> despite its alleged close relationship to RFC 1036, never did that.

B news source:

file rfuncs2.c:

   /*
    * Transmit file to system.
    */
   transmit(sp, file)

...

        if (index(sp->s_flags, 'A') == NULL)
                hwrite(&hh, ofp);

...

The 'A' flag, incidentally, is only present for A news format.
hwrite is defined as a macro in file header.h:

   #define hwrite(hp,fp)   ihwrite(hp,fp,0)
   #define lhwrite(hp,fp)  ihwrite(hp,fp,1)

note that the 3rd argument to ihwrite is zero.

file header.c:

   /*
    * Write header at 'hp' on stream 'fp' in B+ format.  Include received date
    * if wr is 1.  Leave off sysname if wr is 2.
    */
   #ifndef DOXREFS
   /*ARGSUSED*/
   #endif /* !DOXREFS */
   ihwrite(hp, fp, wr)
   register struct hbuf *hp;
   register FILE *fp;
   int     wr;
...

   #ifdef DOXREFS
        if ( wr ==1 && *hp->xref)
                fprintf(fp, "Xref: %s\n", hp->xref);
   #endif /* DOXREFS */
...

If DOXREFS is not defined, nothing happens with Xref fields.  When
defined (the default), wr (the third argument) is checked, and if and
only if its value is one, the field is written.  As noted, the value
is zero when transmitting an article to a remote system.

---

From: Henry Spencer <[email-masked]>
Date: Sat, 25 Jun 2005 16:45:05 -0400 (EDT)

> is zero when transmitting an article to a remote system.

*If* you use the built-in single-article transmission in rfuncs2.c.  Which
nobody has done since *very* early in B News history, because sending
single articles via UUCP -- or most any other store-and-forward path -- is
grossly inefficient.  You're looking at unused vestigial code.

Almost all real B News sites used batch transmission exclusively.  (Bear
in mind that most B News sites joined Usenet only after batching became
standard practice.)  The B News *batcher* never did Xref exclusion; it
sent a byte-for-byte copy of the on-disk article, Xref and all.

                                                          Henry Spencer
                                                       [email-masked]

---

From: Bruce Lilly <[email-masked]>
Date: Sat, 25 Jun 2005 17:16:41 -0400

> important reasons.

I make no claim about the BCP 14 author's intent or about consensus for a
specific interpretation which is not present in the approved document's text.
Here, verbatim, is what the document actually says regarding SHOULD NOT:

4. SHOULD NOT   This phrase, or the phrase "NOT RECOMMENDED" mean that
   there may exist valid reasons in particular circumstances when the
   particular behavior is acceptable or even useful, but the full
   implications should be understood and the case carefully weighed
   before implementing any behavior described with this label.

That text, indeed the full text of BCP 14, fails to include the words
"emphatic", "specific", or "important", with or without "scare" asterisks.
It says what it says, and I have quoted that section verbatim above.
While I would have been happy to defer detailed discussion of suggested
usepro text until we get to that document, I chose the "SHOULD NOT" phrase
carefully and specifically in the context of the suggested text.

> BCP 14 itself calls for using its keywords sparingly, 

The context of that sentence fragment comes from section 6 of BCP 14,
which reads:

6. Guidance in the use of these Imperatives

   Imperatives of the type defined in this memo must be used with care
   and sparingly.  In particular, they MUST only be used where it is
   actually required for interoperation or to limit behavior which has
   potential for causing harm (e.g., limiting retransmisssions)  For
   example, they must not be used to try to impose a particular method
   on implementors where the method is not required for
   interoperability.

"they" in the quoted sentence fragment refers to "Imperatives", and
"SHOULD NOT" is no imperative.  See
http://dictionary.reference.com/search?q=imperative for definitions,
which include the words "command", "obligation", "an order", "compels
a certain behavior", in contrast to the specified meaning of BCP 14
"SHOULD NOT".  In any event, the phrase "SHOULD NOT" in the suggested
text was used sparingly and where security considerations apply; it
imposes no particular method (i.e. imperative or not, the guidelines
for use have been satisfied).  BCP 14 goes on to note:

7. Security Considerations

   These terms are frequently used to specify behavior with security
   implications.  The effects on security of not implementing a MUST or
   SHOULD, or doing something the specification says MUST NOT or SHOULD
   NOT be done may be very subtle. Document authors should take the time
   to elaborate the security implications of not following
   recommendations or requirements as most implementors will not have
   had the benefit of the experience and discussion that produced the
   specification.

While, as noted, I would have been happy to defer details to usepro, I
have taken the time to elaborate the security implications associated with
the suggested recommendation, and have pointed out that it has not been
posed as a requirement.

> all.

The intent is not "a mild recommendation"; it is intended to convey the
fact that there are (perhaps subtle) security effects and that "the full
implications should be understood and the case carefully weighed before
implementing" transmission of an Xref field containing local system data
to a remote system, precisely as BCP 14 indicates for use of SHOULD NOT.

> (And as noted in previous mail, this particular recommendation would 

Your (unsubstantiated by facts, I note) claim about B news appears to
be a false claim.  Statements dependent on the veracity of a false claim
are dubious at best.  There was in fact a point to the specific text
proposed, and there was a reason why "SHOULD NOT" specifically was
proposed, not "MUST NOT" or "MAY" or "should not", and why I have taken
the effort to clarify mischaracterizations of what has been proposed,
at length and in detail.

So to also answer Russ' question, yes there is a point in clear, precise
communication and specifically there is a point in clarifying that the
suggested text is intended as a recommendation with security implications,
but falls short of an imperative (requirement or prohibition).

As I have said a few times, I would have preferred deferring discussion
of this to usepro, but there now seems little more to say on that part
of the suggestion.  That "little" may include the acknowledgment, here
given, that Russ' specific issues indeed qualify as "valid reasons in
particular circumstances when the particular behavior is acceptable or
even useful", but that does not change my position, and indeed is
consistent with it.

Let us now please return to the suggested text for usefor, where I still
don't know if "might be" is acceptable to Russ.

---

From: Matthias Andree <[email-masked]>
Date: Sun, 26 Jun 2005 00:07:12 +0200

>   SHOULD NOT be transmitted to other systems.

The Xref field is usually (i. e. with all upstream servers I can access,
three different software packages and versions) provided through the
RFC-2980-style XOVER, also in the current NNTPEXT-base draft #27 as
OVER, and is somewhat useful to sort out postings that are excessively
crossposted without looking at the HEAD.

Clients might use it by marking a cross-posted article read in
.newsrc-style files in other groups without searching for the
Message-ID.

About transmission to other systems, I'd expect a receiving system that
uses the Xref header to replace it by its own data. Leafnode for
instance does that, and INN appears to do that, too. I wonder if this
issue is so important as to deserve reglementation or strong
recommendations at all.

-- 
Matthias Andree

---

From: Bruce Lilly <[email-masked]>
Date: Sat, 25 Jun 2005 18:55:53 -0400

On Sat June 25 2005 16:06, Forrest J. Cavalier III wrote:

> ID, and then only have a newsgroups header to go on.

For clarification, as there seems to be continued confusion, the proposal
has been carefully considered; "local system" refers to a server *and*
its clients.  The proposal is to state that Xref is used to store
metadata on a local system, and that there are security implications
involved in transmitting that local data with an article to remote
systems, where "remote systems" means a different server where the field
body -- if the optional field is used on that remote system -- would
necessarily have to be different.

> the header is about the same in both cases, and we certainly can't drop

You missed the critical point: the work to "cut information out of the
header" is zero for Xref if there is no Xref field.  Many servers already
index information by message-id, and I see no reason why this sort of
cross-reference information, which exists when an article is stored
locally, could not be part of such data.

> A SHOULD NOT added to USEFOR

The "SHOULD NOT" was for usepro, not usefor, and exists because of
security issues, not "complexity".  See separate discussions.

> A note about how Xref leaks information might be something for USEAGE.

Security issues related to the protocols should be discussed in the
protocol document; that's not a matter that can be deferred to a mere
informative RFC.

---

From: Frank Ellermann <[email-masked]>
Date: Sun, 26 Jun 2005 16:42:50 +0200

> I object to saying that removal of Xref is a SHOULD for
[...] 
> I'm going to now assume that people understand my position
> on this proposal without these sorts of small clarifications.

FWIW "add me".  Bruce's definition of a "local system" as in
'"local system" refers to a server *and* its clients' doesn't
match my concept of NetNews, where a "leafnode" with more than
one "node" is in theory just another "node".

BTW, that's also the reason why I hunted the word "downstream"
in an earlier draft until it was finally removed.  Bye, Frank

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Mon, 27 Jun 2005 10:58:43 +0200

Ticket #1050 created. Based on the following discussion, I have put it into 
state "no change needed".

--On 25. juni 2005 11:14 -0400 Bruce Lilly <[email-masked]> wrote:

> header.  I therefore propose that the following text (subject to
> discussion of course) be added to the usefor draft Xref text:
>

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Mon, 27 Jun 2005 11:07:58 +0200

--On 25. juni 2005 18:55 -0400 Bruce Lilly <[email-masked]> wrote:

> its clients.

Bruce,

usefor-04 does not contain a definition of "system".

In fact the word is used in 2 different senses: "a system" (something that 
passes articles) and "the news system".

Unusually for you, you're using controversial terms without defining them 
first.

---

From: Bruce Lilly <[email-masked]>
Date: Mon, 27 Jun 2005 09:31:14 -0400

> usefor-04 does not contain a definition of "system".

OK.
 
> Unusually for you, you're using controversial terms without defining them 

I did quote the corresponding text from RFC 1036 at the very beginning of
the discussion.  I repeat it for convenience:

Now many of us would agree that 1036 is somewhat vague, but that specific
text in 1036 obviously hasn't prevented a particular (NNTP) server from
communicating cross-reference information to its clients, nor would the
suggested text for usefor.  Indeed, such issues are outside of the scope
of our work.  The scope of our work does include transmission of message
(article) content from one "site" (used but not defined in the draft) or
"server" (ditto) to another.  Given the context of the discussion (viz.
Xref field), it should be clear that the issue is related to the field
body component
   server-name     =  path-identity

So it appears that the usefor draft does not contain a suitably defined
term.  That's another new issue.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Mon, 27 Jun 2005 17:21:57 +0200

--On 27. juni 2005 09:31 -0400 Bruce Lilly <[email-masked]> wrote:

> "server" (ditto) to another.

Now, that's a surprising idea....

>From the charter:

> 1. Produce an Internet Draft (or series of drafts) that describes the
>   core standards for a Usenet article and the features that all Usenet
>   software should take account of.

If you want to argue that Usenet readers are not Usenet software, and 
therefore what goes on between the reader and the server (or "agent" as the 
I'm sure that it would make our work simpler, but in this case, I'm pretty 
sure you're not reflecting a consensus view.

Article formats passed in standardized protocols between a client ("posting 
agent" or "reading agent") and a server ("injecting agent" and "serving 
agent") are within scope.

                        Harald

---

From: Russ Allbery <[email-masked]>
Date: Mon, 27 Jun 2005 09:27:35 -0700

> suggested text for usefor.

Mostly because everyone ignored that part of RFC 1036.

---

From: Bruce Lilly <[email-masked]>
Date: Mon, 27 Jun 2005 13:26:42 -0400

> >   software should take account of.

Something seems to be wrong with the attribution...
 
> If you want to argue that Usenet readers are not Usenet software

I don't.

> agent") are within scope.

That's nice.  Are you suggesting that:
o a posting agent should generate and transmit an Xref field?
o an injecting agent should generate and transmit an Xref field?
o there is some purpose in one "serving agent" sending an Xref field to
  another "serving agent"?

---

From: Russ Allbery <[email-masked]>
Date: Mon, 27 Jun 2005 11:00:05 -0700

> o there is some purpose in one "serving agent" sending an Xref field to

As I believe I previously mentioned, there is indeed some purpose in one
serving agent sending an Xref field to another serving agent, namely Xref
slaving.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 27 Jun 2005 19:50:31 GMT

>of course) be added to the usefor draft Xref text:

Gentlemen,

This is a protocol issue, and therfore it belongs in USEPRO, and most
certainly not in USEFOR.

And USEPRO already says what you want.

Under Duties of a Relaying Agent:

   8. It MAY delete any Xref header that is present.

Under Duties of a Serving Agent:

   7. It MUST remove any Xref header (F-3.2.10) from each article.  It
      then MAY (and usually will) generate a fresh Xref header.

I see little merit in upgrading that MAY to a SHOULD (no interoperability
issue or harm arises). It is a good practice, and to be commended as such,
which is why that MAY is there.

>  methods of handling such metadata include separate databases.

And I take issue with your use of the term "local system". I cannot accept
that currently used terminology would equate a server plus all its
thousands of clients scattered around the globe with a "local system".

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Russ Allbery <[email-masked]>
Date: Mon, 27 Jun 2005 19:30:27 -0700

>       then MAY (and usually will) generate a fresh Xref header.

Please allow for Xref slaving.  (I don't know if the context takes care of
that.)

> I see little merit in upgrading that MAY to a SHOULD (no
> interoperability issue or harm arises). It is a good practice, and to be
> commended as such, which is why that MAY is there.

I think the lack of an Xref header would be surprising to clients, so I
would lean towards SHOULD personally, but I haven't really thought about
it at any great length.

> And I take issue with your use of the term "local system". I cannot
> accept that currently used terminology would equate a server plus all
> its thousands of clients scattered around the globe with a "local
> system".

Now if you'd just apply that same argument to "cooperating subnet"...  :)

---

From: Bruce Lilly <[email-masked]>
Date: Tue, 28 Jun 2005 10:04:45 -0400

>    8. It MAY delete any Xref header that is present.

If you fail to understand not only that "SHOULD NOT" and "MAY" differ,
but that the meanings under BCP 14 differ markedly, you shouldn't be
editing I-Ds intended for the Standards Track.

If you do in fact understand the implications, then please stop wasting
everybody's time by promulgating lies; "MAY" != "SHOULD NOT".  Such lies
are not conducive to rational discussion and are frankly insulting.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 28 Jun 2005 18:22:17 GMT

>Please allow for Xref slaving.  (I don't know if the context takes care of

I think it is OK. If you have a farm of serving agents that keep their
Xrefs in step, do you regard passing articles between those agents as
"relaying" or as some "private protocol" for their mutual cponvenience?
IOW, is that whole farm just "one big serving agent" from the POV of our
draft?

If the answer is "relaying", then the present "relayers MAY delete"
suffices. If it is "one big serving agent", then the question does not
arise. So I think the present USEPRO wording is OK.

If you want some explanatory wording, or a NOTE, then it could be done,
but I don't think it is necessary.

>would lean towards SHOULD personally, but I haven't really thought about

Do you want us to say that is is a required part of the protocol that
serving agents MUST/SHOULD construct an Xref header? I think we all agree
that the Xref header is a bit of a kludge, though it is a very well
established kludge. Changing the wording would make it harder to introduce
innovative solutions as to how a server communicates with its clients.

---

From: Russ Allbery <[email-masked]>
Date: Tue, 28 Jun 2005 19:46:27 -0700

Charles Lindsey <[email-masked]> writes:
> Russ Allbery <[email-masked]> writes:

>> Please allow for Xref slaving.  (I don't know if the context takes care
>> of that.)

> "relaying"

Yes.

> IOW, is that whole farm just "one big serving agent" from the POV of our

No.  There are reasons why you might want to Xref slave to a news feed
that isn't at your site or even run by you.  They're rare, but there's no
particular reason to disallow it.  (Consider, for example, the case where
you run a personal local news server as basically a cache for your ISP's
remote news server and want to be able to switch between servers whenever
you feel like it without renumbering.)

> suffices.

Except that the text you included says:

This is wrong for Xref slaving.  The serving agent does not remove the
Xref header when Xref slaving is being used.

>> would lean towards SHOULD personally, but I haven't really thought
>> about it at any great length.

> serving agents MUST/SHOULD construct an Xref header?

That would be my inclination, yeah.

> I think we all agree that the Xref header is a bit of a kludge, though
> it is a very well established kludge.

Eh.  I could argue that both ways.  It's a kludge from a particularly
purist standpoint of data separation, but it's not clear to me that that
degree of purity is actually useful or desireable in practice.  On the
gripping hand, it's definitely a layering violation.

Anyway, whatever we might think of it, it's not going anywhere.

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Wed, 29 Jun 2005 08:29:45 -0400

> Xref header when Xref slaving is being used.

Well, is that a serving agent or a relay/transport agent?    What
if someone does a SUCK feed and Xref slaving.  Does that make the server
a relay agent?  Does a SUCK feed count as a feed in USEPRO?

As you know, I don't like all the artificial distinctions between
serving/relay/injection agents.

If anything is said, I think it should be a minor note under security
in USEAGE.  I don't think use of RFC2119 language is justified at all,
because there is no interoperability at stake, and the amount and importance
of additional information leaked across a feed in Xref is really, really small.

Downstream sites can already determine how much delay you have
internally, what feeds you get, who you get them from, how complete your
spool is.  So Xref only lets them discover approximately how old your
installation is, modulo your filters, and what newsgroups you carry.
Big deal.  They can look in Whois and look at when you filed articles of
incorporation to get a more accurate determination of when you established
They can look in archives and see when articles with your site name in Path
started appearing.

And you can probably ask them what groups they have.  They will tell you.

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Wed, 29 Jun 2005 08:37:51 -0400

> established kludge. 

I am not sure we have basis for a MUST/SHOULD, but how is it a kludge
to be able to mark cross-posts as read?

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 29 Jun 2005 14:42:53 +0200

This chair has already marked ticket #1050 with "No change needed".

Please stop beating the dead horse.

---

From: [email-masked] (Ralph Babel)
Date: Wed, 29 Jun 2005 17:58:53 +0200

Harald Tveit Alvestrand wrote:

> Ticket #1050 created. Based on the following discussion,
> I have put it into state "no change needed".

NACK, Bruce has a point here: while "Xref" is optional,
so is "Organization", but the former may well be removed
completely upon transfer of an article to the next hop,
while the latter may not. Therefore, something to
the effect of "Xref MAY be removed from in-transit
articles" needs to be added to section 3.2.11.

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 29 Jun 2005 13:49:34 GMT

>Xref header when Xref slaving is being used.

Right! I see your problem now. How about:

   7.   It MUST (exept when specially configured to preserve the
        <article-locator>s set by the sending site) remove any Xref
        header (F-3.2.11) from each article.  It then MAY (and usually
        will) generate a fresh Xref header.

>That would be my inclination, yeah.

OK, then I think you should ask Alexey to raise a ticket if you want to
pursue it.

---

From: Bruce Lilly <[email-masked]>
Date: Wed, 29 Jun 2005 12:40:36 -0400

> Xref header when Xref slaving is being used.

Russ, I think I need some clarification:

Are you saying that the "server-name" part remains unchanged?  If not, how
is the end result conceptually different from remove/generate?  Also, if
not, (i.e. the "server-name" is the same), how is this effectively
different (from the POV of any connecting client) from "one big serving
agent"?
 
It's also unclear exactly how this works in practice: suppose servers A
and B participate in "Xref-slaving".  At some point in time, server A
receives a crossposted article (newsgroups X and Y, with article numbers
12345 and 98765 respectively) from somewhere other than B, includes a
field
  Xref: A X:12345 Y:98765
and forwards it to B, which presumably files the article in the same
newsgroups with the same numbers.  Subsequently, server B receives (not
from A; perhaps from one of those thousands of clients from all over the
globe which are connected as clients to server B) an article posted only
to group Y, and files it as article number 98766.  At the same time,
server A receives (from one of its global clients) an article crossposted
to newsgroups X and Y again, which it duly files as articles 12346 and
98766 on A, with an Xref field
  Xref: A X:12346 Y:98766
and forwards to B.  What happens to the very different article in newsgroup
Y on B which is already numbered 98766?

2005-06-29:3

---

From: Russ Allbery <[email-masked]>
Date: Wed, 29 Jun 2005 09:55:19 -0700

> Are you saying that the "server-name" part remains unchanged?

Yup.

> Also, if not, (i.e. the "server-name" is the same), how is this
> effectively different (from the POV of any connecting client) from "one
> big serving agent"?

The whole point of Xref slaving is to try to make things look the same to
the connecting client.  However, the connecting client isn't the only
point of view or the only connection that we have to think about.  It's
completely different from the perspective of the serving agents
themselves.

> newsgroups with the same numbers.

Right.

> Subsequently, server B receives (not from A; perhaps from one of those
> thousands of clients from all over the globe which are connected as
> clients to server B) an article posted only to group Y, and files it as
> article number 98766.

You can't Xref-slave against more than one server at the same time, nor
can you assign numbers locally.  In an Xref slaving configuration, B will
perform its normal injection agent duties and then rather than storing the
article locally will immediately relay it to A *without* assigning article
numbers.  It will then wait for A to send it back with article numbers
assigned before storing it locally.

---

From: Russ Allbery <[email-masked]>
Date: Wed, 29 Jun 2005 10:00:56 -0700

>         will) generate a fresh Xref header.

That sounds better to me, yes.

---

From: Seth Breidbart <[email-masked]>
Date: Wed, 29 Jun 2005 13:05:55 -0400 (EDT)

> newsgroups with the same numbers.

That means that B is the slave of A.

>  Subsequently, server B receives (not from A; perhaps from one of
> those thousands of clients from all over the globe which are
> connected as clients to server B) an article posted only to group Y,
> and files it as article number 98766.

That would make B a peer of A, not a slave.  Slaves don't get to
assign article numbers.

Seth

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 30 Jun 2005 19:20:01 GMT

>different (from the POV of any connecting client) from "one big serving

I think others have answered your main point. However, even if B is a pure
slave of A, it _might_ still be reasonable for articles passed to B to
change the <server-name> to B, even if they retain the identical
<article-locator>s.

Actually, the wording does not preclude that, since B can always claim to
have removed the old Xref header and replaced it with one that was almost
identical. Indeed, the whole Xref slaving matter could have been covered
by that getout in the old wording, but the new does make it more clear.

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 30 Jun 2005 19:11:20 GMT

>to be able to mark cross-posts as read?

Because, with hindsight, there might have been better ways of achieving
that effect.

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 30 Jun 2005 19:11:20 GMT