Subject: Decision: Let's get draft-allbery-usefor-usepro-00 right and publish it as draft-ietf-usefor-usepro-07

From: Harald Alvestrand <[email-masked]>
Date: Fri, 15 Dec 2006 09:59:28 +0100

All the participants who have stated a preference prefer 
draft-allbery-usefor-usepro as the basis for the next version of the 
USEPRO draft.

Two people have stated opinions that I do not parse as a preference one 
way or the other:

- Charles thinks that we need further discussion
- Frank does not agree (or is not sure he agrees?) with Russ' dividing 
line between "protocol matters" and "best current practice" matters.

Nevertheless, I believe that it's safe to conclude that the next version 
of the draft will be based on draft-allbery-usefor-usepro.

                         Harald

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 18 Dec 2006 19:58:30 GMT

>of the draft will be based on draft-allbery-usefor-usepro.

I do not see how two people in favour, with one person saying he was
against unless the philosophy was changed, plus one asking for more time
for discussion (of which there has been hardly any up until the last
couple of days) can be interpreted as even a "rough" consensus. Possibly
such a consensus might emerge after more discussion, and hopefully some
further opinions.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 18 Dec 2006 20:33:21 +0000

Charles Lindsey wrote:

>I do not see how two people in favour,
>
Charles, some opinions were stated privately, so you can't assume that 
only 2 people were in favor of using Russ' version.

>with one person saying he was

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 19 Dec 2006 11:42:09 +0100

I did not bother listing all the participants in the poll, since not a 
single person had stated a clear preference for continuing with your 
document, Charles.

A number of strong opinions + no opposing opinions constitutes a 
reasonable consensus.

                   Harald

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 19 Dec 2006 15:01:37 GMT

>only 2 people were in favor of using Russ' version.

Well, I can only comment on what I see reported on this list.

Though it does seem, from another thread, that one of your anonymous
voters might be having second thoughts.

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 20 Dec 2006 00:50:50 +0100

Harald Alvestrand wrote:

> not a single person

JFTR, s/not/only/

The duties of an injecting-agent in usepro-06 list
"forward to moderator" as point *_8_*  At that time
the Date, the Message-ID, and other header fields are
already guaranteed to be okay.

Apparently we have three states:  1 - proto-article,
the stuff generated by a newsreader (posting agent or
followup-agent in the convoluted USEPRO terminology),
2 - article, messages relayed between news servers,
or accessed by a newsreader (reading agent), 3 - some
special state between proto-article and article, the
stuff forwarded from injecting agent to a moderator,
from there to another moderator, and eventually it's
either injected or rejected.

 From the POV of the original poster the article was
sent to its normal news server, anything else is an
implementation detail of NetNews for its "moderated"
magic.  We could use the term "submitted article" for
this third state.

Frank

---

From: Russ Allbery <[email-masked]>
Date: Tue, 19 Dec 2006 16:27:04 -0800

Frank Ellermann <[email-masked]> writes:

> The duties of an injecting-agent in usepro-06 list "forward to
> moderator" as point *_8_* At that time the Date, the Message-ID, and
> other header fields are already guaranteed to be okay.

> Apparently we have three states:  1 - proto-article, the stuff generated
> by a newsreader (posting agent or followup-agent in the convoluted
> USEPRO terminology), 2 - article, messages relayed between news servers,
> or accessed by a newsreader (reading agent), 3 - some special state
> between proto-article and article, the stuff forwarded from injecting
> agent to a moderator, from there to another moderator, and eventually
> it's either injected or rejected.

It's still a proto-article, since it hasn't been injected.  The current
protocol does say that you forward after adding the Message-ID and Date
header fields, though.  It's not clear to me how many moderators would
care if things weren't done in this order, but given that this is how news
servers have always done it, it's probably not worth possibly breaking
anything.

So yes, there is an intermediate state of "proto-article but with
mandatory header fields created" that shows up for the moderator, but it's
mostly in there for backward compatibility and is not otherwise hugely
significant from a protocol standpoint.

It's also not hugely significant from a poster standpoint, I believe.  I
doubt many posters are going to care about the difference between a
message ID generated by their local news server and a message ID generated
by the moderator.  The situation only gets interesting when the poster
themselves has provided the message ID, which is an entirely different
case.

> From the POV of the original poster the article was sent to its normal
> news server, anything else is an implementation detail of NetNews for
> its "moderated" magic.  We could use the term "submitted article" for
> this third state.

We could, but I don't see what we gain by giving that state an explicit
label.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: Harald Alvestrand <[email-masked]>
Date: Wed, 20 Dec 2006 10:23:47 +0100

> JFTR, s/not/only/
Frank, are you stating that you think the WG should progress from 
draft-ietf-usefor-usepro-06 and not progress from 
draft-allbery-usefor-usepro?

This is much more than a single issue, which may be worked on in either 
version. I'm asking you to speak clearly now.

Harald

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 20 Dec 2006 21:16:12 +0100

Russ Allbery wrote:
 
> I doubt many posters are going to care about the difference between a
> message ID generated by their local news server and a message ID
> generated by the moderator.

+1  I mentioned 1..8 in the USEPRO duties only as an illustration that
some potential oddities of proto-articles are already handled before a
moderator sees the message (if all goes well as specified).

> The situation only gets interesting when the poster themselves has 
> provided the message ID, which is an entirely different case.

It's the interesting case - I don't care about any provisional M-ID
used for the communication between the server where the article was
submitted, and the moderator(s).  If the first (and only the first)
moderator somehow identifies a provisional M-ID he's free to beautify
it.  But modifying an original M-ID is utter dubious, and it could
cause complete confusion if a later (not the first) moderator starts
to modify an already approved M-ID.

>> From the POV of the original poster the article was sent to its
>> normal news server, anything else is an implementation detail of
>> NetNews for its "moderated" magic.  We could use the term
>> "submitted article" for this third state.
 
> We could, but I don't see what we gain by giving that state an
> explicit label.

It's apparently a part of your philosophy that anything that happens
to proto articles isn't very interesting for the protocol, the show
begins when it's injected.  For my different philosophy the show
begins when a news server didn't reject an article with some kind of
error message.  From my POV the moderators are a part of the protocol
- modbot or human alike.  E.g. "drop" isn't a valid implementation of
"reject".

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 20 Dec 2006 21:37:09 +0100

> Frank, are you stating that you think the WG should progress
> from draft-ietf-usefor-usepro-06 and not progress from

I can't tell what's easier, remove dubious topics from USEPRO,
e.g. extract mvgroup to a different I-D, or add missing items
to Russ' I-D.  Apparently a majority prefers to try the latter.

I'd prefer to go through Charles' long list of differences,
point by point, maybe noted as issues, and get some decisions
what we want.  For "mvgroup" we're apparently almost clear to
remove it from the main I-D.  For the moderator business Russ
offered to remove the offending "arbitrary modifications".

> I'm asking you to speak clearly now.

Russ said twice that I should not pick his draft, because our
philosophies are incompatible, and I agreed, what's unclear
about this ?

Frank

---

From: Russ Allbery <[email-masked]>
Date: Wed, 20 Dec 2006 18:28:55 -0800

Frank Ellermann <[email-masked]> writes:

> For the moderator business Russ offered to remove the offending
> "arbitrary modifications".

I offered to rephrase the current wording.  I didn't offer to change
anything protocol-wise about the current description in my draft, for so
long as it's my draft, which includes the message ID issue.  (If it's a
working group draft, that's another matter.)

---

From: Russ Allbery <[email-masked]>
Date: Fri, 29 Dec 2006 10:28:21 -0800

> It's the interesting case - I don't care about any provisional M-ID used
> for the communication between the server where the article was
> it.  But modifying an original M-ID is utter dubious, and it could cause
> complete confusion if a later (not the first) moderator starts to modify
> an already approved M-ID.

How?  Could you provide some specific examples of what interoperability
problems you believe could result?  As previously mentioned, cancels are
not an insolvable issue; cancels of a message in a moderated group have to
be approved by the moderator anyway, at which point the correct message ID
can be used.

The only interoperability problem that I can think of is if one of the
earlier moderators or the poster made a digital signature that included
the message ID, in which case changing it would invalidate the signature.
In the presence of digital signatures, many different changes, not just
the message ID, are likely to invalidate the signature, and that will
certainly have to be addressed in the specification of that signature
protocol (perhaps by saying "don't change messages" for groups that want
to use such signatures).  But that seems outside the scope of this
document provided that we say nothing that *conflicts*.  (MAY change the
message ID doesn't conflict; SHOULD would conflict.)

> It's apparently a part of your philosophy that anything that happens to
> proto articles isn't very interesting for the protocol, the show begins
> when it's injected.

No, not really.  I consider moderation to be an important part of the
protocol.  However, what happens at the moderation stage is complex enough
and involves enough human judgement that I don't believe much can be
usefully said in a protocol specification, as opposed to a BCP.

> For my different philosophy the show begins when a news server didn't
> reject an article with some kind of error message.  From my POV the
> moderators are a part of the protocol - modbot or human alike.
> E.g. "drop" isn't a valid implementation of "reject".

It *has* to be.  My moderated groups get hundreds of messages a day for
which drop is the *only* valid implementation of reject.  Those messages
are spam with either no contact information at all or forged contact
information.  Any attempt on my part to do anything but drop those
messages would leave me sending unsolicited e-mail or other communication
to someone whose only involvement with the original message was to have
their contact information forged by a spammer.

Numerically, this is the common case.  This case significantly outnumbers
both approvals and human-notified rejects.

Yes, I certainly agree that a human poster should be notified of a
rejection of a message to a moderated group provided that they included
some reasonable means for that notification to be sent, but determining
when this should be done versus dropping spam silently is a judgement
call not amenable to protocol language.  I would be very happy to see a
document that describes those judgement calls, and indeed such a document
was already written, as an I-D, a little over ten years ago.  You can find
it on the web as "NetNews Moderator's Handbook."  It would be great to
polish and update that document and publish it as a BCP or Informational
RFC, but there is a reason why it's over 45 pages long.  If we try to get
into that material in the protocol specification, we're going to end up
wandering *way* off into the weeds.

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 30 Dec 2006 17:21:04 +0100

Russ Allbery wrote:

>> modifying an original M-ID is utter dubious, and it could cause
>> complete confusion if a later (not the first) moderator starts to
>> modify an already approved M-ID.

> problems you believe could result?

The first moderator could note approved Message-IDs and cancel anything
else appearing in his group.

> cancels of a message in a moderated group have to be approved by the
> moderator anyway,

s/anyway/ideally/.  Cancels can be approved by the author as well, they
don't need to be moderated.  Some cases I have in mind:

1 - author submits article, moderator approves it, author decides to
    withdraw his contribution.  The author can send a cancel to the
    moderator (for approval by the moderator), additionally (s)he can
    send a self-approved (author's address) cancel.  For cancels the
    most important issue is speed.

2 - impostor submits article, moderator approves it, alleged author
    cancels the fake.  Somebody abused the From:-address and/or
    Message-ID, therefore cancel it.

> the message ID, in which case changing it would invalidate the signature.

Yes, that's a more elaborated version of "note approved Message-IDs".

> their contact information forged by a spammer.

That procedure has to be outlined in the standard.  You can't just drop
legit submissions without informing the author.  It has to be clear what
the Return-Path is (= an address at the original server), and that you'd
use the From (or Reply-To) for a "normal" reject.

For a "spam" reject you could use the Return-Path, informing the server
admin that one of their users is a spammer or zombie.  They could then
decide to close this account.  If even the Return-Path isn't plausible
(no address you've heard of before, no SPF PASS, the works) you're forced
to drop it, but you'll get false positives with this approach.

> determining when this should be done versus dropping spam silently is
> a judgement call not amenable to protocol language.  I would be very
> happy to see a document that describes those judgement calls, and
> indeed such a document was already written, as an I-D, a little over
> ten years ago.  You can find it on the web as
> "NetNews Moderator's Handbook."

http://www.eff.org/Net_culture/Net_info/Technical/usenet_moderator_handbook.draft

Ten years might be a bit old for issues related to spam.

> If we try to get into that material in the protocol specification,
> we're going to end up wandering *way* off into the weeds.

You could write "the procedure to do X or decide Y is not specified
here", but we need an overall model of what a moderated newsgroup is.

So far I thought it's an NG where "something" can decide to reject
or accept submitted articles.  Apparently your model includes content
modification and silently dropping articles.  That's very different.

Frank

---

From: Russ Allbery <[email-masked]>
Date: Sat, 30 Dec 2006 09:44:54 -0800

> else appearing in his group.

I suppose.  It would be rather pointless to do so at this point, given how
few people support cancels, and those I know of who are doing this use
PGPMoose instead (thus turning this into the digital signature case).

This is a case of cross-posting between moderated groups, which I agree is
kind of iffy already.  In practice, one pretty much always works this out
via special agreement between the moderators.  We have a system described
in our document that sort of works, but it still requires prior
arrangement between the moderators to devise a way of noting that the
message was already approved.  Said prior arrangement could deal with
issues such as this as well.

This area is not going to get any better until there's a strong
authentication system for moderator approvals.  Until then, anything we
write about the mechanics of this is written on quicksand; in practice,
there are a bunch of ad hoc conventions that are not suitable for
standardization.

Again, I'll warn that trying to explore this area is going to go way off
into the weeds for this working group.  Do you want the moderator to
retain the message ID if they've made substantial changes to the post?
What if the moderator decides to remove the newsgroup to which the post
was previously approved?  There's nothing preventing them from doing that,
and there are some moderated newsgroups where the moderation policy
permits that (as much as I think it's a horrible idea for social reasons).
What if the two moderators decide to multipost the message instead of
crosspost it?

Documenting how moderation is done on Netnews and getting consensus around
recommendations is a document of its own and probably at least a year of
work for an active IETF working group.  It took longer than that the last
time, and it's only gotten more complicated.

> don't need to be moderated.

Hm.  Well, I suppose that's a USEAGE question.  I'd consider it
questionable to self-approve *any* message to a moderated newsgroup,
including a cancel, but I can see the argument the other way.  It depends
on how one views the nature of a moderated newsgroup.  Certainly, on a
server that enforces PGPMoose authentication, such a cancel would be
ignored, but then cancels are generally ignored anyway.  (In practice, I'd
just explain to the poster of the cancel that no one honors cancels any
more anyway and they're just creating controversy for themselves without
accomplishing anything useful.)

> Some cases I have in mind:

>     most important issue is speed.

True, to what degree they're useful.  It's really not feasible to withdraw
a message once it's been posted, but ignoring that, I do see your point.

>     Message-ID, therefore cancel it.

Better, here, to notify the moderator since you also want to prevent the
imposter from doing it again (and that's more important than chasing the
first message with a cancel).

>> which drop is the *only* valid implementation of reject.  Those
>> messages are spam with either no contact information at all or forged
>> contact information.  Any attempt on my part to do anything but drop
>> those messages would leave me sending unsolicited e-mail or other
>> communication to someone whose only involvement with the original
>> message was to have their contact information forged by a spammer.

> That procedure has to be outlined in the standard.

Apart from simply disagreeing in general, I don't believe this is feasible
for a standards-track protocol specification.

> You can't just drop legit submissions without informing the author.

Of course I can.  It happens all the time and has for as long as there
have been moderated newsgroups.

You don't *want* moderators to drop legit submissions without informing
the author, but that's a different statement.  I understand why you want
this, and in *most* cases I would agree.  However, in practice, one simply
cannot do this reliably without missing some legitimate submissions unless
one is spamming addresses forged by spammers, since no spam detection
method (including the human eye) is 100% reliable.

> It has to be clear what the Return-Path is (= an address at the original
> server), and that you'd use the From (or Reply-To) for a "normal"
> reject.

The envelope sender (assuming that's what you mean by Return-Path) is
precisely as reliable as the From header, namely not at all.  Spammers are
not that stupid and forging the envelope sender is trivial.

>> I would be very happy to see a document that describes those judgement
>> calls, and indeed such a document was already written, as an I-D, a
>> little over ten years ago.  You can find it on the web as "NetNews
>> Moderator's Handbook."

> Ten years might be a bit old for issues related to spam.

It certainly is.  It's too old for all sorts of things, although that
document addresses many of the other judgement calls that moderators deal
with, but it would desperately need an update.

> here", but we need an overall model of what a moderated newsgroup is.

I believe my current draft provides an accurate overall model of what a
moderated newsgroup is at the Netnews protocol level, namely moderators
*can* do anything they wish, but best practice says to make minimum
modifications to approved posts.

However, I come to this with a fairly intensive knowledge of how
moderation works.  Do you have a specific wording change in mind that
would make the model better for people who don't have that background,
based on what we've discussed?

> So far I thought it's an NG where "something" can decide to reject or
> accept submitted articles.  Apparently your model includes content

Then this discussion has already served an educational purpose.  That's a
very good thing!  I think that many people who have not moderated a
newsgroup don't understand how Netnews moderation works at all, or don't
understand the huge variety of actions different moderators in different
groups take with posts.

---

From: Frank Ellermann <[email-masked]>
Date: Sun, 31 Dec 2006 23:33:03 +0100

Russ Allbery wrote:

> authentication system for moderator approvals.  Until then, anything
> we write about the mechanics of this is written on quicksand; in
> practice, there are a bunch of ad hoc conventions that are not
> suitable for standardization.

If we can't identify a least common denominator or a "KISS" procedure
we're forced to state that X-Posts affecting more than one moderated
group are not covered by the spec. and be done with it.

It's not the only area, we'll have similar problems with spam cancels,
with gateways, and with anything-AUTH.

> Do you want the moderator to retain the message ID if they've made
> substantial changes to the post?

I wouldn't wish to discuss this case.

For Netnews as protocol the combination of Message-ID, newsgroups, path,
and poster IS the PDU, the "content" (body) is only there because the
protocol has to transport something.  Modifying the content is the
business of a higher layer.

 From my POV (moderators as part of the protocol) modifying the content
is an abomination.  But if your "higher-layer moderators" do it anyway,
then yes, of course they MUST use their own Message-ID.  While they're
at it they could also use their own From, and formulate their text as
citation giving credits to the submitter of the original text as needed.

So I'd be happy if you simply write that moderated groups transporting
modified content are out of scope.  We could then focus on the straight
forward accept / reject case, and get that right.

> What if the moderator decides to remove the newsgroup to which the
> post was previously approved?

Not allowed wrt our layer, Netnews as protocol.

> There's nothing preventing them from doing that,

There's also nothing preventing all peers of news.clara.net to reject
all articles containing the word "viagra", nevertheless they don't do
it, and we won't allow it.  And when I found that nanas is empty except
from the X-posted FAQs I complained, and they fixed it, but I digress.

> there are some moderated newsgroups where the moderation policy permits
> that (as much as I think it's a horrible idea for social reasons).

Okay, then let's say (or is that pretend ?) that "moderation policy" is
an incarnation of a higher layer protocol, and out of scope for us.

> What if the two moderators decide to multipost the message instead of

USEPRO must be clear enough that they know why it's imperative to change
the Message-ID for this stunt.  If most readers immediately understand
why that's necessary it's good enough.  (And IMO s-o-1036 managed this).

The minor point that the multi-post is a stupid idea to start with is a
mere corollary.  The protocol standard must be clear about the principles
of operation - figuring out how to break the rules is out of scope.

> In practice, I'd just explain to the poster of the cancel that no one
> honors cancels any more anyway

You can't specify something if you don't believe in the concept, and v.v.

The only case of "cancel doesn't work at all anymore" I'm aware of is
"Supersedes" on news.clara.net.  Otherwise cancels worked when I used
them, but admittedly it might be some years that I used it, and it was
in de.all (not counting local hierarchies).

> Apart from simply disagreeing in general, I don't believe this is
> feasible for a standards-track protocol specification.

We can at least address the "obvious" issues like the format of the mail
sent from a server to a moderator.  It's perfectly annoying to discuss
this issue again and again with folks claiming that all mails they care
about have 2821-From == 2822-From, and anything else is nonsense.

So let's say that I try to post in e.g. nanas (the only moderated NG I've
successfuly used so far, apart from misc.test.moderated), for that I can
in theory use News-From: nobody@

In practice I used Reply-To: nobody@ with a bogus News-From, because the
"higher layer protocol" for nanas allows this, but that's out of scope.

I've no clue how news.t-online or clara.net transported my articles to
nanas-bot, but let's say they used SMTP.  To make it more interesting
let's assume that nanas-bot sits behind an MX enforcing RFC 4408 (SPF).

All goes well if the server uses a Return-Path to itself, i.e, 2821-From
[email-masked] or similar.  It FAILs as designed if they'd use
a 2821-From: nobody@ (forging that based on the News-From).  If they try
to forge a 2821-From based on my bogus News-From it's worse (of course
I made sure that it's TLD invalid as specified in USEFOR I-Ds at this
time, but a decent MX trying to protect nanas-bot would reject this).

So far it's simple, a reverse path in the spirit of the original SMTP
works as designed.  But some folks decided that RFC 821 / 346x / 3834
etc. etc. got it all wrong, besides OE doesn't display a Return-Path,
and some calendar software abused as MUA is allegedly worse.  Therefore
they invented a "PRA" specified in RFC 4407.

With that it starts to get complex:

2821-From: [email-masked] => fine for 3834 + 4408
2822-From: nobody@             => assming I have a PRA policy

Of course I don't have a PRA policy, but IESG and the SenderID folks
claim that I'm wrong, and that my SPF policy is implicity a PRA-policy.
Therefore the mail from my news server to nanas-bot would get a PRA
FAIL, the IP used to talk with the MX before nanas-bot isn't permitted.

It would be similar for users with real PRA-policies, therefore we can
ignore the bit about SPF-policy-abused-as-PRA-policy.  In essence the
news server has to add a Resent-From: [email-masked]

It might get away with adding a Sender: [email-masked], but I've
no clue how I could then submit an article for somebody else (e.g.
Sender: me, News-From: you).  We have to specify at least the first
and simple part (2821-From = server, not 2821-From = poster).

For the PRA stuff we could ignore it, because the IESG found that it's
anyway incompatible with IETF standards.  We should also specify why
nanas-bot is supposed to ignore RFC 3834, and send its ACKs to the
Reply-To or 2822-From, not to the 2821-From.  As it does.

We don't need to go as far as an "updates 3834", but we can't simply
ignore the issue.  The part of NetNews using SMTP has to follow the
rules of engagement in the war on spam.

> The envelope sender (assuming that's what you mean by Return-Path)
> is precisely as reliable as the From header, namely not at all.
> Spammers are not that stupid and forging the envelope sender is
> trivial.

Because they're not stupid they don't do this for SPF FAIL protected
addresses, spammers / SPF publishers / SPF checkers all work together,
a happy family.  But nanas-bot or any other moderator might sit behind
an SPF checker.  Or behind a PRA-checker abusing SPF for its purposes,
for details see <http://purl.net/xyzzy/home/test/senderid-appeal.htm>

> Do you have a specific wording change in mind that would make the
> model better for people who don't have that background, based on what
> we've discussed?

Upper case the "recommended" near "minimal changes".  Don't talk about
content modifications, unless it's a statement that it's out of scope.
If you like the "moderator manual" add it to the informative references.

Let's focus on simple cases, our layer, and more or less known traps and
pitfalls like the "how to forward an article to a moderator with SMTP".
News server admins have to get this right, and we have to tell them how.

Frank

---

From: Russ Allbery <[email-masked]>
Date: Sun, 31 Dec 2006 16:04:09 -0800

> group are not covered by the spec. and be done with it.

I can accept saying that any crossposts between two moderated groups will
have to be worked out between the two moderators via means not covered by
the standard.  I think what we do have now is a minor improvement over
that and was the result of a long working group debate, so I wouldn't
agree with removing it without clear working group consensus and would
oppose that change, but I can see an argument for leaving it
unstandardized.

> forward accept / reject case, and get that right.

We could also say that moderated groups in general are out of scope, which
would be an even more useful simplification and about as productive in
producing a standard that lets people write interoperable software and
understand what assumptions they can and can't make about how Netnews
works.  For me, that's the goal here.  I want to see a protocol standard
published that people can use to write Netnews software that interoperates
with the software that's already out there.

If your software assumes that messages to moderated newsgroups will either
be posted verbatim or rejected with a message to the original poster, your
software will break in the real world.  I want to give you guidance that
you can rely on, rather than guidance that's pure but wrong.

If I were designing a new protocol to fix all the problems with Netnews,
it would be a lot different.

> it, and we won't allow it.

We do allow it and should, because things exactly like that do in fact
happen and software has to be aware that they can and do happen so that it
can make appropriate assumptions and interoperate.

The protocol document is not the appropriate place for us to outlaw all of
the things that have previously annoyed us about the operation of some
news server.  We need to stay focused here on defining the *protocol*,
namely how messages are transmitted if that transmission is authorized and
how protocol commands are communicated, not telling people how they may
and may not configure their software.  See, for instance, RFC 2821, which
similarly makes it quite clear that a given mail server may reject
messsages for whatever reasons it chooses and only defines the protocol
for offering them, processing them if desired, and communicating the
results clearly.

> why that's necessary it's good enough.  (And IMO s-o-1036 managed this).

Indeed.  I would hope that it's an obvious implication of several points
of the duties of relaying and serving agents, not to mention the
definition of a message identifier and the general principle in my draft
about the intent of the Netnews protocol.

> You can't specify something if you don't believe in the concept, and v.v.

Sure I can, but anyway, that's beside the point.  My server is one of the
few that *does* still honor cancel messages, so in this context, I'm the
rare true believer (or at least someone who thinks that the drawbacks
don't actually outweigh the benefits) and would actually be exaggerating
to the person since I'm a counter-example.  But it's still the fastest
approximation of the truth for most of the situations in which this
argument arises.  "Few news servers honor cancels" is probably better,
though.

Plus, this is all from the Usenet perspective.  Cancel messages are still
useful in many other, smaller Netnews networks with better authorization
policies (and in those networks are quite frequently not issued by the
poster).

> about have 2821-From == 2822-From, and anything else is nonsense.

I certainly wouldn't want to attempt to explain in specific detail the
format that you get if you don't use encapsulation.  I doubt it is
describable except in generalizations like "it looks as if someone took a
Netnews message and wedged it into the mail system sideways and then
jumped up and down on it until an SMTP server reluctantly accepted it."

I can tell you exactly how INN constructs an unencapsulated message to a
moderator and how it determines the address to which to send it, but I
don't think the result would be useful to put in a protocol document, nor
would it quite reflect how Diablo does it, or DNews, or C News, or
Typhoon.  We're having to pick our battles here; we're not specifying
pgpverify either, which is in widespread use, or PGPMoose, which is in
lesser but still significant use, or the actsync protocol, or Xref
slaving, or UUCP batching, or feed patterns, or any number of other
interesting things that real news servers do.

> rules of engagement in the war on spam.

I'm pretty sure that, in this area, given the current energy of this
working group, and given the tangle that results as soon as anyone starts
talking about spam filtering, our choices are to ignore this issue or to
give up and never publish.

Right now, my document basically says that the method whereby messages are
conveyed to a moderator is outside the scope of the protocol, except that
if you want to use SMTP, here's how you SHOULD construct the message that
you're going to send.  While this isn't ideal from the perspective of
completely describing how to write a news server, I think it's the best
available compromise towards the goal of being able to publish a protocol
specification.  If we get into things like SPF, we're really going to be
here all day.

Anyway, I have other strong feelings about SPF that are really out of
scope for this working group and don't see any point in investing the time
and energy here to get into a real argument about it, so I'm going to
decline that discussion.

>> Do you have a specific wording change in mind that would make the model
>> better for people who don't have that background, based on what we've
>> discussed?

> If you like the "moderator manual" add it to the informative references.

I believe those changes would make the model considerably worse from the
perspective of predicting what will happen in a real Netnews network.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: Moderation issues
Organization: The Eyrie
Date: Sun, 31 Dec 2006 16:04:09 -0800
Sender: [email-masked]

> group are not covered by the spec. and be done with it.

I can accept saying that any crossposts between two moderated groups will
have to be worked out between the two moderators via means not covered by
the standard.  I think what we do have now is a minor improvement over
that and was the result of a long working group debate, so I wouldn't
agree with removing it without clear working group consensus and would
oppose that change, but I can see an argument for leaving it
unstandardized.

> forward accept / reject case, and get that right.

We could also say that moderated groups in general are out of scope, which
would be an even more useful simplification and about as productive in
producing a standard that lets people write interoperable software and
understand what assumptions they can and can't make about how Netnews
works.  For me, that's the goal here.  I want to see a protocol standard
published that people can use to write Netnews software that interoperates
with the software that's already out there.

If your software assumes that messages to moderated newsgroups will either
be posted verbatim or rejected with a message to the original poster, your
software will break in the real world.  I want to give you guidance that
you can rely on, rather than guidance that's pure but wrong.

If I were designing a new protocol to fix all the problems with Netnews,
it would be a lot different.

> it, and we won't allow it.

We do allow it and should, because things exactly like that do in fact
happen and software has to be aware that they can and do happen so that it
can make appropriate assumptions and interoperate.

The protocol document is not the appropriate place for us to outlaw all of
the things that have previously annoyed us about the operation of some
news server.  We need to stay focused here on defining the *protocol*,
namely how messages are transmitted if that transmission is authorized and
how protocol commands are communicated, not telling people how they may
and may not configure their software.  See, for instance, RFC 2821, which
similarly makes it quite clear that a given mail server may reject
messsages for whatever reasons it chooses and only defines the protocol
for offering them, processing them if desired, and communicating the
results clearly.

> why that's necessary it's good enough.  (And IMO s-o-1036 managed this).

Indeed.  I would hope that it's an obvious implication of several points
of the duties of relaying and serving agents, not to mention the
definition of a message identifier and the general principle in my draft
about the intent of the Netnews protocol.

> You can't specify something if you don't believe in the concept, and v.v.

Sure I can, but anyway, that's beside the point.  My server is one of the
few that *does* still honor cancel messages, so in this context, I'm the
rare true believer (or at least someone who thinks that the drawbacks
don't actually outweigh the benefits) and would actually be exaggerating
to the person since I'm a counter-example.  But it's still the fastest
approximation of the truth for most of the situations in which this
argument arises.  "Few news servers honor cancels" is probably better,
though.

Plus, this is all from the Usenet perspective.  Cancel messages are still
useful in many other, smaller Netnews networks with better authorization
policies (and in those networks are quite frequently not issued by the
poster).

> about have 2821-From == 2822-From, and anything else is nonsense.

I certainly wouldn't want to attempt to explain in specific detail the
format that you get if you don't use encapsulation.  I doubt it is
describable except in generalizations like "it looks as if someone took a
Netnews message and wedged it into the mail system sideways and then
jumped up and down on it until an SMTP server reluctantly accepted it."

I can tell you exactly how INN constructs an unencapsulated message to a
moderator and how it determines the address to which to send it, but I
don't think the result would be useful to put in a protocol document, nor
would it quite reflect how Diablo does it, or DNews, or C News, or
Typhoon.  We're having to pick our battles here; we're not specifying
pgpverify either, which is in widespread use, or PGPMoose, which is in
lesser but still significant use, or the actsync protocol, or Xref
slaving, or UUCP batching, or feed patterns, or any number of other
interesting things that real news servers do.

> rules of engagement in the war on spam.

I'm pretty sure that, in this area, given the current energy of this
working group, and given the tangle that results as soon as anyone starts
talking about spam filtering, our choices are to ignore this issue or to
give up and never publish.

Right now, my document basically says that the method whereby messages are
conveyed to a moderator is outside the scope of the protocol, except that
if you want to use SMTP, here's how you SHOULD construct the message that
you're going to send.  While this isn't ideal from the perspective of
completely describing how to write a news server, I think it's the best
available compromise towards the goal of being able to publish a protocol
specification.  If we get into things like SPF, we're really going to be
here all day.

Anyway, I have other strong feelings about SPF that are really out of
scope for this working group and don't see any point in investing the time
and energy here to get into a real argument about it, so I'm going to
decline that discussion.

>> Do you have a specific wording change in mind that would make the model
>> better for people who don't have that background, based on what we've
>> discussed?

> If you like the "moderator manual" add it to the informative references.

I believe those changes would make the model considerably worse from the
perspective of predicting what will happen in a real Netnews network.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: Control message propagation
Organization: The Eyrie
Date: Sun, 31 Dec 2006 15:30:42 -0800
Sender: [email-masked]

Frank Ellermann <[email-masked]> writes:

> Not really, it just needs a numbered list in 2434bis, it's the 7th point
> in chapter 4.1 of this I-D:

> |  RFC Required - RFC publication (either as IETF Submission or as an
> |                 RFC Editor submission [RFC3932]) suffices.

> Reserving x-verbs would cover points 1 / 2 (private / experimental use).
> We don't need "expert review" or anything more restrictive than "RFC".

I think requiring an RFC is too restrictive and think x-verbs are a bad
idea that cause more problems than they solve in this area, so it would
require more discussion than that.

> It was your idea to add the OPTION of new control verbs, with the odd
> effect of talking about newgroup + rmgroup "for example", as if there
> are others.

No, it was not my idea.  It's how Netnews actually works and has for
longer than I've been involved in it.  From the first version of INN, you
could drop in a handler for a brand new control message by doing nothing
more than putting a shell script in a directory, and people did so to
various ends.  Adding a new group control message would also require
changing one line of code in innd that classifies control messages as
"newgroup-like" for propagation purposes, but that's trivial.  That new
control verbs weren't allowed was a pure artificial creation of previous
documents (assuming one even read them that way) which I have attempted to
bring in line with reality.

Now we can, if the working group wants, discuss whether we want to try to
change reality, but what I did was fix an inconsistency between what was
on paper and how Netnews software actually works.

>> Since relaying agents may reject messages for any reasons they chose,

> This is not the case, compare usepro-06 chapter 6.3 points 1..8.  They
> must / should / may reject articles for the reasons specified in 1..8.

I didn't realize that anyone in the working group didn't consider this
principle implicit and assumed.  I'm glad, in this case, that I made it
explicit in my draft.  Section 3.1 of my draft says:

   No Netnews agent is ever required to accept any article.  It is
   common for injecting, relaying, and serving agents to reject well-
   formed articles for reasons of local policy (such as not wishing to
   carry a particular newsgroup or attempting to filter out unwanted
   articles).  This document specifies how articles are to be treated if
   they are accepted and specifies some cases where they must be
   rejected, but an agent MAY always reject any article for other
   reasons than those stated here.

I consider this principle to be at the core of how Netnews works and will
strenuously object to any attempt to fail to include it in a protocol
specification for Netnews (not to mention that an assumption that agents
will only reject messages for well-defined reasons specified in the
protocol document is catastrophically misleading if the goal if this is to
produce a document that defines how Netnews works in the real world).

>> I want to make it clear that many agents *will* reject unknown control
>> messages (since that's what happens in practice).

> That's fine, but why write so much about these hypothetical new control
> messages if you don't believe in it ?

If I don't believe in what?  I certainly believe that it's possible to
introduce new control messages; it's been done before.  Many agents will
reject them and many won't.  It will take some time and general agreement
with the new control message before they will have the propagation that
one might desire, so in the interim people should have backup procedures
available.  None of this is horribly difficult.  It just takes time and an
idea that's worth the effort.

>> It's standard to include language of this sort for extension
>> mechanisms; something like this has been in every other RFC that I've
>> worked with that has optional extensions.

> I don't recall any "MUST ignore or reject unknown" in RfCs I've read, do
> you have an example ?

By "of this sort" I meant stating explicitly what should be done with
unrecognized extensions.  The common language, as you point out, is "MUST
ignore", which is present in, for example, RFCs 2060, 2251, 2449, 2616,
2919, 3501, 3546, 3977, and 4121, just to pick out a few RFCs that I have
lying around.

Because a Netnews agent MAY reject any message it choses, it MAY reject an
unknown control message rather than ignoring it, so saying in this case
that such messages "MUST be ignored or rejected" is more complete and
avoids even the possible perception of self-contradiction.

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: Moderation issues
Date:  Sun, 31 Dec 2006 23:33:03 +0100
Organization:  <URL:http://purl.net/xyzzy>
Lines: 160
Sender: [email-masked]

Frank

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: Control message propagation
Date:  Sun, 31 Dec 2006 21:24:45 +0100
Organization:  <URL:http://purl.net/xyzzy>
Lines: 67
Sender: [email-masked]

Russ Allbery wrote:

>> IMO that can't fly if relaying agents reject unknown stuff.  We could
>> create a IANA registry of control <verb>s, reserving x-whatever for
>> experimental / unofficial <verb>s, and anything else to be defined in
>> an RFC.

> I don't think this is really necessary and it adds bulk and procedure
> to the document and will mean a fair bit more discussion to figure out
> review and registration procedures.

Not really, it just needs a numbered list in 2434bis, it's the 7th point
in chapter 4.1 of this I-D:

|  RFC Required - RFC publication (either as IETF Submission or as an
|                 RFC Editor submission [RFC3932]) suffices.

Reserving x-verbs would cover points 1 / 2 (private / experimental use).
We don't need "expert review" or anything more restrictive than "RFC".

It was your idea to add the OPTION of new control verbs, with the odd
effect of talking about newgroup + rmgroup "for example", as if there
are others.   If you really want this we can also get it right, we've
all ingredients for a proper IANA registry, some obsolete verbs, some
"real" verbs, and one hypothetical mvgroup to be specified elsewhere.

If you don't really want this let's please get rid of the MAY and "for
example" cruft, it's halfhearted and confusing.

>> With a general "MUST ignore unknown", limiting "MAY reject unknown"
>> to injecting agents.  Or better removing the "reject" option.

> Since relaying agents may reject messages for any reasons they chose,

This is not the case, compare usepro-06 chapter 6.3 points 1..8.  They
must / should / may reject articles for the reasons specified in 1..8.

Your points 1..10 in 3.5 are similar, just add what's missing - maybe
relaying agents need to be free to enforce an UDP no matter what their
peers do, to reject obvious spam, the works.  Don't say "any reason",
that's censorship and evil.

Articles MUST NOT disappear for unknown "any" reasons.  It needs good
reasons, and USEPRO is supposed to provide a list of good reasons, and
ways to check this for observers.

> I want to make it clear that many agents *will* reject unknown control
> messages (since that's what happens in practice).

That's fine, but why write so much about these hypothetical new control
messages if you don't believe in it ?  And it wasn't clear for me, some
lines above I still thought that you might really want to support new
control messages.  So I am (or was) confused.  "Reject crap" is a clear
directive - unlike "MAY send / accept crap + MUST ignore / reject crap".

> It's standard to include language of this sort for extension mechanisms;
> something like this has been in every other RFC that I've worked with
> that has optional extensions.

I don't recall any "MUST ignore or reject unknown" in RfCs I've read, do
you have an example ?  If the idea is to protect the network from unknown
and potentially dangerous crap "reject" is good, if the idea is to allow
unknown future extensions "ignore" is good, but any party doing whatever
it likes better is dubious.  For me it sounds like "MUST do X or not X".

Frank

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: Control message propagation
Organization: The Eyrie
Date: Sun, 31 Dec 2006 09:36:18 -0800
Sender: [email-masked]

Frank Ellermann <[email-masked]> writes:

> IMO that can't fly if relaying agents reject unknown stuff.  We could
> create a IANA registry of control <verb>s, reserving x-whatever for
> experimental / unofficial <verb>s, and anything else to be defined in
> an RFC.

I don't think this is really necessary and it adds bulk and procedure to
the document and will mean a fair bit more discussion to figure out review
and registration procedures.  I can see the merit, but personally I'd
prefer not to.  I'm willing to be convinced otherwise, though.

> With a general "MUST ignore unknown", limiting "MAY reject unknown" to
> injecting agents.  Or better removing the "reject" option.

Since relaying agents may reject messages for any reasons they chose,
listing reject as an option doesn't change anything about the requirements
for a relaying agent and makes it clearer in the text what the options are
for an implementor.  I want to make it clear that many agents *will*
reject unknown control messages (since that's what happens in practice).

> A "MUST ignore or reject unknown" is IMHO pointless, what else should
> they do ?

Attempt to guess at the meaning or bother the news administrator.  It's
standard to include language of this sort for extension mechanisms;
something like this has been in every other RFC that I've worked with that
has optional extensions.

>> Two reasons not to make it a MUST:

>>  * Posting a cancel message to a test newsgroup will result in
>>    autoreplies from some sites, so it's conventional to drop test
>>    newsgroups from cancel messages.

> The first thing I did on any server I've used (all 5 of them :-) was to
> post an article in the local test group and cancel it, checking out if
> if that works as expected, and where I'd find the cancel.  I also did
> that in de.test and de.alt.test (for experiments with X-Posts and with
> Google's "purge" procedure).  No trouble with checkbots anywhere, and I
> think we can ignore checkbot oddities while discussing cancel standards.

I don't agree.  The plural of anecdote is not data and this has been a
known problem for many years.  Autoresponders come and go, different test
newsgroups behave differently, and it's very easy to cause this problem
when writing a naive autoresponder.

>> * Some hierarchies post all spam cancels to a designated newsgroup
>>   to make it easier for people who don't want them to opt out by not
>>   accepting the group.

> Spam cancels intentionally violate official rules, following their own
> standards.

Spam cancels and other sorts of administrative cancels are compliant with
my protocol spec and I want to keep it that way.  They're an authorization
problem, not a protocol problem.

I think the current text is fine.  It reflects what news servers currently
do when they honor cancels, and I don't believe that tweaking the language
will provide sufficient practical benefit for it to be useful.  The
Security Considerations section will have to remain anyway given the
number of existing servers that work that way.

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: Control message propagation
Date:  Sun, 31 Dec 2006 16:41:20 +0100
Organization:  <URL:http://purl.net/xyzzy>
Lines: 19
Sender: [email-masked]

> 1: cancel in A + B: sane
> 2: cancel in A:     bad but acceptable
> 3: cancel in C + B: odd variant of (2)
> 4: cancel in C:     bad, unacceptable

More about that for "Supersedes", if an article was posted in group X,
and the poster a.s.a.p. decided that it should go to group Y instead:

a: NG: X,Y; Fup2: Y   : that's clear for folks only seeing group X
b: NG: Y              : bad, old article mysteriously vanishes from X
c: cancel in X
   new article in Y   : obviously okay

The same rule (Russ' SHOULD match all NGs + my MUST match at least one)
apparently also works for "Supersedes".  Actually "Supersedes" is worse
than "cancel", should we say "MUST match all NGs" for a "Supersedes" ?

Frank

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: Control message propagation
Date:  Sun, 31 Dec 2006 15:11:31 +0100
Organization:  <URL:http://purl.net/xyzzy>
Lines: 65
Sender: [email-masked]

Russ Allbery wrote:

> The standard makes it clear that anyone who wishes can invent a new
> control message and start sending them.

That's apparently a derivation from usepro-06 in your text, you have
"MAY send or accept whatever" with a "MUST ignore or reject unknown".

IMO that can't fly if relaying agents reject unknown stuff.  We could
create a IANA registry of control <verb>s, reserving x-whatever for
experimental / unofficial <verb>s, and anything else to be defined in
an RFC.  With a general "MUST ignore unknown", limiting "MAY reject
unknown" to injecting agents.  Or better removing the "reject" option.

The IANA registry stuff is fairly simple, Harald has written a kind of
"howto" (2434bis), and "defined in an RFC" is straight forward.

A "MUST ignore or reject unknown" is IMHO pointless, what else should
they do ?

#######################################################################

  [A cancel SHOULD match the newsgroups of the original article]
>> Is that good enough for a MUST instead of a SHOULD ?  Or an explicit
>> OPTION to ignore cancels violating the SHOULD ?

> Two reasons not to make it a MUST:

>  * Posting a cancel message to a test newsgroup will result in
>    autoreplies from some sites, so it's conventional to drop test
>    newsgroups from cancel messages.

The first thing I did on any server I've used (all 5 of them :-) was to
post an article in the local test group and cancel it, checking out if
if that works as expected, and where I'd find the cancel.  I also did
that in de.test and de.alt.test (for experiments with X-Posts and with
Google's "purge" procedure).  No trouble with checkbots anywhere, and I
think we can ignore checkbot oddities while discussing cancel
standards.

> * Some hierarchies post all spam cancels to a designated newsgroup
>   to make it easier for people who don't want them to opt out by not
>   accepting the group.

Spam cancels intentionally violate official rules, following their own
standards.  I'd like an informative reference to Skirvin's cancel FAQ,
but for USEPRO we need some minimal "normal" procedure.

Tons of SHOULD not explaining why and when they're violated don't help.

If your justification for a SHOULD instead of a MUST is "spam cancel"
please say so explicitly, with an informative reference.   I think we
have four cases for an article X-posted in unmoderated groups A and B:

1: cancel in A + B: sane
2: cancel in A:     bad but acceptable
3: cancel in C + B: odd variant of (2)
4: cancel in C:     bad, unacceptable

With that I get "at least one of the newsgroups of the cancel MUST
match a newsgroup of the cancelled article" in addition to your SHOULD.

Frank

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Moderation issues (was: Decision: Let's get draft-allbery-usefor-usepro-00 right and publish it as draft-ietf-usefor-usepro-07)
Organization: The Eyrie
Date: Sat, 30 Dec 2006 09:44:54 -0800
Sender: [email-masked]

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: Control message propagation
Organization: The Eyrie
Date: Sat, 30 Dec 2006 09:13:04 -0800
Sender: [email-masked]

>>    Exceptionally, control messages creating or removing newsgroups
>>    (newgroup or rmgroup control messages, for example)

> Ignoring "mvgroup" the example is a complete enumeration, isn't it ?
> The rest of the text doesn't make sense for a "checkgroups".  Maybe
> remove the parentheses.

*Now*, but others may be introduced later and this should still hold, so
it felt right to me to keep it general.  The standard makes it clear that
anyone who wishes can invent a new control message and start sending them.

>> the following under cancel control messages:

>>    A cancel control message SHOULD have the same Newsgroups header field
>>    as the message it is cancelling so that it will attempt to reach the
>>    same news servers as the original message.

>> and the following in Security Considerations:

>>    Cancel control messages are not required to have the same Newsgroups
>>    header field as the messages they are cancelling and, since they are
>>    sometimes processed before the original message is received, it may
>>    not be possible to check that they do.  This allows a malicious
>>    poster to inject a cancel control message for an article in a
>>    moderated newsgroup without adding an Approved header field to the
>>    control message, and to hide malicious cancel control messages from
>>    some reading agents by using a different Newsgroups header field so
>>    that the cancel control message is not accepted by all news servers
>>    that accepted the original message.

> So far it's clear.  Is that good enough for a MUST instead of a SHOULD ?
> Or an explicit OPTION to ignore cancels violating the SHOULD ?

Two reasons not to make it a MUST:

 * Posting a cancel message to a test newsgroup will result in autoreplies
   from some sites, so it's conventional to drop test newsgroups from
   cancel messages.

 * Some hierarchies post all spam cancels to a designated newsgroup to
   make it easier for people who don't want them to opt out by not
   accepting the group.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: More Path notes
Organization: The Eyrie
Date: Sat, 30 Dec 2006 09:10:21 -0800
Sender: [email-masked]

>>    Some existing implementations treat <path-identity> as case-
>>    sensitive, some case-insensitive.  The <path-identity> therefore
>>    SHOULD be all lowercase and implementations SHOULD compare 
>>    identities case-insensitively.

> Messy.  No chance for s/SHOULD/MUST/ ?

I could see a MUST for the second one.  I'm leery of telling people they
have to change their path identity.  What do others think?

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: Decision: Let's get draft-allbery-usefor-usepro-00 right and publish it as draft-ietf-usefor-usepro-07
Date:  Sat, 30 Dec 2006 17:21:04 +0100
Organization:  <URL:http://purl.net/xyzzy>
Lines: 78
Sender: [email-masked]

Frank

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: Control message propagation
Date:  Sat, 30 Dec 2006 15:54:01 +0100
Organization:  <URL:http://purl.net/xyzzy>
Lines: 41
Sender: [email-masked]

Russ Allbery wrote:

>    Exceptionally, control messages creating or removing newsgroups
>    (newgroup or rmgroup control messages, for example)

Ignoring "mvgroup" the example is a complete enumeration, isn't it ?
The rest of the text doesn't make sense for a "checkgroups".  Maybe
remove the parentheses.

>    Group control messages affecting specific groups (newgroup and
>    rmgroup control messages, for example)

Same here, a "checkgroups" listing _all_ newsgroups in the Newsgroups
header field would be odd.  Get rid of "for example", the qualifier is
confusing without "mvgroup".

##########################################################################
> the following under cancel control messages:

>    A cancel control message SHOULD have the same Newsgroups header field
>    as the message it is cancelling so that it will attempt to reach the
>    same news servers as the original message.

> and the following in Security Considerations:

>    Cancel control messages are not required to have the same Newsgroups
>    header field as the messages they are cancelling and, since they are
>    sometimes processed before the original message is received, it may
>    not be possible to check that they do.  This allows a malicious
>    poster to inject a cancel control message for an article in a
>    moderated newsgroup without adding an Approved header field to the
>    control message, and to hide malicious cancel control messages from
>    some reading agents by using a different Newsgroups header field so
>    that the cancel control message is not accepted by all news servers
>    that accepted the original message.

So far it's clear.  Is that good enough for a MUST instead of a SHOULD ?
Or an explicit OPTION to ignore cancels violating the SHOULD ?

Frank

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: More Path notes
Date:  Sat, 30 Dec 2006 15:27:41 +0100
Organization:  <URL:http://purl.net/xyzzy>
Lines: 18
Sender: [email-masked]

Russ Allbery wrote:
 
> FQDN is used without further qualification in USEFOR as well
[...]
> maybe there's a definition somewhere else we can just reference?

Check out RFC 3696 chapter 2.  It doesn't discuss DNS details,
but I don't think we need that for USEPRO.

>    Some existing implementations treat <path-identity> as case-
>    sensitive, some case-insensitive.  The <path-identity> therefore
>    SHOULD be all lowercase and implementations SHOULD compare 
>    identities case-insensitively.

Messy.  No chance for s/SHOULD/MUST/ ?

Frank

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: Injection-Date and reinjection
Organization: The Eyrie
Date: Fri, 29 Dec 2006 18:01:59 -0800
Sender: [email-masked]

I'm not sure the most effective way to respond to this analysis.  I
disagree with many points of Charles's message in one way or another, and
I'm afraid that trying to discuss it line-by-line will just result in a
muddle that will be confusing and too hard to follow.  I think there are
multiple differences of philosophy, definitions, analysis, and use cases
here.

So, what I'm going to try is to write up a description of how I see
reinjection in general and the issues that we need to deal with (and *not*
deal with) here as a coherent document.  Then, hopefully, we can have a
working group discussion comparing and contrasting my view of this with
the one Charles presented and find some solution that takes into account
all the concerns.

First, one issue that's *not* directly part of this.  Even if we keep the
current USEPRO method, we need to have a staleness check on the Date
header field at injection.  We can't make a single leap to allowing stale
Date header fields from not having any Injection-Date header field since
most existing software is going to ignore the Injection-Date header field
entirely and impose staleness checks on Date after injection.  Therefore,
an article with a stale Date is going to propagate very poorly, and
accepting such an article just to let it be dropped silently later by most
of the Netnews network would be a poor interoperability choice.  Far
better to reject the message at the injection point and make the user
replace the Date header field.  We can lower the requirement to SHOULD
from MUST because of Injection-Date, but not drop it entirely.

Now, let's look at the conditions under which this situation arises.

First, note that multiple injection (sending the same proto-article to
multiple injecting agents) is explicitly allowed for and described in our
drafts.  This is the best way to deal with most issues that would
otherwise give rise to reinjection.  For example, most small home servers
are better implemented by proxying local posts to (potentially multiple)
injecting agents rather than injecting the article into one's own local
Netnews server and then gatewaying it later.  A server that has
ineffective relaying relationships and more effective posting access can
inject posts both locally and at remote servers simultaneously and the
result would be much cleaner.

In other words, in general, confusing injecting and relaying or using them
as interchangeable substitutes for each other is a bug.  They do different
things and confusing and strange things can happen when they're not kept
distinct.  Declaring such behavior non-comformant is a feature.  The INN
code to permit people to inject posts via IHAVE is something I committed
only under protest, and I'm entirely content to have its behavior declared
non-comformant by a standard.  It is a *very* specialized configuration
for strange cases where the people involved know just what they're doing.

That being said, there are disjoint Netnews networks that people want to
connect, and multiple injection or proxying injection isn't always the
best approach (for one, it requires a real-time connection with the remote
injecting agent).  My definition of two disjoint Netnews networks is two
Netnews networks between which there is no relaying agent to relaying
agent connection.  Significant examples are:

 * The many people who do not have any relaying agreement with a large
   Netnews network such as Usenet, only a reading agent relationship, but
   still want the features of a full-fledged news server (longer article
   retention, multiple users sharing the same spool, faster service to
   reading agents, local newsgroups without requiring clients use multiple
   servers, off-line operation).  This has become increasingly common over
   the last five years for both personal and small organization use.

 * Ad hoc connections to a stand-alone news server that serves only a
   particular hierarchy.  Relaying is always better for setting up clean
   connections between news servers with the same hierarchy, but sometimes
   people don't want to be bothered to set up a relaying connection or
   (more ethically questionably) people decide they want to propagate in a
   larger Netnews network (like Usenet) groups that the providers don't
   want to propagate.

 * Broken software that only supports relaying and not injecting but which
   can't be trusted with a true relaying agreement.  This is the case that
   sparked the IHAVE support in nnrpd.

The goal is to handle these cases with a solution that has the
following properties:

 * Netnews loop detection and prevention is preserved.

 * Articles are stamped with trustworthy injection information.  Since,
   particularly in the first and third cases, the reason why no relaying
   agreement is in place is precisely because the original injection
   information can't be trusted, this requires replacing injection
   information when the message is gated from one Netnews network to
   another.

 * The protocol is clear about who does what and who is responsible for
   what in these situations.

 * This edge case does not unnecessarily complicate the regular protocol.
   In particular, I want to avoid making agents do complex things to allow
   for possible reinjection when reinjection is a rare case that most
   agents don't have to be concerned with.

My realization when reviewing the draft was that all or nearly all of the
cases where this arises and is not simply a bug (such as treating relaying
and injecting as interchangeable) can be meaningfully treated as disjoint
Netnews networks.  The case which Charles raises of a news server that
acts as a relaying agent to one server and a posting agent to another
server on the same network is an interesting possible exception; it's
rather pathological, but I can see the argument that the end results, if
done properly, are not distinguishable from multiple injection.  But let's
put that aside for a moment and look at the alternatives before us in the
more typical case.

Option 1 (my current draft)

    Proto-articles may not contain Injection-Date.  Injecting agents must
    reject any message that contains Injection-Date.  If an agent needs to
    reinject a message, it must think of itself as a gateway, remove the
    injection header fields (including Injection-Date), perform whatever
    checks it needs to ensure that it's not creating a loop, and then
    reinject the message, at which point it gets a new Injection-Date
    header field with the current date.  This gateway is fully responsible
    for not creating loops.

Option 2 (current USEPRO draft)

    For a given post, an injecting agent may either be doing injection or
    reinjection based on whether injection headers fields are already
    present.  It may decline to do reinjection.  If it doesn't, it may
    rename an existing Injection-Info header to some other name or remove
    it and must retain any existing Injection-Date header field.  It may
    perform a staleness check against the Date header field if no
    Injection-Date header field is present.  (The current USEPRO draft
    doesn't say an injecting agent can perform staleness checks against an
    existing Injection-Date header, but I presume that's simply an
    oversight.)  The injecting agent is responsible for not creating
    loops, not the agent that offers the article for reinjection.

I've already talked about why I think the first option is cleaner and
clearer from a protocol description and implementation standpoint.  Let me
instead focus on failure modes.

The primary risk of the first option is that it could create a slow loop
in the presence of bidirectional gatewaying as follows:  An article exists
in network A.  It is pulled by a reading agent and reinjected into network
B by a gateway which removes the Injection-Date header field.  Later,
another gateway reads the article from network B and reinjects it into
network A, removing the Injection-Date header field again.  In order for
this to occur, enough time will have to have passed between the reading of
the article from network A and the injection back into network A for the
record of the article to have expired, the two gateways would (contrary to
the SHOULD in 3.9) have to have no effective checks against circular
gatewaying, and the Injection-Date header must at each point still be
fresh enough not to trigger any staleness restrictions in either gateway.
Note that this loop cannot at this point continue unless one or the other
gateway also violates the MUST NOT in 3.9.2 against gatewaying the same
message twice; it stops with one duplicate.  (And, currently, the
injecting agent for A would have to not do a staleness check on the
preserved Date header, contrary to a SHOULD, but we're hoping to eliminate
that SHOULD eventually.)

The second option does not have this same failure mode, and indeed it
opens the door to fewer additional problems as long as we still have to
enforce fresh Date header fields.  The failure mode with the second option
comes when we want to drop the staleness check on a Date header field.
Suppose that a stand-alone news server B wants to pull articles from a
Netnews network A for local consumption.  B has a power failure and is
off-line for an extended period of time (a week, say).  When it comes back
up, it wants to catch up on all of the news from A that it missed while it
was down.  Now, as long as we have to enforce Date staleness, it can only
do this by ignoring a SHOULD at reinjection, since those articles will
likely now have stale Dates, but it can be configured to do that.
However, under option 2, catching up on gatewaying has now become
impossible because it is REQUIRED to retain the Injection-Date header of
the original message but that Injection-Date header is now stale and will
therefore be rejected by any relaying or serving agent in B.  Hence,
option 2 makes impossible during reinjection a scenario similar to why we
invented Injection-Date in the first place.

Now, my basic argument, apart from the simplicity of option 1, is that the
failure scenario for option 2 is worse than the failure scenario for
option 1.  I don't consider either to be particularly significant, but
option 2 does fail in a situation that I consider very plausible (if
unusual).  The circumstances for option 1's failure, on the other hand,
are not particularly plausible: the rarer case of bidirectional gatewaying
has to be in effect, we have to already be losing due to lack of
protection from circular gatewaying, *and* this whole thing has to happen
in slow motion.  In any normal circumstances, these gateways are going to
fire within minutes, hours at most, and when the reinjection to A is
attempted, a record of the article will still be present and the article
will be rejected as a duplicate.

Okay, phew.  Now back to Charles's case.

Suppose we have a new server with a relaying agreement to another
unreliable news server and a posting agreement with a much more reliable
news server, and that news server wants to send outgoing articles via both
paths.  Under option 2, the injecting agent that this server talks to must
support reinjection (it's an optional feature in the current USEPRO
draft), must enable it for that client, and then takes responsibility for
doing the appropriate transformation, but Injection-Date is retained.
Under option 1, this news server must either support multiple injection
(far and away the best option) and inject at the remote server at the same
time as the message is injected locally, or it must both relay the message
and gateway it to the remote injecting agent.  The gatewaying is more
complex, but can be done entirely locally and the article will then look
to the remote server like a regular proto-article.  No special support on
the remote server is therefore required; all of the work is born by the
agent doing something unusual.  In either case, two slightly different
copies of the message will exist with different trace headers, but this is
acceptable; the same is true for multiple injection.

I would argue that option 1 is still better for this case.  It's slightly
more work, but it puts the work in the right place and the burden is born
by the agent wanting to do the unusual work.  Furthermore, it's more
likely in practice that option 1 will be *possible*; if the remote
injecting agent isn't willing to set up a relaying agreement with this
news server, it seems unlikely to me that it would be willing to set up an
even more rare reinjection agreement.

Cc: [email-masked]
Date: Fri, 29 Dec 2006 15:54:05 -0800 (PST)
From: Ned Freed <[email-masked]>
Subject: Re: Scope of application/* media types (was: Protocol changes in draft-allbery-usefor-usepro-00)
To: Russ Allbery <[email-masked]>
Sender: [email-masked]

> > > I can't think of an example of how application/news-groupinfo could be
> > > dangerous.  It's nothing but a bit of text giving a newsgroup name and a
> > > description.  In and of itself, it has no force, power, or action
> > > whatsoever.

> > Application types are intended to cause applications to be invoked, with
> > possible side effects.

> Could you provide a reference for this?  I don't see this statement in RFC
> 2046.

That's likely because the invokation of applications is an implementation
artefact rather than a core feature of MIME. In particular, some (but by no
means all) handle different media types through an application dispatch
mechanism. RFC 1343 even defines a specific format to use for such information
- the so-called mailcap format. This is, however, an informational RFC, not a
standard, and not only is there no requirement that such a mechanism be used,
lots of implementations don't work this way.

> The closest that it has is:

>    The "application" media type is to be used for discrete data which do
>    not fit in any of the other categories, and particularly for data to
>    be processed by some type of application program.  This is
>    information which must be processed by an application before it is
>    viewable or usable by a user.

> "To be processed" is far from saying that reception of an application type
> will cause an application to be invoked.  application/octet-stream is an
> obvious counter-example to this interpretation.

Right. Furthermore, there's nothing that says video, image, audio, message or
model types are exempt from processing and the generation of side effects. For
that matter, it may be reasonable in some cases to handle text types by
dispatching to an external application, e.g., in a voice mail system.

> Our case falls into the general scope ("discrete data which do not fit in
> any of the other categories") but not fully into the specific one ("data
> to be processed by some type of application program").  Our media types
> are actually text/* types; the only reason why they're not declared as
> such is because text/* comes with baggage involving CTEs that we can't
> accept.  Declaring them as application/* is in essence a hack around the
> MIME object structure, although unfortunately a necessary one.

Yes, and while there have been various proposals to define an additional
top-level type or types to deal with the baggage associated with text, none of
them have received much support so far.

				Ned

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: application/news-groupinfo in newgroup (was: Protocol changes in draft-allbery-usefor-usepro-00)
Organization: The Eyrie
Date: Fri, 29 Dec 2006 14:20:06 -0800
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:
> Russ Allbery <[email-masked]> writes:
>> Charles Lindsey <[email-masked]> writes:

>>> 31. [-1] (5.2.1) Newgroup message SHOULD include application/group-info.

>>> Was MUST ...

>> Intentional change.

>> Newsgroup descriptions are optional in the protocol and newgroup
>> messages without descriptions work fine with existing software.

> OK, if absence means that it just creates a Newsgroups file entry with
> just the name of the newsgroup, and maybe the TAB, and certainly the
> " (Moderated)" if needed.

Its absence may mean that no newsgroups file entry is created at all.  The
newsgroups file is an optional feature of NNTP and need not be
implemented, nor need the newsgroups file be accurate or complete.  See
section 7.6.6 of RFC 3977:

   The list MAY omit newsgroups for which the information is unavailable
   and MAY include groups not available on the server.  The client MUST
   NOT assume that the list is complete or that it matches the list
   returned by LIST ACTIVE.

Newsgroups are not required to have descriptions unless you want to send a
checkgroups for them.  Some Netnews networks (such as the servers on which
microsoft.* is hosted) simply don't use them.

> Likewise for checkgroups.

I don't see what change you are indicating for checkgroups.  The body of a
checkgroups message can't meaningfully be empty.

> So you need something like:

>     The application/group-info MUST be included if the group is moderated,
>     in which case it MUST include a <moderation-flag>. For other groups, it
>     MAY be omitted if there is no <newsgroup-description> to be provided.

Why do we need something like that?  There's a perfectly usable moderation
flag in the Control header field.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Approved header field content (was: Protocol changes in draft-allbery-usefor-usepro-00)
Organization: The Eyrie
Date: Fri, 29 Dec 2006 14:14:00 -0800
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:
> Russ Allbery <[email-masked]> writes:
>> Charles Lindsey <[email-masked]> writes:

>>> 30. [-1] (5.2) Nothing said about content of Approved header.

>>> Surely, it SHOULD identify the person/identity/role of the issuer, ...

>> Intentional change.

>> The content of the Approved header serves no protocol purpose, and
>> USEFOR already adequately covers the definition of its content.
>> Control message authorization is done on the basis of the Sender or
>> From header (preferrably in combination with a digital signature).

> It asserts that this is an "authorized" message (just what "authorized"
> means is a separate issue which I shall raise later).

The description in USEFOR is:

   The Approved header field indicates the mailing addresses (and
   possibly the full names) of the persons or entities approving the
   article for posting.  Its principal uses are in moderated articles
   and in group control messages; see [I-D.ietf-usefor-usepro].

The Netnews protocol currently does not deal with authorization at all (an
obvious flaw noted in Security Considerations).  Any authorization
information you want to use has to be derived from the underlying
transport protocol or from unstandardized extensions such as digital
signatures.

> As such, it is pretty pointless unless it identifies the entity that is
> authorized, and all news servers that I know of can be configured with
> the identity that is to be recognized for each hierarchy.

If you're referring to group control mesages, said identity is checked
against the *From or Sender* header field, not the Approved header, at
least in INN.  INN ignores the contents of the Approved header.  I don't
know if C News uses the contents of the Approved header field for control
message permissions, but my impression from having maintained control.ctl
for some years is that most everyone uses From/Sender.

> Of course, it really needs to be digitally signed to make it effective,
> but that again is a separate issue.

> I think the WG early on took the view that the Approved header would
> simply fall into disrepute if one could always get away with
>    Approved: kibo
> and that it therefore needed some firmer language (digitally signed in due
> course). I remain of that view.

We can add firmer language when there's some meaning to the contents, such
as when digital signatures are added.  Right now, placing any stricter
requirements would contradict existing practice and be pointless since
there's no possible protocol check on the contents of Approved.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Scope of application/* media types (was: Protocol changes in draft-allbery-usefor-usepro-00)
Organization: The Eyrie
Date: Fri, 29 Dec 2006 14:03:02 -0800
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:
> Russ Allbery <[email-masked]> writes:
>> Charles Lindsey <[email-masked]> writes:

>>> 26. [-1] (4.2,4.3) Removed requirement that
>>> application/news-[groupinfo,checkgroups] MUST NOT be used except with
>>> relevant control messages.

>>> Application types are inherently dangerous ...

>> Intentional change.

>> I can't think of an example of how application/news-groupinfo could be
>> dangerous.  It's nothing but a bit of text giving a newsgroup name and a
>> description.  In and of itself, it has no force, power, or action
>> whatsoever.

> Application types are intended to cause applications to be invoked, with
> possible side effects.

Could you provide a reference for this?  I don't see this statement in RFC
2046.  The closest that it has is:

   The "application" media type is to be used for discrete data which do
   not fit in any of the other categories, and particularly for data to
   be processed by some type of application program.  This is
   information which must be processed by an application before it is
   viewable or usable by a user.

"To be processed" is far from saying that reception of an application type
will cause an application to be invoked.  application/octet-stream is an
obvious counter-example to this interpretation.

Our case falls into the general scope ("discrete data which do not fit in
any of the other categories") but not fully into the specific one ("data
to be processed by some type of application program").  Our media types
are actually text/* types; the only reason why they're not declared as
such is because text/* comes with baggage involving CTEs that we can't
accept.  Declaring them as application/* is in essence a hack around the
MIME object structure, although unfortunately a necessary one.

> In this case, the side efect is that a line gets added/changed in the
> Newsgroups file.

I don't agree with this interpretation, and I don't see anything in the
text of the application/news-groupinfo registration that reflects this.
It sounds to me like you're confusing the newgroup control message with
the application/news-groupinfo MIME media type.  They're not the same
thing.

> OK, in that case what you need to say is:

>     This media type is intended to be used in conjunction with the
>     newgroup control message as described in section 5.2.1. It MUST NOT be
>     automatically invoked in any other context without explicit human
>     intervention.

> and similarly for checkgroups.

I disagree with adding anything like this to our document.  When we just
defined the media type to contain simple text and nothing in the
description of the media type says anything about treating that text as
instructions to a computer program, this seems to me to just be confusing.
There's nothing here to invoke; it's just an association between newsgroup
names and descriptions.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Serving agents and duplicates (was: Protocol changes in draft-allbery-usefor-usepro-00)
Organization: The Eyrie
Date: Fri, 29 Dec 2006 13:51:15 -0800
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:
> Russ Allbery <[email-masked]> writes:
>> Charles Lindsey <[email-masked]> writes:

>>> 21. [-1] (3.6) No mention of processing cancel messages by
>>> serving/storage agents.

>> The same language is here as for relaying agents, but you may have missed
>> it because it's combined with another similar point:

>>   3.  It SHOULD reject any article that has already been successfully
>>       sent to it or that matches an already-received and honored cancel
>>       message or Supersedes header field following the same rules as a
>>       relaying agent (Section 3.5).

> OK, point taken, but it might be better split out. Moreover, the first
> bit corresponds to step 2 of relaying which is a MUST, as is the matter
> of stale articles (which is also a history file matter).

Good point.  So much for trying to save a bit of length.  :)

I now have:

   3.  It MUST reject any article that has already been successfully
       sent to it, based on the Message-ID header field of the article.
       To satisfy this requirement, a relaying agent normally keeps a
       database of message identifiers it has already accepted.

   4.  It SHOULD reject any article that matches an already-received and
       honored cancel message or Supersedes header field following the
       same rules as a relaying agent (Section 3.5).

which is basically the same text as for relaying agents.

> OK, but people will think it odd that you mention the obscure case of
> cancels which arise first here, but make to mention of the common case.
> Perhaps an extra step to say that it SHOULD then process any control
> messages (including cancels) in acccordance with section 5.

This would need to be added to relaying agents as well.  I don't feel like
it's really necessary, but I don't object if others in the WG would like
it to be added.  (I think that's MAY, not SHOULD; many news servers don't
process control messages at all and unless they want to honor cancels,
there's no need for them to do so.  There are numerous other ways to
maintain an active newsgroup list.)

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Xref and relaying agents (was: Protocol changes in draft-allbery-usefor-usepro-00)
Organization: The Eyrie
Date: Fri, 29 Dec 2006 13:42:26 -0800
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:
> Russ Allbery <[email-masked]> writes:
>> Charles Lindsey <[email-masked]> writes:

>>> 20. [00] (3.5) Relaying agent MAY add a new Xref header.

>>> Why might it want to do that?

>> Intentional change.

>> Because the same piece of software also implements a serving agent,
>> which has to add the Xref header, and the same code path is used for
>> all articles received by the server whether they are for further
>> relaying or local serving.

> Well if that is the case it was nominally the serving function that
> added the Xref (section 3.6).

No, because changes made by serving agents would only be seen by posting
agents.  Serving agents do not sent messages to any other news server.
Only relaying agents do that.

> Now whether such an agent first stores the article and then relays what
> it has stored, or whether it relays first and then stores after is none
> of our business.

Our agent description describes the flow, and doesn't allow for this,
conceptually.  This is true even in the current USEPRO draft.

   An "injecting agent" takes the finished article from the posting
   agent (often via the NNTP "POST" command), performs some final checks
   and passes it on to a "relaying agent" for general distribution.

   A "relaying agent" is software which receives allegedly compliant
   articles from injecting agents and/or other relaying agents, and
   possibly passes copies on to other relaying agents and "storage
   agents".

   A "storage agent" receives an article from a relaying agent and files
   it in a "news database". It also provides an interface for reading
   agents to access the news database.

The progression of an article is:

    posting -> injecting -> 1* relaying -> serving -> reading

Articles do not go to serving agents and then back to relaying agents.
This is, admittedly, the only place where it really matters from a
protocol standpoint, but it affects the wording in subtle ways in various
other places and I'd rather keep the clarity.

Because of this, the protocol description has to say that the relaying
agent may add an Xref header, since otherwise nothing in the protocol
permits the Xref header to appear in the article until it reaches a
serving agent.

Few news implementations have a true serving agent as defined in our
document.  INN, for example, splits that function between innd and nnrpd.
Diablo is the only implementation I'm familiar with that has a pure
serving agent as a separate service, although I think the Highwind
commercial servers may also have an equivalent.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Moderation fowarding wording for injecting agents (was: Protocol changes in draft-allbery-usefor-usepro-00)
Organization: The Eyrie
Date: Fri, 29 Dec 2006 13:31:17 -0800
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:
> Russ Allbery <[email-masked]> writes:

>> This section says more than just that, and I think the change makes
>> more sense with more context:

>>   7.   If the Newsgroups header contains one or more moderated groups
>>        and the proto-article does not contain an Approved header field,
>>        the injecting agent SHOULD forward it to a moderator as
>>        specified in Section 3.4.1.  If the article cannot be forwarded
>>        to a moderator for some reason, it MUST be rejected.

> OK, the wording could be better. How about:

>    7.  If the Newsgroups header contains one or more moderated groups and
> 	the proto-article does not contain an Approved header field, the
> 	injecting agent MUST either forward it to a moderator as specified
> 	in Section 3.4.1 or, if that is not possible, reject it.

Yup, that looks better to me as well and it's even shorter.  Changed in my
version.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Injecting agents and From (was: Re: Protocol changes in draft-allbery-usefor-usepro-00)
Organization: The Eyrie
Date: Fri, 29 Dec 2006 13:27:36 -0800
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:
> Russ Allbery <[email-masked]> writes:
>> Charles Lindsey <[email-masked]> writes:

>>> 3. [-1] (3.3.1,3.4,3.9.2) From not omittable in proto-article

>>> There is existing usage where the injecting agent fills in From header
>>> (not possible in NNTP, of course)

>> Intentional change.

>> What injecting agent supports this?

> INN apparently (see below).

> Newsreaders such as rn, trn, nn, and others of that generation had (and
> still have) the capability to interact directly with the newspool
> (either on the same host, or more likely NFS mounted from some server)
> rather than going via NNTP (which did not exist when they were first
> written). They injected articles by calling a program 'inews' which, in
> the absence of an explicit From:, assumed the poster was the user who
> had called 'inews'.

inews as distributed with INN or the common news readers is a (part of a)
posting agent, not an injecting agent.  You can see this by observing what
actions it takes and what agent it talks to.  It sends messages to an
injecting agent via POST and expects that agent to do the injection; it
doesn't perform those actions itself and then send messages directly to
relaying and serving agents the way that an injecting agent does.

Our agent distinctions are largely based on how INN and similar news
servers work.  C News, being a much earlier implementation, may not have
the same distinctions, or they may be much less clear.  Some
implementations written prior to our standard will combine different
agents into one program, so it's possible that in C News inews combines
functions of a posting agent and an injecting agent.  However, I believe
you'll find that treating creation of the From header field as a function
of the posting agent, which is possibly combined with an injecting agent,
will work correctly and not contradict the work flow of any existing
implementation.

INN's inews used to redundantly perform a few (although not all by any
means) of the functions of an injecting agent, such as mailing posts to
the moderators of moderated newsgroups.  This behavior was a bad idea,
causing various problems in practice, and has been removed from current
versions.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: More Path notes (was: Protocol changes in draft-allbery-usefor-usepro-00)
Organization: The Eyrie
Date: Fri, 29 Dec 2006 13:17:17 -0800
Sender: [email-masked]

I'm going to start trying to break this message up into separate subjects
per topic.  I'll also try to only respond to places where there seems to
be more to say or some text change that's needed.

Charles Lindsey <[email-masked]> writes:
> Russ Allbery <[email-masked]> writes:
>> Charles Lindsey <[email-masked]> writes:

>>> 1. [-1] (2.2) <path-identity> - Possibility to use a FQDN not
>>> resolvable in the DNS.

>>> (This was a suggestion by Harald, & 1 of 2 alternatives in USEPRO)

>> Intentional, but I don't object to changing it.

Actually, I didn't make this change, but both of us misread my text.  :)
This case is still allowed (it's my 2nd point).  What changed was the
wording of the 1st point.

>> The intentional change here is that I added a different preferred first
>> option, namely the FQDN of the news server itself.

> Which I did not quite understand. Did you mean the A or AAAA record? In
> which case please be explicit. But I would prefer to allow CNAMEs and
> MXes as well.

Those are both allowed by the second option.

FQDN is used without further qualification in USEFOR as well (section
3.2.8).  I think it's a technical term that readers of standards should be
expected to know and don't think it needs further elaboration.  If it
does, USEFOR also needs to be modified.  One of the reasons why I'd rather
leave it as-is is that it's one of those terms where attempting a formal
definition can quickly get off into the weeds dealing with multihomed
hosts, different DNS RRs, and so forth.  But I can see why not clearly
defining a term in a standard could be a problem, so I can be convinced we
need to say something.  (Ideally, maybe there's a definition somewhere
else we can just reference?)

>>> 2. [00] (3.2) <path-identity>s are case-sensitive.

>>> I was always told that some servers treated them one way and some the
>>> other, so you could not rely on either. If anything, I would have
>>> expected case-insensitive (that is what domain-names are, and what
>>> USEFOR declares <diag-keyword>s are).

>> Intentional change.

>> Declaring them case-sensitive is the conservative choice.  If you treat
>> them as case-sensitive and they're actually compared
>> case-insensitively, everything works fine.  The opposite is not true;
>> if your Path identity varies in case, servers that compare
>> case-sensitively may add incorrect MISMATCH entries or send you
>> articles that you sent them.

> OK, I will buy that.

The more I research this, though, the more I think that while the above is
true, it's not ideal.  Poking around a bit, it looks like a *lot* of
existing implementations are case-insensitive, and I'm not sure I want to
declare that incorrect.  Accordingly, I now have the following text, which
I think is an improvement.  Please, everyone, let me know if you disagree:

   Some existing implementations treat <path-identity> as case-
   sensitive, some case-insensitive.  The <path-identity> therefore
   SHOULD be all lowercase and implementations SHOULD compare identities
   case-insensitively.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Moderator duties
Organization: The Eyrie
Date: Fri, 29 Dec 2006 12:45:28 -0800
Sender: [email-masked]

I went over the text in Duties of a Moderator about what a moderator is
allowed to do and came up with the following rewording.  Do folks feel
that this is an improvement?  Does this address some of the concerns with
the previous language?

   There are no protocol restrictions on what criteria are used for
   accepting or rejecting messages or on what modifications a moderator
   may make to a message (both header fields and body) before injecting
   it.  Recommended best practice, however, is to make the minimal
   required changes.  Moderators need to be aware that modifications
   made to articles may invalidate signatures created by the poster or
   previous moderators.  See [USEAGE] for further best-practice
   recommendations.

   Moderators process articles as follows:

   1.  They decide whether to approve or reject a proto-article, and if
       approved, prepare the proto-article for injection.  If the proto-
       article was received as an unencapsulated email message, this
       will require converting it back to a valid Netnews proto-article.
       If the article is rejected, it is normally rejected for all
       newsgroups to which it was posted and nothing further is done.
       If it is approved, the moderator proceeds with the following
       steps.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: draft-allbery-usefor-usepro-00 errata
Organization: The Eyrie
Date: Fri, 29 Dec 2006 12:33:16 -0800
Sender: [email-masked]

Russ Allbery <[email-masked]> writes:

> As a summary from the recent long threads, here are the changes I
> believe should be made to draft-allbery-usefor-usepro-00 if we go
> forward with that document.  The first one below has not previously been
> discussed on the list.

These errata have now been applied to my working copy.  Where the text was
substantial or I felt it was possibly controversial, I sent the new text
to the list; for the rest of these errata, I made the obvious changes.

>  * USEFOR was changed (correctly, IMO) to allow only WSP between the
>    arguments of control commands, not FWS.  I caught this for newgroup and
>    rmgroup but missed it for checkgroups and cancel, both of which need
>    FWS to WSP changes.

>  * multipart/related in newgroup control messages should be
>    multipart/mixed instead.

>  * application/news-transmission should explicitly note that, contrary to
>    the previous registration, batches are not permitted.

>  * The trimming requirement for References should probably go back to
>    MUST.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Path header field (was: draft-allbery-usefor-usepro-00 errata)
Organization: The Eyrie
Date: Fri, 29 Dec 2006 12:31:59 -0800
Sender: [email-masked]

Russ Allbery <[email-masked]> writes:

>  * A separate sub-section of duties, before the individual agent duties
>    and possibly as part of the same sub-section as the identity
>    discussion, should be added specifying Path header construction.  The
>    Path header example should be moved to be a sub-section of that
>    section.  That section can lay out all the construction requirements
>    for the Path header field and just be referred to by the individual
>    duties sections.

>  * The possibility of adding multiple path-identities should be
>    reintroduced as part of the rework of the Path description.

>  * Serving agents also are not required to modify the Path header field if
>    processing an article from a relaying agent or injecting agent that's
>    part of the same server.  This can be handled more generally in the
>    redone Path section.

As promised, here is the new section.  I also made the corresponding
updates to the other sections (removing the description and replacing it
with a reference to this section) and moved the Path header field example.

3.2.1.  Constructing the Path Header Field

   If a relaying or serving agent receives an article from an injecting
   or serving agent that is part of the same news server, it MAY leave
   the Path header field of the article unchanged.  Otherwise, every
   injecting, relaying, or serving agent that accepts an article MUST
   update the Path header field as follows.  Note that the Path header
   field content is constructed from right to left by prepending
   elements.

   1.  The agent MUST prepend "!" to the Path header field content.

   2.  An injecting agent SHOULD prepend the <path-diagnostic>
       "!.POSTED", optionally followed by "." and the FQDN or IP address
       of the source, to the Path header field content.

   3.  A relaying or serving agent SHOULD prepend a <path-diagnostic> to
       the Path header field content, where the <path-diagnostic> is
       chosen as follows:

       *  If the expected <path-identity> of the source of the article
          matches the leftmost <path-identity> of the Path header
          field's content, use "!" (<diag-match>).

       *  If the expected <path-identity> of the source of the article
          does not match, use "!.MISMATCH." followed by the expected
          <path-identity> of the source or its IP address.

       *  If the relaying or serving agent is not willing or able to
          check the <path-identity>, use "!.SEEN." followed by the FQDN,
          IP address, or expected <path-identity> of the source.

   4.  The agent MUST then prepend its own <path-identity> to the Path
       header field content.

   5.  The agent MAY then prepend additional <path-identity>s for itself
       to the Path header field content, following each <path-identity>
       so added with either "!!" or "!".  This is permitted for agents
       that have multiple <path-identity>s (such as during a transition
       from one to another).  Each of these <path-identity>s MUST meet
       the requirements set out in Section 3.2.

   Any agent which modifies the Path header field MAY fold it by
   inserting FWS immediately after any <path-identity> or <diag-other>
   it added (see section 3.1.5 of [USEFOR] for allowable locations for
   FWS).

>  * (Still under discussion.)  We may wish to reintroduce the possibility
>    of a path-identity that is not a resolvable name in DNS.

This was already in my draft.  I have:

   The <path-identity> used by an agent may be chosen via one of the
   following methods (in decreasing order of preference):

   1.  The fully-qualified domain name (FQDN) of the system on which the
       agent is running.

   2.  A fully-qualified domain name (FQDN) within a domain affiliated
       with the administrators of the agent and guaranteed to be unique
       by the administrators of that domain.  For example, the
       uniqueness of server.example.org could be guaranteed by the
       administrator of example.org even if there is no DNS record for
       server.example.org itself.

   3.  Some other (arbitrary) name in the form <path-nodot> believed to
       be unique and registered at least with all the other news servers
       to which that relaying agent or injecting agent sends articles.
       This option SHOULD NOT be used unless the earlier options are
       unavailable or unless the name is of longstanding usage.

This is the same as what's in the current USEPRO draft except that my 1.
replaces:

   1. A fully qualified domain name (FQDN) that SHOULD be resolvable in
      the DNS (whether via an A, AAAA or MX record or an equivalent
      CNAME), thus guaranteeing a unique identity. Ideally, it will also
      provide a means to contact the administrators by email (according
      to [RFC 2142], the forms "usenet@server" and "news@server" are
      common addresses for a news server administrator).

or

   1. A fully qualified domain name (FQDN) that can be resolved to an
      email server via an MX, A or AAAA record according to the
      procedures of [RFC 2821]; this guarantees that the name is unique,
      and makes it easy to contact the administrators if needed.

The most common current practice is to use the FQDN of the news server,
not a mail server, as a path identity.  If one wants to use a mail server,
2. still allows for that; it doesn't require that the name *not* exist in
DNS.

In my original draft, I intentionally removed any protocol-required
connection between a <path-identity> and a contact address.  After lots of
previous discussion of this point, I don't think the potential gains of
discussing such a connection in this specification are worth the
complexity and divergence from current practice.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: to.* newsgroups (was: draft-allbery-usefor-usepro-00 errata)
Organization: The Eyrie
Date: Fri, 29 Dec 2006 11:36:11 -0800
Sender: [email-masked]

Russ Allbery <[email-masked]> writes:

>  * (Still under discussion.)  We may wish to add a sentence about the
>    construction of newsgroups in the to.* hierarchy.

I'm convinced.  I now have:

   These control messages are normally sent as point-to-point articles
   between two sites and not then sent on to other sites.  Newsgroups
   beginning with "to." are reserved for such point-to-point
   communications and are formed by prepending "to." to a <relayer-name>
   to form a <newsgroup-name>.  Articles with such a group in their
   Newsgroups header fields SHOULD NOT be sent to any news server other
   than the one identified by <relayer-name>.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: ihave/sendme syntax (was: draft-allbery-usefor-usepro-00 errata)
Organization: The Eyrie
Date: Fri, 29 Dec 2006 11:21:49 -0800
Sender: [email-masked]

Russ Allbery <[email-masked]> writes:

>  * We need some resolution of the different conflicting syntaxes for ihave
>    and sendme control messages.  Path of least resistance is probably to
>    revert to the current USEPRO tactic of two separate ABNFs.

I've since reviewed RFC 1036 and Son-of-1036.  The former makes the
relayer name optional and requires that the message IDs be in the control
header field.  The latter makes the relayer name mandatory and RECOMMENDS
that the message IDs be put in the body.  I think the following is the
most reasonable compromise; it makes the relayer-name mandatory if there
are no message IDs (the form never allowed in RFC 1036 but according to
Son-of-1036 the most widely used in practice), and if there are message
IDs, permits the same syntax as RFC 1036.

Let me know if anything looks wrong.

   ihave and sendme control messages share similar syntax for their
   Control header fields and bodies:

       control-command     =/ Ihave-command
       Ihave-command       = "ihave" Ihave-arguments
       Ihave-arguments     = 1*WSP relayer-name
                             / 1*( 1*WSP msg-id ) [ 1*WSP relayer-name ]
       control-command     =/ Sendme-command
       Sendme-command      = "sendme" Sendme-arguments
       Sendme-arguments    = Ihave-arguments
       relayer-name        = path-identity  ; see 3.1.5 of [USEFOR]
       ihave-body          = *( msg-id CRLF )
       sendme-body         = ihave-body

   The body of the article consists of a list of <msg-id>s, one per
   line.  The message identifiers SHOULD be put in the body of the
   article, not in the Control header field, but news servers MAY
   recognize and process message identifiers in the Control header field
   for backward compatibility.  Message identifiers MUST NOT be put in
   the Control header field if they are present in the body of the
   control message.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Control message propagation (was: draft-allbery-usefor-usepro-00 errata)
Organization: The Eyrie
Date: Fri, 29 Dec 2006 11:00:14 -0800
Sender: [email-masked]

Russ Allbery <[email-masked]> writes:

>  * The correct description of special routing for newgroup and rmgroup
>    control messages is something more like:

>      Exceptionally, control messages creating or removing newsgroups
>      (newgroup or rmgroup control messages, for example) SHOULD be relayed
>      if the affected group appears in its Newsgroups header field and the
>      sending agent would have supplied and the receiving agent would have
>      received the newsgroup affected by the control message had it
>      existed, even if it currently does not.

>    The current USEPRO text for construction of the Newsgroups header field
>    for those control messages should be restored (and probably as a
>    general statement about group control messages that affect only one
>    group rather than stated independently in both newgroup and rmgroup).

I now have:

   Exceptionally, control messages creating or removing newsgroups
   (newgroup or rmgroup control messages, for example) SHOULD be relayed
   if the affected group appears in its Newsgroups header field and the
   sending agent would have supplied and the receiving agent would have
   received the newsgroup affected by the control message had it existed,
   even if it currently does not.

for the propagation rule, the following under group control messages:

   Group control messages affecting specific groups (newgroup and
   rmgroup control messages, for example) SHOULD include the <newsgroup-
   name> for the group or groups affected in their Newsgroups header
   field.  Other newsgroups MAY be included in the Newsgroups header
   field so that the control message will reach more news servers, but
   due to the special relaying rules for group control messages (see
   Section 3.5) this is normally unnecessary and may be excessive.

the following under cancel control messages:

   A cancel control message SHOULD have the same Newsgroups header field
   as the message it is cancelling so that it will attempt to reach the
   same news servers as the original message.

and the following in Security Considerations:

   Cancel control messages are not required to have the same Newsgroups
   header field as the messages they are cancelling and, since they are
   sometimes processed before the original message is received, it may
   not be possible to check that they do.  This allows a malicious
   poster to inject a cancel control message for an article in a
   moderated newsgroup without adding an Approved header field to the
   control message, and to hide malicious cancel control messages from
   some reading agents by using a different Newsgroups header field so
   that the cancel control message is not accepted by all news servers
   that accepted the original message.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: Decision: Let's get draft-allbery-usefor-usepro-00 right and publish it as draft-ietf-usefor-usepro-07
Organization: The Eyrie
Date: Fri, 29 Dec 2006 10:28:21 -0800
Sender: [email-masked]

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: charset of application/news-{groupinfo,checkgroups} (was: Re: ASCII)
Organization: The Eyrie
Date: Fri, 29 Dec 2006 10:06:01 -0800
Sender: [email-masked]

>> IMO it's obvious, but that future draft could say that using any
>> charset that's not UTF-8, like Latin-1, is NOT RECOMMENDED if non-ASCII
>> group names are affected.

> Exactly.

> Currently non-ASCII group names are banned entirely by USEFOR, so it
> doesn't make much sense to say this now.

> We could RECOMMEND use of either US-ASCII or UTF-8 for the charset in
> the name of making future transitions easier, though.  I can see some
> merit in going that route, particularly since I think we're fairly
> certain at this point that newsgroup names will either be UTF-8 or some
> ASCII encoding.  I don't see any other non-ASCII character set cropping
> up in the IETF.

While implementing this wording change, I realized that most of the
problem is already handled by the ABNF for these media types.  It already
specifies specific octets for things such as HTAB and the allowable
characters for a newsgroup name.  Because of that, it looked to me like we
could simplify this language considerably and use a sentence closer to
what Charles suggested.

The ABNF was restricting the newsgroup description to 7bit characters, so
that also needed to be changed.

Here's what I currently have:

   The content of the application/news-groupinfo body part is defined
   as:

         groupinfo-body      = [ newsgroups-tag CRLF ]
                                  newsgroups-line CRLF
         newsgroups-tag      = %x46.6F.72 SP %x79.6F.75.72 SP
                                  %x6E.65.77.73.67.72.6F.75.70.73 SP
                                  %x66.69.6C.65.3A
                                  ; case sensitive
                                  ; "For your newsgroups file:"
         newsgroups-line     = newsgroup-name
                                  [ 1*HTAB newsgroup-description ]
                                  [ 1*WSP moderation-flag ]
         newsgroup-description
                             = eightbit-utext *( *WSP eightbit-utext )
         moderation-flag     = %x28.4D.6F.64.65.72.61.74.65.64.29
                                  ; case sensitive "(Moderated)"
         eightbit-utext      = utext / %d127-255

   This unusual format is backward-compatible with the scanning of the
   body of newgroup control messages for descriptions done by Netnews
   implementations that predate this specification.  Although optional,
   the <newsgroups-tag> SHOULD be included for backward compatibility.

   The <newsgroup-description> MUST NOT contain any occurrence of the
   string "(Moderated)" within it.  Moderated newsgroups MUST be marked
   by appending the case sensitive text " (Moderated)" at the end.

   While a charset parameter is defined for this media type, most
   existing software does not understand MIME header fields or correctly
   handle descriptions in a variety of charsets.  Using a charset of US-
   ASCII where possible is therefore RECOMMENDED; if not possible, UTF-8
   [RFC3629] SHOULD be used.  Regardless of the charset used, the
   constraints of the above grammar MUST be met and the <newsgroup-name>
   MUST be represented using the same octets as would be used with a
   charset of US-ASCII.

and under application/news-checkgroups:

   The same restrictions on charset, <newsgroup-name>, and <newsgroup-
   description> apply for this media type as for application/
   news-groupinfo.

Let me know if I'm missing a subtlety that would have been expressed by
Harald's language.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: Leaf node news servers and posting (Re: Message IDs and moderators)
Organization: The Eyrie
Date: Fri, 22 Dec 2006 09:36:12 -0800
Sender: [email-masked]

Russ Allbery <[email-masked]> writes:

> In actual practice, they work the way that I describe them in my
> document, namely as two independent networks and a gateway with all
> gatewaying work done by the stand-alone server.

Someone pointed out to me in private e-mail that there's another
frequently used way of doing this which avoids most of the problems
(including the need to do gatewaying), namely immediately proxying any
POST to a remote server.  Some of the end-point single-host news servers
just keep track of where different groups go and, when receiving a POST,
never store the article locally and instead immediately proxy the POST to
the appropriate server.  Then, when the article shows up on that server,
the normal mechanism pulls it down to the local article store.

This system is much more robust than having to do gatewaying in any
fashion, but has the drawback that it can't function off-line and that it
requires a specialized news server that knows to do unusual things with
POST (as opposed to allowing one to plug a gateway into a generic
all-purpose news server).  If I remember correctly, leafnode works this
way.

This is really the best way to handle this, and avoids having to deal with
the issues in either my draft or in the current USEPRO draft.  There's no
reinjection or gatewaying, just the proxying of the conversation between
the posting agent and the injecting agent.  It just has a few drawbacks
that makes something like INN + suck better in a few situations, mostly
ones where you want to maintain a set of local newsgroups in addition to
gating newsgroups from another Netnews network.

To: [email-masked]
Xref: clerew local.usefor:23944
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Leaf node news servers and posting (Re: Message IDs and moderators)
Date: Fri, 22 Dec 2006 15:34:33 GMT
Lines: 66
Sender: [email-masked]

In <[email-masked]> Harald Tveit Alvestrand <[email-masked]> writes:

>--On torsdag, desember 21, 2006 19:47:43 +0000 Charles Lindsey 
><[email-masked]> wrote:

>> So he posts articles which appear at first in his local serving agent, and
>> then get sent to his ISP's news spool. His ISP will expect him to use
>> POST, and will add an Injection-Info, with complaints directed to the
>> abuse dept. OK so far, and the guy will have to arrange things are done as
>> his ISP requests. And, Shock Horror! maybe it even has an Xref when the
>> ISP gets it.

>can someone who actually runs one of those beasts explain how they handle 
>posting of articles - in particular, whether they really make an article 
>available to their local serving agent before sending it to the ISP's news 
>server?

Yes, my own system runs that way (I run a CNews server, as I am sure you
all know). I download mostly from NIN (news.individual.net), but for some
obscure groups from other servers, and I gateway in this list and some
others.

Anything I post from my newsreader gets injected into my local server
(but, since Injection-Info, Injection-Date and .POSTED are not yet
implemented, the only evidence inside an article is the presence of an
Xref header and a short Path).

For sending stuff out to the wide world, I again use various servers
according to the group. CNews is configurable to use any script you care
to provide for each outgoing feed. So I can arrange for each feed whether
it looks like a relay (using IHAVE) or like an injection (using POST, or
the NNRP "IHAVE" which is really injection) or some mail2news, or a
gateway to a list such as this.

So, for most normal groups I relay to Manchester University and, 30
seconds later, inject to NIN. The University facility is useful because it
is guaranteed not to munge anything that might break a PGPVERIFY, but its
administration is a complete shambles (it has been totally down for the
last couple of weeks). OTOH NIN is extremely reliable, but fussy about
what it accepts (e.g. it insists that all From mail addresses are genuine
and checks that an MX record exists). So when wearing my part-time
uk.net.news.announce moderator's hat and having to post a Call For Votes
from a votetaker who insists on munging his From address to defeat spam
harvesters, I have to rely on the Manchester server to get it out. All
that is, of course, quite untypical of the average Usenet user, but
illustrates some of the stranger corner cases that can arise.

So, technically speaking, when an article is injected at NIN, it is
"reinjection". So when USEFOR/USEPRO are in force I shall have to consider
what Injection-Info etc headers to generate, and whether to include them
in all my outgoing feeds or not, and that will depend I suppose on what
NIN are prepared to accept.

It so happens that I never send out an article that was not generated
locally, otherwise I would need to be much more careful.

Date: Fri, 22 Dec 2006 11:39:26 -0500
From: Dan Schlitt <[email-masked]>
To: Harald Tveit Alvestrand <[email-masked]>
cc: Charles Lindsey <[email-masked]>, [email-masked]
Subject: Re: Message IDs and moderators
Sender: [email-masked]

Customs, traditions, etc. are not protocol issues. They are useage
issues if anything at all is to be said about them.

Defining the boundry as Russ has done seems quite reasonable.

/dan

-- 

Dan Schlitt
[email-masked]

On Fri, 22 Dec 2006, Harald Tveit Alvestrand wrote:

>
>
>
> --On torsdag, desember 21, 2006 19:47:43 +0000 Charles Lindsey
> <[email-masked]> wrote:
>
> >
> > In <[email-masked]> Russ Allbery <[email-masked]>
> > writes:
> >
> >> stanley <[email-masked]> writes:
> >
> >>> Yes, and that is why I expect the attempt to get the codified right to
> >>> edit articles in any way the moderator wishes removed will be a losing
> >>> battle. It would be impossible to prevent, it isn't a matter for the
> >>> protocol, and many people don't care. I cannot accept it on ethical
> >>> grounds, however, and that's why I have to object to it being made
> >>> explicit.
> >
> >> Okay, that makes sense to me.  I'm also happy to try to reword what the
> >> protocol document says so that it specifies that it doesn't constrain
> >> rather than giving explicit blessing to behavior that is controversial.
> >> That can be tricky to do and still be clear, but the purpose, as far as
> >> I'm concerned, is to draw a clear boundary around the protocol and say
> >> that beyond that boundary the protocol doesn't place requirements, not to
> >> say that the protocol is implicitly blessing some particular course of
> >> action.
> >
> > Yes, but the trouble is that "clear boundaries" do not exist in the Real
> > World (TM). Whatever we say, moderators will continue to do whatever they
> > do, and offended readers will send flames and LARTs to ISPs, the B8MB,
> > their lawyers, NANU, and anywhere else they can think of. And eventually
> > the fuss will die down and peace will resume. That is the way that Usenet
> > works. But what we don't want is for any of the parties in the flame war
> > to be able to point to the standard and say "look, it says that moderators
> > can make any alterations they choose".
>
> Poppycock. We do that all the time (define clear boundaries).
>
> And we have tended to put LESS requirements on operators' behaviour as time
> goes by, exactly because our well-meant requirements, stated in an earlier
> time, was used as a LART against people in inappropriate ways.
>
> That's why whe had to revise WHOIS to make it clear that there's no
> PROTOCOL requirement to put the telephone number into a WHOIS record, for
> instance. The "rfc-ignorant" folks were LARTing.
>
> (in case some people haven't seen this term recently: I think LART = Loser
> Attitude Readjustment Tool, also called a "clue-by-4"....)
>
> > The fact is that Usenet has 'customs', 'conventions' and 'rules'
> > established by consensus amongst its users and/or the servers that carry
> > it, and you cannot describe how it works without at least admitting that
> > they exist (though just how they are established is another matter).
> > USEPRO at least admitted that they existed, and hinted at various places
> > that they needed to be followed. Maybe USEPRO said too much, but RUSSPRO
> > has gone to the opposite extreme, and that does not work either.
>
> In the cases cited so far, it works for me.
>

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  OT (was: Message IDs and moderators)
Date:  Fri, 22 Dec 2006 14:00:16 +0100
Organization:  <URL:http://purl.net/xyzzy>
Lines: 39
Sender: [email-masked]

Harald Tveit Alvestrand wrote:

Some potential cases of TIN[WU] (= "There Is No We or Us"):

>> But what we don't want is for any of the parties in the flame war
>> to be able to point to the standard and say "look, it says that
>> moderators can make any alterations they choose".

> Poppycock. We do that all the time (define clear boundaries).

> And we have tended to put LESS requirements on operators' behaviour
> as time goes by, exactly because our well-meant requirements,
> stated in an earlier time, was used as a LART against people in
> inappropriate ways.

> That's why whe had to revise WHOIS to make it clear that there's
> no PROTOCOL requirement to put the telephone number into a WHOIS
> record, for instance. The "rfc-ignorant" folks were LARTing.

They still do this for _wrong_ phone numbers, same idea as in the
WDPRS (ICANN's whois data problem report system).  Killing RFC 954
didn't help "you", I've found RFC 1032 to replace 954 by something
"better" than 3912.  Of course RFC 1591 would be the best, but it's
"only" informational.  John said that's bogus, and RFC 1591 should
be a BCP.  With that I think that I could convince the RFCI folks
to replace RFC 1032 by 1591.  But Brian didn't like John's proposal
to fix the category of 1591, so that's at a stalemate for now.

>> Maybe USEPRO said too much, but RUSSPRO has gone to the opposite
>> extreme, and that does not work either.

> In the cases cited so far, it works for me.

Not for me, like RFC 3912 was a case of "Do Not Publish".  At that
time I still thought that the RFCI folks would know how to stop it
in its "Last Call".  It turned out that they didn't know it... :-(

Frank

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: Leaf node news servers and posting (Re: Message IDs and moderators)
Organization: The Eyrie
Date: Thu, 21 Dec 2006 23:19:49 -0800
Sender: [email-masked]

Harald Tveit Alvestrand <[email-masked]> writes:

> can someone who actually runs one of those beasts explain how they
> handle posting of articles - in particular, whether they really make an
> article available to their local serving agent before sending it to the
> ISP's news server?

In actual practice, they work the way that I describe them in my document,
namely as two independent networks and a gateway with all gatewaying work
done by the stand-alone server.

Take, for instance, the popular news package "suck."  It pulls down
messages from a remote system using reader commands and then injects them
into a local news server using IHAVE (a relaying command).  Then, when
someone posts locally to the server, that posting is accepted by the local
server, immediately made available to the local server's readers, and
queued in what, to the server, looks like a regular outgoing feed, a text
file that contains a queue of message IDs to send to a remote server.
However, rather than running a typical relayer command to do that sending,
one runs rpost instead.

rpost takes the message from the local system, converts it back into a
proto-article by stripping or renaming NNTP-Posting-Host, X-Trace, and
other unstandardized trace headers plus headers such as Xref, and then
offers it to the remote injection agent as a new article.

In practice, this works fine.

Currently, the Date header is conventionally preserved through this
transaction, which helps protect against a badly misconfigured rpost
client posting old messages.  The one place where my document would change
this, which is a side effect of the introduction of Injection-Date, is
that rpost would gain an additional requirement of doing date checks
itself to be sure that it wasn't given a bad batch of ancient articles
that shouldn't be gatewayed.  Currently, injecting agents are still also
required to check the Date header.  Should that requirement ever be
dropped, the onus on preventing regurgitation of ancient articles would be
entirely on rpost.

This has advantages and drawbacks.  In some respects, this scheme is a
serious drawback to introducing Injection-Date; in the process of solving
a not horribly significant problem with delayed postings from off-line
readers that can be worked around in other ways, we're messing with one of
the most basic parts of loop detection and introducing other complexities.
However, this scenario also gains from the exact same enhancements as any
other off-line post does: the actual posting date of the message can be
retained even if the stand-alone server from which the posts are being
gatewayed is off-line for several days.  Most importantly, it puts the
onus of the work on the agent that's introducing the risk and that is in
the best position to know the correct action to take, rather than making
every injection agent deal with this unusual case.

What's in USEPRO contradicts existing practice.  What's in my document is
a translation of existing practice that introduces Injection-Date.  My
*preferred* solution after trying to sort out this whole area would be to
get rid of Injection-Date as a nice idea that's too late to introduce, at
which point we're back to what we have right now and what we know works,
and which we therefore can make some guarantees about safety, but that
requires revisiting USEFOR and it's not the time to do that.  Failing
that, modelling this as a gateway is substantially cleaner than trying to
standardize reinjection and simplifies the picture considerably.  The cost
is one missed opportunity to catch a regurgitation of ancient articles at
the injection agent level (and, in practice, not a missed opportunity in
*this* standard since my document still requires that injection agents
check the Date header, but a potential missed opportunity later should we
ever drop that requirement).  I think from a protocol design perspective,
that tradeoff is worth it.

The closer the protocol design is to what is being done today and is
working today, the more comfortable I am that it doesn't have hidden
gotchas that we're going to regret later.

Date: Fri, 22 Dec 2006 07:48:08 +0100
From: Harald Tveit Alvestrand <[email-masked]>
To: Charles Lindsey <[email-masked]>, [email-masked]
Subject: Leaf node news servers and posting (Re: Message IDs and moderators)
Sender: [email-masked]

--On torsdag, desember 21, 2006 19:47:43 +0000 Charles Lindsey 
<[email-masked]> wrote:

> But then there are still some corner cases. mostly tied up with
> reinjection (which I have written at length about in another thread, and
> which I am still hoping for a reply to). But here is an example of a
> corner case.
>
> Some guy runs a news server on his home machine (quite a lot of people do
> that, because they find it more convenient that fiddling with one of the
> big newsreaders such as Turnpike or Agent).
>
> So he posts articles which appear at first in his local serving agent, and
> then get sent to his ISP's news spool. His ISP will expect him to use
> POST, and will add an Injection-Info, with complaints directed to the
> abuse dept. OK so far, and the guy will have to arrange things are done as
> his ISP requests. And, Shock Horror! maybe it even has an Xref when the
> ISP gets it.

can someone who actually runs one of those beasts explain how they handle 
posting of articles - in particular, whether they really make an article 
available to their local serving agent before sending it to the ISP's news 
server?

                      Harald

Date: Fri, 22 Dec 2006 07:45:43 +0100
From: Harald Tveit Alvestrand <[email-masked]>
To: Charles Lindsey <[email-masked]>, [email-masked]
Subject: Re: Message IDs and moderators
Sender: [email-masked]

--On torsdag, desember 21, 2006 19:47:43 +0000 Charles Lindsey 
<[email-masked]> wrote:

>
> In <[email-masked]> Russ Allbery <[email-masked]>
> writes:
>
>> stanley <[email-masked]> writes:
>
>>> Yes, and that is why I expect the attempt to get the codified right to
>>> edit articles in any way the moderator wishes removed will be a losing
>>> battle. It would be impossible to prevent, it isn't a matter for the
>>> protocol, and many people don't care. I cannot accept it on ethical
>>> grounds, however, and that's why I have to object to it being made
>>> explicit.
>
>> Okay, that makes sense to me.  I'm also happy to try to reword what the
>> protocol document says so that it specifies that it doesn't constrain
>> rather than giving explicit blessing to behavior that is controversial.
>> That can be tricky to do and still be clear, but the purpose, as far as
>> I'm concerned, is to draw a clear boundary around the protocol and say
>> that beyond that boundary the protocol doesn't place requirements, not to
>> say that the protocol is implicitly blessing some particular course of
>> action.
>
> Yes, but the trouble is that "clear boundaries" do not exist in the Real
> World (TM). Whatever we say, moderators will continue to do whatever they
> do, and offended readers will send flames and LARTs to ISPs, the B8MB,
> their lawyers, NANU, and anywhere else they can think of. And eventually
> the fuss will die down and peace will resume. That is the way that Usenet
> works. But what we don't want is for any of the parties in the flame war
> to be able to point to the standard and say "look, it says that moderators
> can make any alterations they choose".

Poppycock. We do that all the time (define clear boundaries).

And we have tended to put LESS requirements on operators' behaviour as time 
goes by, exactly because our well-meant requirements, stated in an earlier 
time, was used as a LART against people in inappropriate ways.

That's why whe had to revise WHOIS to make it clear that there's no 
PROTOCOL requirement to put the telephone number into a WHOIS record, for 
instance. The "rfc-ignorant" folks were LARTing.

(in case some people haven't seen this term recently: I think LART = Loser 
Attitude Readjustment Tool, also called a "clue-by-4"....)

> The fact is that Usenet has 'customs', 'conventions' and 'rules'
> established by consensus amongst its users and/or the servers that carry
> it, and you cannot describe how it works without at least admitting that
> they exist (though just how they are established is another matter).
> USEPRO at least admitted that they existed, and hinted at various places
> that they needed to be followed. Maybe USEPRO said too much, but RUSSPRO
> has gone to the opposite extreme, and that does not work either.

In the cases cited so far, it works for me.

To: [email-masked]
Xref: clerew local.usefor:23940
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Message IDs and moderators
Date: Thu, 21 Dec 2006 19:47:43 GMT
Lines: 115
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>stanley <[email-masked]> writes:

>> Yes, and that is why I expect the attempt to get the codified right to
>> edit articles in any way the moderator wishes removed will be a losing
>> battle. It would be impossible to prevent, it isn't a matter for the
>> protocol, and many people don't care. I cannot accept it on ethical
>> grounds, however, and that's why I have to object to it being made
>> explicit.

>Okay, that makes sense to me.  I'm also happy to try to reword what the
>protocol document says so that it specifies that it doesn't constrain
>rather than giving explicit blessing to behavior that is controversial.
>That can be tricky to do and still be clear, but the purpose, as far as
>I'm concerned, is to draw a clear boundary around the protocol and say
>that beyond that boundary the protocol doesn't place requirements, not to
>say that the protocol is implicitly blessing some particular course of
>action.

Yes, but the trouble is that "clear boundaries" do not exist in the Real
World (TM). Whatever we say, moderators will continue to do whatever they
do, and offended readers will send flames and LARTs to ISPs, the B8MB,
their lawyers, NANU, and anywhere else they can think of. And eventually
the fuss will die down and peace will resume. That is the way that Usenet
works. But what we don't want is for any of the parties in the flame war
to be able to point to the standard and say "look, it says that moderators
can make any alterations they choose".

The fact is that Usenet has 'customs', 'conventions' and 'rules'
established by consensus amongst its users and/or the servers that carry
it, and you cannot describe how it works without at least admitting that
they exist (though just how they are established is another matter).
USEPRO at least admitted that they existed, and hinted at various places
that they needed to be followed. Maybe USEPRO said too much, but RUSSPRO
has gone to the opposite extreme, and that does not work either.

Essentially, the customary expectation is that moderators may fiddle with
"boilerplate" but no more, but how can you say that in precise terms? You
can't, but you can at least point to "established custom", in a NOTE if
needs be. The same is true for group creation and other such things.
RUSSPRO uses the word "authorization" in various places, and such a term
is definitely needed. But nowhere does it define that term, nor say who
authorizes the authorized. USEPRO at least admitted that the problem
existed, and carefully said that it did not attempt to provide an explicit
answer, though it did say that the answer might vary from hierarchy to
hierarchy.

>> As a side issue -- how does one differentiate between a proto-article
>> that has all the mandatory headers and a real article? What should be
>> the result of differentiating between them? I.e., if you have a
>> proto-article that contains all the mandatory headers, how do you treat
>> it differently than a real article?

>My document defines it as follows:

>   A proto-article has the same format as a normal article except that
>   the Injection-Date, Injection-Info, and Xref header fields MUST NOT
>   be present; the Path header field MUST NOT contain a "POSTED" <diag-
>   keyword>; and any of the following mandatory header fields MAY be
>   omitted: Message-ID, Date, and Path.

I think it is not so much a matter of looking at the article as asking
where it was located when you looked at it. If it was still on your
machine waiting to go to the injector, or if it is in some moderator's
input queue, then it is a proto-article. If it is already propagating
around Usenet, then it an article.

But then there are still some corner cases. mostly tied up with
reinjection (which I have written at length about in another thread, and
which I am still hoping for a reply to). But here is an example of a corner
case.

Some guy runs a news server on his home machine (quite a lot of people do
that, because they find it more convenient that fiddling with one of the
big newsreaders such as Turnpike or Agent).

So he posts articles which appear at first in his local serving agent, and
then get sent to his ISP's news spool. His ISP will expect him to use
POST, and will add an Injection-Info, with complaints directed to the
abuse dept. OK so far, and the guy will have to arrange things are done as
his ISP requests. And, Shock Horror! maybe it even has an Xref when the
ISP gets it.

But now his neighbour hears that he has some newsgroups (mayble even some
'local' groups for his neighbourhood) and says "Hey! can you let me have a
feed of those, and BTW can you feed me comp.widgets at the same time as I
am interested in those".

Now what does he do? He really ought to put an Injection-Date on what he
sends to his neighbour, because he cannot be sure whether his neighbour
has other connections (or more likely might establish other connections in
the future without telling him), and the articles he supplies might leak
out onto Usenet that way. So they had better be clean decent articles,
traceable back to him.

So now does he arrange to send different versions of an article to his
neighbour and to his ISP, one already injected and the other not? Or does
he send the same to both? Ah! Now we are into reinjection! Now is it a
proto-article or a real article. Who knows? More importantly, who cares?

Maybe his ISP absolutely forbids it. Fair enough, but if the ISP decides
to reinject it, why not? He just replaces the Injection-Info, if present,
with his own. BUT any Injection-Date MUST then be left as it is.

Date: Thu, 21 Dec 2006 20:46:24 +0000
From: Alexey Melnikov <[email-masked]>
To: [email-masked]
CC: Harald Alvestrand <[email-masked]>, Lisa Dusseault <[email-masked]>
Subject: Status of the USEFOR document
Sender: [email-masked]

Hi folks,
After several discussions between Lisa, Harald and myself it became 
clear that making the reference to USEPRO normative would be the 
easiest/quickest way to remove remaining IESG DISCUSSes. This change 
will delay publication of the USEFOR document until the USEPRO document 
is approved by IESG.

Any objections to doing that?

Alexey

P.S. If the WG finds some serious flaw in USEFOR while working on 
USEPRO, it would be still possible to remove USEFOR from the RFC 
editor's queue and revise it. However this would require another IETF LC 
and another IESG review, so this decision must not be done lightly.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: Decision: Let's get draft-allbery-usefor-usepro-00 right and publish it as draft-ietf-usefor-usepro-07
Organization: The Eyrie
Date: Wed, 20 Dec 2006 18:28:55 -0800
Sender: [email-masked]

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: Decision: Let's get draft-allbery-usefor-usepro-00 right and publish it as draft-ietf-usefor-usepro-07
Date:  Wed, 20 Dec 2006 21:37:09 +0100
Organization:  <URL:http://purl.net/xyzzy>
Lines: 28
Sender: [email-masked]

Frank

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: Decision: Let's get draft-allbery-usefor-usepro-00 right and publish it as draft-ietf-usefor-usepro-07
Date:  Wed, 20 Dec 2006 21:16:12 +0100
Organization:  <URL:http://purl.net/xyzzy>
Lines: 39
Sender: [email-masked]

 

Frank

Date: Wed, 20 Dec 2006 10:23:47 +0100
From: Harald Alvestrand <[email-masked]>
To: Frank Ellermann <[email-masked]>
Cc: [email-masked]
Subject: Re: Decision: Let's get draft-allbery-usefor-usepro-00 right and publish it as draft-ietf-usefor-usepro-07
Sender: [email-masked]

Harald

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: Decision: Let's get draft-allbery-usefor-usepro-00 right and publish it as draft-ietf-usefor-usepro-07
Organization: The Eyrie
Date: Tue, 19 Dec 2006 16:27:04 -0800
Sender: [email-masked]

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: ASCII
Date:  Wed, 20 Dec 2006 01:17:12 +0100
Organization:  <URL:http://purl.net/xyzzy>
Lines: 24
Sender: [email-masked]

Russ Allbery wrote:

>> IMO it's obvious, but that future draft could say that using
>> any charset that's not UTF-8, like Latin-1, is NOT RECOMMENDED
>> if non-ASCII group names are affected.

> Exactly.

Harald proposed MUSTard, he's probably right, and my RECOMMENDED
is too weak.  Buth that's an issue for a future USEPRO-bis in 2010.

> We could RECOMMEND use of either US-ASCII or UTF-8 for the charset
> in the name of making future transitions easier, though.

> I don't see any other non-ASCII character set cropping up in
> the IETF.

BOCU-1 isn't user friendly, UTF-4 is no alternative, SCSU has no
unique encoding, UTF-7 makes no sense in an 8bits environment,
and the s-o-1036 concept of 2047-encoded words was apparently
stillborn,

Frank

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: Decision: Let's get draft-allbery-usefor-usepro-00 right and publish it as draft-ietf-usefor-usepro-07
Date:  Wed, 20 Dec 2006 00:50:50 +0100
Organization:  <URL:http://purl.net/xyzzy>
Lines: 29
Sender: [email-masked]

JFTR, s/not/only/

Frank

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: Message IDs and moderators
Organization: The Eyrie
Date: Tue, 19 Dec 2006 12:10:49 -0800
Sender: [email-masked]

stanley <[email-masked]> writes:
> Russ Allbery <rra@xxxxxxxxxxxx>:

>> If we go with the definitions in my draft, a proto-article is not an
>> article.  There is an intentionally sharp distinction between a
>> proto-article and an article.  A proto-article only becomes an article
>> once it has been injected.

> I would say that any group of bytes that is being produced with the
> intent of becoming a news article, and which follows the standards for
> formatting and definitions, should be subject to those standards.

I do agree with that.

[ snipping things that I don't disagree with ]

> Were Peter to start posting individual articles under other people's
> names to comp.risks, with the same kinds of editing he does in the
> digest, then it would be a corner case. It would then be open to debate
> whether his editing is acceptable for a moderated group and having the
> original author's name applied to a different message than he sent.

> Perhaps a better "corner case" was comp.dcom.telecom. Enough people
> objected to the editing by the moderator of that group that they started
> another one -- comp.dcom.telecom.tech. So, we have at least one
> precedent where moderator editing was considered unacceptable by enough
> people to form a new group.

Good point.  comp.dcom.telecom is a better example.

In that case, we had some people who were comfortable with the editorial
policies of the moderator and some who weren't, and as a result the group
split.  I think that's a good encapsulation of the situation, actually.
Extensive moderator editing is generally frowned upon and it makes a lot
of people unhappy when done on Usenet, but it's also sometimes done
successfully and sometimes particular groups are comfortable with it.
It's that dynamic that, to me, moves it from standard to best practice.

>> We need to make sure that our standard is not incompatible with what
>> people do in practice, including the corner cases, where those cases
>> are accepted.

> Yes, and that is why I expect the attempt to get the codified right to
> edit articles in any way the moderator wishes removed will be a losing
> battle. It would be impossible to prevent, it isn't a matter for the
> protocol, and many people don't care. I cannot accept it on ethical
> grounds, however, and that's why I have to object to it being made
> explicit.

Okay, that makes sense to me.  I'm also happy to try to reword what the
protocol document says so that it specifies that it doesn't constrain
rather than giving explicit blessing to behavior that is controversial.
That can be tricky to do and still be clear, but the purpose, as far as
I'm concerned, is to draw a clear boundary around the protocol and say
that beyond that boundary the protocol doesn't place requirements, not to
say that the protocol is implicitly blessing some particular course of
action.

My currently language doesn't do this anywhere near as well as it could.

> Even if you say "moderators may edit according to group standards",
> since the moderator is free to set the standards, you're saying the same
> thing.

Agreed.

> As a side issue -- how does one differentiate between a proto-article
> that has all the mandatory headers and a real article? What should be
> the result of differentiating between them? I.e., if you have a
> proto-article that contains all the mandatory headers, how do you treat
> it differently than a real article?

My document defines it as follows:

   A proto-article has the same format as a normal article except that
   the Injection-Date, Injection-Info, and Xref header fields MUST NOT
   be present; the Path header field MUST NOT contain a "POSTED" <diag-
   keyword>; and any of the following mandatory header fields MAY be
   omitted: Message-ID, Date, and Path.

Since if one is following the new protocol an injecting agent MUST add an
Injection-Date header, that's the clearest point of distinction.  If it
has an Injection-Date header, it's an article, not a proto-article.

However, since Injection-Date is a new invention of this document, there
is no clear way of determining that a message is a proto-article when you
take into account existing practice.  There are various markers that tell
you that it's an article (presence of any of those three headers or the
POSTED Path diagnostic), and of course if it's missing one of those
mandatory headers it's a proto-article, but there are messages that will
be ambiguous.

Probably the best way of determining whether something is an article or
proto-article currently is to look at whether it has an Xref header field.
That header field is optional, but nearly all servers add it and posting
agents do not, so it's a clear sign that the article has been injected.

Note that inclusion of Xref in this set is new in my document and is a
change from the existing USEPRO draft (but reflects current practice in,
for example, INN, which will reject attempts to post messages with Xref
headers already present).  This is precisely one of the reasons why I
added it to this definition.

Date: Tue, 19 Dec 2006 11:41:37 -0800 (PST)
From: [email-masked]
To: [email-masked]
Subject: Re: Message IDs and moderators
Sender: [email-masked]

Russ Allbery <rra@xxxxxxxxxxxx>:

>If we go with the definitions in my draft, a proto-article is not an
>article.  There is an intentionally sharp distinction between a
>proto-article and an article.  A proto-article only becomes an article
>once it has been injected.

I would say that any group of bytes that is being produced with the intent 
of becoming a news article, and which follows the standards for formatting 
and definitions, should be subject to those standards.

Under that "philosophy", even a proto-article's header fields have the 
meanings that RFC1036 and eventually our standards define for them. That 
includes From and Message-ID fields.

But where I'm taking this is not specifically involving the idea that the 
message id, once applied, cannot be changed by a moderator. It will 
eventually be a discussion of the "moderators may edit articles in any way 
they see fit" concept -- once we've settled on which draft we use.

>I think RISKS Digest is an excellent example of a corner case.

We agree to disagree. comp.risks is a moderated newsgroup, yes. That's 
where the relevance ends. Each "article" that appears in the newsgroup is 
posted by the editor, not individuals, and are posted under the editor's 
name. That editor uses a system of attributing material to the authors he 
quotes in his digest, but at no time does he claim that the digest itself 
was originated by anyone other than himself.

Were Peter to start posting individual articles under other people's names 
to comp.risks, with the same kinds of editing he does in the digest, then 
it would be a corner case. It would then be open to debate whether his 
editing is acceptable for a moderated group and having the original 
author's name applied to a different message than he sent.

Perhaps a better "corner case" was comp.dcom.telecom. Enough people 
objected to the editing by the moderator of that group that they started 
another one -- comp.dcom.telecom.tech. So, we have at least one precedent 
where moderator editing was considered unacceptable by enough people to 
form a new group.

>We need to
>make sure that our standard is not incompatible with what people do in
>practice, including the corner cases, where those cases are accepted.

Yes, and that is why I expect the attempt to get the codified right to 
edit articles in any way the moderator wishes removed will be a losing 
battle. It would be impossible to prevent, it isn't a matter for the 
protocol, and many people don't care. I cannot accept it on ethical 
grounds, however, and that's why I have to object to it being made
explicit.

Even if you say "moderators may edit according to group standards", since 
the moderator is free to set the standards, you're saying the same thing.

As a side issue -- how does one differentiate between a proto-article that
has all the mandatory headers and a real article? What should be the
result of differentiating between them? I.e., if you have a proto-article
that contains all the mandatory headers, how do you treat it differently
than a real article?

Yes, an injecting agent is supposed to throw away certain things, but
that's a specific case based on an action by a user. In general, what's 
the difference?

To: [email-masked]
Xref: clerew local.usefor:23924
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Message IDs and moderators
Date: Tue, 19 Dec 2006 15:19:24 GMT
Lines: 28
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>But one of the goals of this process of comparing two different starting
>points is to determine which philosophy to use as the foundation of the
>draft, which is why I'm making very clear what changes are incidental and
>what changes express that different philosophy.  Adopting my draft but
>then changing all the philosophical differences back to Charles's draft
>makes no sense; if we're going to do that, we should proceed with
>Charles's draft and use mine only as interesting input.

I think that is a fair summary of the situation, and these issues of
philosophy need to be discussed.

Indeed, I shall have a lot to say on that subject presently, but I chose
to start with a list of pure protocol issues, because I had to start
somewhere, and it makes no sense for the WG to discuss all the topics that
need discussing simultaneously.

To: [email-masked]
Xref: clerew local.usefor:23922
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Decision: Let's get draft-allbery-usefor-usepro-00 right and publish it as draft-ietf-usefor-usepro-07
Date: Tue, 19 Dec 2006 15:01:37 GMT
Lines: 24
Sender: [email-masked]

To: [email-masked]
Xref: clerew local.usefor:23926
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: On the issue of publication interval between USEFOR and USEPRO
Date: Tue, 19 Dec 2006 15:28:45 GMT
Lines: 32
Sender: [email-masked]

In <[email-masked]> Lisa Dusseault <[email-masked]> writes:

>--Apple-Mail-5--813014145
>Content-Transfer-Encoding: 7bit
>Content-Type: text/plain;
>	charset=US-ASCII;
>	delsp=yes;
>	format=flowed

>On Dec 15, 2006, at 8:02 PM, Charles Lindsey wrote:

>>
>> But whatever, please don't start tinkering with trying to make  
>> USEPRO a
>> Normative Reference.

>Why not?

Because I don't think any of the items mentioned so far actually require a
normative reference.

To: [email-masked]
Xref: clerew local.usefor:23921
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: ASCII (Re: draft-allbery-usefor-usepro-00 errata)
Date: Tue, 19 Dec 2006 14:59:34 GMT
Lines: 36
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>I think that also describes the same situation, but I prefer the wording
>above.  I think it's clearer and more explicit.

I think we are agreed about the effect to be achieved. But the wording
needs to be such that readers not familiar with the background to this
issue will easily be able to see what it implies. Moreover, if ever we do
go for UTF-8 newsgroup names, then it would be nice to have a wording that
can be adapted, rather than being a complete rewrite.

But I don't think either wordings suggested really works as it stands.

>You only have to require a charset of UTF-8 if you have non-ASCII
>newsgroups; otherwise, anything can be used that satisfies the above
>requirement.  So I don't agree that you would lose backward compatibility.

Ah! So when that happens, existing ASCII newsgroups could continue to use,
say, Iso8859-1 in checkgroups, but any new Utf-8 groups would be
restricted to using UTF-8. Seems slightly weird, but it would work.

BTW, I agree with the suggestion, in another thread, that RECCOMMENDING
(or recommending) UTF-8 at this stage would be a good idea.

To: [email-masked]
Xref: clerew local.usefor:23925
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Responses from the USEFOR WG Chairs on IESG DISCUSSes
Date: Tue, 19 Dec 2006 15:26:55 GMT
Lines: 36
Sender: [email-masked]

In <[email-masked]> Sam Hartman <[email-masked]> writes:

>I guess what I don't understand here is why there is not a sufficient
>justification for "receivers SHOULD be able to handle messages that
>are valid RFC 2822 messages but that do not meet all the additional
>restrictions of this specification."  I'm particularly thinking of the
>generic message header restrictions not the requirements for
>usenet article without those two headers.  There seems to be an
>interoperability justification for being liberal in this area and I
>have not seen an argument advanced why a SHOULD would be
>inappropriate.  I've seen arguments about why some implementations
>might not choose to follow such a SHOULD; while that makes sense, it
>is not an argument against the SHOULD.

Whenever the WG has discussed this topic, it has expressed a very clear
view against any such SHOULD. Why should newsreaders be even RECOMMENDED
to accept constructs which have never ever appeared in News Articles?

I think the view has, rather, been that the next revision of RFC 2822
should seriously consider moving a few of the key differences from the
'generate' syntax (MAY generate) into the 'obsolete' syntax (MUST NOT
generate but MUST accept). The space after the colon is a case in point.
Neither email messages nor news articles are ever generated with that
space missing, so moving it into the obs syntax would have no practical
effect.

To: [email-masked]
Xref: clerew local.usefor:23923
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Message IDs and moderators
Date: Tue, 19 Dec 2006 15:15:13 GMT
Lines: 40
Sender: [email-masked]

In <[email-masked]> [email-masked] writes:

>Russ Allbery <rra@xxxxxxxxxxxx>:

>>From the perspective of the protocol, nothing in the protocol breaks if
>>the moderator changes the message ID.  I know this from a practical basis:
>>many moderators do this now and Usenet still works.

>Agree. Cancels must be delayed until the article arrives at the poster's
>site, but if the poster is cancelling an unapproved forgery he'd have
>to wait anyway.

But please bear in mind that a person wanting to cancel his own article in
a moderated group has to ask the moderator to do it for him, because the
normal protocol will send his cancel to the moderator in the usual way.

But supposing he wants to issue the cancel moments after he has posted it
(2nd thoughts, glaring typo to be fixed, whatever), then he can only send
it using the original Message-ID. That will surely confuse the moderator.

The whole problem with Usenet, which is quite unlike any other IETF
protocol, is that interoperability means more than getting the articles
correctly from A to B. It is a complete system (an "adhocratic anarchy" as
someone wrote in our earlier drafts) that works, but only if numerous
rules which go beyond the normal ideas of IETF "interoperability" are
observed.

So, for Usenet, saying "nothing in the protocol breaks if X happens" is
not always an overriding argument.

Date: Tue, 19 Dec 2006 11:42:09 +0100
From: Harald Alvestrand <[email-masked]>
To: Charles Lindsey <[email-masked]>
Cc: [email-masked]
Subject: Re: Decision: Let's get draft-allbery-usefor-usepro-00 right and publish it as draft-ietf-usefor-usepro-07
Sender: [email-masked]

                   Harald

> such a consensus might emerge after more discussion, and hopefully some

Date: Tue, 19 Dec 2006 11:36:19 +0100
From: Harald Alvestrand <[email-masked]>
To: Russ Allbery <[email-masked]>
Cc: [email-masked]
Subject: Proto-article (Re: Message IDs and moderators)
Sender: [email-masked]

Russ Allbery wrote:
> stanley <[email-masked]> writes:
>   
>> Russ Allbery <rra@xxxxxxxxxxxx>:
>>     
>
>   
>>> When sending proto-articles around to moderators, no Netnews article
>>> has yet been created.
>>>       
>
>   
>> A proto-article is an article. At the least, it is an RFC2822 message.
>>     
>
> If we go with the definitions in my draft, a proto-article is not an
> article.  There is an intentionally sharp distinction between a
> proto-article and an article.  A proto-article only becomes an article
> once it has been injected.
>   

The definition in our finished "usefor" document is:

1.5 Definitions

An "article" is the unit of Netnews, analogous to an [RFC2822]
"message". A "proto-article" is one that has not yet been injected
into the news system. In contrast to an "article", a "proto-article"
may lack some mandatory header fields.

A "message identifier" (Section 3.1.3) is a unique identifier for an
article, usually supplied by the "user agent" that posted it or,
failing that, by the "news server". It distinguishes the article
from every other article ever posted anywhere. Articles with the
same message identifier are treated as if they are the same article
regardless of any differences in the body or header fields.

Note that it says nothing about whether or not a message-id can identify 
a proto-article.

From: Sam Hartman <[email-masked]>
To: Russ Allbery <[email-masked]>
Cc: Harald Alvestrand <[email-masked]>, [email-masked], [email-masked], Alexey Melnikov <[email-masked]>
Subject: Re: Responses from the USEFOR WG Chairs on IESG DISCUSSes
Date: Mon, 18 Dec 2006 18:05:43 -0500
Sender: [email-masked]

>>>>> "Russ" == Russ Allbery <[email-masked]> writes:

    Russ> Sam Hartman <[email-masked]> writes:
    >> I guess what I don't understand here is why there is not a
    >> sufficient justification for "receivers SHOULD be able to
    >> handle messages that are valid RFC 2822 messages but that do
    >> not meet all the additional restrictions of this
    >> specification."

    Russ> Some valid messages under RFC 2822 cannot be transmitted via
    Russ> NNTP, as they break fundamental parts of the protocol.  For
    Russ> example, RFC 2822 allows escaped spaces in the LHS of a
    Russ> message ID, whereas NNTP commands frequently contain message
    Russ> IDs as arguments and always treat a space as an argument
    Russ> separator with no escaping possible.  Given the ubiquity of
    Russ> NNTP as a Netnews transport, I don't believe it makes sense
    Russ> to permit articles that can never be transmitted over NNTP.

    Russ> Other valid messages under RFC 2822 use corners of RFC 2822
    Russ> that are only seen in test messages (such as omitting the
    Russ> space after the colon of a header field), and while we could
    Russ> make a statement that receivers SHOULD be able to handle
    Russ> such messages, realistically such a statement will only make
    Russ> the standard at odds with existing implementations that will
    Russ> probably never be fixed.  Since such messages do not exist
    Russ> in practice, the changes of anyone making the required
    Russ> changes to existing news software to support them seems very
    Russ> low, regardless of what we say in a standards document.

And that would be an argument against should.
Perhaps you could convince Mark, but I doubt it:-)

Anyway, I'm happy to remove that part of my discuss.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: ASCII
Organization: The Eyrie
Date: Mon, 18 Dec 2006 14:21:35 -0800
Sender: [email-masked]

Frank Ellermann <[email-masked]> writes:

> IMO it's obvious, but that future draft could say that using
> any charset that's not UTF-8, like Latin-1, is NOT RECOMMENDED
> if non-ASCII group names are affected.

Exactly.

Currently non-ASCII group names are banned entirely by USEFOR, so it
doesn't make much sense to say this now.

We could RECOMMEND use of either US-ASCII or UTF-8 for the charset in the
name of making future transitions easier, though.  I can see some merit in
going that route, particularly since I think we're fairly certain at this
point that newsgroup names will either be UTF-8 or some ASCII encoding.  I
don't see any other non-ASCII character set cropping up in the IETF.

From: Russ Allbery <[email-masked]>
To: Sam Hartman <[email-masked]>
Cc: Harald Alvestrand <[email-masked]>, [email-masked], [email-masked], Alexey Melnikov <[email-masked]>
Subject: Re: Responses from the USEFOR WG Chairs on IESG DISCUSSes
Organization: The Eyrie
Date: Mon, 18 Dec 2006 14:09:29 -0800
Sender: [email-masked]

Sam Hartman <[email-masked]> writes:

> I guess what I don't understand here is why there is not a sufficient
> justification for "receivers SHOULD be able to handle messages that are
> valid RFC 2822 messages but that do not meet all the additional
> restrictions of this specification."

Some valid messages under RFC 2822 cannot be transmitted via NNTP, as they
break fundamental parts of the protocol.  For example, RFC 2822 allows
escaped spaces in the LHS of a message ID, whereas NNTP commands
frequently contain message IDs as arguments and always treat a space as an
argument separator with no escaping possible.  Given the ubiquity of NNTP
as a Netnews transport, I don't believe it makes sense to permit articles
that can never be transmitted over NNTP.

Other valid messages under RFC 2822 use corners of RFC 2822 that are only
seen in test messages (such as omitting the space after the colon of a
header field), and while we could make a statement that receivers SHOULD
be able to handle such messages, realistically such a statement will only
make the standard at odds with existing implementations that will probably
never be fixed.  Since such messages do not exist in practice, the changes
of anyone making the required changes to existing news software to support
them seems very low, regardless of what we say in a standards document.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: ASCII (Re: draft-allbery-usefor-usepro-00 errata)
Organization: The Eyrie
Date: Mon, 18 Dec 2006 14:03:06 -0800
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:
> Harald Alvestrand <[email-masked]> writes:

[ Proposed new text ]
>>>>     Implementations predating this standard may not understand MIME
>>>>     headers and expect newsgroup names to be in ASCII.  Therefore,
>>>>     regardless of what charset is used, the result of reading each
>>>>     octet of the body and setting bit 8 to zero MUST be a valid
>>>>     message specifying the same newsgroup name [or names for
>>>>     checkgroups].  There is no requirement that the newsgroup
>>>>     description survive this treatment.

>> The language Russ suggested allows one to evaluate a specific message
>> in a specific charset against the criterion - for instance, an
>> ISO-2022-JP message MAY satisfy the criterion, IF the shift to Japanese
>> characters always occurs in the description, and never in the newsgroup
>> name.

> OK, I see what the problem is now. Essentially, the property I suggested
> needs to apply up to the TAB that introduces the description part. After
> that, we don't care.

Right.

> But anything with a bit8 set in the <newsgroup-name> part will break
> existing software, because existing software will not be setting that
> bit to zero before trying to interpret it.

Correct.  Which is why the above wording declares such messages invalid
(since such a message would specify a different newsgroup name after the
treatment described was applied).

> So what you say must be more like:

>       The charset that is specified MUST have the property that any
>       valid <newsgroup-name> appearing at the start of each line, and
>       before the TAB character, is represented by the same sequence of
>       octets both in that charset and in ASCII (this does not preclude
>       that, beyond the TAB, there may be some shift character that
>       changes the interpretation of subsequent octets).

I think that also describes the same situation, but I prefer the wording
above.  I think it's clearer and more explicit.

> That also is wordy and might be improved, but at least it makes clear
> what the issue is.

The first sentence makes that clear, I think.

>> If we ever want to standardize such a thing, I think the logical thing
>> to do would be to say that the charset MUST be UTF-8.

> Yes, but you then lose backward compatibility if you have earlier
> allowed something else.

You only have to require a charset of UTF-8 if you have non-ASCII
newsgroups; otherwise, anything can be used that satisfies the above
requirement.  So I don't agree that you would lose backward compatibility.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: Message IDs and moderators
Organization: The Eyrie
Date: Mon, 18 Dec 2006 13:58:22 -0800
Sender: [email-masked]

stanley <[email-masked]> writes:
> Russ Allbery <rra@xxxxxxxxxxxx>:

>> When sending proto-articles around to moderators, no Netnews article
>> has yet been created.

> A proto-article is an article. At the least, it is an RFC2822 message.

If we go with the definitions in my draft, a proto-article is not an
article.  There is an intentionally sharp distinction between a
proto-article and an article.  A proto-article only becomes an article
once it has been injected.

The forwarding to a moderator may or may not be an RFC 2822 message.
That's the common case on Usenet right now, but we're talking about (and
standardizing) any Netnews system.  For many stand-alone hierarchies, a
moderation submission will never travel via e-mail at all; it will be
stored in a local database of some sort for approval by a moderator.  And
even if it does travel by e-mail, the RFC 2822 message may simply contain
the submission as a sub-part, meaning that the message ID of the mail
message is irrelevant to the submission itself.

>> Again, consider the RISKS digest, which is a very illustrative corner
>> case of the sort of thing that can happen to a submission to a
>> moderated group.

> RISKS Digest is a particularly poor illustration since the Digest is a
> digest of a MAILING list which is distributed by moderated
> newsgroup. Further, each article in that group is posted by the RISKS
> digest author, and the headers of each article clearly represent that.

I think RISKS Digest is an excellent example of a corner case.  We need to
make sure that our standard is not incompatible with what people do in
practice, including the corner cases, where those cases are accepted.  A
moderated group does not necessarily have a one-to-one correspondance
between messages from a poster and messages injected into the group, and
we can't therefore require that the message ID be retained in all cases.

We *could* say something more complex, namely that the message ID SHOULD
be retained in cases where there's a one-to-one correspondance, but I
still think this is a best-practice topic.  (And I realize that this is a
change from what I'd said way back when we first discussed this topic; it
was a change resulting from reading through the entire draft and writing
my own draft.)

>> I'm being a bit hard-nosed here about saying "if you don't like it,
>> maybe my draft isn't for you" because this is typical of multiple
>> changes I made to remove remaining best-practice requirements from the
>> protocol draft and that was one of the significant philosophical
>> differences I have with the existing USEPRO.

> We already have a problem with an editor that makes unilateral changes
> to the draft(s) and ignores consensus. If using this draft means that
> certain things are not negotiable, then I change my vote in the straw
> poll.

I think this is a misunderstanding about what my draft represents.

Should my draft be adopted as a new starting point by the working group,
any changes subsequent to that should not be made unilaterally and will
not be made unilaterally by me, but only as the result of working group
discussion.

However, in the initial rewrite that I did, I made many unilateral
changes.  That's why the document has my name on it, not "ietf".  It is my
document, not a working group document, and it expresses my opinion.  The
purpose of this initial discussion is to decide whether that document is a
better starting point than the current USEPRO document for further working
group work.  If mine is adopted and issued as the new official USEPRO
document, then from that point forward full working group change control
should be exerted over any change.  But as long as the document is my
personal draft, with my name on it, yes, I will make unilateral changes
because that's the whole point of the exercise.

If the draft becomes the working group document, then I will abide by
working group consensus even if I disagree.  If some change is so
egregiously bad in my opinion that I cannot live with it, I may resign any
formal role in the working group, but I won't say you can't use the text.
But one of the goals of this process of comparing two different starting
points is to determine which philosophy to use as the foundation of the
draft, which is why I'm making very clear what changes are incidental and
what changes express that different philosophy.  Adopting my draft but
then changing all the philosophical differences back to Charles's draft
makes no sense; if we're going to do that, we should proceed with
Charles's draft and use mine only as interesting input.

Date: Mon, 18 Dec 2006 20:33:21 +0000
From: Alexey Melnikov <[email-masked]>
To: Charles Lindsey <[email-masked]>
CC: [email-masked]
Subject: Re: Decision: Let's get draft-allbery-usefor-usepro-00 right and publish it as draft-ietf-usefor-usepro-07
Sender: [email-masked]

>such a consensus might emerge after more discussion, and hopefully some

To: [email-masked]
Xref: clerew local.usefor:23906
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Strawman consensus call, mvgroup control message
Date: Mon, 18 Dec 2006 20:02:02 GMT
Lines: 28
Sender: [email-masked]

>>> <http://news.gmane.org/group/gmane.discuss/thread=10363/focus=10384>

>> it looks like Lars has fixed whatever had caused the problem.

>Yes, that's what he said in article number 10384.

>> Perhaps you should try clicking on it again.

AFAICS, GMANE does possess some pairs of newsgroups which are aliassed on
top of each other. So the interesting question is whether, using NNTP,
both versions of such groups appear to contain the same articles, and
whether you can successfully post to the group under either name.

To: [email-masked]
Xref: clerew local.usefor:23904
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Protocol changes in draft-allbery-usefor-usepro-00
Date: Mon, 18 Dec 2006 19:54:30 GMT
Lines: 26
Sender: [email-masked]

In <[email-masked]> Frank Ellermann <[email-masked]> writes:

>Forrest J. Cavalier III wrote:

>> IMHO, the GNKSA approach is probably better to address your concerns.

>It's limited to newsreaders, otherwise I liked it.  Maybe not exactly
>state of the art, it's some years old, but a good start for USEAGE.
>That's what Charles did for useage-00.  I'm less impressed by texts
>like RFC 1855, and weird body conventions like tear lines.

During the short time that we did discuss USEAGE, we did agree that we
should adopt everything in GNKSA, except where it was obviously outmoded,
or where we expclictly decided to be different (which we did for some
cases and may yet do for more). The current USEAGE draft reflects that.

To: [email-masked]
Xref: clerew local.usefor:23907
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Protocol changes in draft-allbery-usefor-usepro-00
Date: Mon, 18 Dec 2006 20:04:53 GMT
Lines: 728
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>Charles Lindsey <[email-masked]> writes:

BTW, you spent a lot of time justifying cases which I had already marked
with [+1], indicating my agreement (though other might want to disagree,
so needed to be made aware of them).

>> 1. [-1] (2.2) <path-identity> - Possibility to use a FQDN not resolvable
>> in the DNS.

>> (This was a suggestion by Harald, & 1 of 2 alternatives in USEPRO)

>Intentional, but I don't object to changing it.

OK. My vote is still [-1].

>The intentional change here is that I added a different preferred first
>option, namely the FQDN of the news server itself.

Which I did not quite understand. Did you mean the A or AAAA record? In
which case please be explicit. But I would prefer to allow CNAMEs and MXes
as well.

>> 2. [00] (3.2) <path-identity>s are case-sensitive.

>> I was always told that some servers treated them one way and some the
>> other, so you could not rely on either. If anything, I would have
>> expected case-insensitive (that is what domain-names are, and what
>> USEFOR declares <diag-keyword>s are).

>Intentional change.

>Declaring them case-sensitive is the conservative choice.  If you treat
>them as case-sensitive and they're actually compared case-insensitively,
>everything works fine.  The opposite is not true; if your Path identity
>varies in case, servers that compare case-sensitively may add incorrect
>MISMATCH entries or send you articles that you sent them.

OK, I will buy that.

>The one exception is if you use a Path header that varies only in case
>from another site and assume that it will be compared case-sensitively.  I
>thought about adding language to specifically say not to do that, but it's
>hard to tell a site how to guarantee that ...

That's the site's problem :-( .

>Anyone doing this is already violating the SHOULD that says path
>identities should be in all lowercase...

Yes, but please add "(since some current usage treats them
case-insensitively)".

>> 3. [-1] (3.3.1,3.4,3.9.2) From not omittable in proto-article

>> There is existing usage where the injecting agent fills in From header
>> (not possible in NNTP, of course)

>Intentional change.

>What injecting agent supports this?

INN apparently (see below).

Newsreaders such as rn, trn, nn, and others of that generation had (and
still have) the capability to interact directly with the newspool (either on
the same host, or more likely NFS mounted from some server) rather than
going via NNTP (which did not exist when they were first written). They
injected articles by calling a program 'inews' which, in the absence of an
explicit From:, assumed the poster was the user who had called 'inews'.

I have been reading and posting to Usenet for around 20 years now, and I
have never yet configured my newseader with what I wanted in my From: line
(except when doing tests with upstart agents such as netscape :-) ). I
don't suppose I am the only one. I suspect that 'inews' still exists on a
lot of systems.

It certainly comes as part of CNews. 'nn' comes with a version of 'inews'
that invokes NNTP with a POST command, but it still checks to see whether
a From: header has been provided, and goes looking in /etc/passwd if not.
And Stan Barber's model implementation of NNTP, when it comes to
implementing the POST command, looks for a program 'inews' on the host it
is running on (though of course it doesn't expect automatic deduction of
From: in that situation).

And now I find that INN seems to have an 'inews'. I just found that the
University offers an 'inews', and I just posted an article with no From:
header using it, and read it back:

kiss% telnet wapping 119
Trying 130.88.198.1...
Connected to wapping.
Escape character is '^]'.
200 cs.man.ac.uk InterNetNews NNRP server INN 2.3.3 ready (posting ok).
group man.test
211 1 1680 1681 man.test
article 1681
220 1681 <[email-masked]> article
Path: cs.man.ac.uk!chl
From: [email-masked] (Charles Lindsey)
Subject: testing inews
Date: Mon, 18 Dec 2006 14:08:25 +0000 (UTC)
Organization: School of Computer Science, University of Manchester, U.K.
Lines: 2
Message-ID: <[email-masked]>
NNTP-Posting-Host: kiss.cs.man.ac.uk
X-Trace: wapping.cs.man.ac.uk 1166450905 2390 130.88.192.64 (18 Dec 2006
14:08:25 GMT)
X-Complaints-To: [email-masked]
NNTP-Posting-Date: Mon, 18 Dec 2006 14:08:25 +0000 (UTC)
Xref: cs.man.ac.uk man.test:1681

called /opt/cs/bin/inews from kiss.
No From line, but it was from Charles Lindsey <[email-masked]>

You will see that it has constructed a suitable From header (that is my
correct email address when I am inside the Manchester firewall).

The wording I wrote in USEPRO was

   "... (and even From if the particular injecting agent can derive that
information from other sources)".

That indicates pretty clearly that it would be an unusual possibility,
but fine if you happen to have it. It ain't broke ...

>....  This doesn't allow for injecting agents that replace the From
>header with one containing the authenticated identity of the user, and I
>do know of some injecting agents which do this, but I don't think that's
>something that we want to support.

Sure, and the wording in USEPRO gives no support whatsoever for that
practice.

>> 5. [-1] (3.3.4) SHOULD trim References to keep <= 998: reduced from MUST.

>> This could allow header lines longer than 998 octets, which is known to
>> severely impact interoperability. It is not disputed that trimming when
>> within the 998 limit is a MAY.

>Intentional change, but I'm dithering.

One might solve the problem by just saying "fold it so that no header line
is more than 998". However, that is not robust enough.

There was a case on this list a few years back (yes, that was email, but
the problem is the same for both). Some thread got so long as to run into
the 998 limit. This is essentially a problem for user agents, not servers,
and some people's MUAs were clearly folding the huge References headers
in a sensible manner. But it was apparent that other MUAs were first
unfolding them, adding the extra entry, and then hopefully refolding them
again. And some agent(s) out there was evidently trying to do all that in
a buffer of 998, and when it overflowed it just truncated things at that
point (or somesuch - I forget the precise details). So you ended up with
an invalid References header which was then sent forth to the list. And
when that reached my 'nn' (I gateway this list into news), it positively
refused to display the article at all (so I could not even see what was
wrong). I had to go to my newspool and hand-edit the References header
into somethng legal (and chop a lot out of it at the same time), and then
it was fine.

>> 6. [00] (3.4) Injecting agents reporting rejection to user agent; was
>> SHOULD, now merely "encouraged".

>> All it needs is an NNTP 44x response or similar.

>Intentional change.

>What this means is that the Netnews protocol would be placing a
>requirement on the transport layer that it have a means of reporting
>errors back to a client.

I would regard an injecting agent that was so detached from its clients as
to be unable to report back to its clients as irretrievably broken, hence
making the system hopelessly inadequate for its users (see my remarks in
response to your "philosophy" claims). Surely that is just the sort of
situation that "SHOULD" is designed for (I agree that if you are
communicating with your injecting agent via carrier pigeon, there is no
way to get the bird to fly back, so "MUST" would be too strong).

Count me as [-1] on this.

>> 8. [-1] (3.4) Requirement for injecting agent to forward articles to
>> moderator groups reduced from MUST to SHOULD.

>> If some small latitude is needed for when the moderated group is in a
>> crosspost to some remote and unknown hierarchy, then this should be
>> stated explicitly.

>Intentional change.

>This section says more than just that, and I think the change makes more
>sense with more context:

>   7.   If the Newsgroups header contains one or more moderated groups
>        and the proto-article does not contain an Approved header field,
>        the injecting agent SHOULD forward it to a moderator as
>        specified in Section 3.4.1.  If the article cannot be forwarded
>        to a moderator for some reason, it MUST be rejected.

OK, the wording could be better. How about:

   7.  If the Newsgroups header contains one or more moderated groups and
	the proto-article does not contain an Approved header field, the
	injecting agent MUST either forward it to a moderator as specified
	in Section 3.4.1 or, if that is not possible, reject it.

>> 9. [-1] (3.4,3.5) Prepending of <path-identity> to Path by injecting
>> agent reduced from MUST to SHOULD.

>> I would accept that prepending the <path-diagnostic> POSTED is a SHOULD.

>And that's exactly what my language says.  Here's the text:

>   9.   The injecting agent SHOULD then prepend the <path-identity> of
>        the injecting agent followed by "!.POSTED", optionally "." and
>        the FQDN or IP address of the source, and a further "!" to the
>        content of the Path header field.  If the injecting agent does
>        not support the use of <diag-keyword>, it MUST instead prepend
>        its <path-identity> followed by "!"; one or the other of these
>        mechanisms MUST be used.

OK, so it's a matter of wording. I think something based on my USEPRO
would be better, such as

   9.  The injecting agent MUST then prepend to the content of the Path
	header field the its own <path-identity> followed by a "!", which
	SHOULD then be followed by a ".", the <diag-keyword> "POSTED" and
	a further "!".

I just noticed there that you also had an optional FQDN or IP address
after the "POSTED". I don't think that was ever the intention of what we
discussed during USEFOR, but as a matter of wording it could easily be
added to the revised text.

>The language is phrased similarly for relaing and serving agents.

Yes, similar changes are needed there, and I would suggest looking again
at my USEPRO version. The essential difference in those cases is that I
described the stuff to be added from left-to-right (which is how I think
people will tend to read it), so that you do the obligatory
<path-identity> first, and then the various possibilities for the
diagnostics.

>Changing the <path-diagnostic> from a MUST to a SHOULD is an intentional
>change.

Yes, I am happy with that (the latest USEPRO left it open for discussion).

>> 10. [00] (3.4) Folding of Path header when length becomes excessive
>> reduced from SHOULD to MAY.

>Intentional change.

I am neutral on this one. Let others decide.

>> 11. [-1] (3.4) Recommendation that each injecting agent SHOULD use a
>> consistent format for Injection-Info removed.

>> Otherwise, the task of chasing the bad guys becomes harder.

>Intentional change.

>I don't see much purpose served by this requirement and the meaning is
>very murky to me.

USEFOR allows a lot of ways for filling in the Injection-Info. I just
don't want to see a single injecting agent, on successive articles,
hopping around at random between various forms of posting-host,
posting-account and logging-data just in order to force its way past my
killfile.

>> 12. [-1] (3.5) The significance of "world" and "local" in the
>> Distribution header are no longer mentioned.

>> We recently took care to reserve these in USEFOR, but that requires
>> backup here to ensure the protocol fulfils that promise.

>Intentional change.

>Most everything useful to be said about "world" and "local" was already
>said in USEFOR, and "world" in particular SHOULD NOT be used anyway (as
>already stated in USEFOR).  Distribution is generally a waste of effort,
>and I'd rather keep the rules related to it to a minimum since in practice
>few sites are ever going to care.

I was more concerned about 'local' than I was about 'world' (which is
unnecessary, though it was seen at one time). If you want to rely on the
wording in USEFOR, then please add something like:

    "See [USEPRO] section 3.2.4 for the significance of the reserved
<dist-name>s "world" and "local".

>> 15. [-1] (3.5) Relaying agent MUST reject any article already already
>> sent to it.

>Intentional change.

>Every relayer I've ever heard of keeps a history file and has to to be
>usable in the real world.  The protocol breaks down badly if you have
>multiple peers and don't do this.  This is a core part of the flood-fill
>protocol.

OK, you have persuaded me. And it is also needed too to check for stale
Injection-Dates, which I _did_ have in USEPRO.

>> 16. [-1] (3.5) Relaying agent SHOULD deal with cancel message (if it
>> chooses to honour it)

>> Yes, USEPRO says the same, but I suspect both are wrong, because it is
>> serving/storage agents that should be doing this (all a pure relaying
>> agent can do is then not to propagate it further).

>Not actually a change, but I think this should stay.  Here's what it says:

>   5.   It SHOULD reject any article that matches an already-received
>        cancel control message or the contents of the Supersedes header
>        field of an accepted article, provided that the relaying agent
>        chose (on the basis of local site policy) to honor that cancel
>        control message or Supersedes header field.

Agreed, but my concern was that this was a matter for serving agents
rather than for relaying agents. If you aren't actially storing the
article, then there is little point in trying to implement cancels. Just
relay it on, and let the agents that _do_ store it decide whether to
honour the cancel or not. Do current relay-only agents currently bother to
do this?

>> 20. [00] (3.5) Relaying agent MAY add a new Xref header.

>> Why might it want to do that?

>Intentional change.

>Because the same piece of software also implements a serving agent, which
>has to add the Xref header, and the same code path is used for all
>articles received by the server whether they are for further relaying or
>local serving.

Well if that is the case it was nominally the serving function that added
the Xref (section 3.6). Now whether such an agent first stores the article
and then relays what it has stored, or whether it relays first and then
stores after is none of our business. So if you were just trying to cover
the first of those cases, where the new Xref might then escape by relaying,
then I think it could be worded better. How about:

   Any Xref header, whether present on input or added by an associated
   local serving agent, MAY be deleted before relaying.

>> 21. [00] (3.6) Storage of control messages in the (pseudo) hierarchy
>> control(.*) is a SHOULD.

>> Is this any of our business?

>Intentional change.  See previous discussion.

I remain neutral [00].

Alert! Duplicated issue number #21!

>> 21. [-1] (3.6) No mention of processing cancel messages by
>> serving/storage agents.

>The same language is here as for relaying agents, but you may have missed
>it because it's combined with another similar point:

>   3.  It SHOULD reject any article that has already been successfully
>       sent to it or that matches an already-received and honored cancel
>       message or Supersedes header field following the same rules as a
>       relaying agent (Section 3.5).

OK, point taken, but it might be better split out. Moreover, the first bit
corresponds to step 2 of relaying which is a MUST, as is the matter of
stale articles (which is also a history file matter).

>Yes, there is no specific mention in the duties section of acting on
>control messages for already received messages.  This is for the same
>reason that there's no specific mention in the duties section of acting on
>newgroup, rmgroup, or checkgroups control messages.

OK, but people will think it odd that you mention the obscure case of
cancels which arise first here, but make to mention of the common case.
Perhaps an extra step to say that it SHOULD then process any control
messages (including cancels) in acccordance with section 5.

>> 22. [00] (3.7) Removed recommendation that reading agent SHOULD have the
>> capability to display the raw article 'as received'.

>Intentional change.

Frank and Forrest have commented on this. Seems controversial. I remain
neutral for now.

>> 23. [-1] (3.8) Moderators merely "encouraged" (was SHOULD) to retain
>> existing <msg-id>.

>Intentional change.

Seems controversial.

>> 24. [00] (3.2.1) Removed provision (MAY) for outgoing gateway to change
>> Content-Transfer-Encoding.

>I don't consider this to be a change.  There's nothing that says that it
>*can't* change the Content-Transfer-Encoding,...

Sure, but it may be helpful to draw attention to it. I remain [00].

>> 26. [-1] (4.2,4.3) Removed requirement that
>> application/news-[groupinfo,checkgroups] MUST NOT be used except with
>> relevant control messages.

>> Application types are inherently dangerous ...

>Intentional change.

>I can't think of an example of how application/news-groupinfo could be
>dangerous.  It's nothing but a bit of text giving a newsgroup name and a
>description.  In and of itself, it has no force, power, or action
>whatsoever.

Application types are intended to cause applications to be invoked, with
possible side effects. In this case, the side efect is that a line gets
added/changed in the Newsgroups file. Fine when it is part of a newgroup
message that you intend to honour, but not anywhere else without explicit
human intervention. Likewise for application/checkgroups.

>Prohibiting use of these media types in all other situations is overly
>broad and seems unnecessary to me.  If I mail a checkgroups for a
>hierarchy to another newsadmin, I don't see any reason why I should be
>prohibited from using application/news-checkgroups for that mail message.

OK, in that case what you need to say is:

    This media type is intended to be used in conjunction with the
    newgroup control message as described in section 5.2.1. It MUST NOT be
    automatically invoked in any other context without explicit human
    intervention.

and similarly for checkgroups.

Alert! Duplicated issue number #27!

>> 27. [-1] (5) Nothing stated regarding Newsgroups header of Control
>> messages.

>> Was SHOULD include the groups affected (& possibly others). Without
>> that, control messages won't propagate to the places they are likely to
>> be needed.

>Intentional change, but after looking at the INN code, I was wrong.

OK.

>> 28. [00] (5) Subjects starting with 'cmsg' not accompanied by a Control
>> header MAY be rejected.

>> We discussed this at some stage, but I don't think we ever agreed it one
>> way or the other.

>Intentional change.

Others have commented, so controversial. I remain [00] for now.

>> 29. [00] (5.2) Newsgroup-names MUST be checked against disallowed names
>> before group control message is honoured.

>Intentional change.

>It's just as important for checkgroups as it is for newgroup.

Point taken.

>> 30. [-1] (5.2) Nothing said about content of Approved header.

>> Surely, it SHOULD identify the person/identity/role of the issuer, ...

>Intentional change.

>The content of the Approved header serves no protocol purpose, and USEFOR
>already adequately covers the definition of its content.  Control message
>authorization is done on the basis of the Sender or From header
>(preferrably in combination with a digital signature).

It asserts that this is an "authorized" message (just what "authorized"
means is a separate issue which I shall raise later). As such, it is
pretty pointless unless it identifies the entity that is authorized, and
all news servers that I know of can be configured with the identity that
is to be recognized for each hierarchy. Of course, it really needs to be
digitally signed to make it effective, but that again is a separate issue.

I think the WG early on took the view that the Approved header would
simply fall into disrepute if one could always get away with
   Approved: kibo
and that it therefore needed some firmer language (digitally signed in due
course). I remain of that view.

>> 31. [-1] (5.2.1) Newgroup message SHOULD include application/group-info.

>> Was MUST ...

>Intentional change.

>Newsgroup descriptions are optional in the protocol and newgroup messages
>without descriptions work fine with existing software.

OK, if absence means that it just creates a Newsgroups file entry with
just the name of the newsgroup, and maybe the TAB, and certainly the
" (Moderated)" if needed. Likewise for checkgroups. So you need something
like:

    The application/group-info MUST be included if the group is moderated,
    in which case it MUST include a <moderation-flag>. For other groups, it
    MAY be omitted if there is no <newsgroup-description> to be provided.

>> 32. [-1] (5.2.1) Newgroup message containing other attachments in
>> addition to news-groupinfo to be multipart/related (was
>> multipart/mixed).

>Unintentional change.

OK.

>> 34. [00] (5.2.3) Checkgroups SHOULD contain sub-hierarchies excluded by
>> chkscope, for backwards compatibility.

>> Fine in principle, but maybe not implementable. If de.alt.* were as
>> chaotic as alt.* (is it?) there would be no way the admins of de.* could
>> include a defnitive list of de.alt.*. So some qualification is needed.

>Intentional change.  See previous discussion.

Then it needs to say:
    and SHOULD contain ... unless no definitive list for such
    sub-hierarchies is available.
which, for a sufficuently disorganized alt-like hierarchy might well be
the case.

>> 36. [00] (5.2.3) Body of checkgroups SHOULD be application/checkgroups,
>> and otherwise SHOULD treat as if it were.

>> Why does that SHOULD differ from the corresponding MUST in newgroups? I
>> would have expected "SHOULD ... and otherwise MUST ...".

>Intentional change.

>What MUST in newgroup?  Scanning the body in newgroup messages without a
>MIME part is a MAY.  It's raised to SHOULD in checkgroups because unlike
>for newgroups, this is actually necessary to honor the checkgroups.

OK, I seem to have misread a MUST somewhere. Objection withdrawn.

>> 37. [00] (5.3) Body of cancel message MAY contain ...explanatory text.

>> Was SHOULD.

>Intentional change.

Frank seems to disagree. So do I, so controversial I think.

>> 38. [-1] (5.3) No mention of who should be in From/Sender of cancel
>> message.

>> Agreed it is not the person who posted the original article. USEPRO says
>> it "should" be the person who issued the cancel. That should really be
>> "SHOULD".

>Intentional change.

>The From/Sender of a cancel message is no different in definition than the
>From/Sender of any other Netnews message, and as such I think USEFOR
>covers the situation sufficiently.  I don't see a reason to call special
>attention to the fields for cancel messages when cancel messages are no
>different than normal articles in this regard.

I think the point is that we have made a change from RFC 1036 here. If you
reworded it as:

    Contrary to [RFC1036], the From and Sender fields SHOULD contain the
    address of the issuer of the cancel, as opposed to that of the poster
    of the cancelled article. That former requirement only ...

then I would be happy. s/SHOULD/should/ if you like.

>> 39. [-1] (5.5) The <relayer-name> in ihave and sendme control messages
>> is now optional.

>> It is required to be present in USEPRO and in Son-of-1036.

>Unintentional change.

>USEPRO offers two different syntaxes, one of which is marked SHOULD NOT,
>and I found that confusing.  I tried to combine them and missed this
>distinction.  (So, actually, it's not required to be present in USEPRO;
>it's only a SHOULD NOT to omit it.)

I think the syntax you need, if you don't want to do it my way, is

    Ihave-command    = "ihave" Ihave-argument
    Ihave-argument   = 1*WSP *( msg-id 1*WSP ) relayer-name

and then say somewhere that , if there is a list of <msg-id>s in the body,
there MUST NOT be any <msg-id>s in the Ihave-argument (ditto for sendme).

The reason I did it the way I did was because it seemed, even at the time
Henry was writing, the practice of putting them in the Ihave-argument was
already long dead, and there was little reason even to continue supporting
it.

>> 40. [-1] (5.5) Format of batched news in response to sendme removed.

>> Note that this was a format 'on the wire' and was intended to be sent on
>> the same 'wire' as the ihave and sendme messages ...

>Intentional change.

>Specifying the behavior of the transport layer is out of the scope of this
>document.  Discussion of the UUCP batch format no more belongs here than
>does a discussion of dot-escaping in NNTP.

If we are allowed to specify the format of news articles 'on the wire'
(which is what USEFOR is all about), then why on earth cannot we specify
the format of the only other thing that is ever put on the same wire? It
is not the method of transport we are specifying, but the format of the
data - otherwise there will be no interoperability.

>> 41. [-1] (5.5) The protocol for using the 'to.' newsgroup-name is
>> omitted.

>> Insofar as this protocol may still occasionally be used, it needs to be
>> documented (as with the batch format).

>I have:

>   These control messages are normally sent as point-to-point articles
>   between two sites and not then sent on to other sites.  Newsgroups
>   beginning with "to." are reserved for such point-to-point
>   communications.

>Son-of has:

>          It is conventional to reserve newsgroup names beginning with
>          "to." for test messages sent  on  an  essentially  point-to-
>          point basis (see also the ihave/sendme protocol described in
>          section 7.2); newsgroup names beginning  with  "to."  SHOULD
>          not be used for any other purpose.  The second (and possibly
>          later) components of such a name should, together,  comprise
>          the  relayer name (see section 5.6) of a relayer.  The news-
>          group exists only at the named relayer  and  its  neighbors.
>          The  neighbors all pass that newsgroup to the named relayer,
>          while the named relayer does not pass it to anyone.

And USEPRO has much the same.

I think the minimum that should be said is that, syntactically, this funny
newsgroup-name consists of "to." followed by a <relayer-name> (we have
already said what <relayer-name>s are), and that control messages using
this notation MUST NOT be propagated to any site other than the sending
site and the <relayer-name> specified. So we leave it to the implementor
to devise how he causes it to propagate to the one site it is allowed to
propagate to (but we have already offered a strong hint that this is all
designed for use with UUCP).

>> 42. [-1] (5.6) Obsolete control messages SHOULD NOT be sent/honoured/

>> Was MUST NOT. Cf the MUST NOT for obsolete headers in USEFOR.

>Intentional change.

>MUST NOT is unwarranted.  Sending them doesn't break anything.

Except cause mailbombs, which seems like severe breakage to me.

We have forbidden the obsolete headers in USEOR, so why not here?

To: [email-masked]
Xref: clerew local.usefor:23903
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: ASCII (Re: draft-allbery-usefor-usepro-00 errata)
Date: Mon, 18 Dec 2006 19:50:26 GMT
Lines: 64
Sender: [email-masked]

In <FACC7B0D595A32D4D72AEDF5@[192.168.1.103]> Harald Alvestrand <[email-masked]> writes:

>--On 15. desember 2006 20:30 +0000 Charles Lindsey <[email-masked]> 
>wrote:

>>
>>>     Implementations predating this standard may not understand MIME
>>>     headers and expect newsgroup names to be in ASCII.  Therefore,
>>>     regardless of what charset is used, the result of reading each octet
>>>     of the body and setting bit 8 to zero MUST be a valid message
>>>     specifying the same newsgroup name [or names for checkgroups].  There
>>>     is no requirement that the newsgroup description survive this
>>>     treatment.

>The language Russ suggested allows one to evaluate a specific message in a 
>specific charset against the criterion - for instance, an ISO-2022-JP 
>message MAY satisfy the criterion, IF the shift to Japanese characters 
>always occurs in the description, and never in the newsgroup name.

OK, I see what the problem is now. Essentially, the property I suggested
needs to apply up to the TAB that introduces the description part. After
that, we don't care.

But anything with a bit8 set in the <newsgroup-name> part will break
existing software, because existing software will not be setting that bit
to zero before trying to interpret it.

So what you say must be more like:

      The charset that is specified MUST have the property that any valid
      <newsgroup-name> appearing at the start of each line, and before the
      TAB character, is represented by the same sequence of octets both in
      that charset and in ASCII (this does not preclude that, beyond the
      TAB, there may be some shift character that changes the
      interpretation of subsequent octets).

That also is wordy and might be improved, but at least it makes clear what
the issue is.

>If we ever want to standardize such a thing, I think the logical thing to 
>do would be to say that the charset MUST be UTF-8.

Yes, but you then lose backward compatibility if you have earlier allowed
something else. The wording I suggested above, with s/ASCII/UTF8/ would
still work, except I doubt there exists any charset with the necessary
property.

>But that is not part of any proposal under discussion, so it's pure 
>speculation at this stage.

True, but we should not paint ourselves into a corner, since I regard that
development as quite likely, and not so far away.

To: [email-masked]
Xref: clerew local.usefor:23905
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Decision: Let's get draft-allbery-usefor-usepro-00 right and publish it as draft-ietf-usefor-usepro-07
Date: Mon, 18 Dec 2006 19:58:30 GMT
Lines: 33
Sender: [email-masked]

Cc: [email-masked]
From: Lisa Dusseault <[email-masked]>
Subject: Re: On the issue of publication interval between USEFOR and USEPRO
Date: Mon, 18 Dec 2006 12:12:26 -0800
To: Charles Lindsey <[email-masked]>
Sender: [email-masked]

--Apple-Mail-5--813014145
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
	charset=US-ASCII;
	delsp=yes;
	format=flowed

On Dec 15, 2006, at 8:02 PM, Charles Lindsey wrote:

>
> But whatever, please don't start tinkering with trying to make  
> USEPRO a
> Normative Reference.

Why not?

It's pretty easy to argue that you need parts of USEPRO in order to  
be able to generate certain headers defined in USEFOR, without which  
(if I understand correctly) you don't have a valid article.  The  
Security Considerations issue is an independent reason to have a  
normative reference.

thx,
Lisa
--Apple-Mail-5--813014145
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=ISO-8859-1

<HTML><BODY style=3D"word-wrap: break-word; -khtml-nbsp-mode: space; =
-khtml-line-break: after-white-space; "><BR><DIV><DIV>On Dec 15, 2006, =
at 8:02 PM, Charles Lindsey wrote:</DIV><BR =
class=3D"Apple-interchange-newline"><BLOCKQUOTE type=3D"cite"><P =
style=3D"margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica; =
min-height: 14.0px"><BR></P> <P style=3D"margin: 0.0px 0.0px 0.0px =
0.0px"><FONT face=3D"Helvetica" size=3D"3" style=3D"font: 12.0px =
Helvetica">But whatever, please don't start tinkering with trying to =
make USEPRO a</FONT></P> <P style=3D"margin: 0.0px 0.0px 0.0px =
0.0px"><FONT face=3D"Helvetica" size=3D"3" style=3D"font: 12.0px =
Helvetica">Normative Reference.</FONT></P> =
</BLOCKQUOTE></DIV><BR><DIV>Why not?=A0=A0</DIV><DIV><BR =
class=3D"khtml-block-placeholder"></DIV><DIV>It's pretty easy to argue =
that you need parts of USEPRO in order to be able to generate certain =
headers defined in USEFOR, without which (if I understand correctly) you =
don't have a valid article.=A0=A0The Security Considerations issue is an =
independent reason to have a normative reference.</DIV><DIV><BR =
class=3D"khtml-block-placeholder"></DIV><DIV>thx,</DIV><DIV>Lisa</DIV></BO=
DY></HTML>=

--Apple-Mail-5--813014145--

From: Sam Hartman <[email-masked]>
To: Harald Alvestrand <[email-masked]>
Cc: [email-masked], [email-masked], Alexey Melnikov <[email-masked]>
Subject: Re: Responses from the USEFOR WG Chairs on IESG DISCUSSes
Date: Mon, 18 Dec 2006 15:11:44 -0500
Sender: [email-masked]

>>>>> "Harald" == Harald Alvestrand <[email-masked]> writes:

    Harald> Greetings, we (the chairs) hvave discussed the IESG
    Harald> feedback on draft-ietf-usefor-usefor.

    Harald> There are two DISCUSS comments that we'd like to discuss
    Harald> further on this document.

    Harald> Sam Hartman filed this:

    >> First, the reference to draft-ietf-usefor-usepro needs to be
    >> normative.  I don't think you can construct an article without
    >> following advice related to path, injection-*, and control from
    >> that document.  Parsing these fields without usepro is
    >> similarly difficult.

    Harald> The WG had a debate specifically about whether there
    Harald> should be a normative reference to USEPRO, and decided
    Harald> that there should not be one.  The WG's belief is that the
    Harald> -usefor document contains enough information to write
    Harald> software that parses an Usenet article and says whether or
    Harald> not it conforms to the format.

    Harald> We believe that the construction of the content of those
    Harald> fields is not a normative reference for the format - any
    Harald> more than the RIR allocation rules for IP addresses forms
    Harald> a normative reference for the definition of the IP header.
I tend to agree with Russ and Forrest here.  I don't think you ever
want to have people implement usefor without usepro and by not making
the reference normative you are sending the mesage that's OK.  I do
not think I could construct an article given only usefuor; parsing is
not enough for interoperability.

    Harald> If there are specific instances of parsing an Usenet
    Harald> message (as opposed to acting on the content of a message)
    Harald> that you think is difficult without USEPRO, we'd like to
    Harald> have the details, so that we can discuss the specifics.

    >> We received last call comments about the subsetting of RFC 2822
    >> messages.  Very late, we realized that the real issue is not
    >> that these requirements are difficult for new posting agents,
    >> but that they are difficult when a email message is injected
    >> into usenet.  First, I think more discussion of whether that is
    >> the right approach for gateways is needed.

    Harald> It is definitely not the right approach.  A normal email
    Harald> message will not be a valid Usenet message - if only
    Harald> because it lacks a Path: header and a Newsgroups: header.

    Harald> The WG has discussed the problem of email-to-news
    Harald> gateways, and made an explicit decision in at least one
    Harald> other case to exclude wording that would seem to be
    Harald> normative for such gateways. We believe that this problem
    Harald> is out of scope for this document, and is (as we read the
    Harald> charter) out of scope for the present charter of the
    Harald> USEFOR WG.

    Harald> However, we have received a proposal for clarifying text
    Harald> about the relationship between the USEFOR format and the
    Harald> mail format; we'll consider this for inclusion at the end
    Harald> of section 2.1 of the draft:

    >> This draft places additional restrictions on RFC2822 message
    >> format rules, in order to be compatible with deployed NetNews
    >> agents.  An entity that generates messages compliant with
    >> RFC2822 might not always generate messages compliant with this
    >> specification, although certainly a single code-base can
    >> generate messages according to both sets of rules if desired.

    >> Gatewaying between email and news is not part of this
    >> specification, but we note that such a gateway would in all
    >> cases have to do more than just copying the message. Ensuring
    >> that the message satisfies the constraints here would be part
    >> of the job of such a gateway.

    Harald> Would this satisfy the DISCUSSant on this point?

Yes.

    Harald> Back to the DISCUSS text:

    >> Second, regardless of what we say here, it will be reasonably
    >> common for messages to be injected that do not meet these
    >> requirements.  In the interests of interoperability we need to
    >> discuss what problems can result and make recommendations for
    >> liberal receivers that minimize interoperability problems.

    Harald> The WG discussed the problem of illegal messages and
    Harald> decided that it was not possible or reasonable to specify
    Harald> handling of messages that do not conform to this syntax in
    Harald> the syntax document. This would also violate long-standing
    Harald> practice in, for instance, RFC 2822.  There is a document
    Harald> on the WG's document list - USEAGE - where such
    Harald> considerations would be appropriate. (However, note that
    Harald> the state of the WG is not such that this document will be
    Harald> finished soon, if at all. See later note.)

I guess what I don't understand here is why there is not a sufficient
justification for "receivers SHOULD be able to handle messages that
are valid RFC 2822 messages but that do not meet all the additional
restrictions of this specification."  I'm particularly thinking of the
generic message header restrictions not the requirements for
Newsgroups: and Path:.  I understand you cannot do anything with a
usenet article without those two headers.  There seems to be an
interoperability justification for being liberal in this area and I
have not seen an argument advanced why a SHOULD would be
inappropriate.  I've seen arguments about why some implementations
might not choose to follow such a SHOULD; while that makes sense, it
is not an argument against the SHOULD.

Date: Mon, 18 Dec 2006 12:03:13 -0800 (PST)
From: [email-masked]
To: [email-masked]
Subject: rE: Message IDs and moderators 
Sender: [email-masked]

Russ Allbery <rra@xxxxxxxxxxxx>:

>When sending proto-articles around
>to moderators, no Netnews article has yet been created.

A proto-article is an article. At the least, it is an RFC2822 message.

>An unknown amount
>of interaction between the moderator and the poster will occur before the
>message is actually sent,

Usually "none" covers the amount before the message is injected, but it is
sent the second the author sends it.

>Again, consider the RISKS digest, which is a very
>illustrative corner case of the sort of thing that can happen to a
>submission to a moderated group.

RISKS Digest is a particularly poor illustration since the Digest is a 
digest of a MAILING list which is distributed by moderated newsgroup. 
Further, each article in that group is posted by the RISKS digest author, 
and the headers of each article clearly represent that.

>From the perspective of the protocol, nothing in the protocol breaks if
>the moderator changes the message ID.  I know this from a practical basis:
>many moderators do this now and Usenet still works.

Agree. Cancels must be delayed until the article arrives at the poster's
site, but if the poster is cancelling an unapproved forgery he'd have
to wait anyway.

>I'm being a bit hard-nosed here about saying "if you don't like it, maybe
>my draft isn't for you" because this is typical of multiple changes I made
>to remove remaining best-practice requirements from the protocol draft and
>that was one of the significant philosophical differences I have with the
>existing USEPRO.

We already have a problem with an editor that makes unilateral changes to 
the draft(s) and ignores consensus. If using this draft means that certain
things are not negotiable, then I change my vote in the straw poll.

Date: Sun, 17 Dec 2006 19:13:34 +0100
From: Harald Alvestrand <[email-masked]>
To: Charles Lindsey <[email-masked]>, [email-masked]
Subject: ASCII (Re: draft-allbery-usefor-usepro-00 errata)
Sender: [email-masked]

--On 15. desember 2006 20:30 +0000 Charles Lindsey <[email-masked]> 
wrote:

>
>>     Implementations predating this standard may not understand MIME
>>     headers and expect newsgroup names to be in ASCII.  Therefore,
>>     regardless of what charset is used, the result of reading each octet
>>     of the body and setting bit 8 to zero MUST be a valid message
>>     specifying the same newsgroup name [or names for checkgroups].  There
>>     is no requirement that the newsgroup description survive this
>>     treatment.
>
> That seems terribly wordy. Surely all you need is that octets 0-127 (or at
> least all those that can occur in newsgroup-names plus TAB) MUST represent
> the corresponding ASCII characters.

MUST or MUST ALWAYS? The set of useful character sets is quite different in 
the two cases.
The language Russ suggested allows one to evaluate a specific message in a 
specific charset against the criterion - for instance, an ISO-2022-JP 
message MAY satisfy the criterion, IF the shift to Japanese characters 
always occurs in the description, and never in the newsgroup name.

>
> OTOH, one slight worry. Suppose we later go to newsgroup-names in UTF8
> (following the path now beng blazed by the EAI WG). Would we than have to
> say that the charset MUST have UTF-8 as a subset? Or merely that the
> charset used must be able to represent all the newsgroup-names in that
> checkgroups? One of the beauties of newsgroup-names in UTF8 is that (as we
> have already established by trying it with the group dk.test.utf8-æøå)
> that existing news servers can cope immediately (user agents would have to
> change, of course).

If we ever want to standardize such a thing, I think the logical thing to 
do would be to say that the charset MUST be UTF-8.

But that is not part of any proposal under discussion, so it's pure 
speculation at this stage.

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: Strawman consensus call, mvgroup control message
Date:  Sat, 16 Dec 2006 15:25:18 +0100
Organization:  <URL:http://purl.net/xyzzy>
Lines: 16
Sender: [email-masked]

Charles Lindsey wrote:

>> <http://news.gmane.org/group/gmane.discuss/thread=10363/focus=10384>

> it looks like Lars has fixed whatever had caused the problem.

Yes, that's what he said in article number 10384.

> Perhaps you should try clicking on it again.

With Ollie's help I found a workaround to get the a link to the wanted
thread before he fixed it.  I wanted the "clickable" link for somebody 
else, I rarely use GMaNe's Web interface.

Frank

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  ASCII (was: draft-allbery-usefor-usepro-00 errata)
Date:  Sat, 16 Dec 2006 15:13:06 +0100
Organization:  <URL:http://purl.net/xyzzy>
Lines: 53
Sender: [email-masked]

Charles Lindsey wrote:

> That seems terribly wordy. Surely all you need is that octets
> 0-127 (or at least all those that can occur in newsgroup-names
> plus TAB) MUST represent the corresponding ASCII characters.

IIRC we've ruled that out as not good enough, an example is
http://source.icu-project.org/repos/icu/data/trunk/charset/data/xml/glibc-SJIS-2.1.2.xml

[...]
 <validity>
  <state type="FIRST" next="VALID" s="0" e="7F" max="FFFF"/>
  <state type="FIRST" next="SECOND" s="81" e="9F" max="FFFF"/>
  <state type="FIRST" next="VALID" s="A0" e="DF" max="FFFF"/>
  <state type="FIRST" next="SECOND" s="E0" e="FC" max="FFFF"/>
  <state type="SECOND" next="VALID" s="40" e="7E" max="FFFF"/>
  <state type="SECOND" next="VALID" s="80" e="FC" max="FFFF"/>
 </validity>
[...]

The s="40" e="7E" in the state type="SECOND" means that the
octets 0x40..0x7E can be the second byte (after a FIRST byte
0x81..0x9F or 0xE0..0xFC) of double-byte characters.

Apparently Ned's example "euc-jp" doesn't have this problem.

Russ' new wording also works for charsets like UTF-7, UTF-1,
or the "SJIS" variant shown above - the online version of ICU
makes it almost impossible to determine what it really is.

---
> OTOH, one slight worry. Suppose we later go to newsgroup-names
> in UTF8 (following the path now beng blazed by the EAI WG).

And 3977.

> Would we than have to say that the charset MUST have UTF-8
> as a subset?

Russ' text doesn't use the term "subset".  There are only two
charsets with UTF-8 as "subset" I'm aware of, the old UTF-8 and
CESU.  Software expecting UTF-8 would panic as soon as it sees
0xFE, 0xFF, or similar crap.

Without explicit charset= the ASCII rule works for ASCII names,
for future UTF-8 names an explicit charset= would be required.

IMO it's obvious, but that future draft could say that using
any charset that's not UTF-8, like Latin-1, is NOT RECOMMENDED
if non-ASCII group names are affected.

Frank

To: [email-masked]
Xref: clerew local.usefor:23895
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: On the issue of publication interval between USEFOR and USEPRO
Date: Fri, 15 Dec 2006 20:02:59 GMT
Lines: 33
Sender: [email-masked]

In <[email-masked]> Harald Alvestrand <[email-masked]> writes:

>After reviewing the linkages between the two, I've concluded that
>formally and practically, we DO have the option of letting USEFOR be
>published at this time (once the IESG is through with it).

>In order to have USEFOR be held, we have to enter an explicit request
>with the IESG to hold it, and we haven't done that yet. We could simply
>not do it.

>(This would, of course, mean that if we were to find something in USEPRO
>that required a change in USEFOR, we would have to issue a new version
>of USEFOR.)

>Thoughts?

I have always said it is better to hold it back, and I think Forrest agrees.

But whatever, please don't start tinkering with trying to make USEPRO a
Normative Reference.

To: [email-masked]
Xref: clerew local.usefor:23897
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: draft-allbery-usefor-usepro-00 errata
Date: Fri, 15 Dec 2006 20:32:59 GMT
Lines: 25
Sender: [email-masked]

In <13D1E93F995F11DE450CDCEB@[192.168.1.103]> Harald Alvestrand <[email-masked]> writes:

>--On 14. desember 2006 08:50 -0800 Russ Allbery <[email-masked]> wrote:

>> * (Still under discussion.)  We may wish to reintroduce the possibility
>>    of a path-identity that is not a resolvable name in DNS.

>Last time I looked at the content of Path: headers, I think that there were 
>a fair number of names in dotted.name form that were not DNS resolvable. So 
>banning those names would be a break with existing practice.

Or, alternatively, do we think existing practice is flawed (especially if
we want to encourage the use of MISMATCH et al)? Whatever we say, it will
not change overnight, of course.

To: [email-masked]
Xref: clerew local.usefor:23896
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: draft-allbery-usefor-usepro-00 errata
Date: Fri, 15 Dec 2006 20:30:16 GMT
Lines: 67
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>As a summary from the recent long threads, here are the changes I believe
>should be made to draft-allbery-usefor-usepro-00 if we go forward with
>that document.  The first one below has not previously been discussed on
>the list.

Mostly agreed, but wordings may need tweaking. But there will surely be
lots of other changes needed in due course, so no great rush with these.

> * USEFOR was changed (correctly, IMO) to allow only WSP between the
>   arguments of control commands, not FWS.  I caught this for newgroup and
>   rmgroup but missed it for checkgroups and cancel, both of which need
>   FWS to WSP changes.

Yes, I spotted that one too, and it was on my list of minor typos, etc.

> * application/news-groupinfo and application/news-checkgroups need
>   language limiting the selection of character sets.  I like Harald's
>   proposal and would tend towards something like:

>     Implementations predating this standard may not understand MIME
>     headers and expect newsgroup names to be in ASCII.  Therefore,
>     regardless of what charset is used, the result of reading each octet
>     of the body and setting bit 8 to zero MUST be a valid message
>     specifying the same newsgroup name [or names for checkgroups].  There
>     is no requirement that the newsgroup description survive this
>     treatment.

That seems terribly wordy. Surely all you need is that octets 0-127 (or at
least all those that can occur in newsgroup-names plus TAB) MUST represent
the corresponding ASCII characters.

OTOH, one slight worry. Suppose we later go to newsgroup-names in UTF8
(following the path now beng blazed by the EAI WG). Would we than have to
say that the charset MUST have UTF-8 as a subset? Or merely that the
charset used must be able to represent all the newsgroup-names in that
checkgroups? One of the beauties of newsgroup-names in UTF8 is that (as we
have already established by trying it with the group dk.test.utf8-���)
that existing news servers can cope immediately (user agents would have to
change, of course).

But if checkstatus was allowed to show "dk.test.utf8-���" in either UTF-8
or in Iso-8859-1, would checkstatus still work on existing news servers
(it probably would in the UTF-8 case)?

> * A separate sub-section of duties, before the individual agent duties
>   and possibly as part of the same sub-section as the identity
>   discussion, should be added specifying Path header construction.  The
>   Path header example should be moved to be a sub-section of that
>   section.  That section can lay out all the construction requirements
>   for the Path header field and just be referred to by the individual
>   duties sections.

Yes, that might be good.

To: [email-masked]
Xref: clerew local.usefor:23894
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Protocol changes in draft-allbery-usefor-usepro-00
Date: Fri, 15 Dec 2006 19:58:44 GMT
Lines: 94
Sender: [email-masked]

In <[email-masked]> "Forrest J. Cavalier III" <[email-masked]> writes:

>Wow, a pile of things in one message is really hard to discuss.

Sure it was a big list. But we need to see which items on it are going to
cause disagreement, and then we can start separate threads/issues on those
ones. A lot of them will turn out to be minor misunderstandings, or
matters of wording, etc.

>>>>23. [-1] (3.8) Moderators merely "encouraged" (was SHOULD) to retain
>>>>existing <msg-id>.
>> 
>> 
>>>>That could cause problems if the original poster later wants to cancel
>>>>it, or if he had forwarded it from some other source.
>> 
>> 
>>>Intentional change.

>I'm on the fence here.  The problems with cancels on USENET are not solvable by 
>message ID magic.  Cancel locks are a better way.  (Better is a relative term.)

The WG looked at Cancel Locks once, and decided they would not work
without corresponding changes in other places. So it needs that "Security
Extension" which we have often mentioned and could, in theory, even get
arund to one day :-) .

>The protocol issue is clear:  is a message with the added approved header the 
>same as the original message?  I say it isn't and retaining the message ID is a 
>hack that sort of makes cancels sort of maybe work, sometimes.

I think it is essentially the same message. You do not need to worry about
the precise identity of a message until it is finally let loose on Usenet.
Generally speaking, small additions or changes to headers do not create a
'different' message; even adding stupid warnings about the viruses you
didn't find to the end of the body doesn't make it different. RFC 2822 has
some useful words on this and, modulo a slightly different set of headers,
works also for news.

>> 
>> Adding a note (like the "IESG notes" in RFCs) is already at the border.
>> Truncating articles (and the note explaining how that was done) is also
>> at the border (and IMHO at the wrong side of this border).  But without
>> some clear published rules for the relevant NG warning posters how their
>> articles might be mutilated the default ought to be SHOULD NOT.

>And the underlying principle is that if you alter the body the message is
>different and the ID should change.

Only if you cause the essential "meaning" of the message to change. Which
"most" body changes would do, and even changes of "Subject".

>I also wonder if the post was to more than one moderated group, are there 
>situations where retaining the message ID fails, because some gateway sees it in 
>history and assumes a loop?  Email message IDs are not unique, but....

There will always be some bizarre borderline cases, which is why it is
SHOULD rather than MUST.

>I recall the WG spent a lot of time on this, and "Re:."

The WG certainly spent a lot of time on "Re:", and a very delicate
compromise was reached. I am relieved to see that Russ has left it
strictly alone.

But "cmsg" is now pretty well dead on the real Usenet, which is why simply
confirming its death is probably enough. I am with Frank on this one, that
the principle of "don't tinker with Subject" should be regarded as more
important here.

>Let's be clear on the reason that is a MAY.  Subject is unstructured, and admins 
>should be aware they have the authority to /dev/null any article for any reason, 
>especially ones that they think could cause problems.

>That is not why I think this deserves a MAY.

>That MAY is there to warn User-Agent implementors that putting cmsg starting
>a subject line could lead to poor propagation.  Don't do it.

Eh? How can rejecting an article cause it to propagate worse?

To: [email-masked]
Xref: clerew local.usefor:23893
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Protocol changes in draft-allbery-usefor-usepro-00
Date: Fri, 15 Dec 2006 19:34:16 GMT
Lines: 92
Sender: [email-masked]

>>> Intentional change.

>>> As mentioned in one of my earlier messages about general philosophy, I
>>> removed all requirements placed on agents that are not limited in
>>> practice by the protocol.  Moderators are given the proto-article and
>>> are free to do anything they wish with it

>> They're free to approve and inject / forward it, or to reject it.  But
>> they are NOT free to manipulate the Message-ID.  They can add one if
>> it's missing, your "proto-article" analogy is fine.  They are not free
>> to do whatever they wish.

>This clearly isn't true, as moderators have been changing the message ID
>routinely for years and continue to do so as I write this.  They are
>entirely free to do so; nothing stops them and Usenet does not break as a
>result.

AFAIR, that bit in USEPRO was put in after some discussion about the habit
of moderators changing Message-IDs (apparently Kent Landfield's FAQ
encourages it :-( ); we decided we didn't like the practice and wanted it
to stop. Hence the wording that appeared in USEPRO. Of course, there may
be some bizarre situations (including when he has significantly changed
the text) where the moderator really has to change it, hence it was SHOULD
rather than MUST.

As to whether things break, they can certainly become highly inconvenient.
If the OP has also submitted it to some mailing list, for example, there
is the possibility that the two versions may come together again at some
point, and the same message-id is an immediate indication that they are
the same message. Also, as someone has mentioned, users like to keep track
of, or be alerted to, followups of their own articles.

>> Alternative:  Move the complete _concept_ of moderated NGs to USEAGe,
>> anything, not only 3.8.

>Extremely strongly opposed.  That would be a show-stopper for me.

+1 to that.

>This is a fundamental philosophical difference about what belongs in the
>protocol specification.  If the working group feels that these sorts of
>restrictions on the actions of a moderator belong in USEPRO rather than in
>a best practices document, you should probably chose to move forward with
>a different draft than mine.

Yes, I think the philosophical difference is the real issue we need to
address. Views on what IETF standards may legitimately say cover a wide
range, and Russ's position is fairly far towards one end of a this
continuum. Essentially, he implies that if nothing breaks, then it doesn't
belong in the standard. OK, that is clearly a gross over-simplification,
so don't take it too literally.

I tried to take a more relaxed view in USEPRO (and the WG was often
pressing me in that direction). Essentially, if some feature, whilst not
actually causing stuff to get lost, makes life terribly difficult for
everyone, then it is better for the standard to disallow it. Security
issues are sometimes an example of this. So are things like Trace Headers.
If Received headers were entirely omitted from RFC 282[12], the mail would
still get through. But sorting out all the awkward things that inevitbly
go wrong would be hellishly difficult without them.

However, it is not all as Black and White as the above might suggest, and
I am not sure we should be addressing this particular issues just yet. I
have lots of differences between USEPRO and RUSPRO that I want to raise
sooner of later, but raising them all at once would just give this group a
huge information overload. So I chose to raise the items related to
protocol changes first, and I think we should mainly stick to those for
the moment.

Ultimately, the document we finally produce will have lots of input from
USEPRO and lots off input from RUSSPRO. Whether it is produced by starting
from RUSSPRO and putting some things back, or by starting with USEPRO and
taking things out, is less important than ensuring that we converge on the
correct document eventually.

To: [email-masked]
Xref: clerew local.usefor:23892
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Strawman consensus call, mvgroup control message
Date: Fri, 15 Dec 2006 17:49:14 GMT
Lines: 27
Sender: [email-masked]

In <[email-masked]> Frank Ellermann <[email-masked]> writes:

>Charles Lindsey wrote:
> 
>> Can you please give us some more details of the things that went wrong?

>It's better if you judge it for yourself, I'm not sure that I understood
>it correctly.  It's a short thread (with a solution of "lost aliases"):

><http://news.gmane.org/group/gmane.discuss/thread=10363/focus=10384>

OK, I looked at that thread, and am not quite sure what it was all about.
However, the link that you claimed to have clicked on and not got what you
expected worked for me when I tried it. So it looks like Lars has fixed
whatever had caused the problem. Perhaps you should try clicking on it
again.

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: Protocol changes in draft-allbery-usefor-usepro-00
Date:  Fri, 15 Dec 2006 12:51:27 +0100
Organization:  <URL:http://purl.net/xyzzy>
Lines: 21
Sender: [email-masked]

Forrest J. Cavalier III wrote:

  [explanation of cancel]
>> It MUST contain an explanation if it's not sent by the original poster,
>> for admin cancels and spam cancels.  MAY is not good enough, unless you
>> intend to discuss such details in USEPRO.  Maybe we should do this (?)

> Frank, do you have any way to justify your opinion of MUSTard here?
> What breaks?

It's hard to decide what's a tolerated spam-cancel or a good admin-cancel
without some hint in the body.  It "breaks" in a certain sense as soon as
admins decide to ignore cancels, when figuring out good vs. rogue takes
too much of their time after complaints.

For some 1st party cancels a hint can be also important, if the article
was posted by somebody else forging the "From".  Not enough for a MUST,
but all in all more than only an option, recommending a hint makes sense.

Frank

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: Protocol changes in draft-allbery-usefor-usepro-00
Date:  Fri, 15 Dec 2006 12:31:47 +0100
Organization:  <URL:http://purl.net/xyzzy>
Lines: 44
Sender: [email-masked]

Forrest J. Cavalier III wrote:
 
> And if your moderator is a sadist, then an RFC isn't going to help.

The four times I tried to post articles in Usenet groups with human
moderators all failed.  Based on that somewhat limited experience the
one moderator I trust - after reading the FAQ several times and testing
it - is the modbot protecting nanas.  A modbot is a piece of software,
an RFC can state what it's expected to do.  

> RFCs are an invitation to cooperate.  Period.  The MUST and MUST NOT
> wording looks stern, but remains nothing more than an invitation to
> cooperate.

They're often used as reference for decisions like "observed behaviour
A is right, not A is wrong, so fix it".  Of course you can ignore some
MUSTs, but...

> "non-compliance breaks something."

> IMHO, the GNKSA approach is probably better to address your concerns.

It's limited to newsreaders, otherwise I liked it.  Maybe not exactly
state of the art, it's some years old, but a good start for USEAGE.
That's what Charles did for useage-00.  I'm less impressed by texts
like RFC 1855, and weird body conventions like tear lines.

> The world suffers from the lack of a specification of how to get
> articles from place to place correctly today.

For moderated NGs the modbot or moderator is one of these places.  It
has duties, and some are critical.  Some UAs allow to "post + mail",
it could have strange effects if something in that procedure makes up
its own Message-IDs.  

Articles can be x-posted in more than one moderated NG.  It won't do
if the 1st moderator approves Message-ID <x1>, and the 2nd moderator
approves the article replacing the Message-ID by <x2>, the cancelbot
of the 1st moderator could kill the unknown <x2>.

The Message-ID is essential for Netnews, also in moderated groups.

Frank

Date: Fri, 15 Dec 2006 09:59:28 +0100
From: Harald Alvestrand <[email-masked]>
To: [email-masked]
Subject: Decision: Let's get draft-allbery-usefor-usepro-00 right and publish it as draft-ietf-usefor-usepro-07
Sender: [email-masked]

                         Harald

Date: Fri, 15 Dec 2006 09:46:58 +0100
From: Harald Alvestrand <[email-masked]>
To: Frank Ellermann <[email-masked]>
Cc: [email-masked]
Subject: Re: Protocol changes in draft-allbery-usefor-usepro-00
Sender: [email-masked]

Frank Ellermann wrote:
> Russ Allbery wrote:
>
>   
>> Philosophical difference.  If the working group wishes to take this
>> approach to a protocol specification, you should not use my draft.
>>     
>
> Okay, apparently I'm better off with USEPRO.  I dislike arbitrariness
> of software or moderators hurting the weakest part of the system, the
> normal posters and users.  
>
>   
I think (technical opinion, not chair hat) that just like USEFOR deals 
with the format and ignores the protocol, USEPRO should deal with the 
protocol and ignore the parts that can only be specified as conventions.

Frank, you may be best served by jumping onto the long-languishing draft 
for USEAGE.

                      Harald

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: Message IDs and moderators
Organization: The Eyrie
Date: Thu, 14 Dec 2006 21:31:43 -0800
Sender: [email-masked]

Forrest J Cavalier <[email-masked]> writes:

> Russ and I are agreeing for different reasons. It is hard to know you
> have covered all the cases for retaining or changing it.

> I understand it isn't a netnews article "yet".  Even if it isn't a
> netnews article, it is an RFC2822 message. Message-ID means
> something....

I don't have much to add here (I think you raise good points) except to
note that if one uses method (a) of forwarding messages to a moderator (or
if one just puts the message into a local database that the moderator then
has access to, as is explicitly permitted in my document), the
proto-article is encapsulated and therefore the mail message identifier
(if any) is irrelevant from a Netnews perspective.

We only run into potential issues with the interaction between the mail
message identifier and the Netnews message identifier when moderation
forwarding is done by pretending the proto-article is an e-mail message
and dropping it into the mail with a minimum of gatewaying, forcing the
moderator to then convert it back to a proto-article before dealing with
it.  I think we're all agreed that this is a wart on the protocol and it
would be great to get rid of it if we can finesse the transition issues.

I'm personally inclined to try to pretend that this ugly bit of gatewaying
involved in the current common method of forwarding messages to moderators
doesn't exist when it comes to considering the disposition of the message
identifier, since otherwise we end up with different rules depending on
how the message is forwarded to the moderator and that feels even uglier
and even harder to explain.

Date: Fri, 15 Dec 2006 00:21:33 -0500
From: "Forrest J. Cavalier III" <[email-masked]>
To: [email-masked]
Subject: Re: Message IDs and moderators
Sender: [email-masked]

Russ Allbery wrote:

> Forrest J Cavalier <[email-masked]> writes:
>>The protocol issue is clear:  is a message with the added approved
>>header the same as the original message?  I say it isn't and retaining
>>the message ID is a hack that sort of makes cancels sort of maybe work,
>>sometimes.
> 
> 
> This isn't actually the direction at which I'm coming to this.  The issue
> that you're concerned with (keeping the message ID the same for the same
> message) is one that I think only applies after the message becomes a real
> article and is present on the network.  When sending proto-articles around
> to moderators, no Netnews article has yet been created.  An unknown amount
> of interaction between the moderator and the poster will occur before the
> message is actually sent, and indeed the poster's message may never be
> posted in that form.  Again, consider the RISKS digest, which is a very
> illustrative corner case of the sort of thing that can happen to a
> submission to a moderated group.  (The protocol needs to cover *every* use
> of Netnews, not just the common case.)

Russ and I are agreeing for different reasons. It is hard to know you have 
covered all the cases for retaining or changing it.

I understand it isn't a netnews article "yet".  Even if it isn't a netnews 
article, it is an RFC2822 message. Message-ID means something....
    The "Message-ID:" field provides a unique message identifier that
    refers to a particular version of a particular message.  The
    uniqueness of the message identifier is guaranteed by the host that
    generates it (see below).  This message identifier is intended to be
    machine readable and not necessarily meaningful to humans.  A message
    identifier pertains to exactly one instantiation of a particular
    message; subsequent revisions to the message each receive new message
    identifiers.

    Note: There are many instances when messages are "changed", but those
    changes do not constitute a new instantiation of that message, and
    therefore the message would not get a new message identifier.  For
    example, when messages are introduced into the transport system, they
    are often prepended with additional header fields such as trace
    fields (described in section 3.6.7) and resent fields (described in
    section 3.6.6).  The addition of such header fields does not change
    the identity of the message and therefore the original "Message-ID:"
    field is retained.  In all cases, it is the meaning that the sender
    of the message wishes to convey (i.e., whether this is the same
    message or a different message) that determines whether or not the
    "Message-ID:" field changes, not any particular syntactic difference
    that appears (or does not appear) in the message.

I'm not an email RFC expert, by any means.  So there may be more relevant
text.  But is a proto-article and the version with an Approved header field the 
"same message?"  Certainly a server is going to treat those two differently.

If a server gets the version without the Approved header, should it cache the 
message-ID in History as a "reject"?  No.  But why?  Only because we can reason 
that would be detrimental.  (Is there advice on this in any of the RFCs?  I find 
that caching rejects is more art than science.)

> 
> So, I think the message ID can be kept the same or changed from a protocol
> level and neither decision violates the protocol.  In essence, the article
> is still being prepared up until the point where the moderator injects it.
> 
> I made this change instead as part of a number of similar changes aimed at
> providing a sharp distinction between the Netnews *protocol* and Usenet
> *best practice*, basically flushing the remaining bits of the latter out
> of the protocol specification and leaving them for USEAGE.
> 
>>From the perspective of the protocol, nothing in the protocol breaks if
> the moderator changes the message ID.  I know this from a practical basis:
> many moderators do this now and Usenet still works.  I know this from a
> theoretical basis: there is no guarantee in the protocol specification
> which is violated by this.  Until the moderator injects the message, only
> one copy of that proto-article exists (well, unless the user injected it
> multiple times, but if the moderation status is consistent that just means
> the moderator gets multiple copies and if it isn't, behavior is not going
> to be specifiable anyway).  It's not public until the moderator injects
> it, so it doesn't matter what message ID it has from a Netnews
> perspective.
> 
> As a matter of best practice, absolutely, the moderator should retain the
> message ID, even if they edit the post, add notes, or do anything else
> they're allowed to do by the protocol.  They should do this because it
> enables a variety of other best-practice behavior, such as allowing users
> to score up their own articles or replies to their own articles easily
> based on message ID patterns.  However, all of the reasons why they should
> do this are best practice questions, and therefore so is the
> recommendation.  There is nothing in the *protocol* that makes doing this
> important, and hence it's not a protocol issue.
> 
> I'll be right there with you arguing for it in USEAGE.
> 
> To require it in the protocol, it has to matter for the protocol.  Since
> there's nothing in the protocol that restricts cancel messages to the
> message IDs that one generates locally, or that requires a poster know or
> be able to predict the message ID of their article for any reason, I
> cannot justify a protocol requirement.
> 
> I'm being a bit hard-nosed here about saying "if you don't like it, maybe
> my draft isn't for you" because this is typical of multiple changes I made
> to remove remaining best-practice requirements from the protocol draft and
> that was one of the significant philosophical differences I have with the
> existing USEPRO.  I just wanted to make sure everyone realized that
> removing these sorts of requirements was one of the core points of my
> rewrite, not just a side issue, so if you disagree with this you may not
> agree with the philosophy that I used to make other changes and wording
> decisions throughout the draft.
> 

Date: Thu, 14 Dec 2006 23:50:49 -0500
From: "Forrest J. Cavalier III" <[email-masked]>
To: [email-masked]
Subject: Re: Protocol changes in draft-allbery-usefor-usepro-00
Sender: [email-masked]

Frank Ellermann wrote:

> Russ Allbery wrote:
> 
> 
>>Philosophical difference.  If the working group wishes to take this
>>approach to a protocol specification, you should not use my draft.
> 
> 
> Okay, apparently I'm better off with USEPRO.  I dislike arbitrariness
> of software or moderators hurting the weakest part of the system, the
> normal posters and users.  

If a User-agent developer can't see how to help normal posters and
users, then an RFC isn't going to help.  And if your moderator is
a sadist, then an RFC isn't going to help.

RFCs are an invitation to cooperate.  Period.  The MUST and MUST NOT wording
looks stern, but remains nothing more than an invitation to cooperate.  RFC MUST 
and MUST NOT are not commands, they are statements of specification.  It
is IETF shorthand for "non-compliance breaks something."

Maybe the world needs an RFC like USEAGE.  I don't really care.  (IMHO, the 
GNKSA approach is probably better to address your concerns.)

The world suffers from the lack of a specification of how to get
articles from place to place correctly today.  That's the need for RUSSPRO.

We don't want GNKSA-type stuff in RUSSPRO, because bulky specs are not
nearly as useful as concise ones.  And even if some of that is there, you
can't use what you like to call RFC2119 MUSTard to get what you want.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Message IDs and moderators (was: Protocol changes in draft-allbery-usefor-usepro-00)
Organization: The Eyrie
Date: Thu, 14 Dec 2006 20:41:57 -0800
Sender: [email-masked]

Forrest J Cavalier <[email-masked]> writes:

> Wow, a pile of things in one message is really hard to discuss.

Yeah, sorry about that.  I thought about breaking up my 700+ line response
into separate messages and just couldn't muster the effort.

>>>> 23. [-1] (3.8) Moderators merely "encouraged" (was SHOULD) to retain
>>>> existing <msg-id>.

>>>> That could cause problems if the original poster later wants to
>>>> cancel it, or if he had forwarded it from some other source.

>>> Intentional change.

> I'm on the fence here.  The problems with cancels on USENET are not
> solvable by message ID magic.  Cancel locks are a better way.  (Better
> is a relative term.)

> The protocol issue is clear:  is a message with the added approved
> header the same as the original message?  I say it isn't and retaining
> the message ID is a hack that sort of makes cancels sort of maybe work,
> sometimes.

This isn't actually the direction at which I'm coming to this.  The issue
that you're concerned with (keeping the message ID the same for the same
message) is one that I think only applies after the message becomes a real
article and is present on the network.  When sending proto-articles around
to moderators, no Netnews article has yet been created.  An unknown amount
of interaction between the moderator and the poster will occur before the
message is actually sent, and indeed the poster's message may never be
posted in that form.  Again, consider the RISKS digest, which is a very
illustrative corner case of the sort of thing that can happen to a
submission to a moderated group.  (The protocol needs to cover *every* use
of Netnews, not just the common case.)

So, I think the message ID can be kept the same or changed from a protocol
level and neither decision violates the protocol.  In essence, the article
is still being prepared up until the point where the moderator injects it.

I made this change instead as part of a number of similar changes aimed at
providing a sharp distinction between the Netnews *protocol* and Usenet
*best practice*, basically flushing the remaining bits of the latter out
of the protocol specification and leaving them for USEAGE.

>From the perspective of the protocol, nothing in the protocol breaks if
the moderator changes the message ID.  I know this from a practical basis:
many moderators do this now and Usenet still works.  I know this from a
theoretical basis: there is no guarantee in the protocol specification
which is violated by this.  Until the moderator injects the message, only
one copy of that proto-article exists (well, unless the user injected it
multiple times, but if the moderation status is consistent that just means
the moderator gets multiple copies and if it isn't, behavior is not going
to be specifiable anyway).  It's not public until the moderator injects
it, so it doesn't matter what message ID it has from a Netnews
perspective.

As a matter of best practice, absolutely, the moderator should retain the
message ID, even if they edit the post, add notes, or do anything else
they're allowed to do by the protocol.  They should do this because it
enables a variety of other best-practice behavior, such as allowing users
to score up their own articles or replies to their own articles easily
based on message ID patterns.  However, all of the reasons why they should
do this are best practice questions, and therefore so is the
recommendation.  There is nothing in the *protocol* that makes doing this
important, and hence it's not a protocol issue.

I'll be right there with you arguing for it in USEAGE.

To require it in the protocol, it has to matter for the protocol.  Since
there's nothing in the protocol that restricts cancel messages to the
message IDs that one generates locally, or that requires a poster know or
be able to predict the message ID of their article for any reason, I
cannot justify a protocol requirement.

I'm being a bit hard-nosed here about saying "if you don't like it, maybe
my draft isn't for you" because this is typical of multiple changes I made
to remove remaining best-practice requirements from the protocol draft and
that was one of the significant philosophical differences I have with the
existing USEPRO.  I just wanted to make sure everyone realized that
removing these sorts of requirements was one of the core points of my
rewrite, not just a side issue, so if you disagree with this you may not
agree with the philosophy that I used to make other changes and wording
decisions throughout the draft.

Date: Thu, 14 Dec 2006 23:12:33 -0500
From: "Forrest J. Cavalier III" <[email-masked]>
To: [email-masked]
Subject: Re: Protocol changes in draft-allbery-usefor-usepro-00
Sender: [email-masked]

Wow, a pile of things in one message is really hard to discuss.

Should all be split to separate topics.  Sigh.

I think Russ gets it right extremely often.  RFC 2119 language
is what MUST and MUST NOT be done avoid things breaking, what SHOULD
be done unless you have a really good reason, and to let people know
how they MAY embellish.

I know raw wishes and opinion have carried the WG in the past, but
there is a reason USEAGE exists: to collect the rest of the opinions.

Inline responses below.   (Liberal cutting with the intent to provide
context, not modify your comments...)

Frank Ellermann wrote:

> Russ Allbery wrote:
> 
> 
>>Charles Lindsey <[email-masked]> writes:
> 
> 
> Thanks for the interesting diff list.
> 
> 
>>>22. [00] (3.7) Removed recommendation that reading agent SHOULD have the
>>>capability to display the raw article 'as received'.
> 
> 
>>Intentional change.

Display of articles is totally outside the scope of the article transfer 
protocol.  USEPRO scope is end-to-end from agent to agent, not end-to-end from
keyboard to eyeball.

> 
> 
> IMO mail and news desperately need manual intervention in the case of
> errors, they're not designed to run "unattended" in the case of errors.

I'm with you 100% here, Frank, but it isn't protocol.  Put it in USEAGE.

> 
>>>23. [-1] (3.8) Moderators merely "encouraged" (was SHOULD) to retain
>>>existing <msg-id>.
> 
> 
>>>That could cause problems if the original poster later wants to cancel
>>>it, or if he had forwarded it from some other source.
> 
> 
>>Intentional change.

I'm on the fence here.  The problems with cancels on USENET are not solvable by 
message ID magic.  Cancel locks are a better way.  (Better is a relative term.)

The protocol issue is clear:  is a message with the added approved header the 
same as the original message?  I say it isn't and retaining the message ID is a 
hack that sort of makes cancels sort of maybe work, sometimes.

> 
> Adding a note (like the "IESG notes" in RFCs) is already at the border.
> Truncating articles (and the note explaining how that was done) is also
> at the border (and IMHO at the wrong side of this border).  But without
> some clear published rules for the relevant NG warning posters how their
> articles might be mutilated the default ought to be SHOULD NOT.

And the underlying principle is that if you alter the body the message is
different and the ID should change.

Now, what if you alter the From and the Subject header fields?   Different 
message.

Add or Alter the Approved header field?  Not clear.  What is clear is that the 
message is going to be treated totally differently when received.  It is treated 
as a different message.  (This distinguishes the case from just adding another 
Received header, or Path identity.)

I also wonder if the post was to more than one moderated group, are there 
situations where retaining the message ID fails, because some gateway sees it in 
history and assumes a loop?  Email message IDs are not unique, but....

> 
> 
>>>28. [00] (5) Subjects starting with 'cmsg' not accompanied by a Control
>>>header MAY be rejected.
> 
> 
>>>We discussed this at some stage, but I don't think we ever agreed it one
>>>way or the other.

I recall the WG spent a lot of time on this, and "Re:."

Let's be clear on the reason that is a MAY.  Subject is unstructured, and admins 
should be aware they have the authority to /dev/null any article for any reason, 
especially ones that they think could cause problems.

That is not why I think this deserves a MAY.

That MAY is there to warn User-Agent implementors that putting cmsg starting
a subject line could lead to poor propagation.  Don't do it.

Why not write it as "User Agents SHOULD NOT" then?  Because then you need
a SHOULD NOT and a MAY in order to be clear.  Stick with the MAY, and the
SHOULD NOT is implied.  Nice and concise.

> 
>>>37. [00] (5.3) Body of cancel message MAY contain ...explanatory text.
>>>Was SHOULD.
> 
> 
>>Intentional change.
> 
> 
>>The body text serves no protocol purpose and therefore shouldn't be
>>subject to a protocol constraint.
> 
> 
> It MUST contain an explanation if it's not sent by the original poster,
> for admin cancels and spam cancels.  MAY is not good enough, unless you
> intend to discuss such details in USEPRO.  Maybe we should do this (?)

Frank, do you have any way to justify your opinion of MUSTard here?
What breaks?

> Frank

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: Protocol changes in draft-allbery-usefor-usepro-00
Date:  Fri, 15 Dec 2006 02:58:51 +0100
Organization:  <URL:http://purl.net/xyzzy>
Lines: 11
Sender: [email-masked]

Russ Allbery wrote:

> Philosophical difference.  If the working group wishes to take this
> approach to a protocol specification, you should not use my draft.

Okay, apparently I'm better off with USEPRO.  I dislike arbitrariness
of software or moderators hurting the weakest part of the system, the
normal posters and users.  

Frank

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: Protocol changes in draft-allbery-usefor-usepro-00
Organization: The Eyrie
Date: Thu, 14 Dec 2006 17:30:20 -0800
Sender: [email-masked]

>> Intentional change.

>> As mentioned in one of my earlier messages about general philosophy, I
>> removed all requirements placed on agents that are not limited in
>> practice by the protocol.  Moderators are given the proto-article and
>> are free to do anything they wish with it

> They're free to approve and inject / forward it, or to reject it.  But
> they are NOT free to manipulate the Message-ID.  They can add one if
> it's missing, your "proto-article" analogy is fine.  They are not free
> to do whatever they wish.

This clearly isn't true, as moderators have been changing the message ID
routinely for years and continue to do so as I write this.  They are
entirely free to do so; nothing stops them and Usenet does not break as a
result.

I believe that what my draft says is the correct statement for the
protocol to make.

> Of course there is, users will see if their article was forged or
> manipulated, and report that.  They won't post in the NG of an abusive
> moderator.  Admins can remove the NGs from their server.

This behavior is not considered abusive and only rarely draws any sort of
complaint.

> Alternative:  Move the complete _concept_ of moderated NGs to USEAGe,
> anything, not only 3.8.

Extremely strongly opposed.  That would be a show-stopper for me.

> Adding a note (like the "IESG notes" in RFCs) is already at the border.
> Truncating articles (and the note explaining how that was done) is also
> at the border (and IMHO at the wrong side of this border).  But without
> some clear published rules for the relevant NG warning posters how their
> articles might be mutilated the default ought to be SHOULD NOT.

This is a fundamental philosophical difference about what belongs in the
protocol specification.  If the working group feels that these sorts of
restrictions on the actions of a moderator belong in USEPRO rather than in
a best practices document, you should probably chose to move forward with
a different draft than mine.

>>> 28. [00] (5) Subjects starting with 'cmsg' not accompanied by a
>>> Control header MAY be rejected.

>>> We discussed this at some stage, but I don't think we ever agreed it one
>>> way or the other.

>> Intentional change.

> USEFOR is now tuned to "don't do odd things with Subject: cmsg", after
> years of discussions, this MAY has to be a MUST NOT.  It has to be noted
> as new requirement, a radical change from 1036.  That was a WG
> consensus, I didn't like it, but the alternative "structured subject"
> was admittedly worse.  Please stick to it, news servers have no business
> to look into the subject at all, end of story.

I don't agree that this was the working group consensus.  I stand by what
my draft says, which is also what existing software is already doing, and
I don't think it contradicts anything in USEFOR.  I don't believe that it
assumes the Subject is structured; I believe, rather, that it is a
reasonable backwards compatibility precaution.

>>> 37. [00] (5.3) Body of cancel message MAY contain ...explanatory text.
>>> Was SHOULD.

>> Intentional change.

>> The body text serves no protocol purpose and therefore shouldn't be
>> subject to a protocol constraint.

> It MUST contain an explanation if it's not sent by the original poster,
> for admin cancels and spam cancels.  MAY is not good enough, unless you
> intend to discuss such details in USEPRO.  Maybe we should do this (?)

Philosophical difference.  If the working group wishes to take this
approach to a protocol specification, you should not use my draft.

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: Responses from the USEFOR WG Chairs on IESG DISCUSSes
Date:  Fri, 15 Dec 2006 02:07:07 +0100
Organization:  <URL:http://purl.net/xyzzy>
Lines: 7
Sender: [email-masked]

Harald Alvestrand wrote:

[...]
>               Harald and Alexey

Thanks.

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: Strawman consensus call, mvgroup control message
Date:  Fri, 15 Dec 2006 01:52:28 +0100
Organization:  <URL:http://purl.net/xyzzy>
Lines: 11
Sender: [email-masked]

Charles Lindsey wrote:
 
> Can you please give us some more details of the things that went wrong?

It's better if you judge it for yourself, I'm not sure that I understood
it correctly.  It's a short thread (with a solution of "lost aliases"):

<http://news.gmane.org/group/gmane.discuss/thread=10363/focus=10384>

Frank

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: Protocol changes in draft-allbery-usefor-usepro-00
Date:  Fri, 15 Dec 2006 01:30:36 +0100
Organization:  <URL:http://purl.net/xyzzy>
Lines: 107
Sender: [email-masked]

Russ Allbery wrote:

> Charles Lindsey <[email-masked]> writes:

Thanks for the interesting diff list.

>> 22. [00] (3.7) Removed recommendation that reading agent SHOULD have the
>> capability to display the raw article 'as received'.

> Intentional change.

> For the reasons spelled out in the section for reading agents, I don't
> think makes sense for us to put protocol requirements on the operation of
> reading agents.  Nothing about the operation of a reading agent affects
> the rest of the network.

IMO mail and news desperately need manual intervention in the case of
errors, they're not designed to run "unattended" in the case of errors.

Only real human users can spot some of those issues, and for that job
they need the complete technical info.  It's a part of the design, it
affects the rest of the network if users have no chance to diagnose
errors.

>> 23. [-1] (3.8) Moderators merely "encouraged" (was SHOULD) to retain
>> existing <msg-id>.

>> That could cause problems if the original poster later wants to cancel
>> it, or if he had forwarded it from some other source.

> Intentional change.

> As mentioned in one of my earlier messages about general philosophy, I
> removed all requirements placed on agents that are not limited in
> practice by the protocol.  Moderators are given the proto-article and
> are free to do anything they wish with it

They're free to approve and inject / forward it, or to reject it.  But
they are NOT free to manipulate the Message-ID.  They can add one if
it's missing, your "proto-article" analogy is fine.  They are not free
to do whatever they wish.

> there is no effective protocol limits on their modifications to
> messages, including whether or not they retain the message ID.

Of course there is, users will see if their article was forged or
manipulated, and report that.  They won't post in the NG of an
abusive moderator.  Admins can remove the NGs from their server.

This is an essential point.  You also wouldn't argue that an
injecting agent is free to do whatever it wishes.  In practice it
is, and the standard says what's good - bad - ugly.

> This doesn't feel to me to be in scope for a protocol specification.

Alternative:  Move the complete _concept_ of moderated NGs to USEAGe,
anything, not only 3.8.

> It's not at all clear to me that posters should have the ability to
> cancel messages that they sent to a moderated group

Of course they can cancel their own articles, or any forged articles
claiming to be "from" them.  It's their article.  If moderators don't
like the content of an article they're free to post their own ideas.

They never need to munge / manipulate / forge articles by somebody
else.  If it's too bad they can reject submitted articles, that's
their job.  But not manipulate it in any way they like.

Adding a note (like the "IESG notes" in RFCs) is already at the border.
Truncating articles (and the note explaining how that was done) is also
at the border (and IMHO at the wrong side of this border).  But without
some clear published rules for the relevant NG warning posters how their
articles might be mutilated the default ought to be SHOULD NOT.

>> 28. [00] (5) Subjects starting with 'cmsg' not accompanied by a Control
>> header MAY be rejected.

>> We discussed this at some stage, but I don't think we ever agreed it one
>> way or the other.

> Intentional change.

USEFOR is now tuned to "don't do odd things with Subject: cmsg", after
years of discussions, this MAY has to be a MUST NOT.  It has to be noted
as new requirement, a radical change from 1036.  That was a WG consensus,
I didn't like it, but the alternative "structured subject" was admittedly
worse.  Please stick to it, news servers have no business to look into
the subject at all, end of story.

If that collides with reality it should be at least clear that it's the
fault of the standard or the server, not the fault of the user or UA.

>> 37. [00] (5.3) Body of cancel message MAY contain ...explanatory text.
>> Was SHOULD.

> Intentional change.

> The body text serves no protocol purpose and therefore shouldn't be
> subject to a protocol constraint.

It MUST contain an explanation if it's not sent by the original poster,
for admin cancels and spam cancels.  MAY is not good enough, unless you
intend to discuss such details in USEPRO.  Maybe we should do this (?)

Frank

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: On the issue of publication interval between USEFOR and USEPRO
Date:  Thu, 14 Dec 2006 23:00:39 +0100
Organization:  <URL:http://purl.net/xyzzy>
Lines: 31
Sender: [email-masked]

Harald Alvestrand wrote:
 
> Thoughts?

I'd prefer to get a USEFOR RFC a.s.a.p.  That needs an approval.
That needs the two [Discuss] cleared or swapped into [Abstain].

To clear the [Discuss] we'd probably need a normative reference,
putting USEFOR on hold.  If we don't add a normative reference
it is already blocked by the [Discuss].

So for my goal doing nothing has probably the same effect as 
asking for a hold.  Doing nothing could also be faster, if Sam
and Russ change their [Discuss] into [Abstain].  

Besides with this WG it's as shaky as with SPF, folks won't stop
to promote wild and wonderful changes until the last microsecond
of AUTH48.

I'll only believe it if it's published with a proper RFC number.
The RFC is also quite important for the sleeping 2822upd draft:

Telling the author "BTW, you got the Message-ID wrong as noted
in I-D.ietf-usefor-usefor-12, and the <unstructured> [FWS] is 
also dubious, noted in an obscure pending errata mbox, and the
same I-D" isn't very convincing.  With a proper RFC number it's
better.  USEFOR should have an RFC number before the discussions
about 2822bis start.

Frank

To: [email-masked]
Xref: clerew local.usefor:23872
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Injection-Date and reinjection (was Protocol changes)
Date: Thu, 14 Dec 2006 21:23:14 GMT
Lines: 228
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>Responding to this one specifically quickly, since it may save you some
>time opening a broader discussion.  I think you may have misunderstood the
>impact of my changes to the definition of proto-articles.

>Charles Lindsey <[email-masked]> writes:

>> 4. [--] (3.2.2,3.4,3.8) Injection-Date can be rewritten.

It is clear that the topics of Gatewaying, Reinjection and the role of the
Injection-Date header are closely intertwined, and need to be considered
together. The WG last considered this in any detail in May of 2004
(archives available on www.imc.org/ietf-usefor).

The essential reason for introducing Injection-Date was that we now follow
the RFC 2822 principle that the Date header indicates when the article was
composed, and hence is unsuited to its role of preventing loops when an
article propagates so slowly that it reaches some server after it has
already been seen and expired there. So a man who composes and Dates an
article and then carries it around on his laptop for 3 weeks before
injecting is not disadvantaged (it may be a stupid thing to do, but that
is no concern of ours).

In the old days, servers had to allow for articles arriving up to 14 days
late (transatlantic propagation delays were regularly over a week then).
Nowadays, the cutoff will be nearer 3 days. Injection-Date MUST ONLY be
created by an injecting agent, but once created it must NEVER be removed
so long as there is any possibility that it may still be seen propagating
on the same network where it was injected (and that includes all bizarre
scenarios of gatewaying it to some other medium/network and later
gatewaying it back again).

So gatewaying and reinjection are the critical issues. We always
understood that reinjection was unnecessary on a perfectly regulated
network, but on a real network it would occasionally happen for a whole
variety of bizarre yet plausible reasons. Here is a list of examples we
produced at that time:

   . a relaying agent unable to relay an article to its usual peers due to
   unanticipated communication problems (for example, an agent on a laptop
   that has been physically disconnected from its usual environment) might
   attempt instead to reinject it at a site which did not recognize it as
   a peer;

   . a relaying agent receiving an attempted relay from a source with
   which it did not have a peering agreement might choose to reinject it
   instead, in order to fulfil its obligation to the rest of the network
   (8.2) to be responsible ...

   . there might be a genuine misunderstanding between two sites as to
   whether the relationship between them was as peers for relaying, or as
   client/server for injecting.

   . an article might be gatewayed out of Netnews into some other medium,
   and then back again into the same, or a different, Netnews network.

But we carefully did not write that list into the draft, because we did
not want to imply that such practices were normal, nor that the list was
exhaustive. For sure, there will be other bizarre but plausible reasons
why it makes sense that we cannot possibly foresee now.

Now the only case for reinjection that Russ seems to recognize is where an
article is gatweayed between two totally disjoint Netnews networks, and
that is just one half of one of the scenarios I gave above.

My belief is that it will most commonly arise where it is being reinjected
into the _same_ network; usually because of some misunderstanding between
a server and a client as to whether it is peering or injecting (especially
likely now that NNRP can accept an IHAVE command and treat it as a POST if
the client does not have peering privileges), or maybe as part of a
deliberate decision to inject an article to two servers (for additional
robustness, or other plausible reasons), where one of them allows peering
and the other doesn't.

As regards truly disjoint Netnews networks (which we indeed claim to cater
for in our document), my belief is that they are extremely rare. Most
private networks of which I am aware share the same servers as the regular
Usenet, and hence have a tendency to leak. Preventing such leaks, whilst
not impossible, is a truly difficult task. Even if one site carefully
injects its two categories of netnews to different servers, one can never
be sure that some other site on those networks does not regard them as two
parts of the same network. And then there can be privately and secretly
constructed unofficial paths between them.

Therefore, fixing the problem by saying reinjection MUST convert the
article back into a proto article (by removing the Injectio-Info and the
Injection-Date and cleaning out the Path) just will not work. It *might*
work if you removed the Message-ID at the same time (note that a proto
article can, and frequently does, already contain a Message-ID). But that
would cause all sorts of other problems, especially in the case of the man
who was injecting the same article to two sites, for it is a fundamental
principle of Usenet that articles with the same(different) message-id ARE
the same(different) article.

Moreover, once having allowed an Injection-Date to be removed, you have
had to reinstate the provision that Dates over 72 hours old SHOULD be
rejected (3.4 Step 3 and again in 3.8 Step 4), thus preventing the case of
the man who wants to carry his already written article around for more
than 72 hours on his laptop, which was one of the reasons for introducing
Injection-Date in the first place.

As for what injecting agents should do if offered an article they
recognize as already injected, that is really a matter of site policy
(they clearly have the right to drop it). But to REQUIRE them to drop it
may cut off many perfectly reasonable cases, and is certainly not current
practice. Moreover, allowing Injection-Date to be removed at a Gateway, as
Russ proposes, is only safe if you can be sure you are gatewaying in a
manner that will NEVER allow it back into the original network. Much safer
to leave it there (assuming there is some way to express it in the other
medium).

When we first discussed all this, back in 2004, we tossed the ideas around
for a few weeks, and when it seemed we has reached a rough consensus I
wrote down and posted what I understood we had agreed (because I did not
want to start cutting new text in the draft until it was quite clear what
was to be done).

I reproduce below what I wrote then. On re-reading it now, I still think
it describes correctly what needs to be done.

---------------------------------------------------------------------------

1. There will be a distinction between the process of "injection" (which will
   be the normal situation) and "reinjection", which should be a rare event
   brought about by exceptional circumstances. We discussed many of these
   circumstances (servers on visiting laptops, use of IHAVE by unauthorized
   users, strange gatewaying). I shall write text to indicate that it is for
   such exceptional situations, and that injecting sites should only allow it
   as a result of a clear policy decision to do so.

2. There shall be a header Injection-Date. Syntax as for Date. 
   MUST ONLY be inserted by injecting agents. This header is intended to 
   replace the currently-used but nowhere-documented header 
   "NNTP-Posting-Date" (cf. NOTE in the present 6.19 regarding 
   NNTP-Posting-Host and X-Trace). 
  
3. The existing Injector-Info header to be renamed as Injection-Info (for 
   consistency of naming), and the posting-date parameter to be removed from 
   it. This header SHOULD be rewritten on reinjection (because the reinjecting
   site takes responsibility for it).
  
4. An injecting agent: 
  
   SHOULD reject any article whose Date-header is more than 24 hours into the
   future.

   MUST reject any article which already has an Injection-Date (unless it is
   convinced that it really ought to be reinjecting).

   MUST NOT remove any Injection-Date or NNTP-Posting-Date header that is 
   already present. [Is that too strong for NNTP-Posting-Date?]
  
   MUST[1] create an Injection-Date-header (except when forwarding to a
   moderator).

   SHOULD create an Injection-Info-header (removing any already present);
   likewise Complaints-To header.

5. A reinjecting agent[3]:

   SHOULD reject any article whose Date-header is more than 24 hours into the
   future.

   MUST NOT remove any Injection-Date or NNTP-Posting-Date header that is 
   already present. 
  
   IF there is no Injection-Date-header already present:
      MUST/SHOULD/MAY[4] reject if Date is more than 72 hours into the past,
      and then MUST create an Injection-Date-header.

   SHOULD create an Injection-Info-header (removing any already present);
   likewise Complaints-To header.

6. A posting agent MUST NOT create an Injection-Date-header.
  
7. A relaying or serving agent: 
  
   MUST look at the Injection-Date (or, if missing, at the Date) and 
   refuse the article if that predates the earliest articles of which it 
   normally keeps record, or if it is more than 24 hours into the future 
   (though they MAY use a margin less than that 24 hours). 
  
5. A moderator: 
  
   MUST remove any Injection-Date header already present (though there
   should be none).
  
   SHOULD[5] always retain the incoming Date-header (it is up to his injector 
   to do the 'right thing' as regards staleness checks and inserting 
   Injection-Date). This actually simplifies the moderator's duties 
   compared to the present text. 

Notes: 

[1] If you want a MUST create Injection-Date-header on injecting, then
   that makes it a Required header, and it goes in Chapter 5. So if anyone
   thinks it should be a SHOULD, then please say so quickly, because I
   don't want to be moving it to Chapter 6 once I have started updating
   the text.

[2] NOTE that other agents cannot rely on the presence of Injection-Date,
   since it will be a new feature of our standard.

[3] I don't actually intend to define the term "reinjecting agent"; that
   was just for this explanation.

[4] I put that staleness check in because, in the case of reinjection, the
   thing might have been wandering around the network for days without any
   Injection-Date. Advice needed regarding the MUST/SHOULD/MAY.

[5] I could be persuaded that should be MUST. 

---------------------------------------------------------------------------

Date: Thu, 14 Dec 2006 21:48:44 +0100
From: Harald Alvestrand <[email-masked]>
To: Russ Allbery <[email-masked]>, [email-masked]
Subject: Re: draft-allbery-usefor-usepro-00 errata
Sender: [email-masked]

--On 14. desember 2006 08:50 -0800 Russ Allbery <[email-masked]> wrote:

> * (Still under discussion.)  We may wish to reintroduce the possibility
>    of a path-identity that is not a resolvable name in DNS.

Last time I looked at the content of Path: headers, I think that there were 
a fair number of names in dotted.name form that were not DNS resolvable. So 
banning those names would be a break with existing practice.

Date: Thu, 14 Dec 2006 12:20:59 -0500
From: "Forrest J. Cavalier III" <[email-masked]>
To: Harald Alvestrand <[email-masked]>
CC: [email-masked]
Subject: Re: On the issue of publication interval between USEFOR and USEPRO
Sender: [email-masked]

Harald Alvestrand wrote:

> 
> Folks,
> 
> after reviewing the archives, it's become clear to me that this WG has
> been operating under the assumption that USEFOR would not be published
> as an RFC before USEPRO rolled out of the WG - or the WG shut down
> without producing USEPRO.

Was this an assumption, or just the consensus of a choice?

> Thoughts?

Hold USEFOR.

To: [email-masked]
Xref: clerew local.usefor:23869
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Strawman consensus call, mvgroup control message
Date: Thu, 14 Dec 2006 14:01:27 GMT
Lines: 18
Sender: [email-masked]

In <[email-masked]> Frank Ellermann <[email-masked]> writes:

>I liked the idea, but the technical details are apparently complex.
>The odd effects of a manual mvgroup-emulation on GMaNe (ASRG list)
>convinced me that "mvgroup" is not trivial.

Can you please give us some more details of the things that went wrong?

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: draft-allbery-usefor-usepro-00 errata
Organization: The Eyrie
Date: Thu, 14 Dec 2006 08:50:27 -0800
Sender: [email-masked]

As a summary from the recent long threads, here are the changes I believe
should be made to draft-allbery-usefor-usepro-00 if we go forward with
that document.  The first one below has not previously been discussed on
the list.

 * USEFOR was changed (correctly, IMO) to allow only WSP between the
   arguments of control commands, not FWS.  I caught this for newgroup and
   rmgroup but missed it for checkgroups and cancel, both of which need
   FWS to WSP changes.

 * application/news-groupinfo and application/news-checkgroups need
   language limiting the selection of character sets.  I like Harald's
   proposal and would tend towards something like:

     Implementations predating this standard may not understand MIME
     headers and expect newsgroup names to be in ASCII.  Therefore,
     regardless of what charset is used, the result of reading each octet
     of the body and setting bit 8 to zero MUST be a valid message
     specifying the same newsgroup name [or names for checkgroups].  There
     is no requirement that the newsgroup description survive this
     treatment.

 * multipart/related in newgroup control messages should be
   multipart/mixed instead.

 * A separate sub-section of duties, before the individual agent duties
   and possibly as part of the same sub-section as the identity
   discussion, should be added specifying Path header construction.  The
   Path header example should be moved to be a sub-section of that
   section.  That section can lay out all the construction requirements
   for the Path header field and just be referred to by the individual
   duties sections.

 * The possibility of adding multiple path-identities should be
   reintroduced as part of the rework of the Path description.

 * Serving agents also are not required to modify the Path header field if
   processing an article from a relaying agent or injecting agent that's
   part of the same server.  This can be handled more generally in the
   redone Path section.

 * application/news-transmission should explicitly note that, contrary to
   the previous registration, batches are not permitted.

 * The trimming requirement for References should probably go back to
   MUST.

 * The correct description of special routing for newgroup and rmgroup
   control messages is something more like:

     Exceptionally, control messages creating or removing newsgroups
     (newgroup or rmgroup control messages, for example) SHOULD be relayed
     if the affected group appears in its Newsgroups header field and the
     sending agent would have supplied and the receiving agent would have
     received the newsgroup affected by the control message had it
     existed, even if it currently does not.

   The current USEPRO text for construction of the Newsgroups header field
   for those control messages should be restored (and probably as a
   general statement about group control messages that affect only one
   group rather than stated independently in both newgroup and rmgroup).

 * We need some resolution of the different conflicting syntaxes for ihave
   and sendme control messages.  Path of least resistance is probably to
   revert to the current USEPRO tactic of two separate ABNFs.

 * (Still under discussion.)  We may wish to reintroduce the possibility
   of a path-identity that is not a resolvable name in DNS.

 * (Still under discussion.)  We may wish to add a sentence about the
   construction of newsgroups in the to.* hierarchy.

Date: Thu, 14 Dec 2006 17:16:00 +0100
From: Harald Alvestrand <[email-masked]>
To: [email-masked]
Subject: On the issue of publication interval between USEFOR and USEPRO
Sender: [email-masked]

Folks,

after reviewing the archives, it's become clear to me that this WG has
been operating under the assumption that USEFOR would not be published
as an RFC before USEPRO rolled out of the WG - or the WG shut down
without producing USEPRO.

After reviewing the linkages between the two, I've concluded that
formally and practically, we DO have the option of letting USEFOR be
published at this time (once the IESG is through with it).

In order to have USEFOR be held, we have to enter an explicit request
with the IESG to hold it, and we haven't done that yet. We could simply
not do it.

(This would, of course, mean that if we were to find something in USEPRO
that required a change in USEFOR, we would have to issue a new version
of USEFOR.)

I realize that this is a new idea. I'm not sure it's a good one.But
I want to make sure it's been aired.

Thoughts?

                  Harald

Date: Thu, 14 Dec 2006 12:08:06 +0100
From: Harald Alvestrand <[email-masked]>
To: Russ Allbery <[email-masked]>
Cc: [email-masked]
Subject: Injection-Date (Re: Protocol changes in draft-allbery-usefor-usepro-00)
Sender: [email-masked]

I read the changes of language around injection-date as resulting from 
Russ' removal of the idea of "reinjection".

Given that reinjection does not happen, I don't see any change in 
injection-date allowed.

                 Harald

Russ Allbery wrote:
> Responding to this one specifically quickly, since it may save you some
> time opening a broader discussion.  I think you may have misunderstood the
> impact of my changes to the definition of proto-articles.
>
> Charles Lindsey <[email-masked]> writes:
>
>   
>> 4. [--] (3.2.2,3.4,3.8) Injection-Date can be rewritten.
>>     
>
>   
>> A major change, which could potentially lead to loops, and has
>> necessitated reverting to reliance on Date header in many
>> situations. The whole point of introuducing Injection-Date was to let
>> Date reflect poster's composition time, even if not injected until weeks
>> later. This is now lost. I shall raise a separate thread for this issue.
>>     
>
> Injection-Date may not be rewritten (except by gateways; see below).  The
> intention of my draft is stronger than the existing USEPRO in that not
> only does it prohibit injecting agents from rewriting Injection-Date, it
> prohibits them from accepting articles that already contain an
> Injection-Date header at all.  See section 3.4, point 2:
>
>    2.   It MUST reject any proto-article that does not have the proper
>         mandatory header fields for a proto-article; that has Injection-
>         Date, Injection-Info, or Xref header fields; that has a Path
>         header field containing the "POSTED" <diag-keyword>; or that is
>         not syntactically valid as defined by [USEFOR].  It SHOULD
>         reject any proto-article which contains a header field
>         deprecated for Netnews.  It MAY reject any proto-article that
>         contains trace header fields indicating that it was already
>         injected by an injecting agent that did not add Injection-Info
>         or Injection-Date.
>
> supported by section 3.3.1, paragraph 2:
>
>    A proto-article has the same format as a normal article except that
>    the Injection-Date, Injection-Info, and Xref header fields MUST NOT
>    be present; the Path header field MUST NOT contain a "POSTED" <diag-
>    keyword>; and any of the following mandatory header fields MAY be
>    omitted: Message-ID, Date, and Path.  In all other respects, a proto-
>    article MUST be a valid Netnews article.
>
> The only place where there is a provision for removing an Injection-Date
> header is when gatewaying a message from one Netnews network to another,
> since such gatewaying involves transforming a message from an article back
> to a proto-article and then reinjecting it.  The agent making such a
> transformation is, in this case, specifically required to perform the
> checks on the Injection-Date header that the injecting agent would
> otherwise be performing:
>
>    In the exceptional case that an article needs to be reinjected for
>    some reason (such as transferring an article from one Netnews to
>    another where those networks have no relaying agreement), the posting
>    agent doing the reinjection MUST convert the article back into a
>    proto-article before passing it to an injecting agent (such as by
>    renaming the Injection-Info and Injection-Date header fields and
>    removing any Xref header field) and MUST perform the date checks on
>    the existing Injection-Date or Date header fields that would
>    otherwise be done by the injecting agent.
>
>    Reinjecting articles may cause loops, loss of trace information, and
>    other problems and should only be done with care and when there is no
>    available alternative.  A posting agent that does reinjection is a
>    limited type of gateway and as such is subject to all of the
>    requirements of an incoming gateway in addition to the requirements
>    of a posting agent.
>
>   

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: Protocol changes in draft-allbery-usefor-usepro-00
Organization: The Eyrie
Date: Wed, 13 Dec 2006 23:43:45 -0800
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:

> 1. [-1] (2.2) <path-identity> - Possibility to use a FQDN not resolvable
> in the DNS.

> (This was a suggestion by Harald, & 1 of 2 alternatives in USEPRO)

Intentional, but I don't object to changing it.

Yeah, I took the first listed option in USEPRO which didn't include this
case.  I have no major problem with including it and no strong opinion
either way.  This isn't a change from USEPRO so much as my picking one of
the two listed options that made the most sense to me, and it's not a
place where I have a strong opinion.

The intentional change here is that I added a different preferred first
option, namely the FQDN of the news server itself.

> 2. [00] (3.2) <path-identity>s are case-sensitive.

> I was always told that some servers treated them one way and some the
> other, so you could not rely on either. If anything, I would have
> expected case-insensitive (that is what domain-names are, and what
> USEFOR declares <diag-keyword>s are).

Intentional change.

Declaring them case-sensitive is the conservative choice.  If you treat
them as case-sensitive and they're actually compared case-insensitively,
everything works fine.  The opposite is not true; if your Path identity
varies in case, servers that compare case-sensitively may add incorrect
MISMATCH entries or send you articles that you sent them.

The one exception is if you use a Path header that varies only in case
from another site and assume that it will be compared case-sensitively.  I
thought about adding language to specifically say not to do that, but it's
hard to tell a site how to guarantee that without some way of psychically
determining what <path-nodot> entries might be used by other sites.
Anyone doing this is already violating the SHOULD that says path
identities should be in all lowercase, so adding an additional restriction
that's difficult to phrase didn't seem to me to add anything worthwhile to
the protocol description.

> 3. [-1] (3.3.1,3.4,3.9.2) From not omittable in proto-article

> There is existing usage where the injecting agent fills in From header
> (not possible in NNTP, of course)

Intentional change.

What injecting agent supports this?  Telling posting agents that this is
allowed when very few injecting agents support it doesn't strike me as a
good idea.  This doesn't allow for injecting agents that replace the From
header with one containing the authenticated identity of the user, and I
do know of some injecting agents which do this, but I don't think that's
something that we want to support.  That's not the purpose of the From
header; it's the purpose of the Sender header or (far better) the
Injection-Info header.

> 4. [--] (3.2.2,3.4,3.8) Injection-Date can be rewritten.

> A major change, which could potentially lead to loops, and has
> necessitated reverting to reliance on Date header in many
> situations. The whole point of introuducing Injection-Date was to let
> Date reflect poster's composition time, even if not injected until weeks
> later. This is now lost. I shall raise a separate thread for this issue.

Addressed in an earlier message.

> 5. [-1] (3.3.4) SHOULD trim References to keep <= 998: reduced from MUST.

> This could allow header lines longer than 998 octets, which is known to
> severely impact interoperability. It is not disputed that trimming when
> within the 998 limit is a MAY.

Intentional change, but I'm dithering.

There is an INN bug in older versions that will cause nnrpd to reject
messages with a single header field in excess of 998 octets, folded.  This
is clearly a bug in INN (and was fixed).

I don't mind changing this to MUST, but I'm not sure there's sufficient
justification for that strength of requirement.  On the other hand, I'm
pretty sure I've argued both sides of this one at different times, so
maybe it's best to just leave good enough alone and not change this.

> 6. [00] (3.4) Injecting agents reporting rejection to user agent; was
> SHOULD, now merely "encouraged".

> All it needs is an NNTP 44x response or similar.

Intentional change.

What this means is that the Netnews protocol would be placing a
requirement on the transport layer that it have a means of reporting
errors back to a client.  NNTP has such a means, but I don't think this is
important enough at the protocol level to require of all transport layers.
I don't see a compelling need for a protocol requirement here.  Not
returning specific rejection errors to a posting agent does not break the
Netnews protocol; it's a quality of implementation issue, and there's
enough rationale still present in the document to make implementors aware
of it.

> 7. [+1] (3.4) Injecting agent SHOULD have access to list of newsgroups.

> So it can reject if not at least one group in Newgroups header exists
> This is an addition to agreed MUST keep list of moderated groups.

Intentional change.

> 8. [-1] (3.4) Requirement for injecting agent to forward articles to
> moderator groups reduced from MUST to SHOULD.

> If some small latitude is needed for when the moderated group is in a
> crosspost to some remote and unknown hierarchy, then this should be
> stated explicitly.

Intentional change.

This section says more than just that, and I think the change makes more
sense with more context:

   7.   If the Newsgroups header contains one or more moderated groups
        and the proto-article does not contain an Approved header field,
        the injecting agent SHOULD forward it to a moderator as
        specified in Section 3.4.1.  If the article cannot be forwarded
        to a moderator for some reason, it MUST be rejected.

In other words, in my language, the standard does not REQUIRE that an
injecting agent be capable of forwarding posts to moderators, but does
REQUIRE that it either forward posts to moderators or reject them and
supporting moderated groups is RECOMMENDED.  I think this strikes a
balance that makes more sense from an implementation perspective than the
current USEFOR language.

> 9. [-1] (3.4,3.5) Prepending of <path-identity> to Path by injecting
> agent reduced from MUST to SHOULD.

> I would accept that prepending the <path-diagnostic> POSTED is a SHOULD.

And that's exactly what my language says.  Here's the text:

   9.   The injecting agent SHOULD then prepend the <path-identity> of
        the injecting agent followed by "!.POSTED", optionally "." and
        the FQDN or IP address of the source, and a further "!" to the
        content of the Path header field.  If the injecting agent does
        not support the use of <diag-keyword>, it MUST instead prepend
        its <path-identity> followed by "!"; one or the other of these
        mechanisms MUST be used.

Adding the <path-identity> with the <path-diagnostic> is a SHOULD.  If it
doesn't add the <path-diagnostic>, it MUST add its own <path-identity>
following the current rules for constructing Path headers.

The language is phrased similarly for relaing and serving agents.

Changing the <path-diagnostic> from a MUST to a SHOULD is an intentional
change.

> 10. [00] (3.4) Folding of Path header when length becomes excessive
> reduced from SHOULD to MAY.

Intentional change.

If Path header folding turns out to be a serious problem in practice, I'd
prefer to leave open to agents the alternative of not doing it.

> 11. [-1] (3.4) Recommendation that each injecting agent SHOULD use a
> consistent format for Injection-Info removed.

> Otherwise, the task of chasing the bad guys becomes harder.

Intentional change.

I don't see much purpose served by this requirement and the meaning is
very murky to me.  I think I understand the general intention, but the
rule is not very specific and I'd have a hard time saying whether a
particular behavior complied with this rule or not.  If the format allows
so much flexibility that we have to make a rule like this, I think that's
a problem with the specification of the header field.

> 12. [-1] (3.5) The significance of "world" and "local" in the
> Distribution header are no longer mentioned.

> We recently took care to reserve these in USEFOR, but that requires
> backup here to ensure the protocol fulfils that promise.

Intentional change.

Most everything useful to be said about "world" and "local" was already
said in USEFOR, and "world" in particular SHOULD NOT be used anyway (as
already stated in USEFOR).  Distribution is generally a waste of effort,
and I'd rather keep the rules related to it to a minimum since in practice
few sites are ever going to care.

If we really have to, we can say something like "If the Distribution
header field contains a <dist-name> of 'world', the article SHOULD be
treated as if the Distribution header field were absent," but note that
this is *not* how existing software behaves.  INN, for instance, doesn't
treat "world" any differently from any other distribution except in its
default configuration files.

I wouldn't argue strongly against that change.

I don't see anything else that should be said about local.  The language
in USEFOR now is better and more accurate, IMO, than the language
currently in USEPRO.

> 13. [+1] (3.5,3.6) Removed possibility of "aliassing out" a site where
> you didn't want an article to go by including it in the Path to the
> right of POSTED.

> This is probably an improvement, but it also needs to be considered
> alongside other conventions for "aliassing out", such as the
> 'cybercancel' convention which is no longer mentioned at all.

Intentional change.

I think this whole area is best deferred to USEAGE or some other best
practice document.  It's thankfully becoming less important, and the whole
pseudo-site convention is complex and strange enough (and enough of a
corner case) that it really belongs in a separate document about the
bizarre world of third-party cancels.

> 14. [+1] (3.5) Relaying agent MUST reject articles without Newgroups or
> Message-ID or (injection-Date or Date).

> Before, it was SHOULD reject for any missing mandatory header.

Intentional change.

This is already done in practice by relaying agents that perform even the
most basic validity checks on articles, and *everyone* rejects (or at
least should be rejecting) articles without Message-ID or
Date/Injection-Date headers since the protocol simply doesn't work without
them.  I can see removing Newsgroups here, but I'd rather keep it.

> 15. [-1] (3.5) Relaying agent MUST reject any article already already
> sent to it.

> This is impossible unless relayer keeps a history file (many don't).
> Nothing breaks if you don't do this, and the usual check on the Path
> header is normally considered sufficient. I might buy a SHOULD here,
> with suitable wording.

Intentional change.

Every relayer I've ever heard of keeps a history file and has to to be
usable in the real world.  The protocol breaks down badly if you have
multiple peers and don't do this.  This is a core part of the flood-fill
protocol.

> 16. [-1] (3.5) Relaying agent SHOULD deal with cancel message (if it
> chooses to honour it)

> Yes, USEPRO says the same, but I suspect both are wrong, because it is
> serving/storage agents that should be doing this (all a pure relaying
> agent can do is then not to propagate it further).

Not actually a change, but I think this should stay.  Here's what it says:

   5.   It SHOULD reject any article that matches an already-received
        cancel control message or the contents of the Supersedes header
        field of an accepted article, provided that the relaying agent
        chose (on the basis of local site policy) to honor that cancel
        control message or Supersedes header field.

This isn't about acting on cancel messages for messages one has already
accepted.  It's about accepting cancels before the message arrives, which
is sufficiently counter-intuitive that it needs to be explicitly
mentioned.  I've seen news server implementations that didn't think about
this, and at least in a spam cancel world it used to be important.

Relaying agents of course always have the option of simply not acting on
cancel messages if they don't want to deal.

> 17. [00] (3.5) Relaying agents SHOULD determine verified/expected source
> of article and construct <path-diagnostics> accordingly.

> USEPRO offers various MUST/SHOULD choices here for discussion. Earlier
> USEPROs had MUST. (It is agreed that the <path-identity> itself is MUST,
> in contrast to the case of injecting agents mentioned above.)

Intentional change.

As above, I made this SHOULD everwhere.  And as mentioned above, I think
you misread the part about injecting agents; adding the <path-identity> is
still a MUST in my language.

> 18. [-1] (3.5) No provision for >1 <path-identity> per relaying agent.

> I think we wrote PATH in USEFOR on the assumption that would be allowed,
> and Richard Clayton had an example where he needed a <path-nodot> in
> addition to a domain-name. I see you also removed this from the example
> in 3.5.1.

Intentional change, but I dithered about it and agree with changing it
back.

The main reason not to add this is just complexity of description, not
anything internal to the protocol.  The protocol itself doesn't really
care how many <path-identity>s you add, so I just need to find a better
way of describing the Path header field manipulation so that adding this
didn't strain the language.

I think moving the Path header field manipulation into a separate section
that's just referred to by the separate agent duties sections will give us
the right layout to clearly add this without making one of these numbered
steps even more complex.

> 19. [00] (3.5) No provision to omit <path-identity> for intermediate
> servers on same site.

> Where a site consists of a large farm of servers (for incoming, outgoing
> and whatever else articles) there should be no need to clutter the Path
> header with them all (unless the site finds that helpful). USEPRO
> therefore relaxed the requirement, provided the outgoing one could be
> verified by the next site.

Unintentional.  Sorry about that.

I handled this case for relaying agents and just missed serving agents.
Relaying agents say:

   7.   If the relaying agent is processing an article from an injecting
        agent that is part of the same news server, it MAY leave the
        Path header field unmodified.

Serving agents similarly need to say:

   7.   If the serving agent is processing an article from an injecting
        or relaying agent that is part of the same news server, it MAY
        leave the Path header field unmodified.

Having a separate section describing Path header construction will make it
easier to catch inconsistencies like this.

> 20. [00] (3.5) Relaying agent MAY add a new Xref header.

> Why might it want to do that?

Intentional change.

Because the same piece of software also implements a serving agent, which
has to add the Xref header, and the same code path is used for all
articles received by the server whether they are for further relaying or
local serving.  Everything already copes with relaying agents adding Xref
or changing Xref header fields and it's common for combined news servers
to do this, so it's easiest to just explicitly allow it.

> 21. [00] (3.6) Storage of control messages in the (pseudo) hierarchy
> control(.*) is a SHOULD.

> Is this any of our business?

Intentional change.  See previous discussion.

> 21. [-1] (3.6) No mention of processing cancel messages by
> serving/storage agents.

> This is odd, given that only a serving agent can actually delete an
> article or otherwise deactivate it. Also, you need to deal here with the
> case where a cancel arrives before the article (it is covered in 5.3,
> but it is the serving agent's duty to implement it).

The same language is here as for relaying agents, but you may have missed
it because it's combined with another similar point:

   3.  It SHOULD reject any article that has already been successfully
       sent to it or that matches an already-received and honored cancel
       message or Supersedes header field following the same rules as a
       relaying agent (Section 3.5).

Yes, there is no specific mention in the duties section of acting on
control messages for already received messages.  This is for the same
reason that there's no specific mention in the duties section of acting on
newgroup, rmgroup, or checkgroups control messages.  My presumption is
that the description of the cancel control message later in its own
section is sufficient.  The case of a control message arriving before an
article is only mentioned because it's a surprising corner case.

> 22. [00] (3.7) Removed recommendation that reading agent SHOULD have the
> capability to display the raw article 'as received'.

Intentional change.

For the reasons spelled out in the section for reading agents, I don't
think makes sense for us to put protocol requirements on the operation of
reading agents.  Nothing about the operation of a reading agent affects
the rest of the network.

I certainly agree with this recommendation as part of the best practices
in USEAGE.

> 23. [-1] (3.8) Moderators merely "encouraged" (was SHOULD) to retain
> existing <msg-id>.

> That could cause problems if the original poster later wants to cancel
> it, or if he had forwarded it from some other source.

Intentional change.

As mentioned in one of my earlier messages about general philosophy, I
removed all requirements placed on agents that are not limited in practice
by the protocol.  Moderators are given the proto-article and are free to
do anything they wish with it; there is no effective protocol limits on
their modifications to messages, including whether or not they retain the
message ID.  This doesn't feel to me to be in scope for a protocol
specification.  It's certainly in scope for a best practices document
about moderation.

It's not at all clear to me that posters should have the ability to cancel
messages that they sent to a moderated group (think RISK digest, for
instance), and I think this is going too far into the realm of network
policy and moderation policy to address in this document.

> 24. [00] (3.2.1) Removed provision (MAY) for outgoing gateway to change
> Content-Transfer-Encoding.

> Given that all other agents, from injecting agents on, are forbidden to
> change the Content-Transfer-Encoding (which will normally be 8bit or
> 7bit in Netnews except for genuine binary objects), it seems desirable
> to mention the exception here, where it may be necessary before the
> article can be passed through some email agents.

I don't consider this to be a change.  There's nothing that says that it
*can't* change the Content-Transfer-Encoding, and the only explicit
statement in my document about CTE is a parenthetical comment for the
injecting agent that says it's not allowed to change the CTE as a
component of the general prohibition on changing the body of the message.
There isn't a general statement about CTE that has to be overridden here.

An outgoing gateway, by its very nature, is not subject to any protocol
restrictions from a Netnews perspective, only recommendations.  It may
change the CTE or do anything else it needs to do.

> 25. [+1] (4.1) Application/news-transmission no longer handles batches.

> That needs to be mentioned explicitly, as it is a change to an existing
> application type.

Intentional change.

Agreed that this difference from the registration needs to be mentioned
explicitly.

> 26. [-1] (4.2,4.3) Removed requirement that
> application/news-[groupinfo,checkgroups] MUST NOT be used except with
> relevant control messages.

> Application types are inherently dangerous if they are executed when
> they should not be. Is there any situation where these two could
> properly be used outside of their intended control messages? Sure, for
> informational purposes, or for passing by private arrangement from one
> newsadmin to another, but best leave it to their common sense if they
> want to break the rules. In any case, news-checkgroups does not make
> sense without its accompanying checkscope parameter.

Intentional change.

I can't think of an example of how application/news-groupinfo could be
dangerous.  It's nothing but a bit of text giving a newsgroup name and a
description.  In and of itself, it has no force, power, or action
whatsoever.

Similarly, application/news-checkgroups is simple text; the only
difference is that it inherently represents a claim that it constitutes a
complete list of newsgroups in a particular hierarchy.  I think that the
security implications of this claim are adequately dealt with by the
security considerations section of the registration:

   Security considerations:  This media type provides only a means for
                             conveying a list of newsgroups and does not
                             provide any information indicating whether
                             the sender is authorized to state which
                             newsgroups should exist within a hierarchy.
                             Such authorization must be accomplished by
                             other means.

Prohibiting use of these media types in all other situations is overly
broad and seems unnecessary to me.  If I mail a checkgroups for a
hierarchy to another newsadmin, I don't see any reason why I should be
prohibited from using application/news-checkgroups for that mail message.
Similarly, if a uk.* hierarchy administrator wanted to convey to uk.*
Control the correct information for a new group, I don't see any reason to
prohibit doing so as an application/news-groupinfo part in a mail message.

> 27. [+1] (4.3) Application/news-[groupinfo,checkgroups] have a charset
> parameter.

> I think we are now agreed abut this, but with a restriction to having
> ASCII as a subset.

Intentional change.

Agreed that we need additional text specifying ASCII as a subset.

> 27. [-1] (5) Nothing stated regarding Newsgroups header of Control
> messages.

> Was SHOULD include the groups affected (& possibly others). Without
> that, control messages won't propagate to the places they are likely to
> be needed.

Intentional change, but after looking at the INN code, I was wrong.

See the duties of a relaying agent, which I modified to reflect what I
thought was the existing practice for newgroup propagation:

   Exceptionally, control messages creating new newsgroups (newgroup
   control messages) SHOULD be relayed if the sending agent has been
   configured to supply and the receiving agent to receive the newsgroup
   affected by the control message, even if that newsgroup does not
   currently exist and even if the control message does not contain that
   group in its Newsgroups header field.

Looking at INN's code, though, that's not exactly how it behaves.  It is
indeed still useful to post the messages to the correct newsgroups.

So, I think we should go back to Charles's previous language for
constructing the Newsgroups header field for newgroup and rmgroup
messages, and what the above should really say is:

   Exceptionally, control messages creating or removing newsgroups
   (newgroup or rmgroup control messages, for example) SHOULD be relayed
   if the affected group appears in its Newsgroups header field and the
   sending agent would have supplied and the receiving agent would have
   received the newsgroup affected by the control message had it existed,
   even if it currently does not.

without that bit that I added at the end, which was based on an incorrect
understanding of the code.

In other words, mostly I messed this up; sorry about that.  The remaining
effective change over the existing USEPRO document is that I made the
special relaying rules a SHOULD rather than a MUST and described them in a
different way.

I'm not horribly happy with my language above; I just think it's a minor
improvement.  I'd welcome an even better improvement.

> 28. [00] (5) Subjects starting with 'cmsg' not accompanied by a Control
> header MAY be rejected.

> We discussed this at some stage, but I don't think we ever agreed it one
> way or the other.

Intentional change.

> 29. [00] (5.2) Newsgroup-names MUST be checked against disallowed names
> before group control message is honoured.

> Was SHOULD. That MUST seems a bit excessive for rmgroup and checkgroups.
> Shouldn't it apply just to newgroup?

Intentional change.

It's just as important for checkgroups as it is for newgroup.  Given that,
I included it for rmgroup as well since I can't think of any reason why it
would hurt and it improves the protocol description to make it a general
statement for all group control messages.  (Which, note, includes any
introduced by other documents, such as mvgroup.)

> 30. [-1] (5.2) Nothing said about content of Approved header.

> Surely, it SHOULD identify the person/identity/role of the issuer, and
> be a working email address. Even in alt.* "Approved: foo" is not really
> good enough. News servers are usually configured with what to expect
> there for each hierarchy.

Intentional change.

The content of the Approved header serves no protocol purpose, and USEFOR
already adequately covers the definition of its content.  Control message
authorization is done on the basis of the Sender or From header
(preferrably in combination with a digital signature).

> 31. [-1] (5.2.1) Newgroup message SHOULD include application/group-info.

> Was MUST (though I wouldn't take that to exclude a "For your newsgroups
> file:" to be scanned for - if allowing that was the intent of the
> SHOULD, then some rewording could cover it). That scanning is currently
> "MAY" which is possbly too weak. Or is it trying to allow the case where
> no description exists for the group (for which I would still have
> expected a Newgroups File entry with just the newsgroup-name, which is
> how a checkgroups would show it).

Intentional change.

Newsgroup descriptions are optional in the protocol and newgroup messages
without descriptions work fine with existing software.  Some hierarchies
do not use newsgroup descriptions at all.  They would have to come up with
something to put there to issue a checkgroups, but not all hierarchies
issue checkgroups control messages.  I think SHOULD strikes the right
balance; MUST is too strong because nothing breaks if it's omitted.

> 32. [-1] (5.2.1) Newgroup message containing other attachments in
> addition to news-groupinfo to be multipart/related (was
> multipart/mixed).

> I don't see what the 'relationship' is.

Unintentional change.

Agreed, this should be multipart/mixed.

> 33. [+1] (5.2.3) Checkgroups message still contains chkscope and
> chksernr arguments.

> No change from USEPRO. I think I finally persuaded Russ this should
> remain so.

I'm not horribly happy about it, and in particular the chksernr parameter
adds a new requirement for a local data store for a news server to keep
track of the previous serial number applied.  I'm not sure if anyone's
going to really implement this.  But the argument in favor is compelling,
so....

I guess I'm reluctantly willing to see if anyone picks it up.

> 34. [00] (5.2.3) Checkgroups SHOULD contain sub-hierarchies excluded by
> chkscope, for backwards compatibility.

> Fine in principle, but maybe not implementable. If de.alt.* were as
> chaotic as alt.* (is it?) there would be no way the admins of de.* could
> include a defnitive list of de.alt.*. So some qualification is needed.

Intentional change.  See previous discussion.

> 35. [+1] (5.2.3) Chksernr MUST increase with successive checkgroups
> message.

> Was SHOULD.

Intentional change.

This was added as a security feature, and it's worthless if there aren't
strong guarantees about how it's used.  I also added language not present
in the USEPRO draft RECOMMENDING that servers implement chksernr
processing, since if we're going to specify it, we need to say it has to
be used, or it's a pointless addition.

> 36. [00] (5.2.3) Body of checkgroups SHOULD be application/checkgroups,
> and otherwise SHOULD treat as if it were.

> Why does that SHOULD differ from the corresponding MUST in newgroups? I
> would have expected "SHOULD ... and otherwise MUST ...".

Intentional change.

What MUST in newgroup?  Scanning the body in newgroup messages without a
MIME part is a MAY.  It's raised to SHOULD in checkgroups because unlike
for newgroups, this is actually necessary to honor the checkgroups.

It's SHOULD and not MUST because I don't see the justification for the
stronger requirement.  If a news server wants to apply a different set of
heuristics to examining untagged checkgroups bodies for some reason, I
don't see any obvious cause to prohibit that.  Treating it the same as
application/news-checkgroups is just the RECOMMENDED approach that will
generally work fine.

> 37. [00] (5.3) Body of cancel message MAY contain ...explanatory text.

> Was SHOULD.

Intentional change.

The body text serves no protocol purpose and therefore shouldn't be
subject to a protocol constraint.

> 38. [-1] (5.3) No mention of who should be in From/Sender of cancel
> message.

> Agreed it is not the person who posted the original article. USEPRO says
> it "should" be the person who issued the cancel. That should really be
> "SHOULD".

Intentional change.

The From/Sender of a cancel message is no different in definition than the
From/Sender of any other Netnews message, and as such I think USEFOR
covers the situation sufficiently.  I don't see a reason to call special
attention to the fields for cancel messages when cancel messages are no
different than normal articles in this regard.

> 39. [-1] (5.5) The <relayer-name> in ihave and sendme control messages
> is now optional.

> It is required to be present in USEPRO and in Son-of-1036.

Unintentional change.

USEPRO offers two different syntaxes, one of which is marked SHOULD NOT,
and I found that confusing.  I tried to combine them and missed this
distinction.  (So, actually, it's not required to be present in USEPRO;
it's only a SHOULD NOT to omit it.)

I'm not sure what to do with this.  I find having two separate ABNFs for
the same field to be very confusing.  I thought I could just use the
varient form as the more general one and mention in the description that
the message IDs SHOULD be in the body and not the header, but apparently
whether the relayer name is optional is also different between the forms,
which makes them not compatible.

This area is annoying.  I hate spending time on these control messages in
the working group when essentially no one uses them.  To make it worse,
I'm almost certainly retreading a discussion that we already had.  Maybe
having two varient ABNFs is the best that we can do.

> 40. [-1] (5.5) Format of batched news in response to sendme removed.

> Note that this was a format 'on the wire' and was intended to be sent on
> the same 'wire' as the ihave and sendme messages (which would usually be
> a UUCP connection). Hence it is a legitimate item to
> standardize. Probably needs some brief mention of its compressed
> versions.

Intentional change.

   Upon receipt of the sendme message (and a decision to honor it), the
   receiving agent sends the article or articles requested.  The
   mechanism for this transmission is unspecified by this document and
   is arranged between the sites involved.

Specifying the behavior of the transport layer is out of the scope of this
document.  Discussion of the UUCP batch format no more belongs here than
does a discussion of dot-escaping in NNTP.

> 41. [-1] (5.5) The protocol for using the 'to.' newsgroup-name is
> omitted.

> Insofar as this protocol may still occasionally be used, it needs to be
> documented (as with the batch format).

I have:

   These control messages are normally sent as point-to-point articles
   between two sites and not then sent on to other sites.  Newsgroups
   beginning with "to." are reserved for such point-to-point
   communications.

Son-of has:

          It is conventional to reserve newsgroup names beginning with
          "to." for test messages sent  on  an  essentially  point-to-
          point basis (see also the ihave/sendme protocol described in
          section 7.2); newsgroup names beginning  with  "to."  SHOULD
          not be used for any other purpose.  The second (and possibly
          later) components of such a name should, together,  comprise
          the  relayer name (see section 5.6) of a relayer.  The news-
          group exists only at the named relayer  and  its  neighbors.
          The  neighbors all pass that newsgroup to the named relayer,
          while the named relayer does not pass it to anyone.

So the question, unless someone has some additional documentation more
thorough than Son-of available, is whether we want to get into how
newsgroups in the to.* hierarchy are named.  The problem with doing so is
that we then have to get into defining the relayer name (to use the Son-of
term), which for the use of to.* groups in practice is a UUCP host name.
That feels to me like wandering rather far into the weeds.

I suppose we can throw in something here like "the components after 'to.'
are taken to be the <path-identity> of the agent to which an article
posted to that to.* newsgroup should be sent."

> 42. [-1] (5.6) Obsolete control messages SHOULD NOT be sent/honoured/

> Was MUST NOT. Cf the MUST NOT for obsolete headers in USEFOR.

Intentional change.

MUST NOT is unwarranted.  Sending them doesn't break anything.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: Protocol changes in draft-allbery-usefor-usepro-00
Organization: The Eyrie
Date: Wed, 13 Dec 2006 13:24:53 -0800
Sender: [email-masked]

Responding to this one specifically quickly, since it may save you some
time opening a broader discussion.  I think you may have misunderstood the
impact of my changes to the definition of proto-articles.

Charles Lindsey <[email-masked]> writes:

> 4. [--] (3.2.2,3.4,3.8) Injection-Date can be rewritten.

> A major change, which could potentially lead to loops, and has
> necessitated reverting to reliance on Date header in many
> situations. The whole point of introuducing Injection-Date was to let
> Date reflect poster's composition time, even if not injected until weeks
> later. This is now lost. I shall raise a separate thread for this issue.

Injection-Date may not be rewritten (except by gateways; see below).  The
intention of my draft is stronger than the existing USEPRO in that not
only does it prohibit injecting agents from rewriting Injection-Date, it
prohibits them from accepting articles that already contain an
Injection-Date header at all.  See section 3.4, point 2:

   2.   It MUST reject any proto-article that does not have the proper
        mandatory header fields for a proto-article; that has Injection-
        Date, Injection-Info, or Xref header fields; that has a Path
        header field containing the "POSTED" <diag-keyword>; or that is
        not syntactically valid as defined by [USEFOR].  It SHOULD
        reject any proto-article which contains a header field
        deprecated for Netnews.  It MAY reject any proto-article that
        contains trace header fields indicating that it was already
        injected by an injecting agent that did not add Injection-Info
        or Injection-Date.

supported by section 3.3.1, paragraph 2:

   A proto-article has the same format as a normal article except that
   the Injection-Date, Injection-Info, and Xref header fields MUST NOT
   be present; the Path header field MUST NOT contain a "POSTED" <diag-
   keyword>; and any of the following mandatory header fields MAY be
   omitted: Message-ID, Date, and Path.  In all other respects, a proto-
   article MUST be a valid Netnews article.

The only place where there is a provision for removing an Injection-Date
header is when gatewaying a message from one Netnews network to another,
since such gatewaying involves transforming a message from an article back
to a proto-article and then reinjecting it.  The agent making such a
transformation is, in this case, specifically required to perform the
checks on the Injection-Date header that the injecting agent would
otherwise be performing:

   In the exceptional case that an article needs to be reinjected for
   some reason (such as transferring an article from one Netnews to
   another where those networks have no relaying agreement), the posting
   agent doing the reinjection MUST convert the article back into a
   proto-article before passing it to an injecting agent (such as by
   renaming the Injection-Info and Injection-Date header fields and
   removing any Xref header field) and MUST perform the date checks on
   the existing Injection-Date or Date header fields that would
   otherwise be done by the injecting agent.

   Reinjecting articles may cause loops, loss of trace information, and
   other problems and should only be done with care and when there is no
   available alternative.  A posting agent that does reinjection is a
   limited type of gateway and as such is subject to all of the
   requirements of an incoming gateway in addition to the requirements
   of a posting agent.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: Protocol changes in draft-allbery-usefor-usepro-00
Organization: The Eyrie
Date: Wed, 13 Dec 2006 13:15:00 -0800
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:

> This list summarizes the normative changes made between USEPRO and
> Russ's document (may we call it "RUSSPRO" for the moment?).

> Some are substantial and some trivial; some are obviously right and some
> (IMO) obviously wrong; and some may be unintended consequences of the
> new wording; many just involve changes between MUST/SHOULD/MAY.

Thank you for the thorough review; you have caught several places that
were simple errors on my part (multipart/related for newgroup mesages, for
instance; I don't know where I got that from, as it had been my intention
to mostly copy USEPRO there).  I'm embarassed that there were a couple of
those.  I suppose that's the inherent peril of rewriting.

I will try to respond to this message in depth tonight noting which of
these were simple errors in my draft that I would correct as you describe
and which of these changes were intentional.

In several cases, you noted a change that I didn't intend to make and that
I thought I *hadn't* made, and I believe it may just be that the wording
moved from one place to another.  I need to study those in more detail
though before I can say for sure what happened.

To: [email-masked]
Xref: clerew local.usefor:23862
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Protocol changes in draft-allbery-usefor-usepro-00
Date: Wed, 13 Dec 2006 20:41:13 GMT
Lines: 337
Sender: [email-masked]

This list summarizes the normative changes made between USEPRO and Russ's
document (may we call it "RUSSPRO" for the moment?).

Some are substantial and some trivial; some are obviously right and some
(IMO) obviously wrong; and some may be unintended consequences of the new
wording; many just involve changes between MUST/SHOULD/MAY.

Nevertheless, if the WG decides to go with RUSSPRO, it needs to be aware
of what changes it has made, and to discuss any that are in doubt.

I have indicated my own opinion of each one with a score ([+1], [-1] or
[00]).  Except for item #4, which is important enought for a separate
thread (which I shall hopefully set in motion tomorrow).

1. [-1] (2.2) <path-identity> - Possibility to use a FQDN not resolvable
in the DNS.

(This was a suggestion by Harald, & 1 of 2 alternatives in USEPRO)

2. [00] (3.2) <path-identity>s are case-sensitive.

I was always told that some servers treated them one way and some the
other, so you could not rely on either. If anything, I would have expected
case-insensitive (that is what domain-names are, and what USEFOR declares
<diag-keyword>s are).

3. [-1] (3.3.1,3.4,3.9.2) From not omittable in proto-article

There is existing usage where the injecting agent fills in From header
(not possible in NNTP, of course)

4. [--] (3.2.2,3.4,3.8) Injection-Date can be rewritten.

A major change, which could potentially lead to loops, and has
necessitated reverting to reliance on Date header in many situations. The
whole point of introuducing Injection-Date was to let Date reflect
poster's composition time, even if not injected until weeks later. This is
now lost. I shall raise a separate thread for this issue.

5. [-1] (3.3.4) SHOULD trim References to keep <= 998: reduced from MUST.

This could allow header lines longer than 998 octets, which is known to
severely impact interoperability. It is not disputed that trimming when
within the 998 limit is a MAY.

6. [00] (3.4) Injecting agents reporting rejection to user agent; was
SHOULD, now merely "encouraged".

All it needs is an NNTP 44x response or similar.

7. [+1] (3.4) Injecting agent SHOULD have access to list of newsgroups.

So it can reject if not at least one group in Newgroups header exists This
is an addition to agreed MUST keep list of moderated groups.

8. [-1] (3.4) Requirement for injecting agent to forward articles to
moderator groups reduced from MUST to SHOULD.

If some small latitude is needed for when the moderated group is in a
crosspost to some remote and unknown hierarchy, then this should be stated
explicitly.

9. [-1] (3.4,3.5) Prepending of <path-identity> to Path by injecting agent
reduced from MUST to SHOULD.

I would accept that prepending the <path-diagnostic> POSTED is a SHOULD.

10. [00] (3.4) Folding of Path header when length becomes excessive
reduced from SHOULD to MAY.

11. [-1] (3.4) Recommendation that each injecting agent SHOULD use a
consistent format for Injection-Info removed.

Otherwise, the task of chasing the bad guys becomes harder.

12. [-1] (3.5) The significance of "world" and "local" in the Distribution
header are no longer mentioned.

We recently took care to reserve these in USEFOR, but that requires backup
here to ensure the protocol fulfils that promise.

13. [+1] (3.5,3.6) Removed possibility of "aliassing out" a site where you
didn't want an article to go by including it in the Path to the right of
POSTED.

This is probably an improvement, but it also needs to be considered
alongside other conventions for "aliassing out", such as the 'cybercancel'
convention which is no longer mentioned at all.

14. [+1] (3.5) Relaying agent MUST reject articles without Newgroups or
Message-ID or (injection-Date or Date).

Before, it was SHOULD reject for any missing mandatory header.

15. [-1] (3.5) Relaying agent MUST reject any article already already sent
to it.

This is impossible unless relayer keeps a history file (many don't).
Nothing breaks if you don't do this, and the usual check on the Path
header is normally considered sufficient. I might buy a SHOULD here, with
suitable wording.

16. [-1] (3.5) Relaying agent SHOULD deal with cancel message (if it
chooses to honour it)

Yes, USEPRO says the same, but I suspect both are wrong, because it is
serving/storage agents that should be doing this (all a pure relaying
agent can do is then not to propagate it further).

17. [00] (3.5) Relaying agents SHOULD determine verified/expected source
of article and construct <path-diagnostics> accordingly.

USEPRO offers various MUST/SHOULD choices here for discussion. Earlier
USEPROs had MUST. (It is agreed that the <path-identity> itself is MUST,
in contrast to the case of injecting agents mentioned above.)

18. [-1] (3.5) No provision for >1 <path-identity> per relaying agent.

I think we wrote PATH in USEFOR on the assumption that would be allowed,
and Richard Clayton had an example where he needed a <path-nodot> in
addition to a domain-name. I see you also removed this from the example in
3.5.1.

19. [00] (3.5) No provision to omit <path-identity> for intermediate
servers on same site.

Where a site consists of a large farm of servers (for incoming, outgoing
and whatever else articles) there should be no need to clutter the Path
header with them all (unless the site finds that helpful). USEPRO
therefore relaxed the requirement, provided the outgoing one could be
verified by the next site.

20. [00] (3.5) Relaying agent MAY add a new Xref header.

Why might it want to do that?

21. [00] (3.6) Storage of control messages in the (pseudo) hierarchy
control(.*) is a SHOULD.

Is this any of our business?

21. [-1] (3.6) No mention of processing cancel messages by serving/storage
agents.

This is odd, given that only a serving agent can actually delete an
article or otherwise deactivate it. Also, you need to deal here with the
case where a cancel arrives before the article (it is covered in 5.3, but
it is the serving agent's duty to implement it).

22. [00] (3.7) Removed recommendation that reading agent SHOULD have the
capability to display the raw article 'as received'.

23. [-1] (3.8) Moderators merely "encouraged" (was SHOULD) to retain
existing <msg-id>.

That could cause problems if the original poster later wants to cancel it,
or if he had forwarded it from some other source.

24. [00] (3.2.1) Removed provision (MAY) for outgoing gateway to change
Content-Transfer-Encoding.

Given that all other agents, from injecting agents on, are forbidden to
change the Content-Transfer-Encoding (which will normally be 8bit or 7bit
in Netnews except for genuine binary objects), it seems desirable to
mention the exception here, where it may be necessary before the article
can be passed through some email agents.

25. [+1] (4.1) Application/news-transmission no longer handles batches.

That needs to be mentioned explicitly, as it is a change to an existing
application type.

26. [-1] (4.2,4.3) Removed requirement that
application/news-[groupinfo,checkgroups] MUST NOT be used except with
relevant control messages.

Application types are inherently dangerous if they are executed when they
should not be. Is there any situation where these two could properly be
used outside of their intended control messages? Sure, for informational
purposes, or for passing by private arrangement from one newsadmin to
another, but best leave it to their common sense if they want to break the
rules. In any case, news-checkgroups does not make sense without its
accompanying checkscope parameter.

27. [+1] (4.3) Application/news-[groupinfo,checkgroups] have a charset
parameter.

I think we are now agreed abut this, but with a restriction to having
ASCII as a subset.

27. [-1] (5) Nothing stated regarding Newsgroups header of Control
messages.

Was SHOULD include the groups affected (& possibly others). Without that,
control messages won't propagate to the places they are likely to be
needed.

28. [00] (5) Subjects starting with 'cmsg' not accompanied by a Control
header MAY be rejected.

We discussed this at some stage, but I don't think we ever agreed it one way
or the other.

29. [00] (5.2) Newsgroup-names MUST be checked against disallowed names before
group control message is honoured.

Was SHOULD. That MUST seems a bit excessive for rmgroup and checkgroups.
Shouldn't it apply just to newgroup?

30. [-1] (5.2) Nothing said about content of Approved header.

Surely, it SHOULD identify the person/identity/role of the issuer, and be a
working email address. Even in alt.* "Approved: foo" is not really good
enough. News servers are usually configured with what to expect there for each
hierarchy.

31. [-1] (5.2.1) Newgroup message SHOULD include application/group-info.

Was MUST (though I wouldn't take that to exclude a "For your newsgroups file:"
to be scanned for - if allowing that was the intent of the SHOULD, then some
rewording could cover it). That scanning is currently "MAY" which is possbly
too weak. Or is it trying to allow the case where no description exists for
the group (for which I would still have expected a Newgroups File entry with
just the newsgroup-name, which is how a checkgroups would show it).

32. [-1] (5.2.1) Newgroup message containing other attachments in addition to
news-groupinfo to be multipart/related (was multipart/mixed).

I don't see what the 'relationship' is.

33. [+1] (5.2.3) Checkgroups message still contains chkscope and chksernr
arguments.

No change from USEPRO. I think I finally persuaded Russ this should remain so.

34. [00] (5.2.3) Checkgroups SHOULD contain sub-hierarchies excluded by
chkscope, for backwards compatibility.

Fine in principle, but maybe not implementable. If de.alt.* were as chaotic as
alt.* (is it?) there would be no way the admins of de.* could include a
defnitive list of de.alt.*. So some qualification is needed.

35. [+1] (5.2.3) Chksernr MUST increase with successive checkgroups message.

Was SHOULD.

36. [00] (5.2.3) Body of checkgroups SHOULD be application/checkgroups, and
otherwise SHOULD treat as if it were.

Why does that SHOULD differ from the corresponding MUST in newgroups? I would
have expected "SHOULD ... and otherwise MUST ...".

37. [00] (5.3) Body of cancel message MAY contain ...explanatory text.

Was SHOULD.

38. [-1] (5.3) No mention of who should be in From/Sender of cancel message.

Agreed it is not the person who posted the original article. USEPRO says it
"should" be the person who issued the cancel. That should really be "SHOULD".

39. [-1] (5.5) The <relayer-name> in ihave and sendme control messages is now
optional.

It is required to be present in USEPRO and in Son-of-1036.

40. [-1] (5.5) Format of batched news in response to sendme removed.

Note that this was a format 'on the wire' and was intended to be sent on the
same 'wire' as the ihave and sendme messages (which would usually be a UUCP
connection). Hence it is a legitimate item to standardize. Probably needs some
brief mention of its compressed versions.

41. [-1] (5.5) The protocol for using the 'to.' newsgroup-name is omitted.

Insofar as this protocol may still occasionally be used, it needs to be
documented (as with the batch format).

42. [-1] (5.6) Obsolete control messages SHOULD NOT be sent/honoured/

Was MUST NOT. Cf the MUST NOT for obsolete headers in USEFOR.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: Strawman consensus call, mvgroup control message
Organization: The Eyrie
Date: Wed, 13 Dec 2006 10:50:57 -0800
Sender: [email-masked]

Frank Ellermann <[email-masked]> writes:

> I liked the idea, but the technical details are apparently complex.
> The odd effects of a manual mvgroup-emulation on GMaNe (ASRG list)
> convinced me that "mvgroup" is not trivial.

Yeah, most of my concerns about how this should work come from talking to
larsi about doing this on Gmane.  I still have a longish list of things
that he'd really like the overview and storage subsystems to do so that
his group and article moving works better that INN just can't handle yet.

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: Strawman consensus call, mvgroup control message
Date:  Wed, 13 Dec 2006 16:36:02 +0100
Organization:  <URL:http://purl.net/xyzzy>
Lines: 15
Sender: [email-masked]

Charles Lindsey wrote:

> If the WG regards automatic resubscription within User Agents as
> an essential feature, then that would be the end of the matter.

I can't judge that.  What about the proposals to extract "mvgroup"
into an experimental document ?  We've to update the Usefor Charter
really soon now anyway, and could add this.

I liked the idea, but the technical details are apparently complex.
The odd effects of a manual mvgroup-emulation on GMaNe (ASRG list)
convinced me that "mvgroup" is not trivial.

Frank

Cc: Harald Alvestrand <[email-masked]>
From: Thorfinn <[email-masked]>
Subject: Re: Decision time: versions of -usepro
Date: Wed, 13 Dec 2006 14:33:35 +1100
To: [email-masked]
Sender: [email-masked]

On 11 Dec 2006, at 19:01, Harald Alvestrand wrote:
> OK, folks have had a little time to look at the two USEPRO versions:
>
> draft-ietf-usefor-usepro-06.txt
> draft-allbery-usefor-usepro-00.txt
>
> It's time to make a decision on which one the group wishes to build  
> upon.
> I'll give three alternatives:
>
> 1) draft-ietf-usefor-usepro-06
> 2) draft-allbery-usefor-usepro-00
> 3) I have another opinion, which is.....
>
> Please - only respond if you have read both documents!
> As usual, send mail to the chairs if you don't want to clutter up  
> the list, but the tally (if needed) will be by name.

Been a *long* while since I posted anything related to USEFOR/USEPRO,  
but I have definitely been lurking and reading.

I do somewhat agree that this is not an ideal time for such a poll,  
but that said, I've read both drafts and the diff.

Prefer option (2).

Thanks,

    Thorfinn

-- 
<a href="http://tertius.net.au/~thorfinn/">[email-masked]</a>
The world is coming to an end.  Please log off.  -- BSD fortune file

Date: Tue, 12 Dec 2006 10:39:05 -0800 (PST)
From: [email-masked]
To: [email-masked]
Subject: Re:  Decision time: versions of -usepro
Sender: [email-masked]

2) draft-allbery-usefor-usepro-00

with some suggested changes in the area of definitions ("distinguished 
into"?), followups, incoming gateways, constructing References, control
messages, and "denial of service", which I will write up further when
I get time.

To: [email-masked]
Xref: clerew local.usefor:23850
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Decision time: versions of -usepro
Date: Tue, 12 Dec 2006 11:12:48 GMT
Lines: 62
Sender: [email-masked]

In <[email-masked]> Harald Alvestrand <[email-masked]> writes:

>OK, folks have had a little time to look at the two USEPRO versions:

>draft-ietf-usefor-usepro-06.txt
>draft-allbery-usefor-usepro-00.txt

STRONGEST POSSIBLE OBJECTIONS to such an early decision.

I have been reading the new draft very carefully, and have not even got
to the end yet.

There are many protocol changes which have not been discussed yet and
which need to be explored, There are subtle changes in meaning which need
to be discussed, understood and resolved. There are changes in style
which, whilst not crucial, we need to form an opinion on.

Ny broad reaction is that the change in order of presentation of topics is
good, and I would happily incorporate most of them into USEPRO. Likewise
many detailed bits and pieces. But there are is also much that I would
disagree with - particularly where the reader has been given insufficient
guidance as to the motivation for what has been specified, and omissions
of things which really need to be said.

There has been no discussion whatsoever on this new draft. The only
discussion so far has been on Russ's preliminary summary of the main
protocol changes. And the reason for this lack of discussion is, I
imagine, that people have not had time to come to grips with the changes
that have been made.

It is my intention (and I am just about ready to start doing it) to start
posting an analysis of the differences. But this needs doing in smallish
stages that can be digested and discussed one at a time. And my intention
was to start with an analysis of the protocol changes, some of which are
quite significant.

>It's time to make a decision on which one the group wishes to build upon.
>I'll give three alternatives:

>1) draft-ietf-usefor-usepro-06
>2) draft-allbery-usefor-usepro-00
>3) I have another opinion, which is.....

I have another opinion, which is that we need time, analysis and discussion.

And then probably further editions of one or both drafts enabling a
properly informed choice to be made between them.

>Please - only respond if you have read both documents!

s/read/read, mark, learn and inwardly digest/ :-)

Date: Tue, 12 Dec 2006 11:03:30 +0000
From: Alexey Melnikov <[email-masked]>
To: Russ Allbery <[email-masked]>
CC: [email-masked]
Subject: Re: USEFOR-11 troubles
Sender: [email-masked]

Russ Allbery wrote:

>>Hi, Usefor-11 didn't make it in the first attempt, it got two [DISCUSS]:
>>    
>>
>>https://datatracker.ietf.org/public/pidtracker.cgi?command=print_ballot&ballot_id=2253&filename=draft-ietf-usefor-usefor
>>    
>>
>>Sam proposes to use a normative reference to USEPRO.  Russ Houley's
>>[DISCUSS] is in a similar direction, he wants to make sure that the
>>"security considerations" in USEPRO (referenced by USEFOR) are not
>>lost, i.e. published in a RFC, not only in an I-D.
>>    
>>
>Reviewing USEPRO brought home to me that it's not possible to implement a
>useful piece of Netnews software other than a pure article reader with
>only the information in USEFOR.  That doesn't necessarily mean that USEFOR
>can't advance separately as an underlying format which is then used by a
>later standard, but that's definitely the approach we'd have to take to
>sever the documents, which means that USEFOR will need to have no
>normative references to USEPRO.
>
>There are several that seem rather normative to me having just gone
>through all of USEPRO, apart from the security considerations.  Section
>3.1.4 says that control messages are defined in USEPRO, section 3.1.5
>defers to USEPRO for the definition of <diag-keyword>, section 3.1.6
>defers to USEPRO for the content of the Subject header (and that reference
>I think could simply be dropped, as the only place that USEPRO discusses
>the Subject header is in the context of a followup and then only as a
>MAY), section 3.2 states that further requirements for optional header
>fields are in USEPRO, section 3.2.3 defers to USEPRO for valid control
>message <verb>s, and section 3.2.12 defines the action of Supersedes in
>terms of cancel control messages defined in USEPRO.
>
>On the surface, it's hard to argue against some or most of those being
>normative references.
>
>Control messages are the hardest problem here, and I expect are the most
>likely to prompt a DISCUSS.  At the least, I think all article syntax
>including acceptable values for control message <verb> and for Path
><diag-keyword> should be defined in USEFOR to make it a standalone format
>document.  That's actually easier than it sounds, since for control
>message verbs *all* values that fit the current syntax are acceptable and
>interpretation is up to the protocol being used, so there's no need to
>defer to USEPRO as aggressively as USEFOR currently does.
>
>For <diag-keyword>, I would move the acceptable list of keywords (SEEN,
>POSTED, and MISMATCH) into USEFOR with *brief* descriptions of their
>meaning and leave to USEPRO only the specific instructions of how to apply
>those meanings.  This would also simplify writing USEPRO, since USEPRO is
>laid out as a list of instructions and doesn't lend itself well to a
>digression on providing additional syntax for Path <diag-keyword>s.
>
>The other references could be dropped or downgraded in their wording
>easily except Supersedes, where the description would need to be rewritten
>without reference to cancel control messages.
>  
>
I think the WG needs to see some specific text before considering moving 
any text from USEPRO to USEFOR.

Date: Tue, 12 Dec 2006 10:54:25 +0000
From: Alexey Melnikov <[email-masked]>
To: "Forrest J. Cavalier III" <[email-masked]>
CC: [email-masked]
Subject: Re: Decision time: versions of -usepro
Sender: [email-masked]

Forrest J. Cavalier III wrote:

> Harald Alvestrand wrote:
>
>> OK, folks have had a little time to look at the two USEPRO versions:
>
> December is a busy time to initiate an important poll. But rather
> than just complain, I did something that can help people evaluate
> the differences.

Thank you!

[...]

Date: Mon, 11 Dec 2006 16:28:25 -0500
From: "Forrest J. Cavalier III" <[email-masked]>
To: Harald Alvestrand <[email-masked]>
CC: [email-masked]
Subject: Re: Decision time: versions of -usepro
Sender: [email-masked]

Harald Alvestrand wrote:

 > It's time to make a decision on which one the group wishes to build upon.
 > I'll give three alternatives:

Having looked at the comparisons I generated today, I prefer this WG proceed 
with draft-allbery-usefor-usepro-00.

Date: Mon, 11 Dec 2006 16:24:30 -0500
From: "Forrest J. Cavalier III" <[email-masked]>
To: [email-masked]
Subject: Re: Decision time: versions of -usepro
Sender: [email-masked]

Harald Alvestrand wrote:
> OK, folks have had a little time to look at the two USEPRO versions:

December is a busy time to initiate an important poll. But rather
than just complain, I did something that can help people evaluate
the differences.

1. I took the TOC from usepro-06 and allbery-00 and matched up the
sections as best I could.  (See below.)

2. I split the documents into individual files, one per section.
(For allbery-00, I had to depaginate first, which consisted of
removing 5 lines per page, using a perl script.)

3. Then I ran a script which cat'ed the usepro-06 section(s) and
to a file, then the allbery-00 section(s) to a second file.  Then I
ran a diff, and appended the result to a file.  The result is linked
below.

A little rough, but you can see the differences almost side by side,
at least for the purposes of choosing.  It may or may not be useful
for sparking discussion.

http://www.mibsoftware.com/userkt/usefor/diff-usepro06-allbery00.txt

Date: Mon, 11 Dec 2006 09:01:07 +0100
From: Harald Alvestrand <[email-masked]>
To: [email-masked]
Subject: Decision time: versions of -usepro
Sender: [email-masked]

OK, folks have had a little time to look at the two USEPRO versions:

draft-ietf-usefor-usepro-06.txt
draft-allbery-usefor-usepro-00.txt

It's time to make a decision on which one the group wishes to build upon.
I'll give three alternatives:

1) draft-ietf-usefor-usepro-06
2) draft-allbery-usefor-usepro-00
3) I have another opinion, which is.....

Please - only respond if you have read both documents!
As usual, send mail to the chairs if you don't want to clutter up the 
list, but the tally (if needed) will be by name.

                       Harald

Date: Sun, 10 Dec 2006 21:09:57 +0100
From: Harald Alvestrand <[email-masked]>
To: "Forrest J. Cavalier III" <[email-masked]>
Cc: [email-masked], [email-masked], Alexey Melnikov <[email-masked]>
Subject: Re: Responses from the USEFOR WG Chairs on IESG DISCUSSes
Sender: [email-masked]

--On 8. desember 2006 10:24 -0500 "Forrest J. Cavalier III" 
<[email-masked]> wrote:

>
> Harald Alvestrand wrote:
>> Greetings,
>>
>> we (the chairs) hvave discussed the IESG feedback on
>> draft-ietf-usefor-usefor.
>>
>> There are two DISCUSS comments that we'd like to discuss further on this
>> document.
>>
>> Sam Hartman filed this:
>>
>>
>>> First, the reference to draft-ietf-usefor-usepro needs to be
>>> normative.  I don't think you can construct an article without
>>> following advice related to path, injection-*, and control from that
>>> document.  Parsing these fields without usepro is similarly difficult.
>>
>>
>> The WG had a debate specifically about whether there should be a
>> normative reference to USEPRO, and decided that there should not be one.
>> The WG's belief is that the -usefor document contains enough information
>> to write software that parses an Usenet article and says whether or not
>> it conforms to the format.
>
> 1. Can we clarify how the "WG decided there should not be" a normative
> reference?

Note: Every single version of the -usefor document since -00 has had USEPRO 
as an informative reference.

WG Last Call on USEFOR, reported June 15:

<http://www.imc.org/ietf-usefor/mail-archive/msg03158.html>

Note the second question - on whether or not to send it to the IESG.
I see that at this time, I assumed that there would be a REF hold on USEPRO 
(which would have been the case had the reference been normative).

Second WG Last Call on USEFOR, reported Sept 15:

<http://www.imc.org/ietf-usefor/mail-archive/msg03431.html>

During this Last Call, we had an explicit discussion of this issue starting 
September 9 of this year, subject line "Is the USEFOR->USEPRO dependency 
normative?". I did not see any consensus of the WG that there should be a 
change, but very few people spoke up at all at this time.

I have not been able to find a previous discussion of the issue using the 
keywords "USEFOR" and "normative".

                 Harald

From: [email-masked] (Ralph Babel)
To: [email-masked], [email-masked]
Subject: Re: Responses from the USEFOR WG Chairs on IESG DISCUSSes
Date: Sat,  9 Dec 2006 12:27:39 +0100 (CET)
Sender: [email-masked]

Forrest J. Cavalier III wrote:

> 1. Can we clarify how the "WG decided there
>    should not be" a normative reference?
>
> I and the USEPRO editor, two of 7 participants in the
> last call poll, argued that USEFOR must wait until USEPRO.

The way I read <[email-masked]> ...

  http://www.imc.org/ietf-usefor/mail-archive/msg03158.html

..., you can probably add Richard and me.

To: [email-masked]
Xref: clerew local.usefor:23840
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Strawman consensus call, mvgroup control message
Date: Fri, 8 Dec 2006 17:22:05 GMT
Lines: 50
Sender: [email-masked]

In <[email-masked]> Harald Alvestrand <[email-masked]> writes:

>As far as I can tell from the list traffic:

>- Charles supports the inclusion of a "mvgroup" control message in 
>-usefor-usepro
>- All other speakers do not support this inclusion
>- Nobody has any big objection to a separate experimental document 
>specifying such a message, despite the fact that several people doubt 
>that it will be useful or adopted.

I think the discussion has been useful in highlighting the implementation
problems and the critical issues.

Essentially, it can be implemented two ways:

1. Keep both groups in existence during the overlap period.

2. Alias the old to the new (or just possibly vice versa). That has the
benefit that users need to be subscribed to only one of them (preferably
the new), but it does require a liberal interpretation of 'aliassing'. It
seems that the way aliassing is currently done in INN makes it extremely
difficult to implement that way, though it works fine in CNEWS. We have no
reports of how other servers would be affected.

There are some details concerning what you put in the Xrefs header that
need attention. But the worst that happens if you get that wrong is that
users may see the same article in both groups instead of just the first
one they read it in (but maybe only for for articles posted before the
mvgroup). That seems fixable.

The real crunch is whether User Agents need to be upgraded. The intention
of the proposal as in the present USEPRO was that it would work with
existing User Agents, with the proviso that users would need to subscribe
manually to the new group and maybe unsubscribe from the old, and that
plenty of announcements, discussion and advice would be posted in the
group(s) to be sure they were aware of this. If the WG regards automatic
resubscription within User Agents as an essential feature, then that
would be the end of the matter.

Date: Fri, 08 Dec 2006 18:32:51 +0000
From: Alexey Melnikov <[email-masked]>
To: "Forrest J. Cavalier III" <[email-masked]>
CC: Harald Alvestrand <[email-masked]>, [email-masked], [email-masked]
Subject: Re: Responses from the USEFOR WG Chairs on IESG DISCUSSes
Sender: [email-masked]

Forrest J. Cavalier III wrote:

> Harald Alvestrand wrote:
>
>> Greetings,
>>
>> we (the chairs) hvave discussed the IESG feedback on 
>> draft-ietf-usefor-usefor.
>>
>> There are two DISCUSS comments that we'd like to discuss further on this
>> document.
>>
>> Sam Hartman filed this:
>>
>>> First, the reference to draft-ietf-usefor-usepro needs to be
>>> normative.  I don't think you can construct an article without
>>> following advice related to path, injection-*, and control from that
>>> document.  Parsing these fields without usepro is similarly difficult.
>>
>> The WG had a debate specifically about whether there should be a 
>> normative reference to USEPRO, and decided that there should not be one.
>> The WG's belief is that the -usefor document contains enough 
>> information to write software that parses an Usenet article and says 
>> whether or not it conforms to the format.
>
> 1. Can we clarify how the "WG decided there should not be" a normative
> reference?
>
> I and the USEPRO editor, two of 7 participants in the last call poll, 
> argued that USEFOR must wait until USEPRO.

"Must wait until USEPRO" is not the same thing as "USEFOR must have a 
normative reference to USEPRO".
There is a consensus to delay publication of USEFOR until after USEPRO 
is done (if it ever gets done). The "if it ever gets done" is an 
important part.

> My objection was more based on
> the observation that this WG was not likely to get the format settled
> correctly until USEPRO was close to being finished.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: Proposing a new editor for the USEPRO document
Organization: The Eyrie
Date: Fri, 08 Dec 2006 09:37:15 -0800
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:
> Russ Allbery <[email-masked]> writes:

>> Well, I as a news administrator don't carry cn.*, so I personally
>> haven't dealt with it one way or the other.  The code that maintains
>> the ftp.isc.org list has a more specific permissions configuration than
>> news servers often enforce that prevents cn.bbs.* checkgroups from
>> affecting anything outside of cn.bbs.* even if they're interpreted
>> protocol-wise as doing so.

> Ah! So you effectively have a private hack in the ftp.isc.org code that
> does essentally what the checkscope parameter is trying to do. And it
> would seem that anyone who currently does carry cn.* must have
> implemented a similar hack also, or else he ignores the cn.bbs.*
> checkgroups.

> So it checkscope remains in the draft, and cn.bbs.* can be persuaded to
> include it, then all the people currently using such a hack could take
> it out and implement checkscope instead.

Well, no, I have a real authorization check in the ftp.isc.org code that
says that the people authorized to send checkgroups for cn.bbs.* aren't
allowed to affect the rest of cn.* *even if they claim to want to*, which
isn't replacable by the chkscope parameter.

But otherwise, yes, I see what you're getting at and it's fairly
convincing to me.

To: [email-masked]
Xref: clerew local.usefor:23837
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: The mvgroup control message
Date: Fri, 8 Dec 2006 16:29:10 GMT
Lines: 31
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>Charles Lindsey <[email-masked]> writes:

>> Yes, full fledged servers normally retain articles from removed groups
>> until they have all expired by natural means.

>Eh?  INN certainly doesn't.  I noticed that in the draft as well and was
>very perplexed by it (and took it out of my version, since I've not heard
>of a news server doing that).

CNEWS has been doing it for years (and BNEWS before it, I imagine). I am
surprised that INN doesn't do it, and would recommend that it did (it is
not hard). And for our draft it should be a SHOULD. It could be very handy
if a newsgroup gets removed by accident.

>Once the group is removed, it's removed.  You can't select it, read it,
>look at it, anything.

Naturally, you disable posting, and even ihaving.

To: [email-masked]
Xref: clerew local.usefor:23835
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Proposing a new editor for the USEPRO document
Date: Fri, 8 Dec 2006 15:12:46 GMT
Lines: 28
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>Well, I as a news administrator don't carry cn.*, so I personally haven't
>dealt with it one way or the other.  The code that maintains the
>ftp.isc.org list has a more specific permissions configuration than news
>servers often enforce that prevents cn.bbs.* checkgroups from affecting
>anything outside of cn.bbs.* even if they're interpreted protocol-wise as
>doing so.

Ah! So you effectively have a private hack in the ftp.isc.org code that
does essentally what the checkscope parameter is trying to do. And it
would seem that anyone who currently does carry cn.* must have implemented
a similar hack also, or else he ignores the cn.bbs.* checkgroups.

So it checkscope remains in the draft, and cn.bbs.* can be persuaded to
include it, then all the people currently using such a hack could take it
out and implement checkscope instead.

To: [email-masked]
Xref: clerew local.usefor:23836
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Proposing a new editor for the USEPRO document
Date: Fri, 8 Dec 2006 15:20:34 GMT
Lines: 34
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>Charles Lindsey <[email-masked]> writes:
>> Russ Allbery <[email-masked]> writes:

>> Of more concern to me is the behaviour of NNTP agents which do not send
>> control messages to users who ask for messages in some particular group.

>That's the way that all widely used news servers behave and there's years
>of existing practice behind that.  I'm going to take a lot of convincing
>to recommend something else.

Sure. It is too entrenched to change now. I was bemoaning that it ever got
like that (as was Frank, I think).

>That's far and away the lesser of two evils, IMO.

And there I disagree with you. I think if I were doing things again from
scratch I would have the control messages visible in their own groups,
with an implict crosspost to the control.* groups as well. And then have a
Distribution 'control' for the benefit of people who did not want to see
them. But that is not a serious proposal for now (though it might be if
ever we try to standardize NOCEMs).

To: [email-masked]
Xref: clerew local.usefor:23839
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: ASCII
Date: Fri, 8 Dec 2006 17:08:16 GMT
Lines: 47
Sender: [email-masked]

>>> Anyone have a suggestion on how one words "MUST have ASCII
>>> as a subset" in fairly precise language?

>> | A charset is said to be "ASCII compatible" if it encodes 
>> | the Unicode points u+0000 up to u+007F as the usual octets
>> | &x00 up to &x7F, and either uses these octets for no other
>> | purpose (e.g. Latin-1, windows-1252, and UTF-8), or offers
>> | clear indications when they are used as (a part of) other
>> | encodings (e.g. UTF-7, SCSU, and TBD).

I don't think you ought to be mentioning Unicode when defining "MUST have
ASCII as a subset". What you need is a code wherein the octets 0-127
always stand for the usual ASCII characters. UTF-8 and the ISO-8859 series
are the obvious examples. I think somebody said the EUC ones were also.

>We do need to support SJIS, though.  I thought that some of the Asian
>encodings like that do shift between ASCII and non-ASCII with escape
>sequences and use ASCII characters within the escaped portion.  But I'm
>not sure.

I don't think you can support anything using escapes, though.

Essentially, when a 'checkgroups' arrives, you take each line, delete the
1st TAB and everything after (but take note of any "(Moderated)" at the
end. Then you compare what you have got with what is in your active file,
and inform the newsadmin of any discrepancies, even offering to fix them
for him. That needs the newsgroup-names to be exactly the same as stored
in the active file. I doubt anyone using strange charsets in a checkgroups
has done anything that would break that behaviour, which is built into
most servers.

To: [email-masked]
Xref: clerew local.usefor:23838
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: The mvgroup control message
Date: Fri, 8 Dec 2006 16:57:55 GMT
Lines: 76
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>Charles Lindsey <[email-masked]> writes:

>Except at least with INN, you can't post to an aliased group (which makes
>sense and seems like the right semantics for aliasing in general to me),
>so if the new group is aliased to the old group, users won't be able to
>post except by going into the old group and posting there.  That seems
>less than ideal.  ...

Yes, it is clear that the aliassing in INN is much more restrictive than
in CNEWS, making full implementation of mvgroup much harder.

>>> For example, unless you rewrite the Newsgroups header of the old
>>> articles or add a Followup-To header...

>> Absolutely not! Otherwise threads are going to get hopelessly confused,
>> and you break the fundamental rule of Usenet ...

>So instead you create a bunch of messages that can't be replied to without
>a bunch of manual fiddling with the headers, which sparks user confusion.

Not at all. Both groups MUST remain postable during the overlap, or else
there is indeed utter confusion. Keeping both groups in existence is one
way to do it, but an alias system that continues to allow posting is
better, on systems where it can be implemented that way.

>> So once a thread starts, it remains in its original group (unless
>> followups are manually changed). Some users may believe they are reading
>> it in newgroup, and some in oldgroup (and may be slightly confused if
>> they inspect the Newsgroup header too closely). But whichever is the
>> case, if you followup it will appear to you to be in your group and to
>> other Usenet users in whichever group their own server has put it in.

>I don't understand what you mean here.  If you follow up, you'll copy the
>Newsgroups header field from the predecessor into the Newsgroups header
>field of your follow-up, and then posting that message will fail because
>you're trying to post to an aliased group.  ...

Only on INN, which is why INN cannot implement it that way.

>> Yes it does, because at the least it gives you the same as the previous
>> system, but with the added benefit that newsadmins are more likely to
>> accept both halves of the change if it comes as one command,

>I don't think I believe this.  If news administrators are acting on
>control messages, there's no reason why they'd act on the newgroup and not
>the rmgroup unless they specifically configured their server that way,

But sadly it seems a lot of them ARE configured that way. Quite illogical,
but since when have news admins ever behaved logically :-( ?

>> Old articles get preserved either way, just that you may have to examine
>> both groups to find them if your server has done the minimal
>> implementation.

>If you implement mvgroup as newgroup and rmgroup, old messages are not
>preserved.

No problem it there is a suitable delay between the newgroup and the
rmgroup (as USEPRO required), so that most oldgroup stuff will have
expired by then (and on CNEWS will aftually be kept until the normal
expiry time anyway).

Date: Fri, 08 Dec 2006 10:24:23 -0500
From: "Forrest J. Cavalier III" <[email-masked]>
To: Harald Alvestrand <[email-masked]>
CC: [email-masked], [email-masked], Alexey Melnikov <[email-masked]>
Subject: Re: Responses from the USEFOR WG Chairs on IESG DISCUSSes
Sender: [email-masked]

Harald Alvestrand wrote:
> Greetings,
> 
> we (the chairs) hvave discussed the IESG feedback on 
> draft-ietf-usefor-usefor.
> 
> There are two DISCUSS comments that we'd like to discuss further on this
> document.
> 
> Sam Hartman filed this:
> 
> 
>>First, the reference to draft-ietf-usefor-usepro needs to be
>>normative.  I don't think you can construct an article without
>>following advice related to path, injection-*, and control from that
>>document.  Parsing these fields without usepro is similarly difficult.
> 
> 
> The WG had a debate specifically about whether there should be a 
> normative reference to USEPRO, and decided that there should not be one.
> The WG's belief is that the -usefor document contains enough information 
> to write software that parses an Usenet article and says whether or not 
> it conforms to the format.

1. Can we clarify how the "WG decided there should not be" a normative
reference?

I and the USEPRO editor, two of 7 participants in the last call poll, argued 
that USEFOR must wait until USEPRO.  My objection was more based on
the observation that this WG was not likely to get the format settled
correctly until USEPRO was close to being finished.

While I cannot speak for the other participants in the poll, most of the
desire seemed to be that "We want a document, some document, out
to IETF last call" and not "USEFOR is set, we have confidence that we
can write companion documents with this."

The current INN maintainer has spent intense hours reviewing the draft
since the poll was taken and is now on record in 
<[email-masked]>, Dec 1, with
     "There are several [sections] that seem rather normative to me having just
     gone through all of USEPRO, apart from the security considerations."

2. I agree with the sentiment that you cannot construct a useful article without
what will be in USEPRO.  You can construct a conforming article, however.  Are
Internet Message RFCs much different in this respect?

3. "Parsing" is an ambiguous term.  USEFOR is complete enough that you can
determine compliance of an article.  So it can be "parsed" in that sense.
But you cannot determine that the article was constructed following USEPRO.

I think this creates an attractive nuisance problem that I don't recall being 
discussed in the WG....

Will releasing USEFOR, without the advice from USEPRO, lead to people following
the syntax, but doing their own thing, e.g. with path diagnostics, leading
to incompatibilities in the wild?  This seems like a real danger to me.

As expected, there was much WG discussion about backwards compatibility with 
some of these features, and there were some areas with very limited flexibility.

If people usurp those methods, creating USEFOR-compliant, but non-interoperating 
implementations due to the lack of guidance of USEPRO (or some early draft of 
USEPRO), then there are not going to be easy alternatives.

Do others think this is a danger?

Date: Fri, 08 Dec 2006 13:47:15 +0100
From: Harald Alvestrand <[email-masked]>
To: [email-masked]
Cc: [email-masked], Alexey Melnikov <[email-masked]>
Subject: Responses from the USEFOR WG Chairs on IESG DISCUSSes
Sender: [email-masked]

Greetings,

we (the chairs) hvave discussed the IESG feedback on 
draft-ietf-usefor-usefor.

There are two DISCUSS comments that we'd like to discuss further on this
document.

Sam Hartman filed this:

> First, the reference to draft-ietf-usefor-usepro needs to be
> normative.  I don't think you can construct an article without
> following advice related to path, injection-*, and control from that
> document.  Parsing these fields without usepro is similarly difficult.

The WG had a debate specifically about whether there should be a 
normative reference to USEPRO, and decided that there should not be one.
The WG's belief is that the -usefor document contains enough information 
to write software that parses an Usenet article and says whether or not 
it conforms to the format.

We believe that the construction of the content of those fields is not a
normative reference for the format - any more than the RIR allocation 
rules for IP addresses forms a normative reference for the definition of 
the IP header.

If there are specific instances of parsing an Usenet message (as opposed 
to acting on the content of a message) that you think is difficult 
without USEPRO, we'd like to have the details, so that we can discuss 
the specifics.

> We received last call comments about the subsetting of RFC 2822
> messages.  Very late, we realized that the real issue is not that
> these requirements are difficult for new posting agents, but that they
> are difficult when a email message is injected into usenet.  First, I
> think more discussion of whether that is the right approach for
> gateways is needed.

It is definitely not the right approach.
A normal email message will not be a valid Usenet message - if only 
because it lacks a Path: header and a Newsgroups: header.

The WG has discussed the problem of email-to-news gateways, and made an
explicit decision in at least one other case to exclude wording that 
would seem to be normative for such gateways. We believe that this 
problem is out of scope for this document, and is (as we read the 
charter) out of scope for the present charter of the USEFOR WG.

However, we have received a proposal for clarifying text about the
relationship between the USEFOR format and the mail format; we'll 
consider this for inclusion at the end of section 2.1 of the draft:

>This draft places additional restrictions on RFC2822 message format rules,
>in order to be compatible with deployed NetNews agents.
>An entity that generates messages compliant with RFC2822 might not always
>generate messages compliant with this specification, although certainly a
>single code-base can generate messages according to both sets of rules if
 >desired.

>Gatewaying between email and news is not part of this specification,
 >but we note that such a gateway would in all cases have to do more than
 >just copying the message. Ensuring that the message satisfies the
 >constraints here would be part of the job of such a gateway.

Would this satisfy the DISCUSSant on this point?

Back to the DISCUSS text:

>  Second, regardless of what we say here, it will
> be reasonably common for messages to be injected that do not meet
> these requirements.  In the interests of interoperability we need to
> discuss what problems can result and make recommendations for liberal
> receivers that minimize interoperability problems.

The WG discussed the problem of illegal messages and decided that it was
not possible or reasonable to specify handling of messages that do not
conform to this syntax in the syntax document. This would also violate
long-standing practice in, for instance, RFC 2822.
There is a document on the WG's document list - USEAGE - where such
considerations would be appropriate. (However, note that the state of 
the WG is not such that this document will be finished soon, if at all. 
See later note.)

>   The Security Considerations say:
>   >
>   > Further security considerations are discussed in
>   > [I-D.ietf-usefor-usepro].
>   >
>   There is some really good information in this document.  Can we make
>   sure that it gets published?  I would like to see an RFC referenced,
>   not an Internet-Draft, when this document becomes an RFC.

This is one of multiple references that really reflect the fact that you
can't make a working Usenet system with just the Usenet article format.

As discussed above, the WG decided against making the reference 
normative - one purpose of this was to make it possible to publish the 
USEFOR document without waiting for USEPRO.

If there are specific concerns for USEPRO that you wish moved into 
USEFOR, we can discuss that - but making the publication of USEFOR 
depend on the publication of USEPRO would definitely be overriding a 
consensus of the working group.

About the state of the working group:
This group is very low energy. The number of active participants is
probably less than 10, and the number of constructive active 
participants less than that. A few tens of people follow the mailing 
list and occasionally make comments.

The process of getting from a draft that was "nearly finished" for 
USEFOR to the present state took 1.5 years; the total process since the 
start of the working group has been more than 9 years.

We have just started a process of deciding if a change of editors for
USEPRO is warranted; once that is settled, the process of making USEPRO
ready for publication could easily take another year. There is also a 
grave danger that the last few constructive active participants will 
just give up and go away, leaving us with no finished USEPRO document at 
all. USEAGE is in even greater danger of that happening, since it is 
later in the WG's schedule, and depends on USEPRO finishing.

In this context, we would very much like to prove to the WG that it is
possible to finish a work item and get it published. We need all the
encouragement we can get.

              Harald and Alexey

Date: Fri, 08 Dec 2006 05:17:53 +0100
From: Harald Alvestrand <[email-masked]>
To: Frank Ellermann <[email-masked]>, [email-masked]
Subject: Re: ASCII (was: Proposing a new editor for the USEPRO document)
Sender: [email-masked]

Strawman, idea. Disregard if nonsense.
Perhaps we should define the operation we want to succeed instead?

For instance, say that:

There is news software that will ignore the charset parameter and parse the 
message as if it was US-ASCII.

Therefore, the result of reading each octet of the message and setting bit 
8 to zero MUST be a valid message specifying the same newsgroup names, as 
long as all newsgroup names are in ASCII. There is no requirement that the 
newsgroup descriptions survive this treatment.

Examples of charsets that allow the message to satisfy this constraint are 
(for example) UTF-8, ISO-8859-1, Shift-JIS and ISO-2022-JP.

Examples of charsets that do not allow the message to satisfy this 
constraint are EBCDIC, UTF-16 and ISO-646-NO.

--On 7. desember 2006 22:16 +0100 Frank Ellermann 
<[email-masked]> wrote:

>
> Russ Allbery wrote:
>
>> Anyone have a suggestion on how one words "MUST have ASCII
>> as a subset" in fairly precise language?
>
>| A charset is said to be "ASCII compatible" if it encodes
>| the Unicode points u+0000 up to u+007F as the usual octets
>| &x00 up to &x7F, and either uses these octets for no other
>| purpose (e.g. Latin-1, windows-1252, and UTF-8), or offers
>| clear indications when they are used as (a part of) other
>| encodings (e.g. UTF-7, SCSU, and TBD).
>
> TBD needs to be some CJK charset.  I'm almost sure that you
> don't want SCSU, UTF-1, or UTF-7, and that could simplify it:
>
>| A charset is said to contain US-ASCII as subset if it encodes
>| the Unicode points u+0000 up to u+007F as the usual octets
>| &x00 up to &x7F, and uses these octets for no other purpose.
>
> Frank
>
>
>

Cc: [email-masked]
Date: Thu, 07 Dec 2006 18:21:28 -0800 (PST)
From: Ned Freed <[email-masked]>
Subject: Re: ASCII
To: Russ Allbery <[email-masked]>
Sender: [email-masked]

> Ned Freed <[email-masked]> writes:

> > However, all of these charsets are ASCII-compatible. Examples of
> > charsets that aren't ASCII-compatible are any UTF-16 variant, any of the
> > EBCDIC charsets, and the old national variant ASCII sets.

> > (What can I say, I've written a lot of charset conversion code.)

> I think that ASCII-compatible is what we actually need.  Basically, we
> have to be assured that the newsgroup *names*, as opposed to the
> descriptions, if read by a naive news server with no knowledge of
> character sets, will still be correct.  As long as that condition is met,
> it doesn't matter what weird stuff is going on in the descriptions as long
> as the character set is properly tagged.

> Maybe we should just say that rather than taking about the character set
> properties.

Seems reasonable - and I can't say that having a protocol specific discussion
of charset terminology and characteristics thrills me.

				Ned

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: ASCII
Organization: The Eyrie
Date: Thu, 07 Dec 2006 18:07:12 -0800
Sender: [email-masked]

Ned Freed <[email-masked]> writes:

> However, all of these charsets are ASCII-compatible. Examples of
> charsets that aren't ASCII-compatible are any UTF-16 variant, any of the
> EBCDIC charsets, and the old national variant ASCII sets.

> (What can I say, I've written a lot of charset conversion code.)

I think that ASCII-compatible is what we actually need.  Basically, we
have to be assured that the newsgroup *names*, as opposed to the
descriptions, if read by a naive news server with no knowledge of
character sets, will still be correct.  As long as that condition is met,
it doesn't matter what weird stuff is going on in the descriptions as long
as the character set is properly tagged.

Maybe we should just say that rather than taking about the character set
properties.

Cc: [email-masked]
Date: Thu, 07 Dec 2006 17:18:22 -0800 (PST)
From: Ned Freed <[email-masked]>
Subject: Re: ASCII
To: Russ Allbery <[email-masked]>
Sender: [email-masked]

> >> Anyone have a suggestion on how one words "MUST have ASCII
> >> as a subset" in fairly precise language?

> > | A charset is said to be "ASCII compatible" if it encodes
> > | the Unicode points u+0000 up to u+007F as the usual octets
> > | &x00 up to &x7F, and either uses these octets for no other
> > | purpose (e.g. Latin-1, windows-1252, and UTF-8), or offers
> > | clear indications when they are used as (a part of) other
> > | encodings (e.g. UTF-7, SCSU, and TBD).

> > TBD needs to be some CJK charset.  I'm almost sure that you
> > don't want SCSU, UTF-1, or UTF-7, and that could simplify it:

How about EUC-JP as an example?

> > | A charset is said to contain US-ASCII as subset if it encodes
> > | the Unicode points u+0000 up to u+007F as the usual octets
> > | &x00 up to &x7F, and uses these octets for no other purpose.

> We do need to support SJIS, though.

Shift-JIS charsets (there are several different ones) do not in general have
ASCII as a subset because there are two-octet sequences in them where the first
octet is >127 but the second is <127.

If memory serves, the various EUC charsets all have ASCII as a subset.

> I thought that some of the Asian
> encodings like that do shift between ASCII and non-ASCII with escape
> sequences and use ASCII characters within the escaped portion.  But I'm
> not sure.

Now you're talking about the various charsets based on ISO 2022. Most if not of
these are 7bit and use various escape sequences to switch between single byte
US-ASCII and some other large coded character set encoded as two successive
US-ASCII characters. These clearly don't have US-ASCII as a subset.

However, all of these charsets are ASCII-compatible. Examples of charsets that
aren't ASCII-compatible are any UTF-16 variant, any of the EBCDIC charsets,
and the old national variant ASCII sets.

(What can I say, I've written a lot of charset conversion code.)

				Ned

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: ASCII
Date:  Thu, 07 Dec 2006 23:31:51 +0100
Organization:  <URL:http://purl.net/xyzzy>
Lines: 13
Sender: [email-masked]

Russ Allbery wrote:
 
>>| A charset is said to contain US-ASCII as subset if it encodes
>>| the Unicode points u+0000 up to u+007F as the usual octets
>>| &x00 up to &x7F, and uses these octets for no other purpose.
 
> We do need to support SJIS, though.

Tricky, maybe "and allows no other encoding for these code points".
That should eliminate UTF-7.  If SJIS is in, then UTF-1 is also in.

Frank

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: ASCII
Organization: The Eyrie
Date: Thu, 07 Dec 2006 13:46:03 -0800
Sender: [email-masked]

>> Anyone have a suggestion on how one words "MUST have ASCII
>> as a subset" in fairly precise language?

> | A charset is said to be "ASCII compatible" if it encodes 
> | the Unicode points u+0000 up to u+007F as the usual octets
> | &x00 up to &x7F, and either uses these octets for no other
> | purpose (e.g. Latin-1, windows-1252, and UTF-8), or offers
> | clear indications when they are used as (a part of) other
> | encodings (e.g. UTF-7, SCSU, and TBD).

> TBD needs to be some CJK charset.  I'm almost sure that you
> don't want SCSU, UTF-1, or UTF-7, and that could simplify it:

> | A charset is said to contain US-ASCII as subset if it encodes
> | the Unicode points u+0000 up to u+007F as the usual octets
> | &x00 up to &x7F, and uses these octets for no other purpose.

We do need to support SJIS, though.  I thought that some of the Asian
encodings like that do shift between ASCII and non-ASCII with escape
sequences and use ASCII characters within the escaped portion.  But I'm
not sure.

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  ASCII (was: Proposing a new editor for the USEPRO document)
Date:  Thu, 07 Dec 2006 22:16:39 +0100
Organization:  <URL:http://purl.net/xyzzy>
Lines: 21
Sender: [email-masked]

Russ Allbery wrote:

> Anyone have a suggestion on how one words "MUST have ASCII
> as a subset" in fairly precise language?

| A charset is said to be "ASCII compatible" if it encodes 
| the Unicode points u+0000 up to u+007F as the usual octets
| &x00 up to &x7F, and either uses these octets for no other
| purpose (e.g. Latin-1, windows-1252, and UTF-8), or offers
| clear indications when they are used as (a part of) other
| encodings (e.g. UTF-7, SCSU, and TBD).

TBD needs to be some CJK charset.  I'm almost sure that you
don't want SCSU, UTF-1, or UTF-7, and that could simplify it:

| A charset is said to contain US-ASCII as subset if it encodes
| the Unicode points u+0000 up to u+007F as the usual octets
| &x00 up to &x7F, and uses these octets for no other purpose.

Frank

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: draft-allbery-usefor-usepro-00 submitted
Organization: The Eyrie
Date: Thu, 07 Dec 2006 10:35:40 -0800
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:
> Russ Allbery <[email-masked]> writes:

>> * I tried to remove all references to policies, hierarchy administrators,
>>   and other terms for the organization of Usenet.  This is partly because
>>   I felt they were out of scope for the protocol and partly because I
>>   tried to make the document general to any Netnews network and avoid any
>>   statements specific or unique to Usenet.  Such matters can be discussed
>>   in USEAGE.

> May I just point out that the name of the WG includes the word "Usenet",
> so defining how Usenet works is hardly off topic.

For USEAGE.  Not in my opinion, for the protocol, which is not specific to
Usenet.  As I say, this is one of those things that people can look at and
see which direction they prefer.

> And private implementations of Netnews need administration just as much
> as the public one. For sure, hierarchy administrators do exist (you and
> I are, or have been, both involved in that) and Usenet would collapse
> without them. The only oddity is that the method of establishing them is
> distinctly murky, but in practice works remarkably well. And for sure,
> the protocols contain features that assume they exist. Which is all
> USEPRO says, if you actually read it carefully.

Well, I dropped all of those references and didn't run into any problems
with features that assumed they existed, so at the protocol level I don't
agree.  The software and protocol are largely agnostic to how those things
are structured, which is what enables hierarchies to have so many very
different policies.

>> I'm not happy with the readability of the instructions for Path
>> construction.  Charles has a new version in his latest draft that may
>> be much better.  It may be even better to pull all Path discussion into
>> a separate section of its own (or a subsection of duties) and just
>> reference it from the duties sections of the specific agents.

> My new version had been published on this List (twice).

Yes, and I've looked at it, and I'm still trying to decide what way feels
the best to me.  I had a limited amount of time to get the draft finished,
and this was one of the things that I didn't spend a lot of time on.  I
think we're all agreed, or nearly agreed, on what agents should do with
the Path header and it's just a matter of finding good wording, which
makes it less important to hash out immediately than the places where
there's disagreement.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: Proposing a new editor for the USEPRO document
Organization: The Eyrie
Date: Thu, 07 Dec 2006 10:29:50 -0800
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:
> Russ Allbery <[email-masked]> writes:

>> The point that I hadn't realized before is that defining systems that
>> are trying to do reinjection as gateways not only makes logical sense
>> (they're gatewaying between two disconnected Netnews networks), ....

> I can imagine situations in which reinjection is nothing like gatewaying,
> but I need to see your draft before commenting further.

Well, one of the reasons why I put it up on my web site was so that people
could look at it before the draft editor got around to publishing it.

>>> It was in application/newstransmission because Henry Spencer put it
>>> there when he registered it.

>> Ah, that explains it.

> Yes, if we remove it from there, then it needs an explicit mention that it
> is a change from the current news/transmission definition. I would have
> no great problem with doing that.

Okay.  That's the direction I'm leaning at the moment.

> And in that case, the batch format is only relevant for describing the
> response to the 'sendme' control message

Well, in my opinion, the batch format is not at all relevant to describing
the response to the sendme control message.  It's outside the scope of our
document to specify how articles are transmitted from one system to
another, and transmitting them in response to sendme is no exception.
Whether they're then sent by NNTP, by UUCP with batches, by magnetic tape,
or by someone walking next door with a floppy disk isn't part of the
sendme specification.  The control mesages are just there to arrange what
should be sent, not how.

That's one of the reasons why I took it out; the only thing in my draft
that referenced it was the application/news-transmission registration.

> (and I hope you are not going to remove mention of the 'ihave' and
> 'sendme' control messages;

No.  I argued for keeping them.

> Also, the ghastly 'to' hack in newsgroup-names needs to be mentioned, if
> only because we mention it in Usefor and refer to USEPRO for the gory
> details.

It's there in all the specificity that I've ever heard for it.  There
really isn't that much to say about it.

>> ...  From the perspective of our draft, it's a very odd sideline that
>> adds about a page of text for something that essentially no one is
>> going to care about,...

> Eh? The batch format is described in 16 lines (though it could do with a
> mention of compression).

Plus additional wording in application/news-transmission.  It's not quite
a page, but it saved a noticable chunk of space.

> I think the least you need to say is that, to be compatible with present
> software, any charset specified here MUST have ASCII as a subset (no
> EBCDIC!). And maybe should/SHOULD be UTF-8 (with mention of NNTP as the
> reason).

> With that restriction, I think a charset parameter might now work.

Hm.  Yeah, that makes sense, and still allows for what's going on in
practice today.  Anyone have a suggestion on how one words "MUST have
ASCII as a subset" in fairly precise language?

>>> I have no real problem with that. But it is introducing a
>>> *new*feature*, horror of horrors :-(, and it is dealing with matters
>>> which are not seen "on the wire" - even greater horrors.

>> This isn't a new feature -- it's already implemented by every widely used
>> news server that I'm aware of.

> Sure, but not always in that form. But surely the point is that that
> form should NEVER appear 'on the wire' and thus is better not normative.
> It is really a private matter between servers and their clients
> (though it is a convention that shold be sidely encouraged, hence my
> NOTE). But I am not really arguing against it.

It appears on the wire between reading agents and serving agents, and our
standard covers that protocol.  How serving agents talk to reading agents
is not a private matter.  It's part of what we're standardizing, even if
we don't have a tremendous amount we have to say about it and not many
restrictions we place on it.

> Of more concern to me is the behaviour of NNTP agents which do not send
> control messages to users who ask for messages in some particular group.

That's the way that all widely used news servers behave and there's years
of existing practice behind that.  I'm going to take a lot of convincing
to recommend something else.

> That behaviour is not required even by RFC 3977,

It's not a topic for NNTP; NNTP doesn't dictate how news servers assign
articles to groups.  It's a topic for USEPRO, which is why I addressed it
there.

> but it seems endemic, and leads to the problem that Frank just mentioned
> that you are forced to bring down the complete control.cancel group if
> you want to see actual cancel messages for some reason.

That's far and away the lesser of two evils, IMO.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: The mvgroup control message
Organization: The Eyrie
Date: Thu, 07 Dec 2006 10:16:47 -0800
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:
> Russ Allbery <[email-masked]> writes:

>> However, moving articles between groups after they're stored pose weird
>> protocol concerns that I am not at all convinced the current proposal
>> has adequately dealt with, even if it were widely implemented.

> No, USEPRO make it clear that you cannot expect to move articles around,
> except when the target group is empty (this is not for merging groups).

My concerns definitely still apply when the target group is empty.

> Essentially, you can keep the articles physically in the old group and
> rename the directory concerned, or move them bodily to the newly created
> group. Only one group ever exists on the server, the other is just an
> alias for it (which might happen either way around). The nature of the
> alias (at least on CNEWS, though probably on others) is that whichever
> alias you ask to see the group by (whether for finding old articles or
> storing new ones, it all happens in the one group).

Except at least with INN, you can't post to an aliased group (which makes
sense and seems like the right semantics for aliasing in general to me),
so if the new group is aliased to the old group, users won't be able to
post except by going into the old group and posting there.  That seems
less than ideal.  That means that if you want the articles to be
accessible, you do need to move them from the old group to the new group
and then alias the old group to the new group, since otherwise no one's
going to be able to get at them.

>> For example, unless you rewrite the Newsgroups header of the old
>> articles or add a Followup-To header or followups to articles that were
>> moved from the old group will be sad (if not on the local server then
>> on someone else's server).

> Absolutely not! Otherwise threads are going to get hopelessly confused,
> and you break the fundamental rule of Usenet that an article appears
> exactly the same (modulo Path, Xref, etc) whichever server you find it
> on.

So instead you create a bunch of messages that can't be replied to without
a bunch of manual fiddling with the headers, which sparks user confusion.

I don't like either alternative here.  This is one of the parts of the
mvgroup proposal that bugs me.  Either way you go, the result doesn't seem
much better than just doing an rmgroup and a newgroup.  Different, but not
clearly better.

> So once a thread starts, it remains in its original group (unless
> followups are manually changed). Some users may believe they are reading
> it in newgroup, and some in oldgroup (and may be slightly confused if
> they inspect the Newsgroup header too closely). But whichever is the
> case, if you followup it will appear to you to be in your group and to
> other Usenet users in whichever group their own server has put it in.

I don't understand what you mean here.  If you follow up, you'll copy the
Newsgroups header field from the predecessor into the Newsgroups header
field of your follow-up, and then posting that message will fail because
you're trying to post to an aliased group.  It makes no difference in how
follow-ups are constructed what group your news reader thinks it's reading
(and in fact, if it uses NEWNEWS, it may not think it's reading any
particular group at all).

> In fact, an NNTP client cannot actually tell which of the two his server
> really keeps it in unless it inspects the Active File, or looks closely
> at the Xref.

You can't even tell with Xref unless you allow messages to be rewritten.
(And I don't see how you can get away without rewriting the message to
modify at least the Xref header field on INN.  Having Xref be wrong in a
message in INN causes a ton of problems, particularly with the recovery
tools.  It's like having a file system in an inconsistent state.  It's not
a stable configuration.)

>> Implementing mvgroup as a clone of newgroup, or even a clone of
>> newgroup and a delayed rmgroup, is pointless.  That doesn't solve any
>> of the problems that mvgroup was introduced to solve.  Unless readers
>> are automatically redirected to the new newsgroup and, ideally, the old
>> articles are also preserved, there's no reason to bother with it, since
>> it won't be any better than what's already available now.

> Yes it does, because at the least it gives you the same as the previous
> system, but with the added benefit that newsadmins are more likely to
> accept both halves of the change if it comes as one command,

I don't think I believe this.  If news administrators are acting on
control messages, there's no reason why they'd act on the newgroup and not
the rmgroup unless they specifically configured their server that way, and
if they did because they don't want to remove old groups on renamings,
they wouldn't act on the mvgroup anyway because it does the same thing.
Or they'd treat mvgroup like newgroup.

If news administrators are not acting on control messages (which is the
more common case), none of this matters.

> and it opens the way to more sophisticated implementations as time goes
> by and user pressure starts to demand them.

That sounds *so* much like experimental track to me.

> Old articles get preserved either way, just that you may have to examine
> both groups to find them if your server has done the minimal
> implementation.

If you implement mvgroup as newgroup and rmgroup, old messages are not
preserved.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: Proposing a new editor for the USEPRO document
Organization: The Eyrie
Date: Thu, 07 Dec 2006 09:50:37 -0800
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:
> Russ Allbery <[email-masked]> writes:

>> True, I can see where that would be useful.  I forgot that the default
>> behavior was hierarchy-scoped, not sub-hierarchy-scoped, so yes, the
>> current cn.bbs.*-only checkgroups would remove other cn.* groups, which
>> isn't desirable.

> Ah! I had thought they were sub-hierarchy-scoped (and that's what USEPRO
> says - I have not yet read that far in your draft yet to see what that
> says).  But I see that s-o-1036 agrees with you (though a bit vague) and
> so does the CNews implementation.

INN as well.

> OTOH, if you are correct then how come the cn.bbs.* checkgroups have not
> destroyed the rest of the cn.* hierarchy long ago? Do you, as a
> newsadmin, get a message telling you that you ought to delete umpteen
> groups in cn.* every time a cn.bbs.* checkgroups comes out?

Well, I as a news administrator don't carry cn.*, so I personally haven't
dealt with it one way or the other.  The code that maintains the
ftp.isc.org list has a more specific permissions configuration than news
servers often enforce that prevents cn.bbs.* checkgroups from affecting
anything outside of cn.bbs.* even if they're interpreted protocol-wise as
doing so.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: The mvgroup control message
Organization: The Eyrie
Date: Thu, 07 Dec 2006 09:48:05 -0800
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:

> Yes, full fledged servers normally retain articles from removed groups
> until they have all expired by natural means.

Eh?  INN certainly doesn't.  I noticed that in the draft as well and was
very perplexed by it (and took it out of my version, since I've not heard
of a news server doing that).

Once the group is removed, it's removed.  You can't select it, read it,
look at it, anything.

> That, I think, is the crux of the matter. Can we agree that the only
> people we need to worry about are those who are already subscribed to
> the old group? In that case, they will surely be aware of what is going
> on. It would be a very strange group whose 'culture' did not encompass
> discussions about the status of the group itself.

This is one of these places where I think the Usenet-centric nature of the
discussion has hurt our ability to come up with a good general protocol.
This may be true on Usenet where there's some sort of formal group change
process, and which is only part of what Netnews is used for.  Internal
newsgroups may get moved without any notification in the group, and in
fact I would consider mvgroup a failure unless it supports that.  That's
the whole point, after all -- to make renaming a group a first-class
operation like creating or removing, one that behaves in a natural manner.

Date: Thu, 7 Dec 2006 17:25:05 +0000
To: [email-masked]
From: Richard Clayton <[email-masked]>
Subject: Re: The mvgroup control message
Sender: [email-masked]

In message <[email-masked]>, Charles Lindsey
<[email-masked]> writes

>>the client would allocate
>>articles to groups in the local database by examining the Newsgroups:
>>header field, so it would not be relevant.
>
>So even if your client issued a GROUP command followed by ARTICLE, it
>would still store it under whichever group was in the Newsgroups header?

yes, for consistency with the NEWNEWS case where this is essential

>>If you mean by "in the first case" the special state "when my system
>>somehow becomes aware of a mvgroup" then yes, some interaction with the
>>user will be necessary -- if only to explain that something weird and
>>wonderful has happened
>
>That, I think, is the crux of the matter. Can we agree that the only
>people we need to worry about are those who are already subscribed to the
>old group? In that case, they will surely be aware of what is going on. 

Which is, of course, exactly what happens at the moment, and why the
world has survived into the 21st century without mvgroup and can
continue to do so

[with apologies to the chair for the repetition in the second comment,
the first one was actually contributing something new]

- -- 
richard                                                   Richard Clayton

Those who would give up essential Liberty, to purchase a little temporary 
Safety, deserve neither Liberty nor Safety. Benjamin Franklin 11 Nov 1755

To: [email-masked]
Xref: clerew local.usefor:23820
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: The mvgroup control message
Date: Thu, 7 Dec 2006 16:39:37 GMT
Lines: 108
Sender: [email-masked]

In <[email-masked]> Richard Clayton <[email-masked]> writes:

>In message <[email-masked]>, Charles Lindsey
><[email-masked]> writes

>>I don't see why that complication is necessary. Either the server has 
>>both groups in separate existence during the overlap period, or it has 
>>them aliased to one group. 

>indeed so ... and my system is effectively a server as well, so
>(depending on design decisions) it needs to alias or overlap in some
>way. In order to do that it needs to become aware of the "mvgroup"
>event, and that's what my issue (a) was saying

The whole 'mvgroup' idea was based on the assumption that existing User
Agents would behave in a sensible (i.e. non-mysterious) manner provided
the servers did sensible things (which would require at least some minimal
upgrade to the server to notice the arrival of the 'mvgroup'). If that
assumption is wrong, then indeed reexamination of the idea is in order.
Hence my reason to enquire whether some existing user agents such as
Turnpike would exhibit 'mysterious' behaviour in their present form (of
course, that does not prevent their implementors from adding extra bells
and whistle to deal with 'mvgroup' if they think it useful).

>>What happens at present if a server ...

>>and later removes an old one on your system? 

>my system remains unaware of removals (this is a flaw in NEWGROUPS)
>unless a full LISTGROUP is requested.  In practice if there are articles
>in the local database then the old group remains in existence so that
>those articles can still be accessed ....  only once all the articles
>vanish will the old group name be forgotten about

Yes, full fledged servers normally retain articles from removed groups
until they have all expired by natural means. Posting of new articles is
normally prevented, though. Nothing 'mysterious' so far.

>>Or if it suddenly 
>>aliases two groups together?

>I've never seen a server do that...  but the client would allocate
>articles to groups in the local database by examining the Newsgroups:
>header field, so it would not be relevant.

So even if your client issued a GROUP command followed by ARTICLE, it
would still store it under whichever group was in the Newsgroups header?
In which case, it would behave essentially as if the server was keeping
both groups separately, so the same as in my first scenario.

FWIW, when I created an alias for a group last week, I see that NEWGROUPS
tells me it has appeared, and shows its Active File line.

>>Presumably in the first case, your client tells its readers that a new 
>>group exists and invites them to subscribe. 

>No -- that's just dumb (and yes I know that some Unix systems do that).
>New groups (often with silly or vaguely obscene names) are created all
>the time, so bothering users with trivia is a waste of time.

My 'nn' reader tells me that. But fortunately my local server has strict
instructions not to accept newgroups from alt.*, so I am not bothered :-).

>If you mean by "in the first case" the special state "when my system
>somehow becomes aware of a mvgroup" then yes, some interaction with the
>user will be necessary -- if only to explain that something weird and
>wonderful has happened

That, I think, is the crux of the matter. Can we agree that the only
people we need to worry about are those who are already subscribed to the
old group? In that case, they will surely be aware of what is going on. It
would be a very strange group whose 'culture' did not encompass
discussions about the status of the group itself. So for sure there would
be articles discussing the forthcoming change, announcements that it had
happened, advice that they would likely need to subscribe to the new group
(and maybe unsubscribe to the old, according to whether and how their user
agent was showing old articles in the new). And of course moaners and
trolls still decrying the whole thing long after it had happened :-).

So, again, nothing particularly 'mysterious' to users who were familiar
enough with how to subscribe and unsubscribe to groups, and to clear out
stored articles from groups they were no longer interested in.

>>.... Eventually the old group disappears, and 
>>presumably copies of articles in the old group are kept or disappear on 
>>the client according to what the user has configured, or whatever your 
>>client normally does with a rmgroup.

>I like the word "presumably" in that sentence.  The bit you have snipped
>with tasks (b) (c) (d) (e) were all about the substantial amount of work
>that would be required at a database level to make it all work

I don't see that. How is it different from what the database level would
have done if there had been a newgroup followed by a rmgroup? For from
what you have described to me, it would seem that is how Turnpike would
perceive it. Other user agents might perceive it differently, of course.

To: [email-masked]
Xref: clerew local.usefor:23819
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Proposing a new editor for the USEPRO document
Date: Thu, 7 Dec 2006 15:10:47 GMT
Lines: 48
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>Charles Lindsey <[email-masked]> writes:
>> Russ Allbery wrote:

>>> If it existed *and were widely deployed*.  My guess is that they'll never
>>> be able to use it no matter what we say in our document, since that use of
>>> it is not backward-compatible.

>> Certainly cn.bbs.* could start using it immediately, since that would
>> merely be what they are currently doing with that extra parameter added
>> (ignored by old systems).

>True, I can see where that would be useful.  I forgot that the default
>behavior was hierarchy-scoped, not sub-hierarchy-scoped, so yes, the
>current cn.bbs.*-only checkgroups would remove other cn.* groups, which
>isn't desirable.

Ah! I had thought they were sub-hierarchy-scoped (and that's what USEPRO
says - I have not yet read that far in your draft yet to see what that
says).  But I see that s-o-1036 agrees with you (though a bit vague) and
so does the CNews implementation. OTOH, if you are correct then how come
the cn.bbs.* checkgroups have not destroyed the rest of the cn.* hierarchy
long ago? Do you, as a newsadmin, get a message telling you that you ought
to delete umpteen groups in cn.* every time a cn.bbs.* checkgroups comes
out?

>I'm saying that the chances are very high that cn.* will never be able to
>issue a checkgroups that omits all of cn.bbs.*.  Ever.  No matter what we
>write in our document.

In that case we might as well write it :-) .

At least, on sites where it was implemented, it might reduce the damage
done by ill-formed checkgroups mesages (such as cn.bbs.* currently appears
to be).

Date: Thu, 07 Dec 2006 13:37:01 +0100
From: Harald Alvestrand <[email-masked]>
To: [email-masked]
Subject: Strawman consensus call, mvgroup control message
Sender: [email-masked]

As far as I can tell from the list traffic:

- Charles supports the inclusion of a "mvgroup" control message in 
-usefor-usepro
- All other speakers do not support this inclusion
- Nobody has any big objection to a separate experimental document 
specifying such a message, despite the fact that several people doubt 
that it will be useful or adopted.

If nobody but Charles speaks in favour of the inclusion of "mvgroup", 
I'll rule that we have rough consensus to exclude it from -usepro, no 
matter who the editor is.

If someone apart from Charles speaks in favour of its inclusion, we'll 
conduct a poll.

                  Harald

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: Proposing a new editor for the USEPRO document
Organization: The Eyrie
Date: Wed, 06 Dec 2006 11:09:42 -0800
Sender: [email-masked]

>> If it existed *and were widely deployed*.  My guess is that they'll never
>> be able to use it no matter what we say in our document, since that use of
>> it is not backward-compatible.

> Certainly cn.bbs.* could start using it immediately, since that would
> merely be what they are currently doing with that extra parameter added
> (ignored by old systems).

True, I can see where that would be useful.  I forgot that the default
behavior was hierarchy-scoped, not sub-hierarchy-scoped, so yes, the
current cn.bbs.*-only checkgroups would remove other cn.* groups, which
isn't desirable.

> Likewise de.alt.*.

de.* is a solved problem.  The de.* checkgroups includes de.alt.* and
there's no separate de.alt.* checkgroups.  They're already doing the best
possible thing; anything involving chkscope would be a step in the wrong
direction.

> The question is what cn.* and de.* might do, and I think you are saying
> that they would not dare use it until they were sure that sufficient
> servers would recognize and act on it.

I'm saying that the chances are very high that cn.* will never be able to
issue a checkgroups that omits all of cn.bbs.*.  Ever.  No matter what we
write in our document.

Date: Wed, 06 Dec 2006 13:14:36 -0500
From: Ken Murchison <[email-masked]>
Organization: Carnegie Mellon University
To: [email-masked]
Subject: Re: USEFOR-11 troubles
Sender: [email-masked]

Frank Ellermann wrote:
> Hi, Usefor-11 didn't make it in the first attempt, it got two [DISCUSS]:
> 
> https://datatracker.ietf.org/public/pidtracker.cgi?command=print_ballot&ballot_id=2253&filename=draft-ietf-usefor-usefor

FYI, I created ticket #1401 in the tracker for the open issues.

-- 
Kenneth Murchison
Systems Programmer
Project Cyrus Developer/Maintainer
Carnegie Mellon University

Date: Wed, 6 Dec 2006 17:53:39 +0000
To: [email-masked]
From: Richard Clayton <[email-masked]>
Subject: Re: The mvgroup control message
Sender: [email-masked]

In message <[email-masked]>, Charles Lindsey
<[email-masked]> writes
>
>Richard Clayton wrote:
>> 
>> ... I use (and designed and helped write) what is effectively an offline
>> news client. This means that it has to have some of the properties of
>> news servers as well (storage, expiry, history, cross-referencing etc).
>> 
>> Even in these broadband days, this type of client is still useful when
>> connectivity to a server is not immediately available.
>> 
>> Dealing with a mvgroup in such a client would involve [and this is just
>> off the top of my head, there's probably a lot more]
>> 
>> (a) receiving the control message [it's not otherwise essential to see
>> any control messages] or extending the NEWGROUP response to be able to
>> tell the client about the event
>
>I don't see why that complication is necessary. Either the server has 
>both groups in separate existence during the overlap period, or it has 
>them aliased to one group. 

indeed so ... and my system is effectively a server as well, so
(depending on design decisions) it needs to alias or overlap in some
way. In order to do that it needs to become aware of the "mvgroup"
event, and that's what my issue (a) was saying

>What happens at present if a server creates a 
>new group 

it is reported via NEWGROUPS

>and later removes an old one on your system? 

my system remains unaware of removals (this is a flaw in NEWGROUPS)
unless a full LISTGROUP is requested.  In practice if there are articles
in the local database then the old group remains in existence so that
those articles can still be accessed ....  only once all the articles
vanish will the old group name be forgotten about

>Or if it suddenly 
>aliases two groups together?

I've never seen a server do that...  but the client would allocate
articles to groups in the local database by examining the Newsgroups:
header field, so it would not be relevant.

>Presumably in the first case, your client tells its readers that a new 
>group exists and invites them to subscribe. 

No -- that's just dumb (and yes I know that some Unix systems do that).
New groups (often with silly or vaguely obscene names) are created all
the time, so bothering users with trivia is a waste of time.

If you mean by "in the first case" the special state "when my system
somehow becomes aware of a mvgroup" then yes, some interaction with the
user will be necessary -- if only to explain that something weird and
wonderful has happened

>After which your readers 
>read both groups and respond as normal (remember that in this case the 
>server has adopted the minimal implementation, so it is not going to be 
>as convenient for users as it might be - but at least nothing happens 
>that should surprise them). Eventually the old group disappears, and 
>presumably copies of articles in the old group are kept or disappear on 
>the client according to what the user has configured, or whatever your 
>client normally does with a rmgroup.

I like the word "presumably" in that sentence.  The bit you have snipped
with tasks (b) (c) (d) (e) were all about the substantial amount of work
that would be required at a database level to make it all work

>The second case is more interesting, but I would expect both groups to 
>be visible on your client, and to contain the same articles. 

that's one possible design, yes -- I can see others

>I have just been experimenting with an aliassed group on my own machine 
>with the two newsreaders available to me.

which are both "online" newsreaders, viz: they don't have any local
article storage, so this isn't especially fascinating to me.  Try
Turnpike or Agent or similar...

>But whatever hapens, the user is no worse off than with separate 
>newgroup and rmgroup commands, and hopefully is much better.

for a small value of "much" and a big value of coding

- -- 
richard                                                   Richard Clayton

Those who would give up essential Liberty, to purchase a little temporary 
Safety, deserve neither Liberty nor Safety. Benjamin Franklin 11 Nov 1755

To: [email-masked]
Xref: clerew local.usefor:23812
Path: clerew!news
From: Charles Lindsey <[email-masked]>
Subject: Re: Proposing a new editor for the USEPRO document
Date: Wed, 6 Dec 2006 16:25:44 GMT
Lines: 28
Sender: [email-masked]

Russ Allbery wrote:
> Charles Lindsey <[email-masked]> writes:
> 
> 
>>Yes, those are probably the ones I have seen. But presumably the rest of
>>cn.* _does_ exist, at which point if they ever did want to issue a
>>checkgroups they would have to ensure that the full cn.bbs.* stuff was
>>included within it (accurately). Or they could use our new checkscope
>>feature if it existed.
> 
> 
> If it existed *and were widely deployed*.  My guess is that they'll never
> be able to use it no matter what we say in our document, since that use of
> it is not backward-compatible.
> 
Certainly cn.bbs.* could start using it immediately, since that would 
merely be what they are currently doing with that extra parameter added 
(ignored by old systems). Likewise de.alt.*.

The question is what cn.* and de.* might do, and I think you are saying 
that they would not dare use it until they were sure that sufficient 
servers would recognize and act on it. Or else they would continue doing 
what de.* have started doing which is to include all the de.alt.* groups 
as well (assuming they have an accurate list). That is a fair point. 
OTOH, it seems like a pretty easy feature to implement, so they might 
not have to wait that long.

I think we need to hear the views of some Germans on this one.

To: [email-masked]
Xref: clerew local.usefor:23813
Path: clerew!news
From: Charles Lindsey <[email-masked]>
Subject: Re: The mvgroup control message
Date: Wed, 6 Dec 2006 16:56:57 GMT
Lines: 71
Sender: [email-masked]

Richard Clayton wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> In message <[email-masked]>, Forrest J. Cavalier III
> <[email-masked]> writes
> 
> 
>>Your recent post doesn't address it being experimental, unproven,
>>expensive to implement, expensive to execute, and requiring
>>coordination of new servers and new clients.
> 
> 
> Since my recent anti-mvgroup email didn't give any reasons, could I
> stress the issue of clients...
> 
> ... I use (and designed and helped write) what is effectively an offline
> news client. This means that it has to have some of the properties of
> news servers as well (storage, expiry, history, cross-referencing etc).
> 
> Even in these broadband days, this type of client is still useful when
> connectivity to a server is not immediately available.
> 
> Dealing with a mvgroup in such a client would involve [and this is just
> off the top of my head, there's probably a lot more]
> 
> (a) receiving the control message [it's not otherwise essential to see
> any control messages] or extending the NEWGROUP response to be able to
> tell the client about the event

I don't see why that complication is necessary. Either the server has 
both groups in separate existence during the overlap period, or it has 
them aliased to one group. What happens at present if a server creates a 
new group and later removes an old one on your system? Or if it suddenly 
aliases two groups together?

Presumably in the first case, your client tells its readers that a new 
group exists and invites them to subscribe. After which your readers 
read both groups and respond as normal (remember that in this case the 
server has adopted the minimal implementation, so it is not going to be 
as convenient for users as it might be - but at least nothing happens 
that should surprise them). Eventually the old group disappears, and 
presumably copies of articles in the old group are kept or disappear on 
the client according to what the user has configured, or whatever your 
client normally does with a rmgroup.

The second case is more interesting, but I would expect both groups to 
be visible on your client, and to contain the same articles. Whether 
local copies stored in your client were stored separately for the two 
groups or mirrored the aliassing on the server depends on how you have 
written it. Again, the user would be informed that a new group had 
appeared, and his best strategy in this case would be to adjust his 
subscription to the new one at the earliest convenient opportunity and 
to forget the old one.

I have just been experimenting with an aliassed group on my own machine 
with the two newsreaders available to me.

'nn' just recognises the existence of one group. If you try to direct 
its focus to the alias, it just goes to the underlying real group and 
that is what you get.

'Mozilla' appears to contain both groups and they contain the same 
articles. But it keeps separate .newsrc entries for them, so it is 
better for the user to retain a subscription to just one of them. He 
probably has to do a 'mark everything as read' when he first moves 
(having finished reading in the old one, or course).

But whatever hapens, the user is no worse off than with separate 
newgroup and rmgroup commands, and hopefully is much better.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: The mvgroup control message
Organization: The Eyrie
Date: Tue, 05 Dec 2006 10:22:27 -0800
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:

> Let me ask just two questions:

> Q1 (for the group)
> 	Other things being equal, do we WANT to see this happen?

Only in conjunction with full support in the NNTP protocol so that clients
can do the right thing, including automatic update of subscriptions.  I
think anything short of that is likely to be more confusing than useful.

However, control messages being what they are, I certainly have no
objection to people issuing mvgroup control messages and sites acting on
them if they disagree with me and think that it's useful.

> If so, then we have a choice between doing it in this standard or as a
> separate experimental protocol. So Q2 is mainly addressed at Russ:

> Q2A
> 	It it appears in the Standard, will it be implemented in INN
> 	(not as first priority on Day 1, but later when more pressing
> 	changes are done)?

> Q2B
> 	If it appears as an Experimental Protocol, will it be implemented
> 	in INN (not as first priority on Day 1, but later when more
> 	pressing changes are done)?

The answer in either case is identical: I don't intend to implement it
myself, but I'll review and commit a decent implementation if someone else
wants it and wants to use it.  That will be the same whether it's in a
standards-track document, an experimental document, or not standardized at
all.

I may have architecture objections to moving articles unless it's done
very carefully.  Even if they're moved into a completely new group, doing
this properly in INN is challenging and requires updating multiple
databases and cross-references properly.  There is also no way of doing
this properly with existing mvgroup-unaware clients without rewriting the
message to, at the very least, change the Xref header, which creates a
nasty conflict between the desire to never modify messages and the
practical implications of having posts for which normal operations fail.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: Proposing a new editor for the USEPRO document
Organization: The Eyrie
Date: Tue, 05 Dec 2006 10:15:40 -0800
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:

> Yes, those are probably the ones I have seen. But presumably the rest of
> cn.* _does_ exist, at which point if they ever did want to issue a
> checkgroups they would have to ensure that the full cn.bbs.* stuff was
> included within it (accurately). Or they could use our new checkscope
> feature if it existed.

If it existed *and were widely deployed*.  My guess is that they'll never
be able to use it no matter what we say in our document, since that use of
it is not backward-compatible.

To: [email-masked]
Xref: clerew local.usefor:23805
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: The mvgroup control message
Date: Tue, 5 Dec 2006 12:53:38 GMT
Lines: 54
Sender: [email-masked]

In <[email-masked]> "Forrest J. Cavalier III" <[email-masked]> writes:

>Yes, this working group already saw explanations of mvgroup.

>Your recent post doesn't address it being experimental, unproven,
>expensive to implement, expensive to execute, and requiring
>coordination of new servers and new clients.  Plus, you admitted
>that there is no real benefit now, only some framework for adding
>future benefit.  Low benefit with High cost doesn't have a chance
>of getting implemented in widely used systems.

It is NOT expensive to implement in its minimal form, and on servers that
already implement aliasing in the 'classical' manner (i.e. as in Bnews and
Cnews, but not apparently INN) it is NOT expensive at all.

Being expensive to execute is irrelevant. Even if they occurred at the
rate of one a month, that hardly amounts to a noticeable quantity of
machine cycles.

Servers and clients? Only servers, I think (people who issue control
messages usually use custom scripts, and generating a mvgroup message is
pretty simple). Yes, different servers will behave differently, so on some
servers you will see two separate groups during the overlap period, and
on others you will see one group with the other aliased to it. That
situation cannot be avoided.

Let me ask just two questions:

Q1 (for the group)
	Other things being equal, do we WANT to see this happen?

If so, then we have a choice between doing it in this standard or as a
separate experimental protocol. So Q2 is mainly addressed at Russ:

Q2A
	It it appears in the Standard, will it be implemented in INN
	(not as first priority on Day 1, but later when more pressing
	changes are done)?

Q2B
	If it appears as an Experimental Protocol, will it be implemented
	in INN (not as first priority on Day 1, but later when more
	pressing changes are done)?

To: [email-masked]
Xref: clerew local.usefor:23804
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Proposing a new editor for the USEPRO document
Date: Tue, 5 Dec 2006 12:33:44 GMT
Lines: 27
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>Charles Lindsey <[email-masked]> writes:

>> The other hierarchy that I know of which currently issues separate
>> checkgroups for a sub-hierarchy is cn.*. I don't know how they prevent
>> these from deleting everything else in cn.*, but presumably they have
>> managed it somehow.

>I don't recall ever seeing a checkgroups for cn.*, only for cn.bbs.*.

Yes, those are probably the ones I have seen. But presumably the rest of
cn.* _does_ exist, at which point if they ever did want to issue a
checkgroups they would have to ensure that the full cn.bbs.* stuff was
included within it (accurately). Or they could use our new checkscope
feature if it existed.

Date: Tue, 5 Dec 2006 11:14:27 +0000
To: [email-masked]
From: Richard Clayton <[email-masked]>
Subject: Re: The mvgroup control message
Sender: [email-masked]

In message <[email-masked]>, Forrest J. Cavalier III
<[email-masked]> writes

>Your recent post doesn't address it being experimental, unproven,
>expensive to implement, expensive to execute, and requiring
>coordination of new servers and new clients.

Since my recent anti-mvgroup email didn't give any reasons, could I
stress the issue of clients...

... I use (and designed and helped write) what is effectively an offline
news client. This means that it has to have some of the properties of
news servers as well (storage, expiry, history, cross-referencing etc).

Even in these broadband days, this type of client is still useful when
connectivity to a server is not immediately available.

Dealing with a mvgroup in such a client would involve [and this is just
off the top of my head, there's probably a lot more]

(a) receiving the control message [it's not otherwise essential to see
any control messages] or extending the NEWGROUP response to be able to
tell the client about the event

(b) juggling around with the local database to relocate articles

(c) dealing specially with ancient articles that have been locally
preserved past their nominal expiry dates

(d) juggling around with the newsgroup "subscriptions" [and deducing
which of several servers should be used for accessing articles within
this group in the future]

and probably, when it was all worked through and it was realised that
clients were ignoring mvgroups unless the change was shoved in their
faces

(e) dealing with a new set of custom response codes for when the group
has apparently disappeared under the client's feet.

This is quite a lot of coding and database redesign for an essentially
trivial feature. I am pleased to see a draft with it removed.

- -- 
richard                                                   Richard Clayton

Those who would give up essential Liberty, to purchase a little temporary 
Safety, deserve neither Liberty nor Safety. Benjamin Franklin 11 Nov 1755

Date: Tue, 05 Dec 2006 02:07:37 -0500
From: "Forrest J. Cavalier III" <[email-masked]>
To: [email-masked]
Subject: Re: The mvgroup control message
Sender: [email-masked]

Yes, this working group already saw explanations of mvgroup.

Your recent post doesn't address it being experimental, unproven,
expensive to implement, expensive to execute, and requiring
coordination of new servers and new clients.  Plus, you admitted
that there is no real benefit now, only some framework for adding
future benefit.  Low benefit with High cost doesn't have a chance
of getting implemented in widely used systems.

Technical merit is not the only consideration of what stays in and
stays out of standards track RFCs.  There have to be interoperating
implementations on widely used news servers.  (CNews is educational,
but it doesn't count as a widely used news server.)

To: [email-masked]
Xref: clerew local.usefor:23799
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: The mvgroup control message
Date: Mon, 4 Dec 2006 19:27:47 GMT
Lines: 114
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>Charles Lindsey <[email-masked]> writes:

>> Now ideally, a mvgroup should, at one stroke, remove the old group, add
>> the new, cause all existing articles magically to move to the new, and
>> take care of running threads. Sounds like hard work, and the WG
>> challenged me to produce an implementation; which I did, on CNEWS, and I
>> tested it pretty thoroughly (though not on the live Usenet, of course,
>> because this is one experiment you cannot do live).

>It's implementable, and I do appreciate you writing an implementation and
>seeing how it can be done.  Unfortunately, the implementation you did is
>in a news server that's not widely used and not particularly maintained,
>so it doesn't count much towards deployed base, although it does count
>towards implementation experience.

>However, moving articles between groups after they're stored pose weird
>protocol concerns that I am not at all convinced the current proposal has
>adequately dealt with, even if it were widely implemented.

No, USEPRO make it clear that you cannot expect to move articles around,
except when the target group is empty (this is not for merging groups).
Essentially, you can keep the articles physically in the old group and
rename the directory concerned, or move them bodily to the newly created
group. Only one group ever exists on the server, the other is just an
alias for it (which might happen either way around). The nature of the
alias (at least on CNEWS, though probably on others) is that whichever
alias you ask to see the group by (whether for finding old articles or storing
new ones, it all happens in the one group).

>  For example,
>unless you rewrite the Newsgroups header of the old articles or add a
>Followup-To header or followups to articles that were moved from the old
>group will be sad (if not on the local server then on someone else's
>server).

Absolutely not! Otherwise threads are going to get hopelessly confused,
and you break the fundamental rule of Usenet that an article appears
exactly the same (modulo Path, Xref, etc) whichever server you find it on.
So once a thread starts, it remains in its original group (unless
followups are manually changed). Some users may believe they are reading
it in newgroup, and some in oldgroup (and may be slightly confused if they
inspect the Newsgroup header too closely). But whichever is the case, if
you followup it will appear to you to be in your group and to other Usenet
users in whichever group their own server has put it in. In fact, an NNTP
client cannot actually tell which of the two his server really keeps it in
unless it inspects the Active File, or looks closely at the Xref. If, at
some stage, users change their subscription from the old group to the new,
they will still see exactly the same articles.

>  Articles crossposted into the old group will need to have their
>Xref headers updated (and therefore will have to have their overview
>records regenerated) or crosspost chasing into the new group will not work
>properly.

I think the best strategy during the overlap period will be to include
both groups in the Xref (the article numbers will be the same, of course).
That way, User Agents that try to avoid showing the same article in two
groups will get it right either way.

Yes, if the .overview included the Xref, then it will have to be rebuilt.
But only if the Xrefs in the existing articles in oldgroup were updated,
which may not be necessary.

Anyway, all that was working in my implementation (I was surprised to find
it was over 5 years ago now). I fired all sorts of strange sequences of
operations at it and it worked fine. Indeed I am still using a couple of
local groups on my server that were moved at that time.

>  News readers need to be told somehow that the group has moved
>so that they can update subscriptions automatically, and there's currently
>no facility for doing that (we'd want to at least standardize the alias
>flag in conjunction with this).

Users will certainly have to change their subscription to the newgroup
sometime before it finally disappears, but I don't think it has to be
automatic. For sure, the actual renaming of the group will be a hot
topic of conversation on the group itself, so people should be well aware
of if. And the hierarchy admins, or the proponents of the change, can be
expected to post frequent reminders if people need to take action (which
is exactly what has to be done under the present newgroup followed by
rmgroup system, so we are certainly no worse off than at present).

>Implementing mvgroup as a clone of newgroup, or even a clone of newgroup
>and a delayed rmgroup, is pointless.  That doesn't solve any of the
>problems that mvgroup was introduced to solve.  Unless readers are
>automatically redirected to the new newsgroup and, ideally, the old
>articles are also preserved, there's no reason to bother with it, since it
>won't be any better than what's already available now.

Yes it does, because at the least it gives you the same as the previous
system, but with the added benefit that newsadmins are more likely to
accept both halves of the change if it comes as one command, and it opens
the way to more sophisticated implementations as time goes by and user
pressure starts to demand them. Old articles get preserved either way,
just that you may have to examine both groups to find them if your server
has done the minimal implementation.

Introducing it netwide is going to take time however we do it, but setting
the "entry level" low is the only way you will ever get it started. And it
it doesn't even get started, then for sure it will never get finished.

To: [email-masked]
Xref: clerew local.usefor:23800
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: draft-allbery-usefor-usepro-00 submitted
Date: Mon, 4 Dec 2006 19:40:44 GMT
Lines: 61
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

> * I tried to keep the document focused tightly on its scope and removed
>   much discussion that I felt was outside its scope.  This includes most
>   of the NOTEs; there are only a few places where I added a NOTE where
>   there wasn't one previously, and many places where I removed a NOTE.  I
>   tried to be very clear about what was within the scope of the protocol
>   and what was not, and omitted discussion of areas that were not even if
>   the discussion was interesting.

> * I tried to remove all references to policies, hierarchy administrators,
>   and other terms for the organization of Usenet.  This is partly because
>   I felt they were out of scope for the protocol and partly because I
>   tried to make the document general to any Netnews network and avoid any
>   statements specific or unique to Usenet.  Such matters can be discussed
>   in USEAGE.

May I just point out that the name of the WG includes the word "Usenet",
so defining how Usenet works is hardly off topic. And private
implementations of Netnews need administration just as much as the public
one. For sure, hierarchy administrators do exist (you and I are, or have
been, both involved in that) and Usenet would collapse without them. The
only oddity is that the method of establishing them is distinctly murky,
but in practice works remarkably well. And for sure, the protocols contain
features that assume they exist. Which is all USEPRO says, if you actually
read it carefully.

> * I have not followed Charles's lead in the renaming of serving agents to
>   storage agents; the former sounds more correct to me.

I was just trying to avoid any possible confusion with news servers, a
term we now use widely in USEFOR.

>I'm not happy with the readability of the instructions for Path
>construction.  Charles has a new version in his latest draft that may be
>much better.  It may be even better to pull all Path discussion into a
>separate section of its own (or a subsection of duties) and just reference
>it from the duties sections of the specific agents.

My new version had been published on this List (twice).

>I'm also not happy with the substantial duplication of instructions
>between injecting, relaying, and serving agents and think that if we could
>find a clean way of not repeating ourselves, we could shorten the draft by
>several pages.

Valid point, though there were often small but necessary differences of
detail.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: Proposing a new editor for the USEPRO document
Organization: The Eyrie
Date: Mon, 04 Dec 2006 10:33:22 -0800
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:

> The other hierarchy that I know of which currently issues separate
> checkgroups for a sub-hierarchy is cn.*. I don't know how they prevent
> these from deleting everything else in cn.*, but presumably they have
> managed it somehow.

I don't recall ever seeing a checkgroups for cn.*, only for cn.bbs.*.

To: [email-masked]
Xref: clerew local.usefor:23798
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: USEFOR-11 troubles
Date: Mon, 4 Dec 2006 14:42:42 GMT
Lines: 62
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>Reviewing USEPRO brought home to me that it's not possible to implement a
>useful piece of Netnews software other than a pure article reader with
>only the information in USEFOR.

Just like you can't write a useful piece of mail software if all you have
is RFC 2822. But that was never intended.

>There are several that seem rather normative to me having just gone
>through all of USEPRO, apart from the security considerations.  Section
>3.1.4 says that control messages are defined in USEPRO, section 3.1.5
>defers to USEPRO for the definition of <diag-keyword>, section 3.1.6
>defers to USEPRO for the content of the Subject header (and that reference
>I think could simply be dropped, as the only place that USEPRO discusses
>the Subject header is in the context of a followup and then only as a
>MAY), section 3.2 states that further requirements for optional header
>fields are in USEPRO, section 3.2.3 defers to USEPRO for valid control
>message <verb>s, and section 3.2.12 defines the action of Supersedes in
>terms of cancel control messages defined in USEPRO.

USEFOR tells you enough about control messages to recognize one when you
see it, and gives a broad syntax to which they must conform. That's all
you need to know for the 'on the wire' format. That, and a hint of what
they are meant for with a pointer to look elswhere for protocol issues.

Much the same for <diag-keyword>. USEFOR tells you enough to recognize
them, but there is no reason so far as USEFOR is concerned why any words
satisfying the syntax should not be used (e.g. it is no pretext for
dropping articles as syntactically incorrect).

The Subject header reference is only to point out that there is sometimes
a miniscule bit of structure there (we probably included it to keep John
Stanley quiet :-) ).

And the other mentions are concerning protocol issues. Essentially, the
only 'semantics' we give in USEFOR are brief mentions of the intended
purpose of each header, with the minimum description of protocol to make
those mentions intelligible.

>On the surface, it's hard to argue against some or most of those being
>normative references.

So I think it is quite proper to regard these as non-normative IMO
(likewise the reference to security considerations, where the only ones
explicitly given in USEFOR relate to syntactic/format matters).

So, if you want to be able to publish USEFORE separately (which I was
against, but that is another story), then I think you leave these things
alone.

To: [email-masked]
Xref: clerew local.usefor:23797
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Proposing a new editor for the USEPRO document
Date: Mon, 4 Dec 2006 14:19:20 GMT
Lines: 125
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>Charles Lindsey <[email-masked]> writes:
>> Russ Allbery <[email-masked]> writes:

>>> * Completely ban reinjection as permitted in the current draft....

>The point that I hadn't realized before is that defining systems that are
>trying to do reinjection as gateways not only makes logical sense (they're
>gatewaying between two disconnected Netnews networks), ....

I can imagine situations in which reinjection is nothing like gatewaying,
but I need to see your draft before commenting further.

>>> * Drop all description of the batch format and don't allow batches to be
>>>   sent via application/news-transmission.  ...

>> Batch format was only ever used for UUCP, where it was indeed necessary.

>It's also used by the NNTP XBATCH extension.  Not that I think many people
>are using that (it was added to INN largely because it was trivial to do
>given that INN already had batch support because of its UUCP support), but
>it's rather neat and is the only current way to transfer compressed news
>via NNTP.

That look like a good reason to write an NNTP extension document.

>> It was in application/newstransmission because Henry Spencer put it
>> there when he registered it.

>Ah, that explains it.

Yes, if we remove it from there, then it needs an explicit mention that it
is a change from the current news/transmission definition. I would have
no great problem with doing that.

And in that case, the batch format is only relevant for describing the
response to the 'sendme' control message (and I hope you are not going to
remove mention of the 'ihave' and 'sendme' control messages; they may be
of mostly archaeological interest now, but they need to be formally
documented somewhere, and I was still seeing them - leaked from some
Japanese site - around 4 years ago, so I suspect they are still around).
Also, the ghastly 'to' hack in newsgroup-names needs to be mentioned, if
only because we mention it in Usefor and refer to USEPRO for the gory
details.

>...  From the perspective of our draft, it's a very
>odd sideline that adds about a page of text for something that essentially
>no one is going to care about,...

Eh? The batch format is described in 16 lines (though it could do with a
mention of compression). If you want to remove the whole
ihave/sendme/batch/to business to an Appendix, then that would be fine,
but you can't just throw it away.

>> I was not aware that the present description of batch format was
>> deficient (it is largely taken from Son-of-1036)....

>I'm surprised, given that you're using UUCP, that you're not using
>compressed batches.

No, I am not using UUCP now. I almost certainly used compressed batches
when I last did.

>>> * Add charset parameters for application/news-groupinfo and
>>>   application/news-checkgroups....

>> No, that would be most unwise until we have resolved how
>> internationalized newsgroup-names are going to look, because they need
>> to appear in the same file, and they really need to appear there in the
>> same form as they do elsewhere.

There was a time, round about when we dropped the idea of
internationalized newsgroup-names, when people actually proposed all sorts
of bizarre encodings of newsgroup-names within ASCII. I think, with the
advent of the EAI WG, the risk of doing it that way has finally receded
which gives us a bit more flexibility. But that was the time when I wrote
in that restriction for checkgroups.

I think the least you need to say is that, to be compatible with present
software, any charset specified here MUST have ASCII as a subset (no
EBCDIC!). And maybe should/SHOULD be UTF-8 (with mention of NNTP as the
reason).

With that restriction, I think a charset parameter might now work.

>There's only something to undo if ...

Yes, but I was trying to avoid putting in things that one might have to
'undo' later.

>>> * Recommend use of the control hierarchy for storing control messages
>>>   rather than storing them in normal groups (upgraded from a NOTE).

>> I have no real problem with that. But it is introducing a *new*feature*,
>> horror of horrors :-(, and it is dealing with matters which are not seen
>> "on the wire" - even greater horrors.

>This isn't a new feature -- it's already implemented by every widely used
>news server that I'm aware of.

Sure, but not always in that form. But surely the point is that that form
should NEVER appear 'on the wire' and thus is better not normative. It is
really a private matter between servers and their clients (though it is a
convention that shold be sidely encouraged, hence my NOTE). But I am not
really arguing against it.

Of more concern to me is the behaviour of NNTP agents which do not send
control messages to users who ask for messages in some particular group.
That behaviour is not required even by RFC 3977, but it seems endemic, and
leads to the problem that Frank just mentioned that you are forced to
bring down the complete control.cancel group if you want to see actual
cancel messages for some reason. But that issue is outside of our scope, I
think.

To: [email-masked]
Xref: clerew local.usefor:23796
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Proposing a new editor for the USEPRO document
Date: Mon, 4 Dec 2006 13:22:24 GMT
Lines: 26
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>If one really wishes to have an administratively separate hierarchy, it's
>far better to do what it.* did in creating it-alt.*.

But de.* didn't. The feature was originally put in at their behest.

The other hierarchy that I know of which currently issues separate
checkgroups for a sub-hierarchy is cn.*. I don't know how they prevent
these from deleting everything else in cn.*, but presumably they have
managed it somehow.

The feature should be pretty easy to implement (it easily translates into
a wildmat). It would then be up to the admins of de.* and cn.* to lean on
providers to implement it (I imagine cn.* would find it easier).

To: [email-masked]
Xref: clerew local.usefor:23795
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Proposing a new editor for the USEPRO document
Date: Mon, 4 Dec 2006 13:15:25 GMT
Lines: 27
Sender: [email-masked]

In <[email-masked]> Frank Ellermann <[email-masked]> writes:

>In theory fine.  But my interpretation of Russ' and your comments
>is that it's pointless:  The USEFOR drafts proposed 'mvgroup' for
>some years, so far nobody implemented it, no additional experiment
>necessary to confirm its death.

The reason nobody implemented it is that they wern't meant to until the
draft became a standard. This is a new feature, and people who implement
new features before they are finally and fully defined usually get into
trouble [1]. It was always envisaged that introducing it would take quite
some time after the standard appeared, hence the provision of transitional
arrangements.

[1] For example, there are some people using an earlier form of
Injection-Info. It would have been better if they hadn't.

Date: Sat, 02 Dec 2006 11:17:07 -0500
From: Ken Murchison <[email-masked]>
Organization: Carnegie Mellon University
To: Russ Allbery <[email-masked]>
CC: [email-masked]
Subject: Re: draft-allbery-usefor-usepro-00 submitted
Sender: [email-masked]

Russ Allbery wrote:

>  * There are numerous minor formatting changes and a few oddities of
>    section organization that are due to use of xml2rfc.  For instance, I
>    couldn't figure out a good way of putting the note to send comments on
>    the draft to the WG somewhere more appropriate than the Scope, such as
>    with the Authors' Addresses.  Much of this is probably due to my lack
>    of fluency with xml2rfc.

I'm not an xml2rfc expert either, but IIRC, you can add a <note 
title="foo"> section in the <front> of the document.

> I'm also not happy with the substantial duplication of instructions
> between injecting, relaying, and serving agents and think that if we could
> find a clean way of not repeating ourselves, we could shorten the draft by
> several pages.

I'll take a look and see if I can come up with some kind of 
consolidation, although I'm not good at making miracles happen -- just 
ask my kids  ;)

> One other general note:  I have, over the past month, put in roughly 25
> hours into this effort, which is a one-time expenditure that I can't
> continue.  Going forward, I'm willing to devote one to two hours a week on
> USEFOR but no more.  Anything that doesn't fit within that time is likely
> to wait until the following week; I just have too many other committments
> that I can't drop for USEFOR work.  I expect this is substantially less
> time than Charles has put in, and as a result I expect I'll be
> substantially less responsive than he has been.  WG participants (and
> chairs) should bear that in mind when deciding on what role for which I
> would be useful.  I'm willing to work on USEFOR-related work, but only
> with that time constraint.

I would be able to find enough cycles to do editing on this draft if it 
moves forward.

-- 
Kenneth Murchison
Systems Programmer
Project Cyrus Developer/Maintainer
Carnegie Mellon University

Date: Sat, 2 Dec 2006 12:40:44 +0000
To: Charles Lindsey <[email-masked]>
Cc: [email-masked]
From: Richard Clayton <[email-masked]>
Subject: Re: The mvgroup control message
Sender: [email-masked]

In message <[email-masked]>, Charles Lindsey
<[email-masked]> writes

>The WG decided right at the start that a mvgroup control message was
>needed. I think the view was that it had to come sooner or later, that it
>might cause some initial pain, but if it had to be done, then nothing was
>gained by postponing it. 

well we seem to have lasted many years since then without it -- so I'm
quite happy to see it ignored even longer

>If we
>don't do it now, then nobody ever will.

I'm happy with that view too

- -- 
richard                                                   Richard Clayton

Those who would give up essential Liberty, to purchase a little temporary 
Safety, deserve neither Liberty nor Safety. Benjamin Franklin 11 Nov 1755

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: The mvgroup control message
Organization: The Eyrie
Date: Sat, 02 Dec 2006 01:09:38 -0800
Sender: [email-masked]

Russ Allbery <[email-masked]> writes:

> Incidentally, the current USEPRO draft contradicts itself; it first says
> that the articles MUST NOT be altered in any way when they're moved, and
> then says that the Xref header may contain entries for the new group.

Oh, I see, it's not a contradiction; I just had to squint at it right.
It's talking about articles that arrived *after* the mvgroup message, and
I was thinking about articles that arrived *before* it.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: draft-allbery-usefor-usepro-00 submitted
Organization: The Eyrie
Date: Sat, 02 Dec 2006 01:06:20 -0800
Sender: [email-masked]

I've just submitted draft-allbery-usefor-usepro-00 to the I-D editor.  You
can also get a copy now from:

    <http://www.eyrie.org/~eagle/usefor/usepro/>

In that directory is the text draft, the HTML version of the draft, and
the original XML source.  I will keep that directory up to date if I
produce any additional versions of the draft.  I haven't had a chance to
wrap any nice HTML around it yet, or update my other pages about USEFOR
activities.

Charles is listed as the co-author because the text is heavily based on
his draft, uses large chunks of his writing, and has benefitted greatly
from the work that he has done.  Please note, however, that every decision
about what was included in this draft is solely my responsibility; please
do not assume that Charles supports its content or wording unless he
explicitly says so.  If you think something in the draft is wrong, I'm the
person to complain to.

This draft is not appreciably shorter and is still organized around
roughly the same lines, although I changed the order of some sections.
There are lots of minor changes to language, and I do not claim that
they're all for the better.  Many are an artifact of the process I used to
produce the draft; rather than editing the existing text, I started with
the current draft in another window and wrote a description of the same
protocol (with those changes that I felt should be made) in mostly my own
terms.  I did, however, freely copy from the existing draft if I felt that
it expressed a particular concept in a way that I liked, so you will see a
great deal of similarity of language.

I already commented separately on the technical changes that I made (and
there are probably others that I missed).  Here's a brief summary of the
editorial changes:

 * I tried to keep the document focused tightly on its scope and removed
   much discussion that I felt was outside its scope.  This includes most
   of the NOTEs; there are only a few places where I added a NOTE where
   there wasn't one previously, and many places where I removed a NOTE.  I
   tried to be very clear about what was within the scope of the protocol
   and what was not, and omitted discussion of areas that were not even if
   the discussion was interesting.

 * I tried to remove all references to policies, hierarchy administrators,
   and other terms for the organization of Usenet.  This is partly because
   I felt they were out of scope for the protocol and partly because I
   tried to make the document general to any Netnews network and avoid any
   statements specific or unique to Usenet.  Such matters can be discussed
   in USEAGE.

 * I attempted to remove mentions of constraints on agents that were not
   protocol constraints.  For example, my draft simply says that a
   moderator may do anything they like to a message; there is no mention
   of compliance with a moderation policy, since that's not part of the
   protocol.  From a protocol perspective, there are no restrictions on
   the actions of a moderator.

 * The Security Considerations section has been substantially modified.  I
   mostly rewrote it, although I think several paragraphs survive mostly
   unscathed.

 * There are numerous minor formatting changes and a few oddities of
   section organization that are due to use of xml2rfc.  For instance, I
   couldn't figure out a good way of putting the note to send comments on
   the draft to the WG somewhere more appropriate than the Scope, such as
   with the Authors' Addresses.  Much of this is probably due to my lack
   of fluency with xml2rfc.

 * I have not followed Charles's lead in the renaming of serving agents to
   storage agents; the former sounds more correct to me.

I'm not happy with the readability of the instructions for Path
construction.  Charles has a new version in his latest draft that may be
much better.  It may be even better to pull all Path discussion into a
separate section of its own (or a subsection of duties) and just reference
it from the duties sections of the specific agents.

I'm also not happy with the substantial duplication of instructions
between injecting, relaying, and serving agents and think that if we could
find a clean way of not repeating ourselves, we could shorten the draft by
several pages.

One other general note:  I have, over the past month, put in roughly 25
hours into this effort, which is a one-time expenditure that I can't
continue.  Going forward, I'm willing to devote one to two hours a week on
USEFOR but no more.  Anything that doesn't fit within that time is likely
to wait until the following week; I just have too many other committments
that I can't drop for USEFOR work.  I expect this is substantially less
time than Charles has put in, and as a result I expect I'll be
substantially less responsive than he has been.  WG participants (and
chairs) should bear that in mind when deciding on what role for which I
would be useful.  I'm willing to work on USEFOR-related work, but only
with that time constraint.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: Proposing a new editor for the USEPRO document
Organization: The Eyrie
Date: Sat, 02 Dec 2006 00:40:05 -0800
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:
> Russ Allbery <[email-masked]> writes:

>> * Completely ban reinjection as permitted in the current draft.  This
>>   allows a much sharper and clearer distinction between articles and
>>   proto-articles and simplifies the descriptions of both posting and
>>   injecting agents.  Instead, state that agents that wish to transfer
>>   articles between two separate Netnews networks are gateways, are
>>   subject to the requirements of a gateway, and must rewrite the articles
>>   into proto-articles and inject them as normal.

> Well it was you who said, when we discussed it a couple of years back,
> that reinjection was going to happen whether we liked it or not and
> occasionally it might even be sensible.

In general, after going through the entire draft, I changed my opinion or
found a better solution to several problems that I'd previously not seen a
good way of handling.  It's a very useful exercise; it gave me a new
perspective on the protocol as a whole.

The point that I hadn't realized before is that defining systems that are
trying to do reinjection as gateways not only makes logical sense (they're
gatewaying between two disconnected Netnews networks), it produces all the
right results in the protocol, applies the correct warnings and
restrictions for doing reinjection safely, and and makes the whole
protocol description fall out more neatly.  It also puts the entire onus
for getting it right on the posting agent that's doing reinjection rather
than requiring the injecting agent to guess at different cases, which
places the complexity on the agent that's doing the unusual thing.

>> * Drop all description of the batch format and don't allow batches to be
>>   sent via application/news-transmission.  The batch format described in
>>   the current draft isn't sufficient to describe existing practice anyway
>>   (it doesn't handle compression), and sending batches via e-mail in
>>   application/news-transmission is a new feature with marginal use.  I'll
>>   describe the full batch format if I ever get around to writing up the
>>   NNTP XBATCH extension.

> Batch format was only ever used for UUCP, where it was indeed necessary.

It's also used by the NNTP XBATCH extension.  Not that I think many people
are using that (it was added to INN largely because it was trivial to do
given that INN already had batch support because of its UUCP support), but
it's rather neat and is the only current way to transfer compressed news
via NNTP.

> It was in application/newstransmission because Henry Spencer put it
> there when he registered it.

Ah, that explains it.

Well, I'd love to know if anyone is actually using it.  So far as I know,
no one is even using application/news-transmission, but of course I may
not have heard about it.  From the perspective of our draft, it's a very
odd sideline that adds about a page of text for something that essentially
no one is going to care about, and adds a few points of complexity that
we've had to deal with before (such as how the batch format counts bytes).

> And if you are going to use tunnelling through email as a method of
> transporting news, which you might still want to do in some unusual
> situations, then it might still be useful. OTOH, you could still get the
> same effect by using a huge multipart/mixed with one article in each
> part, so it is no huge deal to take the feature out. It is in any case
> forbidden for sending to moderators.

Yeah, that's the way I felt about it.

> I was not aware that the present description of batch format was
> deficient (it is largely taken from Son-of-1036). I have quite some
> experience of using batch format, both in UUCP and now in the interface
> between my NEWNEWS suck feed and my server, since CNEWS performs rather
> poorly in the face of large numbers of single-article batches.

I'm surprised, given that you're using UUCP, that you're not using
compressed batches.  I know that most of the INN users who have submitted
bugs and feedback about INN's UUCP batch support are.

The generalized batch format starts with a line of the form:

    #! <processor>

If <processor> is rnews, that line is followed by a byte count of an
article.  Other defined values for <processor>, however, are:

    cunbatch (far and away the most common)
    c7unbatch
    gunbatch

cunbatch indicates that the batch is compressed with standard Unix .Z
compression (although in practice I bet a lot of cunbatch batches are
actually gunbatch batches).  gunbatch indicates that it's compressed with
gzip.  c7unbatch indicates that it's compressed and then a content
transfer encoding is applied to make the data 7-bit; the content transfer
encoding used is unique to news batches so far as I know (it is not base64
or uuencode).  It is "defined" as follows:

**  The encoding uses characters from 0x20 (' ') through 0x7A ('z').
**  (That fits nicely into the UUCP 'f' protocol by Piet Beertema.) First,
**  expand three eight-bit charcters into four six-bit ones.  Collect
**  until we have 13, and spread the last one over the first 12, so that
**  we have 12 6.5-bit characters.  Since there are very few half-bit
**  machines, collect them into pairs, making six 13-bit characters.  We
**  can do this as A * 91 + B where A and B are less then 91 after we add
**  0x20 to make it printable.
**
**  And if you thought that was unclear, then we won't even get into the
**  terminating conditions!

Obviously c7unbatch is not horribly useful these days, but INN still does
support it.  I don't know if it's still used in the wild.

Once you apply the initial processor, you then reiterate on the enclosed
contents, which will generally then begin with #! rnews.

The full batch protocol is not horribly useful when transmitting batches
via e-mail, since you have to encode the e-mail to 8bit anyway and then
you lose most or all of the benefit of compression.  I can see why Henry
would have omitted it in a MIME type registration when those registrations
were basically only for e-mail.  However, when using batches over
binary-clean channels, such as UUCP, it's quite helpful.

>> * Add charset parameters for application/news-groupinfo and
>>   application/news-checkgroups.  We can't just tell everyone to use
>>   ASCII; they aren't now, and they're not going to.  Having explicit
>>   charset information at least allows someone to do something sane while
>>   still meeting the requirements of NNTP to use UTF-8.

> No, that would be most unwise until we have resolved how
> internationalized newsgroup-names are going to look, because they need
> to appear in the same file, and they really need to appear there in the
> same form as they do elsewhere.

No, you can convert between character sets.  But only if you know what
character set you're dealing with.

Tagged data is *always* better than untagged data.  If you know the
charset of the data, you at least have a fighting chance at doing
something with it if it's not in the charset that you need.  If you don't
know the charset of the data, you're doomed.  There is never any reason
not to say what the charset is; it never hurts you, and it stands a good
chance of helping you.

Saying that all checkgroups and newgroup group descriptions are in ASCII
is not helpful, because they're not.  Saying it in our document won't make
it so; it will just confuse implementors when they discover that writing
to our document produces software that doesn't work in the real world.
People are sending non-ASCII newsgroup descriptions right now and aren't
going to stop, since their languages cannot be represented in ASCII.

Given that, the choice is between descriptions tagged with a character set
and descriptions that are in some random 8-bit character set you have to
guess based on the hierarchy the control message affects.  I think that's
an easy choice.

Knowing the character set may not get us everything we need later.  It's
possible that, for reasons of normalization, we will eventually have to
require checkgroups to be in UTF-8.  However, adding a charset parameter
does nothing to hinder that and actually makes it easier to specify and
detect.  If it turns out that's what we have to say, then we can require
that the charset parameter always be either ASCII or UTF-8.  In the
meantime, implementors need to do something with those non-ASCII
descriptions that come right now with ASCII newsgroup names, and in fact
people are already implementing charset guessing code so that they can
convert incoming newsgroup descriptions into UTF-8.  This parameter would
allow us to encourage implementors to make a simple change in their
checkgroups message format and thereby make life massively easier for
those trying to maintain a newsgroups file in a single encoding and
thereby comply with an NNTP SHOULD.

> Years ago, we had plans to put headers, including newsgroup-names, in
> UTF-8. Now the EAI working group is trying to do UTF-8 headers for
> email.  It that endeavour succeeds (it is to be an Experimental Protocol
> according to present plans), then it is most likely that Netnews would
> just pile in behind it. In which case the Newsgroups File would be in
> UTF-8, which would tie in nicely with NNTP. But I would not want to
> allow other charsets in there in the interim, because we might have to
> undo it.

There's only something to undo if one assumes that adding the charset
parameter permits non-ASCII checkgroups when they aren't currently
permitted.  Since that horse has already left the barn, I don't see
anything new introduced by adding the charset parameter that would need to
be undone later in such an eventuality.

> Now it you were to propose that the Checkgroups etc should henceforth be
> in UTF-8 regardless, that would be a different matter which I might well
> support.

That's a much harder migration for existing control message senders, and
therefore much less likely to succeed, than getting them to just add the
charset that their descriptions are already written in.  The new MIME
types are already going to be a hard sell; I almost proposed taking them
out, and only didn't when I contemplated trying to write a formal
description of the newgroup control message body searching that news
servers do now.

>> * Recommend use of the control hierarchy for storing control messages
>>   rather than storing them in normal groups (upgraded from a NOTE).

> I have no real problem with that. But it is introducing a *new*feature*,
> horror of horrors :-(, and it is dealing with matters which are not seen
> "on the wire" - even greater horrors.

This isn't a new feature -- it's already implemented by every widely used
news server that I'm aware of.  It's a feature not described in RFC 1036,
but there are a ton of those.  And the feature is indeed seen on the wire;
it's seen in the dialog between the reading agent and the serving agent.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: The mvgroup control message
Organization: The Eyrie
Date: Sat, 02 Dec 2006 00:04:58 -0800
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:

> Moreover, I don't think experimental protocols are a good idea in
> Usenet, since the whole system really has to sing to the same tune (that
> does not stop you doing experimental things in user agents, such as
> trying out different fancy killfiles, or threading, or whatever else,
> but not with serving agents).

That's not an argument against experimental protocols.  That's an argument
against ever introducing anything new, since there's no way that you could
ever introduce anything new and have the whole system immediately sing the
same tune.  This is actually *not* the case for control messages, which
are particularly amenable to different servers doing different things and
have, from the start, allowed the introduction of completely new control
messages without hurting anything.

> Now ideally, a mvgroup should, at one stroke, remove the old group, add
> the new, cause all existing articles magically to move to the new, and
> take care of running threads. Sounds like hard work, and the WG
> challenged me to produce an implementation; which I did, on CNEWS, and I
> tested it pretty thoroughly (though not on the live Usenet, of course,
> because this is one experiment you cannot do live).

It's implementable, and I do appreciate you writing an implementation and
seeing how it can be done.  Unfortunately, the implementation you did is
in a news server that's not widely used and not particularly maintained,
so it doesn't count much towards deployed base, although it does count
towards implementation experience.

However, moving articles between groups after they're stored pose weird
protocol concerns that I am not at all convinced the current proposal has
adequately dealt with, even if it were widely implemented.  For example,
unless you rewrite the Newsgroups header of the old articles or add a
Followup-To header or followups to articles that were moved from the old
group will be sad (if not on the local server then on someone else's
server).  Articles crossposted into the old group will need to have their
Xref headers updated (and therefore will have to have their overview
records regenerated) or crosspost chasing into the new group will not work
properly.  News readers need to be told somehow that the group has moved
so that they can update subscriptions automatically, and there's currently
no facility for doing that (we'd want to at least standardize the alias
flag in conjunction with this).

When I can come up with three problems like that just off the top of my
head, my impression is that the concept is not fully baked and doesn't
work well with the rest of Usenet without more elaboration of exactly how
it should be done.

Incidentally, the current USEPRO draft contradicts itself; it first says
that the articles MUST NOT be altered in any way when they're moved, and
then says that the Xref header may contain entries for the new group.

> However, it transpired that it was not so easily done in INN, and so we
> backed off and left the full works as an ideal to be aimed at (not even
> a SHOULD), and made the actual requirements much smaller. Indeed, if you
> just implement it as a clone of newgroup (which is surely easily added
> to any existing server) you are still conformant (well, you break a
> "SHOULD", but when did that ever stop anybody :-( ).

Implementing mvgroup as a clone of newgroup, or even a clone of newgroup
and a delayed rmgroup, is pointless.  That doesn't solve any of the
problems that mvgroup was introduced to solve.  Unless readers are
automatically redirected to the new newsgroup and, ideally, the old
articles are also preserved, there's no reason to bother with it, since it
won't be any better than what's already available now.

Honestly, I think the place to start would be standarizing the alias flag
in LIST, introducing a new NNTP response code to GROUP that would act like
an HTTP redirect, and defining the NNTP behavior.  If that were done, the
mvgroup control message implementation would fall out fairly naturally.
(Although introducing a new GROUP response code would be tricky; extension
negotiation mostly doesn't work that way.  One might have to introduce a
new GROUP command to do this properly.  It's going to be rather tricky to
do this in a way that's really effective.)

> So the question is whether you can foresee how it would be introduced.
> Essentially, it will not prosper unless the users press for it. But if
> it is at least in the standard, then some hierarchy-admin can be
> persuaded to give it a go (and it removes the argument of the
> naysayers).

This to me is not a persuasive argument for inclusion of a feature in the
base document.  mvgroup control messages are entirely harmless at sites
that don't implement them; there's nothing about experimental status
instead of standards track that prevents a hierarchy administrator from
giving them a try.

To: [email-masked]
Xref: clerew local.usefor:23775
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Proposing a new editor for the USEPRO document
Date: Fri, 1 Dec 2006 20:12:41 GMT
Lines: 128
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>The specific major technical changes that I am proposing are (and I may
>have missed some, and some that I consider minor others may not):

> * Drop mvgroup and initial articles.

I have dealt with mvgroup in a separate thread.

As for initial articles, I could easily be persuaded to drop them. They
were a fancy idea and the WG seemed willing to go along with them, so they
stayed.

> * Completely ban reinjection as permitted in the current draft.  This
>   allows a much sharper and clearer distinction between articles and
>   proto-articles and simplifies the descriptions of both posting and
>   injecting agents.  Instead, state that agents that wish to transfer
>   articles between two separate Netnews networks are gateways, are
>   subject to the requirements of a gateway, and must rewrite the articles
>   into proto-articles and inject them as normal.

Well it was you who said, when we discussed it a couple of years back,
that reinjection was going to happen whether we liked it or not and
occasionally it might even be sensible. So all that the present draft does
is to make sure that, if it is done, then it is done competently and
consistently (e.g. you don't alter the Injection-Date).

> * Drop all description of the batch format and don't allow batches to be
>   sent via application/news-transmission.  The batch format described in
>   the current draft isn't sufficient to describe existing practice anyway
>   (it doesn't handle compression), and sending batches via e-mail in
>   application/news-transmission is a new feature with marginal use.  I'll
>   describe the full batch format if I ever get around to writing up the
>   NNTP XBATCH extension.

Batch format was only ever used for UUCP, where it was indeed necessary.
It was in application/newstransmission because Henry Spencer put it there
when he registered it. And if you are going to use tunnelling through
email as a method of transporting news, which you might still want to do
in some unusual situations, then it might still be useful. OTOH, you could
still get the same effect by using a huge multipart/mixed with one article
in each part, so it is no huge deal to take the feature out. It is in any
case forbidden for sending to moderators.

I was not aware that the present description of batch format was deficient
(it is largely taken from Son-of-1036). I have quite some experience of
using batch format, both in UUCP and now in the interface between my
NEWNEWS suck feed and my server, since CNEWS performs rather poorly in the
face of large numbers of single-article batches. In any case, we only
describe the batch format to preserve the knowledge of how it is supposed
to be done (and, who knows, when Osama bin Laden hacks his way into the
infrastructure of the Internet and brings the whole edifice to its knees,
we may yet be glad that UUCP is still there :-). But if the batch format
description is materially wrong, then it needs to be fixed.

> * Make use of the new path diag-keywords optional but recommended.

My draft makes it MUST/SHOULD as an issue to be decided.

> * Add charset parameters for application/news-groupinfo and
>   application/news-checkgroups.  We can't just tell everyone to use
>   ASCII; they aren't now, and they're not going to.  Having explicit
>   charset information at least allows someone to do something sane while
>   still meeting the requirements of NNTP to use UTF-8.

No, that would be most unwise until we have resolved how internationalized
newsgroup-names are going to look, because they need to appear in the same
file, and they really need to appear there in the same form as they do
elsewhere.

Years ago, we had plans to put headers, including newsgroup-names, in
UTF-8. Now the EAI working group is trying to do UTF-8 headers for email.
It that endeavour succeeds (it is to be an Experimental Protocol according
to present plans), then it is most likely that Netnews would just pile in
behind it. In which case the Newsgroups File would be in UTF-8, which
would tie in nicely with NNTP. But I would not want to allow other
charsets in there in the interim, because we might have to undo it.

Now it you were to propose that the Checkgroups etc should henceforth be
in UTF-8 regardless, that would be a different matter which I might well
support.

> * Recommend use of the control hierarchy for storing control messages
>   rather than storing them in normal groups (upgraded from a NOTE).

I have no real problem with that. But it is introducing a *new*feature*,
horror of horrors :-(, and it is dealing with matters which are not seen
"on the wire" - even greater horrors.

To: [email-masked]
Xref: clerew local.usefor:23774
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: The mvgroup control message
Date: Fri, 1 Dec 2006 18:51:42 GMT
Lines: 86
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>The specific major technical changes that I am proposing are (and I may
>have missed some, and some that I consider minor others may not):

> * Drop mvgroup ...

OK, I think this is a big enough topic that it needs a separate thread
(maybe even an Issue number).

The WG decided right at the start that a mvgroup control message was
needed. I think the view was that it had to come sooner or later, that it
might cause some initial pain, but if it had to be done, then nothing was
gained by postponing it. Moreover, I don't think experimental protocols
are a good idea in Usenet, since the whole system really has to sing to
the same tune (that does not stop you doing experimental things in user
agents, such as trying out different fancy killfiles, or threading, or
whatever else, but not with serving agents).

I think the argument for mvgroup was that newsadmins could usually be
relied upon to create newgroups, but were extremely reluctant to do
rmgroups (I think even now some of them do not realise that PGVERIFY has
removed the main difficulty with bogus rmgroup commands). So it was
thought that including the rmgroup in the same command as the newgroup
would encourage newsadmins to consider them as a single operation. There
is no doubt that groups need to be renamed from time to time, but every
time someone proposes such a thing (often with good reason) the naysayers
on the config groups say "it will never work", and so it gets dropped (but
in spite of that, we have successfully renamed groups in uk.* on a small
number of occasions - and we have done the other 'impossible' thing, which
is to moderate groups in situ, as well).

Now ideally, a mvgroup should, at one stroke, remove the old group, add
the new, cause all existing articles magically to move to the new, and
take care of running threads. Sounds like hard work, and the WG challenged
me to produce an implementation; which I did, on CNEWS, and I tested it
pretty thoroughly (though not on the live Usenet, of course, because this
is one experiment you cannot do live).

However, it transpired that it was not so easily done in INN, and so we
backed off and left the full works as an ideal to be aimed at (not even a
SHOULD), and made the actual requirements much smaller. Indeed, if you
just implement it as a clone of newgroup (which is surely easily added to
any existing server) you are still conformant (well, you break a "SHOULD",
but when did that ever stop anybody :-( ).

So the question is whether you can foresee how it would be introduced.
Essentially, it will not prosper unless the users press for it. But if it
is at least in the standard, then some hierarchy-admin can be persuaded to
give it a go (and it removes the argument of the naysayers). Of course he
will still have to follow it up with a newgroup, and rmgroups after a
suitable waiting period, and that procedure might well have to continue
for years (indeed, the 'rmgroup' after the decent interval is still a MUST
in my draft for ever). It would be up to hierarchy-admins to decide when
it was safe to abandon the newgroup.

But then, once a few mvgroups were out, users would start to complain to
their news servers that the mvgroup had not been honoured (just as they
frequently have to complain that newgroups have not been honoured at
present). Sadly, the way Usenet currently operates, users leaning on
servers admins is regarded as a normal part of the process. There are
some excellently run sites out there, but many more (usually run by big
companies who are more concerned with making money and keeping up with
the latest fashions that in providing a decent service) that are not.

Anyway, once mvgroups are sort-of-working users will notice that articles
to the old group are not automatically appearing in the new (and they will
contrast that with reports of other providers who have managed to do it).
More leaning on providers....

So yes, it is going to be a slow process - that is inevitable. But it
won't happen at all unless someone sets it going by either creating a
chicken or creating an egg. We have the oppotunity to do that. If we
don't do it now, then nobody ever will.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: Proposing a new editor for the USEPRO document
Organization: The Eyrie
Date: Fri, 01 Dec 2006 12:44:08 -0800
Sender: [email-masked]

>> I am *delighted* that the de.* checkgroups includes de.alt.*.

> That gives you about three binary groups (de.alt.dateien.*), though.

That's fine.

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: USEFOR-11 troubles
Date:  Fri, 01 Dec 2006 20:46:33 +0100
Organization:  <URL:http://purl.net/xyzzy>
Lines: 19
Sender: [email-masked]

Russ Allbery wrote:
 
> For <diag-keyword>, I would move the acceptable list of keywords
> (SEEN, POSTED, and MISMATCH) into USEFOR with *brief* descriptions
> of their meaning and leave to USEPRO only the specific instructions
> of how to apply those meanings.  This would also simplify writing
> USEPRO, since USEPRO is laid out as a list of instructions and
> doesn't lend itself well to a digression on providing additional
> syntax for Path <diag-keyword>s.

That went through a rather complex poll - 2nd poll - rough consensus
procedure, and while I don't recall exactly who said what my proposal
to solve it in USEFOR didn't make it.  Of course it's now harder to
list all details in USEPRO.  The theory was that it's easier to add
further <diag-keyword>s later.  Let's stick to that decision.  Maybe
remove the obscure X-keywords as unnecessary.

Frank

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: Proposing a new editor for the USEPRO document
Date:  Fri, 01 Dec 2006 20:30:52 +0100
Organization:  <URL:http://purl.net/xyzzy>
Lines: 11
Sender: [email-masked]

Russ Allbery wrote:

> I am *delighted* that the de.* checkgroups includes de.alt.*.

That gives you about three binary groups (de.alt.dateien.*), though.
 
> If one really wishes to have an administratively separate hierarchy,
> it's far better to do what it.* did in creating it-alt.*.

There won't be any mass renamings, let alone without 'mvgroup'... :-)

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: USEFOR-11 troubles
Organization: The Eyrie
Date: Fri, 01 Dec 2006 10:21:26 -0800
Sender: [email-masked]

Frank Ellermann <[email-masked]> writes:

> Hi, Usefor-11 didn't make it in the first attempt, it got two [DISCUSS]:

> https://datatracker.ietf.org/public/pidtracker.cgi?command=print_ballot&ballot_id=2253&filename=draft-ietf-usefor-usefor

> Sam proposes to use a normative reference to USEPRO.  Russ Houley's
> [DISCUSS] is in a similar direction, he wants to make sure that the
> "security considerations" in USEPRO (referenced by USEFOR) are not
> lost, i.e. published in a RFC, not only in an I-D.

Reviewing USEPRO brought home to me that it's not possible to implement a
useful piece of Netnews software other than a pure article reader with
only the information in USEFOR.  That doesn't necessarily mean that USEFOR
can't advance separately as an underlying format which is then used by a
later standard, but that's definitely the approach we'd have to take to
sever the documents, which means that USEFOR will need to have no
normative references to USEPRO.

There are several that seem rather normative to me having just gone
through all of USEPRO, apart from the security considerations.  Section
3.1.4 says that control messages are defined in USEPRO, section 3.1.5
defers to USEPRO for the definition of <diag-keyword>, section 3.1.6
defers to USEPRO for the content of the Subject header (and that reference
I think could simply be dropped, as the only place that USEPRO discusses
the Subject header is in the context of a followup and then only as a
MAY), section 3.2 states that further requirements for optional header
fields are in USEPRO, section 3.2.3 defers to USEPRO for valid control
message <verb>s, and section 3.2.12 defines the action of Supersedes in
terms of cancel control messages defined in USEPRO.

On the surface, it's hard to argue against some or most of those being
normative references.

Control messages are the hardest problem here, and I expect are the most
likely to prompt a DISCUSS.  At the least, I think all article syntax
including acceptable values for control message <verb> and for Path
<diag-keyword> should be defined in USEFOR to make it a standalone format
document.  That's actually easier than it sounds, since for control
message verbs *all* values that fit the current syntax are acceptable and
interpretation is up to the protocol being used, so there's no need to
defer to USEPRO as aggressively as USEFOR currently does.

For <diag-keyword>, I would move the acceptable list of keywords (SEEN,
POSTED, and MISMATCH) into USEFOR with *brief* descriptions of their
meaning and leave to USEPRO only the specific instructions of how to apply
those meanings.  This would also simplify writing USEPRO, since USEPRO is
laid out as a list of instructions and doesn't lend itself well to a
digression on providing additional syntax for Path <diag-keyword>s.

The other references could be dropped or downgraded in their wording
easily except Supersedes, where the description would need to be rewritten
without reference to cancel control messages.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: Proposing a new editor for the USEPRO document
Organization: The Eyrie
Date: Fri, 01 Dec 2006 10:07:23 -0800
Sender: [email-masked]

Frank Ellermann <[email-masked]> writes:
> Forrest J. Cavalier III wrote:

>> What is the disadvantage with doing a separate experimental RFC?

> In theory fine.  But my interpretation of Russ' and your comments
> is that it's pointless:  The USEFOR drafts proposed 'mvgroup' for
> some years, so far nobody implemented it, no additional experiment
> necessary to confirm its death.

> Not volunteering to write a separate 'mvgroup' I-D:  Frank

> P.S.:  BTW, we need the WG Charter update soon, Russ proposed to
> add an I-D about news batches.

An I-D covering the XBATCH extension is mostly an NNTP I-D; the batch
format would be ancillary.  If I wrote such a thing, it would be as an
informational RFC via individual submission, not as a WG document.

The batch format really has nothing to do with USEFOR.  It's a transport
artifact.

Prior to working on that, though, I'd instead document the pgpverify and
PGPMoose protocols as informational RFCs, and those are more on-topic for
this WG.  But I have no idea when I'll have time.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: Proposing a new editor for the USEPRO document
Organization: The Eyrie
Date: Fri, 01 Dec 2006 10:05:06 -0800
Sender: [email-masked]

Frank Ellermann <[email-masked]> writes:

> The last time I watched a de.alt vs. de.!alt discussion most users
> claimed that de.alt is an integral part of de.all -- my personal
> impression (arguing on the side of the minority, who considered that as
> unfriendly takeover of de.alt).  We could remove that case as example in
> usepro-06 5.2.4.  </sigh>

As a news server administrator whose users want me to carry de.* but who
cares nothing at all about newsgroup creation policies in de.*, I am
*delighted* that the de.* checkgroups includes de.alt.*.  If it didn't, I
wouldn't carry de.alt.* at all.

If one really wishes to have an administratively separate hierarchy, it's
far better to do what it.* did in creating it-alt.*.

Date: Fri, 01 Dec 2006 17:49:07 +0100
From: Harald Alvestrand <[email-masked]>
To: Frank Ellermann <[email-masked]>, [email-masked]
Subject: Re: USEFOR-11 troubles
Sender: [email-masked]

Thanks Frank - I have "compose a reply" on my TODO list.
I'll CC the WG when I do.

           Harald

--On 1. desember 2006 16:41 +0100 Frank Ellermann 
<[email-masked]> wrote:

>
> Hi, Usefor-11 didn't make it in the first attempt, it got two [DISCUSS]:
>
> https://datatracker.ietf.org/public/pidtracker.cgi?command=print_ballot&b
> allot_id=2253&filename=draft-ietf-usefor-usefor
>
> Sam proposes to use a normative reference to USEPRO.  Russ Houley's
> [DISCUSS] is in a similar direction, he wants to make sure that the
> "security considerations" in USEPRO (referenced by USEFOR) are not
> lost, i.e. published in a RFC, not only in an I-D.
>
> Sam has some questions about mail2news gateways.  For some "more
> discussion" about the Article Format GMaNe offers a list archive
> with about 25,000 articles since 1997,  AFAIK mail2news gateways
> had no serious difficulties to transform 822 to 1036 format in the
> last decades, and MESSFOR to USEFOR is in essence the same issue:
>
> Add magic SP, emulate missing References as explained in 2822 if
> there's an In-Reply-To, fix most obs-stuff because it's illegal in
> NetNews, add missing Message-ID or better reject mails without a
> Message-ID if there can be other gateways of the same article, the
> works, some of the technical details will be discussed in USEPRO.
>
> Discussing gateway considerations in USEFOR would be a bad plan,
> but maybe we're forced to accept a normative USEPRO reference.  If
> that's the case we could also fix one [COMMENT] in the evaluation:
>
> s/192.0.168.1/192.0.2.1/  RFC 4408 got that right, no idea why
> we missed it for USEFOR, is that a missing feature in 'idnits' ?
>
> Frank
>
>
>

Date: Fri, 01 Dec 2006 11:45:10 -0500
From: Ken Murchison <[email-masked]>
Organization: Carnegie Mellon University
To: [email-masked]
Subject: Re: USEFOR-11 troubles
Sender: [email-masked]

Frank Ellermann wrote:

> Discussing gateway considerations in USEFOR would be a bad plan,
> but maybe we're forced to accept a normative USEPRO reference.  If
> that's the case we could also fix one [COMMENT] in the evaluation:
> 
> s/192.0.168.1/192.0.2.1/  RFC 4408 got that right, no idea why
> we missed it for USEFOR, is that a missing feature in 'idnits' ?

Right.  That entire example was already in my copy changed to:

posting-host = "posting.example.com:192.0.2.1"

I will make the change when the I-D gets to AUTH48, unless we need to 
put out a -12 to fix the reference to USEPRO.

-- 
Kenneth Murchison
Systems Programmer
Project Cyrus Developer/Maintainer
Carnegie Mellon University

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: Proposing a new editor for the USEPRO document
Date:  Fri, 01 Dec 2006 17:03:09 +0100
Organization:  <URL:http://purl.net/xyzzy>
Lines: 16
Sender: [email-masked]

Forrest J. Cavalier III wrote:

> What is the disadvantage with doing a separate experimental RFC?

In theory fine.  But my interpretation of Russ' and your comments
is that it's pointless:  The USEFOR drafts proposed 'mvgroup' for
some years, so far nobody implemented it, no additional experiment
necessary to confirm its death.

Not volunteering to write a separate 'mvgroup' I-D:  Frank

P.S.:  BTW, we need the WG Charter update soon, Russ proposed to
add an I-D about news batches.  If anybody here's interested we
could also add the news/nntp URI stuff, otherwise I'll intend to
send a 'publication request' on the day when USEFOR is approved.

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  USEFOR-11 troubles
Date:  Fri, 01 Dec 2006 16:41:31 +0100
Organization:  <URL:http://purl.net/xyzzy>
Lines: 30
Sender: [email-masked]

Hi, Usefor-11 didn't make it in the first attempt, it got two [DISCUSS]:

https://datatracker.ietf.org/public/pidtracker.cgi?command=print_ballot&ballot_id=2253&filename=draft-ietf-usefor-usefor

Sam proposes to use a normative reference to USEPRO.  Russ Houley's
[DISCUSS] is in a similar direction, he wants to make sure that the
"security considerations" in USEPRO (referenced by USEFOR) are not
lost, i.e. published in a RFC, not only in an I-D.

Sam has some questions about mail2news gateways.  For some "more
discussion" about the Article Format GMaNe offers a list archive
with about 25,000 articles since 1997,  AFAIK mail2news gateways
had no serious difficulties to transform 822 to 1036 format in the
last decades, and MESSFOR to USEFOR is in essence the same issue:

Add magic SP, emulate missing References as explained in 2822 if
there's an In-Reply-To, fix most obs-stuff because it's illegal in
NetNews, add missing Message-ID or better reject mails without a
Message-ID if there can be other gateways of the same article, the
works, some of the technical details will be discussed in USEPRO.

Discussing gateway considerations in USEFOR would be a bad plan,
but maybe we're forced to accept a normative USEPRO reference.  If
that's the case we could also fix one [COMMENT] in the evaluation:

s/192.0.168.1/192.0.2.1/  RFC 4408 got that right, no idea why
we missed it for USEFOR, is that a missing feature in 'idnits' ?

Frank

Date: Fri, 01 Dec 2006 10:15:24 -0500
From: "Forrest J. Cavalier III" <[email-masked]>
To: [email-masked]
Subject: Re: Proposing a new editor for the USEPRO document
Sender: [email-masked]

Frank Ellermann wrote:
> Russ Allbery wrote:
> 
>  [mvgroup] 
> 
>>My justification is that it's an entirely experimental feature
>>which has not been implemented in any news server distribution
>>that I'm aware of.  No one is currently issuing them
> 
> 
> <sigh>  Okay, so far for the dream that renaming groups could
> be anything else but a nightmare in future.  Even in standalone
> servers like GMaNe where it would only affect some "leafnodes".

What is the disadvantage with doing a separate experimental RFC?

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: Proposing a new editor for the USEPRO document
Date:  Fri, 01 Dec 2006 13:52:09 +0100
Organization:  <URL:http://purl.net/xyzzy>
Lines: 37
Sender: [email-masked]

Russ Allbery wrote:

 [mvgroup] 
> My justification is that it's an entirely experimental feature
> which has not been implemented in any news server distribution
> that I'm aware of.  No one is currently issuing them

<sigh>  Okay, so far for the dream that renaming groups could
be anything else but a nightmare in future.  Even in standalone
servers like GMaNe where it would only affect some "leafnodes".
 
> The one exception, which I think may have been a mistake to 
> retain, was the additional arguments to checkgroups.

The last time I watched a de.alt vs. de.!alt discussion most
users claimed that de.alt is an integral part of de.all -- my
personal impression (arguing on the side of the minority, who
considered that as unfriendly takeover of de.alt).  We could
remove that case as example in usepro-06 5.2.4.  </sigh>

> USEFOR, which I was treating as sacrosanct.

+1
 
>> Maybe we can add a recommendation to split control.cancel in
>> some appropriate way (for modem users like me when they try
>> to figure out if some rogue cancel bot is at it again)
 
> I think that's more a USEAGE thing.

On big servers it's almost impossible to find anything in their
control.cancel, unless it's divided into sets of TLHs.  If users
can't "control" who or what cancelled their articles it's a Bad
Thing, not only some USEAGE netiquette stuff, it's IMO technical.

Frank