Subject: Re: Proposing a new editor for the USEPRO document

From: Frank Ellermann <[email-masked]>
Date: Fri, 01 Dec 2006 13:52:09 +0100

Russ Allbery wrote:

 [mvgroup] 
> My justification is that it's an entirely experimental feature
> which has not been implemented in any news server distribution
> that I'm aware of.  No one is currently issuing them

<sigh>  Okay, so far for the dream that renaming groups could
be anything else but a nightmare in future.  Even in standalone
servers like GMaNe where it would only affect some "leafnodes".
 
> The one exception, which I think may have been a mistake to 
> retain, was the additional arguments to checkgroups.

The last time I watched a de.alt vs. de.!alt discussion most
users claimed that de.alt is an integral part of de.all -- my
personal impression (arguing on the side of the minority, who
considered that as unfriendly takeover of de.alt).  We could
remove that case as example in usepro-06 5.2.4.  </sigh>

> USEFOR, which I was treating as sacrosanct.

+1
 
>> Maybe we can add a recommendation to split control.cancel in
>> some appropriate way (for modem users like me when they try
>> to figure out if some rogue cancel bot is at it again)
 
> I think that's more a USEAGE thing.

On big servers it's almost impossible to find anything in their
control.cancel, unless it's divided into sets of TLHs.  If users
can't "control" who or what cancelled their articles it's a Bad
Thing, not only some USEAGE netiquette stuff, it's IMO technical.

Frank

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Fri, 01 Dec 2006 10:15:24 -0500

> servers like GMaNe where it would only affect some "leafnodes".

What is the disadvantage with doing a separate experimental RFC?

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 01 Dec 2006 17:03:09 +0100

> What is the disadvantage with doing a separate experimental RFC?

In theory fine.  But my interpretation of Russ' and your comments
is that it's pointless:  The USEFOR drafts proposed 'mvgroup' for
some years, so far nobody implemented it, no additional experiment
necessary to confirm its death.

Not volunteering to write a separate 'mvgroup' I-D:  Frank

P.S.:  BTW, we need the WG Charter update soon, Russ proposed to
add an I-D about news batches.  If anybody here's interested we
could also add the news/nntp URI stuff, otherwise I'll intend to
send a 'publication request' on the day when USEFOR is approved.

---

From: Russ Allbery <[email-masked]>
Date: Fri, 01 Dec 2006 10:05:06 -0800

Frank Ellermann <[email-masked]> writes:

> The last time I watched a de.alt vs. de.!alt discussion most users
> claimed that de.alt is an integral part of de.all -- my personal
> impression (arguing on the side of the minority, who considered that as
> unfriendly takeover of de.alt).  We could remove that case as example in
> usepro-06 5.2.4.  </sigh>

As a news server administrator whose users want me to carry de.* but who
cares nothing at all about newsgroup creation policies in de.*, I am
*delighted* that the de.* checkgroups includes de.alt.*.  If it didn't, I
wouldn't carry de.alt.* at all.

If one really wishes to have an administratively separate hierarchy, it's
far better to do what it.* did in creating it-alt.*.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: Russ Allbery <[email-masked]>
Date: Fri, 01 Dec 2006 10:07:23 -0800

> add an I-D about news batches.

An I-D covering the XBATCH extension is mostly an NNTP I-D; the batch
format would be ancillary.  If I wrote such a thing, it would be as an
informational RFC via individual submission, not as a WG document.

The batch format really has nothing to do with USEFOR.  It's a transport
artifact.

Prior to working on that, though, I'd instead document the pgpverify and
PGPMoose protocols as informational RFCs, and those are more on-topic for
this WG.  But I have no idea when I'll have time.

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 01 Dec 2006 20:30:52 +0100

Russ Allbery wrote:

> I am *delighted* that the de.* checkgroups includes de.alt.*.

That gives you about three binary groups (de.alt.dateien.*), though.
 
> If one really wishes to have an administratively separate hierarchy,
> it's far better to do what it.* did in creating it-alt.*.

There won't be any mass renamings, let alone without 'mvgroup'... :-)

---

From: Russ Allbery <[email-masked]>
Date: Fri, 01 Dec 2006 12:44:08 -0800

> That gives you about three binary groups (de.alt.dateien.*), though.

That's fine.

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 1 Dec 2006 20:12:41 GMT

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>The specific major technical changes that I am proposing are (and I may
>have missed some, and some that I consider minor others may not):

> * Drop mvgroup and initial articles.

I have dealt with mvgroup in a separate thread.

As for initial articles, I could easily be persuaded to drop them. They
were a fancy idea and the WG seemed willing to go along with them, so they
stayed.

> * Completely ban reinjection as permitted in the current draft.  This
>   allows a much sharper and clearer distinction between articles and
>   proto-articles and simplifies the descriptions of both posting and
>   injecting agents.  Instead, state that agents that wish to transfer
>   articles between two separate Netnews networks are gateways, are
>   subject to the requirements of a gateway, and must rewrite the articles
>   into proto-articles and inject them as normal.

Well it was you who said, when we discussed it a couple of years back,
that reinjection was going to happen whether we liked it or not and
occasionally it might even be sensible. So all that the present draft does
is to make sure that, if it is done, then it is done competently and
consistently (e.g. you don't alter the Injection-Date).

> * Drop all description of the batch format and don't allow batches to be
>   sent via application/news-transmission.  The batch format described in
>   the current draft isn't sufficient to describe existing practice anyway
>   (it doesn't handle compression), and sending batches via e-mail in
>   application/news-transmission is a new feature with marginal use.  I'll
>   describe the full batch format if I ever get around to writing up the
>   NNTP XBATCH extension.

Batch format was only ever used for UUCP, where it was indeed necessary.
It was in application/newstransmission because Henry Spencer put it there
when he registered it. And if you are going to use tunnelling through
email as a method of transporting news, which you might still want to do
in some unusual situations, then it might still be useful. OTOH, you could
still get the same effect by using a huge multipart/mixed with one article
in each part, so it is no huge deal to take the feature out. It is in any
case forbidden for sending to moderators.

I was not aware that the present description of batch format was deficient
(it is largely taken from Son-of-1036). I have quite some experience of
using batch format, both in UUCP and now in the interface between my
NEWNEWS suck feed and my server, since CNEWS performs rather poorly in the
face of large numbers of single-article batches. In any case, we only
describe the batch format to preserve the knowledge of how it is supposed
to be done (and, who knows, when Osama bin Laden hacks his way into the
infrastructure of the Internet and brings the whole edifice to its knees,
we may yet be glad that UUCP is still there :-). But if the batch format
description is materially wrong, then it needs to be fixed.

> * Make use of the new path diag-keywords optional but recommended.

My draft makes it MUST/SHOULD as an issue to be decided.

> * Add charset parameters for application/news-groupinfo and
>   application/news-checkgroups.  We can't just tell everyone to use
>   ASCII; they aren't now, and they're not going to.  Having explicit
>   charset information at least allows someone to do something sane while
>   still meeting the requirements of NNTP to use UTF-8.

No, that would be most unwise until we have resolved how internationalized
newsgroup-names are going to look, because they need to appear in the same
file, and they really need to appear there in the same form as they do
elsewhere.

Years ago, we had plans to put headers, including newsgroup-names, in
UTF-8. Now the EAI working group is trying to do UTF-8 headers for email.
It that endeavour succeeds (it is to be an Experimental Protocol according
to present plans), then it is most likely that Netnews would just pile in
behind it. In which case the Newsgroups File would be in UTF-8, which
would tie in nicely with NNTP. But I would not want to allow other
charsets in there in the interim, because we might have to undo it.

Now it you were to propose that the Checkgroups etc should henceforth be
in UTF-8 regardless, that would be a different matter which I might well
support.

> * Recommend use of the control hierarchy for storing control messages
>   rather than storing them in normal groups (upgraded from a NOTE).

I have no real problem with that. But it is introducing a *new*feature*,
horror of horrors :-(, and it is dealing with matters which are not seen
"on the wire" - even greater horrors.

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 1 Dec 2006 18:51:42 GMT

> * Drop mvgroup ...

OK, I think this is a big enough topic that it needs a separate thread
(maybe even an Issue number).

The WG decided right at the start that a mvgroup control message was
needed. I think the view was that it had to come sooner or later, that it
might cause some initial pain, but if it had to be done, then nothing was
gained by postponing it. Moreover, I don't think experimental protocols
are a good idea in Usenet, since the whole system really has to sing to
the same tune (that does not stop you doing experimental things in user
agents, such as trying out different fancy killfiles, or threading, or
whatever else, but not with serving agents).

I think the argument for mvgroup was that newsadmins could usually be
relied upon to create newgroups, but were extremely reluctant to do
rmgroups (I think even now some of them do not realise that PGVERIFY has
removed the main difficulty with bogus rmgroup commands). So it was
thought that including the rmgroup in the same command as the newgroup
would encourage newsadmins to consider them as a single operation. There
is no doubt that groups need to be renamed from time to time, but every
time someone proposes such a thing (often with good reason) the naysayers
on the config groups say "it will never work", and so it gets dropped (but
in spite of that, we have successfully renamed groups in uk.* on a small
number of occasions - and we have done the other 'impossible' thing, which
is to moderate groups in situ, as well).

Now ideally, a mvgroup should, at one stroke, remove the old group, add
the new, cause all existing articles magically to move to the new, and
take care of running threads. Sounds like hard work, and the WG challenged
me to produce an implementation; which I did, on CNEWS, and I tested it
pretty thoroughly (though not on the live Usenet, of course, because this
is one experiment you cannot do live).

However, it transpired that it was not so easily done in INN, and so we
backed off and left the full works as an ideal to be aimed at (not even a
SHOULD), and made the actual requirements much smaller. Indeed, if you
just implement it as a clone of newgroup (which is surely easily added to
any existing server) you are still conformant (well, you break a "SHOULD",
but when did that ever stop anybody :-( ).

So the question is whether you can foresee how it would be introduced.
Essentially, it will not prosper unless the users press for it. But if it
is at least in the standard, then some hierarchy-admin can be persuaded to
give it a go (and it removes the argument of the naysayers). Of course he
will still have to follow it up with a newgroup, and rmgroups after a
suitable waiting period, and that procedure might well have to continue
for years (indeed, the 'rmgroup' after the decent interval is still a MUST
in my draft for ever). It would be up to hierarchy-admins to decide when
it was safe to abandon the newgroup.

But then, once a few mvgroups were out, users would start to complain to
their news servers that the mvgroup had not been honoured (just as they
frequently have to complain that newgroups have not been honoured at
present). Sadly, the way Usenet currently operates, users leaning on
servers admins is regarded as a normal part of the process. There are
some excellently run sites out there, but many more (usually run by big
companies who are more concerned with making money and keeping up with
the latest fashions that in providing a decent service) that are not.

Anyway, once mvgroups are sort-of-working users will notice that articles
to the old group are not automatically appearing in the new (and they will
contrast that with reports of other providers who have managed to do it).
More leaning on providers....

So yes, it is going to be a slow process - that is inevitable. But it
won't happen at all unless someone sets it going by either creating a
chicken or creating an egg. We have the oppotunity to do that. If we
don't do it now, then nobody ever will.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Russ Allbery <[email-masked]>
Date: Sat, 02 Dec 2006 00:04:58 -0800

Charles Lindsey <[email-masked]> writes:

> Moreover, I don't think experimental protocols are a good idea in
> Usenet, since the whole system really has to sing to the same tune (that
> does not stop you doing experimental things in user agents, such as
> trying out different fancy killfiles, or threading, or whatever else,
> but not with serving agents).

That's not an argument against experimental protocols.  That's an argument
against ever introducing anything new, since there's no way that you could
ever introduce anything new and have the whole system immediately sing the
same tune.  This is actually *not* the case for control messages, which
are particularly amenable to different servers doing different things and
have, from the start, allowed the introduction of completely new control
messages without hurting anything.

> take care of running threads. Sounds like hard work, and the WG
> challenged me to produce an implementation; which I did, on CNEWS, and I
> tested it pretty thoroughly (though not on the live Usenet, of course,
> because this is one experiment you cannot do live).

It's implementable, and I do appreciate you writing an implementation and
seeing how it can be done.  Unfortunately, the implementation you did is
in a news server that's not widely used and not particularly maintained,
so it doesn't count much towards deployed base, although it does count
towards implementation experience.

However, moving articles between groups after they're stored pose weird
protocol concerns that I am not at all convinced the current proposal has
adequately dealt with, even if it were widely implemented.  For example,
unless you rewrite the Newsgroups header of the old articles or add a
Followup-To header or followups to articles that were moved from the old
group will be sad (if not on the local server then on someone else's
server).  Articles crossposted into the old group will need to have their
Xref headers updated (and therefore will have to have their overview
records regenerated) or crosspost chasing into the new group will not work
properly.  News readers need to be told somehow that the group has moved
so that they can update subscriptions automatically, and there's currently
no facility for doing that (we'd want to at least standardize the alias
flag in conjunction with this).

When I can come up with three problems like that just off the top of my
head, my impression is that the concept is not fully baked and doesn't
work well with the rest of Usenet without more elaboration of exactly how
it should be done.

Incidentally, the current USEPRO draft contradicts itself; it first says
that the articles MUST NOT be altered in any way when they're moved, and
then says that the Xref header may contain entries for the new group.

> backed off and left the full works as an ideal to be aimed at (not even
> a SHOULD), and made the actual requirements much smaller. Indeed, if you
> just implement it as a clone of newgroup (which is surely easily added
> to any existing server) you are still conformant (well, you break a
> "SHOULD", but when did that ever stop anybody :-( ).

Implementing mvgroup as a clone of newgroup, or even a clone of newgroup
and a delayed rmgroup, is pointless.  That doesn't solve any of the
problems that mvgroup was introduced to solve.  Unless readers are
automatically redirected to the new newsgroup and, ideally, the old
articles are also preserved, there's no reason to bother with it, since it
won't be any better than what's already available now.

Honestly, I think the place to start would be standarizing the alias flag
in LIST, introducing a new NNTP response code to GROUP that would act like
an HTTP redirect, and defining the NNTP behavior.  If that were done, the
mvgroup control message implementation would fall out fairly naturally.
(Although introducing a new GROUP response code would be tricky; extension
negotiation mostly doesn't work that way.  One might have to introduce a
new GROUP command to do this properly.  It's going to be rather tricky to
do this in a way that's really effective.)

> Essentially, it will not prosper unless the users press for it. But if
> it is at least in the standard, then some hierarchy-admin can be
> persuaded to give it a go (and it removes the argument of the
> naysayers).

This to me is not a persuasive argument for inclusion of a feature in the
base document.  mvgroup control messages are entirely harmless at sites
that don't implement them; there's nothing about experimental status
instead of standards track that prevents a hierarchy administrator from
giving them a try.

---

From: Russ Allbery <[email-masked]>
Date: Sat, 02 Dec 2006 00:40:05 -0800

Charles Lindsey <[email-masked]> writes:
> Russ Allbery <[email-masked]> writes:

> occasionally it might even be sensible.

In general, after going through the entire draft, I changed my opinion or
found a better solution to several problems that I'd previously not seen a
good way of handling.  It's a very useful exercise; it gave me a new
perspective on the protocol as a whole.

The point that I hadn't realized before is that defining systems that are
trying to do reinjection as gateways not only makes logical sense (they're
gatewaying between two disconnected Netnews networks), it produces all the
right results in the protocol, applies the correct warnings and
restrictions for doing reinjection safely, and and makes the whole
protocol description fall out more neatly.  It also puts the entire onus
for getting it right on the posting agent that's doing reinjection rather
than requiring the injecting agent to guess at different cases, which
places the complexity on the agent that's doing the unusual thing.

> Batch format was only ever used for UUCP, where it was indeed necessary.

It's also used by the NNTP XBATCH extension.  Not that I think many people
are using that (it was added to INN largely because it was trivial to do
given that INN already had batch support because of its UUCP support), but
it's rather neat and is the only current way to transfer compressed news
via NNTP.

> It was in application/newstransmission because Henry Spencer put it
> there when he registered it.

Ah, that explains it.

Well, I'd love to know if anyone is actually using it.  So far as I know,
no one is even using application/news-transmission, but of course I may
not have heard about it.  From the perspective of our draft, it's a very
odd sideline that adds about a page of text for something that essentially
no one is going to care about, and adds a few points of complexity that
we've had to deal with before (such as how the batch format counts bytes).

> And if you are going to use tunnelling through email as a method of
> transporting news, which you might still want to do in some unusual
> situations, then it might still be useful. OTOH, you could still get the
> same effect by using a huge multipart/mixed with one article in each
> part, so it is no huge deal to take the feature out. It is in any case
> forbidden for sending to moderators.

Yeah, that's the way I felt about it.

> I was not aware that the present description of batch format was
> deficient (it is largely taken from Son-of-1036). I have quite some
> experience of using batch format, both in UUCP and now in the interface
> between my NEWNEWS suck feed and my server, since CNEWS performs rather
> poorly in the face of large numbers of single-article batches.

I'm surprised, given that you're using UUCP, that you're not using
compressed batches.  I know that most of the INN users who have submitted
bugs and feedback about INN's UUCP batch support are.

The generalized batch format starts with a line of the form:

    #! <processor>

If <processor> is rnews, that line is followed by a byte count of an
article.  Other defined values for <processor>, however, are:

    cunbatch (far and away the most common)
    c7unbatch
    gunbatch

cunbatch indicates that the batch is compressed with standard Unix .Z
compression (although in practice I bet a lot of cunbatch batches are
actually gunbatch batches).  gunbatch indicates that it's compressed with
gzip.  c7unbatch indicates that it's compressed and then a content
transfer encoding is applied to make the data 7-bit; the content transfer
encoding used is unique to news batches so far as I know (it is not base64
or uuencode).  It is "defined" as follows:

**  The encoding uses characters from 0x20 (' ') through 0x7A ('z').
**  (That fits nicely into the UUCP 'f' protocol by Piet Beertema.) First,
**  expand three eight-bit charcters into four six-bit ones.  Collect
**  until we have 13, and spread the last one over the first 12, so that
**  we have 12 6.5-bit characters.  Since there are very few half-bit
**  machines, collect them into pairs, making six 13-bit characters.  We
**  can do this as A * 91 + B where A and B are less then 91 after we add
**  0x20 to make it printable.
**
**  And if you thought that was unclear, then we won't even get into the
**  terminating conditions!

Obviously c7unbatch is not horribly useful these days, but INN still does
support it.  I don't know if it's still used in the wild.

Once you apply the initial processor, you then reiterate on the enclosed
contents, which will generally then begin with #! rnews.

The full batch protocol is not horribly useful when transmitting batches
via e-mail, since you have to encode the e-mail to 8bit anyway and then
you lose most or all of the benefit of compression.  I can see why Henry
would have omitted it in a MIME type registration when those registrations
were basically only for e-mail.  However, when using batches over
binary-clean channels, such as UUCP, it's quite helpful.

> No, that would be most unwise until we have resolved how
> internationalized newsgroup-names are going to look, because they need
> to appear in the same file, and they really need to appear there in the
> same form as they do elsewhere.

No, you can convert between character sets.  But only if you know what
character set you're dealing with.

Tagged data is *always* better than untagged data.  If you know the
charset of the data, you at least have a fighting chance at doing
something with it if it's not in the charset that you need.  If you don't
know the charset of the data, you're doomed.  There is never any reason
not to say what the charset is; it never hurts you, and it stands a good
chance of helping you.

Saying that all checkgroups and newgroup group descriptions are in ASCII
is not helpful, because they're not.  Saying it in our document won't make
it so; it will just confuse implementors when they discover that writing
to our document produces software that doesn't work in the real world.
People are sending non-ASCII newsgroup descriptions right now and aren't
going to stop, since their languages cannot be represented in ASCII.

Given that, the choice is between descriptions tagged with a character set
and descriptions that are in some random 8-bit character set you have to
guess based on the hierarchy the control message affects.  I think that's
an easy choice.

Knowing the character set may not get us everything we need later.  It's
possible that, for reasons of normalization, we will eventually have to
require checkgroups to be in UTF-8.  However, adding a charset parameter
does nothing to hinder that and actually makes it easier to specify and
detect.  If it turns out that's what we have to say, then we can require
that the charset parameter always be either ASCII or UTF-8.  In the
meantime, implementors need to do something with those non-ASCII
descriptions that come right now with ASCII newsgroup names, and in fact
people are already implementing charset guessing code so that they can
convert incoming newsgroup descriptions into UTF-8.  This parameter would
allow us to encourage implementors to make a simple change in their
checkgroups message format and thereby make life massively easier for
those trying to maintain a newsgroups file in a single encoding and
thereby comply with an NNTP SHOULD.

> UTF-8. Now the EAI working group is trying to do UTF-8 headers for
> email.  It that endeavour succeeds (it is to be an Experimental Protocol
> according to present plans), then it is most likely that Netnews would
> just pile in behind it. In which case the Newsgroups File would be in
> UTF-8, which would tie in nicely with NNTP. But I would not want to
> allow other charsets in there in the interim, because we might have to
> undo it.

There's only something to undo if one assumes that adding the charset
parameter permits non-ASCII checkgroups when they aren't currently
permitted.  Since that horse has already left the barn, I don't see
anything new introduced by adding the charset parameter that would need to
be undone later in such an eventuality.

> in UTF-8 regardless, that would be a different matter which I might well

That's a much harder migration for existing control message senders, and
therefore much less likely to succeed, than getting them to just add the
charset that their descriptions are already written in.  The new MIME
types are already going to be a hard sell; I almost proposed taking them
out, and only didn't when I contemplated trying to write a formal
description of the newgroup control message body searching that news
servers do now.

> "on the wire" - even greater horrors.

This isn't a new feature -- it's already implemented by every widely used
news server that I'm aware of.  It's a feature not described in RFC 1036,
but there are a ton of those.  And the feature is indeed seen on the wire;
it's seen in the dialog between the reading agent and the serving agent.

---

From: Russ Allbery <[email-masked]>
Date: Sat, 02 Dec 2006 01:09:38 -0800

> then says that the Xref header may contain entries for the new group.

Oh, I see, it's not a contradiction; I just had to squint at it right.
It's talking about articles that arrived *after* the mvgroup message, and
I was thinking about articles that arrived *before* it.

---

From: Richard Clayton <[email-masked]>
Date: Sat, 2 Dec 2006 12:40:44 +0000

In message <[email-masked]>, Charles Lindsey
<[email-masked]> writes

>gained by postponing it. 

well we seem to have lasted many years since then without it -- so I'm
quite happy to see it ignored even longer

>If we

I'm happy with that view too

- -- 
richard                                                   Richard Clayton

Those who would give up essential Liberty, to purchase a little temporary 
Safety, deserve neither Liberty nor Safety. Benjamin Franklin 11 Nov 1755

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 4 Dec 2006 14:19:20 GMT

>>> * Completely ban reinjection as permitted in the current draft....

>gatewaying between two disconnected Netnews networks), ....

I can imagine situations in which reinjection is nothing like gatewaying,
but I need to see your draft before commenting further.

>>>   sent via application/news-transmission.  ...

>it's rather neat and is the only current way to transfer compressed news

That look like a good reason to write an NNTP extension document.

>Ah, that explains it.

Yes, if we remove it from there, then it needs an explicit mention that it
is a change from the current news/transmission definition. I would have
no great problem with doing that.

And in that case, the batch format is only relevant for describing the
response to the 'sendme' control message (and I hope you are not going to
remove mention of the 'ihave' and 'sendme' control messages; they may be
of mostly archaeological interest now, but they need to be formally
documented somewhere, and I was still seeing them - leaked from some
Japanese site - around 4 years ago, so I suspect they are still around).
Also, the ghastly 'to' hack in newsgroup-names needs to be mentioned, if
only because we mention it in Usefor and refer to USEPRO for the gory
details.

>...  From the perspective of our draft, it's a very
>no one is going to care about,...

Eh? The batch format is described in 16 lines (though it could do with a
mention of compression). If you want to remove the whole
ihave/sendme/batch/to business to an Appendix, then that would be fine,
but you can't just throw it away.

>> deficient (it is largely taken from Son-of-1036)....

>compressed batches.

No, I am not using UUCP now. I almost certainly used compressed batches
when I last did.

>>>   application/news-checkgroups....

>> same form as they do elsewhere.

There was a time, round about when we dropped the idea of
internationalized newsgroup-names, when people actually proposed all sorts
of bizarre encodings of newsgroup-names within ASCII. I think, with the
advent of the EAI WG, the risk of doing it that way has finally receded
which gives us a bit more flexibility. But that was the time when I wrote
in that restriction for checkgroups.

I think the least you need to say is that, to be compatible with present
software, any charset specified here MUST have ASCII as a subset (no
EBCDIC!). And maybe should/SHOULD be UTF-8 (with mention of NNTP as the
reason).

With that restriction, I think a charset parameter might now work.

>There's only something to undo if ...

Yes, but I was trying to avoid putting in things that one might have to
'undo' later.

>news server that I'm aware of.

Sure, but not always in that form. But surely the point is that that form
should NEVER appear 'on the wire' and thus is better not normative. It is
really a private matter between servers and their clients (though it is a
convention that shold be sidely encouraged, hence my NOTE). But I am not
really arguing against it.

Of more concern to me is the behaviour of NNTP agents which do not send
control messages to users who ask for messages in some particular group.
That behaviour is not required even by RFC 3977, but it seems endemic, and
leads to the problem that Frank just mentioned that you are forced to
bring down the complete control.cancel group if you want to see actual
cancel messages for some reason. But that issue is outside of our scope, I
think.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 4 Dec 2006 13:15:25 GMT

>necessary to confirm its death.

The reason nobody implemented it is that they wern't meant to until the
draft became a standard. This is a new feature, and people who implement
new features before they are finally and fully defined usually get into
trouble [1]. It was always envisaged that introducing it would take quite
some time after the standard appeared, hence the provision of transitional
arrangements.

[1] For example, there are some people using an earlier form of
Injection-Info. It would have been better if they hadn't.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 4 Dec 2006 13:22:24 GMT

>far better to do what it.* did in creating it-alt.*.

But de.* didn't. The feature was originally put in at their behest.

The other hierarchy that I know of which currently issues separate
checkgroups for a sub-hierarchy is cn.*. I don't know how they prevent
these from deleting everything else in cn.*, but presumably they have
managed it somehow.

The feature should be pretty easy to implement (it easily translates into
a wildmat). It would then be up to the admins of de.* and cn.* to lean on
providers to implement it (I imagine cn.* would find it easier).

---

From: Russ Allbery <[email-masked]>
Date: Mon, 04 Dec 2006 10:33:22 -0800

> these from deleting everything else in cn.*, but presumably they have

I don't recall ever seeing a checkgroups for cn.*, only for cn.bbs.*.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 4 Dec 2006 19:27:47 GMT

>adequately dealt with, even if it were widely implemented.

No, USEPRO make it clear that you cannot expect to move articles around,
except when the target group is empty (this is not for merging groups).
Essentially, you can keep the articles physically in the old group and
rename the directory concerned, or move them bodily to the newly created
group. Only one group ever exists on the server, the other is just an
alias for it (which might happen either way around). The nature of the
alias (at least on CNEWS, though probably on others) is that whichever
alias you ask to see the group by (whether for finding old articles or storing
new ones, it all happens in the one group).

>  For example,
>server).

Absolutely not! Otherwise threads are going to get hopelessly confused,
and you break the fundamental rule of Usenet that an article appears
exactly the same (modulo Path, Xref, etc) whichever server you find it on.
So once a thread starts, it remains in its original group (unless
followups are manually changed). Some users may believe they are reading
it in newgroup, and some in oldgroup (and may be slightly confused if they
inspect the Newsgroup header too closely). But whichever is the case, if
you followup it will appear to you to be in your group and to other Usenet
users in whichever group their own server has put it in. In fact, an NNTP
client cannot actually tell which of the two his server really keeps it in
unless it inspects the Active File, or looks closely at the Xref. If, at
some stage, users change their subscription from the old group to the new,
they will still see exactly the same articles.

>  Articles crossposted into the old group will need to have their
>properly.

I think the best strategy during the overlap period will be to include
both groups in the Xref (the article numbers will be the same, of course).
That way, User Agents that try to avoid showing the same article in two
groups will get it right either way.

Yes, if the .overview included the Xref, then it will have to be rebuilt.
But only if the Xrefs in the existing articles in oldgroup were updated,
which may not be necessary.

Anyway, all that was working in my implementation (I was surprised to find
it was over 5 years ago now). I fired all sorts of strange sequences of
operations at it and it worked fine. Indeed I am still using a couple of
local groups on my server that were moved at that time.

>  News readers need to be told somehow that the group has moved

Users will certainly have to change their subscription to the newgroup
sometime before it finally disappears, but I don't think it has to be
automatic. For sure, the actual renaming of the group will be a hot
topic of conversation on the group itself, so people should be well aware
of if. And the hierarchy admins, or the proponents of the change, can be
expected to post frequent reminders if people need to take action (which
is exactly what has to be done under the present newgroup followed by
rmgroup system, so we are certainly no worse off than at present).

>won't be any better than what's already available now.

Yes it does, because at the least it gives you the same as the previous
system, but with the added benefit that newsadmins are more likely to
accept both halves of the change if it comes as one command, and it opens
the way to more sophisticated implementations as time goes by and user
pressure starts to demand them. Old articles get preserved either way,
just that you may have to examine both groups to find them if your server
has done the minimal implementation.

Introducing it netwide is going to take time however we do it, but setting
the "entry level" low is the only way you will ever get it started. And it
it doesn't even get started, then for sure it will never get finished.

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Tue, 05 Dec 2006 02:07:37 -0500

Yes, this working group already saw explanations of mvgroup.

Your recent post doesn't address it being experimental, unproven,
expensive to implement, expensive to execute, and requiring
coordination of new servers and new clients.  Plus, you admitted
that there is no real benefit now, only some framework for adding
future benefit.  Low benefit with High cost doesn't have a chance
of getting implemented in widely used systems.

Technical merit is not the only consideration of what stays in and
stays out of standards track RFCs.  There have to be interoperating
implementations on widely used news servers.  (CNews is educational,
but it doesn't count as a widely used news server.)

---

From: Richard Clayton <[email-masked]>
Date: Tue, 5 Dec 2006 11:14:27 +0000

In message <[email-masked]>, Forrest J. Cavalier III
<[email-masked]> writes

>coordination of new servers and new clients.

Since my recent anti-mvgroup email didn't give any reasons, could I
stress the issue of clients...

... I use (and designed and helped write) what is effectively an offline
news client. This means that it has to have some of the properties of
news servers as well (storage, expiry, history, cross-referencing etc).

Even in these broadband days, this type of client is still useful when
connectivity to a server is not immediately available.

Dealing with a mvgroup in such a client would involve [and this is just
off the top of my head, there's probably a lot more]

(a) receiving the control message [it's not otherwise essential to see
any control messages] or extending the NEWGROUP response to be able to
tell the client about the event

(b) juggling around with the local database to relocate articles

(c) dealing specially with ancient articles that have been locally
preserved past their nominal expiry dates

(d) juggling around with the newsgroup "subscriptions" [and deducing
which of several servers should be used for accessing articles within
this group in the future]

and probably, when it was all worked through and it was realised that
clients were ignoring mvgroups unless the change was shoved in their
faces

(e) dealing with a new set of custom response codes for when the group
has apparently disappeared under the client's feet.

This is quite a lot of coding and database redesign for an essentially
trivial feature. I am pleased to see a draft with it removed.

- -- 
richard                                                   Richard Clayton

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 5 Dec 2006 12:53:38 GMT

>of getting implemented in widely used systems.

It is NOT expensive to implement in its minimal form, and on servers that
already implement aliasing in the 'classical' manner (i.e. as in Bnews and
Cnews, but not apparently INN) it is NOT expensive at all.

Being expensive to execute is irrelevant. Even if they occurred at the
rate of one a month, that hardly amounts to a noticeable quantity of
machine cycles.

Servers and clients? Only servers, I think (people who issue control
messages usually use custom scripts, and generating a mvgroup message is
pretty simple). Yes, different servers will behave differently, so on some
servers you will see two separate groups during the overlap period, and
on others you will see one group with the other aliased to it. That
situation cannot be avoided.

Let me ask just two questions:

Q1 (for the group)
	Other things being equal, do we WANT to see this happen?

If so, then we have a choice between doing it in this standard or as a
separate experimental protocol. So Q2 is mainly addressed at Russ:

Q2A
	It it appears in the Standard, will it be implemented in INN
	(not as first priority on Day 1, but later when more pressing
	changes are done)?

Q2B
	If it appears as an Experimental Protocol, will it be implemented
	in INN (not as first priority on Day 1, but later when more
	pressing changes are done)?

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 5 Dec 2006 12:33:44 GMT

>I don't recall ever seeing a checkgroups for cn.*, only for cn.bbs.*.

Yes, those are probably the ones I have seen. But presumably the rest of
cn.* _does_ exist, at which point if they ever did want to issue a
checkgroups they would have to ensure that the full cn.bbs.* stuff was
included within it (accurately). Or they could use our new checkscope
feature if it existed.

---

From: Russ Allbery <[email-masked]>
Date: Tue, 05 Dec 2006 10:15:40 -0800

> included within it (accurately). Or they could use our new checkscope

If it existed *and were widely deployed*.  My guess is that they'll never
be able to use it no matter what we say in our document, since that use of
it is not backward-compatible.

---

From: Russ Allbery <[email-masked]>
Date: Tue, 05 Dec 2006 10:22:27 -0800

> 	Other things being equal, do we WANT to see this happen?

Only in conjunction with full support in the NNTP protocol so that clients
can do the right thing, including automatic update of subscriptions.  I
think anything short of that is likely to be more confusing than useful.

However, control messages being what they are, I certainly have no
objection to people issuing mvgroup control messages and sites acting on
them if they disagree with me and think that it's useful.

> 	pressing changes are done)?

The answer in either case is identical: I don't intend to implement it
myself, but I'll review and commit a decent implementation if someone else
wants it and wants to use it.  That will be the same whether it's in a
standards-track document, an experimental document, or not standardized at
all.

I may have architecture objections to moving articles unless it's done
very carefully.  Even if they're moved into a completely new group, doing
this properly in INN is challenging and requires updating multiple
databases and cross-references properly.  There is also no way of doing
this properly with existing mvgroup-unaware clients without rewriting the
message to, at the very least, change the Xref header, which creates a
nasty conflict between the desire to never modify messages and the
practical implications of having posts for which normal operations fail.

---

From: Charles Lindsey <[email-masked]>
Date: Wed, 6 Dec 2006 16:25:44 GMT

> it is not backward-compatible.
Certainly cn.bbs.* could start using it immediately, since that would 
merely be what they are currently doing with that extra parameter added 
(ignored by old systems). Likewise de.alt.*.

The question is what cn.* and de.* might do, and I think you are saying 
that they would not dare use it until they were sure that sufficient 
servers would recognize and act on it. Or else they would continue doing 
what de.* have started doing which is to include all the de.alt.* groups 
as well (assuming they have an accurate list). That is a fair point. 
OTOH, it seems like a pretty easy feature to implement, so they might 
not have to wait that long.

I think we need to hear the views of some Germans on this one.

---

From: Charles Lindsey <[email-masked]>
Date: Wed, 6 Dec 2006 16:56:57 GMT

Richard Clayton wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 

I don't see why that complication is necessary. Either the server has 
both groups in separate existence during the overlap period, or it has 
them aliased to one group. What happens at present if a server creates a 
new group and later removes an old one on your system? Or if it suddenly 
aliases two groups together?

Presumably in the first case, your client tells its readers that a new 
group exists and invites them to subscribe. After which your readers 
read both groups and respond as normal (remember that in this case the 
server has adopted the minimal implementation, so it is not going to be 
as convenient for users as it might be - but at least nothing happens 
that should surprise them). Eventually the old group disappears, and 
presumably copies of articles in the old group are kept or disappear on 
the client according to what the user has configured, or whatever your 
client normally does with a rmgroup.

The second case is more interesting, but I would expect both groups to 
be visible on your client, and to contain the same articles. Whether 
local copies stored in your client were stored separately for the two 
groups or mirrored the aliassing on the server depends on how you have 
written it. Again, the user would be informed that a new group had 
appeared, and his best strategy in this case would be to adjust his 
subscription to the new one at the earliest convenient opportunity and 
to forget the old one.

I have just been experimenting with an aliassed group on my own machine 
with the two newsreaders available to me.

'nn' just recognises the existence of one group. If you try to direct 
its focus to the alias, it just goes to the underlying real group and 
that is what you get.

'Mozilla' appears to contain both groups and they contain the same 
articles. But it keeps separate .newsrc entries for them, so it is 
better for the user to retain a subscription to just one of them. He 
probably has to do a 'mark everything as read' when he first moves 
(having finished reading in the old one, or course).

But whatever hapens, the user is no worse off than with separate 
newgroup and rmgroup commands, and hopefully is much better.

---

From: Richard Clayton <[email-masked]>
Date: Wed, 6 Dec 2006 17:53:39 +0000

In message <[email-masked]>, Charles Lindsey
<[email-masked]> writes
>them aliased to one group. 

indeed so ... and my system is effectively a server as well, so
(depending on design decisions) it needs to alias or overlap in some
way. In order to do that it needs to become aware of the "mvgroup"
event, and that's what my issue (a) was saying

>What happens at present if a server creates a 
>new group 

it is reported via NEWGROUPS

>and later removes an old one on your system? 

my system remains unaware of removals (this is a flaw in NEWGROUPS)
unless a full LISTGROUP is requested.  In practice if there are articles
in the local database then the old group remains in existence so that
those articles can still be accessed ....  only once all the articles
vanish will the old group name be forgotten about

>Or if it suddenly 

I've never seen a server do that...  but the client would allocate
articles to groups in the local database by examining the Newsgroups:
header field, so it would not be relevant.

>group exists and invites them to subscribe. 

No -- that's just dumb (and yes I know that some Unix systems do that).
New groups (often with silly or vaguely obscene names) are created all
the time, so bothering users with trivia is a waste of time.

If you mean by "in the first case" the special state "when my system
somehow becomes aware of a mvgroup" then yes, some interaction with the
user will be necessary -- if only to explain that something weird and
wonderful has happened

>After which your readers 

I like the word "presumably" in that sentence.  The bit you have snipped
with tasks (b) (c) (d) (e) were all about the substantial amount of work
that would be required at a database level to make it all work

>be visible on your client, and to contain the same articles. 

that's one possible design, yes -- I can see others

>with the two newsreaders available to me.

which are both "online" newsreaders, viz: they don't have any local
article storage, so this isn't especially fascinating to me.  Try
Turnpike or Agent or similar...

>newgroup and rmgroup commands, and hopefully is much better.

for a small value of "much" and a big value of coding

- -- 
richard                                                   Richard Clayton

---

From: Russ Allbery <[email-masked]>
Date: Wed, 06 Dec 2006 11:09:42 -0800

> (ignored by old systems).

True, I can see where that would be useful.  I forgot that the default
behavior was hierarchy-scoped, not sub-hierarchy-scoped, so yes, the
current cn.bbs.*-only checkgroups would remove other cn.* groups, which
isn't desirable.

> Likewise de.alt.*.

de.* is a solved problem.  The de.* checkgroups includes de.alt.* and
there's no separate de.alt.* checkgroups.  They're already doing the best
possible thing; anything involving chkscope would be a step in the wrong
direction.

> servers would recognize and act on it.

I'm saying that the chances are very high that cn.* will never be able to
issue a checkgroups that omits all of cn.bbs.*.  Ever.  No matter what we
write in our document.

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 7 Dec 2006 15:10:47 GMT

>current cn.bbs.*-only checkgroups would remove other cn.* groups, which

Ah! I had thought they were sub-hierarchy-scoped (and that's what USEPRO
says - I have not yet read that far in your draft yet to see what that
says).  But I see that s-o-1036 agrees with you (though a bit vague) and
so does the CNews implementation. OTOH, if you are correct then how come
the cn.bbs.* checkgroups have not destroyed the rest of the cn.* hierarchy
long ago? Do you, as a newsadmin, get a message telling you that you ought
to delete umpteen groups in cn.* every time a cn.bbs.* checkgroups comes
out?

>issue a checkgroups that omits all of cn.bbs.*.  Ever.  No matter what we

In that case we might as well write it :-) .

At least, on sites where it was implemented, it might reduce the damage
done by ill-formed checkgroups mesages (such as cn.bbs.* currently appears
to be).

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 7 Dec 2006 16:39:37 GMT

>event, and that's what my issue (a) was saying

The whole 'mvgroup' idea was based on the assumption that existing User
Agents would behave in a sensible (i.e. non-mysterious) manner provided
the servers did sensible things (which would require at least some minimal
upgrade to the server to notice the arrival of the 'mvgroup'). If that
assumption is wrong, then indeed reexamination of the idea is in order.
Hence my reason to enquire whether some existing user agents such as
Turnpike would exhibit 'mysterious' behaviour in their present form (of
course, that does not prevent their implementors from adding extra bells
and whistle to deal with 'mvgroup' if they think it useful).

>>What happens at present if a server ...

>vanish will the old group name be forgotten about

Yes, full fledged servers normally retain articles from removed groups
until they have all expired by natural means. Posting of new articles is
normally prevented, though. Nothing 'mysterious' so far.

>header field, so it would not be relevant.

So even if your client issued a GROUP command followed by ARTICLE, it
would still store it under whichever group was in the Newsgroups header?
In which case, it would behave essentially as if the server was keeping
both groups separately, so the same as in my first scenario.

FWIW, when I created an alias for a group last week, I see that NEWGROUPS
tells me it has appeared, and shows its Active File line.

>the time, so bothering users with trivia is a waste of time.

My 'nn' reader tells me that. But fortunately my local server has strict
instructions not to accept newgroups from alt.*, so I am not bothered :-).

>user will be necessary -- if only to explain that something weird and

That, I think, is the crux of the matter. Can we agree that the only
people we need to worry about are those who are already subscribed to the
old group? In that case, they will surely be aware of what is going on. It
would be a very strange group whose 'culture' did not encompass
discussions about the status of the group itself. So for sure there would
be articles discussing the forthcoming change, announcements that it had
happened, advice that they would likely need to subscribe to the new group
(and maybe unsubscribe to the old, according to whether and how their user
agent was showing old articles in the new). And of course moaners and
trolls still decrying the whole thing long after it had happened :-).

So, again, nothing particularly 'mysterious' to users who were familiar
enough with how to subscribe and unsubscribe to groups, and to clear out
stored articles from groups they were no longer interested in.

>>.... Eventually the old group disappears, and 

>that would be required at a database level to make it all work

I don't see that. How is it different from what the database level would
have done if there had been a newgroup followed by a rmgroup? For from
what you have described to me, it would seem that is how Turnpike would
perceive it. Other user agents might perceive it differently, of course.

---

From: Richard Clayton <[email-masked]>
Date: Thu, 7 Dec 2006 17:25:05 +0000

In message <[email-masked]>, Charles Lindsey
<[email-masked]> writes

>>the client would allocate

yes, for consistency with the NEWNEWS case where this is essential

>old group? In that case, they will surely be aware of what is going on. 

Which is, of course, exactly what happens at the moment, and why the
world has survived into the 21st century without mvgroup and can
continue to do so

[with apologies to the chair for the repetition in the second comment,
the first one was actually contributing something new]

- -- 
richard                                                   Richard Clayton

---

From: Russ Allbery <[email-masked]>
Date: Thu, 07 Dec 2006 09:50:37 -0800

> so does the CNews implementation.

INN as well.

> OTOH, if you are correct then how come the cn.bbs.* checkgroups have not
> destroyed the rest of the cn.* hierarchy long ago? Do you, as a
> newsadmin, get a message telling you that you ought to delete umpteen
> groups in cn.* every time a cn.bbs.* checkgroups comes out?

Well, I as a news administrator don't carry cn.*, so I personally haven't
dealt with it one way or the other.  The code that maintains the
ftp.isc.org list has a more specific permissions configuration than news
servers often enforce that prevents cn.bbs.* checkgroups from affecting
anything outside of cn.bbs.* even if they're interpreted protocol-wise as
doing so.

---

From: Russ Allbery <[email-masked]>
Date: Thu, 07 Dec 2006 09:48:05 -0800

Charles Lindsey <[email-masked]> writes:

> until they have all expired by natural means.

Eh?  INN certainly doesn't.  I noticed that in the draft as well and was
very perplexed by it (and took it out of my version, since I've not heard
of a news server doing that).

Once the group is removed, it's removed.  You can't select it, read it,
look at it, anything.

> people we need to worry about are those who are already subscribed to
> the old group? In that case, they will surely be aware of what is going
> on. It would be a very strange group whose 'culture' did not encompass
> discussions about the status of the group itself.

This is one of these places where I think the Usenet-centric nature of the
discussion has hurt our ability to come up with a good general protocol.
This may be true on Usenet where there's some sort of formal group change
process, and which is only part of what Netnews is used for.  Internal
newsgroups may get moved without any notification in the group, and in
fact I would consider mvgroup a failure unless it supports that.  That's
the whole point, after all -- to make renaming a group a first-class
operation like creating or removing, one that behaves in a natural manner.

---

From: Russ Allbery <[email-masked]>
Date: Thu, 07 Dec 2006 10:16:47 -0800

>> protocol concerns that I am not at all convinced the current proposal
>> has adequately dealt with, even if it were widely implemented.

> except when the target group is empty (this is not for merging groups).

My concerns definitely still apply when the target group is empty.

> alias you ask to see the group by (whether for finding old articles or
> storing new ones, it all happens in the one group).

Except at least with INN, you can't post to an aliased group (which makes
sense and seems like the right semantics for aliasing in general to me),
so if the new group is aliased to the old group, users won't be able to
post except by going into the old group and posting there.  That seems
less than ideal.  That means that if you want the articles to be
accessible, you do need to move them from the old group to the new group
and then alias the old group to the new group, since otherwise no one's
going to be able to get at them.

>> For example, unless you rewrite the Newsgroups header of the old
>> articles or add a Followup-To header or followups to articles that were
>> moved from the old group will be sad (if not on the local server then
>> on someone else's server).

> exactly the same (modulo Path, Xref, etc) whichever server you find it

So instead you create a bunch of messages that can't be replied to without
a bunch of manual fiddling with the headers, which sparks user confusion.

I don't like either alternative here.  This is one of the parts of the
mvgroup proposal that bugs me.  Either way you go, the result doesn't seem
much better than just doing an rmgroup and a newgroup.  Different, but not
clearly better.

> it in newgroup, and some in oldgroup (and may be slightly confused if
> they inspect the Newsgroup header too closely). But whichever is the
> case, if you followup it will appear to you to be in your group and to
> other Usenet users in whichever group their own server has put it in.

I don't understand what you mean here.  If you follow up, you'll copy the
Newsgroups header field from the predecessor into the Newsgroups header
field of your follow-up, and then posting that message will fail because
you're trying to post to an aliased group.  It makes no difference in how
follow-ups are constructed what group your news reader thinks it's reading
(and in fact, if it uses NEWNEWS, it may not think it's reading any
particular group at all).

> In fact, an NNTP client cannot actually tell which of the two his server
> really keeps it in unless it inspects the Active File, or looks closely
> at the Xref.

You can't even tell with Xref unless you allow messages to be rewritten.
(And I don't see how you can get away without rewriting the message to
modify at least the Xref header field on INN.  Having Xref be wrong in a
message in INN causes a ton of problems, particularly with the recovery
tools.  It's like having a file system in an inconsistent state.  It's not
a stable configuration.)

>> Implementing mvgroup as a clone of newgroup, or even a clone of
>> newgroup and a delayed rmgroup, is pointless.  That doesn't solve any
>> of the problems that mvgroup was introduced to solve.  Unless readers
>> are automatically redirected to the new newsgroup and, ideally, the old
>> articles are also preserved, there's no reason to bother with it, since
>> it won't be any better than what's already available now.

> accept both halves of the change if it comes as one command,

I don't think I believe this.  If news administrators are acting on
control messages, there's no reason why they'd act on the newgroup and not
the rmgroup unless they specifically configured their server that way, and
if they did because they don't want to remove old groups on renamings,
they wouldn't act on the mvgroup anyway because it does the same thing.
Or they'd treat mvgroup like newgroup.

If news administrators are not acting on control messages (which is the
more common case), none of this matters.

> and it opens the way to more sophisticated implementations as time goes
> by and user pressure starts to demand them.

That sounds *so* much like experimental track to me.

> Old articles get preserved either way, just that you may have to examine
> both groups to find them if your server has done the minimal
> implementation.

If you implement mvgroup as newgroup and rmgroup, old messages are not
preserved.

---

From: Russ Allbery <[email-masked]>
Date: Thu, 07 Dec 2006 10:29:50 -0800

>> The point that I hadn't realized before is that defining systems that
>> are trying to do reinjection as gateways not only makes logical sense
>> (they're gatewaying between two disconnected Netnews networks), ....

> but I need to see your draft before commenting further.

Well, one of the reasons why I put it up on my web site was so that people
could look at it before the draft editor got around to publishing it.

> no great problem with doing that.

Okay.  That's the direction I'm leaning at the moment.

> response to the 'sendme' control message

Well, in my opinion, the batch format is not at all relevant to describing
the response to the sendme control message.  It's outside the scope of our
document to specify how articles are transmitted from one system to
another, and transmitting them in response to sendme is no exception.
Whether they're then sent by NNTP, by UUCP with batches, by magnetic tape,
or by someone walking next door with a floppy disk isn't part of the
sendme specification.  The control mesages are just there to arrange what
should be sent, not how.

That's one of the reasons why I took it out; the only thing in my draft
that referenced it was the application/news-transmission registration.

> (and I hope you are not going to remove mention of the 'ihave' and
> 'sendme' control messages;

No.  I argued for keeping them.

> only because we mention it in Usefor and refer to USEPRO for the gory

It's there in all the specificity that I've ever heard for it.  There
really isn't that much to say about it.

>> ...  From the perspective of our draft, it's a very odd sideline that
>> adds about a page of text for something that essentially no one is
>> going to care about,...

> mention of compression).

Plus additional wording in application/news-transmission.  It's not quite
a page, but it saved a noticable chunk of space.

> With that restriction, I think a charset parameter might now work.

Hm.  Yeah, that makes sense, and still allows for what's going on in
practice today.  Anyone have a suggestion on how one words "MUST have
ASCII as a subset" in fairly precise language?

>>> I have no real problem with that. But it is introducing a
>>> *new*feature*, horror of horrors :-(, and it is dealing with matters
>>> which are not seen "on the wire" - even greater horrors.

> Sure, but not always in that form. But surely the point is that that
> form should NEVER appear 'on the wire' and thus is better not normative.
> It is really a private matter between servers and their clients
> (though it is a convention that shold be sidely encouraged, hence my
> NOTE). But I am not really arguing against it.

It appears on the wire between reading agents and serving agents, and our
standard covers that protocol.  How serving agents talk to reading agents
is not a private matter.  It's part of what we're standardizing, even if
we don't have a tremendous amount we have to say about it and not many
restrictions we place on it.

> control messages to users who ask for messages in some particular group.

That's the way that all widely used news servers behave and there's years
of existing practice behind that.  I'm going to take a lot of convincing
to recommend something else.

> That behaviour is not required even by RFC 3977,

It's not a topic for NNTP; NNTP doesn't dictate how news servers assign
articles to groups.  It's a topic for USEPRO, which is why I addressed it
there.

> but it seems endemic, and leads to the problem that Frank just mentioned
> that you are forced to bring down the complete control.cancel group if
> you want to see actual cancel messages for some reason.

That's far and away the lesser of two evils, IMO.

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 07 Dec 2006 22:16:39 +0100

Russ Allbery wrote:

> Anyone have a suggestion on how one words "MUST have ASCII
> as a subset" in fairly precise language?

| A charset is said to be "ASCII compatible" if it encodes 
| the Unicode points u+0000 up to u+007F as the usual octets
| &x00 up to &x7F, and either uses these octets for no other
| purpose (e.g. Latin-1, windows-1252, and UTF-8), or offers
| clear indications when they are used as (a part of) other
| encodings (e.g. UTF-7, SCSU, and TBD).

TBD needs to be some CJK charset.  I'm almost sure that you
don't want SCSU, UTF-1, or UTF-7, and that could simplify it:

| A charset is said to contain US-ASCII as subset if it encodes
| the Unicode points u+0000 up to u+007F as the usual octets
| &x00 up to &x7F, and uses these octets for no other purpose.

Frank

---

From: Russ Allbery <[email-masked]>
Date: Thu, 07 Dec 2006 13:46:03 -0800

> | &x00 up to &x7F, and uses these octets for no other purpose.

We do need to support SJIS, though.  I thought that some of the Asian
encodings like that do shift between ASCII and non-ASCII with escape
sequences and use ASCII characters within the escaped portion.  But I'm
not sure.

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 07 Dec 2006 23:31:51 +0100

> We do need to support SJIS, though.

Tricky, maybe "and allows no other encoding for these code points".
That should eliminate UTF-7.  If SJIS is in, then UTF-1 is also in.

Frank

---

From: Ned Freed <[email-masked]>
Date: Thu, 07 Dec 2006 17:18:22 -0800 (PST)

> > don't want SCSU, UTF-1, or UTF-7, and that could simplify it:

How about EUC-JP as an example?

> We do need to support SJIS, though.

Shift-JIS charsets (there are several different ones) do not in general have
ASCII as a subset because there are two-octet sequences in them where the first
octet is >127 but the second is <127.

If memory serves, the various EUC charsets all have ASCII as a subset.

> I thought that some of the Asian

Now you're talking about the various charsets based on ISO 2022. Most if not of
these are 7bit and use various escape sequences to switch between single byte
US-ASCII and some other large coded character set encoded as two successive
US-ASCII characters. These clearly don't have US-ASCII as a subset.

However, all of these charsets are ASCII-compatible. Examples of charsets that
aren't ASCII-compatible are any UTF-16 variant, any of the EBCDIC charsets,
and the old national variant ASCII sets.

(What can I say, I've written a lot of charset conversion code.)

				Ned

---

From: Russ Allbery <[email-masked]>
Date: Thu, 07 Dec 2006 18:07:12 -0800

Ned Freed <[email-masked]> writes:

> However, all of these charsets are ASCII-compatible. Examples of
> charsets that aren't ASCII-compatible are any UTF-16 variant, any of the
> EBCDIC charsets, and the old national variant ASCII sets.

> (What can I say, I've written a lot of charset conversion code.)

I think that ASCII-compatible is what we actually need.  Basically, we
have to be assured that the newsgroup *names*, as opposed to the
descriptions, if read by a naive news server with no knowledge of
character sets, will still be correct.  As long as that condition is met,
it doesn't matter what weird stuff is going on in the descriptions as long
as the character set is properly tagged.

Maybe we should just say that rather than taking about the character set
properties.

---

From: Ned Freed <[email-masked]>
Date: Thu, 07 Dec 2006 18:21:28 -0800 (PST)

> Maybe we should just say that rather than taking about the character set

Seems reasonable - and I can't say that having a protocol specific discussion
of charset terminology and characteristics thrills me.

				Ned

---

From: Harald Alvestrand <[email-masked]>
Date: Fri, 08 Dec 2006 05:17:53 +0100

Strawman, idea. Disregard if nonsense.
Perhaps we should define the operation we want to succeed instead?

For instance, say that:

There is news software that will ignore the charset parameter and parse the 
message as if it was US-ASCII.

Therefore, the result of reading each octet of the message and setting bit 
8 to zero MUST be a valid message specifying the same newsgroup names, as 
long as all newsgroup names are in ASCII. There is no requirement that the 
newsgroup descriptions survive this treatment.

Examples of charsets that allow the message to satisfy this constraint are 
(for example) UTF-8, ISO-8859-1, Shift-JIS and ISO-2022-JP.

Examples of charsets that do not allow the message to satisfy this 
constraint are EBCDIC, UTF-16 and ISO-646-NO.

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 8 Dec 2006 15:20:34 GMT

>to recommend something else.

Sure. It is too entrenched to change now. I was bemoaning that it ever got
like that (as was Frank, I think).

>That's far and away the lesser of two evils, IMO.

And there I disagree with you. I think if I were doing things again from
scratch I would have the control messages visible in their own groups,
with an implict crosspost to the control.* groups as well. And then have a
Distribution 'control' for the benefit of people who did not want to see
them. But that is not a serious proposal for now (though it might be if
ever we try to standardize NOCEMs).

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 8 Dec 2006 16:29:10 GMT

>of a news server doing that).

CNEWS has been doing it for years (and BNEWS before it, I imagine). I am
surprised that INN doesn't do it, and would recommend that it did (it is
not hard). And for our draft it should be a SHOULD. It could be very handy
if a newsgroup gets removed by accident.

>Once the group is removed, it's removed.  You can't select it, read it,

Naturally, you disable posting, and even ihaving.

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 8 Dec 2006 15:12:46 GMT

>anything outside of cn.bbs.* even if they're interpreted protocol-wise as

Ah! So you effectively have a private hack in the ftp.isc.org code that
does essentally what the checkscope parameter is trying to do. And it
would seem that anyone who currently does carry cn.* must have implemented
a similar hack also, or else he ignores the cn.bbs.* checkgroups.

So it checkscope remains in the draft, and cn.bbs.* can be persuaded to
include it, then all the people currently using such a hack could take it
out and implement checkscope instead.

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 8 Dec 2006 17:08:16 GMT

>> | encodings (e.g. UTF-7, SCSU, and TBD).

I don't think you ought to be mentioning Unicode when defining "MUST have
ASCII as a subset". What you need is a code wherein the octets 0-127
always stand for the usual ASCII characters. UTF-8 and the ISO-8859 series
are the obvious examples. I think somebody said the EUC ones were also.

>sequences and use ASCII characters within the escaped portion.  But I'm

I don't think you can support anything using escapes, though.

Essentially, when a 'checkgroups' arrives, you take each line, delete the
1st TAB and everything after (but take note of any "(Moderated)" at the
end. Then you compare what you have got with what is in your active file,
and inform the newsadmin of any discrepancies, even offering to fix them
for him. That needs the newsgroup-names to be exactly the same as stored
in the active file. I doubt anyone using strange charsets in a checkgroups
has done anything that would break that behaviour, which is built into
most servers.

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 8 Dec 2006 16:57:55 GMT

>less than ideal.  ...

Yes, it is clear that the aliassing in INN is much more restrictive than
in CNEWS, making full implementation of mvgroup much harder.

>>> articles or add a Followup-To header...

>> and you break the fundamental rule of Usenet ...

>a bunch of manual fiddling with the headers, which sparks user confusion.

Not at all. Both groups MUST remain postable during the overlap, or else
there is indeed utter confusion. Keeping both groups in existence is one
way to do it, but an alias system that continues to allow posting is
better, on systems where it can be implemented that way.

>you're trying to post to an aliased group.  ...

Only on INN, which is why INN cannot implement it that way.

>the rmgroup unless they specifically configured their server that way,

But sadly it seems a lot of them ARE configured that way. Quite illogical,
but since when have news admins ever behaved logically :-( ?

>If you implement mvgroup as newgroup and rmgroup, old messages are not

No problem it there is a suitable delay between the newgroup and the
rmgroup (as USEPRO required), so that most oldgroup stuff will have
expired by then (and on CNEWS will aftually be kept until the normal
expiry time anyway).

---

From: Russ Allbery <[email-masked]>
Date: Fri, 08 Dec 2006 09:37:15 -0800

>> Well, I as a news administrator don't carry cn.*, so I personally
>> haven't dealt with it one way or the other.  The code that maintains
>> the ftp.isc.org list has a more specific permissions configuration than
>> news servers often enforce that prevents cn.bbs.* checkgroups from
>> affecting anything outside of cn.bbs.* even if they're interpreted
>> protocol-wise as doing so.

> would seem that anyone who currently does carry cn.* must have
> implemented a similar hack also, or else he ignores the cn.bbs.*
> checkgroups.

> include it, then all the people currently using such a hack could take
> it out and implement checkscope instead.

Well, no, I have a real authorization check in the ftp.isc.org code that
says that the people authorized to send checkgroups for cn.bbs.* aren't
allowed to affect the rest of cn.* *even if they claim to want to*, which
isn't replacable by the chkscope parameter.

But otherwise, yes, I see what you're getting at and it's fairly
convincing to me.