Subject: Re: charset of application/news-{groupinfo,checkgroups}

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 1 Jan 2007 16:20:43 GMT

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>Here's what I currently have:

>   The content of the application/news-groupinfo body part is defined
>   as:

...........

>         newsgroup-description
>                             = eightbit-utext *( *WSP eightbit-utext )
>         moderation-flag     = %x28.4D.6F.64.65.72.61.74.65.64.29
>                                  ; case sensitive "(Moderated)"
>         eightbit-utext      = utext / %d127-255

OK, <eightbit-utext> was clearly needed, but it still excludes those ASCII
characters not in <utext> notably CR, LF and HT, so you could not use
EBCDIC (where I am sure those octets correspond to useful characters) nor
any charset which could be escaped into or out of ASCII and in which those
octets had some other meaning when escaped.

>   This unusual format is backward-compatible with the scanning of the
>   body of newgroup control messages for descriptions done by Netnews
>   implementations that predate this specification.  Although optional,
>   the <newsgroups-tag> SHOULD be included for backward compatibility.

>   The <newsgroup-description> MUST NOT contain any occurrence of the
>   string "(Moderated)" within it.  Moderated newsgroups MUST be marked
>   by appending the case sensitive text " (Moderated)" at the end.

>   While a charset parameter is defined for this media type, most
>   existing software does not understand MIME header fields or correctly
>   handle descriptions in a variety of charsets.  Using a charset of US-
>   ASCII where possible is therefore RECOMMENDED; if not possible, UTF-8
>   [RFC3629] SHOULD be used.  Regardless of the charset used, the
>   constraints of the above grammar MUST be met and the <newsgroup-name>
>   MUST be represented using the same octets as would be used with a
>   charset of US-ASCII.

Yes, that almost gets you there, and maybe near enough. But it does result
in some oddities:

1. Suppose you specify a charset that works for the
<newsgroup-description> but renders the <newsgroup-name> as gibberish.

2. Suppose you specify a charset that can be escaped into or out of ASCII.
Then you need to ensure that it starts out in ASCII mode, and returns to
ASCII mode before the moderation-flag, if any.

3. Totally stupid charsets such as EBCDIC are excluded.

4. The ISO 8859-* series will work for now, but not if/when
<newsgroup-name>s in UTF-8 appear.

All those snags are implicit in the wording you have written, but are not
immediately evident to a reader not already familiar with these issues. So
it would be kinder to use a wording which made them more evident, even at
the expense of a little extra text. That would include explicit mention of
ASCII as a subset and of the alternative possibility of escaped charsets,
including the necessity to ensure they were somehow in ASCII mode at the
proper places.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Russ Allbery <[email-masked]>
Date: Mon, 01 Jan 2007 10:20:39 -0800

Charles Lindsey <[email-masked]> writes:
> Russ Allbery <[email-masked]> writes:

> OK, <eightbit-utext> was clearly needed, but it still excludes those
> ASCII characters not in <utext> notably CR, LF and HT,

By HT I assume you mean HTAB?  That's is added back in by WSP in the
newsgroup-description production, so only CR and LF are disallowed, which
is intentional.

> so you could not use EBCDIC (where I am sure those octets correspond to
> useful characters) nor any charset which could be escaped into or out of
> ASCII and in which those octets had some other meaning when escaped.

Right.

> <newsgroup-description> but renders the <newsgroup-name> as gibberish.

Surely this is obviously invalid?  It would mean that the newsgroup name
in the charset specified for the body of this type is different than the
newsgroup name in ASCII, which is precisely what's forbidden.

> 2. Suppose you specify a charset that can be escaped into or out of
> ASCII.  Then you need to ensure that it starts out in ASCII mode, and
> returns to ASCII mode before the moderation-flag, if any.

Well, no, you don't, provided that without the escaping the newsgroup name
is still the same as for ASCII.  If it isn't, then you have a problem
because there's no way to do the escaping before the newsgroup name and
that charset is therefore banned.  Which is, I believe, correct, since
existing software would choke on the initial escaping character.

> <newsgroup-name>s in UTF-8 appear.

Right.

> All those snags are implicit in the wording you have written, but are
> not immediately evident to a reader not already familiar with these
> issues. So it would be kinder to use a wording which made them more
> evident, even at the expense of a little extra text. That would include
> explicit mention of ASCII as a subset and of the alternative possibility
> of escaped charsets, including the necessity to ensure they were somehow
> in ASCII mode at the proper places.

As above, I think escaped charsets don't work as easily as you're
implying.  I'm not sure that I'm convinced for the rest, but if you have a
language suggestion in mind, I'd certainly consider it.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 1 Jan 2007 17:48:20 GMT

>I've since reviewed RFC 1036 and Son-of-1036.  The former makes the
>relayer name optional and requires that the message IDs be in the control
>header field.  The latter makes the relayer name mandatory and RECOMMENDS
>that the message IDs be put in the body.  I think the following is the
>most reasonable compromise; it makes the relayer-name mandatory if there
>are no message IDs (the form never allowed in RFC 1036 but according to
>Son-of-1036 the most widely used in practice), and if there are message
>IDs, permits the same syntax as RFC 1036.

I don't see why the optionality or otherwise of the <relayer-name> should
correlate in any way with whether the list of msg-ids is in the header or
the body (except to ensure that these headers never have empty content,
which does not sound like a particularly good reason).

So I would think it better to follow Son-of-1036 and make it obligatory
regardless. I think we can assume that Henry had sufficient experience of
the way Ihave and Sendme were used at that time that it is safe to follow
his lead.

In any case, the paragraph which follows explaining how to use these two
headers is written on the assumption that a <relayer-name> will be
provided.

If any pair of existing relayers is currently using this protocol without
<relayer-name>s, then good luck to them, and 'lang may their lum reek'.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 1 Jan 2007 17:32:25 GMT

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>I now have:

>   Exceptionally, control messages creating or removing newsgroups
>   (newgroup or rmgroup control messages, for example) SHOULD be relayed
>   if the affected group appears in its Newsgroups header field and the
>   sending agent would have supplied and the receiving agent would have
>   received the newsgroup affected by the control message had it existed,
>   even if it currently does not.

I presume that is intended to go after the current 2nd paragraph of 3.5.

In which case, since that existing paragraph clearly indicates that it is
a matter of the configuration of the sending and receiving relaying
agents, you might as well continue with that terminology. So perhaps:

"...if the affected group appears in its Newsgroups header field and the
sending agent and receiving relaying agents are both configured to relay a
newsgroup of that name (whether or not such a newsgroup yet exists)."

>for the propagation rule, the following under group control messages:

>   Group control messages affecting specific groups (newgroup and
>   rmgroup control messages, for example) SHOULD include the <newsgroup-
>   name> for the group or groups affected in their Newsgroups header
>   field.  Other newsgroups MAY be included in the Newsgroups header
>   field so that the control message will reach more news servers, but
>   due to the special relaying rules for group control messages (see
>   Section 3.5) this is normally unnecessary and may be excessive.

>the following under cancel control messages:

>   A cancel control message SHOULD have the same Newsgroups header field
>   as the message it is cancelling so that it will attempt to reach the
>   same news servers as the original message.

I think I am persuaded by Frank's argument to s/SHOULD/MUST/ in both those
cases (even though USEPRO has "SHOULD"). As to the two exceptions you
mentioned, a man who cancels an article in a test group might actually
prefer to receive confirmation that is had reached the usual
auto-responders, and spam cancellers are in some sense already breaking
the rules, so what the heck if they break some more? But even there, the
spam cancel needs to propagate everywhere the article propagated, and I do
not buy the argument about hierarchies that have special groups for
cancels. Surely NNTP servers are going to put them all in control.cancel
anyway, so that special group would appear to be empty anyway?

>and the following in Security Considerations:

>   Cancel control messages are not required to have the same Newsgroups
>   header field as the messages they are cancelling and, since they are
>   sometimes processed before the original message is received, it may
>   not be possible to check that they do.  This allows a malicious
>   poster to inject a cancel control message for an article in a
>   moderated newsgroup without adding an Approved header field to the
>   control message, and to hide malicious cancel control messages from
>   some reading agents by using a different Newsgroups header field so
>   that the cancel control message is not accepted by all news servers
>   that accepted the original message.

And I do not at all follow the scenarios you envisage there. If the cancel
arrives before its matching article at some server which needs to act on
it, who cares about checking its newsgroups header anymore? And a malicious
canceller who wanted to hide the cancel from certain sites would do better
to preload the Path header.

But if you s/SHOULD/MUST/, then none of this arises anyway.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 1 Jan 2007 17:35:07 GMT

In <[email-masked]> Frank Ellermann <[email-masked]> writes:

>Same here, a "checkgroups" listing _all_ newsgroups in the Newsgroups
>header field would be odd.  Get rid of "for example", the qualifier is
>confusing without "mvgroup".

Hmmmm! Please can somebody say what IS the correct Newsgroups header for a
checkgroups? I don't think any of our documents have mentioned this yet.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 1 Jan 2007 17:51:02 GMT

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>I'm convinced.  I now have:

>   These control messages are normally sent as point-to-point articles
>   between two sites and not then sent on to other sites.  Newsgroups
>   beginning with "to." are reserved for such point-to-point
>   communications and are formed by prepending "to." to a <relayer-name>
>   to form a <newsgroup-name>.  Articles with such a group in their
>   Newsgroups header fields SHOULD NOT be sent to any news server other
>   than the one identified by <relayer-name>.

Fine!

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 1 Jan 2007 20:52:37 GMT

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>As promised, here is the new section.  I also made the corresponding
>updates to the other sections (removing the description and replacing it
>with a reference to this section) and moved the Path header field example.

>3.2.1.  Constructing the Path Header Field

>   If a relaying or serving agent receives an article from an injecting
>   or serving agent that is part of the same news server, it MAY leave
>   the Path header field of the article unchanged.  Otherwise, every
>   injecting, relaying, or serving agent that accepts an article MUST
>   update the Path header field as follows.  Note that the Path header
>   field content is constructed from right to left by prepending
>   elements.

I think that is more restrictive than we intended. The idea was that if
there was a "farm" of servers at one site, splitting the various duties
(incoming articles, outgoing articles, local storage of articles) amongst
themselves in some manner, it was not necessary to include every server
the article actually passed through (i.e. we do not need to see, from the
outside, the detailed internal structure of that site).

>   1.  The agent MUST prepend "!" to the Path header field content.

>   2.  An injecting agent SHOULD prepend the <path-diagnostic>
>       "!.POSTED", optionally followed by "." and the FQDN or IP address
>       of the source, to the Path header field content.

I don't think that we ever agreed to permit a <diag-identity> after
"POSTED", though we did discuss it. I have no great objection if others
want it (semantically, it is rather like "SEEN"). However, if allowed, it
needs to indicate that the FQDN or IP address is indeed a <diag-identity>,
in which case the question arises of why you have not permitted a
<path-nodot> (which the syntax would allow).

>   3.  A relaying or serving agent SHOULD prepend a <path-diagnostic> to
>       the Path header field content, where the <path-diagnostic> is
>       chosen as follows:

>       *  If the expected <path-identity> of the source of the article
>          matches the leftmost <path-identity> of the Path header
>          field's content, use "!" (<diag-match>).

At this point, you need a definition of "expected". I see that you mention
"trusted" sources and agents in 3.4 and 3.5 (but not 3.6). In USEPRO, I
had a generic definition in section 7, but using "verified" rather than
"trusted". I think you either need such a generic definition in this
section, or you need to define "expected" by reference to those earlier
mentions.  And I would have thought that "verified" was a better term than
"expected", but that is less of an issue than ensuring the term in
question is consistently defined somehow.

>       *  If the expected <path-identity> of the source of the article
>          does not match, use "!.MISMATCH." followed by the expected
>          <path-identity> of the source or its IP address.

Again, you are actually constructing a <diag-match> here, but this time
you DO allow a <path-nodot>.

>       *  If the relaying or serving agent is not willing or able to
>          check the <path-identity>, use "!.SEEN." followed by the FQDN,
>          IP address, or expected <path-identity> of the source.

But now the <path-nodot> is gone again :-( .

>   4.  The agent MUST then prepend its own <path-identity> to the Path
>       header field content.

>   5.  The agent MAY then prepend additional <path-identity>s for itself
>       to the Path header field content, following each <path-identity>
>       so added with either "!!" or "!".  This is permitted for agents
>       that have multiple <path-identity>s (such as during a transition
>       from one to another).  Each of these <path-identity>s MUST meet
>       the requirements set out in Section 3.2.

You need to be careful here. Certainly an agent with several
<path-identity>s can prepend a selection of them, but it MUST ensure that
the last (leftmost) prepended one is its "official identity" by which it
will be known to its peers; otherwise, it risks getting MISMATCHed.

>   Any agent which modifies the Path header field MAY fold it by
>   inserting FWS immediately after any <path-identity> or <diag-other>
>   it added (see section 3.1.5 of [USEFOR] for allowable locations for
>   FWS).

OK, the general structure of what you have written is fine. The only extra
item I can think of at the moment is that you need to say that you have
only defined three <diag-keyword>s (there is a pointer to USEPRO regarding
this in USEFOR), and that other <diag-keyword>s MUST/SHOULD NOT be used,
except those beginning with "X" (for sites that claim a need to say
something that cannot be expressed with the three we have provided, or for
other such experimental reasons).

The section on Path Examples presumably follows here here. If Richard
Clayton is listening, could he please repeat his reasons why the
<path-nodot> "demom" was needed (and is still used) by demon's servers,
and can we please then incporate that into the examples somewhere. The
whole <path-nodot> business was heavily influenced by his example, but I
forget the precise reasoning now.

>>  * (Still under discussion.)  We may wish to reintroduce the possibility
>>    of a path-identity that is not a resolvable name in DNS.

>This was already in my draft.  I have:

>   The <path-identity> used by an agent may be chosen via one of the
>   following methods (in decreasing order of preference):

>   1.  The fully-qualified domain name (FQDN) of the system on which the
>       agent is running.

If this means A and AAAA records then please, for the removal of all
doubt, make that explicit.

>   2.  A fully-qualified domain name (FQDN) within a domain affiliated
>       with the administrators of the agent and guaranteed to be unique
>       by the administrators of that domain.  For example, the
>       uniqueness of server.example.org could be guaranteed by the
>       administrator of example.org even if there is no DNS record for
>       server.example.org itself.

Again, if this is meant to include MX records and CNAMES, please make it
explicit, because the wording does not immediately suggest that, and it
will hopefully be the common case.

However, I am still opposed to allowing domain-names for which no DNS
exists, and if that gets taken out then there is little point in keeping
this paragraph, and the MX and CNAMEs may as well go back in #1.

>   3.  Some other (arbitrary) name .....

>The most common current practice is to use the FQDN of the news server,
>not a mail server, as a path identity.  If one wants to use a mail server,
>2. still allows for that; it doesn't require that the name *not* exist in
>DNS.

>In my original draft, I intentionally removed any protocol-required
>connection between a <path-identity> and a contact address.

There hasn't been any such protocol-required connection for a long time.
We decided to let RFC 2142 (muddled though it is) speak for itself.
However, that it no reason not to draw attention to it as USEPRO did
(slightly differently in the two alternatives, of which I prefer the
freestanding second version which came from Harald); observe that it
carefully refrains from saying that you MUST/SHOULD follow it. I would be
equally happy yo see it in a NOTE, but it needs to be mentioned somewhere.

---

From: Russ Allbery <[email-masked]>
Date: Mon, 01 Jan 2007 22:16:34 -0800

Charles Lindsey <[email-masked]> writes:
> Frank Ellermann <[email-masked]> writes:

> checkgroups? I don't think any of our documents have mentioned this yet.

There isn't a correct one from a protocol standpoint -- whatever achieves
the propagation one wants.  Traditionally one uses the administrative
announcement newsgroup for the hierarchy.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 2 Jan 2007 21:00:16 GMT

>> Spam cancels intentionally violate official rules, following their own
>> standards.

>Spam cancels and other sorts of administrative cancels are compliant with
>my protocol spec and I want to keep it that way.  They're an authorization
>problem, not a protocol problem.

Discussion of cancels and who may issue them (1st, 2nd, and 3rd parties
and all that) was moved into USEAGE way back (it was not a clear-cut
decision and I could have been persuaded to move them back, but only one
person ever suggested doing so). So let us leave it that way.

>I think the current text is fine.

But that does not mean I agree with the current text (again, I addressed
this in another thread).

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 2 Jan 2007 20:51:41 GMT

In <[email-masked]> Frank Ellermann <[email-masked]> writes:

>Russ Allbery wrote:

>> The standard makes it clear that anyone who wishes can invent a new
>> control message and start sending them.

>That's apparently a derivation from usepro-06 in your text, you have
>"MAY send or accept whatever" with a "MUST ignore or reject unknown".

For those who are lost, those two statements appear in a single sentence
in section 5, and do seem confusing when put together. Also, I don't
understand what "reject" means (as dintinct from "ignore" or "not honour")
in this context.

>IMO that can't fly if relaying agents reject unknown stuff.  We could
>create a IANA registry of control <verb>s, reserving x-whatever for
>experimental / unofficial <verb>s, and anything else to be defined in
>an RFC.  With a general "MUST ignore unknown", limiting "MAY reject
>unknown" to injecting agents.  Or better removing the "reject" option.

I think what the sentence is really trying to say is:

"Agents MAY make arrangements to accept other control message types than
those specified below, but otherwise MUST ignore all unrecognized types."

>The IANA registry stuff is fairly simple, Harald has written a kind of
>"howto" (2434bis), and "defined in an RFC" is straight forward.

No, please keep IANA out of this. IANA has quite enough to do already :-( .

>#######################################################################

>  [A cancel SHOULD match the newsgroups of the original article]
>>> Is that good enough for a MUST instead of a SHOULD ?  Or an explicit
>>> OPTION to ignore cancels violating the SHOULD ?

>> Two reasons not to make it a MUST:

I basically agree with Frank here, but I have explained my reasons in
another thread.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 2 Jan 2007 21:38:09 GMT

>> It was your idea to add the OPTION of new control verbs, with the odd
>> effect of talking about newgroup + rmgroup "for example", as if there
>> are others.

>No, it was not my idea.  It's how Netnews actually works and has for
>longer than I've been involved in it.  From the first version of INN, you
>could drop in a handler for a brand new control message by doing nothing
>more than putting a shell script in a directory, and people did so to
>various ends.  Adding a new group control message would also require
>changing one line of code in innd that classifies control messages as
>"newgroup-like" for propagation purposes, but that's trivial.  That new
>control verbs weren't allowed was a pure artificial creation of previous
>documents (assuming one even read them that way) which I have attempted to
>bring in line with reality.

The fact that it is easy to drop a new control message type into existing
servers is no argument for making it a readily available feature of the
protocol. It only makes it suitable for use in "cooperating subnets" (a
term which Henry invented for S-o-1036, and which I see you have removed
from your draft). It does not make it suitable for use throughout the
whole of Usenet.

If you want to introduce a new control message type for Usenet as a whole,
then it is essential that everybody knows in advance exactly how it is
supposed to operate, and it needs very careful preparation, since you only
get one chance to get it right. So it really needs an RFC, whether
experimental or standards-track (and I don't think even experimental is
sufficient for Usenet as a whole since, once the experiment is out, there
is no way to put it back in the bag). By all means try out your experiment
on a cooperating subnet, but generally speaking control messages are so
few and far between that it is difficult to obtain evidence of whether the
experiment works, except by artificially creating and removing test groups
in all sorts of contrary ways (which is essentially what I did when I
tested mvgroup in CNews, and an cooperating subnet of 1 sufficed for
that).

So I am essentially dubious about the "MAY accept other control message
types" unless it is coupled with mention on some sort of cooperating
subnet, or with mention of an extension RFC. But maybe I am not as dubious
as Frank :-) .

>>> Since relaying agents may reject messages for any reasons they chose,

>> This is not the case, compare usepro-06 chapter 6.3 points 1..8.  They
>> must / should / may reject articles for the reasons specified in 1..8.

>I didn't realize that anyone in the working group didn't consider this
>principle implicit and assumed.  I'm glad, in this case, that I made it
>explicit in my draft.  Section 3.1 of my draft says:

I agree with Russ here. USEPRO always made it clear that individual sites
could always refuse to accept any article "just because it is Friday" if
they were so minded. The one thing they were NOT allowed to do was to
alter it if they did not like it (beyond a few defined circumstances).

Not that this makes much difference for relayers because, if one site
refuses to relay an article, the flooding algorithm will still find a way
to get it to pretty well every other site.

---

From: Russ Allbery <[email-masked]>
Date: Tue, 02 Jan 2007 23:27:42 -0800

> understand what "reject" means (as dintinct from "ignore" or "not
> honour") in this context.

The same as it means in all other contexts.  Rejecting the message means
refusing to accept it and process it.

> those specified below, but otherwise MUST ignore all unrecognized

No, that wasn't what I was trying to say.  The above is functionally
equivalent to what I have now, but I think explicitly mentioning the
always-present option to reject a message is clearer.

---

From: Richard Clayton <[email-masked]>
Date: Wed, 3 Jan 2007 13:20:10 +0000

In message <[email-masked]>, Charles Lindsey
<[email-masked]> writes

>If Richard
>Clayton is listening, could he please repeat his reasons 

from June 2005, how time flies :(

>why the

What I said in http://www.imc.org/ietf-usefor/mail-archive/msg01381.html
was:

<quote>

It would also be unwise to put any existing servers in a position that
they found themselves feeling that to conform to modern standards they
should change their identity...

Demon Internet (the first UK dialup ISP in the early 1990s) has used
!demon! since before it was called Demon Internet or even before it sent
news over NNTP. When an error was made with a new peering machine and
this tag was omitted a number of customers (with simple single-user
"leaf" systems who only swapped articles with Demon) started fetching
articles and (trying to) re-inject them all. Because servers elsewhere
in the cluster, generally, already had a copy of the article this went
on unnoticed for weeks until occasionally the customer presented the
article earlier than the peering machine.... and when the article was
spam it tripped detectors and we found the problem.

What I'm trying to illustrate is that path identities are not only
important to the few thousand machines that peer news on a high volume
basis, but these identities are nailed into hundred of thousands of end
user systems. We outlaw (or deprecate) existing path identities, whether
dating from UUCP or more recent conventions, at our peril.

</quote>

what I'd add to that now is that any site with multiple machines doing
peering, handling injections etc is very likely to have its own custom
schemes for ensuring that databases are synchronised (or indeed there
may just be a central store of data). These are unlikely to depend on
scanning Path header fields, but of course they may.  However, when
articles depart from these systems to the rest of the world they'd like
to add a site-wide identity to show that they have been dealt with, so
as to discourage them being offered again. Hence the addition of
multiple path identities by such machines...

hence we see

        !newsfeed.gamma.ru!Gamma.RU!
        !supernews.com!corp.supernews.com
        !nx01.iad01.newshosting.com!newshosting.com
        !news.tele.dk!news.tele.dk!small.news.tele.dk

and doubtless many more (those were found in seconds in a current
feed...)

>and can we please then incporate that into the examples somewhere. 

I'm not sure it's to be encouraged for systems that are not so complex
as major peering hubs.  Mind you, I sometimes wonder if there's going to
be all that much left soon that isn't a major peering hub.

>The

- -- 
richard                                              Richard Clayton

They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety.         Benjamin Franklin

---

From: Richard Clayton <[email-masked]>
Date: Wed, 3 Jan 2007 11:51:52 +0000

In message <[email-masked]>, Russ Allbery
<[email-masked]> writes

>   same news servers as the original message.

I don't think messages attempt anything :)

   A cancel control message SHOULD have the same Newsgroups header field
   as the message it is cancelling so as to best ensure that it will
   be relayed to the same news servers as the original message.

though if you dislike split infinitives then yet another phrase will be
necessary :)

- -- 
richard @ highwayman . com                       "Nothing seems the same
                          Still you never see the change from day to day
                                And no-one notices the customs slip away"

---

From: Richard Clayton <[email-masked]>
Date: Wed, 3 Jan 2007 12:42:00 +0000

In message <[email-masked]>, Russ Allbery
<[email-masked]> writes

>          field's content, use "!" (<diag-match>).

If you're not aware of the !! syntax (which is after all "new") then I
think this would confuse, and people will be able to read it so as to
miss out one of the !s

could it say

        #1   All agents MUST always first prepend "!" ..

        #2   An injecting agent SHOULD then prepend ...

        #3   A relaying or serving agent SHOULD then prepend ...

        *  If the expected <path-identity> of the source of the article
           matches the leftmost <path-identity> of the Path header
           field's content, then prepend the <diag-match> string "!". So 
           in this, hopefully most common case, two !s will appear in
           the path.

>          IP address, or expected <path-identity> of the source.

I'd then add

        Note that previous versions of this standard did not require
        <path-diagnostics> so that conformant systems would only add
        single "!" characters between <path-identity> strings.

>       the requirements set out in Section 3.2.

For !! to be of any use (which I continue to feel is entirely dubious,
but that's water under the bridge) then surely the requirement in this
last paragraph ought to be for !! (because the server must surely be
claiming that self-to-self is a "validated" transfer)

- --

---

From: Russ Allbery <[email-masked]>
Date: Wed, 03 Jan 2007 18:56:04 -0800

> sending agent and receiving relaying agents are both configured to relay
> a newsgroup of that name (whether or not such a newsgroup yet exists)."

This seems like an uncontroversial improvement in wording (and it's
shorter!).  Changed in my copy, omitting the "yet" at the end since this
also applies to rmgroups and it may well be that the newsgroup was just
removed.

---

From: Russ Allbery <[email-masked]>
Date: Wed, 03 Jan 2007 19:01:42 -0800

Richard Clayton <[email-masked]> writes:
> Russ Allbery <[email-masked]> writes

> though if you dislike split infinitives then yet another phrase will be

Definite improvement.  The "so as to" still felt awkward, so I also
inverted the sentence.  I now have:

    To best ensure that it will be relayed to the same news servers
    as the original message, a cancel control message SHOULD have the
    same Newsgroups header field as the message it is cancelling.

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 3 Jan 2007 20:07:01 GMT

>and doubtless many more (those were found in seconds in a current

Essentially, we need to allow sites to insert more Path entries than the
number of hosts the article actually passed through, just as we need to
allow them to insert fewer than that number of entries if the internal
structure of the site does not need to be exposed outwardly. Just so long
as the leftmost one is the one most likely to be recognized by their
peers, so that MISMATCH checking proceeds smoothly. What is good for
MISMATCH checking is not necessarily best for preventing articles being
sent to them more than once.

>be all that much left soon that isn't a major peering hub.

I think there are all sort of sites out there with special circumstances
that nobody else is aware of. So if we included an example, we would
probably say something like ...!news.demon.net!demon!incoming.demon.net!...
with and explanation like "although the site news.demon.net does not
actually include a host the with name 'demon', it included that
<path-nodot> because it was aware of a large number of (client) sites that
still knew it by that name". But change "demon" into some other foobarish
name, of course.

Interestingly, we have been talking a lot recently about home sites that
included their own personal newsserver. Demon is unusual in that it
possesses an unusually large number of such clients, since it initially set
all its clients up that way.

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 3 Jan 2007 18:08:37 GMT

>newsgroup-description production, so only CR and LF are disallowed, which

No, I meant that HT is not included in <utext> according to RFC 2822, and
hence is not in <eightbit-utext>. Whether we should care or not I am not
sure.

>newsgroup name in ASCII, which is precisely what's forbidden.

It should be so, but I don't think it is deducible from the wording you
have written. If you
   s/MUST be represented using/MUST be represented in that charset using/
it might work.

>existing software would choke on the initial escaping character.

Suppose the two modes of the charset are ASCII and Japanese. I am not
familiar with such charsets, but I would not be surprised if the default
was to show Japanese characters initially, until an escape character was
encountered. In which case an attempt to display the checkgroups would
render the newsgroup-name as gibberish (it would still work correctly as
regards updating the active file, etc). Japanese users would have to tweak
their displays to show it right (which is surely possible, because it will
be a regular problem that they encounter, so it is not necessarily a show
stopper but still needs drawing attention to).

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 3 Jan 2007 19:07:58 GMT

>announcement newsgroup for the hierarchy.

OK, sounds like a thing to be mentioned in USEAGE. Noted as such.

---

From: Russ Allbery <[email-masked]>
Date: Wed, 03 Jan 2007 21:56:49 -0800

> No, I meant that HT is not included in <utext> according to RFC 2822,
> and hence is not in <eightbit-utext>. Whether we should care or not I am
> not sure.

<utext> doesn't include whitespace, so <eightbit-utext> follows suit.  The
only production that uses it reintroduces non-leading whitespace.

>> Surely this is obviously invalid?  It would mean that the newsgroup
>> name in the charset specified for the body of this type is different
>> than the newsgroup name in ASCII, which is precisely what's forbidden.

>    s/MUST be represented using/MUST be represented in that charset using/

Well, that seems redundant to me, but if others feel like it adds clarity,
I don't mind adding it.

>> Well, no, you don't, provided that without the escaping the newsgroup
>> name is still the same as for ASCII.  If it isn't, then you have a
>> problem because there's no way to do the escaping before the newsgroup
>> name and that charset is therefore banned.  Which is, I believe,
>> correct, since existing software would choke on the initial escaping
>> character.

> regards updating the active file, etc).

In which case you can't use that character set, since the newsgroup name
would be some random Japanese characters rather than a newsgroup that
could be represented in ASCII.

---

From: Ned Freed <[email-masked]>
Date: Wed, 03 Jan 2007 21:29:51 -0800 (PST)

> encountered.

This is not how modal charsets work in practice. I don't have my copy of
ISO-2022 handy to check, but it may well allow the initial bindings to the
G0/G1/C0/C1 regions to be specified externally to be something other than
ASCII. (Heaven knows it allows all sorts of far more preposterous stuff, such
as sequences that do a mode switch only for the next character or creating
something that looks like, say, iso-8859-1 except the role of the 8th bit is
reversed.) Nevertheless, any such capability is not used in the various
iso-2022-* variants that have been defined, none of which allow anything
approaching the full set of capabilities of iso-2022.

In the specific case of Japanese, both iso-2022-jp and iso-2022-jp-2 always
start in ASCII. An escape sequence switches to some other character set. There
is also a requirement that things be switched back to ASCII at the end of each
text segment.

The Korean variant iso-2022-kr also starts in ASCII. It differs from the
Japanese variants in using a different set of capabilites. Specifically,
rather than rebinding the G0 set, iso-2022-kr binds the G1 set to Korean
and then uses SI/SO to switch between G0 and G1. (Note: This is all from
memory so some of the finer points may be wrong.)

The two Chinese charsets iso-2022-cn and iso-2022-cn-ext also start in 
ASCII. They then use an even more elaborate mix of iso-2022 capabilities than
Korean or Japanese.

I don't think iso-2022-tw (traditional as opposed to simplified Chinese) is
well defined, but the few places I've seen it crop up seem to indicate it
starts in ASCII like the others. (If anyone has a pointer to an actual
specification for it I'd like to have it.)

I suppose a fair question to ask is if there are other uses of iso-2022 that
would not be so restricted. AFAIK there are not: Things like X.400 generaltext
(or if you want a really obscure one, iso6937text) do allow the full repetoire
of iso-2022 trickery (although practically nobody ever implemented it all), but
there's no way in these protocols to specify the alternate set of initial
character set bindings that would be needed to start in something other than
ASCII.

Another question is if future charsets will be defined on top of iso-2022 that
behave differently. Of course nobody can predict the future, but I think
this is highly unlikely.

Of course there are lots of other CJK charsets, but these are the modal
ones. Charsets like EUC-TW or GB18030 all use 7bit characters to represent
ASCII and sequences of 8bit characters for everything else.

> In which case an attempt to display the checkgroups would

I don't think this issue exists in practice. See above.

				Ned

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 4 Jan 2007 12:21:54 GMT

>only production that uses it reintroduces non-leading whitespace.

OK, that seems to cover it.

>Well, that seems redundant to me, but if others feel like it adds clarity,

Yes please, though I would really prefer a wording that explicitly spoke
of having ASCII as a subset and/or of using charsets with escapable modes.

Either way, ASCII and UTF-8 would be the RECOMMENDED charsets.

>> Suppose the two modes of the charset are ASCII and Japanese. ...

>could be represented in ASCII.

OK, Ned Freed's explanations seem to confirm that all the current modal
charsets start out in ASCII, and would therefore be suitable.

---

From: Russ Allbery <[email-masked]>
Date: Fri, 05 Jan 2007 10:51:53 -0800

> of having ASCII as a subset and/or of using charsets with escapable

Adding "in that charset" at least seems uncontroversial (and I wanted to
test something else with the commit notification).  Changed in my copy.

> Either way, ASCII and UTF-8 would be the RECOMMENDED charsets.

Yup, that's already there.

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 06 Jan 2007 13:32:17 +0100

>            in this, hopefully most common case, two !s will appear in

Makes sense.  I'd omit "hopefully", and I'd say "two adjacent !s" [...],
or '"!!" (two exclamation marks) will appear' [...]

>         single "!" characters between <path-identity> strings.

Using "conformant" meaning "conformant with RFC 1036" is confusing, the
note would be clearer without this "conforman".  Actully I don't see 
why this note is needed at all, an ordinary "!" is explained in USEFOR.

> the requirement in this last paragraph ought to be for !! (because 
> the server must surely be claiming that self-to-self is a "validated"
> transfer)

ACK

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 06 Jan 2007 13:50:17 +0100

>     same Newsgroups header field as the message it is cancelling.

Proposed addition:

| The groups in the Newsgroups header field MUST match at least one
| of the affected newsgroups.

For Rmgroup and Newgroup we have that anyway, for Checkgroups it's also
sound, and for a Cancel it's required for transparency.  

All other control messages apparently don't have a Newsgroups header 
field.  If that's the case it has to be mentioned explicitly, USEFOR-11
claims that this header field is "mandatory".

Frank

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 06 Jan 2007 14:12:19 +0100

Charles Lindsey wrote:

 [unknown control messages]
> But maybe I am not as dubious as Frank :-) 

I'd understand "MUST ignore" or "MUST reject".  But not simultaneously,
these are conflicting strategies.  If we pick "ignore or reject", then
there can't be a 2119 MUST, because there's no other choice - Russ' idea
"try to guess what it means" is too odd for a "MUST NOT".

We could say "SHOULD ignore" if "MUST ignore" is too strong.  The whole
issue would be much clearer with a registry.  There are already tons of
registries for mail, e.g. a "with" registry (SMTP, ESMTP, ESMTPA, etc.),
it's okay if news has a "control" registry.  

And with an "RFC required" rule the work for IANA is minimal, they read
all "IANA considerations", and note new registrations, like say "mvgroup
is defined in [RFC 5555]", or "LMTPA is defined in [RFC 3848]".

Frank

---

From: Russ Allbery <[email-masked]>
Date: Sat, 06 Jan 2007 11:18:27 -0800

Fixing the header line for you to follow what the chairs asked for.

 

> | of the affected newsgroups.

I'm opposed, unsurprisingly.

> sound,

Does that mean that you're proposing something similar for checkgroups?
If so, you should say so explicitly.

> and for a Cancel it's required for transparency.

> field.

Of course they do.  There are just no special rules, so the standard rules
for a Newsgroups header applies.

---

From: Russ Allbery <[email-masked]>
Date: Sat, 06 Jan 2007 11:50:33 -0800

>> relayer name optional and requires that the message IDs be in the
>> control header field.  The latter makes the relayer name mandatory and
>> RECOMMENDS that the message IDs be put in the body.  I think the
>> following is the most reasonable compromise; it makes the relayer-name
>> mandatory if there are no message IDs (the form never allowed in RFC
>> 1036 but according to Son-of-1036 the most widely used in practice),
>> and if there are message IDs, permits the same syntax as RFC 1036.

> I don't see why the optionality or otherwise of the <relayer-name>
> should correlate in any way with whether the list of msg-ids is in the
> header or the body (except to ensure that these headers never have empty
> content, which does not sound like a particularly good reason).

I didn't think that RFC 1036 allowed message IDs in the body, which would
have created a link, but after reading it more closely, I see that it does
(in a confusing way).  So given that, I agree, there is no obvious link.

> regardless. I think we can assume that Henry had sufficient experience
> of the way Ihave and Sendme were used at that time that it is safe to
> follow his lead.

The primary reason why I had the relayer name be optional when the message
IDs are in the control header is because that's what usepro-06 had.  (I
later checked RFC 1036 and saw that it was optional there, and figured
that's why we'd had it optional.)

> headers is written on the assumption that a <relayer-name> will be

See the bottom of the second paragraph of the text:

   If <relayer-name> is not given, it is determined from the origin of
   the control message.

I think we have three options here:

 * Leave the text alone.  <relayer-name> is optional if the message IDs
   are given in the control message, allowing the primary RFC 1036 syntax,
   and mandatory if the message IDs are in the body.

 * Make <relayer-name> always optional.  This will make our syntax
   identical with RFC 1036.

 * Make <relayer-name> always mandatory and remove the above sentence from
   the text.  This will make our syntax identical with Son-of-1036.

I have a mild preference for the third option but don't really care as
long as we can agree on one choice and stick with it.

For whatever it's worth, INN doesn't support message IDs in the control
header at all.  It requires they be in the body.  INN's implementation
also completely ignores the content of the control header, including any
relayer name, and always uses the origin of the control message.  I've
never gotten any complaints about how it works, which probably just
indicates that few people use these control messages since the current
behavior isn't compliant with any of the various specifications.

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 06 Jan 2007 21:02:37 +0100

> Fixing the header line for you to follow what the chairs asked for.

Thanks, I read 51 pending messages last in - first out, missing the
subject instructions (not that they're new, but I missed it anyway).

> I'm opposed, unsurprisingly.

Yes, IIRC Charles supported the "MUST at least match one" for Cancel.
My proposal to make it a general rule (= not only for Cancel) was a
new idea.
 
> Does that mean that you're proposing something similar for 

Yes.

>> other control messages apparently don't have a Newsgroups header
>> field.
 
> Of course they do.  There are just no special rules, so the standard 
> rules for a Newsgroups header applies.

Good.  There are no "affected newgroups" for a "version", "sendsys",
etc., so that won't conflict with the "SHOULD list all" or my proposed
"MUST match at least one" addition.

Frank

---

From: Russ Allbery <[email-masked]>
Date: Sat, 06 Jan 2007 12:48:05 -0800

> Yes.

Just to be clear, any newsgroup that occurs in the checkgroups message
would satisfy the requirement?

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 06 Jan 2007 22:30:44 +0100

> Just to be clear, any newsgroup that occurs in the checkgroups
> message would satisfy the requirement?

Yes.  As you wrote elsewhere that's typically admin group(s), but
we don't need that detail, let the posters pick what they consider
as the best affected group(s) wrt the checkgroups propagation.

If it's a very small TLH (say less than 15 groups) they might
also get away by listing them all, as in your SHOULD designed for
other types of control messages.

Frank

---

From: Russ Allbery <[email-masked]>
Date: Sun, 07 Jan 2007 16:15:54 -0800

I'm going to break my response up into two parts, one wording change and
one protocol issue.

> think this would confuse, and people will be able to read it so as to

Agreed.

How about:

      *  If the expected <path-identity> of the source of the article
         matches the leftmost <path-identity> of the Path header
         field's content, use "!" (<diag-match>), resulting in two
         consecutive "!"s.

>         single "!" characters between <path-identity> strings.

Taking into account the subsequent discussion, how about:

    Be aware that [RFC1036] did not include <path-diagnostic>.
    Implementations which predate this standard will add only single "!" 
    characters between <path-identity> strings.

If anyone disagrees with these wording changes, please speak up, either to
say that this should be an issue instead or to suggest further tweaks.
Unless this becomes an issue, I'll plan on making the wording changes next
Sunday.

---

From: Russ Allbery <[email-masked]>
Date: Sun, 07 Jan 2007 16:21:48 -0800

> claiming that self-to-self is a "validated" transfer)

Yes, the current wording was a cop-out and I had a feeling it was when
writing it.  It needs to be changed.

Currently, using <path-diagnostic> in general is RECOMMENDED but not
REQUIRED.  As mentioned in an earlier post about philosophy, I did this so
that, should the whole diagnostic thing turn out to be a bad idea for some
reason, or just not worth it, implementations would not be non-conforming
by declining to implement it.  That's what this "either !! or !" business
was trying to capture, poorly.

I think we have two options.  My preferred option would be to say:

  5.  The agent MAY then prepend additional <path-identity>s for itself
      to the Path header field content.  Each <path-identity> so added
      MUST be followed with either "!!" or "!".  Using "!!" is
      RECOMMENDED.  This is permitted for agents that have multiple
      <path-identity>s (such as during a transition from one to another).
      Each of these <path-identity>s MUST meet the requirements set out in
      Section 3.2.

This copies the MUST/SHOULD language of <path-diagnostic> in general,
although it's a bit awkward.  The other alternative is to just require the
<path-diagnostic> in this case:

  5.  The agent MAY then prepend additional <path-identity>s for itself
      to the Path header field content, following each <path-identity> so
      added with "!!".  This is permitted for agents that have multiple
      <path-identity>s (such as during a transition from one to another).
      Each of these <path-identity>s MUST meet the requirements set out in
      Section 3.2.

When configured to add multiple path identities, I believe INN would allow
specifying !! as the delimiter with no changes, but I'm not positive.
Other existing software may well require modification.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 8 Jan 2007 20:58:07 GMT

>that's why we'd had it optional.)

Good Heavens! So it did! Evidently I just blindly copied RFC 1036, on the
grounds that it didn't really matter since I was deprecating it anyway.

>   If <relayer-name> is not given, it is determined from the origin of

Ah!

>long as we can agree on one choice and stick with it.

Agreed. My preference is for the third too. #! introduced apparently
complicated syntax for no apparent reason. #2 could result in an empty
header-body, which USEFOR does not allow.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 8 Jan 2007 21:12:57 GMT

>      Each of these <path-identity>s MUST meet the requirements set out in

Yes, I think that is the effect we want, but there is no need for the
MUSTard. I suggest (even though it is slightly longer):

   5.  The agent MAY then prepend additional <path-identity>s for itself
       to the Path header field content, normally followed by a
       <diag-match> "!" (to form "!!"), since it will surely recognize the
       earlier <path-identity>s as its own.  This is permitted for agents
       that have multiple <path-identity>s (such as during a transition
       from one to another).  Each of these <path-identity>s MUST meet the
       requirements set out in Section 3.2.

Note, however, that I have some other problems with this step and the one
before it, as discussed in another thread.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 8 Jan 2007 20:18:37 GMT

>it's okay if news has a "control" registry.

Yes, I agree the present wording is unsatisfactory.  

>is defined in [RFC 5555]", or "LMTPA is defined in [RFC 3848

But the only likely oddities (until we come to revive 'mvgroup') will be
private arrangements involving cooperating subnets (those seem to be the
ones Russ wants the rest of us to ignore/reject). And IANA does not want
to get involved in the private dealings of cooperating subnets.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 8 Jan 2007 20:13:57 GMT

>| of the affected newsgroups.

+1

Indeed, I would have expected MUST match all of them.

>claims that this header field is "mandatory".

Yes, but checkgroups is the only one where it is not specified
(discounting the obsoleted ones), and I have already agreed that is for
USEAGE to discuss.

---

From: Russ Allbery <[email-masked]>
Date: Thu, 11 Jan 2007 06:36:19 -0800

>        <diag-match> "!" (to form "!!"),

How would that form "!!"?  There's no provision for adding the first "!"
in your language at that point in the flow.  Going into step 5, the head
of the Path header is the first path-identity of the system.  If a "!"
needs to be added, we need to say so explicitly.

>        since it will surely recognize the earlier <path-identity>s as
>        its own.  This is permitted for agents that have multiple
>        <path-identity>s (such as during a transition from one to
>        another).  Each of these <path-identity>s MUST meet the

I don't understand why we would go out of our way to avoid MUST and SHOULD
here, when MUST and SHOULD are used in the rest of the section.  I think
the wording I proposed above is more consistent with the rest of the
section.  I also don't see the justification for downgrading "!!" from
SHOULD to "normally"; that was the flaw that sparked this issue to start
with.

language doesn't explicitly require any delimiter after the additional
path-identities at all; that might be obvious, but I'd rather not write
language, 

> Note, however, that I have some other problems with this step and the
> one before it, as discussed in another thread.

Open an issue.

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 12 Jan 2007 21:22:48 GMT

>needs to be added, we need to say so explicitly.

I was relying on the wording further up. Essentially, anyone who adds a
<path-identity> (even the same site doing it several times) has the
opportunity to add suitable diagnostics, and it would be better if one
piece of wording could cover the case whether it was the same site or a
different site that was doing it. I agree that, with the present
structure, my wording does not quite do it.

>> one before it, as discussed in another thread.

In particular, Steps 4 and 5 do not achieve quite the effect intended.

>Open an issue.

Yes, I shall raise the Steps 4 and 5 problem as an issue, and then maybe
the wording about getting "!!" in the right places will fall out, or can
be reconsidered, as a result of dealing with that.