Subject: #1008 USEFOR 1.5 References: Final text on "followup"

From: Harald Tveit Alvestrand <[email-masked]>
Date: Thu, 09 Jun 2005 16:47:54 +0200

In reviewing the thread, the text that seems to come closest to capturing 
the (VERY) rough consensus of the group is:

 A "followup" is an article containing a response to the contents of
 an earlier article, its "precursor". Every followup includes a
 References header identifying that precursor (but note that non-followup
 articles may also use a References header).

I have also noted that there is no discernible consensus for the text 
Charles suggested addition to the top of the References section (attempting 
to explain what References is used for); it will not be added as part of 
this decision.

Unless there are technical issues that have not been raised before, this is 
the instruction from the WG chairs to the editor of the USEFOR document, 
and the subject of the definition of "followup" is now closed.

Any new messages on the subject of the definition of "followup" that do not 
raise new technical issues will be treated as off-topic postings.

                     Harald & Alexey

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 10 Jun 2005 02:50:41 +0200

Harald Tveit Alvestrand wrote:

> the subject of the definition of "followup" is now closed.

Amen.

> Any new messages on the subject of the definition of
> "followup" that do not raise new technical issues will be
> treated as off-topic postings.

OB on topic:  what's the purpose of ticket #1021 ?  Where does
it come from ?  https://rt.psg.com/Ticket/Display.html?id=1021

                Bye, Frank

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Fri, 10 Jun 2005 14:49:07 +0200

This was an issue that Charles raised off-list, I believe.
Good point - if filing tickets in response to issues raised off-list, we 
need to make sure the WG is notified.....

The text of the ticket is the following:
------------------------------
I suggest to update the Newsgroup ABNF as follows:
1). uppercase letters must be prohibited in newsgroup names
2). single component newsgroups must be prohibited (this also
addresses conflict with the special value "poster")
3). prohibiting all digit components can be done in ABNF, but it would
be ugly. I suggest to address this with an ABNF comment.
4). prohibiting components "ctl"/"all" and newsgroups starting
with "control", "example" or "to" can also be done in ABNF, but that
would be even uglier, as there is no way to exclude something in ABNF
(unless I am mistaken). So I suggest to add another comment(s).
-------------------------------------------
The following comment was added by someone logged in as "ietf":

[[email-masked] - Wed Jun 01 19:18:53 2005]:

 > 2). single component newsgroups must be prohibited (this
> also addresses conflict with the special value "poster")

 But not with pseudo-groups like "junk" or "control", and
the real NG "spamcop" <news://news.spamcop.net/spamcop>

 > 3). prohibiting all digit components can be done in ABNF,
> but it would be ugly.

 Ugly if necessary is acceptable, but why is it necessary ?

 > 4). prohibiting components "ctl"/"all" and newsgroups
> starting with "control", "example" or "to" can also be
> done in ABNF

 From a user's POV control.all behaves like r/o NGs. On
one server I know I "see" control.cancel.de (etc.) for
cancels in various TLHs. I've never heard of "example"
being reserved. And pseudo-TLH "to" is some mail-by-news
trick, from a user's POV like an NG.

 (But actually I'm only testing this RT beast, bye, Frank)

---

From: Russ Allbery <[email-masked]>
Date: Fri, 10 Jun 2005 10:05:15 -0700

Harald Tveit Alvestrand <[email-masked]> writes:

> This was an issue that Charles raised off-list, I believe.  Good point -
> if filing tickets in response to issues raised off-list, we need to make
> sure the WG is notified.....

> The text of the ticket is the following:

It's not completely clear to me whether this is the right thing to do,
since there are some uses of mixed-case newsgroup names in the wild
already and they don't actually break anything beyond making things more
confusing.  The ftp.isc.org list refuses to add them, but the protocol
actually has historically supported them.

Sorry, I should have mentioned that earlier.

> addresses conflict with the special value "poster")

I think they should be allowed; "poster" and "junk" just have to be
reserved.  (Getting the wording right for "junk" without going into what
"junk" is used for is going to be a bit tricky.)

> (unless I am mistaken). So I suggest to add another comment(s).

These both sound good to me.

> the real NG "spamcop" <news://news.spamcop.net/spamcop>

It depends on how one prohibits them, basically.  Definitely, news readers
have to be able to read control and junk, so it has to be able to appear
in the Xref header.  In practice, people post to control and junk
directly, but this often doesn't really do what one wants and I'm not sure
it's a good idea to allow in the standard because the behavior is somewhat
undefined.  Crossposting to those groups is generally fairly harmless,
just confusing.  Creating groups by that name with control messages is a
bad idea and shouldn't happen.

Other single-component newsgroup names work fine at all levels of the
protocol.  They just don't work with the most common control.ctl patterns
and are frowned on because they don't fit into a hierarchy as cleanly as,
say, spamcop.general or the like.  That's not really a protocol issue;
it's more a USEAGE issue.

>  Ugly if necessary is acceptable, but why is it necessary ?

It's necessary to retain backward compatibility with any news server that
uses the traditional directory structure for the news spool, since the
group example.foo.600 and article 600 in example.foo both want to use the
same file name.  Many older news servers will essentially crash if one has
created example.foo.600 and article 600 arrives in example.foo.

Now, one could argue that this is just bad data structures and it's up to
the server to work around this rather than putting this into the protocol,
and I have a lot of sympathy for that argument.  It's just hard to work
around if you want to retain the standard spool structure because normally
the part of the server that numbers articles and the part of the server
that stores them don't have a two-way conversation, and there's a fair bit
of existing software out there that just isn't ever going to cope.

This has been one of the Usenet newsgroup naming rules for many years,
although it's honored in the breech in a few places.  (alt.2600 is the
most famous example -- it gets away with this even on older news servers
since there's no newsgroup named "alt".)

>> 4). prohibiting components "ctl"/"all" and newsgroups starting with
>> "control", "example" or "to" can also be done in ABNF

>  From a user's POV control.all behaves like r/o NGs.

On what server are you seeing control.all?

.all is a wildcard in C News and is equivalent to .*; it's used in feed
pattern specifications.

> On one server I know I "see" control.cancel.de (etc.) for cancels in
> various TLHs.

control.* should be reserved for the local implementation, not reserved in
the sense that it never appears or cannot be read.  Basically, the only
thing you should be forbidden to do is send control messages about it, or
possibly post to it (and I'm very iffy on the latter).

> I've never heard of "example" being reserved.

We're reserving it for the typical IETF "use in example" reasons.  That
will be new with our draft.

> And pseudo-TLH "to" is some mail-by-news trick, from a user's POV like
> an NG.

The to.* newsgroups have extremely special handling with older news
software and were used for point-to-point UUCP feeds and similar sorts of
tricks.  They are special-cased in numerous places in INN and I believe
also in C News, and using them for any normal purposes will make you sad.
This is another case where the client probably gets to read them, and in
this case even post to them, but it should be made clear that they're
special and may do strange things.  (We should probably expand at some
point on just what the strange things are that they're intended to be used
for.)

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 10 Jun 2005 17:13:05 GMT

>to explain what References is used for); it will not be added as part of 

I regret that there will now be no introductory text to the References
header (whether mine, or some other variant).

I also regret that there will be no clear statement of the semantics of
this header, other than what can be gleaned from the process for
constructing it, as set out in USEFOR (Forrest was anxious to have such
semantics included). I would have suggested a text such as the following:

|  The list is composed of message identifiers of ancestors of the
|  current article, sorted so that no article precedes any of its own
|  ancestors. It SHOULD include both the oldest ancestor and the
|  immediate parent of the current article, even if some of the
|  intermediate ones are omitted.

>and the subject of the definition of "followup" is now closed.

In that case, I believe the following represents the definitive result of
our deliberations:

Definitions (for inclusion in USEFOR or USEPRO as appropriate).

   A "followup" is an article containing a response to the contents of an
   earlier article, its "precursor". Every followup includes a References
   header identifying that precursor (but note that non-followup articles
   may also use a References header).    

   A "followup agent" is a combination of reading agent and posting
   agent that aids in the preparation and posting of a followup.

Within USEFOR:

3.2.1  References 

   The References header is the same as that specified in Section 3.6.4
   of [RFC2822] with the added restrictions detailed in Section 2.2 and
   those listed below:

   o  The updated <msg-id> construct defined in Section 3.1.3 MUST
      be used.

   o  Message identifiers MUST be separated with CFWS.

   o  Comments in CFWS between message identifiers can cause
      interoperability problems, so comments SHOULD NOT be generated,
      but MUST be accepted.

   references      =  "References:" SP [CFWS] msg-id *(CFWS msg-id )
                      [CFWS] CRLF

|  The process of generating a References header is set out in [USEPRO].

Within USEPRO:

7.6.  Duties of a Followup Agent

.......

   4. The followup MUST (in accordance with the definition of that term)
      have a References header referring to its precursor, constructed
      in accordance with section 7.6.1 below.

        NOTE: That "MUST" is to be contrasted with the weaker
        recomendation using "SHOULD" applied, in [RFC 2822], to the
        generation of "replies" in email. Moreover, in Netnews, there is
        no expectation of any In-Reply-To header in a followup.
 
7.6.1.  Construction of the References header

   The following procedure is to be used whenever some previous article
   (the "parent") is to be referred to in the References header (F-
   3.2.1) of a new article, whether in the course of generating a
   followup or for some other reason (e.g. the later parts of a
   multipart posting such as a FAQ, or the later parts of a
   message/partial as suggested in [RFC 2046]).

   If the parent did not have a References header, the content of the
   new article's References header MUST be inherited from that of the
   Message-ID header of the parent, otherwise (there already was a
   References header) it MUST be comprised of the parent's References
   header (subject to trimming as described below) followed by CFWS and
   the Message-ID header content of the parent.

   If the resulting References header is excessively long it MAY (and if
   its total length, whether folded or not, exceeds 998 chacacters it
   MUST) then be trimmed, but the first and the last two message
   identifiers MUST NOT be removed.

        NOTE: There is no provision in this standard for an article to
        have more than one parent. The essential property of the
        References header, guaranteed by the procedure above and to be
        preserved in any future extension, is that no article can ever
        precede one of its own parents.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 10 Jun 2005 21:37:52 +0200

Russ Allbery wrote:

> In practice, people post to control and junk directly,
> but this often doesn't really do what one wants

On GMaNe posting to "junk" is my way to check a problem, if
it appears immediately the server is fine, and mail2news is
in trouble - I do this only if a test in gmane.test failed.

> Creating groups by that name with control messages is a

Yes, but these details are IMHO a case for USEPRO.  Trying
to exclude single-component names in the USEFOR ABNF isn't
the best way to address it.

> not really a protocol issue; it's more a USEAGE issue.

Also fine, so it doesn't affect USEFOR.

 [component name 1*DIGIT]   
> necessary to retain backward compatibility with any news
> server that uses the traditional directory structure for
> the news spool, since the group example.foo.600 and article
> 600 in example.foo both want to use the same file name.

Okay, then we need some ldh-string magic to exclude 1*DIGIT.
We could use the ambiguous grammar style as for <id-quote>:

   c-name = *c-text c-okay *c-text
   c-text = DIGIT / c-okay
   c-okay = ALPHA / "-" / "_"

> hard to work around if you want to retain the standard spool
> structure

Adding one special character like "#" to the article numbers
would avoid all potential problems.  Actually I don't care how
we solve it, but don't mention "14" or I'd scream... <g>

> alt.2600 is the most famous example -- it gets away with
> this even on older news servers since there's no newsgroup
> named "alt".

Now that's bad, if our syntax says "not only digits" it will
be used elsewhere, e.g. in my "minimalistic" variant of the
two news-nntp-uri drafts.

> On what server are you seeing control.all?

Sorry, I meant "stuff below control", e.g. control.cancel.de
 
> .all is a wildcard in C News and is equivalent to .*

Yes, that's exactly what I meant.

> We're reserving it for the typical IETF "use in example"
> reasons.  That will be new with our draft.

Was #1021 your ticket ?  RT only says that Alexey added it.

If that's a new idea it's fine - I forgot the URL with the
unofficial "TLH registry", so I hope you checked this.  Bye

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 10 Jun 2005 22:39:19 +0200

Charles Lindsey wrote:

 {USEFOR]
> references  = "References:" SP [CFWS] msg-id *(CFWS msg-id )

ACK.  I'd prefer a msg-id-list version for aesthetical reasons,
same idea as for <newsgroup-list> or <path-list>:

  references  = "References:" SP msg-id-list CRLF
  msg-id-list = [CFWS] *(msg-id CFWS) msg-id [CFWS]

In that order the concept of at least one last msg-id is clear,
reflecting the construction algorithm as good as possible.

 [USEPRO]
> The followup MUST (in accordance with the definition of that
> term) have a References header referring to its precursor,
> constructed in accordance with section 7.6.1 below.

>   generation of "replies" in email. Moreover, in Netnews,
>   there is no expectation of any In-Reply-To header in a
>   followup.

I'd replace this convoluted MUSTard by a simple statement:

  The followup has a References header referring to its
  precursor, constructed in accordance with section 7.6.1
  below.

> in the course of generating a followup or for some other
> reason (e.g. the later parts of a multipart posting such
> as a FAQ, or the later parts of a message/partial as
> suggested in [RFC 2046])

I'd remove these irrelevant examples where other algorithms
might also work, the most important "other reason" is a reply,
not limited to Followup-To: poster.  So that would be:

   in the course of generating a followup or for some other
   reason (e.g. a mail reply).

> NOTE: There is no provision in this standard for an article
> to have more than one parent.

Then don't talk about it, or add the In-Reply-To blurb here:

  NOTE: There is no provision in this standard for an article
  to have more than one parent.  In Netnews, there is no
  expectation of any In-Reply-To header in a followup

> The essential property of the References header, guaranteed
> by the procedure above and to be preserved in any future
> extension, is that no article can ever precede one of its
> own parents.

There is no "guarantee", this is bad advice for implementors.

My UA cannot handle self-referencing articles, and Harald has
reported another case, where this "guarantee" simply crashed.

The essential property of the References is a msg-id-list, and
it's unnecesary to repeat it in USEPRO.

                             Bye, Frank

---

From: Russ Allbery <[email-masked]>
Date: Fri, 10 Jun 2005 13:55:00 -0700

>> In practice, people post to control and junk directly, but this often
>> doesn't really do what one wants

> On GMaNe posting to "junk" is my way to check a problem, if it appears
> immediately the server is fine, and mail2news is in trouble - I do this
> only if a test in gmane.test failed.

The problem is that those articles probably shouldn't propagate in a group
named "junk" and junk will often be full of articles that weren't posted
to that group because of what junk is for.  So it shouldn't be treated the
same as a normal newsgroup.

>> Creating groups by that name with control messages is a bad idea and
>> shouldn't happen.

> Yes, but these details are IMHO a case for USEPRO.  Trying to exclude
> single-component names in the USEFOR ABNF isn't the best way to address
> it.

I think I agree.

>> hard to work around if you want to retain the standard spool structure

> Adding one special character like "#" to the article numbers would avoid
> all potential problems.

That's not retaining the standard spool structure, and doing that will
break software that expects the standard spool structure.  As soon as you
deviate from the standard spool structure there are innumerable ways of
fixing the problem, but that's not the point.

>> alt.2600 is the most famous example -- it gets away with this even on
>> older news servers since there's no newsgroup named "alt".

> Now that's bad, if our syntax says "not only digits" it will be used
> elsewhere, e.g. in my "minimalistic" variant of the two news-nntp-uri
> drafts.

Right, that's why it's an arguable case.

> Was #1021 your ticket ?  RT only says that Alexey added it.

No.

But example had been in our draft for eons, I thought.  If not, my bad
memory; we've certainly talked about it lots before.

---

From: Henry Spencer <[email-masked]>
Date: Fri, 10 Jun 2005 17:04:40 -0400 (EDT)

> same as a normal newsgroup.

However, this is *largely* an internal issue within a news server, not
something that appears "on the wire".  It's reasonable for us to try to
stay out of the way of such internal conventions, but exactly how they
work is not our problem.

Reserving a few name components, and discouraging single-component names,
is not an unreasonable thing to do as part of USEFOR, but trying to do it
in the grammar is a mistake.  Even the most grammar-intensive RFCs have
accompanying text which adds restrictions and constraints.

                                                          Henry Spencer
                                                       [email-masked]

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Fri, 10 Jun 2005 17:14:53 -0400

>    message/partial as suggested in [RFC 2046]).

1. This text appears under "duties of a followup agent."  You have not
shown text in USEPRO that lets non-followups have a references header,
unless they were generated by a followup agent.

Non-followups cannot be generated by a followup agent, because follow up agent is
defined as software used for preparing and posting followups.

I raised this issue at the time of the poll, and was told that the
definition of followup was getting set, and my concerns would be
addressed within the revisions to USEPRO.

I find that my concerns were not addressed at all.

2. Since the semantics of the header were deleted from USEFOR, and the
semantics do not appear in this proposed text for USEPRO, there is
no guidance on how user agents should detect or treat non-followups
that have References headers.

>         precede one of its own parents.

3. Specifying what something is not is by no means a definition of "The essential
property" of anything.

4. RFC2119 language is not called for and is not required.  (E.g. if there
were a specification of the semantics References header, all the "MUST"
formulations are not needed.  The draft should specify that the appearance
of a references header means "blah blah" and then the implementors will
know how to do the right thing.  Specify MUST constructions and it becomes
ambiguous...

5. What should an implementor of a user Agent make of the 7.6
language regarding the References header?  What can that implementor
assume about articles created by agents that pre-date the draft?  Or
is there a flag day?  What can they assume about a multi-part FAQ?

---

From: "Ruud H.G. van Tol" <[email-masked]>
Date: Sat, 11 Jun 2005 09:35:54 +0200

Charles Lindsey schreef:

>    If the resulting References header is excessively long it MAY (and
>    if its total length, whether folded or not, exceeds 998 chacacters
>    it MUST) then be trimmed, but the first and the last two message

The four remaining message identifiers, with their CFWS, can have
a combined length of more than 998 characters.

I also read in that sentence that you are free to remove comments
out of (such long) References (as long as you keep the message-ids).
So the software might decide to try first to shrink all CFWS to a
single space.

I would just take the word 'two' out of it. Or maybe change it into
'few'.

It would seem more practical (to me) to just drop the oldest Message-IDs
(and their CFWS) from the start of the References field, until there
is room for the Message-ID to be added, but I guess that has all been
discussed before and rejected. I think that newer Message-IDs are more
useful (have more entropy/hold more information) than older ones.

-- 
Grtz, Ruud

---

From: "Ruud H.G. van Tol" <[email-masked]>
Date: Sat, 11 Jun 2005 11:25:06 +0200

Russ Allbery schreef:
> Harald Tveit Alvestrand:

> already and they don't actually break anything beyond making things
> more confusing.  The ftp.isc.org list refuses to add them, but the
> protocol actually has historically supported them.

I have messages in my test suite that have the same newsgroup name in
different casings in the Newsgroups: header. This was to test a new
bot that fights excessive crossposting.

-- 
Grtz, Ruud

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Sat, 11 Jun 2005 14:40:09 +0200

> bot that fights excessive crossposting.

which leads to another interesting question:
if net.AAAAAARGH is permitted, is it different from net.aaaaaargh, or are 
they the same?

I HOPE the answer is that "obviously, they are the same", but I had to 
ask...

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 11 Jun 2005 17:09:45 +0200

Harald Tveit Alvestrand wrote:
 
> if net.AAAAAARGH is permitted, is it different from
> net.aaaaaargh, or are they the same?
 
> I HOPE the answer is that "obviously, they are the
> same", but I had to ask...

Yes, I'm very much for discussing _relevant_ questions like
"what's a newsgroup name ?" or "what's a msg-id ?" instead of
wasting time with "what's a FYI 18 header ?".

s-o-1036 says:

| Newsgroup names MUST not contain uppercase letters

(disclaimer about encoded MIME words for I18N removed)

NNTP draft 27 offers some wild fantasies about UTF-8 in its
<wildmat-exact>, if that's related to reality then it's in a
parallel universe far away from my parallel universe:

 [10,2]
| Although this specification allows UTF-8 for newsgroup
| names, they SHOULD be restricted to US-ASCII until a
| successor to RFC 1036 [RFC1036] standardises another
| approach

Thanks but no thanks.  "Stringprep" is black magic while we
seriously consider to support any EST / EDT / UT timestamps.

 [10.3]
| Until such specifications are published, implementations
| SHOULD match newsgroup names octet-by-octet.  It is
| anticipated that any approved scheme will be applied
| "at the edges" and therefore octet-by-octet comparison
| will continue to apply to most, if not all, uses of
| newsgroup names in NNTP.

| In the meantime, any implementation experimenting with
| UTF-8 newsgroup names is strongly cautioned that a future
| specification may require that those names be canonicalized
| when used with NNTP in a way that is not compatible with
| their experiments.

Did I miss an "intended status: experimental" in nntp-27 ?

Anyway, apparently we can just do "the right thing", and in
the light of this draft I'd say "lowercase a..z0..9+-_".  No
uppercase, same idea as IDN.
                            Bye, Frank

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 11 Jun 2005 17:18:33 +0200

Ruud H.G. van Tol wrote:
 
>> If the resulting References header is excessively long
..........^^^^^^^^^
>> the first and the last two message identifiers MUST NOT be
>> removed.
 
> The four remaining message identifiers, with their CFWS, can
> have a combined length of more than 998 characters.

There are only three, the text talks about the result _after_
adding the parent msg-id.
  
> I also read in that sentence that you are free to remove
> comments out of (such long) References (as long as you keep 
> the message-ids). So the software might decide to try first
> to shrink all CFWS to a single space.

Good idea, maybe we should say so explicitly.  Bye, Frank

---

From: Russ Allbery <[email-masked]>
Date: Sat, 11 Jun 2005 09:23:04 -0700

> I HOPE the answer is that "obviously, they are the same", but I had to

Pretty much every piece of news software out there will treat them as
different.

---

From: Henry Spencer <[email-masked]>
Date: Sat, 11 Jun 2005 12:58:47 -0400 (EDT)

> Pretty much every piece of news software out there will treat them as

The "MUST not" in son-of-1036 was motivated by exactly that.  Given that
existing software overwhelmingly does case-sensitive comparisons, the
viable choices are to say that net.AAAAAARGH and net.aaaaaargh are
different names (aaargh!) or to forbid net.AAAAAARGH altogether (with the
usual "a cooperating subnet can do whatever it wants provided it keeps the
curtains closed" escape hatch).

                                                          Henry Spencer
                                                       [email-masked]

---

From: Richard Clayton <[email-masked]>
Date: Sun, 12 Jun 2005 13:36:28 +0100

In message <[email-masked]>, Charles Lindsey
<[email-masked]> writes

>   identifiers MUST NOT be removed.

I've two problems here which would confuse me as an implementor...

a)   does the overall length of 998 include the 12 characters
     "References: " or not (and what of the CR LF?).

     Perhaps this might be best expressed as a requirement that when
     unfolded into a single line including the keyword and terminating
     CRLF that it does not exceed 1000 ?

b)   the ones that can't be removed, is that before or after the message
     ID has been added ?

     I assume it must be after... because otherwise you'd have a
     requirement for at four IDs each of which can be 250 long :)

     It would be helpful to make that crystal clear. I can suggest some
     words, if that's necessary....

>        have more than one parent. 

happy with this

>The essential property of the

nope :(   if you're given rubbish (eg an article with existing circular
references) then this statement suggests that you have an obligation to
fix it up. I don't think that's a reasonable requirement. I suggest that
since this is merely meant to be a helpful deduction then it is either
toned down or removed.

- -- 
richard                                              Richard Clayton

They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety.         Benjamin Franklin

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Sun, 12 Jun 2005 15:30:17 +0200

I've not started recording tickets on USEPRO yet; are you raising a new 
issue on whether semantics of the References: header should be described or 
not in USEFOR (separate from the definition of "followup", which is 
closed), or are you wishing to raise an issue on USEPRO?

> that have References headers.

The implication (as I explicitly stated earlier) is that a reading agent 
cannot mechanically tell the difference between a followup and a 
non-followup.
So all the rules have to be stated in terms of handling articles with a 
References header; the term "followup" is only useful in USEFOR when 
describing how to generate one.

                    Harald

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Mon, 13 Jun 2005 08:26:44 -0400

Harald Tveit Alvestrand wrote:

> issue on whether semantics of the References: header should be described 
> or not in USEFOR (separate from the definition of "followup", which is 

Charles posted text for discussion.  Some of that text was for USEPRO.

Charles previously posted that there is no way to consider the documents
completely independently, and these sections from two documents must
be discussed together.  I agree.

You wrote that you were entering tickets as you saw issues raised on the list.
I did not think that excluded USEPRO.

Why are you not entering tickets for USEPRO?  The point of a ticket
system is to record issues so that work on them can occur at time other
than when they are raised.  Delaying they entry of tickets is completely wrong.

If you don't want to discuss USEPRO now, and you don't want to enter tickets,
then I think it is appropriate to convince Charles that the documents must
be considered independently, and then instruct him not to post USEPRO text
until you will allow discussion.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 13 Jun 2005 13:25:09 GMT

>a combined length of more than 998 characters.

No. Three remaining. That is after the new one has been appended.

>So the software might decide to try first to shrink all CFWS to a

Although not clear from the present wording (because of the way "content"
has been defined), you can keep the old comments, or throw them away, or
even write news ones. Just so long as you keep the msg-ids as directed. I
will make sure that is clear.

And, of course, we are agreed that such comments are at least SHOULD NOT,
and maybe MUST NOT generate.

>useful (have more entropy/hold more information) than older ones.

No, the first one must be kept, because that one is the root of the tree
and may be used by some agents to distinguish between different trees.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 13 Jun 2005 12:39:48 GMT

>unless they were generated by a followup agent.

Eh? It is USEFOR that "lets" you write a References header for a
non-followup. The proposed USEFOR text directs you to USEPRO to find how
to do it properly, and the above section is where it tells you that (it is
a subsection of 7.6 because there did not seem a better place to put it).
That section even gives you examples of non-followup situations that might
arise.

>defined as software used for preparing and posting followups.

Indeed. But that section is not restricted to use by followup agents, as
its wording clearly implies.

>that have References headers.

I have already expressed my regret at the lack of semantics in USEFOR. It
does seem, however, that our chair is willing to permit further discussion
of that. But not of the definition/detection of followups.

As to treatment of References headers, standards do not normally deal with
user interface issues, apart perhaps from sopme brief mention of
possibilites, such as what is currently written in USEPRO 7.7.

>3. Specifying what something is not is by no means a definition of "The essential

That is a NOTE, not a definition. Given the lack of any semantics in
USEFOR, it was the best I could do. An implementor writing a threading
algorithm is entitled to assume that the relevant headers (USEPRO 7.7) have
been constructed "properly". The essential property mentioned there is the
minimum that will enable him to do his job. For sure, it is not enough to
say that the References is "just a list of <msg-id>s, as Frank would have
it.

>is there a flag day?  What can they assume about a multi-part FAQ?

There is no need for any flag day. There is nothing that an implementor
using the References header has to do in consequence of our draft that is
in any way different from what he was doing beforehand.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 13 Jun 2005 13:51:55 GMT

>     "References: " or not (and what of the CR LF?).

Yes for the References, NO for the CRLF (which is true also of all other
places where that magic "998" appears.

>     requirement for at four IDs each of which can be 250 long :)

Correct. It is applied to "the resulting References header" after you have
done all the appending etc from the previous paragraph.

   If the parent did not have a References header, the content of the
   new article's References header MUST be inherited from that of the
   Message-ID header of the parent, otherwise (there already was a
   References header) it MUST be comprised of the parent's References
   header followed by CFWS and the Message-ID header content of the
   parent (subject to trimming as described below).

   If the resulting References header is excessively long it MAY (and if
   its total length, whether folded or not, and including the header
   name but not the final CRLF, exceeds 998 chacacters it MUST) then be
   trimmed, but in such a manner that the first and the last two message
   identifiers always remain.

I am not sure that it really better (and it is certainly longer).

>since this is merely meant to be a helpful deduction then it is either

If all References headers are constructed according to the preceding
paragraphs, then circular references cannot arise. If, by some accident,
one does arise, then "garbage in, garbage out" applies as usual.

But that paragpraph is just a NOTE, so it has no normative effect (and I
would prefer it to have been said in USEFOR - see my reply to Forrest).

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 13 Jun 2005 11:48:39 GMT

>> This was an issue that Charles raised off-list, I believe.

It was raised off list because it related to matters that were contained
in the old draft-13 and which were not believed to be controversial, but
which had not yet found their way into USEFOR.

>actually has historically supported them.

When the WG last discussed this (many years ago) it was claimed that some
software treated newsgroup-names case insensitively and some didn't. We
rejected the idea of allowing injecting agents to "fix it up", and so we
concluded that a ban on uppercase was the safest option.

However, perhaps a NOTE to the effect that some existing groups with UC
names (and also some with all-digit components) DO exist on Usenet, with a
warning that their propagation may be poor and that creation of further
such groups would be inappropriate.

>"junk" is used for is going to be a bit tricky.)

Here is the relevant text from draft-13, which could usefully be copied
into USEFOR more-or-less as it stands:

   The following forms of newsgroup-name MUST NOT be used except for the 
   specific purposes indicated:

     o Newsgroup-names having only one component. These are reserved for 
       newsgroups whose propagation is restricted to a single host or
       local network, and for pseudo-newsgroups such as "poster" (which
       has special meaning in the Followup-To-header - see section 6.7), 
       "junk" (often used by serving agents), and "control" (likewise);
     o Any newsgroup-name beginning with "control." (used as pseudo-
       newsgroups by many serving agents);
     o Any newsgroup-name containing the component "ctl" (likewise);
     o "to" or any newsgroup-name beginning with "to." (reserved for the 
       ihave/sendme protocol described in section 7.4, and for test
       articles sent on an essentially point-to-point basis);
     o Any newsgroup-name beginning with "example." (reserved for
       examples in this and other standards);
     o Any newsgroup-name containing the component "all" (because this
       is used as a wildcard in some implementations).

I think that gives the correct rule for single component newsgroups.

{There used to be a Usenet group "general" which even contained stuff when
I first started to read news, but I don't think it propagates any more.}

>bad idea and shouldn't happen.

I think articles with "control*" and "junk" in the Newsgroups header should
definitely be prohibited, and hence should not be propagated (trolls will
continue to post to control.*, but that's just what trolls do :-( ).

But such "pseudo-groups" can be read from servers, and hence can appear in
Xref headers. Therefore, the proper rule for such <newsgroup-name>s should
be

  MUST NOT appear in Newsgroups and Followup-To headers; MAY appear in
  Xref headers.

In draft-13, we also had names containing <component>s starting with "+"
and "-" which were reserved in the same way, the intention being that
servers who wanted to provide some special service to their own community
of clients (say filtered and unfiltered versions of the same group) could
do so without polluting the official namespace (so, with hindsight, the
"junk" group could have been called "+junk"). I think this would be a
useful feature to retain in USEFOR just in order to prevent further
namespace pollution by "enterprising" server admins.

There was also provision for reserved components begining with "_" for
"future extensions". It is not so clear whether those should be prohibited
from propagating.

>created example.foo.600 and article 600 arrives in example.foo.

I agree this prohibition should remain (but see suggestion for a NOTE
above).

>....  It's just hard to work

We sometimes forget that newsreaders sometimes consult the newsspool
directly, without using NNTP as a conduit. For example, you can do this in
nn (I do) and no doubt in things like trn also.

>also in C News,...

And USEPRO still documents how to use them. And there was certainly
evidence of them still being used a couple of years ago.

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Mon, 13 Jun 2005 10:56:34 -0400

> in any way different from what he was doing beforehand.

These documents will obsolete previous standards. They must be a complete
specification, suitable for implementation of all software which processes
articles.

"Nothing....in any way different from what he was doing beforehand"
is not a specification, much less a complete specification.

---

From: Russ Allbery <[email-masked]>
Date: Mon, 13 Jun 2005 08:49:04 -0700

Charles Lindsey <[email-masked]> writes:
> Russ Allbery <[email-masked]> writes:

> When the WG last discussed this (many years ago) it was claimed that
> some software treated newsgroup-names case insensitively and some
> didn't. We rejected the idea of allowing injecting agents to "fix it
> up", and so we concluded that a ban on uppercase was the safest option.

Yeah, I'm leaning more and more towards prohibiting them because of the
case-insensitivity mess.

> into USEFOR more-or-less as it stands:

[...]

Hm, that sure looks mostly like USEPRO material to me.

USEFOR should deal with what newsgroup names are *syntactically* valid and
discuss newsgroup names that are *always* forbidden.  If the newsgroup
name is just reserved for special purposes, that's exactly the sort of
thing that USEPRO is for.  Given that reasoning, the only ones of these

> {There used to be a Usenet group "general" which even contained stuff
> when I first started to read news, but I don't think it propagates any
> more.}

Why would it have stopped?

newsfeed:~/log> grep ' general$' unwanted.log 
322514 general

Old newsgroups never die; they only become spam traps.

> I think articles with "control*" and "junk" in the Newsgroups header
> should definitely be prohibited, and hence should not be propagated
> (trolls will continue to post to control.*, but that's just what trolls
> do :-( ).

I'm not really seeing what harm is caused by it, other than it being
confusing.  Servers that don't want to can easily refuse to propagate such
articles.

> servers who wanted to provide some special service to their own
> community of clients (say filtered and unfiltered versions of the same
> group) could do so without polluting the official namespace (so, with
> hindsight, the "junk" group could have been called "+junk"). I think
> this would be a useful feature to retain in USEFOR just in order to
> prevent further namespace pollution by "enterprising" server admins.

> "future extensions". It is not so clear whether those should be
> prohibited from propagating.

If we're going to reserve one of these characters for future
standardization purposes, it needs to propagate.  All of the reasons for
reserving a group name that I can think of would need them to propagate
(such as using it for punycode).

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 13 Jun 2005 14:28:40 GMT

>   reason (e.g. a mail reply).

No, if you are generating a mail reply, you use the rules given in RFC
2822, not the rules I have written for USEPRO (though they are near enough
the same).

>Then don't talk about it, or add the In-Reply-To blurb here:

RFC 2822 makes a comparable remark.

And if we do things differently than in RFC 2822, then this needs to be
pointed out (and since this is a protocol difference, not a format
difference, USEPRO is the proper place to do it).

>There is no "guarantee", this is bad advice for implementors.

If you follow the procedure given, then that is the result that you will
get. Of course, I cannot "guarantee" what will happen it you don't follow
the procedure.

>reported another case, where this "guarantee" simply crashed.

No, somebody had evidently not followed the procedure. Garbage in ->
garbage out.

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 13 Jun 2005 20:35:03 +0200

Charles Lindsey wrote:

> if you are generating a mail reply, you use the rules given
> in RFC 2822, not the rules I have written for USEPRO (though
> they are near enough the same).

Followup-To: poster is an issue for news.  If you think that
2822 is good enough, then 2046 is also good enough for partial,
and we get this short statement in USEPRO 7.6:

    [...] in the course of generating a followup.

Delete "or for some other reason" plus examples, because it's
unrelated to Followup agents.

The prose about a "Mail-Copies-To" in usepro-03 (7.6) is also
obsolete,

> if we do things differently than in RFC 2822, then this needs
> to be pointed out

The three differences:  1 - we trim, they don't.  2 - they have
an In-Reply-To, we don't.  3 - they say [CFWS], we say CFWS.

The fourth point, the worst references could be anything, is no
difference, we have it now in the definition, they have it as a
SHOULD (not a MUST, so obviously other references might exist).

> If you follow the procedure given, then that is the result
> that you will get.

No, it is not.  The given procedure gets it right for the LAST
msg-id added to the references.  It cannot guarantee anything
about OLDER msg-ids in the msg-id-list.  Among other problems
there's no guarantee that the FIRST msg-id properly identifies
the start of the thread.

> Of course, I cannot "guarantee" what will happen it you don't
> follow the procedure.

Therefore I recommend to remove this false "guarantee" before
it could confuse implementors.

> Garbage in -> garbage out.

Exactly.  Garbage in -> trimmed garbage with the precusor out.
No guarantees, no graph theory, just a msg-id-list as defined
in USEFOR.
                        Bye, Frank

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 13 Jun 2005 19:55:21 +0200

Charles Lindsey wrote:

> we concluded that a ban on uppercase was the safest option.

Please propose ABNF if that's about USEFOR-05.

> perhaps a NOTE to the effect that some existing groups with
> UC names (and also some with all-digit components) DO exist
> on Usenet

There are apparently three options for the US-ASCII letters:

      ALPHA                ; dubious with NNTP 27
      %x41-5A              ; conflict with RfC 1738
      %x41-5A / %61-7A     ; CAVEAT, case sensitive

Depending on what we pick our news-nntp-uri drafts have to be
updated.  At the moment ALPHA as in 1738 appears to be wrong.

>| The following forms of newsgroup-name MUST NOT be used
>| except for the specific purposes indicated:

>| o Newsgroup-names having only one component.  These are
>|   reserved for newsgroups whose propagation is restricted to
>|   a single host or local network

There's IMHO nothing wrong (syntactically) with the news-URLs
        <news://news.spamcop.net/spamcop>
        <news://news.gmane.org/junk>

So that MUST NOT is not what I'd expect in the ABNF.  Maybe it
is a bad idea like fancy References, but not a syntax problem.

Some prose should cover it, I tend to SHOULD NOT for the class.
In theory a newsgroup "poster" is possible.  Utter dubious and
most probably a very bad idea, but not really MUST NOT harmful.
Or is it harmful ?

> MUST NOT appear in Newsgroups and Followup-To headers; MAY
> appear in Xref headers.

I don't see a technical problem with "junk" in the Newsgroups.

 [only digits]
> I agree this prohibition should remain

Corresponding ABNF proposed elsewhere, essentially two options:
At least one non-digit or (as in s-o-1036) at least one letter.

                           Bye, Frank

---

From: "Ruud H.G. van Tol" <[email-masked]>
Date: Mon, 13 Jun 2005 22:15:22 +0200

>        is used as a wildcard in some implementations).

No doubt old news, but I see several in my newsgroups-list.
Most in alt.*
Other example: sol.lists.freebsd.cvs.all.digest

-- 
Grtz, Ruud

---

From: Richard Clayton <[email-masked]>
Date: Mon, 13 Jun 2005 21:21:17 +0100

In message <[email-masked]>, Charles Lindsey
<[email-masked]> writes

>>>   identifiers MUST NOT be removed.

[snip]

>>     words, if that's necessary....

looks as if it might be :(

I've tried to use header field etc as they should be used, but have not
checked if the other jargon I've chucked in is entirely consistent. That
should however be reasonably simple to adjust

>   parent (subject to trimming as described below).

The field body for the new article's References header field MUST be
constructed as follows. The field body from the parent's References
header field (if any) is placed first followed by CFWS. The field body
is completed by adding the field body from the Message-ID header field
of the parent. Any comments that are present within the field body MAY
be removed.

>   identifiers always remain.

If the total length of the resulting References header field (which
includes the 12 character header field name at the start, but not the
terminating CRLF) exceeds 998 octets it MUST be trimmed and it MAY be
trimmed at any length. Trimming involves the removal of any number of
message identifiers from the field body except that the first and the
last two message identifiers MUST NOT be removed.

>I am not sure that it really better (and it is certainly longer).

well I propose my wording as more direct (and addressing the issues that
I raised, as well as the comments issue someone else mentioned)

>paragraphs, then circular references cannot arise. 

correct, but not the point I was making

>If, by some accident,

that also wasn't the point I was making :(

I was suggesting that implementors who realised that there was a problem
might feel that the note obligated them to arrange that what went out
was not garbage (as would sometimes be possible). Clearly that's not the
intent, so we should arrange that it is clear.

>would prefer it to have been said in USEFOR - see my reply to Forrest).

I would prefer it not to be said at all, except perhaps as a general
hand-waving comment to say that the intent of the References header
field and the rules on trimming are to make it easy to assign articles
to the correct thread and to place them in an appropriate tree-like
structure provided that propagation is not too awful. Of course, we
would add, there may be mangling going on, so programs should take steps
not to be caught out by nonsense.

- -- 
richard @ highwayman . com                       "Nothing seems the same
                          Still you never see the change from day to day
                                And no-one notices the customs slip away"

---

From: "Ruud H.G. van Tol" <[email-masked]>
Date: Mon, 13 Jun 2005 22:48:34 +0200

Charles Lindsey schreef:

> trimmed, but in such a manner that the first and the
> last two message identifiers always remain.

I still can read that as if the first two and the last two message
identifiers (so four) must remain. Of course it is better that they do,
because that enhances the possibility that one of those articles is
still known and/or received. Maybe just remove the 'two'.

-- 
Grtz, Ruud

---

From: Seth Breidbart <[email-masked]>
Date: Mon, 13 Jun 2005 16:59:00 -0400 (EDT)

> identifiers (so four) must remain.

To avoid that possible (incorrect, but possible) ambiguity, how about
wording it as "trimmed, but in such a manner that the first and the
two last message identifiers always remain."?  (I would think that
"first and two last" is harder to misinterpret than "first and last
two" due to the lack of possible parallelism.)

Seth

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Mon, 13 Jun 2005 17:14:45 -0400

> than when they are raised.  Delaying they entry of tickets is completely

Limited time on my hands. I have a hard time just keeping up with the 
USEFOR issues.
More people entering tickets (and tracking followup on tickets) would help, 
if we could avoid stumbling over each others' feet.

> until you will allow discussion.

So far, I'm unconvinced of the need for the strong linkage that Charles 
seems to assume is needed; since he's holding the pen on USEPRO, I can 
understand that he tends to see things in terms of USEPRO changes, but I 
haven't seen a case yet where people (apart from Charles) argued one way or 
another about what was needed in USEFOR based on the USEPRO changes.

                   Harald

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 14 Jun 2005 12:14:26 GMT

>>        is used as a wildcard in some implementations).

Whilst "junk", "control.*", "example.*", plus the ones with conponents
starting with "+" or "-" (if we retain that distinction) are reserved to
special purposes, and might be dealt with in USEPRO, I think "poster" and
"to.*" need to be put alongside "ctl" and "all" in USEFOR, since some very
funny things can happen if people try to post articles into them.

My view is that it is simpler to deprecate them all in one place, but if
the WG or our Chair wants them split between USEFOR and USEPRO in that
manner, then it could be done. More opinions?

>(such as using it for punycode).

The current wording in USEFOR is:

   Components beginning with underline ("_") are reserved for use by
   future versions of this standard and MUST NOT occur in <newsgroup-
   name>s (whether in Newsgroups headers or in newgroup control messages
   [USEPRO]).  However, such names MUST be accepted.

Is that OK? It seems to allow a new standard to say "OK, you can now start
to generate them with the following meaning".

---

From: Seth Breidbart <[email-masked]>
Date: Tue, 14 Jun 2005 12:54:57 -0400 (EDT)

> manner, then it could be done. More opinions?

Deprecate them all in one place (even if some of them are "officially"
deprecated there, and the others are mentioned in a NOTE as being
deprecated elsewhere).  Failing to do so will confuse implementors.

Anyplace that ANY newsgroup names are deprecated MUST mention (at
least by pointer) ALL deprecated newsgroup names.

Seth

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 14 Jun 2005 13:49:59 GMT

>Other example: sol.lists.freebsd.cvs.all.digest

Which is a pity, because alt.* is one hierarchy where you may want to be
extremely picky as to what groups you allow for in your sys file, but in
CNews you could not do so if you wanted to distinguish between similar
groups with and without "all" in them.

I would presume that any INN site will have refused to create that group,
and any others like it.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 14 Jun 2005 13:43:51 GMT

>Please propose ABNF if that's about USEFOR-05.

   component       =  1*component-char ; but containing at least
                                       ; one non-DIGIT

   component-char  =  %x61-7a / DIGIT / "+" / "-" / "_"
                                       ; uppercase letters excluded

>> UC names (and also some with all-digit components) DO exist

     NOTE: the above syntax excludes uppercase letters (because of
     uncertainties as to whether they will be treated case insensitively
     by existing software) and all-digit components (because this can
     cause difficulties for a widely deployed storage technique). The fact
     that a small number of newsgroups have been formed on Usenet in
     violation of these restrictions should not be seen as legitimizing
     the creation of further non-compliant newsgroups.

>updated.  At the moment ALPHA as in 1738 appears to be wrong.

No, those drafts (well, mine at least) allows the wider set of
newsgroup-names that the latest NNTP draft allows (which deliberately
leaves it to the news format standards to specify a more restricted set
for actual use).

So you are entitled to ask for <news://news.example.net/alt.2600> and, in
that particular case, you will likely get it.

>        <news://news.gmane.org/junk>

Indeed, but these are intended to be "Read Only" groups as far as Usenet
is concerned. So by all means use those URI in order to read the groups,
but don't expect to be able to POST to them.

>is a bad idea like fancy References, but not a syntax problem.

If it's a Read Only group, then its name MUST NOT appear in a Newsgroups
header (thogh it MAY appear in an Xref header).

>most probably a very bad idea, but not really MUST NOT harmful.

I think it probably is. Suppose an article has
   Newsgroups: poster,misc.test
and you want to followup to the "poster" group only :-( .

---

From: Russ Allbery <[email-masked]>
Date: Tue, 14 Jun 2005 11:02:11 -0700

> but don't expect to be able to POST to them.

I am fairly certain this is not the case for spamcop.

> header (thogh it MAY appear in an Xref header).

There are plenty of articles out there that have "junk" in the Newsgroups
header.  I understand why you don't like the practice, but the fact of the
matter is that it doesn't break the protocol.  => USEPRO or even USEAGE.

---

From: Russ Allbery <[email-masked]>
Date: Tue, 14 Jun 2005 11:05:00 -0700

Charles Lindsey <[email-masked]> writes:

> special purposes, and might be dealt with in USEPRO, I think "poster"
> and "to.*" need to be put alongside "ctl" and "all" in USEFOR, since
> some very funny things can happen if people try to post articles into
> them.

The special things that happen with to.* are protocol things, not format
things, and should be put into the protocol document; otherwise, the split
that we're creating is rather meaningless.

"ctl" and "all" *are* USEFOR things, but I think it may be an open
question at this point whether we really care to continue to cater to the
obsolete wildcard format of C News, given that apparently some people have
started to ignore this with alt.* groups that are propagating.  Maybe it's
time to move that restriction to a SHOULD NOT instead of a MUST NOT with a
note that the restriction is likely to be removed entirely in a future
revision of the standard.

> manner, then it could be done. More opinions?

I think it should be split.

> Is that OK? It seems to allow a new standard to say "OK, you can now
> start to generate them with the following meaning".

I like Harald's wording better.

---

From: Russ Allbery <[email-masked]>
Date: Tue, 14 Jun 2005 11:08:15 -0700

Charles Lindsey <[email-masked]> writes:

> I would presume that any INN site will have refused to create that
> group, and any others like it.

INN will refuse to honor a control message for it with the default
controlchan configuration (which can be changed).  If the newsgroup were
created manually with ctlinnd, INN would have no problems with it.

INN assigns no special meaning to "all" or "ctl" name components (with one
exception) and treats such groups like any other group internally; the
only place they're handled separately is in auditing newsgroup names in
control messages.  The one exception is that if a control message (with a
Control header) is posted to a group whose name ends in example.foo.ctl,
it is treated as if it were posted to example.foo for propagation
purposes.

---

From: Henry Spencer <[email-masked]>
Date: Tue, 14 Jun 2005 15:13:07 -0400 (EDT)

> least by pointer) ALL deprecated newsgroup names.

I concur.  It is very important that all restrictions and prohibitions be
together, at least logically (i.e., after following cross-reference
pointers).  If the discussion is split, each and every part of it MUST
cross-reference all other parts, no exceptions.

Keeping it all together in one place would make the most sense.  The
format/protocol split is supposed to be *helpful*, not a religious law
which must be followed no matter how much it hurts readability. 

                                                          Henry Spencer
                                                       [email-masked]

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 14 Jun 2005 20:00:51 GMT

>Followup-To: poster is an issue for news.

USEPRO tells you to email it in that case. It is not the job of our drafts
to tell you how to email things.

>unrelated to Followup agents.

No, that section applies to *all* References headers, not just to
followups.

>The prose about a "Mail-Copies-To" in usepro-03 (7.6) is also

Yes, it is gone now.

>an In-Reply-To, we don't.  3 - they say [CFWS], we say CFWS.

And 4 - we MUST have them in followups; 2822 says SHOULD have them in
replies.

>SHOULD (not a MUST, so obviously other references might exist).

No, that is not the meaning of that SHOULD in RFC 2822. They say 'reply
messages SHOULD have "In-Reply-To:" and "References:' fields. I don't
think it ever occurred to them that someone might want to use References
in messages that were not replies. They do not forbid it, but they give no
guidance on how to do it.

>there's no guarantee that the FIRST msg-id properly identifies

If the OLDER msg-ids were not created by the proper procedure, then they
were garbage already. If _every_ msg-id in the header was put there by
some previous agent using the procedure I gave, then the result is indeed
guaranteed.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 14 Jun 2005 19:39:54 GMT

>I've tried to use header field etc as they should be used,

No, our Chair has not ruled on whether to make that change or not. Let us
deal with one issue at once.

>of the parent. Any comments that are present within the field body MAY

OK, but before doing that I want to redefine the term "header content",
within USEPRO as follows:

   The "semantic content" (often abbreviated to just "content") of a
   structured header is its semantic interpretation (cf. use of that
   term in [RFC 2822]); i.e. after unfolding it, ignoring comments and
   other semantically invisible items, replacing white space by a single
   SP, and after decoding and/or recombining any <encoded-word>s or
   <extended-parameter>s [RFC 2231].
[That assumes that "header content" in USEFOR will be redefined as
"header body".]

With that, I can tidy up a whole bunch of places where USEPRO currently
uses the term "content". So, let us start with References headers:

   The (semantic) content of the new article's References header
   consists of the content of the Message-ID header of the parent
   preceded, if the parent had a References header, by the content of
   that References header and a SP (subject to trimming as described
   below).

That gives you carte blanche to refold the new header, to retain or remove
the old comments, and even to insert new ones.

>last two message identifiers MUST NOT be removed.

No, you have omitted the bit about it being a limit on the unfolded length
(because some broken agents out there try to do the whole thing in a
single 998 character buffer).

   If the total length of the resulting References header after
   unfolding (including its header name but not the final CRLF) exceeds
   998 characters, it MUST be trimmed (and otherwise it MAY be trimmed).
   Trimming involves removing any number of message identifiers from its
   content, except that the first message identifier and the last two
   MUST NOT be removed.

OK, I expect all that still needs a little polishing.

>hand-waving comment

Well, being in a NOTE, it was _meant_ to be hand waving ...

> to say that the intent of the References header

And I would like to say all those things too, but in USEFOR. See my reply
to Forrest, and let us see where that leads to.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 14 Jun 2005 18:10:29 GMT

>specification, suitable for implementation of all software which processes

Yes, and there _is_ a complete specification, because USEFOR tells you the
syntax of the References header, and USEPRO tells you how to construct it.
So long as implementors can assume it has been constructed in that way,
implementors (of User Agents, in particular) can go ahead and use it in
any way they see fit. The corresponding header in RFC 2822 was defined in
much the same way, and implementors have managed to figure out how to use
that one.

Now I will grant you that what is said is minimal, and it is said in the
wrong place. Will you join me in asking the Chair to raise a ticket to put
some reasonable semantics back into USEFOR?

>is not a specification, much less a complete specification.

That sentence was not intended as a spedification. It was a simple
statement of fact.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 14 Jun 2005 19:42:38 GMT

>> identifiers (so four) must remain.

Ah! Now I see the ambiguity.

>two" due to the lack of possible parallelism.)

What I have actually put now is

"... that the first message identifier and the last two always remain."

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 15 Jun 2005 08:21:58 -0400

> guidance on how to do it.

Ned was there at the time; he wrote the text about References in MIME 
message/partial. I don't think we should assume he had forgotten it by that 
time......

                   Harald

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 15 Jun 2005 10:35:46 -0400

After exchanging mail with Alexey on the subject, we have agreed that he'll 
take on the task of entering tickets for USEPRO, while I continue entering 
tickets for USEFOR.

We'll continue to follow the <document> <section> naming, so all USEFOR 
tickets will start with USEFOR, while all USEPRO tickets will start with 
USEPRO.

A request from the ticket enterer.....
If you want to raise a new issue, and want to make sure it gets a ticket, 
please use a subject line saying something like

NEW: USEPRO 3.14.159 Section numbering is irrational

This will make the job of entering tickets easier.....

              Harald

--On 13. juni 2005 17:14 -0400 Harald Tveit Alvestrand 
<[email-masked]> wrote:

> More people entering tickets (and tracking followup on tickets) would
> help, if we could avoid stumbling over each others' feet.

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 15 Jun 2005 14:17:45 GMT

>"ctl" and "all" *are* USEFOR things, ...

I don't follow the distinction you are trying to make here. They are all
things that can cause unexpected things to happen if you try to use them
in normal newsgroup names. It seems artificial to try to distinguish
between them.

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 15 Jun 2005 14:09:56 GMT

>I am fairly certain this is not the case for spamcop.

Then it must be a "private arrengement". I think it would have been better
in that case if they had constructed a conventional 2-level hierarchy, but
if it's there, then it's there.

>matter is that it doesn't break the protocol.  => USEPRO or even USEAGE.

Indeed - hence the 'category 2' I just suggested. But is there any actual
purpose in including "junk" in the Newsgroups header (apart from Trolls
just showing that they can do it). For sure, the contents of "junk" will
be entirely different on each server, so it will hardly function as a
normal group.

---

From: Russ Allbery <[email-masked]>
Date: Wed, 15 Jun 2005 11:05:17 -0700

>>> Indeed, but these are intended to be "Read Only" groups as far as
>>> Usenet is concerned. So by all means use those URI in order to read
>>> the groups, but don't expect to be able to POST to them.

> Then it must be a "private arrengement".

Why, because you peresonally think that single-level newsgroup names
shouldn't be allowed?  I see no reason to classify this as a "private
arrangement" rather than a normal use of the news protocols.

> Indeed - hence the 'category 2' I just suggested. But is there any
> actual purpose in including "junk" in the Newsgroups header (apart from
> Trolls just showing that they can do it).

Why is this question even relevant?

> For sure, the contents of "junk" will be entirely different on each
> server, so it will hardly function as a normal group.

On servers that don't use junk for rejected articles (the vast majority of
them), the contents will likely be about as much the same as any other
poorly propagating newsgroup.

---

From: Russ Allbery <[email-masked]>
Date: Wed, 15 Jun 2005 11:06:53 -0700

>> The special things that happen with to.* are protocol things, not
>> format things, and should be put into the protocol document; otherwise,
>> the split that we're creating is rather meaningless.

> in normal newsgroup names. It seems artificial to try to distinguish

The distinction that I'm making here is between the syntactic definition
of a newsgroup name that can be broadly applied in all contexts and the
protocol use of a newsgroup name where particular newsgroups are reserved
in some contexts but not others.  The latter cannot be explained without
reference to the news protocol, and therefore is to me clearly a protocol
issue.

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 15 Jun 2005 22:57:24 +0200

Charles Lindsey wrote:

> I want to redefine the term "header content", within USEPRO
> as follows:

>| The "semantic content" (often abbreviated to just "content")
>| of a structured header is its semantic interpretation (cf.
>| use of that term in [RFC 2822]); i.e. after unfolding it,
>| ignoring comments and other semantically invisible items,
>| replacing white space by a single SP

What else is "invisible" except from replacing as much FWS as
possible by a single SP ?

>| and after decoding and/or recombining any <encoded-word>s
>| or <extended-parameter>s [RFC 2231].

Argh...  the beauty of MIME is that this decoding is normally
unnecessary, and especially in-transit it's _never_ necessary.

> [That assumes that "header content" in USEFOR will be
> redefined as "header body".]

I'm on the brink to join Bruce's "hacker slang" camp if we now
get different terms for the same "structured header field body"
depending on the draft in question.

>| The (semantic) content of the new article's References
>| header consists of the content of the Message-ID header of
>| the parent preceded, if the parent had a References header,
>| by the content of that References header and a SP (subject
>| to trimming as described below).

A completely new terminology instead of simply saying "grab
old msg-id-list, add SP and parent's msg-id, then trim it as
described below" ?  Of course you can replace CFWS by SP, but
that's not a feature restricted to References, it's normal.

> you have omitted the bit about it being a limit on the
> unfolded length (because some broken agents out there try
> to do the whole thing in a single 998 character buffer).

In my case it was an injection-agent (= news server) telling
me when I have to trim excessively long references manually.

                       Bye, Frank

---

From: Richard Clayton <[email-masked]>
Date: Wed, 15 Jun 2005 22:00:31 +0100

In message <[email-masked]>, Charles Lindsey
<[email-masked]> writes

>uses the term "content". 

Accepting that a word for "the meat of what's really there" is useful;
then it's a shame that it isn't a word that resembles the RFC2822 "field
body" terminology; but I've no immediate suggestions that are any better
than "semantic content"   ....  though I really don't think you need to
explicitly appeal to 2822 for the meaning of "semantic" :)

>So, let us start with References headers:
>

OK so far

>the old comments, and even to insert new ones.

OK, but surely I don't need permission to fold anything, I just can.

>No, you have omitted the bit about it being a limit on the unfolded length

erm... the header field _is_ the unfolded form :)

>   998 characters, it MUST be trimmed (and otherwise it MAY be trimmed).

this wording is backwards :(  You've just created the new References
header, so why would you have even thought about folding it yet ? Any
implementation is bound to measure it, trim it, and only then can it
fold it as needed. So what the specification should explain is what the
measurement rule is...

You could just say that the "semantic content" mustn't exceed 986 but I
suspect you don't mean that, you mean the total length including any
comments (ugh!) but you don't want to include anything extra for when
you fold it...    As far as I can see that's just the "field body" plus
12 or in fact the header field (which includes the field name).

>   content, except that the first message identifier and the last two

OK with content here

- -- 
richard                                              Richard Clayton

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 16 Jun 2005 12:05:10 GMT

>reference to the news protocol, and therefore is to me clearly a protocol

Ah! You are saying that "ctl" and "all" MUST NEVER EVER appear, but "To"
is different because it MAY appear if you are running a UUCP feed with the
old IHAVE messages. I still think the distinction is somewhat artificial
when it comes to deciding where to deprecate/discourage these things,
though the exact wording used must be chosen with care.

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 16 Jun 2005 12:11:50 GMT

>possible by a single SP ?

The words "semantically invisible" are used at several places in RFC 2822.

>unnecessary, and especially in-transit it's _never_ necessary.

I am not sure yet whether that bit is needed in USEPRO. I need to go
through and see all the places where "content" is used. When I have done
that, I may be able to simplify that definition.

>depending on the draft in question.

I want USEFOR to use the term "header body" for the syntactic object,
before any unfolding or other semantic simplifications. I have looked at
all the places where USEFOR currently used "header content", and in every
case it is the syntactic, not the semantic, object that is meant.

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 16 Jun 2005 11:48:23 GMT

>body" terminology;

What we actually call the new term can be adjusted when Harald has ruled
on that matter. But keeping "content" in it somewhere seems to agree with
some existing usage of that term.

> but I've no immediate suggestions that are any better

Well RFC 2822 uses "semantic in many places, and I was just trying to show
my credentials :-) . In fact, the precise wording of that definition is
probably an overkill. E.g. the RFC 2231 stuff is not necessary for
applying it to the References header, but until I have gone through the
whole of USEPRO examining existing use of "content", I cannot be sure.

>OK, but surely I don't need permission to fold anything, I just can.

The old version told you to take syntactic objects, such as the header
body (aka header field body, if you must) of an existing References
header, and simply to concatenate them. That buys you whatever folding and
comments were already built into those objects. So, legally speaking, you
were obliged to retain them. Yes, that law was "a ass"; which is why I
want to change it.

>>>If the total length of the resulting References header field .....
>>

>erm... the header field _is_ the unfolded form :)

No, "header field" as defined in RFC 2822 is the syntactic object as it
appears on the wire - folds and all.

>header, so why would you have even thought about folding it yet ?

No, I have just created a References header whose *semantic*content*
contains all the right msg-ids. Who knows whether it is folded or not? Now
I want to say that you had better check that it would be less than 998 if
you were to unfold it (because some poorly designed implementation might
try to unfold it and process it further in a 998 buffer).

Note that there is a general permission in USEPRO that you don't have to
do things in the order stated, so long as the effects achieved are
indistinguishable. So a sensible implementor will probably do his counting
before folding, and indeed before attempting the concatenation.

>.... So what the specification should explain is what the
>measurement rule is...

Yes. That is what I was trying to do: "do a test unfold, and see if it is
too long". Here is another attempt:

   If the resulting References header would, after unfolding, exceed 998
   characters in length (including its header name but not the final
   CRLF), it MUST be trimmed (and otherwise it MAY be trimmed). Trimming
   involves removing any number of message identifiers from its content,
   except that the first message identifier and the last two MUST NOT be
   removed.

---

From: Russ Allbery <[email-masked]>
Date: Thu, 16 Jun 2005 09:41:35 -0700

>> The distinction that I'm making here is between the syntactic
>> definition of a newsgroup name that can be broadly applied in all
>> contexts and the protocol use of a newsgroup name where particular
>> newsgroups are reserved in some contexts but not others.  The latter
>> cannot be explained without reference to the news protocol, and
>> therefore is to me clearly a protocol issue.

> is different because it MAY appear if you are running a UUCP feed with
> the old IHAVE messages. I still think the distinction is somewhat
> artificial when it comes to deciding where to deprecate/discourage these
> things, though the exact wording used must be chosen with care.

I'm saying that "all" and "ctl" are not permissable anywhere, whereas
"to.*" (for example) is a perfectly valid thing to put into the Newsgroups
header (and in fact has to be allowed in the Newsgroups header to work at
all), just not a valid target for control messages and some other
purposes.  (And I don't think the control message restriction should be
handled in the description of the syntax of the control header; that's
really just the wrong place for such things, IMO.)

In other words, I'm looking at the distinction between the ABNF for a
newsgroup name, which has to allow such things as junk or to.* or
control.* and which is used for the Newsgroups header, the Xref header,
and so on, and the rules about what newsgroups you should use for what
purpose.  The latter feels very strongly like a USEPRO issue to me.

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 16 Jun 2005 21:55:25 +0200

> The words "semantically invisible" are used at several places

Found it, the other places are backslashes in quoted-pairs, and
ignoring these "invisible" characters in a msg-id-list is a bad
plan for the References construction and trimming algorithm.

 [2047 decoding] 
> I am not sure yet whether that bit is needed in USEPRO. I
> need to go through and see all the places where "content" is
> used. When I have done that, I may be able to simplify that
> definition.

Better stay away from "semantical content" for the References,
otherwise you'd first remove the backslashes and later reinsert
them.  With Murphy the output differs from the input, and we're
not very interested to fix a non-canonical msg-id in this case.

                       Bye, Frank

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Fri, 17 Jun 2005 12:05:11 +0200

--On torsdag, juni 16, 2005 12:11:50 +0000 Charles Lindsey 
<[email-masked]> wrote:

>  am not sure yet whether that bit is needed in USEPRO. I need to go

The term "header content" (RFC 2822 "field body") is used in 3 places in 
USEFOR, all of which are in this paragraph:

>    o  All agents MUST generate headers so that at least one space
>       immediately follows the ':' separating the header name and the
>       header contents (for compatibility with deployed software).  News
>       agents MAY accept headers which do not contain the required space.
>
>    o  The header contents of every header line (including the first and
>       any that are subsequently folded) MUST contain at least one non-
>       whitespace character.
>
>          NOTE: This means that no header content defined by or
>          referenced by this document can be empty.  As a result, this
>          document updates the <unstructured> construct from Section
>          3.2.6 of [RFC2822] as follows:
>
>    unstructured    =  1*( [FWS] utext ) [FWS]

The first instance seems to say that the space after the colon is not 
considered part of the header content.
The second one is formally wrong (you can't have "header content" of a 
"header line", since "header content" means the whole header field's 
content).
The third one implicitly uses "empty" in the sense of "something that is 
not whitespace". That's not clear either.

Since this paragraph needs some rework anyway to be totally 
self-consistent, perhaps we should free up "content" for what Charles 
wants, and define as follows:

  "field body" (as in RFC 2822) - the characters after the colon up to the 
end of the header field; we don't need to define this, since 2822 does.
The only place we need that is probably the paragraph above.

  "header field content" ("content" for short) - the stuff inside the 
header that is not leading or trailing whitespace, is not a comment, and is 
not the CRLF between header lines (we can surely make up better text for 
this!)

Does that make sense?

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 17 Jun 2005 11:30:51 GMT

>plan for the References construction and trimming algorithm.

No, look at our current syntax for <msg-id>. There is not a <quoted-pair>
in sight. All you have is <mqspecial> and nobody has ever said that
anything in an <mqspecial> is "invisible".

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 17 Jun 2005 13:29:53 GMT

>purpose.  The latter feels very strongly like a USEPRO issue to me.

Except that several people have expressed a strong preference for keeping
all restrictions of this nature at one place, in USEFOR.

But, of course, any prohibition of "To" should be accompanied by a wording
such as:

    (reserved for the ihave/sendme protocol described in [USEPRO], and
    for test articles sent on an essentially point-to-point basis).

I found those words in the old draft-13. Why-oh-why are we spending so
much time and effort on rediscovering wheels which were invented years ago
in our earlier drafts?

---

From: Russ Allbery <[email-masked]>
Date: Fri, 17 Jun 2005 09:44:36 -0700

> Except that several people have expressed a strong preference for
> keeping all restrictions of this nature at one place, in USEFOR.

I know.  I understand their point, but I disagree with those people.  If
we have to put all of the restrictions in the same place, I think putting
them all in USEPRO is actually the best of a couple of bad options and
only put the permissive ABNF into USEFOR (since in practice that's a
common implementation strategy anyway -- namely, to allow anything in the
software but then add restrictions in policy).

> But, of course, any prohibition of "To" should be accompanied by a
> wording such as:

> much time and effort on rediscovering wheels which were invented years
> ago in our earlier drafts?

Because what we had years ago in our earlier drafts was not publishable.
There is no magic pixie dust in the above wording that makes me happy with
calling to.* newsgroups a format issue.

---

From: Henry Spencer <[email-masked]>
Date: Fri, 17 Jun 2005 12:55:48 -0400 (EDT)

> calling to.* newsgroups a format issue.

One of the things which certainly has slowed this process down greatly is
people who obsess about (e.g.) how a "format" document isn't supposed to
talk about anything but "format" (defined in some narrow way), instead of
being willing to make pragmatic compromises to convey the necessary
information most effectively.

                                                          Henry Spencer
                                                       [email-masked]

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 17 Jun 2005 19:28:07 +0200

Harald Tveit Alvestrand wrote:

> The term "header content" (RFC 2822 "field body") is used in
> 3 places in USEFOR, all of which are in this paragraph:

I try to translate this for you, otherwise you'd be forced to
read several hundred rounds of ping pong in the [FWS]-threads:

>| All agents MUST generate headers so that at least one space
>| immediately follows the ':' separating the header name and
>| the header contents (for compatibility with deployed
>| software).

That says we really want SP (not WSP or FWS) after the colon.

>| News agents MAY accept headers which do not contain the
>| required space.

Optionally we'd accept no separator (or a HT) after the colon,
but we never generate header fields this way.

>| The header contents of every header line (including the
>| first and any that are subsequently folded) MUST contain at
>| least one non-whitespace character.

That says we don't allow 2*(CRLF 1*WSP) as in LWSP with an
apparently empty line.  Same concept as in 2822 FWS or CFWS.

It also says that we don't allow a first folding before the
first non-whitespace character in the header field body.

So you can have a Subject:<SP><WSP>hello<CRLF>
<WSP>world but not a Subject:<SP><CRLF>
<WSP>hello world.

That's a major difference from RfC 2822, and it's important.

It's an extension of an 2822 "MUST NOT" about CFWS resulting
in apparently empty lines in the header.

>| NOTE: This means that no header content defined by or
>| referenced by this document can be empty.

That says that we don't have e.g. empty Bcc: headers like mail.

>| As a result, this document updates the <unstructured>
>| construct from Section 3.2.6 of [RFC2822] as follows:

>| unstructured    =  1*( [FWS] utext ) [FWS]

We also have no apparently empty Subject, Summary, or Comments.

We want at least one VCHAR or NO-WS-CTL in every line of the
folded header field body.  Actually some of "us" (= me) think
that NO-WS-CTL is madness, but it's no syntax problem here:

Users are free to experiment with FF, VT, etc. as they see fit,
only for the msg-id we explicitly killed this weird NO-WS-CTL.

Replacing <utext> by ( VCHAR / NO-WS-CTL ) is better:  Without
<obs-utext> that's just what it is.  With a "note" to send all
complaints to the persons listed in RfC 2822 chapter 8.  <beg>

The <obs-utext> bit is about isolated CR or LF outside of CRLF,
so we really don't want this anymore in Netnews header fields.

> The second one is formally wrong (you can't have "header
> content" of a "header line", since "header content" means the
> whole header field's content).

It's about the part of the header field body in a folded header
field line.  Maybe modulo terminology that makes sense for me.

> The third one implicitly uses "empty" in the sense of
> "something that is not whitespace". That's not clear either.

It uses "not empty" in the sense of ( VCHAR / NO-WS-CTL ), "no
header content" [...] "can be empty".  Where "empty" is simply
( WSP / CR / LF ) which is just the same as ( FWS / CR / LF ).

> perhaps we should free up "content" for what Charles wants,
[...]
> Does that make sense?

Yes.  You've almost hit another USEFOR rathole from my POV:
Using [FWS] instead of *WSP only because RfC 2822 says [CFWS].

                            Bye, Frank

---

From: Russ Allbery <[email-masked]>
Date: Fri, 17 Jun 2005 10:37:02 -0700

Henry Spencer <[email-masked]> writes:

> One of the things which certainly has slowed this process down greatly
> is people who obsess about (e.g.) how a "format" document isn't supposed
> to talk about anything but "format" (defined in some narrow way),
> instead of being willing to make pragmatic compromises to convey the
> necessary information most effectively.

I find format vs. protocol to be a useful distinction for writing software
but only if it's done right.  If we just jumble things between the two
documents, it becomes much harder to read than if we had one document.  If
we want to abandon the idea and just write one document, that's something
that can be argued for, but if not, we should do the split in a way that
actually helps keep concepts separate rather than just confuse the issue.

The idea, at least in my opinion, is that one should be able to write an
article parser off the format document, and then put the policy layer of
the software around it while referencing the protocol document.

I know that you opposed splitting the document in the first place, so I'm
not surprised that you consider the discussions that result from trying to
implement the split to be a waste of time.  Certainly, if your feeling is
that it should be one document, it all *is* a waste of time.

However, my understanding of why we split the document in the beginning
was to try to speed up the process of dealing with the format document by
avoiding creeping featurism and try to tighten the scope.  Accordingly,
I'm taking the tack of resisting attempts that I see as trying to
reintroduce all the creeping featurism problems we were having into the
format document, thus miring it in all of the debates we were trying to
sidestep by splitting the documents in the first place.

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 17 Jun 2005 21:26:50 +0200

Charles Lindsey wrote:

  [backslash]
>> ignoring these "invisible" characters in a msg-id-list is a
>> bad plan for the References construction and trimming
 
> No, look at our current syntax for <msg-id>. There is not a
> <quoted-pair> in sight.

Are you sure that a valid 2822 <id-left> or <id-right> never
shows up in the References of a Netnews article ?

As an implementor of any combined MUA and news UA I'd probably
pick a very simple parser, and then the quoted-pair backslash
_is_ "semantically invisible" in both <id-left> and <id-right>,
because 2822 says so.

It's no real problem for the References if some non-canoncial
Message-IDs are mangled by a simple implementation.  But I'm
stunned by your confidence that everybody implements USEFOR
correctly only because we say MUST and hide major differences
from 2822 deep within obscure <mqspecial> or <no-fold-literal>

| The only places in this standard where quoted-pair currently
| appears are ccontent, qcontent, dcontent, no-fold-quote, and
| no-fold-literal.

How did the 2822 author get away with "this standard", Bruce on
vacation ?  Anyway, if you want to stick to a <no-fold-literal>
in your syntax you need again more prose explainig that you've
disabled this rule in USEFOR.  

With <id-quote>, <id-literal>, etc. in my syntax this problem
doesn't exist right from the start.  No extra prose needed.

                             Bye, Frank

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 17 Jun 2005 21:39:28 +0200

Charles Lindsey wrote:

> Why-oh-why are we spending so much time and effort on
> rediscovering wheels which were invented years ago in our
> earlier drafts?

Because so far not one draft back to MESSFOR got the <msg-id>
right, and so it's the natural next step to visit the other
fundamental idea of NetNews again, the <newsgroup-name>.

Apparently Harald has a knack to ask the right questions, if
we couldn't answer it we'd be in serious trouble.  Bye, Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 20 Jun 2005 10:46:28 GMT

>shows up in the References of a Netnews article ?

It might, if some gateway lets it through. But if the References header
contains <ab\[email-masked]> and the article itself contains
<[email-masked]), do not expect a news reader to thread the article in
its proper place. Come to that, I doubt many mail readers would actually
thread it correctly.

>_is_ "semantically invisible" in both <id-left> and <id-right>,

No, a "very simple parser" probably would not treat the backslash
specially at all.

>from 2822 deep within obscure <mqspecial> or <no-fold-literal>

They are not hidden. There is explanation, including the various
<[email-masked]> examples.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 20 Jun 2005 10:02:14 GMT

>considered part of the header content.

Indeed, but the definition of "header field body" in RFC 2822 clearly
includes that SP (that was the initial example which lead to the need to
define something more precise for USEPRO).

>"header line", since "header content" means the whole header field's 

Yes.

>not whitespace". That's not clear either.

Perhaps the word "(in)visible" would help.

>wants, and define as follows:

OK.

>  "header field content" ("content" for short) ...

Well it will be "header content" or "header field content" according to
whether we decide to revert to strict RFC 2822 termonology, or to retain
the "News-style" terminology in the present draft. I still vote for the
latter.

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 20 Jun 2005 22:23:00 +0200

Charles Lindsey wrote:

>> quoted-pair backslash _is_ "semantically invisible" in
>> both <id-left> and <id-right>, because 2822 says so.

> No, a "very simple parser" probably would not treat the
> backslash specially at all.

IBTD.  But this is actually a part of "#1003 fix msg-id".

The following text in 2822 is incorrect for your msg-id:

| Where any quoted-pair appears, it is to be interpreted as the
| text character alone.  That is to say, the "\" character that
| appears as part of a quoted-pair is semantically "invisible".
[...]
| The only places in this standard where quoted-pair currently
| appears are ccontent, qcontent, dcontlnt, no-fold-quote, and
| no-fold-literal.

<no-fold-quote>   is used by <id-left>
<no-fold-literal> is used by <id-right>

The "\"-problem does not exist in the <id-local>, <id-domain>,
<id-quote>, and <id-literal> incarnation of the <msg-id> ABNF.

                             Bye, Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 21 Jun 2005 10:29:37 GMT

><no-fold-literal> is used by <id-right>

But the syntax of <no-fold-quote> and <no-fold-literal> is different in
our draft than it is in RFC 2822. In particular, there is no <quoted-pair>
anywhere within it.

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 21 Jun 2005 19:45:55 +0200

Harald Tveit Alvestrand wrote:

> If you want to raise a new issue, and want to make sure it
> gets a ticket, please use a subject line saying something
> like

Last proposed "security considerations" addition was in
<http://mid.gmane.org/[email-masked]>

 MIME security considerations are discussed in [RFC2046].  Note that
 applying some [RFC2231] extensions for parameters like multi-line
 paramters on a boundary parameter as defined in [RFC2046] might be
 abused to bypass naive methods of handling parameters.  Two examples:

 Content-Type:
  multipart/mixed (; boundary=foo ; xyz=");bOuNdArY*=''next%20part(")
 Content-Type: multipart/digest;
  boundary (not=me) = ("yes ;-) simple (foo;bar") ; x-foo = xyzzy

---

From: Bruce Lilly <[email-masked]>
Date: Tue, 21 Jun 2005 20:36:09 -0400

[...]
>    except that the first message identifier and the last two MUST NOT be

The idea of per-field length limits (which did not exist in 822, 850,
1036 and do not exist in 2822) was discussed near the beginning of this
WG.  For example:
-------------------------------
Date: Mon, 02 Feb 1998 11:50:48 -0800 (PST)
From: Chris Newman <[email-masked]>
Subject: Header/Message Limits
To: [email-masked]
Message-id: <[email-masked]>
MIME-version: 1.0
Content-type: TEXT/PLAIN;
  charset=US-ASCII
Originator-Info: login-id=chris; server=THOR.INNOSOFT.COM
Resent-Message-ID: <"PFEAz3.0.hd5.yCYrq"@ambient>
Resent-From: [email-masked]
X-Mailing-List: <[email-masked]> archive/latest/2643
X-Loop: [email-masked]
Precedence: list
Resent-Sender: [email-masked]
X-UID: 12518
X-Length: 2117

The current limit for messages in email is specified in RFC 1123, section
5.3.8.  I recommend reading it.

Usenet and email currently have no specified limits on headers, and they
seem to work just fine.  Why fix something which isn't broken?

User interfaces will do what is necessary to present headers to the user,
and if they have to truncate them, they will.

Servers which wish to process arbitrary-length headers efficiently can
simply mmap() the articles (or use a fixed buffer at the max message
size) and do in-place processing with minimal calls to allocation 
functions.

If you do add length limits, find the length limits in deployed software
and document them with the same sort of advice as in the above section.

But please don't add length limits which are smaller than those which
function in deployed software.

		- Chris
-------------------------

The result of which was to affirm no such limits, and we moved on from
there.  Now you're trying to introduce a per-field limit. Why -- what
has changed recently to justify reintroducing something that was long
ago rejected?

---

From: Bruce Lilly <[email-masked]>
Date: Tue, 21 Jun 2005 22:35:26 -0400

>   boundary (not=me) = ("yes ;-) simple (foo;bar") ; x-foo = xyzzy

While I see no problem with addressing the issue per se, nor with
the specific examples, it seems that the gist of the proposed wording
is "avoid MIME" (2231 is part of the MIME specification).  I would
much prefer something similar to "it requires complete understanding
and implementation of the normative references to be compliant".  Of
course, the USEFOR draft -04 already says that (section 1.6).  If you
feel strongly that it should be posed as a security consideration, then
I suggest "A naive rather than fully compliant implementation may
make such a naive implementation vulnerable to security breaches.
Specifications have been analyzed for security implications as
documented in the relevant RFCs; deviations from those specifications
may defeat security measures built into the specifications, and the
resulting vulnerabilities might not be obvious to a non-expert".

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 22 Jun 2005 19:25:56 GMT

>has changed recently to justify reintroducing something that was long

It is indeed rather unusual to impose any limit (other than an advisory
one) on the overall length of a header.

I have put the requirement for trimming in there (it was optional before,
though recommended in USEAGE) because Frank asked for something stronger
(and nobody has disputed that).

I put that total length limit in there because (as we have both witnessed)
some existing agents do some most unpleasant things when References
headers get too long. Nobody has queried what I wrote so far, but if you
wish it to be discussed, then please ask Alexey to raise a USEPRO ticket
for it, and we will do so.

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 23 Jun 2005 02:33:42 +0200

Bruce Lilly wrote:
 
> While I see no problem with addressing the issue per se, nor
> with the specific examples, it seems that the gist of the
> proposed wording is "avoid MIME"

It's more like "be careful with insane boundary constructs".

Near the top of the USEFOR memo (2.3) there's already a "MUST":

| User agents MUST meet the definition of MIME-conformance in
| [RFC2049] and MUST also support [RFC2231].

In the "security considerations" (5.) we only need to address
the most critical consequences of this statement.

> If you feel strongly that it should be posed as a security
> consideration

Yes, obviously, because 2046 doesn't discuss comments in the
boundary, let alone any 2231 constructs.

> I suggest "A naive rather than fully compliant implementation
> may make such a naive implementation vulnerable to security
> breaches.

One "naive" too many for my taste, and I very much doubt that
2231 constructs in a boundary are what the authors intended -
in fact they say "don't do this if it's unnecesary" as general
rule.  For this reason it's also better to avoid a statement
about "fully compliant".  Maybe a 2046bis or 2231bis clearly
disallows such boundary-constructs, good riddance.

> Specifications have been analyzed for security implications
> as documented in the relevant RFCs

Both 2046 and 2231 fail miserably for the boundary parameter,
that's why we have to mention it explicitly at all.

> deviations from those specifications may defeat security
> measures built into the specifications, and the resulting
> vulnerabilities might not be obvious to a non-expert

Yes, that's the point.  It's also not obvious for about every
UA claiming to be MIME-compatible, or is it ?  Bye, Frank

---

From: Bruce Lilly <[email-masked]>
Date: Thu, 23 Jun 2005 09:09:17 -0400

> (and nobody has disputed that).

I recall no ticket item for the issue, no discussion, and no declaration
of consensus.  References?

> headers get too long.

There is no such thing as "too long" for a field.  What was observed was
the result of inappropriate folding at 990 octets to stay within the
per-line limit (which does exist).  Broken software is broken software,
and we should not impose limits which will be ineffective (a single-line
References field of 995 octets would still be broken inappropriately at
990 octets by the software in question), which are counter to agreed
upon consensus (at least without reopening discussion first), and which
run counter to established practice per RFCs 822, 850, 1036, and 2822.

> Nobody has queried what I wrote so far, but if you 

The procedure should be discuss first, and change the document (which is
supposed to reflect WG consensus) only after consensus is declared by
the Chair.  Since you are the one who apparently wishes to make a change,
you should request a discussion ticket.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Thu, 23 Jun 2005 15:56:30 +0200

Ticket #1046 assigned.

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 23 Jun 2005 17:41:47 +0200

Bruce Lilly wrote:

>> I have put the requirement for trimming in there (it was
>> optional before, though recommended in USEAGE) because Frank
>> asked for something stronger (and nobody has disputed that).

> I recall no ticket item for the issue, no discussion, and no
> declaration of consensus.  References?

<http://mid.gmane.org/[email-masked]> and many
others, scan the list for "trim".

> There is no such thing as "too long" for a field.

There is, 998 is the precise limit (unfolded without CrLf) for
the References and at least some news servers.

We have some text around this magic number, e.g. the 1+2=3
msg-ids in the References were chosen to be below this limit
in the worst case, 764 = 3*250+2+12, 12=length('References: ').
It also works for 974 = 3*(64+1+255)+14, discussed months ago.

> The procedure should be discuss first, and change the
> document (which is supposed to reflect WG consensus) only
> after consensus is declared

That's the procedure for controversial issues.  In that case
there was no controversy:  Injecting agents rejecting articles
with too long References are bad for the users, therefore the
followup agent SHOULD get it right.  Not only MAY.  That was
the only change.
                          Bye, Frank

---

From: Bruce Lilly <[email-masked]>
Date: Thu, 23 Jun 2005 12:26:03 -0400

> Bruce Lilly wrote:
[quoting C. Lindsey} 
> others, scan the list for "trim".

Thanks for the reference, Frank, but that's not a ticket item nor
a declaration of consensus.
 
> the References and at least some news servers.

The RFC 2822 998 limit is per line, not unfolded.
 
> That's the procedure for controversial issues.

As mentioned earlier, we have a rather long-standing agreement that
there is no field length limit.  Reintroducing one *is* controversial.

> In that case 

Charles' text (quoted a few messages back in the (untrimmed)
References field of this message) says "MUST".  Presumably we
have tickets so there can be organized discussion, a clear decision,
and text reflecting that decision.  Not something completely
different.

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 23 Jun 2005 19:28:55 +0200

Bruce Lilly wrote:
 
> that's not a ticket item nor a declaration of consensus.

2005-04-27, Harald introduced this ticket monster later.  If
you want it to get a ticket number you're supposed to use a
Subject: NEW: USEPRO 7.6 (4) ...

Maybe a bit premature, usepro-03 still says "MAY", the new
text exists only as proposal.  But I know that it's "SHOULD".

Then Alexey is supposed to find any Subject: NEW: USEPRO and
create a usepro-ticket if it's a real issue.

> The RFC 2822 998 limit is per line, not unfolded.

Yes, I know this, it's no 2822 or MIME limit.  Maybe Russ or
Henry have an idea.  S-o-1036 mumbles something about a "length
of header logical lines", maybe that's the source of this limit
1000 for unfolded References on some news servers (?)

All I want is that this SHOULD not be the problem of the user.

Either UAs get it right, or servers stop to reject articles
with longer References.  MAY (or maybe not) isn't good enough.

                         Bye, Frank

---

From: Henry Spencer <[email-masked]>
Date: Thu, 23 Jun 2005 13:57:44 -0400 (EDT)

> 1000 for unfolded References on some news servers (?)

Most likely.  In hindsight, the "if you really must" part of that
discussion in son-of-1036 isn't useful:  an injecting agent wants to know
what it can depend on, so if the standard grants even reluctant permission
to impose a limit, in practice that's a hard limit. 

> with longer References.  MAY (or maybe not) isn't good enough.

Concur.  The standard needs to state what servers *must* accept, and
consequently what injecting agents can *always* generate.

                                                          Henry Spencer
                                                       [email-masked]

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 24 Jun 2005 13:55:00 GMT

>run counter to established practice per RFCs 822, 850, 1036, and 2822.

It is clear that "established practice" is not working (as we have both
seen). Even if you carefully fold your References header, the next person
to followup to it is quite likely to unfold it again. Just what went wrong
in the particular case is not clear; what is clear is that it would not
have happened if some pruning had taken place (though whether my suggested
limit of 998 would have stopped it is not so clear).

>you should request a discussion ticket.

The text in question was written before the ticket system came into being.

Frank asked for something stronger, so I wrote a text and published it here.

It was discussed. People suggested wording changes, and they were made.
Nobody opposed the general idea of a MUST limit on the overall length
until you did so.

So if you want to pursue it, then you should ask for it to be ticketed.

It is clear that threading agents can manage perfectly well with only half
a dozen references (the minimum proposal of 3 is tight, but still
adequate).

So the issues are:
1. Do we propose a definite limit, and if so what number?
2. Is that limit a MUST prune or a SHOULD prune?