Subject: #1008 USEFOR 1.5: Suggested language on "followup"

From: Harald Tveit Alvestrand <[email-masked]>
Date: Thu, 02 Jun 2005 23:28:10 +0200

The thread following my summary doesn't seem to have created many new 
arguments or changes of opinion, so the next step is trying to suggest 
language....

The option among my 4 alternatives that seemed to be least controversial 
was this one:

2) All "followups" have a References: header.
   There exist articles that have References:, but are not "followups"

In order to make this entirely clear in USEFOR, I suggest the following 
edit:

1) To section 1.5:

OLD:

   A "followup" is an article containing a response to the contents of
   an earlier article (the followup's "precursor" indicated in the
   "References" header).

NEW:

   A "followup" is an article containing a response to the contents of
   an earlier article. (the followup's "precursor" indicated in the
   "References" header).
   All "followups" have a "References" header.
   Note: Other articles may also use the "References" header.

I am pretty sure that John Stanley disagrees with this.
If there are other people who violently object to this formulation, or who 
think that the group has or can reach a consensus that is best represented 
by a different formulation, please speak up.

                    Harald

---

From: "Ruud H.G. van Tol" <[email-masked]>
Date: Fri, 3 Jun 2005 00:11:40 +0200

>    Note: Other articles may also use the "References" header.

I can live with that, although I will still call articles "followup"
that have no "References" header but that are obviously reactions on an
earlier article.

So now I have to share this story on something that happened about an
hour ago:

Someone that posts through his ISP's newsserver (news.chello.nl)
suddenly started reacting on older articles but with quoted text from
newer articles, because he could find the newer articles 90 minutes
earlier on google.

So what he did was follow-up to an earlier posting and replacing the
body as if he was reacting on a posting that he still had to received
from his ISP's newsserver. He wrote that he did that to be closer
involved in the discussions. He obviously didn't manually update the
"References" header, so his reactions popped-up in all the wrong places
on people's screens.

Luckily the discussions were in the nl-hierarchy, and he could be
persuaded to start using the good & free news://fb1.euro.net

-- 
Grtz, Ruud

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Thu, 02 Jun 2005 23:37:53 -0400

> I am pretty sure that John Stanley disagrees with this.

I object too, for rational reasons, which have not been addressed.

Surely, "reasons" trump "opinion"?  Oh, wait, this is Usefor,
where people seriously write you can "thread on Subject."

> If there are other people who violently object to this formulation, 

I don't do "violence", unless a formal objection at Last call counts
as violence.

> or 
> who think that the group has or can reach a consensus that is best 
> represented by a different formulation, please speak up.

John and I raised objections which received no adequate response already.
Will doing it again have any different result?

My primary objection at this time is that your proposed text change is
incomplete, because other parts of USEFOR and USEPRO must be changed

   - to provide guidance on how to interpret a references header in a
     received article,

   - to limit conformant applications to insert References headers only
     when appropriate.  I believe the draft documents lack text
     allowing or restricting References in Usefor except for followups.
     That being the case, your change will open up a huge hole unless
     it gets plugged other ways.

If you show us what you intend for those changes, I might be able
to respond.  As it is, your statement "Note: Other articles may also
use the "References" header." will ensure that "threading on References"
will be as unreliable as "threading on Subject."  This direction is contrary
to the entire point of a "draft standard," which is to define formats
and protocols so that you can implement RELIABLE BEHAVIOR.

---

From: Seth Breidbart <[email-masked]>
Date: Fri, 3 Jun 2005 00:11:41 -0400 (EDT)

>   - to provide guidance on how to interpret a references header in a

Exactly as before: thread it as if the article were a followup.
That's the reason for inserting the References header.

>     allowing or restricting References in Usefor except for followups.

Appropriate is defined, roughly, as "the poster intends that this
article be treated as a followup to that one" (or use the term
"threaded" as appropriate).

That's the existing practice.

I agree, it wouldn't be wrong to address those (at least the second,
I'd have to check existing wording to see if there are any changes
needed to accommodate the first).

Seth

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Fri, 03 Jun 2005 07:21:01 +0200

> incomplete, because other parts of USEFOR and USEPRO must be changed

I have deliberately not discussed USEPRO. I believe that we can do one 
thing at a time.

>    - to provide guidance on how to interpret a references header in a

At the moment, USEFOR contains very little information about how to 
interpret ANY header. Are you suggesting to change this?

>      it gets plugged other ways.

I believe the other participants claim that this is a hole that is 
currently open, and should not be closed - people are currently using the 
References: header in messages that can only be called "followup" by a 
rather stretched definition of that term, and should not be prevented from 
doing so.

> and protocols so that you can implement RELIABLE BEHAVIOR.

My personal preference is to have USEPRO (probably) say that it's 
reasonable for newsreaders that see a References: header to make it easy 
for people to find the article referred to by the References: header when 
they are looking at the article containing it, and to find articles with a 
References: header referring to the article they are looking at, and say 
something like "One such mechanism is commonly called "threading"".
(Customarily, IETF standards do NOT specify user interfaces)

I think this description should talk only about how to treat a message with 
a References header, and not attempt to guess whether or not the message is 
a "followup".

But changing USEFOR to contain this level of information would change the 
split between USEFOR and USEPRO, and I don't see a good reason for that.

And - again - at this time, I'm only trying to get USEFOR finished.

                            Harald

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Fri, 03 Jun 2005 01:45:47 -0400

> That's the existing practice.

I agree, but that's exactly my point....

I get the feeling that people have not read the drafts....

The drafts contain no wording that standardizes the existing practice
that Seth described.  And accepting the proposed change with no
other changes is totally unacceptable.

For those who haven't read the draft, I'll spell it out....

1. The References header in usefor-usefor-04 is mentioned in only these sections:

http://www.mibsoftware.com/userkt/usefor/usefor-usefor-04/Definitions.htm
http://www.mibsoftware.com/userkt/usefor/usefor-usefor-04/References.htm
   (which defers to RFC 2822
       http://mibsoftware.com/userkt/usefor/rfc2822/Identification_fields.htm
   )
http://www.mibsoftware.com/userkt/usefor/usefor-usefor-04/Message-ID.htm
http://www.mibsoftware.com/userkt/usefor/usefor-usefor-04/Obsolete_Headers.htm

2. The References header in usefor-usepro-03 is mentioned in only these
sections:

http://www.mibsoftware.com/userkt/usefor/usefor-usepro-03/Transitional_Arrangements.htm
http://www.mibsoftware.com/userkt/usefor/usefor-usepro-03/Duties_of_a_Followup_Agent.htm
http://www.mibsoftware.com/userkt/usefor/usefor-usepro-03/Duties_of_a_Reading_Agent.htm
http://www.mibsoftware.com/userkt/usefor/usefor-usepro-03/Duties_of_a_Relaying_Agent.htm

Note that none of those contain language to allow any articles except followups to
have the header.  The proposed text will standardize that articles which are not
followups can have the header, but nothing in the proposed text or usefor or usepro
limits those articles, nor indicates how they should be treated.

BTW, CFWS in References is a silly unnecessary complication.  (CFWS in any structured header
is a bad idea,IMNSHO. - I implement article processing software.)

usepro Transitional Arrangements even admits comments in References will break existing software.

So, why are comments going to be allowed in the References header?

Is the idea to make References irrelevant and unreliable, and then we don't
care that it cannot be parsed by legacy software?

---

From: Russ Allbery <[email-masked]>
Date: Thu, 02 Jun 2005 23:00:42 -0700

> Note that none of those contain language to allow any articles except
> followups to have the header.  The proposed text will standardize that
> articles which are not followups can have the header, but nothing in the
> proposed text or usefor or usepro limits those articles, nor indicates
> how they should be treated.

I think I'm with Harald on this -- while I agree that USEPRO should say
something about how References should be used in that context, USEFOR
shouldn't, and it's USEFOR that we're trying to finish.

Now, it's a separate (and good) question as to whether publishing USEFOR
without USEPRO should even be considered, given the clarifications like
this that USEPRO would carry, but considering USEFOR in isolation, with
its strictly limited scope, I don't think we should say anything more than
what Harald proposed about References.

Well, maybe a note saying that all References should be treated the same
regardless of their origins, to cover the last part of your paragraph
above, but I'd think that would be fairly obvious.

> BTW, CFWS in References is a silly unnecessary complication.

I agree, but not diverging from mail is even more important to me than
having to deal with that.

> So, why are comments going to be allowed in the References header?

Because mail allows them.  At least that's what I recall.

I think mail was wrong to ever allow them in the first place, but it's
several decades too late to make that argument, alas.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Fri, 03 Jun 2005 02:50:48 -0400

> I have deliberately not discussed USEPRO. I believe that we can do one 

You cannot change a definition in USEFOR and ignore that it requires
a change in USEPRO.  Fragility due to dependency chains is a well-studied
effect.

These are companion documents split only because they were too verbose.
I do not believe they can be edited, issued, or discussed independently.

I do not think it is going to be a "win" to push out USEFOR, and then
see how in the world we can write a USEPRO that requires no changes to
USEFOR.

I don't think it is efficient for the group to consider this problem once
now for USEFOR and then once again later for USEPRO.

> interpret ANY header. Are you suggesting to change this?

I wrote "how to interpret," but I did not mean "how to display."

USEFOR is supposed to be a syntax and semantics document.

More below....

> currently open, and should not be closed - people are currently using 
> the References: header in messages that can only be called "followup" by 
> a rather stretched definition of that term, and should not be prevented 
> from doing so.

So, instead of making an attempt at specifying the conditions that
it is acceptable, we have to allow it without limitation?

That's not current practice.  It is obvious that more than limited
use outside of followups would significantly degrade existing practice,
and that means it is outside the charter.

One of us has a very wrong idea about how useful threading (using
References) is.

> 
> But changing USEFOR to contain this level of information would change 
> the split between USEFOR and USEPRO, and I don't see a good reason for 
> that.

I think that's because you confuse semantics and implementation.

This isn't a hard concept for anyone familiar with writing software
specifications that separate requirements, design, and implementation.

Here is a specification of "semantics" (my proposed References definition
for USEFOR, with some RFC2822 text, and a correct trimming algorithm,
unlike the ambiguity currently in USEPRO.)

    A References header indicates the previously posted context in which
    the article was created.

    The "References:" field will contain the contents of the parent's
    "References:" field (if any) followed by the contents of the parent's
    "Message-ID:" field.  To prevent excessively long lines, message
    identifiers from the parent's References field can be omitted, provided
    that the first and last one or more in that field are retained.

Here is a specification of "implementation" (text from USEPRO-03)

    If the precursor did not have a References header (F-3.2.1), the
    content of the followup's References header MUST be inherited from
    that of the Message-ID header of the precursor. A followup to an
    article which already had a References header MUST have a
    References header comprising the precursor's References header
    (subject to trimming as described below) followed by CFWS and the
    Message-ID header content of the precursor.

    If the resulting References header is excessively long, it MAY be
    trimmed, but the first and the last two message identifiers MUST
    NOT be removed.

By specifying semantics properly, it is clear how to implement
a followup agent, a reading agent, and how to post multi-parts
and FAQs using the References header.

Semantics based is clearer, complete, and shorter already, but the USEPRO
text I showed here is not complete.  Since that is the section on Followup Agent,
there must be some section added for how the references get constructed
in a non-followup.  And then there must be a section about how to
display them.  What advantage is there to that?

I think the current drafts are all terribly written because of this.
They are too long.  They are easy to use for one purpose, but not
another.

---

From: Bruce Lilly <[email-masked]>
Date: Fri, 3 Jun 2005 10:56:30 -0400

On Fri June 3 2005 02:50, Forrest J. Cavalier III wrote:
> a change in USEPRO.

"one thing at a time" != "ignore".

> > a rather stretched definition of that term, and should not be prevented 

This was discussed at length when the change was first made without WG
consensus or prior discussion; it was going on a year ago:
http://www.imc.org/ietf-usefor/2004/Jun/0130.html
That the same argument over the same unrequested change which is 
inconsistent with RFCs 850 and 1036 is still going on today is an
indication that the WG is dysfunctional.

The simple fact is that RFCs 850, 1036 and drafts prior to that change
used "followup" as shorthand for "an article containing a References
field".  And as noted long ago in the ensuing discussion, that shorthand
is objective; it does not require guesses regarding the intent of a
human user, it does not require knowing how the article was prepared or
submitted, and it does not require examination of large numbers of
articles -- it concisely describes a fact about a particular article
which can be objectively assessed by examining that article in isolation.

> and that means it is outside the charter.

The current (RFC 1036) definition of "followup" includes current --
indeed long-standing -- practices such as multi-part FAQs and other
multi-part articles.
 
>     "Message-ID:" field.

If that's supposed to be a shining example of technical writing, we're in
deep trouble:
o terminology is inconsistent. Some of the terminology is OK; other uses
  are inconsistent with definitions in normative references (esp. RFC 2822)
  and in standard glossaries such as FYI 18.
o "previously posted context" is just plain wrong; it does not account for
  multi-part FAQs which may be prepared together (including References fields)
  and posted (nearly) simultaneously.  Likewise for other fragmented messages.
o "the contents of" is at best nebulous; at worst can lead to interoperability
  issues:
    References: <[email-masked]> Message-ID: <[email-masked]>
  is "the contents of" a References field followed by "the contents of" a
  Message-ID field.  We have suitable terminology, viz. "field body", and
  more precisely "msg-ids".

>     To prevent excessively long lines, message 

o "excessively long lines" is a non-issue due to the ability to fold
  physical lines wherever linear whitespace is allowed, as has been the
  case with RFC 822 message format and its predecessors even before that
  format was adopted by RFC 850.
o there is of course no limit to the length of a logical (i.e. logically
  unfolded) field.  Never has been.
 
>     If the precursor did not have a References header (F-3.2.1), the

o where is your definition of "precursor", precisely how does it differ
  from your use above of "parent", and does it account for use with multi-
  part FAQs and fragmented message parts (RFC 2046 section 5.2.2.2)?

>     content of the followup's References header MUST be inherited from

o why "MUST" -- what damage or interoperability problems (see BCP 14)
  are involved? (N.B. "some user doesn't like it" isn't an
  interoperability issue in the same sense as "limiting retransmissions")
  In particular, if some article has a clearly broken field where either
  the first or last identifier is malformed, why "MUST" that error be
  propagated? 
o see above re. "content of"
o where is your definition of "inherited"?  What precisely are the rules
  for ``inheritance'' -- may comments be elided?  Physical lines refolded?
  Etc.

>     References header comprising the precursor's References header

o you are presumably trying to define "followup" but have not yet done
  so.  The current definition means "article with a References field", so
  the text above is at least partially redundant (of course a followup
  must (N.B. lower case, indicating a logical relationship per section 2.14
  of ftp://ftp.rfc-editor.org/in-notes/rfc-editor/instructions2authors.txt)
  have a References field).

>     Message-ID header content of the precursor.

o see above re "content of"
 
>     If the resulting References header is excessively long, it MAY be

o try to define "excessively long".  Then try to achieve consensus for
  your definition.  This WG long ago abandoned that exercise in
  futility.

>     trimmed, but the first and the last two message identifiers MUST

o see above re. BCP 14 and use of imperatives.
 
> and FAQs using the References header.

The text you have supplied is certainly not clear.  It doesn't explicitly
address multi-part articles (FAQs or otherwise) directly, and the
"previously posted" part implies that it may not be used for some types
of multi-part articles.  It is hopelessly vague about "excessively long"
and "content of" and does not address reading agents at all.

> Semantics based is clearer, complete,

In which alternate reality is that either clear or complete, much less
both?

[...]
> in a non-followup.

By definition, a followup contains a References field and an article with
no References field is not a followup.  "how the references get constructed
in a non-followup" is a non sequitur.  Unless of course you are trying to
redefine "followup" to mean something other than what it has meant since
RFC 850, in which case there are many problems.  If it hurts, don't do it.

Complete text regarding References can be a short as:
  Refer to section 3.6.4 of [RFC2822] and section 5.2.2.2 of [RFC2046].

> And then there must be a section about how to 
> display them.

Why "must be"?  We (the IETF) do not usually specify display aspects of
implementation (when we specify implementation detail at all, as opposed
to specifying format and function); exceptions where display is specified
are typically limited to internationalization and/or localization issues
which do not apply to the References field or to display ordering of
articles or to indications of relationships.

> They are too long.  They are easy to use for one purpose, but not

You may be right regarding criticism of current drafts, but frankly your
suggested text (which past history shows would either be ignored or
twisted beyond recognition anyway) hardly seems clear or concise, nor
is delving into implementation choices that do not affect
interoperability (as distinct from product features) appropriate for
Standards Track documents.

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Fri, 03 Jun 2005 11:40:17 -0400

>   and in standard glossaries such as FYI 18.

Yes we are in deep trouble, especially since I copied most of what you objected
to verbatim from RFC2822.

Take a look...

    http://www.mibsoftware.com/userkt/usefor/rfc2822/Identification_fields.htm

If we can't be consistent with RFC2822, then all is lost indeed.

I do take the blame/responsibility for the wording of the "previously posted context"
definition.

The trimming algorithm came from the current USEPRO draft, but I reworded it.

Thanks for the criticism, shall I make an attempt to improve it, and should
I submit some errata for RFC2822 based on your comments?

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Fri, 03 Jun 2005 11:45:24 -0400

Private email.

Did you realize that most of your criticism actually applied
to text that appears in the current USEPRO-03?  It isn't
mine.
    http://www.mibsoftware.com/userkt/usefor/usefor-usepro-03/Duties_of_a_Followup_Agent.htm

I agree with the criticisms you have.

The trouble is that the ENTIRE USEPRO is like that.

If people actually read the drafts, as you were just prompted
to, there would be as much criticism for every section.

---

From: Bruce Lilly <[email-masked]>
Date: Fri, 3 Jun 2005 12:09:36 -0400

>     http://www.mibsoftware.com/userkt/usefor/rfc2822/Identification_fields.htm

I see:
  "Message-ID:" field
  "In-Reply-To:" and "References:" fields
  "Message-ID:" field
  "References:" and "In-Reply-To:" field
  "Message-ID:" field
  header fields
  trace fields
  resent fields
etc., and the definition (consistent with FYI 18):
   A message consists of header fields (collectively called "the header
   of the message") followed, optionally, by a body.  The header is a
   sequence of lines of characters with special syntax as defined in
   this standard. The body is simply a sequence of characters that
   follows the header and is separated from the header by an empty line
   (i.e., a line with nothing preceding the CRLF).

> If we can't be consistent with RFC2822, then all is lost indeed.

Agreed.
 
> Thanks for the criticism, shall I make an attempt to improve it,

Your choice.

> and should 

I see no error in the relevant sections of RFC 2822.

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 3 Jun 2005 15:07:45 GMT

>persuaded to start using the good & free news://fb1.euro.net

Which is a perfect example of how Usenet governs itself. If someone steps
out of line, there is always someone in each newsgroup to bring him back
into the fold.

But sadly, that mechanism does not work for email. Which is why
innovations spread rather rapidly through Usenet (though not necessarily
all useful ones, which is why we have this WG to tidy up the mess).
Whereas in Email the standards people can beaver away for years and still
people do not adopt the new things :-( .

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 3 Jun 2005 15:01:37 GMT

>   Note: Other articles may also use the "References" header.

Yes, that says the right things, but I think the text can be tidied up a
bit. How about:

   A "followup" is an article containing a response to the contents of an
   earlier article, its "precursor". Every followup includes a References
   header identifying that precursor (but note that non-followup articles
   may also use a References header).

And note that either text is consistent with the currently proposed
introduction to the References header:

3.2.1  References

|  The References header is used to refer to earlier articles (e.g.
|  followups refer to their precursors), in order to facilitate the
|  display or retrieval, by reading and other agents, of threads of
|  related articles. It is the same as that specified in Section 3.6.4
|  of [RFC2822] with the added restrictions detailed in Section 2.2 and
|  those listed below:

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Fri, 03 Jun 2005 12:19:17 -0400

> Private email.

Ah, I guess not private.  No big deal.

As you can see I thougth private would avoid extra inconvenience
to Bruce and the list when pointing out that some of the
harsh criticism Bruce had was actually against text
posted from the current drafts.

I think it is clear he did not realize it when he
wrote.  Actually, I think that had an extremely valuable
result....a fresh criticism of the wording in the
current USEPRO draft.

I didn't need to send that publicly, unless other people
also didn't realize where the "implementation" text
from my posting came from.

I guess I assumed that everyone participating here can
recognize text when it comes from RFC2822 and our
drafts.  Wrong assumption.

I admit to not knowing the drafts as well as I could,
but I try to be familiar with the parts as they are
discussed before I comment.  I split the documents into
sections on mibsoftware.com to facilite that.

I have made my opinions on the general state of
the drafts known to every chair over the years, and
I have started being more public with those comments
too.  So that part of my intended private email is
not a revelation.

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Fri, 03 Jun 2005 12:25:38 -0400

> I see no error in the relevant sections of RFC 2822.

This is in RFC2822, which I copied...

    The "References:" field will contain the contents of the parent's
    "References:" field (if any) followed by the contents of the parent's
    "Message-ID:" field (if any).

and one of your criticisms was that "contents of" was nebulous, and
you provided an example.

Are you saying the phrase "contents of" is not nebulous in RFC2822, but
would be nebulous in USEFOR?  Can you explain?

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 03 Jun 2005 18:50:01 +0200

Charles Lindsey wrote:

> Yes, that says the right things

+1

> How about:

Also fine, maybe better.

 [3.2.x propossal]
>| in order to facilitate the display or retrieval, by reading
>| and other agents, of threads of related articles.

It's not only for "threads of", it can be anything the poster
wants as a "relation" for "display or rertrieval", it can be
also a "relation" unknown for most readers (e.g. follwup in a
different TLH, or article posted as reply to a mail in a list)

                          Bye, Frank

---

From: Henry Spencer <[email-masked]>
Date: Fri, 3 Jun 2005 13:03:47 -0400 (EDT)

>    may also use a References header).

Looks good to me.

                                                          Henry Spencer
                                                       [email-masked]

---

From: "Ruud H.G. van Tol" <[email-masked]>
Date: Fri, 3 Jun 2005 19:38:41 +0200

Forrest J. Cavalier III:
> Bruce Lilly:

>     "References:" field (if any) followed by the contents of the
>     parent's "Message-ID:" field (if any).
> 
> Are you saying the phrase "contents of" is not nebulous in RFC2822,
> but would be nebulous in USEFOR?  Can you explain?

The contents of the parent's "Message-ID:"-field will sometimes 
contain more than just a <msg-id>, like trailing whitespace, or 
a comment, or worse.

-- 
Grtz, Ruud

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Fri, 03 Jun 2005 14:22:06 -0400

> contain more than just a <msg-id>, like trailing whitespace, or 

The draft states that comments are forbidden in a Message-ID.

You can read the draft here:

But since the draft allows CFWS in References, it apparently isn't
a concern that comments appear there.  (I disagree.)

I don't know what you mean by "worse."

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 03 Jun 2005 20:52:10 +0200

> I don't know what you mean by "worse."

Take this.

---

From: Bruce Lilly <[email-masked]>
Date: Fri, 3 Jun 2005 15:02:00 -0400

> would be nebulous in USEFOR?  Can you explain?

It's indeed nebulous, but falls short of an error.  It's unfortunate that
it lacks clarity, but this WG isn't tasked with rewriting 2822.  It should
suffice to reference 2822 and hope that the text will be clarified in its
successor.  I don't see much point in copying the text verbatim into our
documents.  At most, we might have a note clarifying the matter.

---

From: Eivind Tagseth <[email-masked]>
Date: Fri, 3 Jun 2005 22:06:05 +0200

> |  followups refer to their precursors), in order to facilitate the

More confusion.  You now say that the references header refers to
"earlier" articles, while an important point of using this complicated
followup/references definitions is to allow for multi-part FAQs to
be posted "at the same time".  "earlier" seems to contradict this.

I'll try this again:

    A "followup" is an article that follows up on the contents of another
    article, its "precursor".  Every followup includes a References header
    identifying that precursor.

    The References header is used to identify the parentage of the article
    by identifying its precursors.

Based on dictionary lookups, I cannot see why multi-part FAQs aren't
covered by this.

www.wordsmyth.net:

Parentage:
 Definition	1.	descent from parents and ancestors; lineage.
 Definition	2.	the condition or relationship of a parent; parenthood.

Follow-up:
 Definition	1.	the act or process of following up, esp. with something that may increase the effectiveness of an earlier action.
 Definition	2.	the means, such as a phone call or letter, used to follow up

merriam-webster:

Follow up:
 1 : to follow with something similar, related, or supplementary <following up his convictions with action -- G. P. Merrill>
 2 : to maintain contact with (a person) so as to monitor the effects of earlier activities or treatments
 3 : to pursue in an effort to take further action <the police follow up leads>

Precursor:
 1 a : one that precedes and indicates the approach of another b : PREDECESSOR
 2 : a substance, cell, or cellular component from which another substance, cell, or cellular component is formed

Eivind

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 6 Jun 2005 16:06:53 GMT

>be posted "at the same time".  "earlier" seems to contradict this.

OK s/earlier/previous/

Clearly, References headers are only appropriate when there as a sequence
of articles with some ordering. "Earlier" implies an ordering in time
(which is true for normal followups). "Previous" is regularly used to
imply an ordering in other senses, e.g. of items in a list.

>    identifying that precursor.

No, I think the consensus of this group is to keep "followup" (and its
converse "precursor") for the genuine reply/response situation, and
Harald's text (or my slight variation of it) expresses that.

The word "parent" may be useful, and indeed I have proposed to use it in
USEPRO.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 6 Jun 2005 15:57:15 GMT

>contain more than just a <msg-id>, like trailing whitespace, or 

Yes, we know that the definition of "content" in USEFOR is not consistent
with the way it is used in USEPRO. That was pointed out on this list
months ago, and the document editors are discussing ways to fix it.

May I suggest that you all stick to the important technical issues under
discussion, and leave it to the document editors to sort out matters of
consistency in the texts.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 6 Jun 2005 15:48:20 GMT

>now for USEFOR and then once again later for USEPRO.

I agree entirely with the above. There is no way that we can publish
USEFOR before we publish USEPRO. Maybe we can leave USEAGE until the other
two are out of the way.

>I wrote "how to interpret," but I did not mean "how to display."

Indeed. EVERY header in USEFOR needs a brief intrpductory section explaining
"what this header is good for". I think References is the only header
lacking that in USEFOR-04, but I have already proposed text to fill that
gap.

Often, that introduction is sufficient for the "semantics" of that header,
especially if the protocols in USEPRO tells you exactly what to do when you
encounter it.

But if agents are going to use it for a variety of purposes, then more may
be needed, since USEPRO is not the place to go into detail of how to
display things (though USEAGE might be).

>it is acceptable, we have to allow it without limitation?

No. We describe the intended "purposes" of the header. In this case to
facilitate "threading", "retrieval" (of parents) and "killfiling" (for
killing whole subthreads) (my proposed introductory text mentions the
first two of those, which I think is enough). If you create a References
header, then you should be aware it is likely to be used for those
purposes, and if those are not what you intend, then you shouldn't be
creating it.

>    A References header indicates the previously posted context in which

That is an "introductory text", but it does not really tell you anything
interesting. The one I proposed goes further.

>    that the first and last one or more in that field are retained.

But that is just telling you how to construct it, and since that is going
to be described in USEPRO, it does not need to be mentioned here
(especially the bit about pruning).

You might argue that if (from USEPRO) you know how to construct it, then
you have sufficient information to work out what it "means". I don't
entirely buy that, and in an earlier draft of my "suggestions" I had some
"semantics" text telling what parent/child relationships you could deduce
from it (effectively, you can deduce a tree or, more generally, a DAG).

But I took that text out because people did not think it necessary. I
could be persuaded to put it back.

>    If the precursor did not have a References header (F-3.2.1), ....

Yes, but that text is way out of date. I have already posted my suggested
text, which tells you how to contruct a References header in the general
case given a "parent" article, and then invokes that for the followup case
in particular. It also ties up the pruning bit more precisely, because
Frank asked for that.

>text I showed here is not complete.

That is because you have ignored the more recent texts posted to this
list.

Come to think of it, it is probably time to post a fresh version of those
texts, taking account of discussions these last few days.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 6 Jun 2005 18:35:31 GMT

>different TLH, or article posted as reply to a mail in a list)

But we don't formally define "threads". We are agreed that the References
heaader should only be used in situations where threading the articles by
reading agents would make sense. So using that word, in what is only meant
as an introductory remark to the References header section, would surely
reinforce that intention.

What does "thread" actually mean in the "Real World" (TM). IN this
context, surely it means a "linear list", which is near enough to its
meaning as used in so called "threaded newsreaders" to be OK in this sort
of context.

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 07 Jun 2005 00:29:33 +0200

Charles Lindsey wrote:
 
> We are agreed that the References heaader should only be used
> in situations where threading the articles by reading agents
> would make sense.

This would be an agreement over my dead body, but I don't feel
like I'm dead.  We "agreed" to describe the followup procedure
as is in USEPRO.  The mere word "thread" is taboo for USEFOR.

> So using that word, in what is only meant as an introductory
> remark to the References header section, would surely

It would surely kill USEFOR for good, yes.  No "threads", no
nonsense, no aclycic graphs.  A msg-id-list is simply a list
of msg-id-s, ready.  Harald's wording (or your version of it)
was fine, any additional comma in 3.2.2 is asking for trouble.

                         Bye, Frank

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 07 Jun 2005 01:00:51 +0200

Charles Lindsey wrote:
 
>> I don't think it is efficient for the group to consider this
>> problem once now for USEFOR and then once again later for
>> USEPRO.
 
> I agree entirely with the above.

And I disagree entirely.  The data structure (USEFOR) is clear,
the algorithms (USEPRO) will fit.

> There is no way that we can publish USEFOR before we publish
> USEPRO.

There is no way that this WG will still exist in ten weeks
without an approved USEFOR RfC submitted to the in-queue.  Or
at the very minimum in "last call".

> EVERY header in USEFOR needs a brief intrpductory section
> explaining "what this header is good for".

Not if it's essentially the same as in 2822.  And saying that
a msg-id-list is a list of msg-ids is a waste of time.  It's
like saying that id-domain is a domain.  The only reason why
you are forced to say the latter is because you insist on the
vague and bad id-right idea found in 2822.

>> A References header indicates the previously posted context
>> in which the article was created.

That's not always and not necessarily the case.

> effectively, you can deduce a tree or, more generally, a
> DAG

Is that a "directed acyclic graph" ?  Whatever it is, this is
not always and not necessarily the case.  The msg-id-list is a
list of msg-ids.
 
> it is probably time to post a fresh version of those texts,
> taking account of discussions these last few days.

That would be the "internal last call" for USEFOR, good.  Bye.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 07 Jun 2005 09:27:32 +0200

New subject....

I was told earlier that the WG had a previous consensus that it would:
a) not change definitions from RFC 2822 (where References has CFWS)
b) allow CFWS wherever possible, with a "MUST accept/SHOULD NOT generate" 
clause for certain headers (including References).

If this is indeed the previously documented consensus, then Forrest's 
comment is out of order - a reopening of past issues.

Did I hear it wrong that this was a previous consensus?

                    Harald

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 07 Jun 2005 09:09:09 +0200

> meaning as used in so called "threaded newsreaders" to be OK in this sort

I have rarely seen an example of a linear thread.....

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Tue, 07 Jun 2005 07:26:06 -0400

Harald Tveit Alvestrand wrote:
> b) allow CFWS wherever possible, with a "MUST accept/SHOULD NOT 
> generate" clause for certain headers (including References).
> 

CFWS was changed to FWS for Xref and Supersedes between April 2005
(usefor-usefor-03) and May 2005 (usefor-usefor-04), so I thought any
"previously documented consensus" must not have been all that important.

If you are you asking me to withhold objections until Last Call,
I will.  We seem to finally have a real push to end this torture,
with either success or failure, in 10 weeks.  I'm all for that.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 07 Jun 2005 14:24:29 +0200

References is defined in 2822, while Xref and Supersedes are defined in 
USEFOR.
I'll open a ticket on whether comments in References should be listed as 
SHOULD NOT generate or MUST NOT generate - saying MUST NOT generate would, 
I think, make it equivalent with replacing CFWS with FWS.

               Harald

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Tue, 07 Jun 2005 08:56:49 -0400

Frank Ellermann wrote:

>>Forrest>A References header indicates the previously posted context
>>Forrest>in which the article was created.
> 
> 

I understand people wish to relax the requirement that an article
be "posted." I disagree.  But, do you have any other cases which are
outside the description?

Frank, it seems like you want to push USEFOR to be basically
ABNF only, with almost no semantics.

Why would such a terse document merit its own separate RFC?

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 7 Jun 2005 12:50:42 GMT

>was fine, any additional comma in 3.2.2 is asking for trouble.

No, the <msg-id-list> in the References header is not any random
collection of <msg-id>s. It is subject to constraints (e.g. that a
precursor does not come later in the list that any followup to it).

You can express this by referring to the rule for constructing it in
USEPOR (which is what my currently proposed text does), or you can express
it by writing some similar rules into USEFOR (which is what Forrest has
suggested), or you can express it by talking about parents coming earlier
than children (which is what my earlier suggestion did, though not using
the words 'parent' and 'child').

What you cannot do is to say nothing at all.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 7 Jun 2005 13:47:23 GMT

>indeed long-standing -- practices such as multi-part FAQs and other

No, the consensus view here is that RFC 1036 does NOT include multi-part
FAQS etc in that manner, which is why the issue needs to be clarified.

>  unfolded) field.  Never has been.

That's the theory, but it ain't the practice.

There was a long running thread on the ietf-822 list, which you may well
remember. The References header got longer and longer. Different agents
folded it in different ways (some even took a whole new line for each
msg-id, which was actually quite nice if you wanted to look through it for
some particular message). Moreover, some agents when replying took the old
References list, added the new item, and refolded the whole thing in a
completely different way. All quite permitted by RFC 2822, of course.

BUT there was at least one agent (I never discovered which) which unfolded
the whole line, added the new entry, and then did not refold, thus undoing
all the good done by the preceding agents. Naturally, the total length
eventually got beyoind 998. At which point some agent (not sure whether it
was the same culprit or the next one in line) truncated the header after
exactly 998 characters (usually in the middle of a <msg-id>) and passed it
on, to the great confusion of the next agent to process it.

I remember that, for some messages, I had to edit the text of the message
by hand before my browser would show it to me (or was it before I could
respond, I am not sure).

So long as such broken agents exist in the wild, the only safe procedure
is to prune the References so that it never exceeds 998, folded or not.

And preferably well before 998, because in practice a reading agent never
needs more than two or three msg-ids in order to generate a proper
threading.

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Tue, 07 Jun 2005 12:59:37 -0400

> FAQS etc in that manner, which is why the issue needs to be clarified.

RFC1036 does not define "follow-up".  Any "definition of followup" from
RFC1036 is only found "between the lines".  I think Bruce and Charles
are wrong to refer to it as a definition.

Here are links to all the RFC1036 sections which use the term:

   http://www.mibsoftware.com/userkt/usefor/rfc1036/References.htm

   http://www.mibsoftware.com/userkt/usefor/rfc1036/Newsgroups.htm

   http://www.mibsoftware.com/userkt/usefor/rfc1036/Subject.htm

   http://www.mibsoftware.com/userkt/usefor/rfc1036/Summary.htm

I find no wording above which definitely excludes or includes
multi-parts.  So appeals to RFC1036 as an authority on this are
rhetoric, not reason.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 7 Jun 2005 13:03:07 GMT

>Not if it's essentially the same as in 2822.

Agreed. There is no need to write introductory text for the From header,
because we have nothing to add to RFC 2822.

But that is not the case for the References header.

>Is that a "directed acyclic graph" ?

Yes.

>  Whatever it is, this is
>not always and not necessarily the case.

Yes it is. If you are not trying to describe at least a DAG (of which
trees and linear lists are a particular case), then you have no business
putting a References header in your article. That is what Forrest is
worried about.

Note that I don't want to include the term "DAG" in any published text.
And note also that my current USEPRO text is only telling you how to
construct a tree (though it does point out, as does RFC 2822, that it has
deliberately limited itself to that).

Nevertheless, you cannot prevent non-trees. Every time someone tries to
followup to two precursors, and manually attempts to fix up the References
to reflect that (and people DO try to do that from time to time), then
that someone has constructed a DAG. We have to live with that, because we
cannot prevent it.

But I am not aware of anyone (trolls apart) who has ever tried to
construct, for example, a References header with a loop in it.

---

From: Russ Allbery <[email-masked]>
Date: Tue, 07 Jun 2005 10:50:36 -0700

> clause for certain headers (including References).

I don't think so.

The consensus, as I understood it, was that we would avoid unnecessary
divergences from RFC 2822 while recognizing that some changes to converge
with RFC 2822 would break existing news software and therefore couldn't be
allowed.  In particular, we HAVE to diverge on Message-ID because the RFC
2822 definition is too lax and will break NNTP.

I would not have agreed with the consensus as stated above, for the
Message-ID issue if for no other reason.

References is a borderline case.  Adding CFWS to References *will* break
existing news software, in the sense that the comments will be interpreted
as message IDs, but the result of the breakage isn't particularly
dramatic.  The open question is whether that degree of breakage warrants
divergence or not.  I don't think we ever really reached a conclusion
everyone was satisfied with, although we've gone around on the topic
several times.

---

From: Bruce Lilly <[email-masked]>
Date: Tue, 7 Jun 2005 14:12:42 -0400

> Here are links to all the RFC1036 sections which use the term:

Thanks.

The bottom line is that the net result is what our drafts said
for quite some time and which enjoyed consensus, viz.

   A followup MUST have a References-header, and an article that is not
   a followup MUST NOT have a References-header.

Modulo capitalization of "NOT", that text also appeared in Henry's
news.txt.Z.

That text provided a convenient shorthand ("followup" == article with a
References field) and is an objective fact about a given article which
can be determined directly from that article without the need to examine
an arbitrarily large collection of other articles, without having to 
break out a Ouija board to divine whether or not the "subject" changed or
what the authors' or poster's intentions were, etc.

Until it was arbitrarily changed w/o prior WG discussion or consensus.

> multi-parts.

We have been directed to base our work on RFC 2822 and MIME, and
MIME (specifically RFC 2046 section 5.2.2.2) permits use of a
References field to reference parts of a fragmented message. Such
a message with a References field is a "followup" per the consensus
text quoted above.  RFC 2822 and its predecessors defines use of
the References field for "message identifiers of other messages"
without restriction, and that applies to multi-part FAQs.

I move that we restore the text quoted above which was present in
our drafts prior to the arbitrary change, and which enjoyed WG
consensus and was subjected to internal last calls.  Anybody care
to second the motion?

---

From: Bruce Lilly <[email-masked]>
Date: Tue, 7 Jun 2005 14:19:08 -0400

> FAQS etc in that manner, which is why the issue needs to be clarified.

It was eminently clear in earlier drafts which passed internal last call;
see separate message.

> on, to the great confusion of the next agent to process it.

For the purpose of the USEFOR (syntax & semantics) document, it suffices to
specify the syntax and semantics (obviously?).  For the purpose of the
USEPRO (protocol) document, one could add an implementation note such as:

  When modifying a References field, care must be taken to ensure that the
  result complies with the syntax specification for the field, specifically
  that line length limits are observed and that no msg-id is truncated or
  subjected to internal line folding.

---

From: Russ Allbery <[email-masked]>
Date: Tue, 07 Jun 2005 11:23:40 -0700

Bruce Lilly <[email-masked]> writes:

> break out a Ouija board to divine whether or not the "subject" changed
> or what the authors' or poster's intentions were, etc.

> Until it was arbitrarily changed w/o prior WG discussion or consensus.

Whether there was consensus before it was changed or not, we just went
through this and there's a fair bit of consensus to leave it the way that
it is now.  That's simply not how the word "followup" is used in software
and in discussion of Usenet, for better or for worse.

> We have been directed to base our work on RFC 2822 and MIME, and MIME
> (specifically RFC 2046 section 5.2.2.2) permits use of a References
> field to reference parts of a fragmented message. Such a message with a
> References field is a "followup" per the consensus text quoted above.

I'm not sure what you're trying to demonstrate by inserting the word
"consensus" into this sentence.  I would instead argue that RFC 1036 did
not anticipate uses such as MIME when that text was drafted.

> RFC 2822 and its predecessors defines use of the References field for
> "message identifiers of other messages" without restriction, and that
> applies to multi-part FAQs.

I agree with you on this, as previously mentioned.

> I move that we restore the text quoted above which was present in our
> drafts prior to the arbitrary change, and which enjoyed WG consensus and
> was subjected to internal last calls.  Anybody care to second the
> motion?

I think this is a really bad argument to use to support the change.

---

From: "Ruud H.G. van Tol" <[email-masked]>
Date: Tue, 7 Jun 2005 20:23:03 +0200

> construct, for example, a References header with a loop in it.

A have seen the shortest variant a few times: self-referencing.

The last time even because somebody considered it useful.
See news:[email-masked]

http://groups.google.co.uk/groups?selm=[email-masked]&fwc=1
http://groups.google.co.uk/groups?selm=[email-masked]&fwc=1
http://groups.google.co.uk/groups?selm=[email-masked]&fwc=1

The URL with 465 will not return a message, I guess that google filters
such messages.

-- 
Grtz, Ruud

(Bcc: that somebody)

---

From: Bruce Lilly <[email-masked]>
Date: Tue, 7 Jun 2005 14:35:53 -0400

> everyone was satisfied with, although we've gone around on the topic

A good example of where Mark Crispin's suggestion of an implementation
note would be appropriate, e.g.:

  Although comments have always been permitted in the References field
  per RFC 822 syntax as adopted by RFCs 850 and 1036, some implementations
  are known to attempt to interpret comments as msg-ids.  Consequently,
  implementations concerned about such matters should avoid comments
  when generating References fields. Implementers concerned about
  resources used for handling msg-ids should ensure that comments are
  not misinterpreted as msg-ids.

---

From: Bruce Lilly <[email-masked]>
Date: Tue, 7 Jun 2005 15:11:18 -0400

> not anticipate uses such as MIME when that text was drafted.

Apologies if it was unclear; let me clarify.  The "consensus text" referred
to is what appeared in draft-ietf-usefor-article versions 06 through 13 at
least (possibly others), viz.:

That text was subject to WG review including internal last calls, and
including after a WG Chair clarified that our work was to be based on
2822 + MIME.  Clearly that text survived at least eight iterations of
the draft, including last calls, and while we were considering MIME.

It also happens to be the same as in Henry's news.txt document modulo
capitalization of "NOT", and is a reasonable interpretation of 1036 (which
does have issues with clarity).  I am not claiming that the consensus
applied to those other documents or to text other than that quoted above.
The text does not appear verbatim as a unit in 1036; that is not the claim.

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 08 Jun 2005 06:26:24 +0200

Charles Lindsey wrote:

  [DAG]

> Yes.

Then it's too general, if you're lucky you can deduce a rooted
tree, and then the References are the way from the root to the
article.

In a DAG the direction "from root to leave" isn't clear, it's
just a tree with arbitrary directions attached to the edges...

Wait, I see, you want multiple trees, all the msg-ids are by
definition direct or indirect "precursors" of the article in
question, and for two msg-ids m and in this list with m < n
you have either m is an (in)direct "precursor" of n, or it is
unrelated.  In any case all listed msg-ids are "precursors" of
the article in question.

Defining "precursor" as "existed before (pray)".  Fine, that
makes sense, but we are in trouble with the "pray".  And it's
not guaranteed, if I pick msg-ids (x, y, z) as my "references",
and x was a reply or followup of z, then it's not your DAG.

And while I do such stupid things no text in the world forces
me to sort these msg-ids as ((y), (z, x)) or ((z, x), y) or
(z, y, x) only because you like to see "z before x".  There's
a serious chance that I don't know this even if I'm willing to
follow your decree.  Maybe I found x, y, z individually.

>> Whatever it is, this is not always and not necessarily the
>> case.

> Yes it is.

No, it's not.  It's a msg-id-list.  Only if you are lucky it's
a way from "the" root to the article in question.  Often you
don't know what it is (= references from an invisible context).

> If you are not trying to describe at least a DAG (of which
> trees and linear lists are a particular case), then you have
> no business putting a References header in your article.
> That is what Forrest is worried about.

My UA doesn't care what Forrester is worried about, because his
definition is not correct.  My UA _normally_ grabs whatever it
sees in the References, a msg-id-list, adds the msg-id of the
parent article, and uses this as references in the followup or
reply.

It has _no_ idea what the old references were.  And that's only
the normal case.  Under less normal conditions I could put some
"related Message-IDs" manually in the references, for "retrieval"
as you said.  It's strictly my business.

And I'd bet that no UA in the world has a crystal ball to check
these references against theoretical trees, they just do what
my UA does:  grab it, add parent Message-ID, ready.  Exactly
this is the correct behaviour.  If you want to say more about
it put it in 1855bis / USEAGE.

> my current USEPRO text is only telling you how to construct a
> tree

It doesn't, it's only a theoretical tree by induction, if the
_complete_ references were constructed this way in each step.
That is not guaranteed, and it's generally neither possible
nor desirable to check it.

What you have in USEPRO is the one correct step to create a
followup given a parent msg-id and its old msg-id-list.  This
old msg-id-list can be anything, it can be some "See-Also".

> I am not aware of anyone (trolls apart) who has ever tried
> to construct, for example, a References header with a loop
> in it.

I tried this here, References containing my own Message-ID:

<http://article.gmane.org/gmane.ietf.usenet.format/27603/raw>
<news://news.gmane.org/[email-masked]>

It's not that I'd "recommend" this practice, it's a bad idea,
but it's still something UAs have to deal with.  Bye, Frank

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 08 Jun 2005 06:49:19 +0200

> ABNF only, with almost no semantics.

For a "msg-id-list" the semantics is "list of msg-ids", for
a more elaborated answer about Charles' DAG idea (which also
does not fly) see my parallel article.
 
> Why would such a terse document merit its own separate RFC?

There are some interesting and important differences from RfC
2822, one header field is irrelevant in news (In-Reply-To),
many header fields are only relevant in news, not mentioned
in 2822, above all "we" have the one and only msg-id syntax.

In two slightly different variants, but syntactically the same
msg-id, _very_ different from 2822.  There are at the moment
at least three drafts building on "our" fixed msg-id:

The 26th NNTP monster in last call, Charles' news-nntp-uri-00,
my news-nntp-uri-00.  They all say "NO-WS-CTL is not correct,
RfC 2822 got it wrong".
                           Bye, Frank

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 08 Jun 2005 07:32:20 +0200

> I have rarely seen an example of a linear thread.....

Some [FWS] and msg-id threads here were rather linear.
In the latter case there was still a sort of result -
or maybe two almost equivalent results.

The threads about references were in fact loops, they
started from scratch periodically.  No surprising new
arguments for years.
                     Bye, Frank

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 08 Jun 2005 07:53:41 +0200

> What you cannot do is to say nothing at all.

Actually RFC 2822 gets away with only giving a construction algorithm.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 08 Jun 2005 08:06:51 +0200

> construct, for example, a References header with a loop in it.

I have been told by someone familiar with News (I think it was by Arnt 
Gulbrandsen, the implementor of the Leafnode newsserver, but it's so many 
years ago that it could easily be someone else) that he's seen at least one 
example of a References: loop.

I believe this was a case of two messages having References referring to 
each other. Had "interesting" effects on some news reading software he was 
writing that expected the References chain to be a DAG.

                       Harald

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 08 Jun 2005 07:59:06 +0200

>    a followup MUST NOT have a References-header.

Bruce,

remember the article I posted on May 27?

> 1) All "followups" have a References: header.
>    All articles with References: header are "followups".
> - Eivind Tagseth: "Because I view... multi-part..special type of followup"
>
> - Henry Spencer: "Existing practice"
> - Richard Clayton: "captures my perception of useful current practice"
> - Seth Breidbarth: "current reality"
> - Russ Albery: "but would be content with 1)"
> - Frank Ellermann

(I also had 3 people arguing other positions).

Whatever the basis is for your claim of an earlier consensus, we don't seem 
to have consensus for your position now.

                     Harald

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 8 Jun 2005 14:00:20 GMT

>  subjected to internal line folding.

I agree that any mention of pruning is a matter for USEPRO rather than
USEFOR. What I currently propose to say, at Frank's request (see suggested
text posted earlier) is that you MAY prune (subject to keeping the first
and the last two), and that you MUST prune to keep it within 998
(including folded lines, because of the kind of breakage I described).

I think it goes without saying that whatever results must be syntactically
correct, and so there can be no folding within a msg-id.

There is some additional advice on pruning in USEAGE.

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 8 Jun 2005 12:11:53 GMT

>Actually RFC 2822 gets away with only giving a construction algorithm.

Yes, but Alexey pointed out a couple of weeks back that RFC 2822 is both a
protocol and a format document, with the implication that that
construction algorithm belonged in USEPRO (where it indeed is).

Which leaves the question whether it is sufficient, in USEFOR, to point
you at that text in USEPRO, or whether you also need some explicit
semantics text in USEFOR, most likely worded in terms of parents coming earlier
than children (though maybe not using the words 'parent' and 'child').

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 8 Jun 2005 12:05:12 GMT

>I have rarely seen an example of a linear thread.....

I think the use of the metaphor "thread" came about in the following way.

Consider a tree (or a list of a DAG or a graph) with a bead at each node.

Take a piece of cotton thread, and thread it through some/all of those
beads in some manner. That defines a linear ordering on the nodes in the
tree (DAG/graph/whatever).

A useful threading order for a tree of news articles is a depth first
traversal, and newsreaders that present articles in that order (and
sometimes with some indentation to make it even prettier) are hence known
as "threading newsreaders".

But the question immediately facing us is whether or not it is useful to
use "thread" informally in some text that serves to introduce the
References header in USEFOR.

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 8 Jun 2005 12:18:06 GMT

>remember the article I posted on May 27?

........

>to have consensus for your position now.

And may I also say that the text that Bruce has quoted from the earlier
drafts was written *before* someone came on this list to point out the use
of References in multi-part FAQs (a practice I had not been aware of, but
which seemed eminently sensible).

The *whole* of our discussions, proposed changes, etc. since then have
arisen as a direct result of what that someone (I cannot recall who it
was) said.

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 8 Jun 2005 13:52:02 GMT

>I think, make it equivalent with replacing CFWS with FWS.

I think you need to consider a more general issue.

There are various places where our draft mandates behaviour that is not in
accordance with current practice; in all such cases, it is necessary to
consider what transitional arrangements need to be put in place.

The issue here is where we are mandating that something MUST be accepted,
and where the transitional behaviour is of the form "but do not generate
it yet", for some value of "yet". The cases in this category are:

1. <comment>s in References.
2. Naked '.' in <phrase>s.
3. Folding of the Newsgroups header.

(There were also matters of <comment>s in some other headers, but these
are mostly gone now - there remain cases like 
   From: (rhubarb) Joe Doe (rhubarb) <[email-masked]> (Joe Doe)
where the first two <comment>s might confuse some existing newsreaders,
but I could perhaps live with that).

So what is an appropriate transitional arrangement for those three?

Our first reaction was to propose some cutoff date after which they could
be freely generated, but we were told that the IESG did not like Flag
Days.

OTOH, we did not like the idea of "do not generate until the next standard
comes out" (expressible as "MUST NOT generate") because #3, in particular,
really needed a more urgent fix than that (the Newsgroups header is the
only one of all the mail and news headers that cannot be folded, and
people are still astounded to be told that it still breaks existing
software - even INN has only fixed it in the last 12 months AFAIAA).

So the position we took was that people should start generating them when,
in their opinion, they would work well enough that the benefit would
outweigh the risk that articles would not be seen at some sites. That was
expressed by saying "SHOULD NOT generate", plus some verbiage such as:

   ....  Posting agents should be aware that such postings
   may be rejected by overly-critical old-style relaying agents. When a
   sufficient number of relaying agents are in conformance, posting
   agents SHOULD generate such whitespace in the form of <CRLF WSP> so
   as to keep the length of lines in the relevant headers (notably
   Newsgroups and Followup-To) to a reasonable length (such as 79
   characters, which is likely to be displayed satisfactorily by most
   current reading agents).  Before such critical mass occurs, injecting
   agents MAY reformat such headers by removing whitespace inserted by
   the posting agent, but relaying agents MUST NOT do so.

which is a bit wordy, but conveys the essential idea.

So we either say "MUST NOT generate" for them all, or
we say "SHOULD NOT generate" plus verbiage, or
we say that Newsgroups (for example) is urgent, and gets "SHOULD NOT",
whereas nobody is actually wanting to put <comment>s in References, so
"MUST NOT" will do there.

But it has also been pointed out (Harald will have seen an email from
Alexey on this) that even "SHOULD NOT" is too strong for Newsgroups
folding, since it gives no incentive to implement the "MUST accept".

So there is the gamut of possibilities. They need to be considered
together.

---

From: Russ Allbery <[email-masked]>
Date: Wed, 08 Jun 2005 12:11:38 -0700

Charles Lindsey <[email-masked]> writes:

> OTOH, we did not like the idea of "do not generate until the next
> standard comes out" (expressible as "MUST NOT generate") because #3, in
> particular, really needed a more urgent fix than that (the Newsgroups
> header is the only one of all the mail and news headers that cannot be
> folded, and people are still astounded to be told that it still breaks
> existing software - even INN has only fixed it in the last 12 months
> AFAIAA).

I think I've mentioned this before, but maybe not.  INN still does not
accept folded newsgroup headers, since that would mean accepting an
article that's just going to get rejected by the next news server and fail
silently.  Be conservative in what one generates, etc.

---

From: Bruce Lilly <[email-masked]>
Date: Wed, 8 Jun 2005 18:45:47 -0400

> No, it's not.  It's a msg-id-list.

With the (draft) provision(s) for eliding random ids, it is
just a list.

Without that provision (i.e. if all ids are required to be
retained), *AND* with sufficient external information
(gathered from all referenced articles, therefore requiring
access to all such articles), one could theoretically generate
a directed graph.  It might plausibly contain one or more
cycles.

Even with such extreme measures to generate a graph, the ids
comprising a References field body remain a list.

> > trees and linear lists are a particular case),

No. A tree is not necessarily directed (unrooted trees are not,
rooted trees are).  Lists are sets, and do not possess edges,
much less directed edges. See
http://www.csse.monash.edu.au/~lloyd/tildeAlgDS/Graph/

> as you said.  It's strictly my business.

Yes, "other messages" (RFC 2822), not "earlier messages" or
"previous messages".

---

From: Bruce Lilly <[email-masked]>
Date: Wed, 8 Jun 2005 18:51:23 -0400

> Actually RFC 2822 gets away with only giving a construction algorithm.

But N.B. that is given for a particular case:

   When creating a reply to a message,...

which is not the only case:

   ...and the message identifiers of other messages (for example,
   in the case of a reply to a message which was itself a reply).

(N.B. "for example"). [not arguing for anything other than "[RFC2822]",
just cautioning against reading too much into what 2822 says]

---

From: Bruce Lilly <[email-masked]>
Date: Wed, 8 Jun 2005 19:11:01 -0400

> remember the article I posted on May 27?

Yes, I remember; I suspect that there are more than 9 people on the
mailing list, and I know some (e.g. me) didn't respond.  Some of us
monkeys are tired of chasing the same weasel around the same mulberry
bush, year after year.  IIRC, the discussion after your message went
down the usual "thread by Subject" rathole.

If you like you can put me in the "1)" column with a note that there
is disagreement with the alternatives for the reasons mentioned in
recent discussion (those alternatives do not provide an objective
term, hence not a useful term).

---

From: Bruce Lilly <[email-masked]>
Date: Wed, 8 Jun 2005 23:37:09 -0400

> correct, and so there can be no folding within a msg-id.

The specific case you mentioned involved a transport agent that folded a
long line at 990 (N.B. not 998) octets, which happened to be in the middle
of a msg-id.  The suggested text above specifically covers that case.  If
you're not now concerned about that, one wonders why you brought it up...

---

From: Bruce Lilly <[email-masked]>
Date: Thu, 9 Jun 2005 00:36:36 -0400

On Wed June 8 2005 09:52, Charles Lindsey wrote:

> The cases in this category are:
> 
[...] 
> really needed a more urgent fix than that [...]

It is possible to go from Proposed to Draft in six months.

> in their opinion, they would work well enough [...].

I suspect that in a Standards Track document "do whatever you please"
isn't going to pass muster.  Less so, I suspect, than a flag day.
  
> So there is the gamut of possibilities. They need to be considered

No, there are other possibilities, including, but not limited to:
o provide a roadmap (X at Proposed, Y at Draft, Z at Full Standard).
  Harald can probably give some guidance as to how well that would be
  received by the IESG, but failing something like that, there is a
  good chance that each such change would require a reset to Proposed
o Implementation notes as suggested by Mark Crispin:
    http://www.imc.org/ietf-usefor/2003/Feb/0486.html
    http://www.imc.org/ietf-usefor/2003/Feb/0493.html
    http://www.imc.org/ietf-usefor/2003/Feb/0554.html
o Implementations are of course free to ignore the specification provided
  they don't claim to conform to it
o As already suggested in discussion, specifically w.r.t. dot in a
  phrase, "do nothing" (aside from "[RFC2822]") may be a reasonable
  option

Specifically regarding the References field and comments, what is the
scope of the issue?  Transport agents are unaffected.  Any
considerations appear to be confined to UAs, and as the field plays
no critical role in protocol (certainly not warranting BCP 14
imperatives), "[RFC2822]" (for implementations that wish to claim
conformance) may suffice [note that a) comments have always been
permitted per RFC 822 as adopted by 850/1036, b) many UAs are combined
mail/news UAs and already necessarily handle comments (unless they're
broken UAs), and c) message-generating UAs generally do not insert
comments in References fields, so the only likely source is a human
manually editing a field].  It doesn't seem worth spending much time
on -- not beyond an implementation note at most.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Thu, 09 Jun 2005 09:04:18 +0200

--On 8. juni 2005 19:11 -0400 Bruce Lilly <[email-masked]> wrote:

> bush, year after year.

Yep. That's why I'm trying to nail it to the tree. And make it stay nailed.

FWIW, the mailing list had 49 members a few weeks ago, and 20 people have 
so far responded to my call for participants.

> IIRC, the discussion after your message went

Thanks.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Thu, 09 Jun 2005 16:14:40 +0200

> "MUST NOT" will do there.

After this many years of this WG trying to provide *one* revision, I wonder 
about the likelyhood of there ever being another......

What's more likely to happen is that someone produces a News extension, 
which updates USEFOR saying "this MUST NOT generate is now changed to a 
MAY".
This is standard practice for protocol extensions.

> So there is the gamut of possibilities. They need to be considered

IMHO, and speaking for myself-as-participant only:

Realistically, either newsgroups folding is useful or it is not.
If it is not, there is no incentive to implement it no matter what the 
standard says.
If it is useful, there is incentive to implement it.
If it is very useful, someone might even be bothered to write an 1-page RFC 
saying "Newsgroup folding is now permitted".

But I very much doubt it.

We have better things to do than to encourage the implementation of useless 
features.

                         Harald

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 9 Jun 2005 16:39:08 GMT

>silently.  Be conservative in what one generates, etc.

My understanding was that INN was now accepting folded Newsgroups headers
when relaying.

I agree that it would be foolish, at this time, to accept them when
injecting.

---

From: Russ Allbery <[email-masked]>
Date: Thu, 09 Jun 2005 13:01:54 -0700

Charles Lindsey <[email-masked]> writes:
> Russ Allbery <[email-masked]> writes:

>> article that's just going to get rejected by the next news server and
>> fail silently.  Be conservative in what one generates, etc.

> My understanding was that INN was now accepting folded Newsgroups
> headers when relaying.

Ah, yes, it's done that since 2.4.

It doesn't accept colons, btw.

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 9 Jun 2005 17:04:27 GMT

>    http://www.imc.org/ietf-usefor/2003/Feb/0554.html

AIUI, Mark Crispin's idea is that you give guidance to implementors to the
effect that such and such behaviour, although permitted, will cause such
and such problems on the present network, and that implementors should
consider this carefully before deciding whether to provide it.

If that is correct, I would have no particular problem with it. But it is,
in essence, what the article-* drafts were trying to say (they were
certainly not saying "do whatever you please", any more than Mark was
saying that).

>on -- not beyond an implementation note at most.

The problems are indeed with UAs, and it is true that such comments are
most unlikely to appear in Real Life (so getting agents to accept them
comes fairly low in implementors' priority orders).

However, it would be unwise to assume that existing mail UAs "already
necessarily handle comments". Have you actually tried it? Sadly, broken
agents are a fact of life, however much King Canute might wish them to go
away.

For sure, it is known for a fact that some agents (news-only ones maybe)
do NOT currently handle these comments correctly.

---

From: Bruce Lilly <[email-masked]>
Date: Thu, 9 Jun 2005 16:50:31 -0400

> Ah, yes, it's done that since 2.4.

Q1: for relaying only, or for relaying and injection?

Q2: any estimate of the number of pre-2.4 installations still active?

> It doesn't accept colons, btw.

Q3: In the Newsgroups field?

Q4: What about colon in Path; does "doesn't accept" mean:
   a) rejects articles
   b) interprets colon as a delimiter
   c) does not interpret colon as a delimiter

---

From: Bruce Lilly <[email-masked]>
Date: Thu, 9 Jun 2005 17:25:16 -0400

> consider this carefully before deciding whether to provide it.

There were several specific examples, none of which used anything
resembling "in their opinion".
   
> certainly not saying "do whatever you please", any more than Mark was

There is a huge difference between documenting specific issues and
making specific recommendations vs. leaving matters to the opinions
of non-experts.

Another example of how issues can be handled is the following text
taken from the rfc2440bis draft:

        While this attack is somewhat obscure, requiring a special set
        of circumstances to create it, it is nonetheless quite serious
        as it permits someone to trick a user to decrypt a message.
        Consequently, it is important that:

         1. Implementers treat MDC errors and decompression failures as
            security problems.

         2. Implementers implement Modification Detection with all due
            speed and encourage its spread.

         3. Users migrate to implementations that support Modification
            Detection with all due speed.

Something similar could certainly be used where there are issues
corresponding to what Mark calls "baby programmer errors", e.g.
inability to handle msg-ids with a legal 255-octet domain name.  There
is no good reason to be stuck with such an absurdity forever; implementers
can and should be encouraged to fix the problems "with all due speed".
 
> necessarily handle comments".

No assumption necessary; comments have always been legal in RFC 724/733/822
structured fields -- any supposed "UA" that doesn't handle them properly is
not conformant.  Some software is crappy; fact of life.  Absent other
considerations, that's no reason to degrade specifications to the level of
crappiness of the lowest common denominator.

> do NOT currently handle these comments correctly.

Nobody is going to force them to claim compliance with our specification.
As RFCs 850 and 1036 refer to 822 and make no exception regarding comments
in References fields, such agents were not 850- or 1036-conforming.

---

From: Russ Allbery <[email-masked]>
Date: Thu, 09 Jun 2005 14:59:52 -0700

> Q1: for relaying only, or for relaying and injection?

Only for relaying.

> Q2: any estimate of the number of pre-2.4 installations still active?

Certainly at least hundreds.  My guess would be that something in the
range of 50-60% of the INN installations out there have upgraded at this
point, based on the questions that I get.

> Q3: In the Newsgroups field?

Right.  Sorry, that was rather unimportant; I ended up getting confused
between this and the Path header.  Of course it doesn't accept colons in
Newsgroups.  :)

>    c) does not interpret colon as a delimiter

INN considers [a-zA-Z0-9._-] to be valid hostname characters in the Path
header and all other characters are treated as delimiters.  So in other
words, the following path header:

    Path: f-oo.com%  bar_bar.com*10.10.10.10![::45af:89]=not-for-mail

is treated by INN as completely identical to:

    Path: f-oo.com!bar_bar.com!10.10.10.10!45af!89!not-for-mail

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 10 Jun 2005 03:28:58 +0200

Russ Allbery wrote:

> INN considers [a-zA-Z0-9._-] to be valid hostname characters
[...]

>     Path: f-oo.com!bar_bar.com!10.10.10.10!45af!89!not-for-mail

I see a hack for our IPv6 literals, replace colon by underscore
- but that's only a very bad dream, right ?

                         Bye, Frank

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Fri, 10 Jun 2005 08:02:34 +0200

> - but that's only a very bad dream, right ?

The colon is giving everyone bad dreams, some of which are RFCs :-(

RFC 3986 (URI syntax) put square brackets around IPv6 addresses to 
disambiguate them from hostnames; at the time, a number of solutions were 
suggested, including representing the IPv6 address as 32 hex digits without 
punctuation and percent-escaping the colons.

In the end, the people doing that syntax decided to use the colon and let 
the parsers fail where they may.

silly (?) question:

is there a failure mode in any known deployed software in which Path: 
bang![dead::beef]!not-for-mail breaks things worse than Path: 
bang!dead::beef!not-for-mail?

Russ' description seems to indicate that INN would cause both paths to be 
treated identically..... and to create trouble for hosts "dead" and 
"beef"....

                         Harald

---

From: Russ Allbery <[email-masked]>
Date: Thu, 09 Jun 2005 23:10:45 -0700

> Russ' description seems to indicate that INN would cause both paths to
> be treated identically..... and to create trouble for hosts "dead" and
> "beef"....

Not that I'm aware of; I think they're equivalent.  Either software is
only going to use ! as a delimiter or software is going to use the
"standard" method of considering basically everything to be a delimiter.

Note that hosts dead and beef shouldn't happen in normal operations, since
Path entries should be FQDNs unless they're in the UUCP maps (and really,
these days, that latter excuse is getting less and less relevant).
However, of course, people do use random junk as their path identity from
time to time.

---

From: Russ Allbery <[email-masked]>
Date: Fri, 10 Jun 2005 12:02:46 -0700

Charles Lindsey <[email-masked]> writes:

> IMO, Newsgroups folding would be very useful. Newsgroups headers in
> excess of 80 characters are very common. They display badly in many user
> agents; e.g. there is just a '!' to indicate "rest of line truncated",
> or they are are wrapped arbitrarily in the middle of a newsgroup (or
> between newsgroups if you are lucky).

Surely this is a quality of implementation issue.  It's trivial to wrap
the newsgroups header for display; it's not like it's hard to parse.  My
user agent does this without any difficulty.

> Even where they are displayed "properly" folded, the user is given the
> impression that it would be OK to generate new articles like that, and
> copy/paste from what is displayed to a new article might or might not
> result in a valid header.

Surely this too is a quality of implementation issue; my user agent
removes folding and whitespace from the header when posting.

> Moreover, this is the only header, within News or Mail, that cannot be
> folded, so it sticks out like a sore thumb.

Er, no.  Message-ID cannot be folded, Control cannot be folded, Date will
probably break in practice if folded, Followup-To similarly cannot be
folded, and that's just off the top of my head.  And I'm still concerned
that folding Path might not actually work in practice, although it does
work with the software I personally have checked.

> Note that the problem is with relaying agents. I think all current user
> agents would accept and display them properly.

I'm pretty sure you're wrong.

> Therefore, I wish to see as much incentive as possible to get relaying
> agents upgraded. MUST accept in our draft is a start. SHOULD NOT
> generate would give more incentive than MUST NOT generate, and something
> even weaker (as in the article-* drafts) would give more incentive
> again.

If a new RFC differs from existing practice by too much, more requirements
that differ from existing practice cease to be motivation to change
software and become motivation to just ignore the document.

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 10 Jun 2005 16:42:41 GMT

>saying "Newsgroup folding is now permitted".

IMO, Newsgroups folding would be very useful. Newsgroups headers in excess
of 80 characters are very common. They display badly in many user agents;
e.g. there is just a '!' to indicate "rest of line truncated", or they are
are wrapped arbitrarily in the middle of a newsgroup (or between
newsgroups if you are lucky). Even where they are displayed "properly"
folded, the user is given the impression that it would be OK to generate
new articles like that, and copy/paste from what is displayed to a new
article might or might not result in a valid header.

The problem is that (most) relaying agents need to examine this header
during transit. To upgrade them to accept the folding is not rocket
science, but it is only now beginning to get done. So at present such
articles would not propagate well. Note that the problem is with relaying
agents. I think all current user agents would accept and display them
properly.

Therefore, I wish to see as much incentive as possible to get relaying
agents upgraded. MUST accept in our draft is a start. SHOULD NOT generate
would give more incentive than MUST NOT generate, and something even
weaker (as in the article-* drafts) would give more incentive again.

---

From: Graham Drabble <[email-masked]>
Date: Sat, 11 Jun 2005 00:29:20 +0100

On 10 Jun 2005 "Charles Lindsey" <[email-masked]> wrote in
news:[email-masked]: 

> The problem is that (most) relaying agents need to examine this
> header during transit. To upgrade them to accept the folding is
> not rocket science, but it is only now beginning to get done. So
> at present such articles would not propagate well. 

How much of a problem is this going to cause big sites? I imagine the 
extra overhead of unfolding the header could be significant but have no 
data to go on.

-- 
Graham Drabble
http://users.ox.ac.uk/~sjoh1646/

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 10 Jun 2005 19:11:45 GMT

>Ah, yes, it's done that since 2.4.

OK, let's try it.

I have just submitted an article with the following headers to a server
using INN 2.4.1. It certainly got stored on that server. I invite you all
to see whether it arrived on any of the groups it was sent to, and
especially indications of how long it took and the route it took would be
interesting, since propagation using only INN 2.4+ servers may be rather
slow. If you do not see it in the named groups, it might be worth looking
to see if your server has filed it under "junk".

Newsgroups: man.test,
   uk.test,
   misc.test,
   de.test 
Path: chl
From:  "Charles Lindsey" <[email-masked]>
Subject: Testing folded newsgroups.
Message-ID: <[email-masked]>
Date: Fri, 10 Jun 2005 18:32:18 GMT
Lines: 14

It would also be helpful if others who have relaying privileges would try
the same experiment.

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 10 Jun 2005 18:44:59 GMT

>making specific recommendations vs. leaving matters to the opinions

I am impressed by your classification of all news-server implementors as
"non-experts".

>can and should be encouraged to fix the problems "with all due speed".

The restriction of msg-ids t0 250 characters is not a problem to be fixed
"with all due speed". It was extensively discussed by the NNTP working
group and it was discussed in this WG also.

It is now hard coded into the NNTP-draft which has just completed its IETF
Last Call (I note that you did not object to it).

It is a Done Deal.

>crappiness of the lowest common denominator.

Taking note of the lowest common denominator is, in practice, the way
things often work in the Real World. It is more politely expressed as
"being liberal".

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 11 Jun 2005 05:03:29 +0200

> Message-ID: <[email-masked]>

I don't see it on news.clara.net (about 0300Z), it arrived on
Google <http://purl.net/net/msgid/[email-masked]>
- but there you don't see the Path :-(

                          Bye, Frank

---

From: Russ Allbery <[email-masked]>
Date: Fri, 10 Jun 2005 20:34:57 -0700

> OK, let's try it.

I don't find these exercises useful.  I don't think they really prove what
you're trying to prove.  Propagation only measures whether or not the vast
majority of servers reject it; if only a smaller number of servers reject
the message, it could still be a significant problem in practice but won't
show on this sort of a test.

Furthermore, most transit servers do nearly no syntax validation of the
article whatsoever, or only the most minimal amount possible, so this sort
of test proves that all the new syntax we want to add works fine, also
proves that all the new syntax we don't want to add works fine, and
additionally proves that it's just fine to use "MEZT" as your time zone.
:)  You're lucky transit servers check even Message-ID and Date.  So
proving that things can propagate really is pretty meaningless.

The real test is how many software packages might have trouble when trying
to read the message, trying to construct followups, trying to post a
similar message back into a newsgroup, trying to archive that message,
trying to gate it into mail, etc.

---

From: Bruce Lilly <[email-masked]>
Date: Fri, 10 Jun 2005 23:34:53 -0400

> I am impressed by your classification of all news-server implementors as

I have not done so and I do not appreciate your false assertion.

---

From: Russ Allbery <[email-masked]>
Date: Fri, 10 Jun 2005 20:38:45 -0700

>> making specific recommendations vs. leaving matters to the opinions of
>> non-experts.

> I am impressed by your classification of all news-server implementors as

I have no difficulty whatsoever classifying most news server implementors
as non-experts.  One thing that's become abundantly clear over the years
is that most news implementors in general do not bother to read the
standards.  I'm not sure how to fix this, although getting a new, more
modern standard out the door would certainly help.

>> is no good reason to be stuck with such an absurdity forever;
>> implementers can and should be encouraged to fix the problems "with all
>> due speed".

Sorry, I'm just completely unenthused about trying to fix this
non-problem.  It offends people's sense of cleanliness to limit the length
of anything.  I understand, but that's not a reason to change the
standard.  Interoperability is more important than that sort of
aesthetics.  There is no earthly reason why anything needs to generate
251-octet message IDs.

---

From: Richard Clayton <[email-masked]>
Date: Sun, 12 Jun 2005 13:16:58 +0100

In message <[email-masked]>, Charles Lindsey
<[email-masked]> writes

>IMO, Newsgroups folding would be very useful.

cross-posting of articles is seldom a wise thing to do; it doesn't seem
to be wise to encourage it any more than at present :-(

One of the advantages of not having any folding is that we do not get
articles spammed to more than a dozen or so newsgroups.

With folding, we will see articles being sent to hundreds of newsgroups
at a time on a regular basis ... if that's not an issue for server
software, then I don't suppose its any worse for the readers than the
current spam situation.

However, I'd like to see some comment as to whether dealing with a
single article that needs to be inserted into hundreds or thousands of
groups at a time is likely to be affect performance, fragility, etc

> Newsgroups headers in excess
>of 80 characters are very common.

yes

> They display badly in many user agents;
>newsgroups if you are lucky). 

mayhap

>Even where they are displayed "properly"

really ?  could you give an example of where this has happened recently?

The way in which most user software works, you don't get to copy and
paste in truncated lists of newsgroups ...  and most people just press
the reply button and their software does the right thing with the
unmangled copy of the line (rather than the displayed version)

>agents. I think all current user agents would accept and display them

That's far from obvious!!   I had to read the code to believe this
statement about "my" client. As it happens, it looks as if it _is_ well
structured enough to accept a folded Newsgroups header field.

I have limited faith that other code would be as solidly written :(

>weaker (as in the article-* drafts) would give more incentive again.

"my" client, Turnpike, generates folded References: header fields and so
we now have ten years plus experience of this issue.

It has been quite common to see our References: mangled by other
software -- that assumed that just the first physical line needed to be
considered. When this made a mess of threading (or worse) then their
users complained and the authors of this other software then complained
to us, entirely incorrectly, that the standards had been ignored and we
had done the wrong thing :(

I too would like to see better handling of incoming material, but years
of experience suggest that this is a long haul :(  ALSO, the aim here
ought to be stated explicitly rather than buried in some subtle usage of
SHOULDs or whatever. Some implementors are, I'm afraid, dim :(

- -- 
richard                                              Richard Clayton

They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety.         Benjamin Franklin

---

From: Richard Clayton <[email-masked]>
Date: Sun, 12 Jun 2005 13:53:46 +0100

In message <[email-masked]>, Charles Lindsey
<[email-masked]> writes

>to see whether it arrived on any of the groups it was sent to

the article

>Message-ID: <[email-masked]>

is not currently (almost 2 days later) available on news.demon.co.uk
(well connected UK ISP server), news.mit.edu (MIT's server) or
news.cam.ac.uk (Cambridge University's server)

retention is sufficient for the article to be seen if it managed to
reach these servers by any flood-fill route (direct routes would have
meant that it arrived in seconds)

>It would also be helpful if others who have relaying privileges would try

the existing result speaks for itself

"Broken" articles like this don't propagate at all well :(

hence any suggestion that folded newsgroups header lines are current
state of the art (rather than some future possibility) seems premature

- -- 
richard                                              Richard Clayton

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Sun, 12 Jun 2005 14:59:57 +0200

> show on this sort of a test.

Actually I find them useful - but of course one has to be careful to check 
what they really test.

If this test had uncovered significant problems in distribution (query: did 
anyone see it in group man.test, but NOT in group misc.test, where both 
where carried?), newsgroup folding would have been dead in this round; it's 
clear that a standard meant to be useful in present reality has to be a 
"MUST NOT generate".

If it should uncover no significant problem in distribution, we can still 
come to the conclusion that it should be MUST NOT generate, because they 
cause other problems.

My opinion....

                    Harald

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 13 Jun 2005 12:08:14 GMT

>structured enough to accept a folded Newsgroups header field.

My expectation was based on a presumption that the code used to display
the Newsgroups header would be no different that that used for other
headers, where UAs already cope happily with folded headers, either
displaying them "as received", of wrapping/refolding them to suit the
particular display device.

I could well believe that some agents might have a problem when generating
a followup, though even then the "naive" approach of just copying the
header as received into the new article would be correct.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 13 Jun 2005 14:18:38 GMT

>you're trying to prove.

It wasn't meant to _prove_ anything. It was intended to find out how well
such articles would propagate at the present time.

Not as well as I would have expected, it seems, but since it got as far as
Google it certainly travelled some considerable distance from where I sent
it. But it would be nice to see some Paths.

>  Propagation only measures whether or not the vast

What I am curious to know is whether some servers decided not to store it
even after they had received it (or whether they stored it in "junk").
Looking in server logs might be instructive.

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 13 Jun 2005 14:42:54 GMT

>Surely this is a quality of implementation issue.

Then why does RFC 2822 make such an issue of it, giving encouragement to
fold everything within 78 characters (rather than 998), presumablt because
that's how user agents prefer it.

Generally speaking, I have a great dislike if what user agents show me has
been munged beyond ehat was actually on the wire.

>work with the software I personally have checked.

Message-ID cannot be folded simply because the only folding points are at
the beginning and end, and we don't allow empty header content or empty
folded lines.

Control _can_ be folded (it has FWS in it).

Folding of Date is possible, but hevily deprecated in RFC 2822 (except
before any final comment). We inherit that deprecation.

Folding of Followup-To stands or falls with folding of Newsgroups.

I believe Paths are regularly folded in the Real Usenet.

>I'm pretty sure you're wrong.

See reply to Richard.

>software and become motivation to just ignore the document.

There is a balance to be struck. Hence this discussion.

---

From: Russ Allbery <[email-masked]>
Date: Mon, 13 Jun 2005 09:37:04 -0700

>>> excess of 80 characters are very common. They display badly in many
>>> user agents; e.g. there is just a '!' to indicate "rest of line
>>> truncated", or they are are wrapped arbitrarily in the middle of a
>>> newsgroup (or between newsgroups if you are lucky).

> fold everything within 78 characters (rather than 998), presumablt
> because that's how user agents prefer it.

Because, all other things being equal, it's better to fold stuff.  As RFC
822 allowed folding all over the place, all other things are equal.  All
other things are most decidedly not equal for Usenet, which has never
allowed folding in some headers.

> Generally speaking, I have a great dislike if what user agents show me
> has been munged beyond ehat was actually on the wire.

That's really not a good reason to require in the protocol that articles
be pre-formatted for your personal preferences.  :)

> Control _can_ be folded (it has FWS in it).

We haven't fixed that yet?  Consider the same objections lodged as to
folding Newsgroups, but with considerably more vehemence.  Consider, for
instance, the behavior of:

    Control: newgroup example.some.really.long.newsgroup.name
      moderated

in different existing implementations.

> I believe Paths are regularly folded in the Real Usenet.

I think you're confusing what was written into this draft years ago with
what actually happens out on the wire where no one is paying attention to
these drafts.  :)

Since it happens to work with at least some major servers, a few people
may fold Path headers just for grins, but it is not in my experience an
event that could be described with the term "regularly."

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Mon, 13 Jun 2005 17:07:19 -0400

> that's how user agents prefer it.

In the Beginning Of Time, certain email systems ran on IBM mainframes, and 
stored email in record-oriented files, one record per line, which had a 
hard limit of 80 characters per line (think punch-card hard).
I believe (hope!) that all these systems are now dead.

A number of email readers ran by dumping their output on 80-character 
terminals, some of which had an irritating bug where, if you sent it 80 
characters and a CRLF, the cursor would move 2 lines down, not one - making 
column 80 "dangerous territory".
I hope these terminals are now disposed of in an environmentally friendly 
fashion, and no longer interesting.

When variable-width textboxes became available, non-folding headers 
sometimes caused a long scrollbar at the bottom of the header-display box. 
Uncomfortable.
(This happens also if you pull up a message with a long header in text 
format inside a text editor, btw)
Thus, a number of people grew to like everything folding within 80 
characters, because it gave a consistent display when you could guarantee 
that you had 80 characters available.

Summary: It used to be an interoperability issue. Nowadays I hope it's just 
an user preference issue.
(note on terminology: in my parlance, user agents are software; they 
execute, and work or fail to work. users are the ones who have 
preferences....)

                    Harald

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 14 Jun 2005 17:02:55 GMT

>in different existing implementations.

We discussed this recently, and it was agreed AIUI to replace CFWS by FWS
in the Control header.

>event that could be described with the term "regularly."

You may well be right about regular use, but certainly I have seen folded
Paths propagate in previous experiments, and I had always assumed this was
OK because it seems clear from RFC 1036 that the Path header may be
folded.

---

From: Russ Allbery <[email-masked]>
Date: Tue, 14 Jun 2005 14:37:00 -0700

>> folding Newsgroups, but with considerably more vehemence.  Consider,
>> for instance, the behavior of:

> We discussed this recently, and it was agreed AIUI to replace CFWS by
> FWS in the Control header.

It needs to be WS if you want to have a prayer of interoperating with
existing software.  Not even the latest development version of INN can
cope with a folded Control header.

> You may well be right about regular use, but certainly I have seen
> folded Paths propagate in previous experiments, and I had always assumed
> this was OK because it seems clear from RFC 1036 that the Path header
> may be folded.

I guarantee that software breaks if you fold the path header.  Software
will break with anything other than host names separated by !.  Now, we
may decide that we don't care about a lot of that software (such as ad-hoc
parsing scripts), and major news software should not break, but the Path
header format *in practice*, and therefore what much software has been
written to during this long interregnum without an effective Usenet
standard, is significantly more restrictive than what RFC 1036 says.

---

From: Bruce Lilly <[email-masked]>
Date: Wed, 15 Jun 2005 12:11:05 -0400

Russ, thanks for the clarifications.  One more:

>     Path: f-oo.com!bar_bar.com!10.10.10.10!45af!89!not-for-mail

Does INN treat numeric/hexadecimal/dotted path components as IP
addresses (e.g. comparing against the server's IP address(es) and/or
doing reverse DNS lookups), or are they treated purely as "names"?
I.e in the example above (ignoring ::45af:89 since we know there's a
colon issue), is "10.10.10.10" treated as an IP address?

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 15 Jun 2005 14:26:43 GMT

>standard, is significantly more restrictive than what RFC 1036 says.

It all the major news software does not break, then I could live with
problems with a few ad hoc scripts.

Two things are needed.

1. The folded Path header must propagate without bits of it getting lost.
That is a problem for relaying and serving agents to get right.

2. When a relaying agent reads through the Path, to avoid sending an
article where it has already been sent before, it must search the whole of
the Path. That also is a problem for relaying and serving agents to get
right.

If existing relaying agents in fact satisy both of those, then there is no
problem. Moreover, one can tolerate occasional failure of #2, because it
causes nothing to break beyond a bit of redundant transmission.

---

From: Russ Allbery <[email-masked]>
Date: Wed, 15 Jun 2005 11:09:14 -0700

Bruce Lilly <[email-masked]> writes:

> doing reverse DNS lookups), or are they treated purely as "names"?  I.e
> in the example above (ignoring ::45af:89 since we know there's a colon
> issue), is "10.10.10.10" treated as an IP address?

INN currently does no verification of path identities whatsoever.  All
path identities are used only as opaque tokens that are compared
case-insensitively against the recorded path identity for peers to avoid
sending them articles they already got.  DNS is never used (and in fact
anything at all related to DNS is carefully avoided in all core INN code,
since DNS lookups can take minutes and INN is single-threaded).

---

From: [email-masked] (Kai Henningsen)
Date: 17 Jun 2005 08:58:00 +0200

> I believe (hope!) that all these systems are now dead.

Well, the filesystems on which these files are stored insist on records -  
there is no non-record-oriented format. The best you could do is variable  
length format, and IIRC you still need a max length ...

And while I don't know about CMS or TSO or similar mainframe systems, the  
AS/400 is alive and well and has a very similar filesystem (actually, it  
seems to be something like a thin layer above a SQL database), and I'm  
pretty certain there are people doing mail on it. When doing my small bit  
of C programming on that machine, I certainly need to cope with fixed-80  
format for C source.

And yes, that's all EBCDIC.

> terminals, some of which had an irritating bug where, if you sent it 80

80 character screens certainly still exist.

> column 80 "dangerous territory".

Curses fortunately can handle that. However, I'm not at all sure I'd agree  
to calling that a bug - it's pretty natural behaviour if you automatically  
wrap the line at the end. You actually need to do unnatural stuff to avoid  
that.

> fashion, and no longer interesting.

I certainly doubt that, too. But see the comment about curses - the  
*problem* ought to be gone, even if the terminals aren't.

The one annoying problem left is that these terminals usually cannot write  
into the lower right corner (because doing so causes immediate scrolling).  
Sometimes, this can be done with a trick (using horizontal or vertical  
insertion, if that capability is available).

MfG Kai

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 17 Jun 2005 20:13:30 +0200

Kai Henningsen wrote:

> When doing my small bit of C programming on that machine, I
> certainly need to cope with fixed-80 format for C source.

And with the RfC format they could punch a serial card number
in the columns 73..80, and that is exactly as it should be :-)

> 80 character screens certainly still exist.

Every normal text mode window uses it, 80 is better than 40.

>> making column 80 "dangerous territory".
> Curses fortunately can handle that

On a PC with a straight forward BIOS interface, virtual or
real, I found no way around "stay clear of row 25 column 80
in the string output function".  Yes, PCcurses handles it,
but internally it has to care about it.

> The one annoying problem left is that these terminals
> usually cannot write into the lower right corner (because
> doing so causes immediate scrolling).

It works if you use single char output in that position -
test Mark's PDcurses or for DOS test PCcurses (sorry, the
latter has no mouse support).  For ncurses it would be a
problem in the termcap-layer, not in ncurses.

                       Bye, Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 20 Jun 2005 09:53:34 GMT

>Every normal text mode window uses it, 80 is better than 40.

Indeed. I have seen lots of 80 column windows which make life complicated
if you actually try to put a character in column 80 (hence the advice to
keep lines at or below 79).

But I have never seen such a window in which complicated things happened
if you tried to put a character in column 79. Hence I never understood the
"78" figure in RFC 2822, in spite of asking for a justification on several
occasions.

---

From: Eivind Tagseth <[email-masked]>
Date: Mon, 20 Jun 2005 16:12:44 +0200

> "78" figure in RFC 2822, in spite of asking for a justification on several

I always thought 78 was used to allow for a limited set of quotation 
characters?

When following up on a message with 79/80 characters wide lines, formatting
any quoted text is a pain.

Eivind

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 20 Jun 2005 21:33:50 +0200

Charles Lindsey wrote:
 
> I have never seen such a window in which complicated things
> happened if you tried to put a character in column 79. Hence
> I never understood the "78" figure in RFC 2822, in spite of
> asking for a justification on several occasions.

It's an ordinary "off by one" thing, I have exactly one tool
where it hits me.  If you port *NIX software to DOS, Windows,
or OS/2, CrLf is one character more than Lf, and then a simple
if ( 80 <= n ) strikes for 78 + 2.
                                    Bye, Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 21 Jun 2005 11:11:16 GMT

>I always thought 78 was used to allow for a limited set of quotation 

Hmmmm! I have never seen that explanation offered before, and it is pretty
weak insofar as it only allows for one level of quotation.

USEAGE recommends a limit of 72 if you want to allow for quotation, and 79
if you don't really care about being quoted.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 21 Jun 2005 11:06:24 GMT

>if ( 80 <= n ) strikes for 78 + 2.

What is this tool that hits you, and exactly what is the nature of the
breakage that you see?

All the reasons I have seen in attempting to justify a 78 or 79 limit have
nothing to do with the number of characters needed to represent an EOL,
and have everthing to do with the space occupied by visible characters in
windows, terminals, etc.

---

From: Eivind Tagseth <[email-masked]>
Date: Tue, 21 Jun 2005 13:57:24 +0200

> if you don't really care about being quoted.

Ah... sorry, got those two numbers mixed up.

Eivind

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 21 Jun 2005 18:00:55 +0200

Charles Lindsey wrote:

> What is this tool that hits you, and exactly what is the
> nature of the breakage that you see?

It's a "norton commander clone" with among many other features
a built-in text viewer.  That beast folds lines if they reach
column 79.  I have many "magic" 78 and 79 in my REXX scripts.

> the reasons I have seen in attempting to justify a 78 or
> 79 limit have nothing to do with the number of characters
> needed to represent an EOL

Is there any other (and preferably less ridiculous) reason...

> the space occupied by visible characters in windows,
> terminals, etc.

...what's that, something with proportional fonts and 79 "M" ?

                          Bye, Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 22 Jun 2005 21:01:26 GMT

>column 79.  I have many "magic" 78 and 79 in my REXX scripts.

And it does that even if the window is wide enough to accomodate 80
characters? Weird!

>Is there any other (and preferably less ridiculous) reason...

Yes, lots of viewers do what your norton commander clone does, but not
until they reach column 80.