Subject: ISSUE #1414: Call for consensus

From: Russ Allbery <[email-masked]>
Date: Fri, 18 May 2007 19:20:44 -0700

Okay, I think we're all agreed on the specific wording change for this
issue.  Here is the diff one more time.  Unless someone tells me that they
don't agree with the resolution, I'm going to commit this next Tuesday,
May 22nd.

--- usepro.xml	2007-05-18 17:08:35.000000000 -0700
+++ usepro-1414.xml	2007-05-18 17:26:33.000000000 -0700
@@ -373,16 +373,22 @@
               specification will add only single "!" characters between
               &lt;path-identity> strings.</t>
 
-              <t>The agent MUST then prepend its own &lt;path-identity> to
-              the Path header field content.</t>
-
-              <t>The agent MAY then prepend additional &lt;path-identity>s
-              for itself to the Path header field content, following each
-              &lt;path-identity> so added with either "!!" or "!".  This
-              is permitted for agents that have multiple
-              &lt;path-identity>s (such as during a transition from one to
-              another).  Each of these &lt;path-identity>s MUST meet the
-              requirements set out in <xref target="path" />.</t>
+              <t>The agent MAY then prepend to the Path header field
+              content "!" or "!!" followed by an additional
+              &lt;path-identity> for itself other than its primary one.
+              Using "!!", and thereby adding a &lt;diag-match> since the
+              &lt;path-identity> clearly is verified, is RECOMMENDED.  This
+              step may be repeated any number of times.  This is permitted
+              for agents that have multiple &lt;path-identity>s (such as
+              during a transition from one to another).  Each of these
+              &lt;path-identity>s MUST meet the requirements set out in
+              <xref target="path" />.</t>
+
+              <t>Finally, the agent MUST prepend its primary
+              &lt;path-identity> to the Path header field content.  The
+              primary &lt;path-identity> is the &lt;path-identity> it
+              normally advertises to its peers for their use in generating
+              &lt;path-diagnostic>s as described above.</t>
             </list>
           </t>
 
-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 21 May 2007 16:33:01 GMT

>+++ usepro-1414.xml	2007-05-18 17:26:33.000000000 -0700

Yes, I think that is all agreed.

There remains the related ISSUE of whether agents may OMIT some entries
from the Path (e.g. when they merely record some inner communications
within the one site). But the proposed alteration to allow that affects a
different (earlier) piece of wording.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 22 May 2007 05:33:01 +0200

Looks Good To Me.

--On 18. mai 2007 19:20 -0700 Russ Allbery <[email-masked]> wrote:

> -              <t>The agent MUST then prepend its own &lt;path-identity>
> to -              the Path header field content.</t>
> -
> -              <t>The agent MAY then prepend additional
> &lt;path-identity>s -              for itself to the Path header field
> content, following each -              &lt;path-identity> so added with
> either "!!" or "!".  This -              is permitted for agents that
> have multiple
> -              &lt;path-identity>s (such as during a transition from one
> to -              another).  Each of these &lt;path-identity>s MUST meet
> the -              requirements set out in <xref target="path" />.</t> +
> <t>The agent MAY then prepend to the Path header field +
> content "!" or "!!" followed by an additional
> +              &lt;path-identity> clearly is verified, is RECOMMENDED.
> This +              step may be repeated any number of times.  This is
> permitted +              for agents that have multiple
> &lt;path-identity>s (such as +              during a transition from one
> to another).  Each of these +              &lt;path-identity>s MUST meet
> the requirements set out in +              <xref target="path" />.</t>
> +
> +              normally advertises to its peers for their use in
> generating +              &lt;path-diagnostic>s as described above.</t>
>              </list>
>            </t>
>
> --

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 23 May 2007 14:14:44 +0200

> different (earlier) piece of wording.

If I were being pedantic, I'd claim that a system of entities that adds one 
and only one entry to the path must be regarded as one agent, no matter how 
many computers or intermediate communication hops it consists of.

In the same way that an (IP) router will decrement the TTL of IP packets 
passing through it; a box or set of boxes with a summary function that 
decrements the TTL by 1 will have to be regarded as one (IP) router.

Actually, I don't see a reason to depart from pedantry at this point. Seen 
in that light, the "remaining issue" can't be stated in a meaningful way, 
so it must be regarded as resolved.

                 Harald

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 24 May 2007 08:46:37 GMT

>many computers or intermediate communication hops it consists of.

I think the administrator of the agent(s) concerned is best placed to
decide whether his complicated server farm is best regarded as a single
agent, when viewed from the outside, or not.

One observes many cases where a Path contains three, or even four, entries
evidently placed there by the one site. Often, there may be good reasons
why those entries all need to be visible from the outside - no problem
with that. But one suspects that often it just arises from blind adherence
to the idea that each host a message passes through needs to add its own
Path entry. I just want to make it clear that sites have some discretion
in this matter.

>so it must be regarded as resolved.

It is easily stated. My suggested wording was

Except possibly when relaying to other hosts on the same site, every
injecting, relaying, or serving agent that accepts an article MUST
update the Path header field ....

And it turns out that this is part of Issue #1415, not #1414
(though those two issues have now got well mixed).

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Thu, 24 May 2007 09:45:06 -0400

Charles Lindsey wrote:

[snip]
 >I just want to make it clear that sites have some discretion
 >in this matter.
[snip]

> update the Path header field ....

Not better, in my view.

1. It seems to indicate that "hosts" can be "relays" without being agents.  Can 
you explain?

2. How is this a statement of discretion? Your wording is a narrow exception to 
a MUST.  That makes it more likely that they add them, doesn't it?

3. I like Harald's pendatry on this one.  A server farm could be a mixture of 
NATing firewalls, load-balancing routers, front-end NNTP caches, multi-cast 
relayers, backend servers attached to NFS, archive storage boxes.  All of these 
are hosts.  I think that introducing "host" is not going to bring clarity.

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 31 May 2007 10:31:58 GMT

>1. It seems to indicate that "hosts" can be "relays" without being agents.  Can 

s/hosts/agents/ if you like.

>a MUST.  That makes it more likely that they add them, doesn't it?

"Except possibly" indicates that the MUST can be overridden in the given
circumstance. As to whether that makes it more or less likely to add them,
I don't really care. It gives them the discretion of they care to use it.
Further advice would be for USEAGE, it anywhere.

>are hosts.  I think that introducing "host" is not going to bring clarity.

And I don't really want all those to be visible in the Path if they are of
no real interest to outside sites. But USEAGE would be the place where
that might be discussed. USEPRO just needs to set the rules.

To: [email-masked]
Xref: clerew local.usefor:24704
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: #1415 (was ISSUE #1414: Call for consensus)
Date: Thu, 31 May 2007 10:31:58 GMT
Lines: 49
Sender: [email-masked]

>1. It seems to indicate that "hosts" can be "relays" without being agents.  Can 

s/hosts/agents/ if you like.

>a MUST.  That makes it more likely that they add them, doesn't it?

"Except possibly" indicates that the MUST can be overridden in the given
circumstance. As to whether that makes it more or less likely to add them,
I don't really care. It gives them the discretion of they care to use it.
Further advice would be for USEAGE, it anywhere.

>are hosts.  I think that introducing "host" is not going to bring clarity.

And I don't really want all those to be visible in the Path if they are of
no real interest to outside sites. But USEAGE would be the place where
that might be discussed. USEPRO just needs to set the rules.

Date: Thu, 24 May 2007 09:45:06 -0400
From: "Forrest J. Cavalier III" <[email-masked]>
To: [email-masked]
Subject: Re: #1415 (was ISSUE #1414: Call for consensus)
Sender: [email-masked]

Charles Lindsey wrote:

[snip]
[snip]

Not better, in my view.

To: [email-masked]
Xref: clerew local.usefor:24702
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: #1415 (was ISSUE #1414: Call for consensus)
Date: Thu, 24 May 2007 08:46:37 GMT
Lines: 49
Sender: [email-masked]

To: [email-masked]
Xref: clerew local.usefor:24701
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: #1416 Injection-Date: current wording proposal (corrected)
Date: Thu, 24 May 2007 08:25:23 GMT
Lines: 24
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>I now have:

>            <t>It MUST reject any article that has already been accepted.
>            If it implements the mechanism described in <xref
>            target="history" />, this means that it MUST reject any
>            article whose date falls outside a cutoff interval since it
                                              ^
                                             the
>            won't know whether such articles had been accepted or not.</t>
                                                               ^
                                                         earlier/previously

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: #1416 Injection-Date: current wording proposal (corrected)
Organization: The Eyrie
Date: Wed, 23 May 2007 20:05:23 -0700
Sender: [email-masked]

Forrest J Cavalier <[email-masked]> writes:
> Harald Tveit Alvestrand wrote:

>> Going to 2 sentences would help me, I think.
>> 
>>   It MUST reject any article that has already been successfully sent to it.
>>   If it implements the mechanism described in <xref history>, this means
>> that it
>>   will have to reject any article where its date falls outside a cutoff
>> interval,
>>   since it doesn't know whether it has been successfully sent to it or not.
>> 
>> If shorter text is desired, drop the part after the comma.
>> 

> I really like that this expresses the idea that if you can't know if you
> accepted it or not, you must reject.

> I would wordsmith:

> 	successfully sent to it -> accepted.

> (There have been some propagation algorithms which trust articles from
> certain peers, but not others.)

> And I don't have a strong opinion, but should this part also change:
> 	will have to reject -> MUST reject.

I now have:

            <t>It MUST reject any article that has already been accepted.
            If it implements the mechanism described in <xref
            target="history" />, this means that it MUST reject any
            article whose date falls outside a cutoff interval since it
            won't know whether such articles had been accepted or not.</t>

since indeed I think that rejecting such articles is a MUST *if* the
server is using that algorithm.

Date: Wed, 23 May 2007 14:14:44 +0200
From: Harald Tveit Alvestrand <[email-masked]>
To: Charles Lindsey <[email-masked]>, [email-masked]
Subject: Re: ISSUE #1414: Call for consensus
Sender: [email-masked]

                 Harald

To: [email-masked]
Xref: clerew local.usefor:24697
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: #1416 Injection-Date: current wording proposal (corrected)
Date: Wed, 23 May 2007 10:02:01 GMT
Lines: 22
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>This header has been around for years.  Has this ever happened?  If it
>has, I have no objections to mentioning it in Security Considerations, but
>if it hasn't, it seems like wasted verbiage to me.

Not that I know of. But the inventors of that header wisely included a
SHOULD NOT use it in News, and we should either invoke that by reference,
or specify it explicitly. And it is a good idea for injecting agents to
check for its absence, and for sure it needs a mention in the Security
Considerations.

To: [email-masked]
Xref: clerew local.usefor:24698
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: #1416 Injection-Date: current wording proposal (corrected)
Date: Wed, 23 May 2007 10:07:32 GMT
Lines: 34
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>Charles Lindsey <[email-masked]> writes:

>> Yes, but I am not expecting posting agents to be routinely adding
>> Injection-Date (except in the special case where they are
>> multi-injecting).

>Agents that add Date and Message-ID should also add Injection-Date.  I'd
>be happy to add wording to that effect.

That would certainly help if Option IR is to be retained, but it all gets
rather messy. Option IC is much cleaner, even though it may occasionally
make things worse in the short term.

>The current protocol lets posting agents add Date, which has the same
>protocol role.  Having the posting agent add Injection-Date doesn't make
>matters any worse than it is now.

My worry is that poor implementations will just create Injection-Date as a
copy of Date, which defeats the whole purpose. Your wording would need to
make it very clear that that is not good enough (and why).

To: [email-masked]
From: [email-masked]
Subject: Commit in docs/usefor (usepro.xml)
Date: Tue, 22 May 2007 11:47:37 -0700 (PDT)
Sender: [email-masked]

    Date: Tuesday, May 22, 2007 @ 11:47:36
  Author: eagle
Revision: 2992

Require a space before (Moderated) in application/news-groupinfo and
clarify the ABNF for the moderation-flag.

Modified:
  docs/usefor/usepro.xml

Modified: docs/usefor/usepro.xml
===================================================================
--- docs/usefor/usepro.xml	2007-05-22 18:45:26 UTC (rev 2991)
+++ docs/usefor/usepro.xml	2007-05-22 18:47:36 UTC (rev 2992)
@@ -1379,11 +1379,11 @@
                                ; "For your newsgroups file:"
       newsgroups-line     = newsgroup-name
                                [ 1*HTAB newsgroup-description ]
-                               [ 1*WSP moderation-flag ]
+                               [ *WSP moderation-flag ]
       newsgroup-description
                           = eightbit-utext *( *WSP eightbit-utext )
-      moderation-flag     = %x28.4D.6F.64.65.72.61.74.65.64.29
-                               ; case sensitive "(Moderated)"
+      moderation-flag     = SP "(" %4D.6F.64.65.72.61.74.65.64 ")"
+                               ; SPACE + case sensitive "(Moderated)"
       eightbit-utext      = utext / %d127-255
           </artwork>
         </figure>

To: [email-masked]
From: [email-masked]
Subject: Commit in docs/usefor (usepro.xml)
Date: Tue, 22 May 2007 11:45:28 -0700 (PDT)
Sender: [email-masked]

    Date: Tuesday, May 22, 2007 @ 11:45:26
  Author: eagle
Revision: 2991

Resolution of #1414.  Be clearer about how additional path identities
are prepended, be clear about using !! rather than !, and put the most
canonical Path identity left-most.

Modified:
  docs/usefor/usepro.xml

Modified: docs/usefor/usepro.xml
===================================================================
--- docs/usefor/usepro.xml	2007-05-22 18:35:49 UTC (rev 2990)
+++ docs/usefor/usepro.xml	2007-05-22 18:45:26 UTC (rev 2991)
@@ -373,16 +373,22 @@
               specification will add only single "!" characters between
               &lt;path-identity> strings.</t>

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: #1416 Injection-Date: current wording proposal (corrected)
Organization: The Eyrie
Date: Tue, 22 May 2007 09:42:22 -0700
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:
> Russ Allbery <[email-masked]> writes:

>> I don't see how this would be the case.  Either posting agents add
>> Injection-Date or they don't add Date and Message-ID and the injecting
>> agent does.

> Yes, but I am not expecting posting agents to be routinely adding
> Injection-Date (except in the special case where they are
> multi-injecting).

Agents that add Date and Message-ID should also add Injection-Date.  I'd
be happy to add wording to that effect.

> My view is that it is really the injecting agent's job, and that
> injecting agents are more likely to do it right than posting agents in
> the hands of unskilled users.

The current protocol lets posting agents add Date, which has the same
protocol role.  Having the posting agent add Injection-Date doesn't make
matters any worse than it is now.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: #1416 Injection-Date: current wording proposal (corrected)
Organization: The Eyrie
Date: Tue, 22 May 2007 09:41:03 -0700
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:

> I think the problem is combined mail/news user agents, or which there
> are a lot around. The naive user is likely to configure his agent to
> always include this header ("because it seems like a Good Idea"), and
> for the badly written user agent (or which, again, there are many) not
> to notice that the option is not supposed to be activated for News.

This header has been around for years.  Has this ever happened?  If it
has, I have no objections to mentioning it in Security Considerations, but
if it hasn't, it seems like wasted verbiage to me.

Including it as an example of a deprecated header is certainly fine.

To: [email-masked]
Xref: clerew local.usefor:24689
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: #1416 Injection-Date: current wording proposal (corrected)
Date: Tue, 22 May 2007 15:22:23 GMT
Lines: 40
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>Charles Lindsey <[email-masked]> writes:
>> Russ Allbery <[email-masked]> writes:

>>>>> +            not syntactically valid as defined by <xref target="USEFOR"
>>>>> +            />.  It SHOULD reject any proto-article which contains a
>>>>> +            header field deprecated for Netnews.  It MAY reject any
>>>>                                                  ^
>>>>                                       (see e.g. <xref RFC2298>)

>>> RFC 2298 is obsolete, so I don't really want to generate even an
>>> informative reference to it, and the superseding RFC (RFC 3798) makes no
>>> mention of Netnews, so I'm not sure this is a great example or
>>> particularly helpful for the reader.

>> The problem is that use of that header in Netnews can give rise to the
>> Mother of all Mailbombs. This is also a Security Consideration, and I
>> think it is also mentioned as such (or should be).

>I think it's overkill to be that worried about an obscure header that no
>one has ever implemented for Netnews.

I think the problem is combined mail/news user agents, or which there are
a lot around. The naive user is likely to configure his agent to always
include this header ("because it seems like a Good Idea"), and for the
badly written user agent (or which, again, there are many) not to notice
that the option is not supposed to be activated for News.

To: [email-masked]
Xref: clerew local.usefor:24688
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: #1416 Injection-Date: current wording proposal (corrected)
Date: Tue, 22 May 2007 15:18:31 GMT
Lines: 28
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>Charles Lindsey <[email-masked]> writes:

>> No, it's worse than that. As currently worded, that situation will
>> persist FOR EVER, even after every agent on the Net implements
>> Injection-Date fully.

>I don't see how this would be the case.  Either posting agents add
>Injection-Date or they don't add Date and Message-ID and the injecting
>agent does.

Yes, but I am not expecting posting agents to be routinely adding
Injection-Date (except in the special case where they are
multi-injecting). My view is that it is really the injecting agent's job,
and that injecting agents are more likely to do it right than posting
agents in the hands of unskilled users.

Date: Tue, 22 May 2007 08:01:21 -0400
From: "Forrest J. Cavalier III" <[email-masked]>
To: [email-masked]
Subject: Re: #1416 Injection-Date: current wording proposal (corrected)
Sender: [email-masked]

Harald Tveit Alvestrand wrote:
> Going to 2 sentences would help me, I think.
> 
>   It MUST reject any article that has already been successfully sent to it.
>   If it implements the mechanism described in <xref history>, this means 
> that it
>   will have to reject any article where its date falls outside a cutoff 
> interval,
>   since it doesn't know whether it has been successfully sent to it or not.
> 
> If shorter text is desired, drop the part after the comma.
> 

I really like that this expresses the idea that if you can't know if you 
accepted it or not, you must reject.

I would wordsmith:

	successfully sent to it -> accepted.

(There have been some propagation algorithms which trust articles from certain 
peers, but not others.)

And I don't have a strong opinion, but should this part also change:
	will have to reject -> MUST reject.

Date: Tue, 22 May 2007 05:33:01 +0200
From: Harald Tveit Alvestrand <[email-masked]>
To: Russ Allbery <[email-masked]>, [email-masked]
Subject: Re: ISSUE #1414: Call for consensus
Sender: [email-masked]

Looks Good To Me.

> +++ usepro-1414.xml	2007-05-18 17:26:33.000000000 -0700

Date: Tue, 22 May 2007 05:41:31 +0200
From: Harald Tveit Alvestrand <[email-masked]>
To: Russ Allbery <[email-masked]>, [email-masked]
Subject: Re: #1416 Injection-Date: current wording proposal (corrected)
Sender: [email-masked]

--On 18. mai 2007 20:47 -0700 Russ Allbery <[email-masked]> wrote:

>
> Forrest J Cavalier <[email-masked]> writes:
>> Russ Allbery wrote:
>
>>>>> +            <t>It MUST reject any article that has already been
>>>>
>>>>               successfully sent to it, or alternatively where its date
>>>>               falls outwith any cutoff interval as described in <xref
>>>>               history>.
>
>>> I now have:
>>>
>>>    3.  It MUST reject any article that has already been successfully
>>>        sent to it, which may include rejecting articles whose dates fall
>>>        outside a cutoff interval as described in Section 3.3.
>
>> This changes the meaning significantly.  Is that right?
>
> Probably not.  I'm failing to find the right wording here.
>
> News servers MUST reject articles that have already been successfully sent
> to them.  They're given an allowance to reject messages that have not
> already been successfully sent to them as part of this if that's needed
> for their algorithm (which doesn't change the protocol, since they can
> always reject more articles for any reason at all).  *If* the server uses
> the schemes in 3.3, *then* they must reject articles whose dates lie
> outside the cutoff interval.
>
> Suggestions for wording that captures this?

Going to 2 sentences would help me, I think.

  It MUST reject any article that has already been successfully sent to it.
  If it implements the mechanism described in <xref history>, this means 
that it
  will have to reject any article where its date falls outside a cutoff 
interval,
  since it doesn't know whether it has been successfully sent to it or not.

If shorter text is desired, drop the part after the comma.

                Harald

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: #1416 Injection-Date: current wording proposal (corrected)
Organization: The Eyrie
Date: Mon, 21 May 2007 21:59:57 -0700
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:
> Russ Allbery <[email-masked]> writes:

>>>> +            not syntactically valid as defined by <xref target="USEFOR"
>>>> +            />.  It SHOULD reject any proto-article which contains a
>>>> +            header field deprecated for Netnews.  It MAY reject any
>>>                                                  ^
>>>                                       (see e.g. <xref RFC2298>)

>> RFC 2298 is obsolete, so I don't really want to generate even an
>> informative reference to it, and the superseding RFC (RFC 3798) makes no
>> mention of Netnews, so I'm not sure this is a great example or
>> particularly helpful for the reader.

> No, RFC 3798 still contains the relevant wording:

>    Messages posted to newsgroups SHOULD NOT have a Disposition-
>    Notification-To header.

Ah, newsgroups.  That's one of the keywords I didn't search for.

Added to my draft text.

> The problem is that use of that header in Netnews can give rise to the
> Mother of all Mailbombs. This is also a Security Consideration, and I
> think it is also mentioned as such (or should be).

I think it's overkill to be that worried about an obscure header that no
one has ever implemented for Netnews.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: #1416 Injection-Date: current wording proposal (corrected)
Organization: The Eyrie
Date: Mon, 21 May 2007 21:55:23 -0700
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:

> No, it's worse than that. As currently worded, that situation will
> persist FOR EVER, even after every agent on the Net implements
> Injection-Date fully.

I don't see how this would be the case.  Either posting agents add
Injection-Date or they don't add Date and Message-ID and the injecting
agent does.

To: [email-masked]
Xref: clerew local.usefor:24683
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: #1416 Injection-Date: current wording proposal (corrected)
Date: Mon, 21 May 2007 16:27:29 GMT
Lines: 49
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>Harald Alvestrand <[email-masked]> writes:

>> There are 3 cases:

>> - Message is singly injected. Injection-Date: does no harm.
>> - Message is multiply injected, Injection-Date: on all copies are quite
>> close to each other, because they all got injected in rapid
>> succession. Injection-Date: does no harm.
>> - Message is multiply injected, and some of the injections got
>> significantly delayed.

>Right, the problems only potentially happen in the third case, but that
>third case includes many serial injection cases.

>> So the net result of catering for the minority third case is that *most*
>> messages on the Net will be sent without an Injection-Date.

>Correct, until posting agents are updated to generate it.  ...

No, it's worse than that. As currently worded, that situation will persist
FOR EVER, even after every agent on the Net implements Injection-Date
fully.

This is essentially the difference between options IR and IC, where IC
allows that things may temporarily get slightly worse in order that the
final steady-state situation will be cleaner.

We need to resolve the IR vs IC issue soon, but I think it best to get all
the other aspects of Injection-Date sorted out, which we now seem to be
well on the way to doing.

> ...This is the way
>that I think it should work, since all the benefits of Injection-Date are
>born by the posting agents, not by the injecting agents (which don't care
>at all).

To: [email-masked]
Xref: clerew local.usefor:24682
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: #1416 Injection-Date: current wording proposal (corrected)
Date: Mon, 21 May 2007 16:12:47 GMT
Lines: 100
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>Charles Lindsey <[email-masked]> writes:

>> OK, I have finally got around to reading the whole of this. Many of my
>> points are minor textual niggles, but some are quite substantive.

>>> @@ -478,48 +565,70 @@
>>> 
>>>           <t>A proto-article has the same format as a normal article
>>> +          except that the Injection-Info and Xref header fields MUST NOT
>>> +          be present; the Path header field MUST NOT contain a "POSTED"
>>> +          &lt;diag-keyword>; ........

>> I disagree with forbidding POSTED (already raised in a separate thread),
>> and am dubious about Xref ("SHOULD" would be quite strong enough).

>I'm not sure we've reached a conclusion on those other threads yet.  So
>far, I've not made any changes here.

OK, that is an issue still waiting resolution. At least all the places
affected are now documented.

>> However, this does not really convey the scenario as it is anticipated to
>> occur. In essence, the first network is typically small and private and
>> the second network is usually large, and probably Usenet. And one does not
>> usually just "find" articles there - one usually placed them there
>> oneself.

>> So how about:

>>             ....... (such as when gatewaying, after the fact, articles
>>             from a small private network, supposedly with no other
>> 	    connections, to a larger network, such as Usenet). ......

>Well, that's one of the cases where offering the same proto-article to all
>injecting agents *is* possible, normally, so I'm not sure it's a great
>example.  That's the use case where I really wish people would just use
>normal multiple injection.

Yes, but the case arises where the small private network contains several
sites some of which, by misunderstanding, misconfiguration, or the result
of some anticipated cross-posting also inject the article (so the
"official" injecting channel does not get the option of multi-injecting it
properly, even if he wanted to).

So I think such examples need mentioning, but maybe in different words.
The problem with "small private networks" is that they tend to grow
without the knowledge of the network administrator :-( .

>>> @@ -644,23 +753,24 @@
>>> 
>>>             <t>It MUST reject any proto-article that does not have the
>>>             proper mandatory header fields for a proto-article; that has
>>> +            Injection-Info or Xref header fields; that has a Path header
>>               ^              ^^^^^^^^^^^^^^^^^^^^^  ^^^^^^^^^^^^^^^^^^^^^^
>>              any                   field
>>> +            field containing the "POSTED" &lt;diag-keyword>; or that is
>>               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

>Likewise here.

>>> +            not syntactically valid as defined by <xref target="USEFOR"
>>> +            />.  It SHOULD reject any proto-article which contains a
>>> +            header field deprecated for Netnews.  It MAY reject any
>>                                                  ^
>>                                       (see e.g. <xref RFC2298>)

>RFC 2298 is obsolete, so I don't really want to generate even an
>informative reference to it, and the superseding RFC (RFC 3798) makes no
>mention of Netnews, so I'm not sure this is a great example or
>particularly helpful for the reader.

No, RFC 3798 still contains the relevant wording:

   Messages posted to newsgroups SHOULD NOT have a Disposition-
   Notification-To header.

and we need to draw attention to that (or, alternatively, explicitly
prohibit that header ourselves). The problem is that use of that header in
Netnews can give rise to the Mother of all Mailbombs. This is also a
Security Consideration, and I think it is also mentioned as such (or
should be).

I am not aware of any other headers that are explicitly deprecated for
Netnews by other standards, but there might be.

To: [email-masked]
Xref: clerew local.usefor:24685
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: ISSUE #1414: Call for consensus
Date: Mon, 21 May 2007 16:33:01 GMT
Lines: 40
Sender: [email-masked]

To: [email-masked]
Xref: clerew local.usefor:24684
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: #1416 Injection-Date: current wording proposal (corrected)
Date: Mon, 21 May 2007 16:30:12 GMT
Lines: 35
Sender: [email-masked]

In <[email-masked]> "Forrest J. Cavalier III" <[email-masked]> writes:

>Russ Allbery wrote:
>>>>+            <t>It MUST reject any article that has already been
>>>
>>>               successfully sent to it, or alternatively where its date
>>>               falls outwith any cutoff interval as described in <xref
>>>               history>.
>> 
>> 
>> I now have:
>> 
>>    3.  It MUST reject any article that has already been successfully
>>        sent to it, which may include rejecting articles whose dates fall
>>        outside a cutoff interval as described in Section 3.3.
>> 

>This changes the meaning significantly.  Is that right?

Not really. The second half of the sentence only applies IF the agent
maintains a "cutoff interval" (in practice, they all do, but if they don't
then neither wording places any REQUIREMENT on them).

But I think Frank's suggested wording covers it well.

To: [email-masked]
Xref: clerew local.usefor:24681
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: #1416 Injection-Date - Summary of options
Date: Mon, 21 May 2007 15:51:42 GMT
Lines: 30
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>Okay, my proposed draft text now has:

>   ....  It SHOULD similarly reject any article
>        whose Date header field is too far in the past, since not all
>        news servers support Injection-Date and only the injecting agent
>        can provide a useful error message to the posting agent.  In
>        either case, this interval SHOULD NOT be any shorter than 72
>        hours into the past.

>The third sentence is the one that remains controversial.  I think we're
>fairly comfortable with the rest of the paragraph.

The wording is now much weaker than in the present draft. It is more of a
recommendation to "think of a number greater than 72 hours", which does
not preclude thinking of "infinity". So I still marginally do not like it,
but will not press the point if others are happy.

To: [email-masked]
From: [email-masked]
Subject: Commit in docs/usefor (usepro.xml)
Date: Sat, 19 May 2007 20:29:10 -0700 (PDT)
Sender: [email-masked]

    Date: Saturday, May 19, 2007 @ 20:29:08
  Author: eagle
Revision: 2986

Clarify the wording of the exception for control message propagation by
relaying agents.

Modified:
  docs/usefor/usepro.xml

Modified: docs/usefor/usepro.xml
===================================================================
--- docs/usefor/usepro.xml	2007-05-20 03:18:07 UTC (rev 2985)
+++ docs/usefor/usepro.xml	2007-05-20 03:29:08 UTC (rev 2986)
@@ -783,10 +783,9 @@
         header field (if present).  Exceptionally, control messages
         creating or removing newsgroups (newgroup or rmgroup control
         messages, for example) SHOULD be relayed if the affected group
-        appears in its Newsgroups header field and and the sending agent
-        and receiving relaying agents are both configured to relay a
-        newsgroup of that name (whether or not such a newsgroup
-        exists).</t>
+        appears in its Newsgroups header field and both the sending and
+        receiving relaying agents are configured to relay a newsgroup of
+        that name (whether or not such a newsgroup exists).</t>
 
         <t>In order to avoid unnecessary relaying attempts, an article
         SHOULD NOT be relayed if the &lt;path-identity> of the receiving

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Minor Change: Wording tweak in Duties of a Relaying Agent
Organization: The Eyrie
Date: Sat, 19 May 2007 20:28:51 -0700
Sender: [email-masked]

Julien ÉLIE suggested the following change, which seems a clear
improvement.  The current text had various grammatical problems.

Index: usepro.xml
===================================================================
--- usepro.xml	(revision 2983)
+++ usepro.xml	(working copy)
@@ -783,10 +783,9 @@
         header field (if present).  Exceptionally, control messages
         creating or removing newsgroups (newgroup or rmgroup control
         messages, for example) SHOULD be relayed if the affected group
-        appears in its Newsgroups header field and and the sending agent
-        and receiving relaying agents are both configured to relay a
-        newsgroup of that name (whether or not such a newsgroup
-        exists).</t>
+        appears in its Newsgroups header field and both the sending and
+        receiving relaying agents are configured to relay a newsgroup of
+        that name (whether or not such a newsgroup exists).</t>
 
         <t>In order to avoid unnecessary relaying attempts, an article
         SHOULD NOT be relayed if the &lt;path-identity> of the receiving

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: #1416 Injection-Date: current wording proposal (corrected)
Organization: The Eyrie
Date: Sat, 19 May 2007 17:03:08 -0700
Sender: [email-masked]

Frank Ellermann <[email-masked]> writes:
> Russ Allbery wrote:

>>>> It MUST reject any article that has already been successfully
>>>> sent to it, which may include rejecting articles whose dates fall
>>>> outside a cutoff interval as described in Section 3.3.
 
>> Suggestions for wording that captures this?

> s/which may include/and this typically requires/ or similar.

That sounds pretty good to me.

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: #1416 Injection-Date: current wording proposal (corrected)
Date:  Sun, 20 May 2007 00:51:21 +0200
Organization:  <URL:http://purl.net/xyzzy>
Lines: 12
Sender: [email-masked]

Russ Allbery wrote:

>>> It MUST reject any article that has already been successfully
>>> sent to it, which may include rejecting articles whose dates fall
>>> outside a cutoff interval as described in Section 3.3.
 
> Suggestions for wording that captures this?

s/which may include/and this typically requires/ or similar.

Frank

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: #1416 Injection-Date: current wording proposal (corrected)
Organization: The Eyrie
Date: Fri, 18 May 2007 20:47:17 -0700
Sender: [email-masked]

Forrest J Cavalier <[email-masked]> writes:
> Russ Allbery wrote:

>>>> +            <t>It MUST reject any article that has already been
>>> 
>>>               successfully sent to it, or alternatively where its date
>>>               falls outwith any cutoff interval as described in <xref
>>>               history>.

>> I now have:
>> 
>>    3.  It MUST reject any article that has already been successfully
>>        sent to it, which may include rejecting articles whose dates fall
>>        outside a cutoff interval as described in Section 3.3.

> This changes the meaning significantly.  Is that right?

Probably not.  I'm failing to find the right wording here.

News servers MUST reject articles that have already been successfully sent
to them.  They're given an allowance to reject messages that have not
already been successfully sent to them as part of this if that's needed
for their algorithm (which doesn't change the protocol, since they can
always reject more articles for any reason at all).  *If* the server uses
the schemes in 3.3, *then* they must reject articles whose dates lie
outside the cutoff interval.

Suggestions for wording that captures this?

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: ISSUE: 4.2. Moderation flag in checkgroups
Organization: The Eyrie
Date: Fri, 18 May 2007 20:10:06 -0700
Sender: [email-masked]

Forrest J Cavalier <[email-masked]> writes:
> Russ Allbery wrote:

>> Okay, the current proposal is to change this to:
>> 
>>          newsgroups-line     = newsgroup-name
>>                                   [ 1*HTAB newsgroup-description ]
>>                                   [ *WSP moderation-flag ]
>>          moderation-flag     = SP "(" %x28.4D.6F.64.65.72.61.74.65.64 ")"
>>                                   ; SPACE + case sensitive "(Moderated)"
>> 
>> If there are any objections, speak up now.  Unless there are objections by
>> Tuesday, May 22nd, I'm going to declare consensus and make this change.

> Please fix the obvious typo.

Bleh, sorry.

         newsgroups-line     = newsgroup-name
                                  [ 1*HTAB newsgroup-description ]
                                  [ *WSP moderation-flag ]
         moderation-flag     = SP "(" %4D.6F.64.65.72.61.74.65.64 ")"
                                  ; SPACE + case sensitive "(Moderated)"

is the correct text.  Thank you for the catch.

Date: Fri, 18 May 2007 23:08:08 -0400
From: "Forrest J. Cavalier III" <[email-masked]>
To: [email-masked]
Subject: Re: #1416 Injection-Date: current wording proposal (corrected)
Sender: [email-masked]

Russ Allbery wrote:
>>>+            <t>It MUST reject any article that has already been
>>
>>               successfully sent to it, or alternatively where its date
>>               falls outwith any cutoff interval as described in <xref
>>               history>.
> 
> 
> I now have:
> 
>    3.  It MUST reject any article that has already been successfully
>        sent to it, which may include rejecting articles whose dates fall
>        outside a cutoff interval as described in Section 3.3.
> 

This changes the meaning significantly.  Is that right?

Date: Fri, 18 May 2007 23:00:59 -0400
From: "Forrest J. Cavalier III" <[email-masked]>
To: [email-masked]
Subject: Re: ISSUE: 4.2. Moderation flag in checkgroups
Sender: [email-masked]

Russ Allbery wrote:

> Okay, the current proposal is to change this to:
> 
>          newsgroups-line     = newsgroup-name
>                                   [ 1*HTAB newsgroup-description ]
>                                   [ *WSP moderation-flag ]
>          moderation-flag     = SP "(" %x28.4D.6F.64.65.72.61.74.65.64 ")"
>                                   ; SPACE + case sensitive "(Moderated)"
> 
> If there are any objections, speak up now.  Unless there are objections by
> Tuesday, May 22nd, I'm going to declare consensus and make this change.

Please fix the obvious typo.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: ISSUE #1414: Call for consensus
Organization: The Eyrie
Date: Fri, 18 May 2007 19:20:44 -0700
Sender: [email-masked]

 

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: #1416 Injection-Date: current wording proposal (corrected)
Organization: The Eyrie
Date: Fri, 18 May 2007 18:09:36 -0700
Sender: [email-masked]

Harald Alvestrand <[email-masked]> writes:

> There are 3 cases:

> - Message is singly injected. Injection-Date: does no harm.
> - Message is multiply injected, Injection-Date: on all copies are quite
> close to each other, because they all got injected in rapid
> succession. Injection-Date: does no harm.
> - Message is multiply injected, and some of the injections got
> significantly delayed.

Right, the problems only potentially happen in the third case, but that
third case includes many serial injection cases.

> I think most user agents will add Date: and Message-ID: by
> default. (Anyone have data on this, rather than supposition?)

Charles had some data, I believe.  If I remember it correctly, "most" is
iffy, but many do.

> So the net result of catering for the minority third case is that *most*
> messages on the Net will be sent without an Injection-Date.

Correct, until posting agents are updated to generate it.  This is the way
that I think it should work, since all the benefits of Injection-Date are
born by the posting agents, not by the injecting agents (which don't care
at all).

> If so, what's the point of claiming that we're "mandating" it?

We're mandating checking it instead of Date so that reading agents can
start using the feature.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: #1416 Injection-Date: current wording proposal (corrected)
Organization: The Eyrie
Date: Fri, 18 May 2007 18:06:54 -0700
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:

> OK, I have finally got around to reading the whole of this. Many of my
> points are minor textual niggles, but some are quite substantive.

Changes snipped without remark have been accepted and made in my draft
text.

>> +            <t>Agents that enforce such a cutoff MAY then drop records of
>> +            articles that had dates older than the cutoff from their
>> +            history databases.  If such an article were offered to the
>> +            agent again, it would be rejected due to the cutoff date, so
>> +            the history record is no longer required to suppress the
>> +            duplicate.</t>
>> +
>> +            <t>As an optimization for easier history database
>> +            manipulation, agents MAY instead drop history records written
>> +            longer ago than the cutoff interval plus one day.  If this
>> +            retention mechanism is used, the history retention period MUST
>> +            be longer than the cutoff interval to allow for articles dated
>> +            in the future unless the agent rejects all articles dated in
>> +            the future.  One day is the maximum allowed error into the
>> +            future for article dates, so it is a convenient and safe
>> +            extension for the retention interval.</t>

> It takes a lot of VERY careful reading to spot the curicial diference
> between those two cases (and Frank seems to have the same problem). I
> therefore suggest the following alternative for the 2nd one:

>               Alternatively, agents MAY drop history records according to
> 	      the date when the article was received, in which case they
> 	      MUST retain them for 24 hours longer than the cutoff
> 	      interval, since the date that accompanies the article is
> 	      permitted to be up to 24 hours later than its actual time of
> 	      injection (see <xref duties-injecting>).

> {BTW, I prefer using 24 hours rather than 1 day for such short time
> intervals, especially since the discussion under duties-injecting speaks
> of even shorter intervals}

I couldn't resist wordsmithing this a bit, but I think I retained the
intent.  I now have:

   o  Alternatively, agents MAY drop history records according to the
      date when the article was first seen by that agent rather than the
      date of the article.  In this case, the history retention interval
      MUST be at least 24 hours longer than the cutoff interval to allow
      for articles dated in the future.  This interval matches the
      allowable error in the date of the article (see Section 3.5).

>> +        <t>This is just one implementation strategy for article history,
>              ^^^^^^^^^^^^^^^^                ^^^^^^^^
>              These are just two             strategies
>> +        albeit the most common one.  Relaying and serving agents are not
>                                  ^^^
>                                  ones
>> +        required to use this strategy, only to meet the requirement of not
>> +        accepting an article more than once. ....

> However, the next bit of the sermon is a bit overdone, and the use of
> "default" seems not quite right:

>> +        ......... However, implementors of
>> +        general-purpose Netnews relaying and serving agents who do not
>> +        have extensive experience with Netnews and the subtle effects of
>> +        its flood-fill algorithm are encouraged to use the above algorithm
>> +        by default.</t>

> So I would suggest:

>           ......... However, implementors are advised normally to use
> 	  these strategies, especially if they do not have extensive
> 	  experience with Netnews and the subtle effects of its flood-fill
> 	  algorithm.

I now have:

   These are just two implementation strategies for article history,
   albeit the most common ones.  Relaying and serving agents are not
   required to use these strategies, only to meet the requirement of not
   accepting an article more than once.  However, these strategies are
   safe and widely deployed and implementors are encouraged to use one
   of them, especially if they not have extensive experience with
   Netnews and the subtle effects of its flood-fill algorithm.

going with explaining why this is a good idea rather than just emphasizing
it.

>> @@ -478,48 +565,70 @@
>> 
>>           <t>A proto-article has the same format as a normal article
>> +          except that the Injection-Info and Xref header fields MUST NOT
>> +          be present; the Path header field MUST NOT contain a "POSTED"
>> +          &lt;diag-keyword>; ........

> I disagree with forbidding POSTED (already raised in a separate thread),
> and am dubious about Xref ("SHOULD" would be quite strong enough).

I'm not sure we've reached a conclusion on those other threads yet.  So
far, I've not made any changes here.

>>           <t>If a posting agent intends to offer the same proto-article to
>> +          multiple injecting agents, the header fields Message-ID, Date,
>> +          and Injection-Date MUST be present and identical in all copies
>> +          of the proto-article.  See <xref target="multi-injection" />.</t>
>                ^^^^^^^^^^^^^^^^^
>                   it

This was intentional.  It reads awkwardly, but it's clearer.  "It" is
somewhat ambiguous and I kept stumbling over thinking it referred to the
posting agent for a moment.

>> +          <t>Whenever possible, multiple injection SHOULD be done by
>> +          offering the same proto-article to multiple injecting agents.
>> +          The posting agent MUST supply the Message-ID, Date, and
>> +          Injection-Date header fields, and the proto-article offered to
>                                                   ^^^^^^^^^^^^^
>                                                proto-articles as
>> +          each injecting agent MUST be identical.</t>

I'm intentionally not pluralizing proto-article, since the whole point is
that there's only one of them, sent to multiple locations.  Adding "as"
does make the sentence read better to me, though, so I did that.

>> +          <t>In some cases, offering the same proto-article to all
>> +          injecting agents may not be possible (such as when transferring,
>                                                                ^^^^^^^^^^^^
>                                                                gatewaying
>> +          after the fact, articles found on one Netnews network to
>> +          another, unconnected one). .........
>             ^^^^^^^^^^^^^^^^^^^^^^^^
>             another, supposedly unconnected, one

Okay.

> However, this does not really convey the scenario as it is anticipated to
> occur. In essence, the first network is typically small and private and
> the second network is usually large, and probably Usenet. And one does not
> usually just "find" articles there - one usually placed them there
> oneself.

> So how about:

>             ....... (such as when gatewaying, after the fact, articles
>             from a small private network, supposedly with no other
> 	    connections, to a larger network, such as Usenet). ......

Well, that's one of the cases where offering the same proto-article to all
injecting agents *is* possible, normally, so I'm not sure it's a great
example.  That's the use case where I really wish people would just use
normal multiple injection.

>> +          ...   In this case, the posting agent MUST
>> +          convert the article back into a proto-article before passing it
>> +          to another injecting agent, but it MUST retain unmodified the
>> +          Message-ID, Date, and Injection-Date header fields.  It MUST NOT
>> +          add an Injection-Date header field if it is missing from the
>> +          existing article.  It MUST remove any Xref header field and
>                                   ^^^^                              ^^^
>                                MAY/SHOULD                           but
>> +          either rename or remove any Injection-Info header field and
>             ^
>            MUST
>> +          other trace fields.</t>
>                               ^
>                    (such as NNTP-Posting-Host and XTrace)

> There is also the question of what to do about Path. My preference would
> be for "MAY/SHOULD retain"

As discussed above, I haven't made any changes here.

>> @@ -644,23 +753,24 @@
>> 
>>             <t>It MUST reject any proto-article that does not have the
>>             proper mandatory header fields for a proto-article; that has
>> +            Injection-Info or Xref header fields; that has a Path header
>               ^              ^^^^^^^^^^^^^^^^^^^^^  ^^^^^^^^^^^^^^^^^^^^^^
>              any                   field
>> +            field containing the "POSTED" &lt;diag-keyword>; or that is
>               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Likewise here.

>> +            not syntactically valid as defined by <xref target="USEFOR"
>> +            />.  It SHOULD reject any proto-article which contains a
>> +            header field deprecated for Netnews.  It MAY reject any
>                                                  ^
>                                       (see e.g. <xref RFC2298>)

RFC 2298 is obsolete, so I don't really want to generate even an
informative reference to it, and the superseding RFC (RFC 3798) makes no
mention of Netnews, so I'm not sure this is a great example or
particularly helpful for the reader.

>> +            proto-article that contains trace header fields indicating
>                                                              ^
>                                         (e.g. NNTP-Posting-Host or Xtrace)

Added just NNTP-Posting-Host, since USEFOR already mentions and explicitly
obsoletes it.

>> +            that it was already injected by an injecting agent that did
>> +            not add Injection-Info or Injection-Date.</t>
>> 
>>             <t>It SHOULD reject any article whose Date header field is
>>             more than 24 hours into the future (and MAY use a margin less
>>             than 24 hours).  It SHOULD reject any article whose Date
>> +            header is too far into the past (older than the cutoff
>                     ^
>                   field
>> +            interval of a relaying agent the injecting agent is using, for
>> +            example), .......

> Would that apply to Injection-Date, if any? Or would you allow a stale
> Date if there was a plausible Injection-Date?

Covered in another thread.

>> +            <t>It MUST reject any article that has already been
>> +            successfully sent to it or that is dated older than its cutoff
>> +            date, as described in <xref target="history" />.</t>

> No, that won't work. You have not actually defined the term "cutoff date".
> But, in any case even providing a "cutoff interval" is not a REQUIRTEMENT.

> How about:

>> +            <t>It MUST reject any article that has already been
>                successfully sent to it, or alternatively where its date
>                falls outwith any cutoff interval as described in <xref
>                history>.

I now have:

   3.  It MUST reject any article that has already been successfully
       sent to it, which may include rejecting articles whose dates fall
       outside a cutoff interval as described in Section 3.3.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: #1416 Injection-Date - Summary of options
Organization: The Eyrie
Date: Fri, 18 May 2007 17:33:00 -0700
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:

> We are agreed (I think) that Dates up to 24 hours in the future do
> little harm, and injecting agents are supposed to reject anything
> longer.  Clearly, they SHOULD reject Injection-Dates over 24 hours into
> the future (I think I already suggested that needed to be added to
> Russ's text).

I must have missed that.  Indeed.

Okay, my proposed draft text now has:

   3.   It SHOULD reject any article whose Injection-Date or Date header
        field is more than 24 hours into the future (and MAY use a
        margin less than 24 hours).  It SHOULD reject any article whose
        Injection-Date header field is too far in the past (older than
        the cutoff interval of a relaying agent the injecting agent is
        using, for example).  It SHOULD similarly reject any article
        whose Date header field is too far in the past, since not all
        news servers support Injection-Date and only the injecting agent
        can provide a useful error message to the posting agent.  In
        either case, this interval SHOULD NOT be any shorter than 72
        hours into the past.

to be very clear about what header field we're talking about here.  I've
explicitly mentioned rejecting based on a stale Injection-Date because
injecting agents that accept articles and then have their upstream
relaying agent reject them may have lost the opportunity to return a
meaningful error message and injecting agents that just proxy to the
relaying agent and return its error message satisfy this requirement.

The third sentence is the one that remains controversial.  I think we're
fairly comfortable with the rest of the paragraph.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: #1416 Injection-Date - Summary of options
Organization: The Eyrie
Date: Fri, 18 May 2007 17:25:43 -0700
Sender: [email-masked]

Harald Alvestrand <[email-masked]> writes:

> one (contrived) example would be someone who wishes to make a posting
> that is visible at one site, but not at another site, where he knows
> that one site discards based on Date: and the other side discards based
> on Injection-Date:

> He would then put either Date: or Injection-Date: a week or so in the
> past, depending on which site he wishes his message to be "invisible"
> at.

> If the WG thinks that this isn't very worrying (I can't say I find it
> very worrying either), it's fine.

Ah, I see what you're saying.  Yes, I don't find that particularly
worrying.  I could see mentioning it in the Security Considerations
section, though, since a few similar things are already mentioned.  How do
you feel about that?

> (minor guard against this particular kind of mischief: Insist that
> Injection-Date: is no earlier than Date:, and check BOTH of them against
> the 72-or-so-hour limit....)

In the current draft, we are checking the Date header field, so that would
limit this (although that isn't the main reason why it's being done).
This is one of the things that Charles feels should be changed.

To: [email-masked]
From: [email-masked]
Subject: Commit in docs/usefor (usepro.xml)
Date: Fri, 18 May 2007 17:11:16 -0700 (PDT)
Sender: [email-masked]

    Date: Friday, May 18, 2007 @ 17:11:15
  Author: eagle
Revision: 2983

Add a missing close parenthesis to the explanation of the Path example.

Modified:
  docs/usefor/usepro.xml

Modified: docs/usefor/usepro.xml
===================================================================
--- docs/usefor/usepro.xml	2007-05-18 02:15:53 UTC (rev 2982)
+++ docs/usefor/usepro.xml	2007-05-19 00:11:15 UTC (rev 2983)
@@ -431,7 +431,7 @@
           [2001:DB8:0:0:8:800:200C:417A].  (This is not to say that
           bar.isp.example was not a correct &lt;path-identity> for that
           source but simply that that identity did not match the
-          expectations of foo-news.</t>
+          expectations of foo-news.)</t>
 
           <t>foo-news then passed the article to foo.isp.example, which
           declined to validate its &lt;path-identity> and instead appended

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Minor Change: Missing paren in path explanation
Organization: The Eyrie
Date: Fri, 18 May 2007 17:10:52 -0700
Sender: [email-masked]

Spotted by Julien ÉLIE.

           [2001:DB8:0:0:8:800:200C:417A].  (This is not to say that
           bar.isp.example was not a correct &lt;path-identity> for that
           source but simply that that identity did not match the
-          expectations of foo-news.</t>
+          expectations of foo-news.)</t>
 
           <t>foo-news then passed the article to foo.isp.example, which
           declined to validate its &lt;path-identity> and instead appended

From: Russ Allbery <[email-masked]>
To: Usefor Mailing List <[email-masked]>
Subject: Re: ISSUE: 4.2. Moderation flag in checkgroups
Organization: The Eyrie
Date: Fri, 18 May 2007 17:03:23 -0700
Sender: [email-masked]

Forrest J Cavalier <[email-masked]> writes:

> I like the %x20.  Servers are not implemented as BNF parsers.  They look
> for %x28.4D.6F.64.65.72.61.74.65.64.29, not EBCDIC.  They need the
> preceding %x20.

Yeah, but we're already using SP in the "For your newsgroups file:" and no
one had noticed before.  And servers actually are looking for
strcmp(" (Moderated)"), which isn't any of those things.  You lose
something in translation to ABNF and the leading space is really easy to
miss in the mess of hex digits.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: ISSUE: 4.2. Moderation flag in checkgroups
Organization: The Eyrie
Date: Fri, 18 May 2007 17:01:58 -0700
Sender: [email-masked]

Frank Ellermann <[email-masked]> writes:
> Harald Alvestrand wrote:

>> I'd prefer to say
>>   moderation-flag = SP %x28.4D.6F.64.65.72.61.74.65.64.29
>>                    ; SPACE + case sensitive "(Moderated)"

> Yes, that's clearer.  We could also de-obscure 0x28 and 0x29,
> and use hex. only for case-sensitive stuff like "Moderated"
> or "poster":

>     moderation-flag = SP "(" %x4D.6F.64.65.72.61.74.65.64 ")"
>                      ; SPACE + case sensitive "(Moderated)"

I like this version best.

Okay, the current proposal is to change this to:

         newsgroups-line     = newsgroup-name
                                  [ 1*HTAB newsgroup-description ]
                                  [ *WSP moderation-flag ]
         moderation-flag     = SP "(" %x28.4D.6F.64.65.72.61.74.65.64 ")"
                                  ; SPACE + case sensitive "(Moderated)"

If there are any objections, speak up now.  Unless there are objections by
Tuesday, May 22nd, I'm going to declare consensus and make this change.

I'm not following the previous procedure of requiring an open ticket and a
chair resolution for this, but it seemed like people didn't feel that was
necessary for this issue.  If anyone does, say so, and I'll go back to
requiring that.

To: [email-masked]
Xref: clerew local.usefor:24662
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: #1416: Reinjection and Injection-Date proposed text
Date: Fri, 4 May 2007 11:34:10 GMT
Lines: 50
Sender: [email-masked]

In <[email-masked]> "Forrest J. Cavalier III" <[email-masked]> writes:

>Russ wrote:
> >  I don't think it makes as much sense to end up in a
>>mixed situation where Injection-Date is a MUST for injecting agents but
>>only a should for multi-injecting posting agents, or vice versa.

>I don't understand this.

>There are lots of cases where SHOULD/MUST are conditional (apply to
>certain situations.)

Yes. All Russ is asking for here is that two comparable situations should
bother say either MUST or both not say it. There are still other cases
where SHOULD would be strong enough.

>It's nice that Russ is spending so much time hammering out language
>and details.  But this really should be someone else's burden to get
>a working solution.  It is obvious that whoever conceived Injection-Date,
>did not understand the implications, and got it wrong, and is not
>working here to get it right.

The people who conceived injection date are the same people who are
discussing it now. All that has changed is that we have seen some benefit
(especially if option IR wins) in allowing Posting Agents to insert it,
and are working out the details.

>I'd still recommend:
>   Contrary to RFC-2822, the contents of the Date header field in an
>   article MUST be close to the time of injection, and MUST be identical
>   in all copies of an article.  A "Composition-Date:" header field MAY
>   be used to indicate the time an article was composed.  Servers MAY
>   choose to act on the contents of Composition-Date.

But Canute cannot stop the tide. User agents are already routinely
inserting the Date at composition time, and you will have a hard struggle
to stop them.

To: [email-masked]
Xref: clerew local.usefor:24663
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: #1416 Injection-Date: current wording proposal (corrected)
Date: Fri, 4 May 2007 11:50:35 GMT
Lines: 63
Sender: [email-masked]

In <[email-masked]> Harald Alvestrand <[email-masked]> writes:

>>
>> @@ -704,8 +814,14 @@
>>              the source of the article and possibly other trace information
>>              as described in Section 3.2.8 of <xref target="USEFOR" />.</t>
>>  
>> -            <t>The injecting agent MUST then add an Injection-Date header
>> -            field containing the current date and time.</t>
>> +            <t>If the proto-article already had an Injection-Date header
>> +            field, it MUST NOT be modified or replaced.  If the
>> +            proto-article had both a Message-ID header field and a Date
>> +            header field, an Injection-Date header field MUST NOT be
>> +            added, since the proto-article may have been multiply injected
>> +            by a posting agent that predates this standard.  Otherwise,
>> +            the injecting agent MUST add an Injection-Date header field
>> +            containing the current date and time.</t>
>>  
>>              <t>Finally, the injecting agent forwards the article to one or
>>              more relaying agents, and the injection process is
>>   
>I (still) am not convinced of the need for not adding an Injection-Date: 
>header when a message-ID and Date is present, but Injection-Date is not.

Then you are essentially supporting option IC rather than option IR.

>There are 3 cases:

..........

>- Message is multiply injected, and some of the injections got 
>significantly delayed.

That, in combination with legacy user agents which do not insert an
Injection-Date when multiply injecting, is what may cause a later relaying
agent to re-accept an article it has already accepted and expired once
before (which would not have happened under the present arrangements).

That is Russ's argument for Option IR. I am not convinced, on the grounds
that it will be a very rare occurrence and causes little harm (someone
finds himself reading an article he had already read a week earlier).

>I think most user agents will add Date: and Message-ID: by default. 
>(Anyone have data on this, rather than supposition?)

I think most will add Date, but fewer will add Message-ID.

>So the net result of catering for the minority third case is that *most* 
>messages on the Net will be sent without an Injection-Date.

Indeed, which will mean a slower adoption of it, but also an anomaly which
will remain long after it eventually is in universal use.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: #1416 Injection-Date: current wording proposal (corrected)
Organization: The Eyrie
Date: Thu, 03 May 2007 10:18:22 -0700
Sender: [email-masked]

Frank Ellermann <[email-masked]> writes:
> Russ Allbery wrote:
 
> [a definition of future]
>> Here, I do want to use 24 hours.  The definition of a "day" may be 25
>> hours or 23 hours depending on daylight saving time transitions, and I
>> don't think we want the ambiguity.  It doesn't really matter once we're
>> talking about longer periods, but it does matter at the interval of a
>> day.

> If your implementations get 24 hours right for the two interesting days
> in a year I withdraw my objection... :-) I didn't know that I'd need the
> complete timezone stuff to implement a news server.

For INN, I wrote my own RFC 2822 date parser that converts to seconds
since epoch and deals with various non-RFC-2822 dates that are seen in the
wild, including ones that give a time zone without an offset.

/* Additional non-numeric time zones supported because the old parsedate
   parser supported them.  These aren't legal in RFC 2822, but are supported
   in lax mode. */
static const struct {
    const char name[5];
    long offset;
} OBS_ZONE_OFFSET[] = {
    { "UTC",    0 },                 /* Universal Coordinated */
    { "CUT",    0 },                 /* Coordinated Universal */
    { "WET",    0 },                 /* Western European */
    { "BST",    1 * 60 * 60 },       /* British Summer */
    { "NDT",  (-2 * 60 + 30) * 60 }, /* Newfoundland Daylight */
    { "NST",  (-3 * 60 + 30) * 60 }, /* Newfoundland Standard */
    { "ADT",   -3 * 60 * 60 },       /* Atlantic Daylight */
    { "AST",   -4 * 60 * 60 },       /* Atlantic Standard */
    { "YDT",   -8 * 60 * 60 },       /* Yukon Daylight */
    { "YST",   -9 * 60 * 60 },       /* Yukon Standard */
    { "AKDT",  -8 * 60 * 60 },       /* Alaska Daylight */
    { "AKST",  -9 * 60 * 60 },       /* Alaska Standard */
    { "HADT",  -9 * 60 * 60 },       /* Hawaii-Aleutian Daylight */
    { "HAST", -10 * 60 * 60 },       /* Hawaii-Aleutian Standard */
    { "HST",  -10 * 60 * 60 },       /* Hawaii Standard */
    { "CES",    2 * 60 * 60 },       /* Central European Summer */
    { "CEST",   2 * 60 * 60 },       /* Central European Summer */
    { "MEZ",    1 * 60 * 60 },       /* Middle European */
    { "MEZT",   2 * 60 * 60 },       /* Middle European Summer */
    { "CET",    1 * 60 * 60 },       /* Central European */
    { "MET",    1 * 60 * 60 },       /* Middle European */
    { "EET",    2 * 60 * 60 },       /* Eastern European */
    { "MSK",    3 * 60 * 60 },       /* Moscow Winter */
    { "MSD",    4 * 60 * 60 },       /* Moscow Summer */
    { "WAST",   8 * 60 * 60 },       /* Western Australian Standard */
    { "WADT",   9 * 60 * 60 },       /* Western Australian Daylight */
    { "HKT",    8 * 60 * 60 },       /* Hong Kong */
    { "CCT",    8 * 60 * 60 },       /* China Coast */
    { "JST",    9 * 60 * 60 },       /* Japan Standard */
    { "KST",    9 * 60 * 60 },       /* Korean Standard */
    { "KDT",    9 * 60 * 60 },       /* Korean Daylight (no change?) */
    { "CAST",  (9 * 60 + 30) * 60 }, /* Central Australian Standard */
    { "CADT", (10 * 60 + 30) * 60 }, /* Central Australian Daylight */
    { "EAST",  10 * 60 * 60 },       /* Eastern Australian Standard */
    { "EADT",  11 * 60 * 60 },       /* Eastern Australian Daylight */
    { "NZST",  12 * 60 * 60 },       /* New Zealand Standard */
    { "NZST",  13 * 60 * 60 },       /* New Zealand Daylight */
};

Whee.

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: #1416 Injection-Date: current wording proposal (corrected)
Date:  Thu, 03 May 2007 17:28:33 +0200
Organization:  <URL:http://purl.net/xyzzy>
Lines: 15
Sender: [email-masked]

Russ Allbery wrote:
 
 [a definition of future]
> Here, I do want to use 24 hours.  The definition of a "day" may be 25
> hours or 23 hours depending on daylight saving time transitions, and I
> don't think we want the ambiguity.  It doesn't really matter once we're
> talking about longer periods, but it does matter at the interval of a
> day.

If your implementations get 24 hours right for the two interesting days
in a year I withdraw my objection... :-)  I didn't know that I'd need
the complete timezone stuff to implement a news server.

Frank

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: ISSUE: 4.2. Moderation flag in checkgroups
Date:  Thu, 03 May 2007 17:12:22 +0200
Organization:  <URL:http://purl.net/xyzzy>
Lines: 14
Sender: [email-masked]

Harald Alvestrand wrote:

> I'd prefer to say
>   moderation-flag = SP %x28.4D.6F.64.65.72.61.74.65.64.29
>                    ; SPACE + case sensitive "(Moderated)"

Yes, that's clearer.  We could also de-obscure 0x28 and 0x29,
and use hex. only for case-sensitive stuff like "Moderated"
or "poster":

    moderation-flag = SP "(" %x4D.6F.64.65.72.61.74.65.64 ")"
                     ; SPACE + case sensitive "(Moderated)"
Frank

Date: Thu, 03 May 2007 08:10:21 -0400
From: "Forrest J. Cavalier III" <[email-masked]>
To: Usefor Mailing List <[email-masked]>
Subject: Re: ISSUE: 4.2. Moderation flag in checkgroups
Sender: [email-masked]

Harald Alvestrand wrote:

> 
> I'd prefer to say
> 
>   moderation-flag = SP %x28.4D.6F.64.65.72.61.74.65.64.29
>                    ; SPACE + case sensitive "(Moderated)"
> 
> since it's the only place in all our grammars (that I can remember 
> offhand) that we specify a SPACE using a hex value.

I like the %x20.  Servers are not implemented as BNF parsers.
They look for %x28.4D.6F.64.65.72.61.74.65.64.29, not EBCDIC.  They need
the preceding %x20.

I do like the extra comment noting there is a SPACE there because it
gives attention.  How about...

    moderation-flag = %x20.28.4D.6F.64.65.72.61.74.65.64.29
                     ; SPACE + case sensitive "(Moderated)"

Date: Thu, 03 May 2007 08:04:50 -0400
From: "Forrest J. Cavalier III" <[email-masked]>
To: [email-masked]
Subject: Re: #1416: Reinjection and Injection-Date proposed text
Sender: [email-masked]

Russ wrote:
 >  I don't think it makes as much sense to end up in a
>mixed situation where Injection-Date is a MUST for injecting agents but
>only a should for multi-injecting posting agents, or vice versa.

I don't understand this.

There are lots of cases where SHOULD/MUST are conditional (apply to
certain situations.)

Our effort is to agree on language to optimize what happens with delayed 
injections, which happens rarely.

So a "mixed situation" seems OK, especially if the burden is shouldered
by those who intend to reap the benefit.

Ha!  I see a large irony with that statement, because the underlying
principle of fairness is violated within this WG all the time.

It's nice that Russ is spending so much time hammering out language
and details.  But this really should be someone else's burden to get
a working solution.  It is obvious that whoever conceived Injection-Date,
did not understand the implications, and got it wrong, and is not
working here to get it right.

USEFOR/USEPRO has been delayed countless months/years because someone
suggested a half-baked idea, and then expects the WG to figure out
a fully-baked solution.  I don't see how the charter indicates that
was supposed to happen.  But here we are again.

Is this feature really so important that we need all this discussion,
adding paragraphs and paragraphs about the internal working mechanisms
of history files?

Based on all the discussion, isn't it obvious that trying to make 
"Injection-Date" work is a big problem?  Doesn't that indicate a
half-baked idea, not ready for standard track?

I'd still recommend:
   Contrary to RFC-2822, the contents of the Date header field in an
   article MUST be close to the time of injection, and MUST be identical
   in all copies of an article.  A "Composition-Date:" header field MAY
   be used to indicate the time an article was composed.  Servers MAY
   choose to act on the contents of Composition-Date.

Date: Thu, 03 May 2007 13:35:45 +0200
From: Harald Alvestrand <[email-masked]>
To: Russ Allbery <[email-masked]>
Cc: [email-masked]
Subject: Re: ISSUE: 4.2. Moderation flag in checkgroups
Sender: [email-masked]

Russ Allbery wrote:
> Julien ÉLIE pointed out in private e-mail a minor problem with how the
> moderation flag is specified in section 4.2.  Currently, we have:
>
>          newsgroups-line     = newsgroup-name
>                                   [ 1*HTAB newsgroup-description ]
>                                   [ 1*WSP moderation-flag ]
>          newsgroup-description
>                              = eightbit-utext *( *WSP eightbit-utext )
>          moderation-flag     = %x28.4D.6F.64.65.72.61.74.65.64.29
>                                   ; case sensitive "(Moderated)"
>          eightbit-utext      = utext / %d127-255
>
> This allows a tab before "(Moderated)" instead of a space.  I'm not sure
> that existing software supports that, it doesn't match the informal
> description on ftp.isc.org, and it contradicts the text in the following
> paragraph:
>
>    The <newsgroup-description> MUST NOT contain any occurrence of the
>    string "(Moderated)" within it.  Moderated newsgroups MUST be marked
>    by appending the case sensitive text " (Moderated)" at the end.
>
> I propose changing the newsgroups-line production to instead read:
>
>          newsgroups-line     = newsgroup-name
>                                   [ 1*HTAB newsgroup-description ]
>                                   [ *WSP moderation-flag ]
>
> and moderation-flag to read:
>
>          moderation-flag     = %x20.x28.4D.6F.64.65.72.61.74.65.64.29
>                                   ; case sensitive " (Moderated)"
>
> which makes it clearer that the space is part of the flag.  I had
> considered just changing the 1*WSP to *WSP SP, but while writing this
> message, I decided I liked the above approach better.
>
>   
I'd prefer to say

  moderation-flag = SP %x28.4D.6F.64.65.72.61.74.65.64.29
                   ; SPACE + case sensitive "(Moderated)"

since it's the only place in all our grammars (that I can remember 
offhand) that we specify a SPACE using a hex value.

Otherwise, no problem.

                Harald

To: [email-masked]
Xref: clerew local.usefor:24653
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: #1416 Injection-Date - Summary of options
Date: Thu, 3 May 2007 10:46:52 GMT
Lines: 55
Sender: [email-masked]

In <[email-masked]> Harald Alvestrand <[email-masked]> writes:

>Charles Lindsey wrote:
>>
>>> After writing the text, I think it's weird, but it's not quite as bad as I
>>> thought it was going to be.  It does have the merit of closely mirroring
>>> how Date works today, and if we're going to change something in this area,
>>> sticking as close to what we know works as possible seems like a good idea
>>> to me.
>>>     
>>
>> +1
>>
>> It may well be that most 'normal' posting agents leave it to the injecting
>> agent, but for the guy who is doing something slightly out of the ordinary
>> (multiple injection, delayed injection, etc) it does have some advantages,
>> and I can't see any disadvantage.
>>   
>As long as we assume that no posting agent ever sees an advantage to 
>creating an injection-date with false information, and that the number 
>of posting agents who generate wrong injection-date parameters is not 
>significantly higher than the number of injecting agents to do so, there 
>are no disadvantages I can see.

A fair question that needs to be examined.

First, let us look at the presently used Date.

We are agreed (I think) that Dates up to 24 hours in the future do little
harm, and injecting agents are supposed to reject anything longer.
Clearly, they SHOULD reject Injection-Dates over 24 hours into the future
(I think I already suggested that needed to be added to Russ's text).

If a man, with hindsight, uses a Date header months into the past, then he
may be able to claim, falsely, "See, I already said that months ago". Of
course, that may also cause his article to propagate poorly (which is his
problem). But, in any case, this is not really a protocol issue.

But if a man uses an Injection-Date into the past, then the only effect
will be that his article propagates worse, so I do not see how that can
possibly give him any advantage, except possibly when he knows the exact
cutoff policy of his immediate peers, and is trying to ensure that his
scam is only seen by sites very close to himself (e.g. by colleagues in
his own organization). I think I could live with that.

To: [email-masked]
Xref: clerew local.usefor:24652
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: #1416: Reinjection and Injection-Date proposed text
Date: Thu, 3 May 2007 10:31:14 GMT
Lines: 24
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>I'd prefer to make a similar decision one way or the other for
>Injection-Date.  Either the entire feature is only a SHOULD, in which case
>it should be possible to implement any agent completely ignoring
>Injection-Date and never violate a MUST, or we consider the feature
>mandatory, in which case both posting agents and injecting agents MUST use
>it when appropriate.  I don't think it makes as much sense to end up in a
>mixed situation where Injection-Date is a MUST for injecting agents but
>only a should for multi-injecting posting agents, or vice versa.

In that case, I think I prefer MUSTs in the relevant places (essentially
what is in your proposed text).

To: [email-masked]
Xref: clerew local.usefor:24651
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: ISSUE: 4.2. Moderation flag in checkgroups
Date: Thu, 3 May 2007 10:28:37 GMT
Lines: 33
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>I propose changing the newsgroups-line production to instead read:

>         newsgroups-line     = newsgroup-name
>                                  [ 1*HTAB newsgroup-description ]
>                                  [ *WSP moderation-flag ]

>and moderation-flag to read:

>         moderation-flag     = %x20.x28.4D.6F.64.65.72.61.74.65.64.29
>                                  ; case sensitive " (Moderated)"

>which makes it clearer that the space is part of the flag.  I had
>considered just changing the 1*WSP to *WSP SP, but while writing this
>message, I decided I liked the above approach better.

+1

It might be odd to see yadda-yadda<HTAB><SP>(Moderated), but then it would
be odd (though harmless) to see <HTAB>s there at all (except for the
obligatory ones after the <newsgroup-description>).

Date: Thu, 03 May 2007 09:59:48 +0200
From: Harald Alvestrand <[email-masked]>
To: Russ Allbery <[email-masked]>
Cc: [email-masked]
Subject: Re: #1416 Injection-Date: current wording proposal (corrected)
Sender: [email-masked]

>
> @@ -704,8 +814,14 @@
>              the source of the article and possibly other trace information
>              as described in Section 3.2.8 of <xref target="USEFOR" />.</t>
>  
> -            <t>The injecting agent MUST then add an Injection-Date header
> -            field containing the current date and time.</t>
> +            <t>If the proto-article already had an Injection-Date header
> +            field, it MUST NOT be modified or replaced.  If the
> +            proto-article had both a Message-ID header field and a Date
> +            header field, an Injection-Date header field MUST NOT be
> +            added, since the proto-article may have been multiply injected
> +            by a posting agent that predates this standard.  Otherwise,
> +            the injecting agent MUST add an Injection-Date header field
> +            containing the current date and time.</t>
>  
>              <t>Finally, the injecting agent forwards the article to one or
>              more relaying agents, and the injection process is
>   
I (still) am not convinced of the need for not adding an Injection-Date: 
header when a message-ID and Date is present, but Injection-Date is not.

There are 3 cases:

- Message is singly injected. Injection-Date: does no harm.
- Message is multiply injected, Injection-Date: on all copies are quite 
close to each other, because they all got injected in rapid succession. 
Injection-Date: does no harm.
- Message is multiply injected, and some of the injections got 
significantly delayed.

I think most user agents will add Date: and Message-ID: by default. 
(Anyone have data on this, rather than supposition?)

So the net result of catering for the minority third case is that *most* 
messages on the Net will be sent without an Injection-Date.

If so, what's the point of claiming that we're "mandating" it?

                        Harald

Date: Thu, 03 May 2007 09:54:54 +0200
From: Harald Alvestrand <[email-masked]>
To: Russ Allbery <[email-masked]>
Cc: [email-masked]
Subject: Re: #1416 Injection-Date - Summary of options
Sender: [email-masked]

Russ Allbery wrote:
> Harald Alvestrand <[email-masked]> writes:
>
>   
>> As long as we assume that no posting agent ever sees an advantage to
>> creating an injection-date with false information, and that the number
>> of posting agents who generate wrong injection-date parameters is not
>> significantly higher than the number of injecting agents to do so, there
>> are no disadvantages I can see.
>>     
>
>   
>> Those two assumptions are fairly big.
>>     
>
> I don't see any possible advantage for a posting agent creating a false
> Injection-Date header over a posting agent right now creating a false Date
> header.  They should be equivalent.  The Injection-Date header is
> functional, rather than a trace header.
>
> Trace information should be put in Injection-Info instead.
>
>   
one (contrived) example would be someone who wishes to make a posting 
that is visible at one site, but not at another site, where he knows 
that one site discards based on Date: and the other side discards based 
on Injection-Date:

He would then put either Date: or Injection-Date: a week or so in the 
past, depending on which site he wishes his message to be "invisible" at.

If the WG thinks that this isn't very worrying (I can't say I find it 
very worrying either), it's fine.

(minor guard against this particular kind of mischief: Insist that 
Injection-Date: is no earlier than Date:, and check BOTH of them against 
the 72-or-so-hour limit....)

                       Harald

To: [email-masked]
Xref: clerew local.usefor:24645
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: #1416 Injection-Date: current wording proposal (corrected)
Date: Wed, 2 May 2007 18:02:06 GMT
Lines: 327
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>Gr.  Ignore the previous version of this message.  Saving the file before
>regenerating the diff is useful.  Please ignore the previous version and
>use this one instead.

OK, I have finally got around to reading the whole of this. Many of my
points are minor textual niggles, but some are quite substantive.

>--- usepro.xml	2007-02-19 22:20:06.000000000 -0800
>+++ usepro-1416.xml	2007-04-28 19:30:11.000000000 -0700
>@@ -446,6 +447,68 @@
> 
>+      <section anchor="history"
>+               title="Article History and Duplicate Suppression">
>+
>+        <t>Relaying and serving agents therefore MUST keep a record of
>+        articles they have already seen and use that record to reject
>+        additional offers of the same article.  This record is called the
>+        history file or database.</t>
          ^^^^^^^
         "history"
{since you have jsut defined a new term}
>+
>+            <t>Agents that enforce such a cutoff MAY then drop records of
>+            articles that had dates older than the cutoff from their
>+            history databases.  If such an article were offered to the
>+            agent again, it would be rejected due to the cutoff date, so
>+            the history record is no longer required to suppress the
>+            duplicate.</t>
>+
>+            <t>As an optimization for easier history database
>+            manipulation, agents MAY instead drop history records written
>+            longer ago than the cutoff interval plus one day.  If this
>+            retention mechanism is used, the history retention period MUST
>+            be longer than the cutoff interval to allow for articles dated
>+            in the future unless the agent rejects all articles dated in
>+            the future.  One day is the maximum allowed error into the
>+            future for article dates, so it is a convenient and safe
>+            extension for the retention interval.</t>

It takes a lot of VERY careful reading to spot the curicial diference
between those two cases (and Frank seems to have the same problem). I
therefore suggest the following alternative for the 2nd one:

              Alternatively, agents MAY drop history records according to
	      the date when the article was received, in which case they
	      MUST retain them for 24 hours longer than the cutoff
	      interval, since the date that accompanies the article is
	      permitted to be up to 24 hours later than its actual time of
	      injection (see <xref duties-injecting>).

{BTW, I prefer using 24 hours rather than 1 day for such short time
intervals, especially since the discussion under duties-injecting speaks
of even shorter intervals}

>+        <t>This is just one implementation strategy for article history,
             ^^^^^^^^^^^^^^^^                ^^^^^^^^
             These are just two             strategies
>+        albeit the most common one.  Relaying and serving agents are not
                                 ^^^
                                 ones
>+        required to use this strategy, only to meet the requirement of not
>+        accepting an article more than once. ....

However, the next bit of the sermon is a bit overdone, and the use of
"default" seems not quite right:

>+        ......... However, implementors of
>+        general-purpose Netnews relaying and serving agents who do not
>+        have extensive experience with Netnews and the subtle effects of
>+        its flood-fill algorithm are encouraged to use the above algorithm
>+        by default.</t>

So I would suggest:

          ......... However, implementors are advised normally to use
	  these strategies, especially if they do not have extensive
	  experience with Netnews and the subtle effects of its flood-fill
	  algorithm.

>@@ -453,9 +516,33 @@
> 
>+        applicable policies.  They MUST NOT create an Injection-Info
                                                     ^^
                                                     any
>+        header field; this header field will be added by the injecting
>+        agent.</t>
>+
>+        <t>If a proto-article contains both Message-ID and Date header
                ^              ^
               the           already
>+        fields, posting agents MAY add an Injection-Date header field to
>+        that proto-article immediately before passing that proto-article
>+        to an injection agent.  They SHOULD do so if the Date header field
>+        (representing the composition time of the proto-article) is more
>+        than a day in the past at the time of injection.  If the
>+        proto-article is being sent to more than one injecting agent, see
                                 ^^^^
                               submitted
>+        <xref target="multi-injection" />.</t>

{Note that this paragraph might change if we adopt option IC rather than
IR}
>+
>+        <t>The Injection-Date header field is new in this revision of the
>+        Netnews protocol and is designed to permit the Date header field
                                              ^^^^^^
                                              allow
>+        to hold the composition date (as defined in section 3.6.1 of <xref
                                           ^^^^^^^
                                         recommended
>+        target="RFC2822" />), even if the proto-article is not injected
>+        for some time after its composition.  However, note that all
>+        implementations predating this specification ignore the
>+        Injection-Date header field and use the Date header field in its
>+        stead for rejecting articles older than their cutoff (see <xref
>+        target="history" />), and injecting agents predating this
>+        specification do not add an Injection-Date header.  Articles with
>+        a Date header field substantially in the past will still be
>+        rejected by implementations predating this specification,
>+        regardless of the Injection-Date header field, and may suffer poor
                                                            ^           ^^^^
                                                          hence         poorer
>+        propagation.</t>
> 
>@@ -478,48 +565,70 @@
> 
>           <t>A proto-article has the same format as a normal article
>+          except that the Injection-Info and Xref header fields MUST NOT
>+          be present; the Path header field MUST NOT contain a "POSTED"
>+          &lt;diag-keyword>; ........

I disagree with forbidding POSTED (already raised in a separate thread),
and am dubious about Xref ("SHOULD" would be quite strong enough).

> 
>           <t>If a posting agent intends to offer the same proto-article to
>+          multiple injecting agents, the header fields Message-ID, Date,
>+          and Injection-Date MUST be present and identical in all copies
>+          of the proto-article.  See <xref target="multi-injection" />.</t>
               ^^^^^^^^^^^^^^^^^
                  it
> 
>+        <section anchor="multi-injection"
>+                 title="Multiple Injection of Articles">
>+          <t>Under some circumstances (posting to multiple disjoint
>+          networks, injecting agents with spotty connectivity, or
                                                                    ^
                                                                   for
>+          redundancy, for example), a posting agent may wish to offer the
>+          same article to multiple injecting agents.  In this unusual
>+          case, the goal is to not create multiple independent articles
>+          but rather to inject the same article at multiple points and let
>+          the normal duplicate suppression facility of Netnews (see
>+          <xref target="history" />) ensure that any given agent only
                                                                   ^^^^
>+          accepts the article once.</t>
                               ^
                              only
>+
>+          <t>Whenever possible, multiple injection SHOULD be done by
>+          offering the same proto-article to multiple injecting agents.
>+          The posting agent MUST supply the Message-ID, Date, and
>+          Injection-Date header fields, and the proto-article offered to
                                                  ^^^^^^^^^^^^^
                                               proto-articles as
>+          each injecting agent MUST be identical.</t>
>+
>+          <t>In some cases, offering the same proto-article to all
>+          injecting agents may not be possible (such as when transferring,
                                                               ^^^^^^^^^^^^
                                                               gatewaying
>+          after the fact, articles found on one Netnews network to
>+          another, unconnected one). .........
            ^^^^^^^^^^^^^^^^^^^^^^^^
            another, supposedly unconnected, one

However, this does not really convey the scenario as it is anticipated to
occur. In essence, the first network is typically small and private and
the second network is usually large, and probably Usenet. And one does not
usually just "find" articles there - one usually placed them there
oneself.

So how about:

            ....... (such as when gatewaying, after the fact, articles
            from a small private network, supposedly with no other
	    connections, to a larger network, such as Usenet). ......

>+          ...   In this case, the posting agent MUST
>+          convert the article back into a proto-article before passing it
>+          to another injecting agent, but it MUST retain unmodified the
>+          Message-ID, Date, and Injection-Date header fields.  It MUST NOT
>+          add an Injection-Date header field if it is missing from the
>+          existing article.  It MUST remove any Xref header field and
                                  ^^^^                              ^^^
                               MAY/SHOULD                           but
>+          either rename or remove any Injection-Info header field and
            ^
           MUST
>+          other trace fields.</t>
                              ^
                   (such as NNTP-Posting-Host and XTrace)

There is also the question of what to do about Path. My preference would
be for "MAY/SHOULD retain"

I think the next paragraph should be a NOTE. There is nothing normative
in it.

>+          <t>Multiple injection inherently risks duplicating articles.
>+          Multiple injection after the fact, by converting an article back
>+          to a proto-article and injecting it again, additionally risks
>+          loops, loss of trace information, and other problems and should
                                             ^
                      unintended reinjection into the same network,
>+          be done with care and only when there is no alternative. ......

In fact, it is usually "unintended reinjection" that is the cause of
"loops".

>+          <t>Multiple injection of an article listing one or more
>+          moderated newsgroups in its Newsgroups header field SHOULD only
>+          be done by a moderator and MUST only be done after the
>+          proto-article is approved for all moderated groups to which it
>+          is posted and has an Approved header field.  .......
              ^                                       ^
            to be                       (see <xref duties-moderator>)

>@@ -644,23 +753,24 @@
> 
>             <t>It MUST reject any proto-article that does not have the
>             proper mandatory header fields for a proto-article; that has
>+            Injection-Info or Xref header fields; that has a Path header
              ^              ^^^^^^^^^^^^^^^^^^^^^  ^^^^^^^^^^^^^^^^^^^^^^
             any                   field
>+            field containing the "POSTED" &lt;diag-keyword>; or that is
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

>+            not syntactically valid as defined by <xref target="USEFOR"
>+            />.  It SHOULD reject any proto-article which contains a
>+            header field deprecated for Netnews.  It MAY reject any
                                                 ^
                                      (see e.g. <xref RFC2298>)
>+            proto-article that contains trace header fields indicating
                                                             ^
                                        (e.g. NNTP-Posting-Host or Xtrace)
>+            that it was already injected by an injecting agent that did
>+            not add Injection-Info or Injection-Date.</t>
> 
>             <t>It SHOULD reject any article whose Date header field is
>             more than 24 hours into the future (and MAY use a margin less
>             than 24 hours).  It SHOULD reject any article whose Date
>+            header is too far into the past (older than the cutoff
                    ^
                  field
>+            interval of a relaying agent the injecting agent is using, for
>+            example), .......

Would that apply to Injection-Date, if any? Or would you allow a stale
Date if there was a plausible Injection-Date?

>+            ...... since not all news servers support Injection-Date
>+            and only the injecting agent can provide a useful error
>+            message to the posting agent.  This interval SHOULD NOT be any
>+            shorter than 72 hours into the past.</t>

Yes, I can accept the 72 hours in that last sentence. I could also accept
omitting the sentence entirely.
> 
>@@ -704,8 +814,14 @@
> 
>+            <t>If the proto-article already had an Injection-Date header
>+            field, it MUST NOT be modified or replaced.  If the
>+            proto-article had both a Message-ID header field and a Date
>+            header field, an Injection-Date header field MUST NOT be
>+            added, since the proto-article may have been multiply injected
>+            by a posting agent that predates this standard.  Otherwise,
>+            the injecting agent MUST add an Injection-Date header field
>+            containing the current date and time.</t>

That paragrpaph is one of the ones that would change if we adopt IC rather
than IR. ALternaitvely, one might consider s/MUST NOT be added/SHOULD NOT
be added/. My concern with the IR option is that it leaves no room for
some future day when everybody routinely implements Injection-Date and it
would be sensible for all injecting agents to add it regardless (unless
already present, of course).
> 
>@@ -801,18 +917,15 @@
>+
>+            <t>It MUST reject any article that has already been
>+            successfully sent to it or that is dated older than its cutoff
>+            date, as described in <xref target="history" />.</t>

No, that won't work. You have not actually defined the term "cutoff date".
But, in any case even providing a "cutoff interval" is not a REQUIRTEMENT.

How about:

>+            <t>It MUST reject any article that has already been
               successfully sent to it, or alternatively where its date
               falls outwith any cutoff interval as described in <xref
               history>.

>@@ -886,16 +999,13 @@
> 
>             <t>It MUST reject any article that has already been
>+            successfully sent to it or that is dated older than its cutoff
>+            date, as described in <xref target="history" />.</t>

Same problem - same cure.

I think some mention of Injection-Date also needs to be made in section
3.9.2 (incoming Gateways), though I am not quite sure what.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: #1416 Injection-Date: current wording proposal (corrected)
Organization: The Eyrie
Date: Wed, 02 May 2007 20:05:25 -0700
Sender: [email-masked]

Frank Ellermann <[email-masked]> writes:

> Apparently the definition of "retention" is based on the article date,
> not the arrival date, is that as it should be ?  [[In the bullet with
> the "optimization for easier history manipulation"]]

> Maybe add a qualifier "if using the article date and not the arrival
> date" to the "history retention" in this paragraph.

Is this part clear now?

> For the "MUST NOT contain POSTED", in theory this would be fine for an
> injecting-agent not creating its own POSTED because it doesn't know the
> new diagnostics.  But when it doesn't know them it can't identify them,
> so I guess the MUST NOT is unnecesary.  Or I forgot why we wanted this.

I'm not sure that it was a matter of a lot of on-list discussion.  It feel
out of a tighter definition of just what a proto-article is, and I may
well have made it too tight.  It's a trace diagnostic, so having it
multiple times doesn't hurt anything.

I think we either say that an injecting agent MUST reject articles that
already have POSTED or we drop that wording entirely; I don't think a
SHOULD would make sense.

> You still have 72 hours, can we use 3 days, getting rid of "hours"
> everywhere ?  And MUST NOT instead of SHOULD NOT about the minimum, I
> don't see how it can work as expected for a seriously shorter periods,
> or what's a good excuse for shorter periods.

I don't mind changing this to three days.

I'm still uncomfortable about making this a MUST NOT in general.  The
interoperability problems of using a shorter interval don't feel like
MUST-level problems to me.  But I'd certainly welcome other opinions and
can be convinced.

> Similarly, s/24 hours/one day/ for the definition of "future".  The "MAY
> reject" about less than 24 hours could run into timezone issues, please
> delete it.  It MAY reject depending on the phase of the moon, or
> anything from odd IPs on Tuesdays ((c) JCK), "less than 24 hours in the
> future" needs no special MAY.

Here, I do want to use 24 hours.  The definition of a "day" may be 25
hours or 23 hours depending on daylight saving time transitions, and I
don't think we want the ambiguity.  It doesn't really matter once we're
talking about longer periods, but it does matter at the interval of a day.

Any other feelings on whether or not to keep the MAY about shorter
rejection periods for articles dated in the future?

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: #1416: Reinjection and Injection-Date proposed text
Organization: The Eyrie
Date: Wed, 02 May 2007 19:59:17 -0700
Sender: [email-masked]

Charles Lindsey <[email-masked]> writes:
> Russ Allbery <[email-masked]> writes:

>> Just for clarity in the discussion (I don't think a wording change is
>> needed), note that the cutoff date for history retention is independent
>> of the period of article retention in the spool.  Usually the latter is
>> longer than the former, but it may also be shorter.  They don't need to
>> be related.

> You surprise me. I would have expected the article period retention to
> be used if longer than the cutoff, on the grounds that, if you still
> have the article stored, then it is easy to detect that anotherone that
> arrives should be ignored.

Since articles are not usually stored by message ID, most implementations
will not find it any easier to verify that they already have an article
when the article is still in the spool.

However, related to that, INN and CNews both use the history file for
various protocol operations, such as retrieving articles by message ID.
There's no inherent need to use the history file for this purpose; it just
happened to be convenient to extend it with a few additional fields since
it was already keyed by message ID.  This is just an implementation
detail, and it's fair game to have the history entirely independent of the
local spool.

That wasn't really what I was driving at, though.  What I was saying was
that the cutoff period for history entries (how long they're retained
regardless of whether the article is still stored) is often shorter than
and independent of the article retention time for articles that were
received.  It so happens that entries for articles still in the spool are
also retained in history past the cutoff, but that's irrelevant for the
section of the protocol that we're talking about.  Since not *all* seen
articles are retained for that longer period, the existence of a few
longer-term entries doesn't mean the cutoff is any longer.

> Yes, that is what CNews does. If the actual retention is less than the
> cutoff, the article is removed, but the history file remembers it to
> prevent duplicates. But if the actual retention is longer, then that
> longer period is used for both purposes.

INN works the same way.

>> This was a bit of a mess.  In my current draft, I've simplified this
>> whole situation by saying that posting agents doing multiple injection
>> MUST add the Injection-Date header field.  This makes all existing
>> multiple injection posting agents non-compliant, but we also did the
>> same thing with injecting agents and their Injection-Date requirement,
>> so this seems symmetric.

>> I would prefer that the whole Injection-Date feature be either a SHOULD
>> or a MUST, rather than using differing levels of requirement for
>> different agents.  Thoughts?  And does anyone want to argue that the
>> whole feature should be a SHOULD?

> In the case of a simple singly-injecting agent that does not delay
> injection, then I don't think we care whether it adds Injection-Date or
> not, so MUST would be too strong there. We say that if it is >1day late,
> then he SHOULD add it (but it's his own fault if he doesn't).

I didn't mean that I would ever say a posting agent MUST add
Injection-Date unconditionally.  What I mean is more like what we have
with <path-diagnostic>.  Right now, the draft has the feature as a whole
at the SHOULD level.  It's possible to ignore <path-diagnostic> completely
and implement the protocol without violating any MUSTs, only SHOULDs.

I'd prefer to make a similar decision one way or the other for
Injection-Date.  Either the entire feature is only a SHOULD, in which case
it should be possible to implement any agent completely ignoring
Injection-Date and never violate a MUST, or we consider the feature
mandatory, in which case both posting agents and injecting agents MUST use
it when appropriate.  I don't think it makes as much sense to end up in a
mixed situation where Injection-Date is a MUST for injecting agents but
only a should for multi-injecting posting agents, or vice versa.

> I think I have lost the context here, but if this id multiple-injection,
> then MUST is needed.

Yeah, that's the context.  That's the direction that I went, since my
feeling was that we wanted Injection-Date to be a MUST implement feature.

>> It's more the former, plus backwards compatibility.  Existing injecting
>> agents will reject articles with Xref rather than remove the header.
>> It seems simple enough given the other context to have posting agents
>> remove it, and existing posting agents remove it, so I just said MUST
>> instead of adding the explanation that existing injecting agents will
>> reject and then go through and try to change the language to allow
>> injecting agents to cope with it.

> Xrefs only become of interest to clients of the serving agent that
> generated them. Removal of redundant Xrefs by earlier agents is just a
> matter of tidiness and possible removal of confusion, so I don't think
> you have any grounds for MUST there.

I think interoperability is fine grounds for a MUST.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: ISSUE: 4.2. Moderation flag in checkgroups
Organization: The Eyrie
Date: Wed, 02 May 2007 10:51:05 -0700
Sender: [email-masked]

Julien ÉLIE pointed out in private e-mail a minor problem with how the
moderation flag is specified in section 4.2.  Currently, we have:

         newsgroups-line     = newsgroup-name
                                  [ 1*HTAB newsgroup-description ]
                                  [ 1*WSP moderation-flag ]
         newsgroup-description
                             = eightbit-utext *( *WSP eightbit-utext )
         moderation-flag     = %x28.4D.6F.64.65.72.61.74.65.64.29
                                  ; case sensitive "(Moderated)"
         eightbit-utext      = utext / %d127-255

This allows a tab before "(Moderated)" instead of a space.  I'm not sure
that existing software supports that, it doesn't match the informal
description on ftp.isc.org, and it contradicts the text in the following
paragraph:

   The <newsgroup-description> MUST NOT contain any occurrence of the
   string "(Moderated)" within it.  Moderated newsgroups MUST be marked
   by appending the case sensitive text " (Moderated)" at the end.

I propose changing the newsgroups-line production to instead read:

         newsgroups-line     = newsgroup-name
                                  [ 1*HTAB newsgroup-description ]
                                  [ *WSP moderation-flag ]

and moderation-flag to read:

         moderation-flag     = %x20.x28.4D.6F.64.65.72.61.74.65.64.29
                                  ; case sensitive " (Moderated)"

which makes it clearer that the space is part of the flag.  I had
considered just changing the 1*WSP to *WSP SP, but while writing this
message, I decided I liked the above approach better.

From: Russ Allbery <[email-masked]>
To: [email-masked]
Subject: Re: #1416 Injection-Date - Summary of options
Organization: The Eyrie
Date: Wed, 02 May 2007 10:40:46 -0700
Sender: [email-masked]

Harald Alvestrand <[email-masked]> writes:

> As long as we assume that no posting agent ever sees an advantage to
> creating an injection-date with false information, and that the number
> of posting agents who generate wrong injection-date parameters is not
> significantly higher than the number of injecting agents to do so, there
> are no disadvantages I can see.

> Those two assumptions are fairly big.

I don't see any possible advantage for a posting agent creating a false
Injection-Date header over a posting agent right now creating a false Date
header.  They should be equivalent.  The Injection-Date header is
functional, rather than a trace header.

Trace information should be put in Injection-Info instead.

Date: Wed, 02 May 2007 16:08:34 +0200
From: Harald Alvestrand <[email-masked]>
To: Charles Lindsey <[email-masked]>
Cc: [email-masked]
Subject: Re: #1416 Injection-Date - Summary of options
Sender: [email-masked]

Charles Lindsey wrote:
>
>> After writing the text, I think it's weird, but it's not quite as bad as I
>> thought it was going to be.  It does have the merit of closely mirroring
>> how Date works today, and if we're going to change something in this area,
>> sticking as close to what we know works as possible seems like a good idea
>> to me.
>>     
>
> +1
>
> It may well be that most 'normal' posting agents leave it to the injecting
> agent, but for the guy who is doing something slightly out of the ordinary
> (multiple injection, delayed injection, etc) it does have some advantages,
> and I can't see any disadvantage.
>   
As long as we assume that no posting agent ever sees an advantage to 
creating an injection-date with false information, and that the number 
of posting agents who generate wrong injection-date parameters is not 
significantly higher than the number of injecting agents to do so, there 
are no disadvantages I can see.

Those two assumptions are fairly big.

                  Harald

To: [email-masked]
Xref: clerew local.usefor:24638
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: RESOLUTION: #1482: USEPRO 3.2: Possibility to use non-resolvable domain name as path-identity
Date: Mon, 30 Apr 2007 16:40:15 GMT
Lines: 24
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>It's a minor point, and somewhat off-topic, but I feel obligated to point
>out whenever it seems appropriate that hosts running SMTP daemons with A
>records do not need MX records.  A remarkable number of people (which I
>expect does *not* include you -- this was just a rant trigger) seem to be
>confused about this these days and write stupid spam filtering software
>under the assumption that A records are not sufficient for delivering
>e-mail.

Yes, I knew that, but a pure news server with an A record might not be
listening on Port 25 at all. In that case, an MX (probably pointing to the
site's mailhost) would be a good idea.

To: [email-masked]
Xref: clerew local.usefor:24637
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: #1416: Reinjection and Injection-Date proposed text
Date: Mon, 30 Apr 2007 16:37:55 GMT
Lines: 192
Sender: [email-masked]

In <[email-masked]> Russ Allbery <[email-masked]> writes:

>Frank Ellermann <[email-masked]> writes:
>> Russ Allbery wrote:

>>> + it ought not be aggressively short.  For For Usenet, for example,
>>> + a cutoff interval of no less than seven days is conventional.

>> Yes, no MUSTard needed here, the consequences are clear.  Commercial
>> servers with shorter cutoff intervals for "binary" newsgroups may
>> need the support staff to answer complaints by their angry customers.

>Just for clarity in the discussion (I don't think a wording change is
>needed), note that the cutoff date for history retention is independent of
>the period of article retention in the spool.  Usually the latter is
>longer than the former, but it may also be shorter.  They don't need to be
>related.

You surprise me. I would have expected the article period retention to be
used if longer than the cutoff, on the grounds that, if you still have the
article stored, then it is easy to detect that anotherone that arrives
should be ignored. OK, I suppose the site might keep two history files one
of which (in a more abbreviated form) was just for cutoff.

>Normally, the cutoff date is set uniformly across the server regardless of
>the group, but the article retention period may be set much shorter for
>binary groups.  The requirement is then that the server keep a record of
>those received and expired binary articles in the history file even after
>deleting the article off disk until they meet the cutoff requirements.

Yes, that is what CNews does. If the actual retention is less than the
cutoff, the article is removed, but the history file remembers it to
prevent duplicates. But if the actual retention is longer, then that
longer period is used for both purposes.

>This was a bit of a mess.  In my current draft, I've simplified this whole
>situation by saying that posting agents doing multiple injection MUST add
>the Injection-Date header field.  This makes all existing multiple
>injection posting agents non-compliant, but we also did the same thing
>with injecting agents and their Injection-Date requirement, so this seems
>symmetric.

>I would prefer that the whole Injection-Date feature be either a SHOULD or
>a MUST, rather than using differing levels of requirement for different
>agents.  Thoughts?  And does anyone want to argue that the whole feature
>should be a SHOULD?

In the case of a simple singly-injecting agent that does not delay
injection, then I don't think we care whether it adds Injection-Date or
not, so MUST would be too strong there. We say that if it is >1day late,
then he SHOULD add it (but it's his own fault if he doesn't).

>>> + the goal is to not create multiple independent articles but
>>> + rather to inject the same article at multiple points and use the
>>> + normal duplicate suppression facility of Netnews (see Section 3.3)
>>> + ensure that any given agent only accepts the article once.

>> Nit: s/use/let/

>Fixed.

>> The move from "reinjection" to "multiple injection" resulted in a
>> far better readable story.

>Yes, I was rather happy with that.  Thanks to Harald for the suggestion.
>I think that casting of the idea makes everything much simpler.

>>> -  A posting agent SHOULD normally reject such articles.

>> Odd, was that really a "posting agent" rejecting articles ?

>Yeah.  Dumb idea on my part, in retrospect.  It was discussed in some of
>the side-threads of this overall discussion, and I decided to drop it as
>part of these changes since the bits that it was referring to have now
>substantially changed anyway.

>>> + The posting agent MUST supply the Message-ID, Date, and
>>> + Injection-Date header fields, and the proto-article offered to
>>> + each injecting agent MUST be identical.

>> Some lines above it was a "multi-SHOULD", now you have a "multi-MUST"
>> for the Injection-Date.  How about this:

>> | The posting agent MUST supply the Message-ID and the Date, it
>> | SHOULD also supply an Injection-Date, and the proto-article
>> | offered to each injecting agent MUST be identical.

>> Arguably you can also use MUST here as you have it, and replace
>> the "multi-SHOULD" at the end of 3.4.1 by MUST.  But that would
>> render all existing posting agents with no idea about the new
>> Injection-Date as "non-conforming" when they try multi-Injection.

>As mentioned above, I went the MUST route, but I think it's a debatable
>point and I'd love to hear other comments.

I think I have lost the context here, but if this id multiple-injection,
then MUST is needed.

I have no problem with existing agents that move from "conforming" to
"legacy" when our standard comes out, provided we arrange (as we have)
that they continue to interoperate.

>>> + It MUST remove any Xref header field and either rename or remove
>>> + any Injection-Info header field and other trace fields.

>> I see why they MUST remove Injection-Info, an old injecting agent would
>> let it pass causing havoc.  But why MUST a posting agent remove Xref,
>> this job could be handled by the injecting agent ?

>> Is it "a simple MUST is better than more convoluted statements",
>> or do I miss a serious Xref-problem here ?

>It's more the former, plus backwards compatibility.  Existing injecting
>agents will reject articles with Xref rather than remove the header.  It
>seems simple enough given the other context to have posting agents remove
>it, and existing posting agents remove it, so I just said MUST instead of
>adding the explanation that existing injecting agents will reject and then
>go through and try to change the language to allow injecting agents to
>cope with it.

Xrefs only become of interest to clients of the serving agent that
generated them. Removal of redundant Xrefs by earlier agents is just a
matter of tidiness and possible removal of confusion, so I don't think you
have any grounds for MUST there.

>Here's what it looked like before I started writing this section of my
>reply:

>Legacy posting agent, legacy injecting agent:
>    No one will add Injection-Date since neither agent knows about it,
>    although as a generic unknown header, either agent is permitted to
>    do so.

>Legacy posting agent, compliant injecting agent:
>    Posting agent won't add Injection-Date.  Injecting agent MUST add
>    Injection-Date if one or both of Date and Message-ID are not
>    specified, otherwise MUST NOT (since multiple injection may be
>    happening).

>Compliant posting agent, legacy injecting agent:
>    Posting agent MAY add Injection-Date any time, SHOULD if the Date
>    is more than a day in the past at the time of injection, MUST if
>    doing multiple injection.  The injecting agent doesn't know about
>    the header and therefore will do nothing with it.

>Compliant posting agent, compliant injecting agent:
>    Posting agent MAY add Injection-Date any time, SHOULD if the Date
>    is more than a day in the past at the time of injection, MUST if
>    doing multiple injection.  Injecting agent generally MUST add the
>    header if the posting agent doesn't.  The one exception is if the
>    posting agent provides both Message-ID and Date but not
>    Injection-Date, which it is always allowed to do and which doesn't
>    even violate a SHOULD if the Date is fresh.  In this case, the
>    injecting agent MUST NOT add it either.

>Looking at that table, and given that we have spelled out elsewhere how to
>handle articles with no Injection-Date, I think I agree with your point
>and don't see a reason for the injection agent requirement to be MUST.  We
>already have a scenario in which a post only handled by compliant agents
>gets no Injection-Date and it doesn't break anything, so it seems like
>addition of Injection-Date by an injecting agent need only be a SHOULD.

>Opinions?

If we are trying to move to a position where all articles routinely carry
an Injection-Date, then MUST is appropriate. Under option IR, which is
what you describe above, things are complicated by that MUST NOT, so under
IR a SHOULD might be enough.

But under IC, it is a simple MUST add it regardless (and put up with a
little pain from some hopefully rare legacy situations).

>This situation might change if we went with Charles's approach where
>injecting agents are permitted to add Injection-Date to posts that already
>have both Message-ID and Date.  In that case, there is no gap.

Indeed. I think we need to resolve the IR/IC issue before we can finalize
any wording here.

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: #1416: Reinjection and Injection-Date proposed text
Date:  Tue, 01 May 2007 10:40:04 +0200
Organization:  <URL:http://purl.net/xyzzy>
Lines: 112
Sender: [email-masked]

Russ Allbery wrote:

> Just for clarity in the discussion (I don't think a wording change is
> needed), note that the cutoff date for history retention is independent
> of the period of article retention in the spool.  Usually the latter is
> longer than the former, but it may also be shorter.  They don't need to
> be related.

Oops, now I've read and answered the corrected diff before I read this
article, some of my remarks there might be obsolete...

Yes, I thought that there's only one meaning of "retention", so please
ignore the remark / question about that.

>>| The posting agent MUST supply the Message-ID and the Date, it
>>| SHOULD also supply an Injection-Date, and the proto-article
>>| offered to each injecting agent MUST be identical.
 
>> Arguably you can also use MUST here as you have it, and replace
>> the "multi-SHOULD" at the end of 3.4.1 by MUST.  But that would
>> render all existing posting agents with no idea about the new
>> Injection-Date as "non-conforming" when they try multi-Injection.
 
> As mentioned above, I went the MUST route, but I think it's a debatable
> point and I'd love to hear other comments.

Okay, if it's consistent I can certainly live with both, and the
MUST route results in clearer text.  

>> why MUST a posting agent remove Xref, this job could be handled
>> by the injecting agent ?  Is it "a simple MUST is better than
>> more convoluted statements", or do I miss a serious Xref-problem
>> here ?
 
> It's more the former, plus backwards compatibility.  Existing
> injecting agents will reject articles with Xref rather than remove
> the header.

Okay, "reject" is serious enough...  Reminds me of some cmsg: cancel
threads, we're not going to discuss Xref removal for a year or so :-)

>>> + This interval SHOULD NOT be any shorter than 72 hours into the
>>> + past.
 
>> That's a rather short minimum, barely enough for a normal weekend.
>> For this radical approach, how about a MUST NOT ?  Disclaimer, I've
>> no idea about what's usual for "binary" newsgroups.  But I doubt
>> that they have any use for the 2822-Date concept like "text" groups
>> or, more important, mail2news gateways.
 
> On Usenet, I think a MUST NOT would probably be warranted, but I wasn't
> comfortable enough saying that there would be no reason to ever use a
> shorter interval in any Netnews implementation.  I don't really feel
> strongly about this, but I mildly lean towards SHOULD NOT here just on
> the grounds of permitting an out for some reason that we didn't
> anticipate.

If Usenet needs a MUST NOT I'm tempted to ignore others where shorter
might also work.  The text makes it clear that implementors can't
hardwire "three days", and as soon as it's configurable admins can 
also try shorter periods, and get what they asked for.  

> Legacy posting agent, legacy injecting agent:
>  No one will add Injection-Date since neither agent knows about it,
>  although as a generic unknown header, either agent is permitted to
>  do so.
 
> Legacy posting agent, compliant injecting agent:
>  Posting agent won't add Injection-Date.  Injecting agent MUST add
>  Injection-Date if one or both of Date and Message-ID are not
>  specified, otherwise MUST NOT (since multiple injection may be
>  happening).
 
> Compliant posting agent, legacy injecting agent:
>  Posting agent MAY add Injection-Date any time, SHOULD if the Date
>  is more than a day in the past at the time of injection, MUST if
>  doing multiple injection.  The injecting agent doesn't know about
>  the header and therefore will do nothing with it.

> Compliant posting agent, compliant injecting agent:
>  Posting agent MAY add Injection-Date any time, SHOULD if the Date
>  is more than a day in the past at the time of injection, MUST if
>  doing multiple injection.  Injecting agent generally MUST add the
>  header if the posting agent doesn't.  The one exception is if the
>  posting agent provides both Message-ID and Date but not
>  Injection-Date, which it is always allowed to do and which doesn't
>  even violate a SHOULD if the Date is fresh.  In this case, the
>  injecting agent MUST NOT add it either.

Thanks.
 
>>> + It MAY reject articles with dates in the future with a smaller
>>> + margin than 24 hours.
 
>> Does that really help ?
 
> It's there to make it clear that some agents do not give you 24 hours
> of leeway and this is permitted in the protocol.  It's redundant with
> the general rule that anyone can reject any article for any reason
> they please, but I felt like it was worth drawing special attention
> to this so that people don't think they can rely on having 24 hours
> of slop in the date checking.

Yes, but we want to give them that leeway for timezone issues.  The
warning is okay, but an explicit MAY sounds like a *permission* to
be more restrictive, not like a *warning* "some servers out there
might be more restrictive".  

Keep the MAY if you like it, it's no important point.

Frank

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: RESOLUTION: #1482: USEPRO 3.2: Possibility to use non-resolvable domain name as path-identity
Date:  Tue, 01 May 2007 09:41:25 +0200
Organization:  <URL:http://purl.net/xyzzy>
Lines: 28
Sender: [email-masked]

Charles Lindsey wrote:

> USEAGE could try to pick up the pieces.

If it starts by recommending _one_ news role acoount out of the
three proposals (two in 2142 + two in s-o-1036) it can try that.

In a parallel article you said about the "zone cut" odditites:

| Interestingly, the DKIM WG has been looking at this, and to
| do it properly you really need some widely publicised and
| maintained list of TLDs/zones.

The SURBL list is nice, but it's no IANA registry or something,
some entries in this list are there because I submitted them
(e.g. enum.arpa), IOW it's far from complete or reliable.  No
uri.arpa and no urn.arpa (examples) if it's still at the state
I know.

The CSV strategy was also cute, but it was a kludge, something
in the direction of "try direct, if that fails strip to seven
labels or what you have minus the already tested direct FQDN,
test that, and then test up to the SLD, never bother the TLD".

Don't nail me about CSV details, please.

Frank

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: OT: nslookup -q=soa bogus.bogus.com
Date:  Tue, 01 May 2007 09:30:49 +0200
Organization:  <URL:http://purl.net/xyzzy>
Lines: 16
Sender: [email-masked]

Harald Alvestrand wrote:

>> `nslookup -q=soa bogus.bogus.com` results in an error
>> with my version of nslookup, it gives up for NXDOMAIN.

> I was talking about Charles' tree-climbing exercise; if you cut off
> labels until you find something, you will eventually look up ".com".

In his example he'd arrive at bogus.com (that exists) with tree
walking.  It's odd that there is no clear distinction between
the two cases "yes, this is my domain, and what I do below it
is none of your business" and "yes, this is my domain, but I'm
not responsible for anything below it that doesn't exist (yet)."

Frank

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: Reject vs. fix
Date:  Tue, 01 May 2007 09:23:12 +0200
Organization:  <URL:http://purl.net/xyzzy>
Lines: 13
Sender: [email-masked]

Harald Alvestrand wrote:
 
> My technical opinion:
 
> I think it's reasonable for a posting agent (under the control of the
> poster) to "fix" articles.
> I think it's unreasonable for an injecting agent (under the control of
> the news admin) to do so.

We didn't disagree, or did we ?  Maybe we're in violent agreement.

Frank

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Subject:  Re: #1416 Injection-Date: current wording proposal (corrected)
Date:  Tue, 01 May 2007 09:14:40 +0200
Organization:  <URL:http://purl.net/xyzzy>
Lines: 45
Sender: [email-masked]

Russ Allbery wrote:

> I think it may be worthwhile to reach consensus on several parts of
> this separately.

Yes, e.g. I've no clue what precisely you adopted from what I wrote,
or what I wrote in the first place.  Sometimes there are situations
where I think B is better than A, and as soon as I see B I think A
is better than B, etc. until somebody says enough.

> For example, if we decide that the whole current history section is
> fine, I can incorporate that into the draft and then stop including
> it in further diffs for discussion.

Checking what I recall, the "one day in the future" business is clear.

Apparently the definition of "retention" is based on the article date,
not the arrival date, is that as it should be ?  [[In the bullet with
the "optimization for easier history manipulation"]]

Maybe add a qualifier "if using the article date and not the arrival
date" to the "history retention" in this paragraph.

For the "MUST NOT contain POSTED", in theory this would be fine for
an injecting-agent not creating its own POSTED because it doesn't
know the new diagnostics.  But when it doesn't know them it can't
identify them, so I guess the MUST NOT is unnecesary.  Or I forgot
why we wanted this.

You still have 72 hours, can we use 3 days, getting rid of "hours"
everywhere ?  And MUST NOT instead of SHOULD NOT about the minimum,
I don't see how it can work as expected for a seriously shorter
periods, or what's a good excuse for shorter periods.

Similarly, s/24 hours/one day/ for the definition of "future".  The
"MAY reject" about less than 24 hours could run into timezone issues,
please delete it.  It MAY reject depending on the phase of the moon,
or anything from odd IPs on Tuesdays ((c) JCK), "less than 24 hours
in the future" needs no special MAY.

The XML diff listed the 24 hours in two overlapping contexts (just
an observation).

Frank