Subject: Responses from the USEFOR WG Chairs on IESG DISCUSSes

From: Harald Alvestrand <[email-masked]>
Date: Fri, 08 Dec 2006 13:47:15 +0100

Greetings,

we (the chairs) hvave discussed the IESG feedback on 
draft-ietf-usefor-usefor.

There are two DISCUSS comments that we'd like to discuss further on this
document.

Sam Hartman filed this:

> First, the reference to draft-ietf-usefor-usepro needs to be
> normative.  I don't think you can construct an article without
> following advice related to path, injection-*, and control from that
> document.  Parsing these fields without usepro is similarly difficult.

The WG had a debate specifically about whether there should be a 
normative reference to USEPRO, and decided that there should not be one.
The WG's belief is that the -usefor document contains enough information 
to write software that parses an Usenet article and says whether or not 
it conforms to the format.

We believe that the construction of the content of those fields is not a
normative reference for the format - any more than the RIR allocation 
rules for IP addresses forms a normative reference for the definition of 
the IP header.

If there are specific instances of parsing an Usenet message (as opposed 
to acting on the content of a message) that you think is difficult 
without USEPRO, we'd like to have the details, so that we can discuss 
the specifics.

> We received last call comments about the subsetting of RFC 2822
> messages.  Very late, we realized that the real issue is not that
> these requirements are difficult for new posting agents, but that they
> are difficult when a email message is injected into usenet.  First, I
> think more discussion of whether that is the right approach for
> gateways is needed.

It is definitely not the right approach.
A normal email message will not be a valid Usenet message - if only 
because it lacks a Path: header and a Newsgroups: header.

The WG has discussed the problem of email-to-news gateways, and made an
explicit decision in at least one other case to exclude wording that 
would seem to be normative for such gateways. We believe that this 
problem is out of scope for this document, and is (as we read the 
charter) out of scope for the present charter of the USEFOR WG.

However, we have received a proposal for clarifying text about the
relationship between the USEFOR format and the mail format; we'll 
consider this for inclusion at the end of section 2.1 of the draft:

>This draft places additional restrictions on RFC2822 message format rules,
>in order to be compatible with deployed NetNews agents.
>An entity that generates messages compliant with RFC2822 might not always
>generate messages compliant with this specification, although certainly a
>single code-base can generate messages according to both sets of rules if
 >desired.

>Gatewaying between email and news is not part of this specification,
 >but we note that such a gateway would in all cases have to do more than
 >just copying the message. Ensuring that the message satisfies the
 >constraints here would be part of the job of such a gateway.

Would this satisfy the DISCUSSant on this point?

Back to the DISCUSS text:

>  Second, regardless of what we say here, it will
> be reasonably common for messages to be injected that do not meet
> these requirements.  In the interests of interoperability we need to
> discuss what problems can result and make recommendations for liberal
> receivers that minimize interoperability problems.

The WG discussed the problem of illegal messages and decided that it was
not possible or reasonable to specify handling of messages that do not
conform to this syntax in the syntax document. This would also violate
long-standing practice in, for instance, RFC 2822.
There is a document on the WG's document list - USEAGE - where such
considerations would be appropriate. (However, note that the state of 
the WG is not such that this document will be finished soon, if at all. 
See later note.)

>   The Security Considerations say:
>   >
>   > Further security considerations are discussed in
>   > [I-D.ietf-usefor-usepro].
>   >
>   There is some really good information in this document.  Can we make
>   sure that it gets published?  I would like to see an RFC referenced,
>   not an Internet-Draft, when this document becomes an RFC.

This is one of multiple references that really reflect the fact that you
can't make a working Usenet system with just the Usenet article format.

As discussed above, the WG decided against making the reference 
normative - one purpose of this was to make it possible to publish the 
USEFOR document without waiting for USEPRO.

If there are specific concerns for USEPRO that you wish moved into 
USEFOR, we can discuss that - but making the publication of USEFOR 
depend on the publication of USEPRO would definitely be overriding a 
consensus of the working group.

About the state of the working group:
This group is very low energy. The number of active participants is
probably less than 10, and the number of constructive active 
participants less than that. A few tens of people follow the mailing 
list and occasionally make comments.

The process of getting from a draft that was "nearly finished" for 
USEFOR to the present state took 1.5 years; the total process since the 
start of the working group has been more than 9 years.

We have just started a process of deciding if a change of editors for
USEPRO is warranted; once that is settled, the process of making USEPRO
ready for publication could easily take another year. There is also a 
grave danger that the last few constructive active participants will 
just give up and go away, leaving us with no finished USEPRO document at 
all. USEAGE is in even greater danger of that happening, since it is 
later in the WG's schedule, and depends on USEPRO finishing.

In this context, we would very much like to prove to the WG that it is
possible to finish a work item and get it published. We need all the
encouragement we can get.

              Harald and Alexey

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Fri, 08 Dec 2006 10:24:23 -0500

> it conforms to the format.

1. Can we clarify how the "WG decided there should not be" a normative
reference?

I and the USEPRO editor, two of 7 participants in the last call poll, argued 
that USEFOR must wait until USEPRO.  My objection was more based on
the observation that this WG was not likely to get the format settled
correctly until USEPRO was close to being finished.

While I cannot speak for the other participants in the poll, most of the
desire seemed to be that "We want a document, some document, out
to IETF last call" and not "USEFOR is set, we have confidence that we
can write companion documents with this."

The current INN maintainer has spent intense hours reviewing the draft
since the poll was taken and is now on record in 
<[email-masked]>, Dec 1, with
     "There are several [sections] that seem rather normative to me having just
     gone through all of USEPRO, apart from the security considerations."

2. I agree with the sentiment that you cannot construct a useful article without
what will be in USEPRO.  You can construct a conforming article, however.  Are
Internet Message RFCs much different in this respect?

3. "Parsing" is an ambiguous term.  USEFOR is complete enough that you can
determine compliance of an article.  So it can be "parsed" in that sense.
But you cannot determine that the article was constructed following USEPRO.

I think this creates an attractive nuisance problem that I don't recall being 
discussed in the WG....

Will releasing USEFOR, without the advice from USEPRO, lead to people following
the syntax, but doing their own thing, e.g. with path diagnostics, leading
to incompatibilities in the wild?  This seems like a real danger to me.

As expected, there was much WG discussion about backwards compatibility with 
some of these features, and there were some areas with very limited flexibility.

If people usurp those methods, creating USEFOR-compliant, but non-interoperating 
implementations due to the lack of guidance of USEPRO (or some early draft of 
USEPRO), then there are not going to be easy alternatives.

Do others think this is a danger?

---

From: Alexey Melnikov <[email-masked]>
Date: Fri, 08 Dec 2006 18:32:51 +0000

Forrest J. Cavalier III wrote:

>> The WG's belief is that the -usefor document contains enough 
>> information to write software that parses an Usenet article and says 
>> whether or not it conforms to the format.
>
> I and the USEPRO editor, two of 7 participants in the last call poll, 
> argued that USEFOR must wait until USEPRO.

"Must wait until USEPRO" is not the same thing as "USEFOR must have a 
normative reference to USEPRO".
There is a consensus to delay publication of USEFOR until after USEPRO 
is done (if it ever gets done). The "if it ever gets done" is an 
important part.

> My objection was more based on

---

From: [email-masked] (Ralph Babel)
Date: Sat, 9 Dec 2006 12:27:39 +0100 (CET)

Forrest J. Cavalier III wrote:

> 1. Can we clarify how the "WG decided there
>    should not be" a normative reference?
>
> I and the USEPRO editor, two of 7 participants in the
> last call poll, argued that USEFOR must wait until USEPRO.

The way I read <[email-masked]> ...

  http://www.imc.org/ietf-usefor/mail-archive/msg03158.html

..., you can probably add Richard and me.

---

From: Harald Alvestrand <[email-masked]>
Date: Sun, 10 Dec 2006 21:09:57 +0100

> 1. Can we clarify how the "WG decided there should not be" a normative

Note: Every single version of the -usefor document since -00 has had USEPRO 
as an informative reference.

WG Last Call on USEFOR, reported June 15:

<http://www.imc.org/ietf-usefor/mail-archive/msg03158.html>

Note the second question - on whether or not to send it to the IESG.
I see that at this time, I assumed that there would be a REF hold on USEPRO 
(which would have been the case had the reference been normative).

Second WG Last Call on USEFOR, reported Sept 15:

<http://www.imc.org/ietf-usefor/mail-archive/msg03431.html>

During this Last Call, we had an explicit discussion of this issue starting 
September 9 of this year, subject line "Is the USEFOR->USEPRO dependency 
normative?". I did not see any consensus of the WG that there should be a 
change, but very few people spoke up at all at this time.

I have not been able to find a previous discussion of the issue using the 
keywords "USEFOR" and "normative".

                 Harald

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 15 Dec 2006 02:07:07 +0100

Harald Alvestrand wrote:

[...]
>               Harald and Alexey

Thanks.

---

From: Sam Hartman <[email-masked]>
Date: Mon, 18 Dec 2006 15:11:44 -0500

>>>>> "Harald" == Harald Alvestrand <[email-masked]> writes:

    Harald> Greetings, we (the chairs) hvave discussed the IESG
    Harald> feedback on draft-ietf-usefor-usefor.

    Harald> There are two DISCUSS comments that we'd like to discuss
    Harald> further on this document.

    Harald> Sam Hartman filed this:

    >> following advice related to path, injection-*, and control from
    >> that document.  Parsing these fields without usepro is
    >> similarly difficult.

    Harald> The WG had a debate specifically about whether there
    Harald> should be a normative reference to USEPRO, and decided
    Harald> that there should not be one.  The WG's belief is that the
    Harald> -usefor document contains enough information to write
    Harald> software that parses an Usenet article and says whether or
    Harald> not it conforms to the format.

    Harald> We believe that the construction of the content of those
    Harald> fields is not a normative reference for the format - any
    Harald> more than the RIR allocation rules for IP addresses forms
    Harald> a normative reference for the definition of the IP header.
I tend to agree with Russ and Forrest here.  I don't think you ever
want to have people implement usefor without usepro and by not making
the reference normative you are sending the mesage that's OK.  I do
not think I could construct an article given only usefuor; parsing is
not enough for interoperability.

    Harald> If there are specific instances of parsing an Usenet
    Harald> message (as opposed to acting on the content of a message)
    Harald> that you think is difficult without USEPRO, we'd like to
    Harald> have the details, so that we can discuss the specifics.

    >> messages.  Very late, we realized that the real issue is not
    >> that these requirements are difficult for new posting agents,
    >> but that they are difficult when a email message is injected
    >> into usenet.  First, I think more discussion of whether that is
    >> the right approach for gateways is needed.

    Harald> It is definitely not the right approach.  A normal email
    Harald> message will not be a valid Usenet message - if only
    Harald> because it lacks a Path: header and a Newsgroups: header.

    Harald> The WG has discussed the problem of email-to-news
    Harald> gateways, and made an explicit decision in at least one
    Harald> other case to exclude wording that would seem to be
    Harald> normative for such gateways. We believe that this problem
    Harald> is out of scope for this document, and is (as we read the
    Harald> charter) out of scope for the present charter of the
    Harald> USEFOR WG.

    Harald> However, we have received a proposal for clarifying text
    Harald> about the relationship between the USEFOR format and the
    Harald> mail format; we'll consider this for inclusion at the end
    Harald> of section 2.1 of the draft:

    >> This draft places additional restrictions on RFC2822 message
    >> format rules, in order to be compatible with deployed NetNews
    >> agents.  An entity that generates messages compliant with
    >> RFC2822 might not always generate messages compliant with this
    >> specification, although certainly a single code-base can
    >> generate messages according to both sets of rules if desired.

    >> Gatewaying between email and news is not part of this
    >> specification, but we note that such a gateway would in all
    >> cases have to do more than just copying the message. Ensuring
    >> that the message satisfies the constraints here would be part
    >> of the job of such a gateway.

    Harald> Would this satisfy the DISCUSSant on this point?

Yes.

    Harald> Back to the DISCUSS text:

    >> Second, regardless of what we say here, it will be reasonably
    >> common for messages to be injected that do not meet these
    >> requirements.  In the interests of interoperability we need to
    >> discuss what problems can result and make recommendations for
    >> liberal receivers that minimize interoperability problems.

    Harald> The WG discussed the problem of illegal messages and
    Harald> decided that it was not possible or reasonable to specify
    Harald> handling of messages that do not conform to this syntax in
    Harald> the syntax document. This would also violate long-standing
    Harald> practice in, for instance, RFC 2822.  There is a document
    Harald> on the WG's document list - USEAGE - where such
    Harald> considerations would be appropriate. (However, note that
    Harald> the state of the WG is not such that this document will be
    Harald> finished soon, if at all. See later note.)

I guess what I don't understand here is why there is not a sufficient
justification for "receivers SHOULD be able to handle messages that
are valid RFC 2822 messages but that do not meet all the additional
restrictions of this specification."  I'm particularly thinking of the
generic message header restrictions not the requirements for
Newsgroups: and Path:.  I understand you cannot do anything with a
usenet article without those two headers.  There seems to be an
interoperability justification for being liberal in this area and I
have not seen an argument advanced why a SHOULD would be
inappropriate.  I've seen arguments about why some implementations
might not choose to follow such a SHOULD; while that makes sense, it
is not an argument against the SHOULD.

---

From: Russ Allbery <[email-masked]>
Date: Mon, 18 Dec 2006 14:09:29 -0800

Sam Hartman <[email-masked]> writes:

> justification for "receivers SHOULD be able to handle messages that are
> valid RFC 2822 messages but that do not meet all the additional
> restrictions of this specification."

Some valid messages under RFC 2822 cannot be transmitted via NNTP, as they
break fundamental parts of the protocol.  For example, RFC 2822 allows
escaped spaces in the LHS of a message ID, whereas NNTP commands
frequently contain message IDs as arguments and always treat a space as an
argument separator with no escaping possible.  Given the ubiquity of NNTP
as a Netnews transport, I don't believe it makes sense to permit articles
that can never be transmitted over NNTP.

Other valid messages under RFC 2822 use corners of RFC 2822 that are only
seen in test messages (such as omitting the space after the colon of a
header field), and while we could make a statement that receivers SHOULD
be able to handle such messages, realistically such a statement will only
make the standard at odds with existing implementations that will probably
never be fixed.  Since such messages do not exist in practice, the changes
of anyone making the required changes to existing news software to support
them seems very low, regardless of what we say in a standards document.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: Sam Hartman <[email-masked]>
Date: Mon, 18 Dec 2006 18:05:43 -0500

>>>>> "Russ" == Russ Allbery <[email-masked]> writes:

    Russ> Sam Hartman <[email-masked]> writes:
    >> I guess what I don't understand here is why there is not a
    >> sufficient justification for "receivers SHOULD be able to
    >> handle messages that are valid RFC 2822 messages but that do
    >> not meet all the additional restrictions of this
    >> specification."

    Russ> Some valid messages under RFC 2822 cannot be transmitted via
    Russ> NNTP, as they break fundamental parts of the protocol.  For
    Russ> example, RFC 2822 allows escaped spaces in the LHS of a
    Russ> message ID, whereas NNTP commands frequently contain message
    Russ> IDs as arguments and always treat a space as an argument
    Russ> separator with no escaping possible.  Given the ubiquity of
    Russ> NNTP as a Netnews transport, I don't believe it makes sense
    Russ> to permit articles that can never be transmitted over NNTP.

    Russ> Other valid messages under RFC 2822 use corners of RFC 2822
    Russ> that are only seen in test messages (such as omitting the
    Russ> space after the colon of a header field), and while we could
    Russ> make a statement that receivers SHOULD be able to handle
    Russ> such messages, realistically such a statement will only make
    Russ> the standard at odds with existing implementations that will
    Russ> probably never be fixed.  Since such messages do not exist
    Russ> in practice, the changes of anyone making the required
    Russ> changes to existing news software to support them seems very
    Russ> low, regardless of what we say in a standards document.

And that would be an argument against should.
Perhaps you could convince Mark, but I doubt it:-)

Anyway, I'm happy to remove that part of my discuss.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 19 Dec 2006 15:26:55 GMT

>is not an argument against the SHOULD.

Whenever the WG has discussed this topic, it has expressed a very clear
view against any such SHOULD. Why should newsreaders be even RECOMMENDED
to accept constructs which have never ever appeared in News Articles?

I think the view has, rather, been that the next revision of RFC 2822
should seriously consider moving a few of the key differences from the
'generate' syntax (MAY generate) into the 'obsolete' syntax (MUST NOT
generate but MUST accept). The space after the colon is a case in point.
Neither email messages nor news articles are ever generated with that
space missing, so moving it into the obs syntax would have no practical
effect.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5