Subject: Old ticket status - USEFOR - Jan 3, 2006

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 03 Jan 2006 13:24:37 +0100

Happy new year, folks!

At the moment, I think we have 6 tickets left from "the old stuff":

1032 USEFOR general: Document changes from RFC 1036

   Text in Appendix B. Please speak up if this is not sufficient, or needs
   modification - otherwise, I'll just close this.

1047 USEFOR 3.1.6: Path field delimiters and components

   Proposed text incorporated in -06.

   Still open whether USEFOR should specify syntax that allows one to
   tell the difference between a path-identity and a diagnostic.

1078 USEFOR 3.1.5: Need to describe meaning of Newsgroups header field in 
email

   was "no consensus" Nov 28. I think "proposed no change" would be a
   reasonable current status, based on the 2 comments at that time.
   That is - do not say anything, leave this for USEAGE.

1080 USEFOR 3.2.14 - MIME parameters for Injection-Info and Archive header 
field need more text/updated syntax

  was "text proposed" on Nov 28, text incorporated in -06.
  If this text is not OK, speak up - otherwise, I'll just close this.

1084 USEFOR 2.1, 3 Names for ABNF productions redefining 822 constructs

  was "proposed no change" on Nov 28

1132 USEFOR 3.1.6: Outlaw IP address in path-identity?

  was "text accepted" on Nov 28 - this is closely enough linked to 1047
  that they should be considered together, but I regard the specific
  issue as closed.

I'll open some new tickets with the issues subsequently raised on the list.

                     Harald

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 03 Jan 2006 19:53:01 +0100

>    Text in Appendix B. Please speak up if this is not
>    sufficient, or needs modification - otherwise, I'll
>    just close this.

Add:

| The obsolete convention to interpret subjects starting
| with the word "cmsg" as control message was removed.

> 1078 USEFOR 3.1.5:
[...]
> do not say anything, leave this for USEAGE.

+1

> 1080 USEFOR 3.2.14
[...]
> just close this.

+1

> 1084 USEFOR 2.1, 3
[...]
> "proposed no change" on Nov 28

Without some fresh opposition supporting id-unique @ id-domain
etc. (instead of id-left @ id-right) there's nothing to do.
If somebody else doesn't like id-right I post my variant again.

> 1132 USEFOR 3.1.6:
[...]
> I regard the specific issue as closed.

+1

#1047 moved to a separate article.

---

From: Frank Ellermann <[email-masked]>
Date: Tue, 03 Jan 2006 20:00:35 +0100

[splitting my attempted all-in-one-reply, ideas about #1047]

[...]
> Still open whether USEFOR should specify syntax that allows
> one to tell the difference between a path-identity and a
> diagnostic.

As it is the syntax is ambiguous.  We've seen at least two
more complete solutions, one by Charles, another by me, but
it's somewhat hard to get the ABNF right while it's still
unclear what we really want.

Points I recall:  Compatible with existing practice, that's
AFAIK ...!...!MISMATCH!... or ...!POSTED!... or ...!... and
we add ...!!... (as shorthand for ...!SEEN!... but without
mentioning / allowing this long form).

Bareword (funny underscores) vs. FQDN with a <toplabel>.
IPv4 or IPv6 only in conjunction with ...!ip!MISMATCH!...

Minimize potential FWS, it can come after ...!! or after
...!...!MISMATCH! or after ...!POSTED! or after ...! but
not elsewhere.

No "multi-entries", they are too confusing as part of the
syntax.

Could we somehow agree that that's precisely what we want ?
Then Charles could cook up some corresponding syntax, and
I debug it, or vice versa,

[...]

If we could agree on the requirements for #1047 the solution
should be clear, and we're almost ready.  I think (not sure)
the key is to consider the four possible constructs...

...!old-path
...!!old-path
...!...!MISMATCH!old-path
...!POSTED!old-path

...as units.  With "diagnostics" only appearing as ... before
a MISMATCH.  Or add a fifth construct...

...!...!SEEN!old-path

...also with a second slot for "diagnostics" (IIRC Russ' idea).

The <path-identity> always in the first slot, no IP.  The only
[FWS] either before !old-path, or behind "!" before old-path.

It MUST be possible to get this right, assuming that it really
is what we want (the main problem, what should the Path be ?).

                          Bye, Frank

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Tue, 03 Jan 2006 22:16:29 +0100

> ...also with a second slot for "diagnostics" (IIRC Russ' idea).

In addition:
What's out there is a mess of

...!server!diagnostic!....

...!server!server!....

...!server!POSTED!.... (possibly diagnostic before the final-entry, not 
sure)

I'm not sure if we ever have

...!server!diagnostic!diagnostic!....

Should the syntax bless, curse, grudgingly permit, or ignore these 
practices?

                      Harald

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 4 Jan 2006 10:44:38 GMT

>...!server!POSTED!.... (possibly diagnostic before the final-entry, not=20
>sure)

>Should the syntax bless, curse, grudgingly permit, or ignore these=20
>practices?

So long as you can tell which is a server and which is a diagnostic and
which is a keyword, it does not really matter whether these oddities are
allowed or not. In the interests of tidiness, one would allow each
"server!" to be followed by zero or one "diagnostic!" (maybe with
keyword), and for sure USEPRO will tell you to construct just that. But
whether you write all that into the syntax is a secondary matter (my last
proposed syntax did do it).

But for sure, we don't expect software which reads this header to waste
time parsing it in that detail - all it needs to do it to pick out the
delimiters, pick out the stuff between them (it might or might not worry
about illegal characters in the 'stuff'), and then see if the 'stuff' is
recorded in its 'sys' file as a place that it need not relay the article
to.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 4 Jan 2006 11:28:41 GMT

>| with the word "cmsg" as control message was removed.

No, you cannot say that in Appendix B of USEFOR, because the change is not
mentioned in USEFOR anywhere.

It IS mentioned in USEPRO, which has its own list of differences from
1036, and I will see that it gets mentioned in that list.

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 4 Jan 2006 11:25:08 GMT

>   modification - otherwise, I'll just close this.

It says:

      Whitespace is permitted in Newsgroups header fields.

One might say "Whitespace (and hence folding) is permitted ..."

One might also add that it SHOULD NOT be generated (yet).

BTW, USEPRO has always said that "SHOULD NOT be generated" restriction
might be lifted in some future standard. Should we say that in 3.1.5, just
to encourage implemetors to take the "MUST be accepted" seriously? Ditto
for the comments in the References header.

And on the subject of 3.1.5, I think I have already mentioned that the
wording about "harming propagation" is not strictly correct as it is
written (was that one of the things Ken agreed to fix?).

>  If this text is not OK, speak up - otherwise, I'll just close this.

Yes.

BTW, some way back I asked for an extra parameter
      authentication = whatever
to indicate how the user had identified himself to the injecting agent.
Discussion went as far as concluding that it was "authentication" as
opposed to "authorization" that was required, but it never got alloated a
ticket number. But if it it too late to raise it again now, then no
matter.

All other actions mentioned seem OK to me.

---

From: Ken Murchison <[email-masked]>
Date: Wed, 04 Jan 2006 08:44:12 -0500

> written (was that one of the things Ken agreed to fix?).

It doesn't look like I've made any changes.  What was your suggested text?

-- 
Kenneth Murchison
Project Cyrus Developer/Maintainer
Carnegie Mellon University

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 04 Jan 2006 14:51:40 +0100

> No, you cannot say that in Appendix B of USEFOR, because the
> change is not mentioned in USEFOR anywhere.

It is a side effect of 3.1.4 <subject> and 2.2 <unstructured>,
and adding the proposed statement to appendix B will fix the
issue that it's not yet explicitly mentioned anywhere.

Appendix B lists all differences in the "article format" from
1036, and this is a very important difference.  I used the old
"Subject: cmsg cancel" as feature on one news server in 2002,

I got an "administrative restriction" on IIRC GMaNe about it
later, and the Google archive ignored all articles with this
form of a subject, even including "Subject: Re: cmsg cancel"
in ordinary articles (no control messages).  That's broken,
and USEFOR fixes it, it has to be mentioned in appendix B.

> It IS mentioned in USEPRO

That's good, it's important enough to mention it everywhere.

Testing to change the subject here. new subject:
cmsg cancel (was: Old ticket status - USEFOR - Jan 3, 2006)

--- new ---
Well, if I didn't screw up that simply didn't work, I don't
see that article <[email-masked]> on GMaNe:

http://mid.gmane.org/[email-masked]

Maybe you got it, maybe you didn't, the list archive doesn't
work in real time, so far I don't see it:

http://www.imc.org/ietf-usefor/mail-archive/maillist.html

Maybe it's a GMaNe bug.  But above all it's a 1036 oddity
removed by USEFOR for good.  Therefore we MUST mention this.

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 04 Jan 2006 16:08:56 +0100

--- new ---

Reposted, apparently the string "cmsg cancel" anywhere in the
subject, even as "Subject: #1032 (was: cmsg cancel)", does not
work.  That's the third attempt to send this article after
http://mid.gmane.org/[email-masked] and
http://mid.gmane.org/[email-masked]

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 4 Jan 2006 13:29:27 GMT

>unclear what we really want.

Exactly. The things we need to know (in decreasing order of urgency) are:

1. Do we want to be able to tell the difference betweeen a diagnostic and
a server?

1a. If so, is another keyword ("SEEN" has been suggested) the way to do
it?

2. What is the full list of keywords we want (in particular, do we omit
"MATCH")? Or do we make the list extensible as Harald proposed?

3. Which keyword notation? (!foo.bar!MISMATCH! or !foo.bar..MISMATCH!, or
whatever else).

4. Do we enforce ':' ONLY in Ipv6Addresses (whether by explicit syntax or
MUSTard)?

5. Remaining syntactic niggles, such as maybe disallowing
!diagnostic!diagnostic! and where [FWS] is allowed (certainly before
delimiters, and maybe afterwards as well).

>I debug it, or vice versa,

Indeed. Even answering the first two questions would enable some progress.

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 5 Jan 2006 12:35:40 GMT

>http://mid.gmane.org/[email-masked]

I received <[email-masked]> as well as
<[email-masked]> (which is the article I am replying to).

I did not see <[email-masked]>

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 5 Jan 2006 16:40:17 GMT

It was on Dec 19 when commenting on your list of diffs in 06, when I said:

>>    Folding the Newsgroups header field over several lines has been shown
>>    to harm propagation significantly......
> 
>Actually, it is worse than that. Even WSP will break existing usage, and
>it is more than "significantly". I think you have to make it clear that
>this is a newly introduced feature, hence "MUST accept" but "MUST NOT
>generate" (yet, for some value of "yet" :-) ).

[though I should have said "...SHOULD NOT generate (yet...)..."]

The present text says

   Folding the Newsgroups header field over several lines has been shown
   to harm propagation significantly.  Folded Newsgroups header fields
   SHOULD NOT be generated, but MUST be accepted.

and there are two things wrong with that:

1. It is not just folding that doesn't work, it is any whitespace anywhere
amongst the newsgroups.

2. It has not been "shown to harm propagation", but rather it never
propagated at all in strictly conforming software, because RFC 1036 just
did not permit it (and the decision to introduce it was one of the first
things the WG decided all those years ago).

So it would be more accurate to say something like:

   The [FWS] within the list of <newsgroup-name>s is newly introduced in
   this standard and will not be accepted by most current implementations.
   Therefore, although it MUST now be accepted, it SHOULD NOT be generated
   [though this restriction may well be removed in a future version of
   this standard].

where the words in [...] is an addition that I would like to see, just to
encourage implementors to take the MUST accept seriously (there has been
such wording in USEPRO for some time, but USEPRO is not the place for it).

And the restriction on <comment>s in References headers might say
something similar.

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 11 Jan 2006 11:47:20 +0100

Harald Tveit Alvestrand wrote:

> ...!server!POSTED!.... (possibly diagnostic before the
> final-entry, not> sure)

AFAIK my UA could submit articles as xyzzy!frank (at least in
a pre-s-o-1036 parallel universe), and that would then result
in server!POSTED!xyzzy!frank

That's shown in USEPRO 7.3.1, no problem for your third case.

Is your second case Russ' idea of unverified diagnostics,
only still without "SEEN", because that's not yet specified ?

For legacy formats we can only offer an unambiguous better
format.  The "multi-entry" is no real problem, a server is
free to pretend that its different identities talk with each
other.

And your first case ...!server!diagnostic!...

> Should the syntax bless, curse, grudgingly permit, or
> ignore these practices?

...would be cursed, if the diagnostics is an IP, because we
offer an unambiguous solution with MISMATCH or SEEN.  But I'm
of course not sure, I don't have a collection of "paths ITW".

Fresh attempt added below, Bill's parser would propose *"-"
instead of *( "-" ), but that's only an aesthetical nit.  I
hope the "features" are obvious.
                                Bye, Frank

path            =  "Path:" SP *WSP path-list tail-entry *WSP CRLF
path-list       =  *( path-entry "!" [FWS] )
path-entry      =  path-identity / path-verified / path-unverified /
                   path-mismatch / path-posted

path-verified   =  path-identity "!"             ; an additional "!"
path-unverified =  path-identity "!SEEN!"     path-diagnostic
path-mismatch   =  path-identity "!MISMATCH!" path-diagnostic
path-posted     =  path-identity "!POSTED!"   path-diagnostic

path-diagnostic =  path-identity / IPv4address / IPv6address
path-identity   =  ( 1*( label "." ) toplabel ) / path-legacy

path-legacy     =  1*( ALPHA / DIGIT / "-" / "_" )
tail-entry      =  path-legacy                   ; allows underscore

label           =  letdig [ *( letdig / "-" ) letdig ]
letdig          =  ALPHA / DIGIT                 ; compare RFC 3696

toplabel        =  ( [ label *( "-" ) ] ALPHA   *( "-" ) label   ) /
                   (   label *( "-" )   ALPHA [ *( "-" ) label ] ) /
                   (   label          1*( "-" )          label   )

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 12 Jan 2006 18:05:14 GMT

>in server!POSTED!xyzzy!frank

Yes, there are all sorts of things that might appear between the POSTED
and the <tail-entry>, such as sites you wanted to 'alias out',
"cybercancel"s, and even assorted hosts between you and the true injector
(such as 'xyzzy').

>of course not sure, I don't have a collection of "paths ITW".

Are you agreeing that the 'diagnostic' needs to be recognisable as such,
and that the presence of MISMATCH or SEEN a reasonable way to go about it.
If so, then maybe we have a consensus on that and can proceed to the
details.

But I think it a bit premature to be writing syntax if that consensus is
not there. Nervertheless I shall comment on your syntax.

>path-list       =  *( path-entry "!" [FWS] )

No, if we are not going to allow [FWS] on both sides of a '!', then Russ
expressed a preference for having it before the '!' (so a folded line will
always start with a '!' (after the WSP, of course).

>path-mismatch   =  path-identity "!MISMATCH!" path-diagnostic

OK, so you have omitted "MATCH", and I have no great problem with that.
However, the intention has always been that the MISMATCH/SEEN should come
_after_ the diagnostic, not before it.

>path-posted     =  path-identity "!POSTED!"   path-diagnostic

However, I don't think the thing after a POSTED is a diagnostic. In your
'xyzzy' example you (the owner of the host xyzzy) were asserting that the
article had passed through that host, so it is a <path-identity>.
Likewise, a 'cybercancel' in there has always meant "we pretend this
article passed through a host 'cybercancel'; if you don't want to receive
these cancels, then please arrange that 'cybercancel' is an alias for your
own host".  There is a feature in INN (maybe others) that facilitates that
usage (see description in USEPRO).

>path-diagnostic =  path-identity / IPv4address / IPv6address

Yes, that paves the way for minimizing the dead:beef problem, and the rest
of your syntax seems OK (modulo minor niggles).

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 12 Jan 2006 20:57:54 +0100

Charles Lindsey wrote:

> there are all sorts of things that might appear between the
> POSTED and the <tail-entry>, such as sites you wanted to
> 'alias out', "cybercancel"s, and even assorted hosts between
> you and the true injector (such as 'xyzzy')

Also the re-injection discussed in USEPRO.  Assuming that we
still want some kind of POSTED, Richard proposed to forget it.

> Are you agreeing that the 'diagnostic' needs to be
> recognisable as such, and that the presence of MISMATCH or
> SEEN a reasonable way to go about it. If so, then maybe we
> have a consensus on that and can proceed to the details.

IFF (sic!) we want diagnostics they have to be recognizable,
otherwise they would be bogus.  IIRC we had a poll about "IP
anywhere or only in diagnostics", there aren't too many ways
to get this right.

> I think it a bit premature to be writing syntax if that
> consensus is not there

Without syntax I'm lost.  Harald proposed "diagnostics" months
ago, and our previous ABNF attempts didn't properly reflect the
new concept.

> No, if we are not going to allow [FWS] on both sides of a
> '!', then Russ expressed a preference for having it before
> the '!'

With "!" at the end a top down parser knows that there should
be a folded continuation line.

It also guarantees that the "!!" case doesn't have the FWS in
the middle.  And it allows to take an old very long path, add
a space and be done with that part, before creating your own
Path: <path-entry> "!" line.  All old stuff in the next lines.

You can't swap it, you would either get "!" [FWS] "!" if you
try, or you're thrown back to our old attempts

> so a folded line will always start with a '!'

Yes, Russ said that's what an "!" as operator looks like, but
in a Path it's more like a comma.  You wouldn't put a comma at
the start of the line.

BTW, that's all about limiting the places where an [FWS] is
allowed in a path.  The only way to limit it is "at the end of
a path-entry".

> the intention has always been that the MISMATCH/SEEN should
> come _after_ the diagnostic, not before it.

No problem, swap it.  And remove <diagnostics> from POSTED if
that's not the idea.  I'm only trying to translate this debate
in ABNF, otherwise I don't see where potential traps might be -
like IPv4.MISMATCH also matching <path-identity> in Richard's
fallback plan.

With your changes - ignoring the [FWS] for the moment until we
have a state where it's again possible to do something for it -
I would get:

path-verified   =  path-identity "!"             ; an additional "!"
path-unverified =  path-identity "!" path-diagnostic "!SEEN"
path-mismatch   =  path-identity "!" path-diagnostic "!MISMATCH"
path-posted     =  path-identity "!POSTED"

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Fri, 13 Jan 2006 11:50:29 +0100

> like IPv4.MISMATCH also matching <path-identity> in Richard's

You're probably right to translate it into ABNF, Frank.
I don't know whether Charles means "after" as "prepended afterwards" or "to 
the right of", which is two opposite meanings; USEPRO-04 says:

   1. It MUST establish the trusted identity of the source of the
      article and compare it with the leftmost <path-identity> of the
      Path header field's content. If it matches it MUST then prepend
      its own <path-identity> and a '!!' <path-delimiter> to that
      content. If it does not match then it prepends instead two entries
      to that content; firstly the true established <path-identity> of
      the source followed by a '!', the <path-keyword> "MISMATCH" and a
      further '!', and then, to the left of that, its own <path-
      identity> followed by a '!!' <path-delimiter> as usual.

It took me multiple readings to parse that as something that resulted in 
"foo!!1.2.3.4!MISMATCH!", but now I agree that that's what Charles seems to 
be trying to say.

                     Harald

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 16 Jan 2006 11:09:05 GMT

>You're probably right to translate it into ABNF, Frank.

OK, does that now mean we are agreed that diagnostics need to be
syntactically identifiable (I see that Russ now accepts that point) and we
are moving on to fixing the details (presumably using the keywords
MISMATCH and SEEN, since no other method seems to be on the table)?

Next question. Are MISMATCH and SEEN (plus the '!!') sufficient, or do we
want an open-ended extensible list as Harald suggested at one stage
(nobody else has picked up that idea AFAICS, though I can see it has some
possibilities)? Default answer is stick with just those two.

And then there is the question of the notation for the keywords

version 1:
   Path: downstream.example.com!123.123.123.123!MISMATCH
   !injector.example.com!POSTED!not-for-mail
version 2:
   Path: downstream.example.com!123.123.123.123..MISMATCH
   !injector.example.com!POSTED!not-for-mail

Where version 1 is what we have been using mostly in recent discussions,
and is what is in the present draft, and hence is the default the version
unless we consciously decide to change it. Does anyone want to do that (FX
timidly raises my own hand)?

Once those two questions are decided (or if we stick with the default),
then I can dust down my syntax and post it again.

>the right of", which is two opposite meanings; USEPRO-04 says:

Yes, it is easy to confuse oneself when building a list from right to left
:-( .

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 19 Jan 2006 17:05:50 +0100

> <[email-masked]> (which is the article I

Okay, I checked the official list archive for further lost
articles from GMaNe's POV, but apparently the list of losses
is complete:

One "Subject: cmsg cancel" never made it to the list, and
one "Subject: #1032 (was: cmsg cancel)" made it to the list,
but not back to GMaMe:

<http://www.imc.org/ietf-usefor/mail-archive/msg02708.html>

So far that's three additional points for (#1032) appendix B:

== 1: cmsg ==

| The old convention about interpreting all articles with a
| subject starting with a string "cmsg " in the header field
| body as control message was obsoleted by s-o-1036, and it
| has been completely removed by this standard.

With an explanation in 3.1.4 "Subject":

| The header field body of subjects SHOULD NOT be interpreted
| by news servers, it is <unstructured> text.

Complete rant in any desired length up to ten pages quoting
RFC 4096 on demand, but an absolute minimum is stated above.

== 2: path punctuation ==

| The only recognized delimiter in a path is an exclamation
| mark reflecting common practice already stated in s-o-1036.
| This allows underscores in legacy names and colons in IPv6
| address literals.

== 3: path identities ==

| Path identities, formerly known as relayer names, are now
| restricted to complete Internet domain names for hosts.
| This reflects a requirement in s-o-1036.  Legacy constructs
| are still accepted.
                              Bye, Frank