Subject: #1047 - Path Syntax

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 10 Nov 2005 20:05:23 GMT

Things have gone quiet. In an effort to get things moving, I have been
through the outstanding issues and extracted those which appear to be
holding us up. Here is where I think we are at with the Path Syntax:

There are two orthogonal choices for the WG to make in the following
proposal:

1. Syntactic style for the <source-keyword>s "MISMATCH" etc:

version 1:
   Path: downstream.example.com!123.123.123.123!MISMATCH
   !injector.example.com!POSTED!not-for-mail
version 2:
   Path: downstream.example.com!123.123.123.123..MISMATCH
   !injector.example.com!POSTED!not-for-mail

Observe that POSTED has '!'s on either side in both versions, because
peering sites still need to be able to recognize that articles should
not be sent back to injector.example.com.

2. The list of <source-keyword>s.

The proposed set is "SEEN", "MATCH", "MISMATCH". The WG may want to
reduce this (MATCH is redundant because of "!!") or extend it (Harald
even suggested letting sites invent additional ones).

I would suggest that we resolve these two main issues first, before
proceeding to any minor tuning.

3.1.6  Path
 
   The Path header field indicates the route taken by an article since
   its injection into the Netnews system.  Each agent that processes an
|  article is required to prepend a <path-entry> to this header field
   body. .......

[1st version of syntax, following the usage of MISMATCH in the present
USEFOR.]

   path = "Path" ":" SP [FWS] *( path-entry [ source-entry / "!" ] )
             tail-entry [FWS] CRLF

   path-entry = 1*( path-identity [FWS] "!" ) [ "POSTED" [FWS] "!" ]

   source-entry = source-identity [FWS] "!" source-keyword  [FWS] "!"

   source-keyword = "SEEN" / "MATCH" / "MISMATCH"

[2nd version of syntax, with MISMATCH etc incorporated into the
identity.]

   path = "Path" ":" SP [FWS] *( path-entry [ source-entry / "!" ] )
             tail-entry [FWS] CRLF

   path-entry = 1*( path-identity [FWS] "!" ) [ "POSTED" [FWS] "!" ]

   source-entry = source-identity source-keyword [FWS] "!"

   source-keyword = ( "..SEEN" / "..MATCH" / "..MISMATCH" )

[syntax common to both versions.]

   path-identity = fqdn / bareword

   fqdn = 1*( label "." ) toplabel

   label = alphanum / alphanum *( alphanum / "-" ) alphanum

   toplabel = [ label *( "-" ) ] ALPHA [ label *( "-" ) ]    ; at least one ALPHA

   alphanum = ALPHA / DIGIT

   bareword = ALPHA / ALPHA *( alphanum / [ "-" / "_" ] ) alphanum )

   tail-entry = fqdn / bareword

[how about tail-entry = 1*( alphanum / "-" / "_" / "." ) ?]

   source-identity = fqdn / IP-address / bareword

   IP-address = IPv4address / IPv6address ;  see [RFC 3986]

   Each <path-entry> (but not the <tail-entry>) indicates, from right to
   left, the successive sites through which the article has passed.  The
   keyword "POSTED" indicates that the site to its left injected the
   article.

   A <source-entry> indicates the true source from which the site to its
   left acquired the article, with the <source-keyword> showing its
   relationship with the <path-entry> inserted by the site to its right
   thus:
       MATCH    they are asserted to be the same, modulo known aliases,
       MISMATCH they are asserted not to be the same,
       SEEN     no asertion is made.
   The presence of two "!"s in succession (the second being an
   alternative to a <source-entry>) is also an assertion that the true
   source matched the <path-entry> to the right (hence the use of the
   <source-keyword> "MATCH" is redundant, and it was only included for
   completeness).

   The full procedure for constructing a <path-entry> as well as the
   preferred format to be used in a <path-identity> are discussed in
   [USEPRO].

      NOTE: An <IPv6address> is the only place where a ":" can occur
      within a <path-entry>. Since some current news-servers may treat
      ":" as a delimiter (in addition to the customary "!"), it is
      inadvisable for any <bareword> to be composed only from
      hexadecimal digits.

      NOTE: Historically, the <tail-entry> .........

      NOTE: Although case-insensitive, it is intended that "SEEN",
      "POSTED" and "MISMATCH" should be in upper case, to distinguish
      them from <bareword>s and the components of <fqdn>s which are
      traditionally in lower case.

In the following examples, "upstream.example.com" is a site which has
just sent an article (or so it is claimed) to "downstream.example.com".
"123.123.123.123" is the IP address from which downstream received it,
which may or may not be the IP address of upstream. The examples show the
full <path-entry> inserted by downstream, followed by (at least the start
of) the <path-entry> inserted by upstream.
 
1. downstream.example.com!123.123.123.123..SEEN!upstream.example.com...
2. downstream.example.com!123.123.123.123..MISMATCH!upstream.example.com...
3. downstream.example.com!123.123.123.123..MATCH!upstream.example.com...
4. downstream.example.com!!upstream.example.com...
5. downstream.example.com!upstream.example.com...
 
All the diagnostic <source-entry>s are immediately distinguishable from
the <path-entry>s by the presence of a big uppercase keyword.
 
Even if you thought 123.123.123.123 was a site to which you were not
supposed to send articles, no current implementation would recognize it as
such in any of those examples (though that would not be the case with
the version 1 syntax).
 
Ex1. Downstream recorded that 123.123.123.123 was the source of the
article "just for the hell of it". It makes no claim to have verified that
it belonged to upstream.
 
Ex2. Downstream recorded that 123.123.123.123 was the source of the
article and asserted that it no way matched any known address associated
with upstream. Some current practice uses almost exactly that notation.

Ex3. Downstream recorded that 123.123.123.123 was the source of the
article and asserted that it was indeed a known address used by upstream.
In which case you may ask why it bothered to clutter the Path by quoting
it in full.

Ex4. Downstream noted the IP (whatever) of the source and asserts that it
was indeed a known address used by upstream. But it did not clutter the
Path with unnecessary detail. That is the notation intended for normal
use.

Ex5. Downstream made no checks and makes no assertions. That is
essentially current practice which we hope will be replaced by Ex4 over
time.

NOTE: The use of an IP address for the <source-entry> in the examples
does not imply that a corresponding <fqdn> or <bareword> could not have
been used instead (indeed, it would be more user-friendly to have done
so). So Ex3 could as well have been written:

3. downstream.example.com!upstream.example.com..MATCH!upstream.example.com...

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Thu, 10 Nov 2005 17:14:04 -0800

I'm at the IETF this week, and at IEEE next week. So I can't follow up much.

But consider this for a moment:

Given the total inability this group has had on getting consensus on even 
minor tightenings or changes to the current models - what's the *minimum* 
amount of specification that can get this piece finished?

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 25 Nov 2005 18:21:29 GMT

In <215B75666F1D0DE711CBA5C5@B50854F0A9192E8EC6CDA126> Harald Tveit Alvestrand <[email-masked]> writes:

>I'm at the IETF this week, and at IEEE next week. So I can't follow up =
>much.

OK, but you seem to have done a good job on the IMA stuff.

>Given the total inability this group has had on getting consensus on even=20
>minor tightenings or changes to the current models - what's the *minimum*=20

Well I went through the list of issues on the website, and there seemed
three that still needed significant work:
    #1047
    #1080
    #1102
So I wrote proposals on the first two to try and get some movement
(haven't gotten around to the 3rd yet). The rest of the issues seemed like
they were mostly agreed (though the web site didn't seem quite up to date
on a few minor niggles).

Now, as to #1047, what are we to do?

We could say "leave it as it stands in the current draft, since we have no
agreement on anything further" (indeed, that would suit me fine).

However, the WG had agreed on some changes:
   a) That some entries were "diagnostic" (I prefer to call them
"source-entries" but that is a minor detail), and diagnostic entries were
to be the only ones where IP addresses were allowed - normal path-entries
were to be restricted to domains or barewords.
   b) The only place where a ":" was to be permitted was within IP
addresses (and specifically within IPv6 addresses). That would reduce the
possible occurrences of the "dead:beef" problem to an acceptable level.

If those changes are agreed, then they need text to describe them. Before
writing that text, there are some minor details to sort out:

   (i) Do we want it to be obvious which entries are "diagnostic" so that
humans can spot them (they are also to be ignored when deciding whether to
send an article to some peer). There was some desire expressed for this
(but note that diagnostics are not necessarily IP addresses, because some
articles may arrive other than by NNTP).

   (ii) If so, then what notation to use? Keywords seem the obvious
answer, since the present draft (and some existing usage) already uses
"MISMATCH".  Present usage seems to use IP addresses (at least)
diagnostically without any indication at all (contrary to (i)), so I
suggested using the keyword "SEEN". I also suggested "MATCH" to complete
the set, but that is not essential given that we also have the "!!"
notation. There was also a suggestion to leave the set of keywords
open-ended.

   (iii) We also need to decide the format of keywords. But at least, if
we cannot achieve consensus on that we can just fall back to the present
draft which uses, for example, "...!123.123.123.123!MISMATCH!...". The
alternative I have suggested is "...!123.123.123.123..MISMATCH!...", which
is closer to some existing usage.

But clearly, we MUST answer question (i), and if that point is agreed,
then we MUST answer question (ii) as well. Question (iii) is less
important.

The text I proposed is easily tweaked to fit any answers to those
questions, but I suggest tweaking details is less important than agreeing
the answers in principle.

As to #1080, the present draft is plainly buggy. The text I suggested is
the only show currently in town, as Frank has said, but probably omitting
the "gibbous" paragraph. The only alternative (minor tweaks excepted) is a
completely new notation for Injection-Info, and I don't think the WG
really has the stomache for that.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Mon, 28 Nov 2005 09:46:17 +0100

I think you have correctly identified the contentious items. I've added 
this text as a comment to the tracker.

> articles may arrive other than by NNTP).

My work on names used in current messages revealed fairly heavy use of path 
components that weren't hostnames.... the choices before us are (in my 
opinion):

- Declare that "anything that isn't a [known?] hostname can be treated like 
a diagnostic" (my proposal the last time around). This effectively blesses 
all existing practice - maximum interoperability, but makes life hard for 
parsers that are looking for diagnostics.
- Declare a specific syntax for diagnostic, and say "anything else is a 
hostname" - also blesses existing practice, but relaxes the rules for 
hostnames quite a bit
- Declare a specific syntax for diagnostic, keep a strict formulation for 
hostnames, and declare that anything else is a protocol error (that's how I 
interpret your proposals).

I think it's relatively simple to declare the field of "diagnostics" wide 
open and later say "here is a specific set of diagnostics, with a 
standardized meaning" - that could even be done in USEPRO, without anything 
in USEFOR. It's a little messy (problem should be obvious), but has the 
procedural advantage of letting us finish USEFOR without answering (ii) or 
(iii).

> notation. There was also a suggestion to leave the set of keywords

I think a more basic issue is to decide between 2-element diagnostics 
(!diag!keyword! or !keyword!diag!) and 1-element diagnostics 
(!diag..keyword!). That folds in your (iii)....
> then we MUST answer question (ii) as well. Question (iii) is less

Yes, we have to answer question (i) (in my opinion).

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 29 Nov 2005 14:25:08 GMT

>> humans can spot them  .....

>components that weren't hostnames.... the choices before us are (in my 

Yes, it happens, but I son't think we should do anything to encourage the
continuation of the practice. I recently proposed, as a compromise, that
the text in USEPRO should say that the domain name in a path-identity
SHOULD resolve to something in the DNS (which is the same thing as saying
that domain-names that don't resolve SHOULD NOT be used). That would not
stop it from happening, but would hopefully lead to a reduction in the
practice. Nothing breaks if it happens, but it does make life harder for
humans who are trying to diagnose some problem that has arisen. Would you
accept the wording I proposed?

>interpret your proposals).

Not quite. The existing practice will surely continue for some time, but
if we provide a notation to make it clear which entries are diagnostic,
then we should prefer that it should be used in future. I am sure I
could add some wording to my proposal to make it clear that such existing
practices should continue to be accepted. Nobody currently rejects articles
with unparseable Path headers (unless the errors are exceedingly gross)
because it is just too much work in a critical part of the code.

>(!!keyword! or !keyword!diag!) and 1-element diagnostics 

I prefer the 1-element form because it reduces the small possibility that
some agent would try to use the diagnostic as a path-identity (though that
agent would have to be pretty stupidly configured as well). It also
removes any possibility of confusing a keyword with a <bareword>.
However, as I said, the 2-element form is in the present draft, so it
would remain if we cannot agree to change it (and "POSTED" would still
have to come in the 2-element form because the thing to the left of it is
a genuine path-identity - not a diagnostic).

>Yes, we have to answer question (i) (in my opinion).

Indeed. My view is that diagnostics should be readily identifiable as
such.  Currently that is not the case. The fact that IP addresses are
currently 99% certain to be diagnostic is not written down anywhere, nor
is it widely understood (we only know because we have researched it). It
seems that the new notations proposed (e.g. a keyword SEEN) can improve
the situation if they become adopted over time, and can do so without
preventing any existing software from continuing to work.