Subject: ABNF

From: Frank Ellermann <[email-masked]>
Date: Thu, 22 Dec 2005 16:35:34 +0100

Hi, I tried to validate the ABNF:

| injection-info  =  "Injection-Info:" SP [CFWS] path-identity
|                    [CFWS] *(';' parameter) CRLF

s/';'/";"/

Bill's parser accepts fields =/ etc.  I just added anything
from 2822 we need not redefined by us on top of the collected
USEFOR ABNF.  

I got a "lines defined but not used" warning, please add this
to fields =/
                           Bye, Frank

;;;;;;;;;;;;;;;;; -2822- ;;;;;;;;;;;;;;;;;;;;;;;;

NO-WS-CTL       =       %d1-8 /         ; US-ASCII control characters
                        %d11 /          ;  that do not include the
                        %d12 /          ;  carriage return, line feed,
                        %d14-31 /       ;  and white space characters
                        %d127

text            =       %d1-9 /         ; Characters excluding CR and LF
                        %d11 /
                        %d12 /
                        %d14-127 /
                        obs-text

quoted-pair     =       ("\" text) / obs-qp

FWS             =       ([*WSP CRLF] 1*WSP) /   ; Folding white space
                        obs-FWS

ctext           =       NO-WS-CTL /     ; Non white space controls
                        %d33-39 /       ; The rest of the US-ASCII
                        %d42-91 /       ;  characters not including "(",
                        %d93-126        ;  ")", or "\"

ccontent        =       ctext / quoted-pair / comment

comment         =       "(" *([FWS] ccontent) [FWS] ")"

CFWS            =       *([FWS] comment) (([FWS] comment) / FWS)

atext           =       ALPHA / DIGIT / ; Any character except controls,
                        "!" / "#" /     ;  SP, and specials.
                        "$" / "%" /     ;  Used for atoms
                        "&" / "'" /
                        "*" / "+" /
                        "-" / "/" /
                        "=" / "?" /
                        "^" / "_" /
                        "`" / "{" /
                        "|" / "}" /
                        "~"

atom            =       [CFWS] 1*atext [CFWS]

dot-atom        =       [CFWS] dot-atom-text [CFWS]

dot-atom-text   =       1*atext *("." 1*atext)

qtext           =       NO-WS-CTL /     ; Non white space controls
                        %d33 /          ; The rest of the US-ASCII
                        %d35-91 /       ;  characters not including "\"
                        %d93-126        ;  or the quote character

qcontent        =       qtext / quoted-pair

quoted-string   =       [CFWS]
                        DQUOTE *([FWS] qcontent) [FWS] DQUOTE
                        [CFWS]

word            =       atom / quoted-string

phrase          =       1*word / obs-phrase

utext           =       NO-WS-CTL /     ; Non white space controls
                        %d33-126 /      ; The rest of US-ASCII
                        obs-utext

date-time       =       [ day-of-week "," ] date FWS time [CFWS]

day-of-week     =       ([FWS] day-name) / obs-day-of-week

day-name        =       "Mon" / "Tue" / "Wed" / "Thu" /
                        "Fri" / "Sat" / "Sun"

date            =       day month year

year            =       4*DIGIT / obs-year

month           =       (FWS month-name FWS) / obs-month

month-name      =       "Jan" / "Feb" / "Mar" / "Apr" /
                        "May" / "Jun" / "Jul" / "Aug" /
                        "Sep" / "Oct" / "Nov" / "Dec"

day             =       ([FWS] 1*2DIGIT) / obs-day

time            =       time-of-day FWS zone

time-of-day     =       hour ":" minute [ ":" second ]

hour            =       2DIGIT / obs-hour

minute          =       2DIGIT / obs-minute

second          =       2DIGIT / obs-second

zone            =       (( "+" / "-" ) 4DIGIT) / obs-zone

address         =       mailbox / group

mailbox         =       name-addr / addr-spec

name-addr       =       [display-name] angle-addr

angle-addr      =       [CFWS] "<" addr-spec ">" [CFWS] / obs-angle-addr

group           =       display-name ":" [mailbox-list / CFWS] ";"
                        [CFWS]

display-name    =       phrase

mailbox-list    =       (mailbox *("," mailbox)) / obs-mbox-list

address-list    =       (address *("," address)) / obs-addr-list

addr-spec       =       local-part "@" domain

local-part      =       dot-atom / quoted-string / obs-local-part

domain          =       dot-atom / domain-literal / obs-domain

domain-literal  =       [CFWS] "[" *([FWS] dcontent) [FWS] "]" [CFWS]

dcontent        =       dtext / quoted-pair

dtext           =       NO-WS-CTL /     ; Non white space controls
                        %d33-90 /       ; The rest of the US-ASCII
                        %d94-126        ;  characters not including "[",
                                        ;  "]", or "\"

message         =       (fields / obs-fields)
                        [CRLF body]

body            =       *(*998text CRLF) *998text

;;;;;;;;;;;;;;;;; fields ;;;;;;;;;;;;;;;;;;;;;;;;

fields          =       *(trace
                          *(resent-date /
                           resent-from /
                           resent-sender /
                           resent-to /
                           resent-cc /
                           resent-bcc /
                           resent-msg-id))
                        *(orig-date /
                        from /
                        sender /
                        reply-to /
                        to /
                        cc /
                        bcc /
                        message-id /
                        in-reply-to /
                        references /
                        subject /
                        comments /
                        keywords /
                        optional-field)

fields          =/ *( newsgroups /
                      path /
                      injection-date /
                      followup-to /
                      expires /
                      control /
                      supersedes /
                      distribution /
                      summary /
                      approved /
                      organization /
                      xref /
                      archive /
                      user-agent /
                      injection-info )

;;;;;;;;;;;;;;;;; USEFOR ;;;;;;;;;;;;;;;;;;;;;;;;

unstructured    =  1*( [FWS] utext ) [FWS]

from            =  "From:" SP mailbox-list CRLF

orig-date       =  "Date:" SP date-time CRLF

message-id      =  "Message-ID:" SP [FWS] msg-id [FWS] CRLF

msg-id          =  "<" id-left "@" id-right ">"
                   ; maximum length is 250 octets

id-left         =  dot-atom-text / no-fold-quote

id-right        =  dot-atom-text / no-fold-literal

no-fold-quote   =  DQUOTE
                      ( "." *mqtext /
                        *mqtext "." /
                        *mqtext mqspecial *mqtext )
                      DQUOTE

mqtext          =  atext / "." / mqspecial

mqspecial       =  "(" / ")" /      ; same as specials except
                   "<" /            ; "\" and DQUOTE quoted
                   "[" / "]" /      ; "." doubled and ">" omitted
                   ":" / ";" /
                   "@" / "," /
                   ".." / "\\" / "\" DQUOTE

no-fold-literal =  "[" *( mdtext / "\[" / "\]" / "\\" ) "]"

mdtext          =  %d33-61 /        ; The rest of the US-ASCII
                   %d63-90 /        ; characters not including
                   %d94-126         ; ">", "[", "]", or "\"

subject         =  "Subject:" SP unstructured CRLF

newsgroups      =  "Newsgroups:" SP newsgroup-list CRLF

newsgroup-list  =  [FWS] newsgroup-name
                   *( [FWS] "," [FWS] newsgroup-name ) [FWS]

newsgroup-name  =  component *( "." component )

component       =  1*component-char

component-char  =  ALPHA / DIGIT / "+" / "-" / "_"

path            =  "Path:" SP path-list CRLF

path-list       =  [FWS]
                   *( ( path-identity / path-keyword /
                        path-diagnostic ) [FWS]
                      path-delimiter [FWS] )
                   tail-entry [FWS]

path-identity   =  ( ALPHA / DIGIT )
                   *( ALPHA / DIGIT / "-" / "." / "_" )

path-keyword    =  "POSTED" / "MISMATCH"

path-diagnostic =  1*( ALPHA / DIGIT / "-" / "." / ":" / "_" )

tail-entry      =  path-identity

path-delimiter  =  "!" / "!!"

sender          =  "Sender:" SP mailbox CRLF

reply-to        =  "Reply-To:" SP address-list CRLF

comments        =  "Comments:" SP unstructured CRLF

keywords        =  "Keywords:" SP phrase *("," phrase) CRLF

injection-date  =  "Injection-Date:" SP date-time CRLF

references      =  "References:" SP [CFWS] msg-id *(CFWS msg-id)
                   [CFWS] CRLF

followup-to     =  "Followup-To:" SP ( newsgroup-list / poster-text )
                   CRLF

poster-text     =  [FWS] %d112.111.115.116.101.114 [FWS]
                   ; "poster" in lower-case

expires         =  "Expires:" SP date-time CRLF

control         =  "Control:" SP [FWS] control-command [FWS] CRLF

control-command =  verb *( [FWS] argument )

verb            =  token

argument        =  value

supersedes      =  "Supersedes:" SP [FWS] msg-id [FWS] CRLF

distribution    =  "Distribution:" SP dist-list CRLF

dist-list       =  [FWS] dist-name
                   *( [FWS] "," [FWS] dist-name ) [FWS]

dist-name       =  ALPHA / DIGIT
                   *( ALPHA / DIGIT / "+" / "-" / "_" )

summary         =  "Summary:" SP unstructured CRLF

approved        =  "Approved:" SP mailbox-list CRLF

organization    =  "Organization:" SP unstructured CRLF

xref            =  "Xref:" SP [FWS] server-name
                   1*( FWS location ) [FWS] CRLF

server-name     =  path-identity

location        =  newsgroup-name ":" article-locator

article-locator =  1*( %x21-27 / %x29-3A / %x3C-7E )
                   ; US-ASCII printable characters
                   ; except '(' and ';'

archive         =  "Archive:" SP [CFWS] ("no" / "yes")
                   *( [CFWS] ";" archive-param ) CRLF

archive-param   =  parameter

user-agent      =  "User-Agent:" SP 1*product [CFWS] CRLF

product         =  [CFWS] token [ [CFWS] "/" product-version ]

product-version =  [CFWS] token

injection-info  =  "Injection-Info:" SP [CFWS] path-identity
                   [CFWS] *(";" parameter) CRLF

host-value      =  dot-atom-text / [ dot-atom-text ":" ]
                   ( IPv4address / IPv6address ) ;  see [RFC 3986]

sender-value    =  mailbox / "verified"

lines           =  "Lines" ":" SP [FWS] 1*DIGIT [FWS] CRLF

---

From: Ken Murchison <[email-masked]>
Date: Thu, 22 Dec 2005 11:03:48 -0500

> I got a "lines defined but not used" warning, please add this

Both issues fixed.

-- 
Kenneth Murchison
Project Cyrus Developer/Maintainer
Carnegie Mellon University

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Fri, 23 Dec 2005 11:15:14 +0100

> I got a "lines defined but not used" warning, please add this

Thanks Frank!
That's one of the steps that the WG chair is supposed to make sure has been 
done before submission - very good to have it done!

               Harald

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 23 Dec 2005 11:23:42 +0100

> Both issues fixed.

Thanks.  I think we should add an "import interface" for all 21
terms defined elsewhere.  We use five sources 2045, 2231, 2822,
3986, and 4234, and the 2231 terms are redefinitions of 2045.

Readers would be hard pressed to merge these sources with our
ABNF to get something that's valid.  We could offer the source
(plus section) by 21 ABNF lines like (unverified examples):

               CRLF     = <see RFC 4234 section 2  > 
               token    = <see RFC 2231 section 2  >
               fields   = <see RFC 2822 section 3.6>

Grouped by RFC number this should help, not only for Bill's
validator.

--- host-value ---

I've seen another minor problem in the ABNF (in addition to
<article-locator> )

| host-value =  dot-atom-text / [ dot-atom-text ":" ]
|               ( IPv4address / IPv6address ) ;  see [RFC 3986]

Alternatives without grouping are discouraged in 4234.  To fix
that you could say:

| host-value  =  dot-atom-text 
|                [ ":" ( IPv4address / IPv6address ) ]

We discussed to replace ":" by a "better" separator, but I
forgot the details.  Could we use "[" ( IPv4 / IPv6 ) "]" ?

--- Lines ---

Bill's validator proposes "Lines:" instead of "Lines" ":", and
that's also the style already used for all other header fields.

                              Bye, Frank

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 23 Dec 2005 11:46:58 +0100

> |                [ ":" ( IPv4address / IPv6address ) ]

No, that's not the same before, sorry.  Next attempt:

| host-value  =  dot-atom-text
|             / ( [ dot-atom-tet ":" ]
|                 ( IPv4address / IPv6address ))

Won't win us a price for beautiful ABNF.  Third atempt:

| host-value  = dot-atom-text / IPv4address / IPv6address /
|               ( dot-atom-text ":" ( IPv4address / IPv6address ))

That's somewhat clearer.  If the ":" is a problem we could
try something completely different:

| IP-literal  = "[" ( IPv4address / IPv6address ) "]"
|
| host-value  = dot-atom-text / IP-literal /
|               ( dot-atom-text IP-literal )

That's not bad, or is it ?  Bye, Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 28 Dec 2005 19:10:24 GMT

>3986, and 4234, and the 2231 terms are redefinitions of 2045.

I am all for giving the hard-pressed reader as much help as possible.
There are many ways one might do this. At one extreme, one could include
all the cited rules from all the documents referred to as part of the
"collected syntax" appendix. Indeed, if you look at
draft-ietf-usefor-article-13, you will see that is exactly what was done
(non-normatively, of course), complete with a notation to show which RFC
each rule had been taken from, and whether it had been changed/adapted
from its original. At the other extreme, a simple list of the rules
'borrowed' from other documents. And there are many possible stages in
between.

>forgot the details.  Could we use "[" ( IPv4 / IPv6 ) "]" ?

The syntax using the ":" was taken from the presently used
NNTP-Posting-Host header, which <host-value> is intended to replace.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 30 Dec 2005 05:33:35 +0100

Charles Lindsey wrote:

> if you look at draft-ietf-usefor-article-13, you will see
> that is exactly what was done (non-normatively, of course),

Yes, I liked that, but a later consensus (?) was to drop it.

> At the other extreme, a simple list of the rules 'borrowed'
> from other documents. And there are many possible stages in
> between.

Let's pick the ABNF-prose imports, otherwise Bill would need
hours to check our ABNF.

 <host-value> 
> NNTP-Posting-Host header, which <host-value> is intended to

Okay.  But note that colon is a 2045 <tspecial>, so there will
a major difference for "FQDN or IPv4 only" vs. "FQDN:IPv4", in
the latter case the <host-value> MUST be in a <quoted-string>.

With square brackets for all IP literals (as in 2821) or at the
very minimum for IPv6 and beyond (as in STD 66) we could avoid
all potential IPv6 trouble:

If you (collective you) don't like this we should at least use
grouping notation as recommended in 4234 3.5:

The latter is equivalent with <host-value> in draft -06.  Bye