Subject: Re: #1080 - MIME parameters for Injection-Info and Archive header field need more text/updated syntax

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 1 Aug 2005 13:37:28 GMT

In <[email-masked]> Frank Ellermann <[email-masked]> writes:

>Okay.  Reason I stumbled over it:  It starts (optionally) with
><host> ":" followed by the IP.  Hard to read for an IPv6, e.g.
>need.more.cafe:cafe:cafe::cafe for a future gTLD cafe.  OTOH
>need.more.cafe:[cafe:cafe::cafe] could be clearer.

Ah! I see the problem. We have:

   host-value      =  dot-atom /
                      [ dot-atom ":" ] ( IPv4address / IPv6address )
                                       ;  see [RFC 3986]

and the ":" used as a separator can also appear in an IPv6address.

It is not actually ambiguous, but it is not tidy. We need a separator that
cannot occur in either a dot.atom (well, dot.atom.text maybe) or in an IP
address. That gives us:

    ( ) < > [ ] ; @ \ ,

none of which is really suitable. So maybe square brackets as you
suggested:

    posting-host = example.com
    posting-host = example.com[123.123.123.123]
    posting-host = example.com[beef:beef:beef:beef:....]
    posting-host = 123.123.123.123
    posting-host = [123.123.123.123]

probably make the [...] optional if there it no dot-atom[.text].

Or else we leave it (not-quite) ambiguous as it is.

Comments before I propose syntax?

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Bruce Lilly <[email-masked]>
Date: Wed, 3 Aug 2005 20:32:41 -0400

On Fri July 29 2005 14:40, Frank Ellermann wrote:
> 
> Bruce Lilly wrote:
>  
> > I proposed something over a year ago, based on the Received
> > field syntax.
> 
> Did I miss, forget it, or was it incomplete ?

You probably forgot it.  Search the archives for
Message-ID: <[email-masked]>
IIRC, you're the "examples-are-better-than-syntax" guy :-).

Special offer; three for the price of one!

> Your article 
> some days ago was "only" the general structure, but for the
> Injection-Info we need also the specific details, the most
> complex part is probably mail-complaints-to.

I'd still do Complaints-To as a bracketed URI list.  Same general
form as List-Post; plenty of reusable code for that.
 
---------
> > Please reread RFCs 2017, 2045, and 2231.
> 
> Come on, I did that more than a year ago after you told me so,
> and more than once.
> 
> > (e.g. 40-character limit).
> 
> BTW unnecessary, I hate new magic numbers without good reasons.

RFC 2017 is hardly "new".
  
> > You asked where the 255 number came from.  It's the limit on
> > a fully-qualified domain name length.
> 
> Yes, I didn't get your point about 255 > 78, because 255 < 998.
> The 78 is "only" a SHOULD, I'd ignore it e.g. for long URLs in
> the body of an article.

See comp.mail.mime for an example of an ISP that likes to re-wrap
(at coincidentally ~ 255) during transport.
  
----------
> >> Sub-strings reflecting 2047-words are allowed,
> > Not clear; would have to be quoted,
> 
> Yes. = and ? are no specials, but if there's a blank => quoted.

Remember that RFC 2045 that I suggested rereading?  Look up
"tspecial".

> > and encoded-words aren't allowed in quotes-strings.
> 
> No, = and ? are allowed in quoted-strings, in any combination.

Yes, but then there's no encoded-word.

---

From: Bruce Lilly <[email-masked]>
Date: Wed, 3 Aug 2005 20:52:11 -0400

On Fri July 29 2005 10:23, Charles Lindsey wrote:

> Anyway, nobody has spoken against my basic methodology, so that seems
> sort of accepted.

That is a false assertions, as can plainly be seen from the mailing list
archives.

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 12 Aug 2005 05:45:36 +0200

> IIRC, you're the "examples-are-better-than-syntax" guy :-).

Omigod, July 10 2004.  Unfortunately still the old list, so
it's not easy to check what else happened at this time.

July 1, Russ tells me that 2231 exists, strange, I thought
that it was you.  Maybe 2231 made me so angry that I plonked
somebody (whatever that means with my "mozilla 3").  Lots of
articles about back-references, complaints-to, cmsg <sigh />
oops, yes, here it is, July 3 and 4:

<http://mid.gmane.org/[email-masked]>
<http://mid.gmane.org/[email-masked]>

Scanning until July 19, no, I really missed or ignored your
proposal.  We discussed URL folding and 2017 at this time.

Harald just closed the Injection-Date ticket, so maybe this
is no good time to reintroduce the <date-time> part of your
idea.  Otherwise it's very much like "Received", apparently 
you want to use some of the same <item-name>.

It doesn't use 2231 <parameter>, it uses 2822 <name-val-list>
with <item-name> and <item-value>.  Fine, so why don't we do
it this way ?

> Special offer; three for the price of one!

Compared with 2231 even #1 is better.

> I'd still do Complaints-To as a bracketed URI list.  Same
> general form as List-Post; plenty of reusable code for that.

Okay, I like your #1..#3 idea, and maybe it's then better to
go back to a separate Complaints-To.

~~~  
 
> Remember that RFC 2045 that I suggested rereading?  Look up

Sure, I know it.
SPECIALS : " ( ) , .   : ; <   >   @ [ \ ]
TSPECIALS: " ( ) ,   / : ; < = > ? @ [ \ ]

We discussed solutions without <parameter> like <name-val-list>
or a similar construct in the Received-SPF header field IIRC.

So 2045 rules about <parameter> don't enter the picture.  If we
want <quoted-string> all of <qtext> is okay, incl. "=" and "?".

Your idea is much better than my idea, let's forget the latter.

In essence what you propose is to replace all 'name = "value"'
by 'name value'.  CFWS separated words.  So far that's KISS and
better than 2231, where are the problems ?

                            Bye, Frank

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 12 Aug 2005 05:54:59 +0200

> Comments before I propose syntax?

Apparently not only I didn't comment Bruce's idea, nobody did,
or it was off list, or a case of bad timing beause we were
busy with tons of other details at this time (July 10 2004).

<http://article.gmane.org/gmane.ietf.usenet.format/27216>

What do you think of it ?  It avoids RfC 2231.  What are the
disadvantages ?
                           Bye, Frank

---

From: Bruce Lilly <[email-masked]>
Date: Sun, 14 Aug 2005 10:04:15 -0400

On Thu August 11 2005 23:45, Frank Ellermann wrote:

> Otherwise it's very much like "Received", apparently 

John has identified the persistently recurring problem in messages
<[email-masked]> and
<[email-masked]>.

Consider the fourth definition of NIH at
http://www.acronymfinder.com/af-query.asp?p=dict&String=exact&Acronym=NIH
The answer to your question is NICHL; one person's fetish for
parameters despite WG consensus against their use.

> TSPECIALS: " ( ) ,   / : ; < = > ? @ [ \ ]

Anything with any tspecial has to be quoted.  '?' is a tspecial.
therefore any parameter with '?' has to be quoted.  And encoded-words
are not permitted in quoted-strings.  Consequently one cannot have an
RFC 2047/2231 encoded-word in an RFC 2045/2231 parameter.

> want <quoted-string> all of <qtext> is okay, incl. "=" and "?".

Fine as text in a quoted-string, but encoded-words never appear in
quoted-strings, only in unstructured fields, words in a phrase
(specifically excluding the other component of phrase), and within
parenthesized comments.

> better than 2231, where are the problems ?

See above and John's messages.

---

From: Frank Ellermann <[email-masked]>
Date: Sun, 14 Aug 2005 22:10:19 +0200

Bruce Lilly wrote:

 [Somebody]
> has identified the persistently recurring problem

Okay, if you think that there are serious problems with your
idea, then we can as well use 2231 instead of rolling our own.

> Consider the fourth definition of NIH

Yes, I know NIH.  Neither 2231 nor 2822-Received was invented
here, for definitions of "here = NetNews" or "here = USEFOR".

> one person's fetish for parameters despite WG consensus
> against their use.

I'm not sure about a consensus, so far I'd _guess_ that Russ
doesn't like the parameter-solution.  I can't tell what you
want, and I don't like 2231, but draw the line at inventing
something that's not really better.

> Anything with any tspecial has to be quoted.

For a 2045-parameter value.  Not in say local parts.  It's
beside the point, we were talking about a <quoted-string>,
and the only characters you've to worry about in this context
are DQUOTE and "\", maybe some CTL, but not "/", "=", "?", or
<specials>.

DQUOTE and "\" are relevant for all arbitrary <quoted-string>s,
other <specials> or <tspecials> are only relevant if it's in
a context where they mean something.

E.g. "(" and ")" are only relevant for comments, but there are
no comments within a <quoted-string>.  If you take "something"
containing comments and quote it, then the "quoted something"
does not more contain a comment.

"(I am no comment while quoted)"
"=?us-ascii?q?I_am_no_encoded_word_while_quoted?="
"\"Goedel Escher Bach was fun\""

> And encoded-words are not permitted in quoted-strings.

Exactly that's the point, there simply are no <encoded-word>s
in a <quoted-string>, that's no question of legal vs. illegal
or a _permission_ , it's the _definition_ of a <quoted-string>.

They are not permitted as parameter-value in an unquoted form.
But of course you can use any <quoted-string> - including the
three examples shown above.

> Consequently one cannot have an RFC 2047/2231 encoded-word in
> an RFC 2045/2231 parameter.

If you really think than one or more of my three examples is
an invalid / illegal <parameter> value I'd be very surprised.
We could ask Keith and Ned what they think about it.

Only for parameters, 2045 and 2231, not for boundary strings:

<bchars> has no "\" for <quoted-pair>, so the Goedel-example
is no legal boundary.  But AFAIK it's a legal parameter value.

                       Bye, Frank

---

From: Bruce Lilly <[email-masked]>
Date: Sun, 14 Aug 2005 18:17:36 -0400

>  [Somebody]
= John
> > has identified the persistently recurring problem

And others have mentioned the problem, on list and off, for years.

> idea

The problem isn't with the specific idea, it's with getting ideas
into (and in some cases out of) our documents when they happen to
not coincide (or to coincide) with one person's preferences.

> here, for definitions of "here = NetNews" or "here = USEFOR".

"Here = USEFOR [WG]" isn't the issue; several options for injection
information fields have been discussed here.  Reread John's
messages.
 
> doesn't like the parameter-solution.

There was a poll and striking results several years ago.  That was
mentioned and discussed more recently, and finally (after repeated
directions from the Chair) most of the parameters are gone.  But
because of one person's fetish, we're still discussing problems
with Injection-Info, and that's holding up progress.

IMO, we need to select syntax that avoids the many problems with
parameters (as noted, there have been such proposals) or yank the
entire field out (as it's not an existing field), leaving it for a
future extension.  We're way behind schedule, and if we don't
resolve this issue and move on, the IESG would be justified in
disbanding the WG (which has produced none of its chartered
deliverable work products after 8 years, 4 months, and 4 days
(and counting)).

> I can't tell what you 
> want

I don't see why not; I've been clear about proposals for unproblematic
fields for injection information and about Complaints-To.

> and I don't like 2231, but draw the line at inventing 

No need to invent; we can reuse existing field syntax that is known
not to have problems with parameters (borrowing syntax from Received
and List-Post for injection and complaint-reporting information).
 
> For a 2045-parameter value.  Not in say local parts.

But you wanted to put address-lists in a parameter, at which time
any encoded-words in display names would cease to be encoded-words,
creating a mess.

[...]
> does not more contain a comment.

Another reason why an address-list (which can of course contain
comments) is a very bad idea for a parameter.

> "\"Goedel Escher Bach was fun\""
[...]

> an invalid / illegal <parameter> value I'd be very surprised.

They are legal quoted-strings which could appear as parameter values.
However, they are then exactly that -- parameter values -- and cease to
be comments (first example) and encoded-words (second example).
Conforming parsers will never see them as comments or encoded-words.

> is no legal boundary.

Backslash is irrelevant in that case; it's a mechanism for putting
DQUOTE in a quoted-string.  bchars doesn't permit DQUOTE, so you're
right about the third example not being a legal boundary value (for
the wrong reason).

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 15 Aug 2005 04:55:51 +0200

Bruce Lilly wrote:

> =
[...]  My killfile is a mental thing.  As long as $Somebody
and me are not obliged to recognize each other this works
as expected (incl. "not so well").

> The problem isn't with the specific idea, it's with getting
> ideas into (and in some cases out of) our documents when they
> happen to not coincide (or to coincide) with one person's
> preferences.

In that case it's tricky, your $Somebody is the editor or his
co-author.  In that case you're forced to handle it through
others or the Chair.

> Reread
[Somebody's]
> messages.

No, thank you.  I think that $Somebody is not interested in
any progress or WG output.  No big deal if I'm wrong.

> There was a poll and striking results several years ago.

Probably I missed it.  I only know a rather small part of the
WG history, mainly the first "last-last-last-call" (the old
draft -07 or -08) and Ralph Babel's poll, where you found that
there was no consensus for the UTF-8 stuff, and then again
last year (?) after the decision to split the document.

Which was a good idea, although I didn't like it for some time.

> because of one person's fetish, we're still discussing
> problems with Injection-Info, and that's holding up progress.

I'm disussing it because I have problems with 2231, but there
is some "progress", I won't opt-in for something worse.  So if
we really want or need most features of 2231 then let's use it.
If that's not the case let's avoid it.

It's no fetish for me.  I like the idea of one header field
instead of a header field zoo, and I like a complaint address
instead of URLs.

At the moment we have two Injection-* header fields.  With your
idea we'd get Complaints-To plus something-like-Received, also
two header fields, so far not worse or better.

Your idea allows URLs.  While I hate this I know that Alexey and
Russ might like it, so ignore me.  And _if_ you can avoid 2231
that's much more important than the minor complaint URL issue.

But you have to make sure that even simple-minded algorithms
find the timestamp.  That's critical, unlike the URL business.

Your option #3 and maybe also #2 addreses this point.  So far
your proposal is apparently better from my POV.

> IMO, we need to select syntax that avoids the many problems
> with parameters (as noted, there have been such proposals)

ACK so far...

> or yank the entire field out (as it's not an existing field)
> leaving it for a future extension

...but NAK here.

> We're way behind schedule

Yes, let's focus on real alternatives and then decide it.  I've
already said all I can contribute to this issue.  If the result
is "let's keep it as it is" I can live with it.  But we cannot
kill Injection-* without proper replacement, that's just wrong.

> if we don't resolve this issue and move on, the IESG would be
> justified in disbanding the WG

Following the procedures listed in RfC 2048 chapter 4.  Are you
perhaps interested in my unpublished "3710 is obsolete" I-D ?

>> I can't tell what you want
> I don't see why not

Simply put I'm not sure whether you want any result, because it
won't be what you like in several points like the magic SP, and
a few other NetNews oddities.  You care about the holy grail
"Internet message format", I care about NetNews.  Different
priorities, your number one is my number two (and vice versa).

> I've been clear about proposals for unproblematic fields for
> injection information and about Complaints-To.

It would be easier for me to second it if it's complete ABNF
ready to replace the Injection-*.  Or if Russ or Henry or both
like it.  By my appointment they can "veto" everything I say -
at least when they more or less agree.

A somewhat unreliable oracle, IIRC both refused to comment the
<id-right> vs. <id-domain> issue.  But it's also interesting
when they say nothing.

>> I don't like 2231, but draw the line at inventing something
>> that's not really better.

> No need to invent; we can reuse existing field syntax that is
> known not to have problems with parameters (borrowing syntax
> from Received and List-Post for injection and complaint-
> reporting information).

Okay, see above.  Get support from some relevant contributors.
As you said we're late, we can't discuss this for another year,
we need concrete proposals, not vague or abstract ideas.

 [Goedel Escher Bach]
> But you wanted to put address-lists in a parameter

No, I wanted something like 'name "=" value' with a <quoted-
string> as a value, same idea as in Received-SPF.  We already
agree that it was a bad idea, but it was precisely to avoid
the MIME <parameter> syntax.

> at which time any encoded-words in display names would cease
> to be encoded-words, creating a mess.

It's not messy.  a = "b" ; c = "d" ; e = "f" is simply a list
of names and values separated by semicolon, and at this moment
it's irrelevant what b d f are outside of a quoted-string.

Your idea results in a b c d e f, we have to make sure that
we can parse b and d and f in this form, even if b is in fact
b' c' d' resulting in a b' c' d' c d e f.  The parser has to
identify c' d' as belongig to b'.  With quoted strings that's
obvious: a = "b' c' d'" ; c = "d" ; e = "f".

> Conforming parsers will never see them as comments or

Okay, so we agree about that detail, a <quoted-string> can be
almost anything, as long as all DQUOTE and "\" within it are
escaped.

 [bchars]
> Backslash is irrelevant in that case; it's a mechanism for
> putting DQUOTE in a quoted-string.  bchars doesn't permit
> DQUOTE, so you're right about the third example not being a
> legal boundary value (for the wrong reason).

Neither DQUOTE nor "\" are allowed, we're both right.  Unless
you think that "\\hcaB rehcsE ledeoG\\" is a legal boundary,
then it would be only me who's right... ;-)  Bye, Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 17 Aug 2005 14:36:54 GMT

>(specifically excluding the other component of phrase), and within

Actually, you could allow encoded-words within quoted-strings (since a
quoted-string is just an opaque collection of ASCII characters). For sure,
the reader would see them in all their ugliness, because reading agents
would not decode them (unless we go to a lot of trouble to allow that
explicitly, which I would not recommend). OTOH, if a user were to do a
simple copy-and-paste of, say, a mail-complaints-to value into the To:
field of a new email, the encoded word would magically become valid again
and the mail would be sent to the correct destination.

Note that I neither propose nor oppose this idea - I merely point out that
it is possible.

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 17 Aug 2005 14:29:50 GMT

>What do you think of it ?  It avoids RfC 2231.  What are the

Essentially, it is a different format for the same wheel. It has pairs of
names and values separated by CFWS. The values are 1*angle-addr /
addr-spec / atom / domain / msg-id, some of which are appropriate for our
needs and some not. It provides nothing obviously suitable for the
posting-account and logging-data parameters, and it makes no provision for
any parameters which might contain whitespace within themselves, or for
I18N (both of which might well arise for the two parameters mentioned). So
there is no way that a value could be folded.

It makes no provision for display-names to go with addr-specs (which are
not essential, but I can see they might be handy).

It makes no provision for future parameters in extensions (to be ignored
in the meanwhile), nor for ad-hoc parametets with x-names.

By the time you have fixed all those things, you are more or less back at
MIME parameters. That would bring up the dreaded 2231, of course, but at
least you get semi-decent I18N that way (the only thing wrong with the
2231 I18N is that it is different from all the other I18N fixes that have
been invented over the years). So the only problem left is the line
splitting feature of 2231, which I agree we do not want, and could happily
deprecate.

---

From: Bruce Lilly <[email-masked]>
Date: Sun, 21 Aug 2005 11:48:33 -0400

On Wed August 17 2005 10:36, Charles Lindsey wrote:

> Actually, you could allow encoded-words within quoted-strings

No.  RFC 2047, section 5:

   + An 'encoded-word' MUST NOT appear within a 'quoted-string'.

> Note that I neither propose nor oppose this idea - I merely point out that

You are mistaken.

---

From: Bruce Lilly <[email-masked]>
Date: Sun, 21 Aug 2005 12:34:11 -0400

On Wed August 17 2005 10:29, Charles Lindsey wrote:
> needs and some not.

Since the Injection-Info field is optional, there is no "need".  Even if
field were mandatory, the components (other than the identifier of the
injecting systems (i.e. "domain")) are optional.  So, your claim, Charles,
in untrue.

> It provides nothing obviously suitable for the 
> posting-account and logging-data parameters,

angle-addr or addr-spec or atom, and atom or msg-id, respectively, are
suitable.  Another untrue claim, Charles.  Moreover, as already discussed,
"logging-data" is a waste of bandwidth and disk space, as it is only
meaningful at the injection site; it is therefore pointless to transmit it
to thousands of systems where it will occupy valuable disk space -- it
need only be associated with the article (via msg-id) at the injecting site.
Ditto for posting-account.

> and it makes no provision for 
> any parameters which might contain whitespace within themselves

Untrue; angle-addr, addr-spec, and domain may all contain whitespace.
Also, there is a conflict between RFCs 2821 and 2822 w.r.t. quoted-strings
(allowed in 2821 but omitted from 2822); if that is resolved in favor of
2821, that is another mechanism. [incidentally, both angle-addr and addr-spec
already include quoted-sting via local-part, but 2822 as written does not
permit a stand-alone quoted-string as an item-value]

> , or for 
> I18N (both of which might well arise for the two parameters mentioned).

More untrue assertions; look up "IDN".  Also, "atom" includes things such as
=?us*en?q?foo?= (indeed, RFC 2047/2231 encoded-words are intentionally
designed to be RFC 822 atoms); however that is somewhat unclear as MIME
(2047/2231) is based on 822 rather than 2822, and 2822 neglects mention of
MIME encoded-words.  While RFC 2047 contains a curious prohibition against
encoded-words in the Received field, that obviously would not apply to a
new field (i.e. different field name) using similar field body syntax. 

> So 

A patently and obviously false claim; all item-value constructs explicitly
permit folding, and folding is allowed between an item-name and item-value as
well as between name-val-pairs.  As the syntax is modeled on the Received
field syntax and Received fields are commonly folded, there is no excuse for
making such a false claim.

> not essential, but I can see they might be handy).

Nor does use of 2231 parameters; indeed that was recently discussed as a
serious problem with that proposed scheme vs. a separate Complaints-To
field.
 
> in the meanwhile),

Yet another falsehood; the syntax provides for any item-name which meets
the syntax requirements.

> nor for ad-hoc parametets with x-names.

That is a feature. [though names beginning with "x-" match the item-name
syntax and could therefore be used if we wanted to open that Pandora's box]

> MIME parameters.

Absolutely untrue.  There are not the parsing complexities of MIME parameters,
no quoting vs. encoded-word issues, and existing reusable code (vs. having to
write code from scratch including handling RFC 2045/2047/2231 issues).

> That would bring up the dreaded 2231, of course, but at 
> least you get semi-decent I18N that way

See IDN and RFC 2047/2231.

> (the only thing wrong with the 
> been invented over the years).

Another falsehood; Q encoding is essentially the same as that used in URIs
and differs from that used in encoded-words only in the choice of the initial
character ('%' vs. '='), and there is evidence that the latter difference
was not originally intended.

> So the only problem left is the line 

More falsehoods.  We have no authority to modify RFC 2231.