Subject: Re: Does 2231 break MIME 1.0?

From: Frank Ellermann <[email-masked]>
Date: Wed, 01 Dec 2004 06:13:28 +0100

Bruce Lilly wrote:

> According to my local copy of the official language-tags
> document:

| i-klingon    Klingon  [Everson]   Deprecated use ISO
|                                   639-2 tlh, registered

Strange, I don't have a local copy of "official" tags, but
an <URL:http://www.iana.org/assignments/lang-tags/i-klingon>

The last I heard about the new scheme was a draft with some
problems like X-Y-Z ~ X-Y but X-Y-Z !~ X-Z, and all the old
i-tags like i-default etc. as grandfathered.  IMHO unusable,
unless they add a frozen 3166 list (before the big CS oops).

> the complete code change necessary to print the parameters:

Sorry, I don't have the C source for this patch, but it's a
bad sign that you had to change something.  The patch does
not show how you handle any duplicated resp. missing pieces.

> Implementation in UAs is not difficult given the right set
> of tools

It's a PITA if all you want is to find the next MIME part,
because that's not the precise moment where you want any:

boundary*1="pa\\rt " ; dup*1="two" ; mia*0="missing" ;
dup*0=one ; mia*3=action ; dup*1=dup ; boundary*0=multi ;

Just in case, maybe you want more weird test cases ;-)

Don't forget to get rid of the trailing space in this
boundary (another trick used against SpamCop, of course
fixed now).  In one of my scripts I remove all \, that
is a bad idea for \\, maybe I'll fix it later.

> ISPs often wish to add a tracking identification tag
> to each individual message.

Not in the case of 50 worms from the same IP. ;-)  Yes,
it depends on the recipient, but multipart/digest is in
the "minimal MIME conformance" RfC.

Joining our two 2231 threads again:

>>> Since there cannot be multiple instances of the same
>>> attribute in the same field due to other considerations,
>>> it's not incompatible.
 
>> What "other considerations" ?
 
> Insignificance of order, as Ned explained in the ietf-822
> message that I cited earlier.

"Insignificance of order" and "unique" are quite different.

The latter is a "set" (any item is either IN or OUT), the
former is a "bag" (IIRC, please correct me if I'm wrong).

> Are you referring to something other than the published
> errata?

That's too substantial for the "errata".  Now give the poor
authors a break, it's less than two weeks that you found
this bug.  The proposed strategy (add "MUST be unique" and
limit *-hacks to parameters where it can't cause harm) may
need a year, it affects at least two RfCs.       Bye, Frank

---

From: [email-masked] (Claus Färber)
Date: 01 Dec 2004 12:41:00 +0100

Frank Ellermann <[email-masked]> schrieb/wrote:
> What "other considerations" ?  The author saying that he had something
> like an environment in mind explains why he forgot to mention it
> explicitly.  But we've already seen that it could make sense, e.g. if
> I'd say...

> charset=windows-1252 ; charset=iso-8859-1

> ...it could stand for "I didn't use any 128..159, it works
> with whatever you have,...

You could still do this:

Content-Type: text/plain; charset=iso-8859-1; alt-charsets*1=" iso-8859"
  alt-charsets*0*=''iso-8859-1; alt-charsets*2="-15 iso-8859-2 windows-"
  alt-charsets*3=1252

But why, if you just label your content with the most common charset;  
e.g. ISO-8859-1.

> and if you really insist on it I'm ready to accept replies with
> windows-1252".

No. That would be a very bad idea.

Claus
-- 
http://www.faerber.muc.de

---

From: [email-masked]
Date: Wed, 01 Dec 2004 07:22:42 -0800 (PST)

> No. That would be a very bad idea.

The understatement of the year arrives just in time for Christmas ;-)

				Ned

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 1 Dec 2004 12:37:39 GMT

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>We could, provided we provide a clear and complete specification.
>We should consider liklihood of implementation.  It has been
>claimed (unconvincingly) that because there may exist parsing
>of MIME parameters for Content-Type and Content-Disposition
>fields in some software, parsing of such parameters in other
>fields will magically happen as well.

You weren't convinced. I don't recall anyone else agreeing with you.

But in any case, there is never any need for any regular news agent to
parse the Injection-Info header to that level of detail. For injection
agents, a cursory examination to ensure that it does no gross violation of
the RFC 2045 syntax is quite sufficient.

The Injection-Info header is provided for sysadmins, geeks, spam fighters
and the like who will want to examine it only when something appears to
have gone wrong (e.g. so that they can generate abuse complaints in the
particular cases under discussion).

Usually, they will just read it. Maybe they will cut and paste bits out of
it (IP addresses, URLS, etc). If they are really keen, they will write
scripts to assist their tasks, though I doubt few of them will bother to
build full RFC 2231 capability into their scripts - simpler to disentangle
them by hand in the rare cases they will appear.

>RFC 2369 handles URIs (per RFC 1738 syntax, compatible
>with 2396 syntax), but is incompletely specified (no ABNF!).
>If we wish to use multiple URIs in a separate field similar to
>the List- fields described in RFC 2369, we could use that
>mechanism, but we should completely specify it including
>ABNF.  That's complicated by incompatible changes in the
>URI syntax draft which will probably be approved before
>we get around to a specification.

>> They simply say "internal whitespace being ignored" in 2369.

>RFC 2369 also says "MUST NOT insert whitespace within the
>brackets".  Ignoring whitespace when parsing is apparently
>intended to cope with mangling during transit, not as a
>mechanism for deliberately folding long URIs when generating
>a field.  RFC 2369 completely ignores (the common case of)
>long URIs.

Ah! So implementations that "ignore internal whitespace" like the RFC says
must first check whether the mangling occurred in transit and not do it
otherwise. How clever and sensible! NOT.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: Bruce Lilly <[email-masked]>
Date: Wed, 1 Dec 2004 12:39:48 -0500

> not show how you handle any duplicated resp. missing pieces.

The change was solely to print the (reassembled) parameters;
normally a parameter is simply used for its defined purpose.

> Just in case, maybe you want more weird test cases ;-)

Do you mean something like the example that I posted to
this WG mailing list on 19 Mar 2002? :-) That contained:

Content-type: multipart/mixed;
  boundary*0* =''sim ;
  boundary*1* = ple%20
 ;
  boundary*2* = boundary
 
> former is a "bag" (IIRC, please correct me if I'm wrong).

Repetition ("foo=x ;foo=x") is not likely to be a problem.
Because order is insignificant (and always has been),
"foo=x ; foo=y" is a problem, because there is no way
to  determine the value associated with the attribute
foo.  One cannot depend on order, because order is
insignificant (changing order shouldn't change the
semantics).  That's why Ned wrote:

"It may not be explicitly stated that duplicate parameters aren't allowed, but
given that parameter order is required not to be significant (RFC 2046 section
1) is is hard to come up with any other result that makes sense."

> this bug.

I've never claimed that there's a "bug" associated
with continuation parameters; that's your claim,
but I don't think you've convinced anybody.

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 02 Dec 2004 05:32:55 +0100

Claus Färber wrote:
 
> But why, if you just label your content with the most common
> charset; e.g. ISO-8859-1.

It was a hypothetical example, Bruce said that all unordered
lists are sets, and I needed something where that's obviously
not the case.

>> I'm ready to accept replies with windows-1252".

With my old MUA it would be an excellent idea, it allows me to
discuss with Hažek about “€-crats” ;-)

                         Bye, Frank

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 02 Dec 2004 06:18:06 +0100

> Content-type: multipart/mixed;

Nice... ;-)  Fortunately my stupid script tries to guess where
a MIME part might begin ("--"), then it accepts any Content-*
incl. folding, a blank line, and a base64 line starting with
TV or UE.  Anything else is irrelevant for its purposes.
  
OTOH it scans only TOP 50 body lines, and then tries to fix the
outer boundary after 50 lines (for an abuse complaint, that's a
truncated message/rfc822).  It won't add a --simple boundary--
in your example, and abuse@WannaSpew would be very unhappy. :-(

>| It may not be explicitly stated that duplicate parameters
>| aren't allowed, but given that parameter order is required
>| not to be significant (RFC 2046 section 1) is is hard to
>| come up with any other result that makes sense.

Yes, that was wrong, it was not hard to find counter-examples.
In fact we stumbled over an example without looking for it, our
complaint-url= case.  Alexey only said "I want more than one".
 
> I've never claimed that there's a "bug" associated with
> continuation parameters; that's your claim, but I don't think
> you've convinced anybody.

My original claim was "okay, let's simply have zero or more of
these complaint-url= beasts".  I checked that it should be okay
for MIME-2049.  And then you demonstrated that it's _not_ okay
for MIME-2231.  I was immediately convinced by your proof, and
changed the subject of this thread.  JFTR, bye, Frank  <beg>

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 2 Dec 2004 15:47:39 GMT

In <[email-masked]> Bruce Lilly <[email-masked]> writes:

>... That's why Ned wrote:

>1) is is hard to come up with any other result that makes sense."

So should I understand that the present standards do not actually prohibit
duplicate parameters anywhere, but that they do not make any sense in a
lot of cases.

In that case, it depends on the purpose of the "foo" parameter, and
whether it makes sense to say "foo might be x; alternatively foo might be
y".

The case we were considering was where we might put, in Injection-Info:

... mail-complaints-to="[email-masked]";
    mail-complaints-to="[email-masked]";

OTOH, there are easier ways to do the same thing, such as by saying that
the syntax of that parameter is:

    mail-complaints-parameter = "mail-complaints-to" "=" address-list

which also happens to be the syntax in the Complaints-To header as
currently proposed.

---

From: [email-masked]
Date: Thu, 02 Dec 2004 09:30:45 -0800 (PST)

> duplicate parameters anywhere, but that they do not make any sense in a

There is no case I am aware of where a parameter has been defined that allows
for duplication. If such a case has made it through, it was done in error and
if pointed out will be rectified ASAP.

I can assure you that any media type application that calls for parameter
duplication would be rejected by the media types reviewer (me) or by the IESG
in the case of a standards tree registration. Any attempt to define a
content-disposition parameter of this sort would be treated similarly.

The bottom line is that duplicate parameters don't make sense, the intent from
the get-go was that they not be allowed, the fact that this didn't get stated
explicitly is no reason it should be assumed to be allowed. Stuff like this
happens from time to time. And more generally, to assume that every concievable
form of bad behavior has been anticipated and explicitly prohibited in the
relevant standards documents is nothing short of pure folly.

> which also happens to be the syntax in the Complaints-To header as

It is axiomatic that a completely new field which defines a completely new set
of parameters gets to choose its own set of rules for how those parameters will
behave. As such, reasoning on the basis of what is or isn't allowed by
content-type or content-disposition is simply silly. The point was, is, and
always will be that choices such as whether or not duplication is or isn't
allowed in some new set of things has implications in regards to syntax,
semantics, and implementation, and therefore must be carefully considered.

				Ned

---

From: Alexey Melnikov <[email-masked]>
Date: Sun, 19 Dec 2004 20:55:32 +0000

Charles Lindsey wrote:

>>>>I would prefer #2, followed by #1 and #3.
>>>>
>>>>As a chair I would note that I see no consensus to use mail-complaint-to 
>>>>over complaints-url.
>>>>        
>>>>
>
>I think the question is whether to provide both of them initially.
>
This WG has enough work to do already, so I am Ok with defining 
complaints-url in a separate document. However the syntax for 
Injection-Info (or whatever we end up using) should allow for multiple 
URLs.

>>>Ah! You mean that those who want to give a (non-mailto) URL SHOULD give a
>>>mail-address as well. Yes, I could live with that.
>>>      
>>>
>>Yes.
>>Note, that this SHOULD belongs to USEAGE.    
>>
>
>Which is where we disagree.
>
The key reason for keeping syntax/protocol separate from policy is to be 
able to revise documents independently. Policy is more likely to change.

>  
>
>>And I am strictly against it becoming a MUST.    
>>
>
>Indeed.
>
Alexey

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 21 Dec 2004 13:48:52 GMT

>Injection-Info (or whatever we end up using) should allow for multiple 

OK, so it seems that a mail-complaints-to parameter in Injection-Info is
the front runner. So we would need, in USEFOR:

   inj-info-param  =  post-host-param /
                      post-account-param /
                      sender-param /
                      logging-param /
                      mail-complaints-param

   mail-complaints-param = "mail-complaints-to" "=" address-list

and then:

   The specified <address>(es) is for sending complaints concerning the
   behaviour of the poster of the article; it SHOULD NOT be used for
   matters concerning propagation, protocol problems, etc. which should be
   addressed to "usenet@" or "news@" the <path-identity> (see [RFC 2142]).
   In the absence of this parameter, complaints concerning a poster's
   behaviour MAY be addressed to "abuse@" that <path-identity> (again, see
   [RFC 2142], though that address may not be available at injecting
   agents not provided for the use of the general public).

If we are agreed on that, then we can ask Ken to incorporate it into
USEFOR.

[Note that some related texts from draft-13 relating to mailability of
<path-identity>s have not yet made their way into USEFOR.]

The only other alternative remaining on the table is the Complaints-To
header as in draft-13. I received some mail from Alexey about this which I
am reproducing here, because it is part of the same discussion:

>....Charles Lindsey wrote:
 
>>Incidentally, one of the advantages of a Complaints-To header over a
>>parameter of Injection-Info is that it would be suitable for use in
>>Emails, if the email community wants to use it. 
  
>Can you elaborate on potential usage scenarios for email?

The situations I had in mind were where some ISP operated an email
submission agent, and had an AUP which forbade harrassment, spamming, and
other such evils. So it might add a Complaints-To header to emails passing
through it. Or else it might be added by a mailing list expander which had
rules about what could be posted to the list.

It is not for us to say whether that header should be used for email or
not, but it would clearly be suitable, whereas the Injection-Info header
is no way suited to email. Whether this factor should sway our decision on
the matter is for the WG to decide.

---

From: Russ Allbery <[email-masked]>
Date: Tue, 21 Dec 2004 09:20:19 -0800

Charles Lindsey <[email-masked]> writes:
> Alexey Melnikov <[email-masked]> writes:

>    mail-complaints-param = "mail-complaints-to" "=" address-list

That looks like it's allowing multiple addresses, not multiple URLs.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 22 Dec 2004 14:24:06 GMT

>That looks like it's allowing multiple addresses, not multiple URLs.

Yes, that was the intention. The proposal, as already discussed, was to
have separate parameters for the pure email case (mail-complaints-to=) and
for the url case (complaints-url= or somesuch). Our chair proposed (see
above) leaving complaints-url to a future extension, and that gave me the
opportunity to write up the mail-complaints-to properly.

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 22 Dec 2004 17:27:41 +0000

>opportunity to write up the mail-complaints-to properly.
My question is: do we *want* to allow for multiple complaints-to email 
addresses?

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 23 Dec 2004 18:57:35 GMT

>My question is: do we *want* to allow for multiple complaints-to email 

The syntax I gave already allows for multiple complaints-to email
addresses, because it contains an address-list (the Complaints-To header
likewise). OK, we have to make it clear that it means "send your complaint
to one of these addresses", and not "send it to all of them", and that is
indeed not quite the usual interpretation of <address-list>. So first, let
us be clear which of those interpretations you have in mind, and then we
can ask whether <address-list> is the proper syntax to do it.

If a future document introduces a complaints-url (whatever) parameter, as
you have suggested, then it will be up to that document to make provision
for multiple URLs.

---

From: Alexey Melnikov <[email-masked]>
Date: Fri, 24 Dec 2004 11:58:47 +0000

>can ask whether <address-list> is the proper syntax to do it.
I think that only "send to one of those addresses" interpretation makes 
sense.
However, I can't even see a practical need for multiple addresses. Can 
somebody give an example?

>you have suggested, then it will be up to that document to make provision
Of course.