Subject: #1003 USEFOR 3.1.3 Message-ID: Proposed text

From: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 29 Jun 2005 11:36:26 +0200

OK, here's the proposed text for the problems I outlined yesterday:

OLD (start of section):

   The Message-ID header contains a single unique message identifier.
   This document updates the <msg-id> construct from Section 3.6.4 of
   [RFC2822] so as to ensure that Internet Message Format Message-IDs
   are usable in widely deployed news software.  The global uniqueness
   requirement for <msg-id> in [RFC2822] is to be understood as applying
   across all protocols using such message identifiers, and across both
   Email and Netnews in particular.  A revised syntax for <msg-id> is
   given below, but the requirements and descriptive text from Section
   3.6.4 of [RFC2822] still apply.

NEW:

   The Message-ID header contains a single unique message identifier.
   News is more dependent on msg-id uniqueness and fast comparision than
   email is, and some news software would have trouble with the full range
   of possible msg-ids permitted by RFC 2822; this section therefore
   restricts the syntax of <msg-id> compared to Section 3.6.4 of
   [RFC2822].
   The global uniqueness
   requirement for <msg-id> in [RFC2822] is to be understood as applying
   across all protocols using such message identifiers, and across both
   Email and Netnews in particular.

OLD (ABNF):

   no-fold-quote   =  DQUOTE
                         ( "." mqtext /
                           *mqtext "." /
                           *mqtext mqspecial *mqtext )
                         DQUOTE

NEW:

   no-fold-quote   =  DQUOTE
                         ( "." *mqtext /
                           *mqtext "." /
                           *mqtext mqspecial *mqtext )
                         DQUOTE

OLD (end of section):

      NOTE: It is RECOMMENDED in [RFC2822] that, for ensuring global
      uniqueness, the <id-right> be some domain identifier within whose
      scope the uniqueness of the <id-left> can be guaranteed.  When
      following this recommendation, any <dot-atom-text> or <no-fold-
      literal> used for the <id-right> are to be interpreted as
      <domain>s as described in Section 3.4.1 of [RFC2822].

NEW:

   Some software will try to match the <id-right> of a msg-id in
   a case-insensitive fashion; some will match it in a case sensitive
   fashion. Implementations MUST NOT generate two message-IDs where the
   only difference is the case of characters in the <id-right> part.

   If domain literals are used, the syntax found in [RFC2821] section
   4.1.3 is RECOMMENDED.

     NOTE: [RFC2822] section 3.6.4 recommends that the <id-right> should
     be a domain name or a domain literal. Domain literals are troublesome
     since many IP addresses are not globally unique; domain names are
     more likely to generate unique message-IDs.
     In no circumstance is it reasonable for software receiving a <msg-id>
     to change behaviour based on a theory about how the <msg-id> was
     generated.

I think this makes it clear that in message-id, the question of "what" the 
RHS is matters only to the generator, not to the parsing entity; all we 
care about after generation is that the message-ids are unique.

(the 2821 reference was added when I discovered that in 2822, this was 
punted to "the transport" - this makes sense for addressing, but not for 
message-IDs, where uniqueness is the thing that matters)

Makes sense?

                    Harald

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Wed, 29 Jun 2005 08:33:34 -0400

>      to change behaviour based on a theory about how the <msg-id> was

What about "<cancel.", and other schemes (maybe multiparts.)?

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 29 Jun 2005 14:40:52 +0200

--On onsdag, juni 29, 2005 08:33:34 -0400 "Forrest J. Cavalier III" 
<[email-masked]> wrote:

>>      In no circumstance is it reasonable for software receiving a
>>      <msg-id> to change behaviour based on a theory about how the
>>      <msg-id> was generated.
>
>

Sorry, I don't understand. Is there software that tries to *parse* a 
message-ID?

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 29 Jun 2005 15:34:01 +0200

Harald Tveit Alvestrand wrote:

> Is there software that tries to *parse* a message-ID?

It's a convention for spam cancel Message-IDs.  In that case
an intentional collision, one <cancel.old@id> is enough to
cancel the <old@id> spam.  In a more elaborated form it's
<cancel.secret.old@id>, because the bad guys started to send
a dummy <cancel.old@id> before the <old@id> spam, this dummy
blocked the spam cancel.  IIRC, bye, Frank

---

From: Henry Spencer <[email-masked]>
Date: Wed, 29 Jun 2005 10:11:15 -0400 (EDT)

On Wed, 29 Jun 2005, Harald Tveit Alvestrand wrote:
> OK, here's the proposed text for the problems I outlined yesterday...

Seems about right.

>    News is more dependent on msg-id uniqueness and fast comparision than

Should be "comparison".

                                                          Henry Spencer
                                                       [email-masked]

---

From: Graham Drabble <[email-masked]>
Date: Wed, 29 Jun 2005 15:23:43 +0100

On 29 Jun 2005 Harald Tveit Alvestrand <[email-masked]> wrote
in news:[email-masked]: 

> --On onsdag, juni 29, 2005 08:33:34 -0400 "Forrest J. Cavalier
> III" <[email-masked]> wrote:
> 
>>
>>>      In no circumstance is it reasonable for software receiving
>>>      a <msg-id> to change behaviour based on a theory about how
>>>      the <msg-id> was generated.
>>
>>
> Sorry, I don't understand. Is there software that tries to *parse*
> a  message-ID?

I think Forest is talking about the $alz convention
(http://www.killfile.org/faqs/cancel.html section IIB). I think
you're note is still OK with this as you don't parse the message-id
as much as add cancel. to what was already there. 

-- 
Graham Drabble
http://www.drabble.me.uk/

---

From: "Forrest J. Cavalier III" <[email-masked]>
Date: Wed, 29 Jun 2005 11:08:45 -0400

> as much as add cancel. to what was already there. 

Sorry, I thought that there was more widespread understanding
among the WG that some packages have a "refuse cybercancels" setting that
causes a relay agent to not accept messages with IDs which start
"<cancel"

I thought, but could be mistaken, that there were some multi-part
naming schemes which used formatted message-IDs, but I am less
sure that anything acts on that.

---

From: Richard Clayton <[email-masked]>
Date: Wed, 29 Jun 2005 16:52:28 +0100

In message <[email-masked]>, Forrest J. Cavalier III
<[email-masked]> writes

>causes a relay agent to not accept messages with IDs which start

Although I can imagine that there exists software that does practically
anything -- I believe that the intent of the conventions is that the
$alz scheme (which means that one is able to predict the cancel message
ID) is to ensure that only one cancel floods out, no matter how many
robots generate it (in the extended scheme with a secret, the cabal of
cancellers share the secret amongst themselves -- same difference!)

Servers that wish to ignore cancels (or wish to distinguish amongst the
various parts) are clearly _intended_ to process the last few elements
of the Path: header field data, looking for the "pseudosite" strings
that they are prepared to accept (or not)  [since they're not in the
UUCP message maps they're some of the most used non-DNS based strings
... cyberspam, mmfcancel, bincancel, spewcancel and doubtless others
I've forgotten]

viz: since we're not producing a taxonomy of strange behaviour, I don't
see a need to document the exciting things that people have dreamt up on
their own to (not) process cancels ...

>sure that anything acts on that.

Since there are better ways of tying multiparts together, I don't think
that leaving a hole for such a scheme is desirable or necessary

- -- 
richard                                              Richard Clayton

They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety.         Benjamin Franklin

---

From: [email-masked] (Ralph Babel)
Date: Wed, 29 Jun 2005 18:16:11 +0200

Frank Ellermann wrote:

> In that case an intentional collision, one <cancel.old@id>
> is enough to cancel the <old@id> spam. In a more
> elaborated form it's <cancel.secret.old@id>, because the
> bad guys started to send a dummy <cancel.old@id> before
> the <old@id> spam, this dummy blocked the spam cancel.
> IIRC,

Actually, you don't recall correctly. The bad guys are the
ones that pollute other people's message-ID space instead
of agreeing on a canonical way of encoding the original
message ID in the cancel message ID _without_ resorting
to the original RHS, e.g. by moving it to the LHS of the
cancel's message ID. Gee, folks, it's not rocket science.

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 29 Jun 2005 19:16:28 +0200

Ralph Babel wrote:

> Actually, you don't recall correctly.

<http://www.xs4all.nl/~rosalind/faq-care.html#cancel-message>
<http://www.killfile.org/faqs/cancel.html#II.B.> $alz-convention
<http://www.killfile.org/faqs/cancel.html#II.C.> cancel example

> Gee, folks, it's not rocket science.

Not guilty, convince Rosalind and Skirvin.  I already had my
fun with trying to cancel two unjustified cancels.  Your idea
also answers Harald's question, sometimes a <msg-id> is parsed.

And you implicitly said that the RHS is important, how about
joining the s/id-left@id-right/id-local@id-domain/ club ?

The news-nntp-uri drafts want at least the "@" (unlike NNTP):
http://purl.net/xyzzy/home/test/draft-lindsey-news-nntp-uri-00
http://purl.net/xyzzy/home/test/draft-ellermann-news-nntp-uri-00

                           Bye, Frank

---

From: Russ Allbery <[email-masked]>
Date: Wed, 29 Jun 2005 10:27:29 -0700

> that they are prepared to accept (or not) [since they're not in the UUCP
> message maps they're some of the most used non-DNS based strings

You have to parse the whole article before you can reject based on the
Path header.  Parsing the message ID lets you decline cancels you're
uninterested in before the article is sent to your server.

INN does indeed support this (off by default) due to significant popular
demand.

The $alz convention is a kludge, though, and I'm not sure that the
environment in which this sort of thing was important, back in the days of
hundreds of thousands of cancels a day, is still as much of a concern.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: Russ Allbery <[email-masked]>
Date: Wed, 29 Jun 2005 10:28:26 -0700

> OK, here's the proposed text for the problems I outlined yesterday:

This looks good to me.

---

From: Bruce Lilly <[email-masked]>
Date: Wed, 29 Jun 2005 15:25:34 -0400

>    The Message-ID header contains a single unique message identifier.

Seems to need "field" per the decision on "header".

>    email is, and

I'm not convinced about "more"; email and news use several of the same
(i.e. combined functionality) UAs and protocols (e.g. IMAP) and the same
use of Message-ID, References, and Supersedes fields, the same "threading"
algorithms in those UAs, etc.  Uniqueness is certainly an issue for email
and for other applications layered on the Message Format, which itself
mentions global uniqueness.  I agree with Henry about "fast" also.

>    some news software would have trouble with the full range 

Some news software may indeed have trouble, but I believe it is important
to distinguish between broken implementations with few global implications
e.g. UAs and isolated sites that have never complied with RFC 822, 850,
1036 syntax through more serious problems (NNTP constraints) to global
problems (none?).  More detail below.
 
>    Email and Netnews in particular.

OK.  I wouldn't mind the latter part expanded to "all applications which
use the Internet Message Format [RFC2822]".
 
> OLD (ABNF): [...] NEW:

As with orig-date and zone, a redefinition of 2822-defined grammar
productions is a problem.  Changing the production name won't help in
this case, as a msg-id is used in precisely three places: the Message-ID,
References, and Supersedes fields, and those are all defined in external
specifications (2822 and 2156).

That leaves implementation notes to deal with any concrete issues.

>    only difference is the case of characters in the <id-right> part.

That's OK as one implementation note.
 
>    If domain literals are used, the syntax found in [RFC2821] section

I'm not sure about that; the 2821 tagging scheme doesn't seem to have
generated much traction.  I hear that a 2821bis draft might appear soon,
perhaps some off-list communication with JCK and IPv6 experts might be
in order.
 
>      more likely to generate unique message-IDs.

OK.  Informative references to RFCs 1054, 1918, 2715, 3330, 3849 would be
helpful to implementers.

>      to change behaviour based on a theory about how the <msg-id> was

Pick British or American spelling; I don't care as long as it's consistent.
 
> care about after generation is that the message-ids are unique.

It would seem that the same reasoning would apply to the LHS, i.e. there's
no problem with <"foo"@example.edu> so long as example.edu doesn't generate
ids like <[email-masked]>, <f\[email-masked]>, etc.

Lacking any concrete statements from other WG participants about specific
issues, and noting that whitespace is already prohibited in a msg-id by
2822 syntax, here's a stab at some more implementation notes:

  NOTE: The NNTP protocol [RFC977] has no provision for quoting and uses
        the space character as a delimiter.  While msg-id syntax prohibits
        CFWS in a msg-id, a space character is permitted within a
        quoted-pair within a quoted-string in id-left, e.g.
        <"foo\ bar"@example.org>.   Implementations SHOULD therefore avoid
        such an id-left construct.

"SHOULD" because a "private" installation of "cooperating" "systems" known
not to use NNTP need not be concerned about NNTP foibles.

My understanding is that the NNTP issues affects only the Message-ID field
directly, so such a note could go in that field's section.  For that
matter, it's an external protocol issue, so it could go in USEPRO or USEAGE.

  NOTE: The NNTP protocol [RFC977] transmits msg-ids in a fixed-length
        command-line.  While the length of a msg-id is constrained (to 997
        octets including angle brackets by the line length limit of
        [RFC2822]), NNTP cannot handle a msg-id longer than 500 octets
        due to the command-line limit.  Implementations SHOULD therefore
        not generate a msg-id longer than 500 octets including angle
        brackets.

Same rationale as above for "SHOULD".  Same regarding placement.

  NOTE: Although the length of a msg-id is constrained to 997 octets, and
        a legal domain name may be up to 255 octets in length, some news
        software implementations silently truncate a msg-id at 255 octets.
        Such implementations SHOULD be upgraded as soon as practicable to
        support a msg-id length of 997 octets.  Implementations that
        exchange messages which may be sent to a legacy implementation
        should limit msg-id length to no more than 255 octets if possible.

Clearly the problem here is some legacy implementations, since a domain name
has always been allowed to be up to 255 octets (RFC 882).  The onus is
placed on ancient implementations to modernize to 1980s technology, while
interoperating implementations are encouraged to play nice where possible
(clearly a site with a 252-octet domain name cannot do so).  Placement in
USEFOR or USEAGE would probably be fine.

2005-06-29:5

---

From: Seth Breidbart <[email-masked]>
Date: Wed, 29 Jun 2005 16:05:23 -0400 (EDT)

> uninterested in before the article is sent to your server.

Telling your neighbor "cyberspam is a synonym for me" means you don't
even have to look at the message ID  because your neighbor won't
attempt to send you those cancels in the first place.

Seth

---

From: Russ Allbery <[email-masked]>
Date: Wed, 29 Jun 2005 13:25:05 -0700

> even have to look at the message ID because your neighbor won't attempt
> to send you those cancels in the first place.

Yes, that's even better, but it assumes cooperative upstream peers that
can do this sort of configuration.  Per the feature requests we got for
INN, this was often not the case.  (Not to mention that many news sites
have hundreds of peers, and contacting them all is rather difficult.)

---

From: Frank Ellermann <[email-masked]>
Date: Wed, 29 Jun 2005 22:35:23 +0200

Bruce Lilly wrote:

> As with orig-date and zone, a redefinition of 2822-defined
> grammar productions is a problem.  Changing the production
> name won't help in this case, as a msg-id is used in
> precisely three places: the Message-ID, References, and
> Supersedes fields,

Also in Control: cancel, the historical s-o-1036 Also-Control,
in article bodies, in news URLs (1738 + several drafts), and
in paths and parameters of http-URLs for various archives,
some of them based on NNTP.

That's why the old idea <msg-id-core> was changed to <msg-id>,
as you say _this_ production name should match RfC 2822.

> all defined in external specifications (2822 and 2156).

As well as 1036, 1738, s-o-1036, and 977bis, all with their
own POV, often some "maximal" concept incompatible with the
other "maximal" ideas.  We need some maximal "least common
denominator" for both syntax and semantics without conflicts.

>> If domain literals are used, the syntax found in [RFC2821]
>> section 4.1.3 is RECOMMENDED.

> I'm not sure about that; the 2821 tagging scheme doesn't seem
> to have generated much traction.  I hear that a 2821bis draft
> might appear soon

ACK, I hope it adopts the STD 66 syntax for domain literals,
maybe modulo the 2821-legacy "IPv6:" tag.

> Informative references to RFCs 1054, 1918, 2715, 3330, 3849
> would be helpful to implementers.

Yes, that's the Bruce as I know him, five RfCs I've never heard
of, I only check it, you say if any important errata exist.

1054: obsoleted by 1112 (STD 5), updated etc., multicasting IPs
1918: private IPs, I didn't recognize the number, shame on me
2715: informational (multicasting again)
3330: informational (special use IPs), added to my local disk,
      apparently the complete IPv4 zoo, nice
3849: IPv6 examples, I didn't recognize the number

Harald's wording for "stay away from domain literals" might be
good enough.  I still like STD 11, Dave's STRONGLY DISCOURAGED,
and those who aren't discouraged simply must know what they do.

>> I think this makes it clear that in message-id, the question
>> of "what" the RHS is matters only to the generator, not to
>> the parsing entity; all we care about after generation is
>> that the message-ids are unique.

But here I strongly disagree, I care _very much_ if somebody
generates a <[email-masked]> without my permission
because it's my name space.  And therefore <id-right> is a bad
name.

> It would seem that the same reasoning would apply to the LHS

No, the name space abuse case is a matter of the <id-domain>,
not of the <id-local>.  As the names say.  As they say it in
various relevant standards in different ways.

Excl. RfC 2822, that dropped the ball with its bad <id-right>
obscuring the real meaning and purpose.  It's not only about
the good idea "unique", it's also about the evil idea "abuse".

 [Further notes incompatible with 997bis and many UAs skipped]

                             Bye, Frank

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Thu, 30 Jun 2005 09:04:20 +0200

> because it's my name space.  And therefore <id-right> is a bad

ah - now I get the concern..... the requirement for uniqueness is not only 
that you generate names in a namespace that you control, but that I do 
*not* generate names in that namespace.

Should we add a "MUST NOT generate msg-ids in id-rights you don't control"?

(That's why I'm concerned with the syntax of RHS domain-literals; unless I 
know what other people are doing, I don't know what namespace I can claim 
without anyone else stomping on me...)

                    Harald

---

From: Russ Allbery <[email-masked]>
Date: Thu, 30 Jun 2005 06:48:42 -0700

Harald Tveit Alvestrand <[email-masked]> writes:

> ah - now I get the concern..... the requirement for uniqueness is not
> only that you generate names in a namespace that you control, but that I
> do *not* generate names in that namespace.

> Should we add a "MUST NOT generate msg-ids in id-rights you don't

The $alz convention is still used for spam cancels, so I think that's at
most a SHOULD NOT.  (I know I was just arguing against caring that much,
but that was more an argument against documenting the convention in at
least the standards-track portions of the drafts.  I'm not sure that we
want to be encouraging it, particularly, but it does solve some real
problems.)

> (That's why I'm concerned with the syntax of RHS domain-literals; unless
> I know what other people are doing, I don't know what namespace I can
> claim without anyone else stomping on me...)

Yup.

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 30 Jun 2005 18:40:29 +0200

Harald Tveit Alvestrand wrote:
 
> now I get the concern..... the requirement for uniqueness is
> not only that you generate names in a namespace that you
> control, but that I do *not* generate names in that
> namespace.

Yes.  Russ' point about the $alz convention not withstanding:

A conventional bid "4 SA" doesn't mean that you want to play
sans atout, or at least not only 4 SA.  And it's a convention
how to break some 1036-rules intentionally and efficiently,
it's the same situation with s-o-1036 or its future grandson.

> Should we add a "MUST NOT generate msg-ids in id-rights you

It's probably clear that I prefer a simple <id-domain> instead
of tons of MUSTard with the same effect for the bad <id-right>.

Yes, that MUST could fix one of these id-left / right problems.
 
> That's why I'm concerned with the syntax of RHS
> domain-literals

And <no-fold-quote> and <no-fold-literal> are also bad names,
we don't want to inherit the 2822-rule about their "semantical
content" (= invisible backslash in a quoted-pair).

A clean minimal interface is a new <msg-id> and from there use
new productions, don't try to overload more old RfC 2822 names
if the syntax or semantics differs from 2822.  Especially if
those differences are very subtle like a "visible" backslash.

                           Bye, Frank

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 30 Jun 2005 16:47:50 GMT

>because it's my name space.  And therefore <id-right> is a bad

But that is not a violation of USEFOR, which is to be a format standard.

It might be a proper matter for attention in USEPRO, but IMO it is really
an argument between you and the abuse dept. of ISP of the perpetrator.

Moreover, if it is someone issuing a cancel for one of your spammed
articles, it might even be a right and proper thing to do :-) .

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 30 Jun 2005 16:21:33 GMT

>   Email and Netnews in particular.

There are still people (even, apparently, on this list) who seem to think
that it is only "legacy software" that will "have trouble", and look
forward to the day when these irksome restrictions can be lifted (and hence
seek to express all these limitations by verbiage). Therefore I would like
to see something a little stronger, such as:

"Netnews is critically dependent on the uniqueness of <msg-id>s and on the
ability to compare them rapidly. Retaining the full range of possible
<msg-id>s permitted by RFC 2822 would introduce severe performance
penalities, and would also be in conflict with [NNTP]; this section
therefore ..."

>                           *mqtext mqspecial *mqtext )
Correct.

>   only difference is the case of characters in the <id-right> part.

That seems a protocol issue to me, and therefore should go in USEPRO.
Indeed, USEPRO already contains (7.3):

        NOTE: Some old software may treat message identifiers that
        differ only in case within their <id-right> part as equivalent,
        and implementors of agents that generate message identifiers
        should be aware of this.

That could be altered if people want it stronger.

>   If domain literals are used, the syntax found in [RFC2821] section

Except that may change in 2821bis, and RFC 2822 took great care to allow
for all sorts of domain literals that might be invented in the future. So
I think we should stick to the RFC 2822 line. Perhaps:

   If <no-fold-literal>s are used, they SHOULD be restricted to the
   formats allowed for Internet addresses by the relevant standards.

(Which is slightly stronger that what is said in 3.4.1 of RFC 2822, but
uses some of the same terminology.)

>     be a domain name or a domain literal. Domain literals are troublesome
                       ^^^^^^^^^^^^^^^^^^^
		       (or a domain literal IP address)
>     since many IP addresses are not globally unique; domain names are
            ^^^^
	    some
>     more likely to generate unique message-IDs
                                     ^^^^^^^^^^^
				     message identifiers.
>     to change behaviour based on a theory about how the <msg-id> was

Just using correct terminology from RFC 2822. "message-ID" is not a
defined technical term. "Message identifier" is the semantic concept
(ignoring '<' and '>'), and <msg-id> is its syntactic embodiment.

The only genuine not-globally-unique IP addresses are the ones set aside
for local use (127.0.0.1 is the best-known). Even IP addresses obtained on
a DHCP lease are uniquely yours so long as you hold them. Hence
s/many/some/.

And just to cover some conventions in current and private use,
s/In no circumstance is it reasonable/It would be unwise{unreasonable}/.
"In no circumstances" is, in any case, a bit too strong to go in a NOTE.

>Makes sense?

Yes indeed.

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 30 Jun 2005 16:39:12 GMT

>use the Internet Message Format [RFC2822]".

Be careful there. The present wording already covers "all applications
which use the Internet Message Format", but it also includes anything
else (such as X.400) which might use something compatible with our
"message identifiers".

>has always been allowed to be up to 255 octets (RFC 882).

No, the limit (250, not 255) is hard coded into many implementations and,
more to the point, it is hard coded into the new NNTP standard-to-be,
which has just passed its IESG Last Call and is now in the RFC Editor's
queue.