Subject: [apps-discuss] [Technical Errata Reported] RFC6729 (3337)

From: RFC Errata System <[email-masked]>
Date: Thu, 6 Sep 2012 14:36:46 -0700 (PDT)

The following errata report has been submitted for RFC6729,
"Indicating Email Handling States in Trace Fields".

--------------------------------------
You may review the report below and at:
http://www.rfc-editor.org/errata_search.php?rfc=6729&eid=3337

--------------------------------------
Type: Technical
Reported by: D. Stussy <[email-masked]>

Section: 3

Original Text
-------------
In Section 6, the "state" clause is added to the Additional-
   Registered-Clauses IANA sub-registry.  The ABNF for this clause is:

      State = CFWS "state" FWS queue-state-keyword [ "/" value ]
...

Corrected Text
--------------

Notes
-----
Clarification needed:  Does this "state" clause keyword place before, after, or in either order with the "priority" clause keyword specified in RFC 6710?  RFC 5321 does not specify an order for additional "Received:" header clauses defined after its publication.  The order of additional clauses may need to be defined for proper parsing to determine validity of messages for spam classification or determination of other subversive purposes (as invalid values may indicate bad messages).

(See also the note filed in errata #1 filed for RFC 6710.)

Instructions:
-------------
This errata is currently posted as "Reported". If necessary, please
use "Reply All" to discuss whether it should be verified or
rejected. When a decision is reached, the verifying party (IESG)
can log in to change the status and edit the report, if necessary. 

--------------------------------------
RFC6729 (draft-ietf-appsawg-received-state-04)
--------------------------------------
Title               : Indicating Email Handling States in Trace Fields
Publication Date    : September 2012
Author(s)           : D. Crocker, M. Kucherawy
Category            : PROPOSED STANDARD
Source              : Applications Area Working Group
Area                : Applications
Stream              : IETF
Verifying Party     : IESG

---

From: Ned Freed <[email-masked]>
Date: Thu, 06 Sep 2012 15:52:11 -0700 (PDT)

I strongly disagree that this is in any way, shape, or form an appropriate
errata for RFC 6729 and I believe it should be rejected outright.

RFC 5321 neither requires any ordering nor attaches any significance to the
ordering of additional Received: field clauses. To change that would require
updating RFC 5321 itself, and that in turn would require figuring out some
ordering scheme that works across future registrations.

> Notes
> -----
> Clarification needed:  Does this "state" clause keyword place
> before, after, or in either order with the "priority" clause keyword specified
> in RFC 6710?

What semantics would attach to such an ordering anyway? That's the only reason
I can concieve of for requiring an ordering of these clauses.

> RFC 5321 does not specify an order for additional "Received:"
> header clauses defined after its publication.

Exactly.

> The order of additional clauses
> may need to be defined for proper parsing

If that's indeed the case then it's a flaw in RFC 5321 and would have to be
corrected there. Consider: Since the list of additional clauses can be amended
at any time, software that intends to parse these clauses has to be able to
accomodate previously unknown clauses. Moreoever, knowing the relative ordering
of the clauses your software is written to understand doesn't help you at all
with unknown clauses.

However, this can be seen to be a nonissue simply by looking at the
definition in RFC 5321:

   Additional-Registered-Clauses  = CFWS Atom FWS String

So additional clauses always consists of either a pair of atoms or an  atom
following by a quoted string. No other syntaxes are allowed and this is
trivially easy to parse.

> to determine validity of messages for
> spam classification or determination of other subversive purposes (as invalid
> values may indicate bad messages).

First, while it is true that agency may sometimes be inferred indirectly
through syntax use, both legal and illegal, this is *lightyears* away from
anything that we should be standardizing. The slope here is perilous in the
extreme and in many cases we're already sliding down it: It is not at all
uncommon for perfectly legal messages to be rejected by overzealous antispam
software simply because they employed some bit of syntax that's commonly
associated with some spam agent or other. 

The last thing we want to do is specify some ordering requirement that could be
amended at any time by the addition of a new clause.

				Ned

---

From: Barry Leiba <[email-masked]>
Date: Thu, 6 Sep 2012 19:29:15 -0400

> errata for RFC 6729 and I believe it should be rejected outright.

Already has been, hours ago.

> ordering scheme that works across future registrations.

's what I said when I rejected it.

Barry

---

From: Ned Freed <[email-masked]>
Date: Thu, 06 Sep 2012 17:23:51 -0700 (PDT)

> Already has been, hours ago.

Good.

> 's what I said when I rejected it.

The one action I would consider in regards to this - and it does NOT rise
to the level of an errata - is to make the lack of ordering, as well as the
*reason* why ordering doesn't help, clear in a future revision of RFC 5321,
assuming that ever happens.

Future-proofing and all that.

				Ned

---

From: John C Klensin <[email-masked]>
Date: Thu, 06 Sep 2012 21:09:20 -0400

I doubt that Ned's note needs any additional reinforcement but,
for the record and as editor, if he hadn't responded to this
first, my response would have been among much that same lines
(although probably not as eloquent).  The one thing I might add
is that we have known for about as long as I can remember that
the switch in parsing rules in the middle of the "Received:"
header field from keyword-dependent with per-keyword syntax to
strict Atom-String pairs up to the semicolon that sets off the
date-time was a bit inelegant and perhaps even a trap for the
unwary.  It was already old (as part of <Opt-info>" when we
added the explicit provision for Additional-Registered-Clauses
in RFC 5321.  I vaguely recall discussing it then with the
conclusion that nothing could be done about it that would not
have been worse.   So complaints about it at this stage are not
only incorrect, but are well over a decade too late. 

best,
   john

--On Thursday, September 06, 2012 15:52 -0700 Ned Freed
<[email-masked]> wrote:

> I strongly disagree that this is in any way, shape, or form an
> appropriate errata for RFC 6729 and I believe it should be
> rejected outright.
> 
> RFC 5321 neither requires any ordering nor attaches any
> significance to the ordering of additional Received: field
> clauses. To change that would require updating RFC 5321
> itself, and that in turn would require figuring out some
>>       State = CFWS "state" FWS queue-state-keyword [ "/"
>>       value ] ...
> 
> 
> 
>> Corrected Text
>> --------------
> 
> 
>> Notes
>> -----
>> before, after, or in either order with the "priority" clause
>> keyword specified in RFC 6710?
> 
> What semantics would attach to such an ordering anyway? That's
> the only reason I can concieve of for requiring an ordering of
> these clauses.
> 
> If that's indeed the case then it's a flaw in RFC 5321 and
> would have to be corrected there. Consider: Since the list of
> additional clauses can be amended at any time, software that
> intends to parse these clauses has to be able to accomodate
> previously unknown clauses. Moreoever, knowing the relative
> ordering of the clauses your software is written to understand
> doesn't help you at all with unknown clauses.
> 
> However, this can be seen to be a nonissue simply by looking
> at the definition in RFC 5321:
> 
> So additional clauses always consists of either a pair of
> atoms or an  atom following by a quoted string. No other
> syntaxes are allowed and this is trivially easy to parse.
> 
>> spam classification or determination of other subversive
>> purposes (as invalid values may indicate bad messages).
> 
> First, while it is true that agency may sometimes be inferred
> indirectly through syntax use, both legal and illegal, this is
> *lightyears* away from anything that we should be
> standardizing. The slope here is perilous in the extreme and
> in many cases we're already sliding down it: It is not at all
> uncommon for perfectly legal messages to be rejected by
> overzealous antispam software simply because they employed
> some bit of syntax that's commonly associated with some spam
> agent or other. 
> 
> The last thing we want to do is specify some ordering
> requirement that could be amended at any time by the addition
> of a new clause.
> 
> 				Ned
> _______________________________________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss

---

From: [email-masked]
Date: Thu, 6 Sep 2012 17:43:17 -0700 (PDT)

--- On Thu, 9/6/12, Ned Freed <[email-masked]> wrote:
> [snip]

Nor should it.  Order should be introduced as additional clauses are defined, even if that order is "any."  As it appears in RFC 5321, it was merely a reservation of a place-holder for future use.
 
> of the clauses your software is written to understand doesn't help you at all

Tell me why "previously unknown clauses" not registered with IANA (i.e. as a result of the RFC process) should be accepted?  To allow such is the same as accepting random text, so one may as well say, "Spam me."

> following by a quoted string. No other syntaxes are allowed and this is

Don't be so certain.  The "for" clause may take (even if it's not the "best current practice") multiple (even "infinite") values.  Should I assume that the first value that does not match the appropriate syntax (a mailbox or a list) must then be a literal matching an optional additional clause keyword added by a later RFC (such as 6710 or 6729)?  If these additional keywords may then appear in any order but only once, I now also have to remember whether I've encountered it before; a complication to the linear parsing we had before the second of these additions.

It doesn't matter if one is parsing left-to-right through the "for" clause (if present) or right-to-left from the semicolon separating the timestamp; the problem of multiple parsing paths encountered due to "any" order occurs.  For "n" additional clauses appearing in any order, there are n! permutations not counting omissions.

Try writing a Turing machine for the syntax.  Without multiple "for" clauses, the machine is finite and without loops.  At the clause level, the machine is even linear in design.  Now, add these additional clauses.  If one loops back to the state "after-for-value", one must now also remember whether we've travelled an additional keyword path to the "found-X" state (where "X" is an additional clause), and that violates TM design.  A proper TM will suffer the n! parsing path problem, and although state-finite, it gets very complex quickly.
 
> associated with some spam agent or other. 

I would say that the mere existence of the "best current practices" series of RFCs would contradict that position.

> amended at any time by the addition of a new clause.

We already have an ordering requirement defined in the syntax itself for the first six clauses defined in RFC 5321.  Messages with these clauses out of order are not RFC (or standards*) compliant and I have no problem rejecting them as such.  I see nothing wrong with defining a default order of future additions to be in the order they were published by the RFC process.

All we have per RFC 5321 is that additional clauses added by future changes or additions follow the existing ones.  It is up to these future documents to define the characteristics of these additional items.  Simply put, this RFC left out a characteristic which was deferred to it.

* - A standard is a policy, usage, or custom which is ENFORCED.  Standards track RFC documents are merely proposals.

---

From: Ned Freed <[email-masked]>
Date: Fri, 07 Sep 2012 13:04:02 -0700 (PDT)

> Nor should it.  Order should be introduced as additional clauses are defined, even if that order is "any."  As it appears in RFC 5321, it was merely a reservation of a place-holder for future use.
 
No, it should not. What should happen is that additional clauses that require
ordering should be rejected as being badly designed.

> Tell me why "previously unknown clauses" not registered with IANA (i.e. as a
> result of the RFC process) should be accepted? 

Read what I said again. I was talking about clauses registered after some piece
of software was written and deployed. Registration can occur at any time, and
in the case of 

> To allow such is the same as
> accepting random text, so one may as well say, "Spam me."

On what planet does it make sense for spammers to introduce new Received:
clauses that will get them noticed as such?

> Don't be so certain.

My certainty, if you can call it that, derives from the way RFC 5321 is
written. It only allows the addition of clauses with these forms and it
carefully defines what preceeds them.

> The "for" clause may take (even if it's not the "best
> current practice") multiple (even "infinite") values.

RFC 5321 (which is a draft standard, not a BCP), defines "for" clauses
thusly:

   For            = CFWS "FOR" FWS ( Path / Mailbox )
   Path           = "<" [ A-d-l ":" ] Mailbox ">"
   Mailbox        = Local-part "@" ( Domain / address-literal )

Sure looks like it only takes one value to me. And this is actually unchanged
from RFC 822, which also specified single value "for" clauses.

In other words, the syntax you are now describing is currently invalid and has
been a standards violation since at least 1982.

> Should I assume that the
> first value that does not match the appropriate syntax (a mailbox or a list)
> must then be a literal matching an optional additional clause keyword added by
> a later RFC (such as 6710 or 6729)?

What you should do is parse the crap people put in Received: fields as best you
can given that all sorts of syntax violations are common. (I haven't seen the
multiple for value one myself, but I've seen many others and have no problem
believing it exists.)

What you appear to be doing instead is assuming that despite the fact that
rules about the values the clauses take as well as the order the clauses 
appear in have been, let's say less than scrupulously followed for 30 years,
that all of a sudden we can reject the consensus behind three standards track
documents, one a draft standard, and instead implement a complex administrative
procedure for forcing an ordering and this time things will be different and
people will pay attention to the new rules while continuing to violate the
old ones?

> If these additional keywords may then
> appear in any order but only once, I now also have to remember whether I've
> encountered it before; a complication to the linear parsing we had before the
> second of these additions.

Yep, that's how it works.

> It doesn't matter if one is parsing left-to-right through the "for" clause
> (if present) or right-to-left from the semicolon separating the timestamp; the
> problem of multiple parsing paths encountered due to "any" order occurs.  For
> "n" additional clauses appearing in any order, there are n! permutations not
> counting omissions.

Yes, but requiring an ordering doesn't help, because unless you've figured out
a way to magically update all your software in the field the instant a new
registered field shows up it is going to fail when that happens. And then
there's the possibility of unregistered fields, not to mention out of order
fields and various other messes, which are commonly produced by completely
legitimate agents.

> Try writing a Turing machine for the syntax.  Without multiple "for" clauses,
> the machine is finite and without loops.  At the clause level, the machine is
>  even linear in design.  Now, add these additional clauses.  If one loops back
> to the state "after-for-value", one must now also remember whether we've
> travelled an additional keyword path to the "found-X" state (where "X" is an
> additional clause), and that violates TM design.  A proper TM will suffer the
> n! parsing path problem, and although state-finite, it gets very complex
> quickly.
 
I would not waste my time on such a pointless exercise. Our software parses
Received: fields all the time using a variety of tecniques and a handful of
heuristics. It seems to work pretty well.

> RFCs would contradict that position.

This plus your earlier comment means you need to reread RFC 2026 to better
understand document statuses and what BCPs are actually for.

> We already have an ordering requirement defined in the syntax itself for the
> first six clauses defined in RFC 5321.  Messages with these clauses out of
> order are not RFC (or standards*) compliant and I have no problem rejecting
> them as such.  I see nothing wrong with defining a default order of future
> additions to be in the order they were published by the RFC process.

That's your opinion, which of course you're entitled to have. What I'm not
seeing is a groundswell of support, let alone a strong consensus, to change an
existing standard to say what you want it to say.

There are quite a few things about RFC 5321 and RFC 5322, not to mention
several other documents I actually coauthored, that I don't like and would
change if I were King of the Standards for a day. But I'm not, and neither are
you.

> All we have per RFC 5321 is that additional clauses added by future changes
> or additions follow the existing ones.  It is up to these future documents to
> define the characteristics of these additional items.  Simply put, this RFC
> left out a characteristic which was deferred to it.

I see no text in RFC 5321 to support this view.

> * - A standard is a policy, usage, or custom which is ENFORCED.  Standards
> track RFC documents are merely proposals.

Even if grant this is true, which I don't, I completely fail to see the
relevance.

But in any case, I'm done with this. According to Barry your errata was
rejected  even before I posted my response and I see no point in discussing
this further.

				Ned