Subject: [spfbis] Possible ABNF nit with Received-SPF - #53

From: Scott Kitterman <[email-masked]>
Date: Tue, 05 Feb 2013 17:50:04 -0500

Need some help with this one:

http://trac.tools.ietf.org/wg/spfbis/trac/ticket/53

 Message posted by Barry Leiba on 4 Sept 2012:

    that was intended to support unknown mechanisms. So, if you had an SPF
    record from draft-mengwong-spf-00:

        v=spf1 a mx ptr domainkeys:_dk.%{d} -all

    The Received-SPF header would look like:

        Received-SPF: unknown domainkeys=_dk.example.com

If that's the case, the text needs work to make it clear, because it's 
decidedly not. The text below the grammar lists key names and the meanings of 
the values. "mechanism" is listed as a key name in that text. So there's a 
problem somewhere that needs to be cleared up.

​http://www.ietf.org/mail-archive/web/spfbis/current/msg02608.html

Downthread from the referenced message, there's a proposed simplification from 
Arthur Thisell, but I think it's based on the (I believe incorrect) assumption 
that mechanism isn't used.

Suggestions please.

Scott K

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 5 Feb 2013 21:57:09 -0800

I think the ABNF is correct as-is, but there is no example that illustrates
the use of mechanisms within a Received-SPF, which would clear up the
confusion.

The current ABNF is mean to enable things like:

Received-SPF: pass mx=something; a=something; ...

So "mechanism" doesn't need to be quoted, because it expands into all of
the available mechanism names.  What Barry was doing is assuming one wants
to produce:

Received-SPF: pass mechanism=mx; ...

...which was not the intention.

Does that sound right, Barry?  Scott?

There is an error in the example upthread though, because it's impossible
to generate "Received-SPF: unknown ..." as "unknown" isn't a valid result
string.

-MSK

On Tue, Feb 5, 2013 at 2:50 PM, Scott Kitterman <[email-masked]> wrote:

> http://www.ietf.org/mail-archive/web/spfbis/current/msg02608.html
>
> _______________________________________________
> spfbis mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/spfbis
>

---

From: Scott Kitterman <[email-masked]>
Date: Wed, 06 Feb 2013 01:26:36 -0500

I fear not.  I looked in the internal pyspf tests and it has things like 
mechanism=mx/24, so at best it's not clear.  Mail::SPF seems to only put the 
mechanisms in comments and not use this bit of the ABNF.

Unknown is the old name for temperror, so it used to be a valid result.  
Perhaps a copy/paste from an old example.

Scott K

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 5 Feb 2013 22:35:03 -0800

That makes it an open topic.  Ugly.

Do we have any indication of a consensus of what various implementations
do?  Do we need to undertake such a survey?

-MSK

On Tue, Feb 5, 2013 at 10:26 PM, Scott Kitterman <[email-masked]> wrote:

> > > decidedly not. The text below the grammar lists key names and the
> meanings
> > > of
> > > Downthread from the referenced message, there's a proposed
> simplification
> > > from

---

From: Scott Kitterman <[email-masked]>
Date: Wed, 06 Feb 2013 01:40:46 -0500

I think we can pick what we want.  Mechanism is there for forensics, not for 
filtering, so changing it won't affect interoperability.  Personally, I think 
mx=example.com is a lot more useful than mechanism=mx since there may be more 
than one of a mechanism type in a record, but I'm open to either approach.  I 
do think we should make it clear and then give an example.

Scott K

On Tuesday, February 05, 2013 10:35:03 PM Murray S. Kucherawy wrote:
> > > I think the ABNF is correct as-is, but there is no example that
> > 
> > illustrates
> > 
> > > the available mechanism names.  What Barry was doing is assuming one
> > 
> > wants
> > 
> > > to produce:
> > > 
> > > On Tue, Feb 5, 2013 at 2:50 PM, Scott Kitterman <[email-masked]>
> > 
> > wrote:
> > > >     that was intended to support unknown mechanisms. So, if you had an
> > 
> > SPF
> > 
> > > > the values. "mechanism" is listed as a key name in that text. So
> > 
> > there's a
> >

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Thu, 7 Feb 2013 00:45:04 -0800

It's true that changing this doesn't affect interoperability of policy
evaluation, but it does affect anything that parses this field one way if
we change it to the other.  Consumers of this field are mostly what I'm
concerned with here.

However, if nobody's heard of something other than humans that eats
Received-SPF fields, or if those that do exist already agree on how it's
done, then I guess we just come to consensus on which of the two syntaxes
we want and run with it.

-MSK

On Tue, Feb 5, 2013 at 10:40 PM, Scott Kitterman <[email-masked]> wrote:

> > > mechanism=mx/24, so at best it's not clear.  Mail::SPF seems to only
> put
> > > the

---

From: Scott Kitterman <[email-masked]>
Date: Thu, 07 Feb 2013 07:26:23 -0600

There are non-human consumers of Received-SPF.  None that I'm aware of care about mechanism.

Scott K

"Murray S. Kucherawy" <[email-masked]> wrote:

>> I think we can pick what we want.  Mechanism is there for forensics,
>not
>> for
>> filtering, so changing it won't affect interoperability.  Personally,
>I
>> think
>> mx=example.com is a lot more useful than mechanism=mx since there may
>be
>> more
>> than one of a mechanism type in a record, but I'm open to either
>approach.
>>  I
>> > On Tue, Feb 5, 2013 at 10:26 PM, Scott Kitterman
><[email-masked]>
>> wrote:
>> > > I fear not.  I looked in the internal pyspf tests and it has
>things
>> like
>> > > > So "mechanism" doesn't need to be quoted, because it expands
>into
>> all of
>> > > > to generate "Received-SPF: unknown ..." as "unknown" isn't a
>valid
>> > > > result
>> > > > string.
>> > > >
>> > > > -MSK
>> > > >
>> > > > >     that was intended to support unknown mechanisms. So, if
>you
>> had an
>> > >
>> > > SPF
>> > >
>> > > > > If that's the case, the text needs work to make it clear,
>because
>> it's
>> > > > > Arthur Thisell, but I think it's based on the (I believe
>incorrect)
>> > > > > assumption
>------------------------------------------------------------------------
>

---

From: Scott Kitterman <[email-masked]>
Date: Sun, 10 Feb 2013 01:20:13 -0500

I think the mechanism=value construct is what we should go with as it provides 
the most information.  We end up with:

key              = "client-ip" / "envelope-from" / "helo" /
                   "problem" / "receiver" / identity /
                    mechanism / name

In the ABNF, if I'm reading it right.  That is unchanged from RFC 4408, which 
is good.  That some people read it wrong is not good, so I'll add an example.

Scott K

On Thursday, February 07, 2013 07:26:23 AM Scott Kitterman wrote:
> >On Tue, Feb 5, 2013 at 10:40 PM, Scott Kitterman <[email-masked]>
> >
> >wrote:
> >> > Do we have any indication of a consensus of what various
> >
> >implementations
> >
> >> > > mechanism=mx/24, so at best it's not clear.  Mail::SPF seems to
> >
> >only
> >
> >> put
> >> 
> >> > > the
> >> > > Unknown is the old name for temperror, so it used to be a valid
> >
> >result.
> >
> >> > > On Tuesday, February 05, 2013 09:57:09 PM Murray S. Kucherawy
> >
> >wrote:
> >> > > > the use of mechanisms within a Received-SPF, which would clear
> >
> >up the
> >
> >> > > > confusion.
> >> > > > 
> >> > > > the available mechanism names.  What Barry was doing is
> >
> >assuming one
> >
> >> > > wants
> >> > > 
> >> > > > to produce:
> >> > > > 
> >> > > > On Tue, Feb 5, 2013 at 2:50 PM, Scott Kitterman
> >
> ><[email-masked]>
> >
> >> > > wrote:
> >> > > > > decidedly not. The text below the grammar lists key names and
> >
> >the
> >
> >> > > meanings
> >> > > 
> >> > > > > of
> >> > > > > the values. "mechanism" is listed as a key name in that text.
> >
> >So
> >
> >> > > there's a
> >> > >

---

From: Stuart D Gathman <[email-masked]>
Date: Mon, 11 Feb 2013 16:38:21 -0500

> is good.  That some people read it wrong is not good, so I'll add an example.
Well, I am one of those people that read it wrong.  Pyspf generates 
mechanism=mx:example.com, not mx=example.com .  I suppose for syntax 
errors, problem=ip:1.2.3.4 is used rather than mechanism=ip:1.2.3.4 ?  
Also, pyspf generates identity=mfrom or identity=helo, not 
mfrom=[email-masked] (which would be redundant with 
envelope-from=[email-masked]) or helo=mail.example.com (which is also 
redundant).  I assumed "identity" identified which identity was 
responsible for the result.  But that is because "identity" is actually 
quoted in RFC4408, and the above is wrong/changed!  Are we removing 
x-name for extended keys?

---

From: Stuart D Gathman <[email-masked]>
Date: Mon, 11 Feb 2013 16:50:07 -0500

On 02/11/2013 04:38 PM, Stuart D Gathman expounded in part:
> responsible for the result.  But that is because "identity" is 
> actually quoted in RFC4408, and the above is wrong/changed!  Are we 
> removing x-name for extended keys?
(Sorry about typos in the above)

Given that rfc4408 [7] says, "if no mechanisms matched, substitute the 
word "default", I think my interpretation is the more reasonable, and 
"mechanism" should in fact be quoted.

Here is a production Received-SPF:

Received-SPF: Neutral (mail.bmsi.com: 67.23.163.235 is neither permitted 
nor denied by domain of dotperfectionprinting.com) 
client-ip=67.23.163.235; 
envelope-from="bounce-2703-1431319507-gmeagher=[email-masked]"; 
helo=smtp.dotperfectionprinting.com; receiver=mail.bmsi.com; 
mechanism=?all; x-helo-spf=neutral; identity=mailfrom

If we do what I think people are saying, it would look like this:

Received-SPF: Neutral (mail.bmsi.com: 67.23.163.235 is neither permitted 
nor denied by domain of dotperfectionprinting.com) 
client-ip=67.23.163.235; 
envelope-from="bounce-2703-1431319507-gmeagher=[email-masked]"; 
helo=smtp.dotperfectionprinting.com; receiver=mail.bmsi.com; all; 
x-helo-spf=neutral; identity=mailfrom

which is *not* an improvement.

I vote that "mechanism" should be quoted in the key list.  Is there any 
existing practice the other way?

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 11 Feb 2013 16:52:40 -0500

On Monday, February 11, 2013 04:38:21 PM Stuart D Gathman wrote:
> > I think the mechanism=value construct is what we should go with as it
> > provides the most information.  We end up with:
> > 
> > In the ABNF, if I'm reading it right.  That is unchanged from RFC 4408,
> > which is good.  That some people read it wrong is not good, so I'll add
> > an example.

Identity tells you if it was mail from or helo:

   identity         = "mailfrom"   ; for the "MAIL FROM" identity
                      / "helo"     ; for the "HELO" identity
                      / name       ; other identities

It being quoted in 4408 was a bug in the ABNF.  It should actually be expanded 
as listed in the ABNF.

We removed any references on X- due to RFC 6648.

Scott K

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 11 Feb 2013 17:04:09 -0500

On Monday, February 11, 2013 04:50:07 PM Stuart D Gathman wrote:
> envelope-from="bounce-2703-1431319507-gmeagher=schayer.com@dotperfectionprin
> ting.com"; helo=smtp.dotperfectionprinting.com; receiver=mail.bmsi.com;
> envelope-from="bounce-2703-1431319507-gmeagher=schayer.com@dotperfectionprin
> ting.com"; helo=smtp.dotperfectionprinting.com; receiver=mail.bmsi.com; all;

I can see your point.  The bar "all" isn't very understandable as is.  I 
hadn't thought about the all case when I made my proposal.  I had been 
thinking that in the case of matching a mechanism you'd be trading between 
something like mechanism=mx and mx=smtp.example.com.  I think the latter is 
more interesting, particularly in the case where there's more than one 
mechanism of a type in a record (or maybe it matched something from an 
included record).

Is there a better way to present the "all" case than what we have proposed now 
without losing the added information one gets with a literal interpretation of 
4408?

We have an ambiguity we have to resolve here and so I think we have some 
freedom to solve it the 'best' way.

Scott K

---

From: Stuart D Gathman <[email-masked]>
Date: Mon, 11 Feb 2013 18:08:50 -0500

> freedom to solve it the 'best' way.
Here is another production example:

Received-SPF: Pass (mail.bmsi.com: domain of usexp.bmsi.com designates 
50.56.123.192 as permitted sender) client-ip=50.56.123.192; 
envelope-from="SRS0=ivzp/=MD=usexp2.usexpressfreight.com=[email-masked]"; 
helo=fwd1.xif.com; receiver=mail.bmsi.com; mechanism="a:fwd1.xif.com"; 
identity=mailfrom

The full mechanism is included for diagnostic purposes.

"Mechanism" should be a literal.

---

From: Stuart D Gathman <[email-masked]>
Date: Mon, 11 Feb 2013 18:12:42 -0500

> It being quoted in 4408 was a bug in the ABNF.  It should actually be expanded
Are you proposing that instead of "identity=mailfrom", R-S should say 
just "mailfrom" ?   That works better than unquoting "mechanism", but I 
still like the clarity of identity=<one of several standard values>.

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 11 Feb 2013 18:22:07 -0500

On Monday, February 11, 2013 06:08:50 PM Stuart D Gathman wrote:
> >> envelope-from="bounce-2703-1431319507-gmeagher=schayer.com@dotperfectionp
> >> rin ting.com"; helo=smtp.dotperfectionprinting.com;
> >> receiver=mail.bmsi.com; mechanism=?all; x-helo-spf=neutral;
> >> identity=mailfrom
> >> 
> >> envelope-from="bounce-2703-1431319507-gmeagher=schayer.com@dotperfectionp
> >> rin ting.com"; helo=smtp.dotperfectionprinting.com;
> >> receiver=mail.bmsi.com; all; x-helo-spf=neutral; identity=mailfrom
> >> 
> > Is there a better way to present the "all" case than what we have proposed
> > now without losing the added information one gets with a literal
> > interpretation of 4408?
> > 
> envelope-from="SRS0=ivzp/=MD=usexp2.usexpressfreight.com=[email-masked]
> om"; helo=fwd1.xif.com; receiver=mail.bmsi.com; mechanism="a:fwd1.xif.com";
> identity=mailfrom
> 

OK.  I think that clearly either identity or mechanism is wrong in RFC 4408:

   key              = "client-ip" / "envelope-from" / "helo" /
                      "problem" / "receiver" / "identity" /
                       mechanism / "x-" name / name

It also seems to me like we ought to take a step back and reconsider which one 
that is.

Scott K

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 11 Feb 2013 18:26:53 -0500

>still like the clarity of identity=<one of several standard values>.

I liked mailfrom="value", but now I'm not sure.

Scott K

---

From: Stuart Gathman <[email-masked]>
Date: Mon, 11 Feb 2013 22:11:00 -0500

> I liked mailfrom="value", but now I'm not sure.
That is already covered by envelope-from="value".  Your choices are 
identity=mailfrom or just mailfrom (if identity should not have been 
quoted).

---

From: Alessandro Vesely <[email-masked]>
Date: Tue, 12 Feb 2013 12:39:58 +0100

On Mon 11/Feb/2013 23:04:09 +0100 Scott Kitterman wrote:
>> Received-SPF: [...]; mechanism=?all; [...]
>> 
>> Received-SPF: [...]; all; [...]
>> 

How about:

   "mechanism" / mechanism
?

At least, that can stimulate some readers to think about it :-)

---

From: Stuart D Gathman <[email-masked]>
Date: Tue, 12 Feb 2013 12:38:18 -0500

> At least, that can stimulate some readers to think about it :-)
Then what do you do with -mx/24 ?   (emails are *not* sent from the MXes 
or their subnet for this domain)

Received-SPF: [...]; mx="-/24'; [...]
or
Received-SPF: [...]; "-mx/24"; [...]

Look, "mechanism" is quoted, it is *supposed* to a keyword.  This 
mx=<value> idea is half baked, not in rfc4408, and just doesn't work.  
Can we please drop it?

The correct answer is:
Received-SPF: [...]; mechanism="-mx/24"; [...]

End of story.

---

From: Stuart D Gathman <[email-masked]>
Date: Thu, 21 Feb 2013 18:03:15 -0500

> do think we should make it clear and then give an example.
I disagree.  What rfc4408 calls for is mechanism="mx:example.com" (if 
mx:example.com was indeed the mechanism that matched).  It don't know 
where the ideas of "mechanism=mx" (unless there truly was a bare mx) or 
"mx=example.com" come from.  All the keys listed are keys.  Including 
"identity".  I believe the missing quotes on mechanism were a typo.

---

From: Scott Kitterman <[email-masked]>
Date: Thu, 21 Feb 2013 18:10:50 -0500

On Thursday, February 21, 2013 06:03:15 PM Stuart D Gathman wrote:
> > I think we can pick what we want.  Mechanism is there for forensics, not
> > for filtering, so changing it won't affect interoperability.  Personally,
> > I think mx=example.com is a lot more useful than mechanism=mx since there
> > may be more than one of a mechanism type in a record, but I'm open to
> > either approach.  I do think we should make it clear and then give an
> > example.
> 

I'm somewhat inclined to agree with you after the last discussion on this.

I think the ABNF does need to (additionally) be clarified to make it clear 
that:

mechanism="mx:example.com"

is preferred over:

mechanism="mx"

I think the ABNF would lead one towards the latter now (since 
mechanism="mx:example.com" is mechanism:domain-spec, not mechanism.

Scott K

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Fri, 22 Feb 2013 12:57:27 -0800

I don't think I'm partial to either form but we need to ensure that, as
much as possible, any change we make now doesn't render existing
implementations invalid.

-MSK

---

From: Scott Kitterman <[email-masked]>
Date: Sat, 23 Feb 2013 18:26:44 -0500

> mechanism="mx:example.com" is mechanism:domain-spec, not mechanism.

Now that I've had time to read the ABNF and think it through, I believe it 
already does this.  The ABNF definition of each mechanism includes the domain-
spec, so no further change is needed.  I believe all that's needed, now that I 
understand the issue better, is:

=== modified file 'trunk/ScottK/draft-ietf-spfbis-4408bis-11.xml'
--- trunk/ScottK/draft-ietf-spfbis-4408bis-11.xml       
revid:[email-masked]
+++ trunk/ScottK/draft-ietf-spfbis-4408bis-11.xml       2013-02-23 23:23:42 
+0000
@@ -1897,8 +1897,8 @@
 key-value-pair   = key [CFWS] "=" ( dot-atom / quoted-string )
 
 key              = "client-ip" / "envelope-from" / "helo" /
-                   "problem" / "receiver" / identity /
-                    mechanism / name
+                   "problem" / "receiver" / "identity" /
+                    "mechanism" / name
 
 identity         = "mailfrom"   ; for the "MAIL FROM" identity
                    / "helo"     ; for the "HELO" identity
@@ -2708,8 +2708,8 @@
 key-value-pair   = key [CFWS] "=" ( dot-atom / quoted-string )
 
 key              = "client-ip" / "envelope-from" / "helo" /
-                   "problem" / "receiver" / identity /
-                    mechanism / name
+                   "problem" / "receiver" / "identity" /
+                    "mechanism" / name

I think Stuart's analysis of where the error lay is correct.

Scott K

---

From: Alessandro Vesely <[email-masked]>
Date: Mon, 25 Feb 2013 10:15:22 +0100

>> mechanism="mx:example.com"

Fine.  I guess you are going to add something like:

OLD
  mechanism     the mechanism that matched (if no mechanisms matched,
                substitute the word "default")

NEW
  mechanism     the mechanism that matched (if no mechanisms matched,
                substitute the word "default"; otherwise if the match
                involved a <target-name> and a dual-cidr-length, they
                can be reported, e.g. mechanism="mx:example.com/24"

>                     / "helo"     ; for the "HELO" identity

The latter looks a dangling rule to me.  Actually, it is a standalone
rule, recalled later down in Section 9.1.  How about moving it there?

OLD
  identity      the identity that was checked; see the <identity> ABNF
                rule

NEW

Such move does not change the fact that identity has a formal
definition while the other values are described informally:  It just
makes it visible, thereby improving readability.  Collected ABNF in
the appendix remain unaltered.

jm2c

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 25 Feb 2013 08:53:16 -0500

>the appendix remain unaltered.

I hadn't thought such additional changes were needed.  Anyone else? 

Scott K