Subject: Re: [spfbis] #33: Marked Failed Mail Exposure

From: Arthur Thisell <[email-masked]>
Date: Sat, 8 Sep 2012 06:46:52 -0700 (PDT)

> According to the SPFBIS charter, draft-ietf-spfbis-4408bis is intended as a 
> standards track document defining SPF, based on RFC 4408 and as amended 
> above (the charter). There isn't any mention of any SPF drafts. The work is 
> only on RFC 4408 instead of RFC 4408 and SPF drafts. It is not the purpose 
> of this working group to convince or force anyone to change their 
> operations.

This almost scary wrong.   Yes, this WG is using kitterman-4408bis as a base, as the charter calls for.  No, other drafts aren't mentioned in the draft.   But the only thing I can think of worse than not looking at earlier SPF drafts to give insight into how SPF should be updated is not looking at the real-world use of SPF.   If you are going to constrain the WG to looking only at RFC4408, we are going to have serious problems.

Just recently, someone mentioned that Courier-MTA returned "unknown" as an SPF result, which can only be understood by knowning earlier SPF drafts.  RFC4408 was written by people, and people make mistakes.  Some differences between RFC4408 and earlier SPF drafts are deliberate choices and some are mistakes, 

IMHO, when deciding what belongs in RFC4408bis, the most important thing is real-world usage, then RFC4408, then pre-marid SPF documents, then post made to various mailing lists, and last, marid drafts.   The later two are pretty weak sources, but still useful.

OK, that said, the earlier SPF drafts DID NOT have "a deterministic, powerful, near zero false positive method of SPOOF protection."  It DID NOT require rejecting on SPF fail.   There was *always* a debate about what SPF should do with SPF fail, even in 2003, and at best, the earlier drafts had contradictory language.

Now, back to the charter.  During the MARID WG, it was decided to not have any demands in the SPF draft to require SPF verifiers to handle SPF fail in any particular way.   This group's charter says we are not supposed to rehash arguments from MARID.

This discussion needs to be closed, not because insight was brought up from early SPF drafts, but because it brought up issues that were resolved in MARID.

---

From: Hector Santos <[email-masked]>
Date: Sat, 08 Sep 2012 10:35:13 -0400

I strongly believe there is a serious underestimation of how much 
current implementations and deployments since 2003 support the 
original specification "MAY REJECT" with a reject-on-fail 
implementation and deployment despite if is optional or not.  RFC4408 
simply relaxed the semantics to make it non-normative to appease the 
IETF adoption process.  It was already very hard to get any mail 
protocol to reject mail standardized and for good technical reason and 
with zero false positives.  We had CAM-SPAM in the works and these 
things were very important once SORBIG hit the industry and showed how 
nasty the accept/bounce mail problem was.

Anyway, take a look at all the available SPF APIs, open source and 
otherwise and see what they are using for -ALL policies results.  I 
can see this off hand for Libspf2 for the EXIM ACL

http://duncanthrax.net/exiscan-acl/exiscan-acl-spec.txt

  and it appears to me that it does not have a ACCEPT+MARK option at 
all.  I can't say its customize for specific sites, the main point is 
I believe there is a serious underestimation and why ISSUE #33 is 
important to resolve.

EXIM SPF support:

You can now run SPF checks in incoming SMTP by using the "spf"
ACL condition  in either  the MAIL,  RCPT or  DATA ACLs.  When
using it in the RCPT ACL, you can make the checks dependend on
the RCPT  address (or  domain), so  you can  check SPF records
only  for   certain  target   domains.  This   gives  you  the
possibility  to opt-out  certain customers  that do  not want
their mail to be subject to SPF checking.

The spf condition  takes a list  of strings on  its right-hand
side. These strings describe the outcome of the SPF check  for
which the spf condition should succeed. Valid strings are:

   o pass      The SPF check passed, the sending host
               is positively verified by SPF.
   o fail      The SPF check failed, the sending host
               is NOT allowed to send mail for the domain
               in the envelope-from address.
   o softfail  The SPF check failed, but the queried
               domain can't absolutely confirm that this
               is a forgery.
   o none      The queried domain does not publish SPF
               records.
   o neutral   The SPF check returned a "neutral" state.
               This means the queried domain has published
               a SPF record, but wants to allow outside
               servers to send mail under its domain as well.
   o err_perm  This indicates a syntax error in the SPF
               record of the queried domain. This should be
               treated like "none".
   o err_temp  This indicates a temporary error during all
               processing, including exim's SPF processing.
               You may defer messages when this occurs.

You can prefix each string with an exclamation mark to  invert
is meaning,  for example  "!fail" will  match all  results but
"fail".  The  string  list is  evaluated  left-to-right,  in a
short-circuit fashion.  When a  string matches  the outcome of
the SPF check, the condition  succeeds. If none of the  listed
strings matches the  outcome of the  SPF check, the  condition
fails.

Here is a simple example to fail forgery attempts from domains
that publish SPF records:

/* -----------------
deny message = $sender_host_address is not allowed to send mail from 
$sender_address_domain
      spf = fail
--------------------- */

You can also give special treatment to specific domains:

/* -----------------
deny message = AOL sender, but not from AOL-approved relay.
      sender_domains = aol.com
      spf = fail:neutral
--------------------- */

Explanation: AOL  publishes SPF  records, but  is liberal  and
still allows  non-approved relays  to send  mail from aol.com.
This will result in a "neutral" state, while mail from genuine
AOL servers  will result  in "pass".  The example  above takes
this into account and  treats "neutral" like "fail",  but only
for aol.com. Please note that this violates the SPF draft.

When the spf condition has run, it sets up several expansion
variables.

   $spf_header_comment
   This contains a human-readable string describing the outcome
   of the SPF check. You can add it to a custom header or use
   it for logging purposes.

   $spf_received
   This contains a complete SPF-Received: header that can be
   added to the message. Please note that according to the SPF
   draft, this header must be added at the top of the header
   list. Please see section 10 on how you can do this.

   $spf_result
   This contains the outcome of the SPF check in string form,
   one of pass, fail, softfail, none, neutral, err_perm or
   err_temp.

   $spf_smtp_comment
   This contains a string that can be used in a SMTP response
   to the calling party. Useful for "fail".

Arthur Thisell wrote:
> _______________________________________________
> spfbis mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/spfbis
> 
> 

-- 
HLS

---

From: S Moonesamy <[email-masked]>
Date: Sat, 08 Sep 2012 10:20:32 -0700

Hi Arthur,
At 06:46 08-09-2012, Arthur Thisell wrote:
>charter calls for.  No, other drafts aren't mentioned in the 
>draft.   But the only thing I can think of worse than not looking at 
>earlier SPF drafts to give insight into how SPF should be updated is 
>not looking at the real-world use of SPF.   If you are going to 
>constrain the WG to looking only at RFC4408, we are going to have 
>serious problems.

There isn't any constraint to ignore operational input.  Information 
about operational issues are taken into account as that is a good way 
to understand how people actually use a specification.

>Just recently, someone mentioned that Courier-MTA returned "unknown" 
>as an SPF result, which can only be understood by knowning earlier 
>SPF drafts.  RFC4408 was written by people, and people make 
>mistakes.  Some differences between RFC4408 and earlier SPF

Yes.

>IMHO, when deciding what belongs in RFC4408bis, the most important 
>thing is real-world usage, then RFC4408, then pre-marid SPF 
>documents, then post made to various mailing lists, and last, marid 
>drafts.   The later two are pretty weak sources, but still useful.

I would prefer not to get into arguments based on pre-marid SPF 
documents or marid drafts.

>OK, that said, the earlier SPF drafts DID NOT have "a deterministic, 
>powerful, near zero false positive method of SPOOF protection."  It 
>DID NOT require rejecting on SPF fail.   There was *always* a debate 
>about what SPF should do with SPF fail, even in 2003, and at best, 
>the earlier drafts had contradictory language.

It is mentioned in the SPFBIS Charter that:

   * Revisiting past technical arguments where consensus was reached in
     the MARID working group, except where review is reasonably
     warranted based on operational experience.

That is the idea I wanted to convey.

>Now, back to the charter.  During the MARID WG, it was decided to 
>not have any demands in the SPF draft to require SPF verifiers to 
>handle SPF fail in any particular way.   This group's charter says 
>we are not supposed to rehash arguments from MARID.

Yes.

>This discussion needs to be closed, not because insight was brought 
>up from early SPF drafts, but because it brought up issues that were 
>resolved in MARID.

This discussion has been going on for a while.  It has been argued 
that there is a security issue.  If someone raises a security issue, 
it is better not to close it without assessing whether there is an 
actual issue and what is the impact on the protocol.  There is a 
rough summary of the discussion at 
http://www.ietf.org/mail-archive/web/spfbis/current/msg02378.html  up 
to the date of that message.

It would be most welcome if someone suggested a way to resolve the issue.

Regards,
S. Moonesamy (speaking for myself)

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Sat, 8 Sep 2012 23:08:36 -0700

> This almost scary wrong.   Yes, this WG is using kitterman-4408bis as a base, as the charter calls for.  No, other drafts aren't mentioned in the draft.   But the only thing I can think of worse th…

I think the use of pre-RFC drafts of SPF to make claims of the form
"SPF was always about..." also fails to accept current realities.  SPF
evolved beyond some of those earlier ideas, and unless we can show
such evolution was harmful, we shouldn't hold up older drafts as
evidence that the earlier ideas were right.

> OK, that said, the earlier SPF drafts DID NOT have "a deterministic, powerful, near zero false positive method of SPOOF protection."  It DID NOT require rejecting on SPF fail.   There was *always*…

That's what I'm talking about, primarily.

> This discussion needs to be closed, not because insight was brought up from early SPF drafts, but because it brought up issues that were resolved in MARID.

+1

-MSK

---

From: Hector Santos <[email-masked]>
Date: Sun, 09 Sep 2012 04:15:12 -0400

Murray S. Kucherawy wrote:
> "SPF was always about..." also fails to accept current realities.

In fact, if read carefully, the thread is exactly showing how the 
current realities perhaps makes issue #33 more relevant.

< SPF evolved beyond some of those earlier ideas, and unless we can show
> evidence that the earlier ideas were right.

The semantics did not change - only relaxed RFC2119 wise.

    MAY REJECT -->   "can mark or reject outright"

RFC4408 was by design 100% (or nearly) compatible with the early 
adopter establishment - it was a prerequisite.  In any case, this 
thread has turned to something that was never meant to be - not 
surprising, but lets try not to add more to it than issue #33 states 
and proposes as a non-normative security precautionary note which I 
believe does match current realities. No mandate of what you need to 
do no - just be aware of.

As you eluded to, there was perhaps no harm when updating to RFC4408 
was done. It was mostly fine tuning (off the top of my head - the 
limits added). The old spec reference was made only to show perhaps 
the maturity of early adopters most likely did not change from the 
standpoint of how the relevant issue of how a SPF -ALL policy is 
modeled by various implementations.

I also suggested that maybe looking at all the published APIs and open 
source to see what is the OUT OF THE BOX default logic.   I am 
speculating that if indeed a good many followed RFC4408 which is 
semantically UNCHANGED from early specs with an "MAY REJECT" and no 
definition toward MARKING, then maybe any new emphasis for MARKING may 
be be a barrier for some (due to what is described in issue #33).

I already stated we can't support this option without a critical 
change to our code.  I could be the only system out there with a 
REJECT-ON-FAIL behavior. I doubt it though.

-- 
HLS

---

From: S Moonesamy <[email-masked]>
Date: Sun, 09 Sep 2012 12:09:45 -0700

>    MAY REJECT -->   "can mark or reject outright"

There is the following sentence in RFC 4408:

   "The checking software can choose to mark the mail based on this
    or to reject the mail outright."

It does not contain any RFC 2119 key word.

>REJECT-ON-FAIL behavior. I doubt it though.

There isn't any normative text in Section 2.5.4 of RFC 4408 which 
prohibits the rejection of messages.

Regards,

---

From: Barry Leiba <[email-masked]>
Date: Sun, 9 Sep 2012 17:18:35 -0400

> It does not contain any RFC 2119 key word.

Nevertheless, what often causes confusion with sentences like that is
that it's not always clear whether there are other alternatives to the
ones mentioned in the sentence.  In other words, which of the
following does the quoted sentence mean?:

1. The checking software can choose between the following two: (a)
mark the mail based on this or (b) reject the mail outright.

2. The checking software can choose (a) to mark the mail based on
this, (b) to reject the mail outright, or (c) to do something else
that we're not specifying here.

I've seen many comments related to that sort of confusion in IESG
evaluation of various documents, and it's usually a good idea to
re-word the sentence to clarify what's meant, so no one has to guess.
I believe that in this case, the intent was (2), but that some people
quite understandably read it as (1).

Barry

---

From: S Moonesamy <[email-masked]>
Date: Sun, 09 Sep 2012 14:52:45 -0700

Hi Barry,
>that we're not specifying here.

The sentence could be clarified as it can be read as (1).  I read it 
as (2) is not ruled out.  Another possible issue is the meaning of "mark".

Regards,
S. Moonesamy

---

From: Commerco WebMaster <[email-masked]>
Date: Sun, 09 Sep 2012 16:11:34 -0600

SM,

On 9/9/2012 3:52 PM, S Moonesamy wrote:
> The sentence could be clarified as it can be read as (1). I read it as
> (2) is not ruled out. Another possible issue is the meaning of "mark".
>
> Regards,
> S. Moonesamy

How about this:

1. The checking software can choose between the following two: (a)
reject the mail outright or (b) process the mail based on local policy.

When you think about it, b is exactly what an SPF unaware MTA is really 
doing.

Best,

Alan M.

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Sun, 9 Sep 2012 15:45:14 -0700

> When you think about it, b is exactly what an SPF unaware MTA is really

(I presume we're talking about something to go in Section 1 or so, or
maybe the top of 2.5, to set context.)

Note that rejection can also be the result of local policy.  I'd
prefer something like:

The result of the check_host() function (see Section 4) is typically
used as input to an operator's final handling decision for the message
under evaluation.  For sites that choose to be very strict, this can
involve SMTP rejection of a message for which check_host() returns a
"fail" result.  An operator might also use the result as a factor in a
more complex handling decision process before choosing to accept or
reject the message.  More commonly, operators tend simply to "mark"
the message by affixing a header field (see Section 7) that relays
details of the SPF evaluation to other handling agents such as MUAs
and downstream filters, and then accept it.  Due to the variance of
needs from one operator to the next, this document establishes no
specific handling requirement.

-MSK

---

From: S Moonesamy <[email-masked]>
Date: Sun, 09 Sep 2012 16:14:43 -0700

>reject the mail outright or (b) process the mail based on local policy.

The suggested text is a step forward.  The problem with the above 
text is that "reject" may be interpreted as not part of local 
policy.  An alternative would be reword the text as an example.

>When you think about it, b is exactly what an SPF unaware MTA is really doing.

Yes.

Another way to look at the issue is to see what implementations are 
doing and take it from there.

>maybe the top of 2.5, to set context.)

It's about Section 2.5.4.  That doesn't mean that the text has to go 
in that part of the draft.

>specific handling requirement.

I would drop "very strict" because it is debatable.  The idea of 
having "mark" as adding a header field is interesting.  There is the 
following text in Section 4.2 of RFC 4408:

   "Based on the result, the action to be taken is determined by
    the local policies of the receiver."

The above text, or a variation of it could be fitted in there.  One 
issue is that Section 2.5 is titled "interpreting the 
results".  Here's what Section 2.5.4 says about SPF "fail":

   'A "Fail" result is an explicit statement that the client is not
    authorized to use the domain in the given identity.'

There seems to be agreement about the text.  It is the tie-in to 
specific actions which may be the issue.

At 15:34 09-09-2012, Arthur Thisell wrote:
>                                                 MTAs, MDAs, and MUAs may
>    choose to accept, classify, discard, or reject messages based on the
>    result of an SPF test.

    For example, an action such as accepting, classifying, 
discarding, or rejecting
    a message may be taken based on the result of a SPF evaluation.

Regards,
S. Moonesamy

---

From: Hector Santos <[email-masked]>
Date: Sun, 09 Sep 2012 20:14:14 -0400

Although I have stated basically the same you have here in this or 
adjunct thread, where the open-ended statement can basically mean 
anything,  this is not all what my real concern is.  Obviously, local 
policy rules here like its always is else where, but we are talking 
about protocol logic, and whether its 1, 2 or 3 options:

   1) Reject
   2) Mark (presumption: Add Received-SPF and now A-R in 4408BIS)
   3) Other undefined

This section 2.5.4, in my technical opinion MUST BE applied with what 
4408 section 2.3 states about the intent of -ALL (SPF fail) policy for 
domains:

    2.3.  Publishing Authorization

    ..

    If domain owners choose to publish SPF records, it is RECOMMENDED
    that they end in "-all", or redirect to other records that do, so
    that a definitive determination of authorization can be made.

    Domain holders may publish SPF records that explicitly authorize no
    hosts if mail should never originate using that domain.

    ..

and if you look at 4408BIS version, it is revised in my view with even 
stronger semantics regarding negative/positive determinations or as I 
may label it as mail classification lingo:

    SPF results can be used to make both positive (source is authorized)
    and negative (source is not authorized) determinations.  If domain
    owners choose to publish SPF records and want to support receivers
    making negative authorization determinations, then they MUST publish
    records that end in "-all", or redirect to other records that do,
    otherwise, no definitive determination of authorization can be made.
    Potential issues and mitigations associated with negative
    determinations are discussed in Section 9.

The irony as I stated  is that in the name of not trying to have any 
compliancy language with the idea of "deterministic rejection" ideas, 
with 4408bis, it actually added more strength in 2.3.

I don't care what mode is chosen - its about the protocol logic that 
can be extracted from the functional specification into a technical 
specification. The logic added by an developer to get the local 
operators the local policy options.

Regardless of what option is selected, IMTO, each "SHOULD" satisfy the 
intent of 2.3, and if not, then we have issue #33.

Finally, my reference to MARID and pre-MARID with early adopters is 
the new emphasis in 4408BIS.  If there is a single "big new feature" 
for SPF developers, it is the idea of adding A-R support and in 
addition, speaking for myself, realizing in this WG that there are 
some ESPs that do an ACCEPT+MARK and follow 100% the security 
precautions in issue #33.  We want to offer the same optional feature 
offering in 2.5.4. of using ACCEPT+MARK instead of just REJECT.  For 
us to do this, it will be a major change to make sure our GATEWAY can 
now move "Received-SPF: fail" or "Authentication-Result: spf=fail" 
marked messages into spam folders.  Its not doing that now and if we 
don't we have a SPF security leak for the -ALL domains that are no 
longer rejected.

Thanks

-- 
HLS

---

From: "John Levine" <[email-masked]>
Date: 10 Sep 2012 00:47:49 -0000

>reject the mail outright or (b) process the mail based on local policy.

The checking software can process the mail based on local policy.

If it rejects, that's local policy, too.  I really think that we need
to prune out the advice that implies that recipients are required to
do anything specific with SPF results.

R's,
John

---

From: S Moonesamy <[email-masked]>
Date: Tue, 11 Sep 2012 00:39:28 -0700

Hello,

I haven't discussed this issue with Andrew yet.  I don't see 
agreement on Issue #33 ( 
http://www.ietf.org/mail-archive/web/spfbis/current/msg02195.html 
).   Murray Kucherawy suggested some text where it is argued that the 
attack on SPF is outside the scope of the specification ( 
http://www.ietf.org/mail-archive/web/spfbis/current/msg02661.html 
).  I don't see agreement on the text which Murray proposed.  During 
the discussion, Barry Leiba raised an interesting point about a 
sentence in Section 2.5.4 ( 
http://www.ietf.org/mail-archive/web/spfbis/current/msg02708.html 
).  I reopened Issue #29 based on that ( 
http://trac.tools.ietf.org/wg/spfbis/trac/ticket/29#comment:5 ).

There has been a lot of discussion about rejecting or "marking" a 
message based on a SPF result of "fail".  Section 1 of RFC 4408 
mentions the following:

   "An additional benefit to mail receivers is that after the use of an
    identity is verified, local policy decisions about the mail can be
    made based on the sender's domain, rather than the host's IP address.
    This is advantageous because reputation of domain names is likely to
    be more accurate than reputation of host IP addresses.  Furthermore,
    if a claimed identity fails verification, local policy can take
    stronger action against such E-Mail, such as rejecting it."

Section 4.2 of RFC 4408 mentions that:

   "Based on the result, the action to be taken is determined by the
    local policies of the receiver."

I opened Issue #58 to find out whether there is agreement or not 
about whether a message rejection or any other action following a SPF 
evaluation is a matter of local policy.  My opinion is that an action 
based on the result of a SPF evaluation is up to the local policy of 
the receiver.  The receiver may take stronger action such as 
rejecting a message.

My recommendation is not to add any text about Issue #33 to the draft 
unless there is strong agreement on the mailing list for the proposed 
change.  I also recommend reverting the following text in Section 2.3 
of draft-ietf-spfbis-4408bis-07:

   "SPF results can be used to make both positive (source is authorized)
    and negative (source is not authorized) determinations.  If domain
    owners choose to publish SPF records and want to support receivers
    making negative authorization determinations, then they MUST publish
    records that end in "-all", or redirect to other records that do,
    otherwise, no definitive determination of authorization can be made.
    Potential issues and mitigations associated with negative
    determinations are discussed in Section 9."

to:

   "If domain owners choose to publish SPF records, it is RECOMMENDED
    that they end in "-all", or redirect to other records that do, so
    that a definitive determination of authorization can be made."

and not including any change in Section 2.3 unless there is strong 
agreement on the mailing list for the proposed change.

I will close Issue #33 in the tracker unless there is an explanation 
about whether the attack (see issue #33) is out of scope and what 
countermeasures can be applied to defend against it.  I suggest 
basing the explanation on text which is in RFC 4408 only as it is my 
understanding that existing SPF implementations use that specification.

Regards,
S. Moonesamy
SPF WG co-chair

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 11 Sep 2012 11:41:32 -0700

On Tue, Sep 11, 2012 at 12:39 AM, S Moonesamy <[email-masked]> wrote:
> I haven't discussed this issue with Andrew yet.  I don't see agreement on
> Issue #33 (
> http://www.ietf.org/mail-archive/web/spfbis/current/msg02195.html ).
> Murray Kucherawy suggested some text where it is argued that the attack on
> SPF is outside the scope of the specification (
> http://www.ietf.org/mail-archive/web/spfbis/current/msg02661.html ).  I
> don't see agreement on the text which Murray proposed.  During the
> discussion, Barry Leiba raised an interesting point about a sentence in
> Section 2.5.4 (
> http://www.ietf.org/mail-archive/web/spfbis/current/msg02708.html ).  I
> reopened Issue #29 based on that (
>[...]

Someday when we have searchable archives I will be able to provide
links like you do, but I'm apparently less patient than you are at
finding them with the current system.

Some time ago Scott pointed out that the RFC that lays out guidance
for Security Considerations sections specifically calls out message
injection attacks as something that needs to be documented.  The
description it uses seemed to point almost directly at this issue.  I
think that compels us to say something about it, and not close it
unaddressed.

I'd like to hear more comments about the text I proposed, which I
recall at least Scott liked.

-MSK

---

From: S Moonesamy <[email-masked]>
Date: Tue, 11 Sep 2012 13:46:13 -0700

Hi Murray,
>finding them with the current system.

It's a hit and miss affair with search engines.  I read the messages 
locally to identify what to look for.  It can be quite tedious.

>think that compels us to say something about it, and not close it

I would not describe it as a message injection attack as that line of 
argument may create more issues than it solves.  I wrote 
draft-moonesamy-mail-security-00 (expired) to document some arguments 
about mail content.  It was in response to a review where the 
reviewer asked the following question:

   "Presumably you mean to say that binary attachments that are ideal for
    delivering malware are supported irrespective of the use of this feature,
    right?"

That is somewhat along the lines of what your text argues about.  I 
would keep it in mind for the Security Considerations section.

Regards,
S. Moonesamy

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 11 Sep 2012 15:02:49 -0700

On Tue, Sep 11, 2012 at 1:46 PM, S Moonesamy <[email-masked]> wrote:
> That is somewhat along the lines of what your text argues about.  I would
> keep it in mind for the Security Considerations section.

I think the proposed text takes this into account.  It specifically
calls out that there are cases where the feature can admit bad stuff,
or block good stuff.  One's mileage may vary, caveat emptor, etc.
That's basically it.

-MSK

---

From: Scott Kitterman <[email-masked]>
Date: Tue, 11 Sep 2012 18:03:58 -0400

> recall at least Scott liked.

It's my recollection that I did as well.

Scott K

---

From: Scott Kitterman <[email-masked]>
Date: Tue, 11 Sep 2012 18:12:41 -0400

> understanding that existing SPF implementations use that specification.

I don't understand why you want to revert the Section 2.3 changes?  

Those changes were not primarily made in response to issue #33.  They were 
made because of comments that in RFC 4408 the specification RECOMMENDS 
something without a rationale for the recommendation.  Given the lack of 
rationale, it's hard to know what might be an exceptional case for not 
following the recommendation.  

The change is also made because of operational experience gained since RFC 
4408 was published.  At the time RFC 4408 was published, it was envisioned 
that there would be a transition to a paradigm where most organizations 
published -all SPF records.  This didn't happen.  The revised text gives some 
guidance on the implications of not following the recommendation.  Since it is 
common for large entities to ignore it, I think that is important.  That's 
more true now than I would have envisioned when RFC 4408 was being developed.

Scott K

---

From: S Moonesamy <[email-masked]>
Date: Tue, 11 Sep 2012 16:12:10 -0700

Hi Scott,
>following the recommendation.

If there isn't a rationale for a (RFC 2119) RECOMMENDS, it is better 
not to replace that with a (RFC 2119) MUST unless the change has 
strong agreement.  I don't see that in the mailing list discussions.

Even though those changes were not made primarily in response to 
Issue #33, it has an impact on the issue ( 
http://www.ietf.org/mail-archive/web/spfbis/current/msg02715.html 
).  That makes it very difficult to figure out a path forward to 
resolve issues.

I'll open a new issue based on the message you posted about the 
change to Section 2.3.

Regards,
S. Moonesamy
SPFBIS WG co-chair

---

From: Scott Kitterman <[email-masked]>
Date: Tue, 11 Sep 2012 22:56:22 -0400

> I'll open a new issue based on the message you posted about the

I think it's unrelated.

This is a change in description, not an increase in requirements.

RFC 4408 says a -all records are RECOMMENDED

4408bis currently says if senders ... "want to support receivers making 
negative authorization determinations, then they MUST publish records that end 
in -all".

The statement in 4408bis has always been true even if it was not explicitly 
stated that way.

4408 assumed that senders would generally want to support "making negative 
authorization determination" and based on that assumed general goal made a 
general recommendation.  In the intervening time we've learned that there are 
quite a few senders that aren't particularly worried about this and so now 
I've proposed it say "If you want X then you MUST Y" instead of "Y is 
RECOMMENDED".

There is no new requirement here and the proposed language better fits the 
current operational environment than reverting to what was in RFC 4408.

Scott K

---

From: S Moonesamy <[email-masked]>
Date: Tue, 11 Sep 2012 20:45:55 -0700

Hi Scott,
>I think it's unrelated.

The change you proposed is being tracked as Issue #60.

>current operational environment than reverting to what was in RFC 4408.

 From RFC 2119:

  'MUST   This word, or the terms "REQUIRED" or "SHALL", mean that the
   definition is an absolute requirement of the specification.

   SHOULD   This word, or the adjective "RECOMMENDED", mean that there
   may exist valid reasons in particular circumstances to ignore a
   particular item, but the full implications must be understood and
   carefully weighed before choosing a different course.'

My reading of that RFC is that "MUST" and "RECOMMENDED" are not the 
same.  I do not see strong agreement for the text change.

Regards,
S. Moonesamy
SPFBIS WG co-chair

---

From: Andrew Sullivan <[email-masked]>
Date: Wed, 12 Sep 2012 00:00:21 -0400

No hat.

On Tue, Sep 11, 2012 at 08:45:55PM -0700, S Moonesamy wrote:

> [Scott said]:

It seems that these two mails talk past each other.  Scott's point, as
I read him, is that the 4408 text just says "you probably want to use
these records", and the new proposed text says "if you want to achieve
something, then you MUST do some other thing".

Now, there are two questions here, both relevant to the WG.

First, is this really a clarification?  That is, does the new proposed
text expans on or otherwise clarify what the old text said implicitly,
as Scott argues it does?

Second, is the new text both true and a correct use of the RFC 2119
keyword?  It seems to me that in the new proposed text, it isn't a
2119 MUST at all.  It's really just stating an entailment.  It could
be as correctly phrased as "if senders ... 'want to support receivers
making negative authorization determinations, it is necessary for them
to publish records that end in -all'".  If I understand correctly,
this is a fact about how the protocol works, and not actually a
normative requirement for interoperation.  What have I missed?

Best,

A

-- 
Andrew Sullivan
[email-masked]

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 11 Sep 2012 23:06:02 -0700

> normative requirement for interoperation.  What have I missed?

My understanding is that your proposed wording here is more
appropriate for a protocol document (a "technical specification" in
RFC2026 terms), but the MUST might be appropriate in an applicability
statement.  Since RFC4408bis is the former and not the latter, I'm
inclined to go with the non-2119 version.

I don't know if it's possible to have one document be both of those
things at the same time, but I don't think I've ever seen one.

For those unfamiliar: An applicability statement is a proposed
standard document that explains how to use a protocol to achieve a
particular goal that wasn't described in the protocol itself, such as
a novel application or innovation.  RFC2119 language is use a bit
differently there, to mean "To achieve your goal, you have to take
these steps to make the protocol afford you this additional
capability."  A recent example is RFC6647, which is an applicability
statement about how to apply SMTP to achieve what's commonly known as
"greylisting".

-MSK

---

From: Hector Santos <[email-masked]>
Date: Wed, 12 Sep 2012 12:30:39 -0400

> normative requirement for interoperation.  What have I missed?

Nothing if I read you right.

This is all very easy.  In MARID, there were four productions:

     rfc4405 - SMTP Extension SUBMITTER protocol (SPF Extension?)
     rfc4406 - Sender ID: Authenticating E-Mail (SPF Extension?)
     rfc4407 - PRA Algorithm for RFC4406 (SPF Extension?)
     rfc4408 - SPF v1.0 "Base Protocol"

Prior to MARID, we had LMAP related based protocols:

    http://tools.ietf.org/html/draft-irtf-asrg-lmap-discussion-01

which RMX, DMP, CEP and eventually SPF and SENDERID were all pattens 
of.  LMAP tried to be a baseline protocol for who these DOMAIN::IP 
assertion based protocols.  Keep in that LMAP was not a protocol but a 
Philosophy and looking at section 1.2:

    1.3. Interpretation of LMAP Data

    Recipient MTAs are free to interpret LMAP data as they wish.
    Possibilities range from rejecting email with a 550 error code to
    using LMAP data as one input to a multi-criterion filter.  Domains
    may also optionally use LMAP data to whitelist or give higher passing
    values for email in their filters.

NOTE, it starts with the OBVIOUS of all things mail related - you are 
in the control, the local operator/policy decides, but what is 
includes as the possible design options is the #1 idea with the 
highest protection value; REJECTION.

    E-mail from LMAP domains that do not publish LMAP data should NOT be
    rejected since there is no requirement that domains do so, and many
    will not, either for policy reasons or from lack of resources.  E-
    mail from non-LMAP domains should be treated as e-mail is treated
    currently.

    All local policy decisions remain with the recipient's MTA.  Readers
    are reminded that recipients have the right to refuse any
    communication from anyone, for any reason.

So as you see, this is all NATURAL.   Whats going on here is we are 
using this idea of LOCAL POLICY to mandate that its actually a BAD 
IDEA to rejection.

Now. Is that wrong?

No, but from from a software developer standpoint, he need to decide 
what are the switches, knobs, and default values for operators to use 
and do so with the ability of adhering with their company mail 
policies and services.

My point we should just spell out all the options and describe 
accepting fail mail can have some [some] consequences the backend 
needs to make sure its part of SPF integration and usage.   I 
personally see this as an ethical engineering issue because not to do 
so, needs to be explained in terms of what benefits are gained by not 
separating mail over the harm that is possible.  If is not done, then 
IMO, its poor protocol engineering (even mal-practice in the sense 
that we knew of the problem, but decided not to do anything about it).

It just seems to be that we have a battle of two camps that really as 
preexisted since MARID (SPF, SENDERID), DKEYS, CVS and DKIM,

   - Those that do not like SMTP level rejection ideas, in principal,
     and lean towards the "Let User Decide" philosophy. I have seen this
     more with those who are in DMA, ESP, Bulk Mailer software market.

   - Those who believe strong SMTP compliancy, transaction (online)
     level rejections when possible IFF (if and only if) there is a
     very high degree of confidence, zero-false positive and a DOMAIN
     publicized policy of authorization using protocols 100% designed
     to address the email SPOOFING problem.

Remember, CAN-SPAM, my overall summarize of this LAW:

     It gives SPAMMERS the "god given" natural law capitalistic right
     to  do business with user-vendor contracts as long as they follow
     all the EMAIL STANDARD rules properly, and you read CAN-SPAM, it
     indicated that it left that responsibility to the IETF to define
     what at the email standards for the spammers to follow and be
     compliant

We distribute this stock/optional SMTP 220 welcome response:

220-************** WARNING:  FOR AUTHORIZED USE ONLY! 
**********************
220-* THIS SYSTEM DO NOT AUTHORIZE THE USE OF ITS PROPRIETARY 
COMPUTERS    *
220-* AND COMPUTER NETWORKS TO ACCEPT, TRANSMIT, OR DISTRIBUTE 
UNSOLICITED *
220-* BULK E-MAIL SENT FROM THE INTERNET. THIS SYSTEM WILL RESTRICT 
ACCESS *
220-* TO CAN-SPAM (US S. 877) COMPLIANT CLIENTS ONLY. 
      *
220 
************************************************************************

So as long as the anonymous sender is:

   - 100% Compliant with SMTP
      - including CBV (Callback Verification for 5321.MailFrom)
   - Is not restricted per SPF -ALL policies

My strong long time mail system engineering opinion (covers 3 decades 
over several mail networks) is that a TRANSPORT software has no 
business in making decisions for local operation as it to relates to 
contain (the payload).  But at the network level, the MTA IMO has some 
strong client/server protocol negotiations they can follow in the 
state machine.   We tried add more rule to the above:

   - Is not restricted per DKIM+ADSP DISCARD policies

but we screwed up the purpose and confidence in using DKIM with a 
strong domain policy security wrapper such as ADSP=DISCARD that we 
were left with just recording it with A-R and this my friends, is that 
I am overall worry about can happen with SPF. This series of recent 
threads if purely SPF theory that was long settled and my opinion, the 
management of the WG need to begin to steer the group in the right 
direction and remain focus.  I must admit I am baffled by what goals 
are being sort.

I accept the idea that accepting+marking is a viable idea for SPF - 
but only in my engineering opinion that it is separated from the user 
in some implementation own way, i.e. issue #33.

I believe if we can update 2.5.4 to describe what "mark" meant, then I 
believe we could then avoid highlighting it as a security issue (issue 
#33), making it appear better for IESG review.  But maybe the cat is 
out of the bag.

Perhaps can state the current practice this include accept+mark PLUS 
separate among those implementations that already support accept+mark. 
  Otherwise, the other current practice is to reject outright.

--
HLS

---

From: Scott Kitterman <[email-masked]>
Date: Wed, 12 Sep 2012 13:07:23 -0400

> statement about how to apply SMTP to achieve what's commonly known as

I confess to some confusion about how these different types of documents are 
supposed to work, so I probably got it wrong.

If I understand your message correctly, a protocol document should only use 
2119 words for protocol elements.  An applicability statement should use 2119 
words about how to use a technology to achieve a specific goal for the cases of 
things that are not described in the protocol document.

>From that, I draw the conclusion that one can say "If you want X you MUST Y 
using Z protocol" in a that is NOT the document that defines Z, but you can't 
use the same MUST in the document that defines protocol Z?

That may be correct, but it seems completely counter intuitive.

Is that correct?

Scott K

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 12 Sep 2012 10:38:31 -0700

> That may be correct, but it seems completely counter intuitive.

The short answer is "yes".  See: http://tools.ietf.org/html/rfc2026#section-3.2

However, RFC2026 doesn't explain the appropriate use of RFC2119
language in an AS (obviously; note the order of the numbers).  I base
that part of what I said on precedent (e.g., RFC6647).  Anyone with
more grey in the beard than me want to clarify?

-MSK

---

From: Hector Santos <[email-masked]>
Date: Wed, 12 Sep 2012 13:44:55 -0400

> things that are not described in the protocol document.

Keep in my that RFC6647 is not a document that is agreed upon by all, 
only by those that happen to be part of the review/WG.  Thats fine, 
but there are many issues with it and in my view, missed out on some 
real current practices.  So I don't think its a good idea to begin 
judging this document with others that do not represent wide spread of 
engineering writing experiences and technical writing styles.

> That may be correct, but it seems completely counter intuitive.

Scott, you have functional specs (what we want) and the technical 
specs (how its done). The beauty of the RFC format (and sometimes its 
curse), is it is often a MERGE of the two disciplines.  Some can do it 
better than others, but it takes real experience of BOTH to do cater 
properly to all the proper audiences.

If people what to argue "2119" language, that is only to be able to 
throw the book on people, and yes, does help the designer with "IF TO 
ADD, NOT ADD" decisions to make.   But lets not play down the 
intelligence of people:

     CAN MARK OR REJECT

is pretty clear to ANY software developer as far as what an API can 
offer and also pretty to the operators to run.

It could of been written:

     SOFTWARE MUST provide two options:

       - REJECT
       - ACCEPT + MARK

Instead, the above (the way 4408 has it) allows it be read:

     SOFTWARE should provide one of two options or both:

       - REJECT
       - ACCEPT + MARK

We naturally, without thinking much about, only did a REJECT option 
because that was the fundamental idea that the entire SPF era along 
with SUBMITTER, PRA, SENDERID was dealing with (for the -ALL policy).

Without a doubt, for an ESP that was service free email accounts, like 
GMAIL, YAHOO and others, of course, they has user-based concepts and 
foremost they did great jobs in presenting frontends with SPAM folders.

The question is for the ACCEPT+MARK implementators:

     Did a SPF -ALL fail become one of the RULES to move the failed
     message to the ESP SPAM folder?

According to my gmail testing, using a facebookmail.com spoof 
transaction, it did move the mail into my gmail account spam folder.

The question for SPFBIS is really two options:

   1) Do you present the ACCEPT+MARK PLUS SEPARATION as a
      recommended insight in section 2.5.4  (ISSUE #29)

   2) Do you say nothing in 2.5.4 and present as a
      precautionary security note (ISSUE #33).

It has nothing to with with LOCAL POLICY.  It is the SPF specification 
that MUST cover the entire scope, and when it comes to security 
related things, it is a no brainer.   You don't need 5321 to tell you 
about it or any other document - security is security.  Either we see 
it or we don't.

-- 
HLS