Subject: [dmarc-ietf] Lenient SPF

From: Rick van Rein <[email-masked]>
Date: Sun, 08 Oct 2017 23:51:37 +0200

Hello,

This is a solution for SPF after forwarding and lists that I've been
aware of for a while; I wonder if this is already commonly done?

Forwarding is an action on the receiving end, and can only be solved
reliably by the recipient.  Notably, a mailbox user could specify
addresses that are forwarded to their mailbox.  Mailing list
subscriptions may be seen as a special form of forwarding.

When incoming mail is SPF-validated and the envelope sender does not
match the sending MTA address, those forwarded domains can be looked up
for the envelope _recipient_ address.  Lenient SPF validation would add
permission to sending MTAs from these forwarded domains.  The result
would be that forwarding never breaks, when configured on the receiving
mailbox.

I am not sure if this is part of the "mark as NOT spam" procedures in
common mailbox services, but AFAIK it would fix quite a bundle of problems.

I suppose this could also lead to a definition of Lenient DMARC.

Cheers,
 -Rick

---

From: "John Levine" <[email-masked]>
Date: 9 Oct 2017 02:29:01 -0000

>subscriptions may be seen as a special form of forwarding.

Sorry but this is another WKBI.  Off the top of my head some of the
reasons it doesn't work are:

* There is no way to tell who forwarded a message, in particular you
cannot expect the recipient to be in a To: or Cc: header.  You could
invent a Forwarded-By: header, but of course bad guys add fake
headers, so you'd have to track who's supposed to be forwarding what
and you'd end up with a very complex and fragile forwarder whitelist.

* There is a kludge called SRS that embeds the original bounce address
in the forwarder's bounce address, but it doesn't help.

* Mailing lists invariably replace the incoming bounce address with
their own bounce address so they can handle the bounces, which among
other things means there you can't tell that a mailing list message is
a mutated forwarded message from whoever originally sent it.  Skipping
ahead, you might at best end up with a very complex and fragile
mailing list whitelist.

* Asking users to manage their own security settings doesn't work.
See prior discussion of MUA highlighting and browser warnings.

R's,
John

PS: Most courtesy forwards don't break DKIM signatures.  That's not an
accident.

---

From: Rick van Rein <[email-masked]>
Date: Mon, 09 Oct 2017 19:49:23 +0200

Hi,

> reasons it doesn't work are:

Thank you for your comments, John!

> and you'd end up with a very complex and fragile forwarder whitelist.

The Received: header actually does most of this -- it contains the
EHLO and MAIL FROM information of previous envelopes.  It does not
always contain the RCPT TO (max one ends up in optional "for").

Given that SPF approves of a sender [because it is configured as a
reliable provider], one might trust the topmost Received: header to
have been delivered by that reliable party.  That assures us of the
sender one step down, and after a few steps we've left the entrance
points to our own realm [over which we have control, so not all bad]
and then... the customary SPF magic can take place, based on the
envelope data available there.  The last IP address logged comes from
the last trusted Received: header and that's the one that did the
crossover to our realm of control.

The devil will be in the details, I realise that.  But a lot of the
info you mentioned actually is there already, and is even obliged.
Things like one domain leaving multiple Received: headers, and
complex MTA infra that doesn't share input and output addresses.

> in the forwarder's bounce address, but it doesn't help.

I agree.  And unpacking the address from SRS doesn't sound like a
dream of interoperability either.

One might actually rely on the From: header more than SPF does now;
unlike To: and Cc: this is a reliable address source [except for
bounces, which will normally transport directly and SPF-compliant].
The From: address can then be used at the end of the unraveled
envelope history from Received: headers.  Whee, it almost sounds
like DMARC is teaching SPF how it's done :)

> ahead, you might at best end up with a very complex and fragile

Same as SRS, really.

> See prior discussion of MUA highlighting and browser warnings.

Maybe.  They do manage their spam folders, and this may piggyback on that.

> PS: Most courtesy forwards don't break DKIM signatures.  That's not an

I know; I am not focussing on these "most" cases, but on the rest.
Like this list :-) that requires DKIM re-signing.

Thanks for your thoughtful thoughts!

-Rick

---

From: "John R Levine" <[email-masked]>
Date: 9 Oct 2017 14:22:16 -0400

> The devil will be in the details, I realise that.

Really, this is not a new idea.  It doesn't work.  Please go back and read 
the archives from a decade ago when all of these ideas were considered and 
rejected.

R's,
John

> Maybe.  They do manage their spam folders, and this may piggyback on that.

Very, very badly.  People at large webmail providers tell me that users 
retrieve phishes out of their spam folders all the time and complain when 
it doesn't let them click on the links.

---

From: Dave Crocker <[email-masked]>
Date: Mon, 9 Oct 2017 11:25:53 -0700

On 10/9/2017 11:22 AM, John R Levine wrote:
> Really, this is not a new idea.  It doesn't work.  Please go back and 
> read the archives from a decade ago when all of these ideas were 
> considered and rejected.

If we want to give constructive guidance to folk who offer ideas that 
have previously been rejected, we need to document them and the reasons 
for rejecting them, rather than give this kind of generic, vague pointer 
to the past.

I'm glad to help, but don't feel I have enough of the detail to be 
primary for the effort.

d/
-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: Brandon Long <[email-masked]>
Date: Tue, 10 Oct 2017 13:53:25 -0700

> complex MTA infra that doesn't share input and output addresses.

So, to a certain extent, sure.  Or, you can just do the SPF evaluation on
the edge, and pass that evaluation into your ADMD via the
Authentication-Results header.  Or, use some sort of specific local
equivalent.

GSuite does allow customers to specify the IP addresses of their inbound
gateways, so that when the message finally reaches GSuite, we'll walk the
received headers for the first one that specifies an IP address not on
their list and evaluate SPF based on that.  This works ok for most use
cases, though we've recently had some issues because the list of IPs in an
ADMD can be fairly large (O365 is moving to using non-private IPv6
addresses for internal servers, which can make maintaining the list
challenging).

Note that ARC is the end result of thinking of this, that we can create a
signed Received header chain containing the info we need to pass an
observed spf=pass at hop 1 and pass that information on to follow on's
across trust/ADMD boundaries and know who the actors are so we can trust
the information passed.

Brandon

---

From: Hector Santos <[email-masked]>
Date: Wed, 11 Oct 2017 10:11:48 -0400

> I suppose this could also lead to a definition of Lenient DMARC.

Hi Rick,

A few highly opinionated points:

Keep in mind that SPF is a PAYLOAD (DATA smtp state) technology. In 
other words, it is generally processed, as it was originally intended, 
before the DATA smtp state.  Therefore, there are limited process 
parameters available pre-DATA:

   CIP  Connection/Client IP address
   CDN  Client Domain Name (HELO/EHLO)
   RP   Return Path (MAIL FROM)
   FP   Forwarding Path (RCPT)

So SPF is basically a function of the CIP, CDN and/or RP and generally 
not the FP, however some implementations, lioe ours, do not process 
SPF until FP is first acceptable. i.e. no need to perform the SPF 
Overhead if the user does not exist or the user or domain is not 
locally hosted.

Hence there is no PAYLOAD data to be used in an SPF method unless the 
implementation is designed to wait until the DATA is transferred to 
begin an SPF.   This was a latter design, in particular when Payload 
technologies were proposed, like ADSP and "Super ADSP" (DMARC).  DMARC 
combined SPF, so to support DMARC, your implementation would delay SPF 
processing until DATA is transferred.   But a key point is that no one 
can dictate how/when SPF should be employed during an SMTP session. 
High scale optimization is still important.

Since 2003, my statistics showed very early on that 83% of the time, 
it would be a high overhead waste because if SPF was to fail at DATA, 
it would of also failed at PRE-DATA.

This is where the old Microsoft SPF-clone  Sender-ID with its PRA 
algorithm came into play where it took the 5322.From into account. It 
was a DATA technology and as with my early data analysis, if Sender-ID 
failed at DATA, so would SPF at PRE-DATA.   It was rare to see the 
difference.  By far, more overhead in SMTP processing by delaying SPF 
and waiting until the potentially wasteful DATA payload was 
transferred.  It was certainly not optimal.  But there were the 
conditions where there were false positives.

Finally, what if the bad guy created the Headers you wanted to see? 
How can you stop it?  When DKIM finally come along (after SPF), it 
helped deal with the integrity of the payload AS DEFINED by the 
originating author domain (5322.From).   But when it was altered by 
the 3rd party mailers, it threw the proverbial "wrench" into the 
process.  How to trust the 3rd party "interfering" system became the 
ultimate problem we have been trying to resolve.

At the end of the day, nothing works when the required information 
needed to "answer the authorizing question" is not coming from the 
authoring/originating domain.

ADSP and its successor, DMARC. all fail because it lacks 3rd party 
authorization methods.

ATPS was proposed to augment ADSP to address 3rd party resigners. 
Still scratching head why it wasn't carried on to DMARC.  It will not 
address all scales, but it will for most of them them.

In the mean time, we continue to look for extremely high head PAYLOAD 
algorithms that 80% of the time, is just waste when it comes to hard 
rejection exclusive SPF policies or 5322.From Policies like ADSP or DMARC.

It is that 20% or less, of dealing with the middle ware list server 
that is ignoring SPF, ADSP, DMARC and just blindly altering and 
resigning with DKIM.   If we allow that, then any 3rd party can do the 
same.  There is no trust. We don't have an authorization process. 
Thats been the dearth of tbe problem of completing this 14+ years of 
DKIM R&D project.

We never completed a sound POLICY layer.

-- 
HLS

---

From: Hector Santos <[email-masked]>
Date: Wed, 11 Oct 2017 10:18:28 -0400

sorry, meant to say SPF is NOT a payload technology.

-- 
HLS

---

From: Grant Taylor <[email-masked]>
Date: Thu, 12 Oct 2017 12:53:38 -0600

> in the forwarder's bounce address, but it doesn't help.

Where can I find out more about how / why SRS does not help.

I've been aware of SRS for quite some time and I always thought that it 
did address SPF issues caused by forwarding messages.

-- 
Grant. . . .
unix || die

---

From: Hector Santos <[email-masked]>
Date: Thu, 12 Oct 2017 15:17:12 -0400

On 10/12/2017 2:53 PM, Grant Taylor wrote:
>> * There is a kludge called SRS that embeds the original bounce
>> address in the forwarder's bounce address, but it doesn't help.
>
> I've been aware of SRS for quite some time and I always thought that
> it did address SPF issues caused by forwarding messages.

SRS "didn't work" (catch on) because there is/was a fundamental 
concept/idea/methodology/philosophy/taboo that the Return Path 
(5321.MailFrom) should never be changed/altered/tampered with along 
the transport path/route.

It's the same long time mail engineering ethics mindset against 
altering the 5322.From original/authoring address to deal with 3rd 
party issues and ADSP/DMARC strong policies.

-- 
HLS

---

From: Grant Taylor <[email-masked]>
Date: Thu, 12 Oct 2017 14:58:10 -0600

On 10/12/2017 01:17 PM, Hector Santos wrote:
> (5321.MailFrom) should never be changed/altered/tampered with along the 
> transport path/route.

Interesting.

That's new to me.  But I haven't administered email as a day job in a 
number of years.  Just personal / friend / family things now.

I can see why people might view the Return Path (RFC 5321 Mail From) and 
 From header (RFC 5322 From) as sacrosanct.  I can somewhat get behind 
it.  At least for the normal delivery path, including relays.

I guess I personally view things like forwarding email and mailing lists 
as being the end of one delivery path and the start of a second delivery 
path.

> It's the same long time mail engineering ethics mindset against altering 
> the 5322.From original/authoring address to deal with 3rd party issues 
> and ADSP/DMARC strong policies.

I strongly view things like the original recipient's .forward file and 
mailing lists to be entities unto themselves.  I deliver messages to 
them, and they can do what ever they want to with them.  If that means 
that they generate new messages (Auto-Submitted: auto-generated) based 
on the incoming messages, so be it.  -  I view these new messages as 
being from said entity, thus think the 5321.MailFrom and 5322.From 
/should/ reflect said entity.

I know that I am *not* personally sending /individual/ messages to all 
of the DMARC list subscribers.  -  I /am/ sending a single message the 
DMARC mailing list.  Subsequently the DMARC mailing list is sending many 
individual messages to all the list subscribers.  ;-)

But, to each his / her own.

Thank you for the enlightenment Hector.

-- 
Grant. . . .
unix || die

---

From: "John Levine" <[email-masked]>
Date: 13 Oct 2017 16:54:19 -0000

>Where can I find out more about how / why SRS does not help.

SRS is one of those magic bullets that might have worked if everyone
in the world implemented it at the same time, and the design were
written by someone who understood the security issues.  In the real
world, any change to e-mail is implemented gradually, so you have to
be prepared for the old way and new way to work in parallel for a long
time.  So if you have to be prepared to ignore overenthusiastic SPF
-all anyway, who needs SRS?  

There's no technical way to tell an incoming message with a real
rewritten SRS address from one with a fake SRS address intended to
make spam look less spammy, other than by checking against a whitelist
of known virtuous forwarders.  But if you have that whitelist, you
don't need SRS, you just skip the SPF check on mail from those
forwarders.  (Gmail approximately does that.)

The other problem that SRS is supposed to fix, forwarding back bounce
messages, barely exists any more.  Due to the floods of spam with
forged return addresses, bounce messages are now mostly reflected spam
going back to someone who didn't send the message in the first place.
MTAs (other than those written in Redmond WA) try hard to reject
undeliverable mail during the initial SMTP session rather than accept
and bounce.

The comment about some rule against changing bounce addresses is
wrong.  Every mailing list rewrites the bounce address on mail it
forwards.  That makes sense for lists, since they need to handle
subscriber bounces, not the person who originally sent the message.

R's,
John

---

From: Grant Taylor <[email-masked]>
Date: Fri, 13 Oct 2017 11:24:22 -0600

On 10/13/2017 10:54 AM, John Levine wrote:
> written by someone who understood the security issues.

Would you please elaborate on (or point me to existing documentation) on 
the security issues that you're referring to?

> In the real 
> time.
Agreed.

> So if you have to be prepared to ignore overenthusiastic SPF 

Maybe it's just me, but I've never felt the desire to ignore 
overenthusiastic SPF.  Rather I've always been more of the opinion to 
push back and have the (purported) sending domain adjust their SPF 
record or persuade the forwarder to use something like SRS.

I feel that if a sending domain sets "-all" they (hypothetically) know 
what they are doing and that they do in fact want me to treat things 
that fail SPF as spam.  -  Or the first few failed deliveries turns into 
a learning opportunity for the sending domain and they adjust their policy.

> of known virtuous forwarders.

What's wrong with using SPF to filter fake messages coming from an 
invalid source?  I.e. -all on the SRS domain.  (I typically have each 
MTA use it's hostname as the SRS domain and publish an SPF record for 
the host that only allows it to send messages from it's name.)

> But if you have that whitelist, you

I've never felt the need for such a white list.  -  Maybe it's the 
nature of my small mail server having different use cases compared to 
the bigger players.

> undeliverable mail during the initial SMTP session rather than accept

I agree that's how things should be.  Sadly, I still see people not 
doing so, and sending bounces after the fact.

> subscriber bounces, not the person who originally sent the message.

I assume you're referring to Hector's comment about the "Return Path 
(5321.MailFrom)" being sacrosanct.

-- 
Grant. . . .
unix || die

---

From: Rick van Rein <[email-masked]>
Date: Fri, 13 Oct 2017 21:36:32 +0200

Hello Hector, Brandon, Dave and John,

Thanks for the detailed responses to what may be naive questions that
pop up again and again... It is quite clear that SPF is more problematic
to make work through unknown relays and lists than DKIM, and so that my
proposed work on Lenient DKIM is a better thing to work on.

As for responding to such questions, it is indeed helpful to know *why*
things were decided against, especially if this was 10 years ago. 
History should not repeat itself, but the grounds for a decision are
useful to weigh again every now and then.  This is one of the reasons
why I was asking around.

It's interesting to hear that rolling back of Received: headers is done
at Google [apparently with /128 prefixes for IPv6, and resulting trouble
with private address changes] and the reason I find it interesting is
that we're designing an identity system that would also manage external
aliases, so it could do the thing of which we discussed that [at least
some] users might use to shoot themselves in the foot.

Very clear in all this is that SPF is riddled with problems and should
not be included lightly.  SRS puts a strain on intermediates and won't
solve things IMHO.  I wish I could say otherwise for DKIM, but that too
breaks more than I like, and so I am really really looking forward to
the next ARC draft and hope to also come up with a MIME-aware
canonicalisation proposal in proze/draft and poetry/code.

Thanks,
 -Rick

---

From: Hector Santos <[email-masked]>
Date: Fri, 13 Oct 2017 17:21:32 -0400

> (5321.MailFrom)" being sacrosanct.

Note, a "Mailing List" is a whole different horse in this matter. 
That would be a 1::Many (one to many) group-ware distribution process 
where historically, the return path is (almost always) set to the 
bounce/error/list-admin/distributor/mailer-daemon address -- never the 
original message author submitter. It didn't make technical mail sense 
to do that.  There was never any ambiguity about that with List Server 
development.  That was not the case in a 1::1 direct/private transport 
where persistency was expected along the SMTP transport path.

-- 
HLS

---

From: "John Levine" <[email-masked]>
Date: 16 Oct 2017 01:39:19 -0000

In article <[email-masked]> you write:
>-=-=-=-=-=-

See the paragraph about forged domains and whitelists in the original message.

>overenthusiastic SPF.

It's just you.  I talk to people who run large mail systems, and
without exception they tell me that they do not reject on spf -all
other perhaps a plain -all which means someone sends no mail at all..
The false positive rate would just be too high.

>what they are doing ...

Wow, you're the optimist.

R's,
John

---

From: Hector Santos <[email-masked]>
Date: Sun, 15 Oct 2017 22:49:49 -0400

> The false positive rate would just be too high.

Expressing a different point of view since the world of electronic 
mail is not completely made of "large mail systems."

Its not just him.

Big difference on whether its a public ESP with forwarding, hosting 
operations or a private (any size) domain who generally deal with 
final destination operations only and are not in the business of 
forwarding/hosting or they got out of the hosting business like so 
many did.

 From a SPF functional specs standpoint, if a domain has a -ALL 
policy, it would be an operational mistake to presume or expect 
receivers will not be rejecting their mail on SPF failures.  A mistake 
on their SPF policy holder part to believe that receivers will be 
accepting failures when the specs (old and new) has it burned in to be 
a rejectable condition.  The older, original spec was very  clear 
about, the newer spec appropriately expounded on the potential risk 
for false positives. Nonetheless, the SPF hard fail intention is to 
REJECT, not to ACCEPT.  Therefore a mistake to believe receivers will 
always accept the failed SPF message.

It would also be a operational mistake to presume rejection will not 
take place at SMTP immediately, before DATA and before DMARC is 
processed at DATA.  Do not presume that a DMARC receiver will delay 
the rejection until the payload is transferred.

For our package, out of the box, our mostly private operators have 
SMTP reject on hard fail policies. That is what the domain wanted, 
that is what the spec indicated can happen, that is what they can get 
-- an immediate dynamic rejection.   In 14 years of using SPF, it has 
never been a big support discussion. SRS came up here and there but I 
don't think operators have been changing the default, but if they did, 
no big deal. Just do it because SPF was not for their operation. There 
have been operators who turned off SPF completely because they have 
combined their mail frontends but overall, 14 years worth of stats 
have shown a low to nil false positive, i.e. hardly ever any support 
concerns.

To me, SPF has been highly successful with a high confidence payoff of 
between 4% to 8% dynamic rejection rate over the years.

-- 
HLS

---

From: Grant Taylor <[email-masked]>
Date: Sun, 15 Oct 2017 21:52:26 -0600

> See the paragraph about forged domains and whitelists in the original message.

*chuckle*

Okay.

I feel like having each forwarding host use SRS with it's own FQDN as 
the from domain -and- publishing an SPF record for the host with -ALL 
covers that.

However, given what I think I've learned about your opinion of SPF -ALL, 
I can see why you say what you say.

To each his / her own.

> The false positive rate would just be too high.

I doubt that it's just me.  I may well be in the extreme minority.  But 
I find it hard to believe that it's just me.  (Disregard if you were 
re-using the words that I poorly chose.)

> Wow, you're the optimist.

I am, but I think that's independent of this.

If you tell me that you will only send email from given location(s) and 
that I should treat everything from anywhere else as suspect, that's 
exactly what I'm going to do.

I believe that the onus is on you to either adhere to what you say, or 
change what you say to reflect your new sending location(s).

I believe in Taylor Mali's "policy about honesty and ass-‐kicking: if 
you ask for it, then I have to let you have it."

Link - What Teachers Make
  - https://taylormali.com/poems/what-teachers-make/

If you put "-ALL" in your SPF record, I'm going to reject messages 
purportedly from you that fail your SPF record.  ;-)

-- 
Grant. . . .
unix || die

---

From: Hector Santos <[email-masked]>
Date: Mon, 16 Oct 2017 02:23:41 -0400

On 10/15/2017 11:52 PM, Grant Taylor wrote:
> If you tell me that you will only send email from given location(s)
> and that I should treat everything from anywhere else as suspect,
> that's exactly what I'm going to do.
>

+1.

For failures, the SPF domain declaring an -ALL is expecting rejection 
and if not, due to a receiver local policy, the SPF domain will|should 
expect the accepted message to be separated from the target user's 
main online mail in-box stream and/or POP3 mail stream which is 
generally a single pickup channel, not one combined with normal+spam 
folders. Don't presume IMAP is always users or Online only.  POP3 is 
still active.

See the RFC7208 security considerations section 11.7.

    https://tools.ietf.org/html/rfc7208#section-11.7

    11.7.  Delivering Mail Producing a "Fail" Result

    Operators that choose to deliver mail for which SPF produces a "fail"
    result need to understand that they are admitting content that is
    explicitly not authorized by the purported sender.  While there are
    known failure modes that can be considered "false negatives", the
    distinct choice to admit those messages increases end-user exposure
    to likely harm.  This is especially true for domains belonging to
    known good actors that are typically well-behaved; unauthorized mail
    from those sources might well be subjected to much higher skepticism
    and content analysis.

    SPF does not, however, include the capacity to distinguish good
    actors from bad ones, nor does it handle the concept of known actors
    versus unknown ones.  Those notions are out of scope for this
    specification.

-- 
HLS