Subject: [dmarc-ietf] Known DMARC failures of legitimate email

From: Terry Zink <[email-masked]>
Date: Wed, 5 Feb 2014 23:43:23 +0000

Hello,

We're currently building DMARC support and before we go live, I am looking for some numbers around known cases of DMARC quarantining/rejecting legitimate email. Here are cases I can think of:

1. Case 1 - sender publishes DMARC and only authenticates with SPF, user auto-forwards their email

Example: bulk sender -> Hotmail -> Gmail
This would pass DMARC at Hotmail (since bulk sender publishes SPF) but fail at Gmail (since Gmail will see Hotmail's IP but bulk sender in the 5321.MailFrom)

According to Google in a blog post - http://googleonlinesecurity.blogspot.com/2013/12/internet-wide-efforts-to-fight-email.html, around 75% of messages authenticate with DKIM and SPF. 15% authenticate with SPF only, 2% with DKIM only and about 9% no authentication.

The only ones that would be a problem in Case 1 is SPF-only, and only those that publish DMARC records with p=quarantine/none. Does anyone know how much email this might be?

2. Case 2 - discussion/mailing lists. This is a known limitation of DMARC and there are workarounds, but if no one does anything, the day after we turn on DMARC how much email would this affect?

Does anyone have numbers on how much this would affect?

3. Case 3 - anything else? Possibly broken DKIM signatures? Based upon threads, DKIM seems to pass anywhere between 10% and 90% of the time. Seems to be that the higher number is more realistic since if your DKIM is breaking, you'd want to know about it. This would be especially true for those who publish DMARC.

Thanks.

-- Terry

---

From: "J. Trent Adams" <[email-masked]>
Date: Wed, 05 Feb 2014 17:38:40 -0700

Terry -

> Does anyone have numbers on how much this would affect?

I think that this depends on the type of mail being sent. Our data
indicates this is an issue affecting less than 2% of our customers (and
it's likely much less than that). The mail we send, however, is largely
transactional (i.e., mail sent from our system directly to customers
such as receipts and account updates), so YMMV.

- Trent

> _______________________________________________
> dmarc mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dmarc

-- 
J. Trent Adams

Profile: http://www.mediaslate.org/jtrentadams/
LinkedIN: http://www.linkedin.com/in/jtrentadams
Twitter: http://twitter.com/jtrentadams

---

From: Franck Martin <[email-masked]>
Date: Wed, 5 Feb 2014 20:45:19 -0600 (CST)

----- Original Message -----
> From: "Terry Zink" <[email-masked]>
> To: [email-masked]
> Sent: Wednesday, February 5, 2014 3:43:23 PM
> Subject: [dmarc-ietf]  Known DMARC failures of legitimate email
> 
> Hello,
> 
> We're currently building DMARC support and before we go live, I am looking
> for some numbers around known cases of DMARC quarantining/rejecting
> legitimate email. Here are cases I can think of:
> 
> This would pass DMARC at Hotmail (since bulk sender publishes SPF) but fail
> at Gmail (since Gmail will see Hotmail's IP but bulk sender in the
> 5321.MailFrom)

hotmail does not break DKIM when forwarding AFAIK. So if a sender do DMARC with p!=none with no DKIM, they shoot themselves in the foot, and they will know quickly that it hurts.

> 
> According to Google in a blog post -
> http://googleonlinesecurity.blogspot.com/2013/12/internet-wide-efforts-to-fight-email.html,
> around 75% of messages authenticate with DKIM and SPF. 15% authenticate with
> SPF only, 2% with DKIM only and about 9% no authentication.
> 
> The only ones that would be a problem in Case 1 is SPF-only, and only those
> that publish DMARC records with p=quarantine/none. Does anyone know how much
> email this might be?
> 
Once again, with DMARC if you shoot yourself in the foot, you will know it hurts.

> 
> 2. Case 2 - discussion/mailing lists. This is a known limitation of DMARC and
> there are workarounds, but if no one does anything, the day after we turn on
> DMARC how much email would this affect?
> 

mailing list traffic is low in all legitimate mail (<10% I have heard), and some mailing lists keep DKIM.

> 
> 
> 3. Case 3 - anything else? Possibly broken DKIM signatures? Based upon
> threads, DKIM seems to pass anywhere between 10% and 90% of the time. Seems
> to be that the higher number is more realistic since if your DKIM is
> breaking, you'd want to know about it. This would be especially true for
> those who publish DMARC.
> 

In all these cases, the problem is not on the receiver side, but on the sender side. When a sender moves to p!=none it does it perfectly knowing what will be the impact based on the aggregate reports, while making sure everything is DKIM signed with SPF also on hostnames.

As a sender, I would confirm the numbers Trent gave which is about 2 to 1% email loss (even lower). Which is nothing in comparison of all the spoofing mitigated and I'm not even talking about controlling mail streams and third party vendor on boarding.

As a mailbox provider, you don't want to publish anything than p=none for your domain.

As a receiver, you save a lot by checking DMARC on incoming emails and let the sender put the right policies.

---

From: Michael Adkins <[email-masked]>
Date: Thu, 6 Feb 2014 19:19:45 +0000

On 2/5/14 3:43 PM, "Terry Zink" <[email-masked]> wrote:

>Hello,
>
>We're currently building DMARC support and before we go live, I am
>looking for some numbers around known cases of DMARC
>quarantining/rejecting legitimate email. Here are cases I can think of:
>
>This would pass DMARC at Hotmail (since bulk sender publishes SPF) but
>fail at Gmail (since Gmail will see Hotmail's IP but bulk sender in the
>5321.MailFrom)
>
>https://urldefense.proofpoint.com/v1/url?u=http://googleonlinesecurity.blo
>gspot.com/2013/12/internet-wide-efforts-to-fight-email.html&k=ZVNjlDMF0FEl
>m4dQtryO4A%3D%3D%0A&r=fAk2HhpwqneloqGEFXhAtQ%3D%3D%0A&m=pRSJ4QiEBWv2%2BArW
>%2FhQZZWTcTmL7J2ianTBs7FiroIw%3D%0A&s=8cdd07fe693bdd2263ba26b7e1c918fdba1c
>ed24fb57fcb7718ce61758fe3f14, around 75% of messages authenticate with
>DKIM and SPF. 15% authenticate with SPF only, 2% with DKIM only and about
>9% no authentication.
>
>The only ones that would be a problem in Case 1 is SPF-only, and only
>those that publish DMARC records with p=quarantine/none. Does anyone know
>how much email this might be?

I suspect that this is going to vary a lot from one system to the next, so
I don't know that anyone else can give you great advice on it.  What I can
tell you is that a large scale DMARC system needs a mechanism to
automatically detect legitimate forwarding and exempt them from reject and
quarantine actions.  DMARC has the notion of 'trusted forwarders' defined
in the spec and accounted for in the reporting.  There are a few different
ways to do that, some that border on secret sauce, and are mostly a
combination of a reputation or trust system combined with something
observable in the header or the traffic patterns.  For example, you can
look for authentication results headers that were added by the upstream
system and choose to use their results if you trust them (aka transitive
trust).  You can look for other headers that are typically added to
forwarded email (X-Original-To, etc) and decide to use them as a signal if
you trust the upstream system.  If you sign your outbound email, and you
have enough volume, you will see some amount of your own signed email
forwarded back to your system with the signatures still intact.  That's a
good signal that the upstream system legitimately forwards stuff.

>
>
>2. Case 2 - discussion/mailing lists. This is a known limitation of DMARC
>and there are workarounds, but if no one does anything, the day after we
>turn on DMARC how much email would this affect?
>

Again, this is going to vary from system to system.  I would recommend
that you have a 'log only' mode for your DMARC implementation so you can
roll it out and have it report what action it would have taken to you
without actually impacting delivery.  This is the other case where you
need an automated mechanism to detect legitimate lists based on their
behavior and exempt them.  From experience, I can tell you that if you
exempt email signed by google groups, yahoo groups and it's international
flavors, and microsoft live groups, then you will have solved the problem
for 90% of users.

>threads, DKIM seems to pass anywhere between 10% and 90% of the time.
>Seems to be that the higher number is more realistic since if your DKIM
>is breaking, you'd want to know about it. This would be especially true
>for those who publish DMARC.

Those are the two big ones.  I haven't really run into other problems.

>https://urldefense.proofpoint.com/v1/url?u=https://www.ietf.org/mailman/li
>stinfo/dmarc&k=ZVNjlDMF0FElm4dQtryO4A%3D%3D%0A&r=fAk2HhpwqneloqGEFXhAtQ%3D
>%3D%0A&m=pRSJ4QiEBWv2%2BArW%2FhQZZWTcTmL7J2ianTBs7FiroIw%3D%0A&s=7f67d6d41
>adc94b6e45861fa6dbdf4beae357311b2f59043d361ce6e47ca59fb

---

From: Roland Turner <[email-masked]>
Date: Fri, 07 Feb 2014 13:30:52 +0800

> We're currently building DMARC support and before we go live, I am looking for some numbers around known cases of DMARC quarantining/rejecting legitimate email. Here are cases I can think of:

No-one is in a better position than you are to survey the DMARC breakage 
that affects you, and to get real numbers about the impact of each 
cause. It would be sensible to implement DMARC initially with no action 
taken on the results (apart from logging). You then have source data to 
mine to:

  * Identify DKIM-breaking forwarders that you already know have sound
    filtering, so DMARC processing can be skipped for them
  * Identify heavily spoofed domains that should be in the first batch
    to have Domain Owner policy execution turned on
  * Identify ...overly enthusiastic Domain Owners who are publishing
    quarantine/reject policies despite not having their message streams
    under control; their policies should of course not be executed while
    that is the case.

Presumably you'll want to do this on an ongoing basis. Initially it 
would make sense to turn on policy execution only for those domains 
which are heavily phished, over time you're likely to be able to lower 
the threshold to the point where you're executing all Domain Owner 
policies apart from those observed to be broken.

(See also Dave's Using DMARC 
<http://tools.ietf.org/html/draft-crocker-dmarc-bcp-03#section-6> draft)

- Roland

---

From: Roland Turner <[email-masked]>
Date: Fri, 07 Feb 2014 13:34:42 +0800

> let the sender put the right policies.

I suspect this depends to some extent upon how captive your users are. 
Yours clearly are, Terry's are to a large extent, others' are not. For 
the latter group, user churn is a big concern.

- Roland

---

From: Roland Turner <[email-masked]>
Date: Fri, 07 Feb 2014 13:40:32 +0800

On 02/07/2014 03:19 AM, Michael Adkins wrote:

> DMARC has the notion of 'trusted forwarders' defined in the spec and 
> accounted for in the reporting. There are a few different ways to do 
> that, some that border on secret sauce, and are mostly a combination 
> of a reputation or trust system combined with something observable in 
> the header or the traffic patterns.
...
> From experience, I can tell you that if you exempt email signed by 
> google groups, yahoo groups and it's international flavors, and 
> microsoft live groups, then you will have solved the problem for 90% 
> of users. 

Most of the others are also pretty easy to find: any source which has a 
relatively stable long term message volume and a very low detected spam 
rate but authentication failures for a large number of domains is almost 
certainly a well-behaved forwarder that, at least as a first 
approximation, can be exempted from DMARC processing.

- Roland

---

From: Terry Zink <[email-masked]>
Date: Fri, 7 Feb 2014 17:42:45 +0000

Thanks, everyone for the advice. This has been useful.

-- Terry

-----Original Message-----
From: dmarc [mailto:[email-masked]] On Behalf Of Roland Turner
Sent: Thursday, February 6, 2014 9:41 PM
To: [email-masked]
Subject: Re: [dmarc-ietf] Known DMARC failures of legitimate email

...
> microsoft live groups, then you will have solved the problem for 90% 

Most of the others are also pretty easy to find: any source which has a relatively stable long term message volume and a very low detected spam rate but authentication failures for a large number of domains is almost certainly a well-behaved forwarder that, at least as a first approximation, can be exempted from DMARC processing.

- Roland

---

From: "J. Gomez" <[email-masked]>
Date: Sat, 8 Feb 2014 21:19:59 +0100

On Thursday, February 06, 2014 12:43 AM [GMT+1=CET], Terry Zink wrote:

> This would pass DMARC at Hotmail (since bulk sender publishes SPF)
> but fail at Gmail (since Gmail will see Hotmail's IP but bulk sender
> in the 5321.MailFrom)  

I think in that case Gmail would see the bulk sender in the 5322.MimeFrom and Hotmail in the 5321.MailFrom. Threfore, the SPF-itself check would pass at Gmail (because that check would validate that the email is Hotmail-authentic), although DMARC would fail its (version of the) SPF-check at Gmail (because that check would try to validate whether the email is BulkSender-authentic).

Yes, it is a mess.

The solution is to check SPF-itself before checking DMARC, and then exempt from DMARC processing those emails which get an SPF-itself result of pass AND come from known-good domains (according to your locally maintained whitelist of known-good domains).

> 2. Case 2 - discussion/mailing lists. This is a known limitation of
> DMARC and there are workarounds, but if no one does anything, the day
> after we turn on DMARC how much email would this affect?  
> 

You will have to implement mailing-list detection logic on your side, and then exempt those emails from DMARC processing. It is either that, or breaking mailing lists for your users (they won't be happy about it). The other option is to never obey DMARC's p=reject policy.

Regards,

J.Gomez

---

From: Mike Jones <[email-masked]>
Date: Mon, 10 Feb 2014 15:28:30 -0800

On Wed, Feb 5, 2014 at 3:43 PM, Terry Zink <[email-masked]>wrote:

> around 75% of messages authenticate with DKIM and SPF. 15% authenticate
> with SPF only, 2% with DKIM only and about 9% no authentication.
>

I've been looking into the DMARC data we have to see if I can come up with
a number here.  Only a few DMARC reporters use the "mailing_list" policy
override reason in their DMARC reports.  Looking at just those reporters,
about 4.8% of DMARC failures are reported with a mailing_list override.
That number is for all DMARC failures, not just those from domains with a
reject or quarantine.

Doing some further digging I looked at the traffic reported with a source
IP that was identified as "mailing_list" at least once. From that list, if
at least 75% of the IP's messages were tagged as "mailing_list" by those
DMARC reporters, then I assumed that IP was primarily used for mailing
lists.  So I took that list of IPs and applied it to all DMARC reporter's
data, whether they report a mailing_list reason or not.  What I came up
with was that across all DMARC failures reported by all DMARC reporters,
about 1.4% of DMARC failures came from this list of "mailing_list" source
IPs.

So somewhere in the range of 1.4% - 4.8% of DMARC failures are due to
mailing lists. Again, this is for all DMARC failures, regardless of the
policy on the domain. So turning on DMARC would affect whatever fraction of
that % is coming from domains with a reject policy/quarantine policy.

Mike

---

From: "J. Gomez" <[email-masked]>
Date: Fri, 14 Feb 2014 16:07:51 +0100

From: Mike Jones
> On Wed, Feb 5, 2014 at 3:43 PM, Terry Zink
> <[email-masked]> wrote: 
> 
> quarantining/rejecting legitimate email.
> 
> mailing lists. Again, this is for all DMARC failures, regardless of
> the policy on the domain. So turning on DMARC would affect whatever
> fraction of that % is coming from domains with a reject
> policy/quarantine policy.

Yes, but that pertcentage is the "percentage of messages affeced of total email", not the "percentage affected users". I think the percentage of affected users (those who use at least one mailing list, even if seldom), is going to be higher than that.

Regards,

J.Gomez