Subject: [dmarc-ietf] Limiting what goes in aggregate reports

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 21 Jan 2014 12:55:26 -0800

I've received a feature request for OpenDMARC to limit what goes into
aggregate reports.  Specifically, the suggestion is not to store any data
regarding messages that pass the DMARC test and thus only report on things
that fail.

I don't believe the current specification says anything requiring that all
messages be recorded, so I think this wouldn't violate the specification,
but it might violate the spirit of what was intended or what's desirable to
Domain Owners.

Comments?

-MSK

---

From: Scott Kitterman <[email-masked]>
Date: Tue, 21 Jan 2014 16:09:59 -0500

> but it might violate the spirit of what was intended or what's desirable to

I've done statistical analysis on DMARC aggregate reports (and similar 
reporting) that can't be done without knowing the total population.  If the 
results in the aggregate report are filtered, I think it'd be very useful to 
know that.  I don't think there's currently any way to indicate it though.

Scott K

---

From: Franck Martin <[email-masked]>
Date: Tue, 21 Jan 2014 15:20:38 -0600 (CST)

----- Original Message -----

> From: "Murray S. Kucherawy" <[email-masked]>
> To: [email-masked]
> Sent: Tuesday, January 21, 2014 12:55:26 PM
> Subject: [dmarc-ietf] Limiting what goes in aggregate reports

> Comments?

I think it is starting to build too much complexity in what receivers must do to generate aggregate reports. We should keep it light for receivers, it is up to senders to process the data. 

It seems important to me to be able to look at the ratio pass/fail to know if you have a phishing problem, or if you have something in your infrastructure that would reject too many good emails, compared to your phishing problem, if you move to an active policy. 

I think the spec should indicate, if not done already, that the receiver SHOULD(MUST?) report all emails in the aggregate report, but if people decide to go for this proposed option then at least notify it dumped some certain categories. 

However if you don't have the report of emails that pass, how do you find sending IPs that may have your DKIM key? 

So I'm not for limiting what goes in aggregate reports.

---

From: Mike Jones <[email-masked]>
Date: Tue, 21 Jan 2014 13:26:57 -0800

I echo Scott's position.  Understanding certain reasons for a daily
increases (or decreases) in authentication failures is possible when you
know the passing volume as well as the failing volume.  Domains with
sending volumes that vary widely day to day for example, will have failure
volumes that vary widely.  Is the increase due to a problem or is there is
just more mail being sent and the failure rate is normal?

Mike

On Tue, Jan 21, 2014 at 1:09 PM, Scott Kitterman <[email-masked]>wrote:

> _______________________________________________
> dmarc mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dmarc
>

---

From: Steven M Jones <[email-masked]>
Date: Tue, 21 Jan 2014 13:32:04 -0800

Bare minimum, you need a clear way to indicate tat the report is truncated in this manner.

As a domain owner I'd prefer the whole picture. But as I'm mobile, a good reason for the receiver will have to wait...

--S.

Sent from my highly functional, highly vulnerable smartphone

> On Jan 21, 2014, at 12:55, "Murray S. Kucherawy" <[email-masked]> wrote:
> 
> I've received a feature request for OpenDMARC to limit what goes into aggregate reports.  Specifically, the suggestion is not to store any data regarding messages that pass the DMARC test and thus only report on things that fail.
> 
> I don't believe the current specification says anything requiring that all messages be recorded, so I think this wouldn't violate the specification, but it might violate the spirit of what was intended or what's desirable to Domain Owners.
> 
> Comments?
> 
> -MSK

---

From: "Popowycz, Alex" <[email-masked]>
Date: Tue, 21 Jan 2014 17:04:08 -0500

The more that’s left up to the receivers as to what does/does not end up in the aggregate report, the less valuable those reports become. As it stands already, it’s difficult enough to compare the results coming from receivers today let alone having the reports become even more fragmented and less deterministic. Is this suggestion being driven by work effort concerns (in creating a “complete and accurate” aggregate report?

True, the reporting recommendations are fairly open in the specification, but an intrinsic value of DMARC is reduced (IMHO materially) if the reports are diluted.

From: dmarc [mailto:[email-masked]] On Behalf Of Franck Martin
Sent: Tuesday, January 21, 2014 4:21 PM
To: Murray S. Kucherawy
Cc: [email-masked]
Subject: Re: [dmarc-ietf] Limiting what goes in aggregate reports

________________________________
From: "Murray S. Kucherawy" <[email-masked]<mailto:[email-masked]>>
To: [email-masked]<mailto:[email-masked]>
Sent: Tuesday, January 21, 2014 12:55:26 PM
Subject: [dmarc-ietf] Limiting what goes in aggregate reports

---

From: John Sweet <[email-masked]>
Date: Tue, 21 Jan 2014 14:34:29 -0800

I'm not sure whether the feature being requested is a) providing the sender
with another DMARC tag to specify that they only want aggregate reports to
include failures, or b) receivers, at their discretion, being able to
choose to only report failures.

I can imagine a valid use case for a), but not for b).

On Tue, Jan 21, 2014 at 2:04 PM, Popowycz, Alex <[email-masked]>wrote:

> The more that’s left up to the receivers as to what does/does not end up
> in the aggregate report, the less valuable those reports become. As it
> stands already, it’s difficult enough to compare the results coming from
> receivers today let alone having the reports become even more fragmented
> and less deterministic. Is this suggestion being driven by work effort
> concerns (in creating a “complete and accurate” aggregate report?
>
>
>
> True, the reporting recommendations are fairly open in the specification,
> but an intrinsic value of DMARC is reduced (IMHO materially) if the reports
> are diluted.
>
>
>
>
>
>
>
> *From:* dmarc [mailto:[email-masked]] *On Behalf Of *Franck Martin
> *Sent:* Tuesday, January 21, 2014 4:21 PM
> *To:* Murray S. Kucherawy
> *Cc:* [email-masked]
> *Subject:* Re: [dmarc-ietf] Limiting what goes in aggregate reports
>
>
> ------------------------------
>
> *From: *"Murray S. Kucherawy" <[email-masked]>
> *To: [email-masked]
> *Sent: *Tuesday, January 21, 2014 12:55:26 PM
> *Subject: *[dmarc-ietf] Limiting what goes in aggregate reports
>
>
>
> I think it is starting to build too much complexity in what receivers must
> do to generate aggregate reports. We should keep it light for receivers, it
> is up to senders to process the data.
>
>
>
> It seems important to me to be able to look at the ratio pass/fail to know
> if you have a phishing problem, or if you have something in your
> infrastructure that would reject too many good emails, compared to your
> phishing problem, if you move to an active policy.
>
>
>
> I think the spec should indicate, if not done already, that the receiver
> SHOULD(MUST?) report all emails in the aggregate report, but if people
> decide to go for this proposed option then at least notify it dumped some
> certain categories.
>
>
>

---

From: Greg Colburn <[email-masked]>
Date: Tue, 21 Jan 2014 16:22:10 -0700

As others have said before, the value of the total aggregate numbers is significant.  They provide a comparison point to measure scale of malicious or fraudulent activity.  The initial request to the openDMARC list suggests a solution to a implementation problem that I believe presents a compromise to the value of the Aggregate Report.

"not to store any data regarding messages that pass the DMARC test and thus only report on things that fail”

The last line in 7.2 covers this requirement casually.  I think that we should consider strengthening this language in the next rev of the draft.

7.2. Aggregate Reports

   Visibility comes in the form of daily (or more frequent) Mail
   Receiver-originated feedback reports that contain aggregate data on
   message streams relevant to the Domain Owner.  This information
   includes data about messages that passed DMARC authentication as well
   as those that did not.

Greg C

On Jan 21, 2014, at 3:34 PM, John Sweet <[email-masked]<mailto:[email-masked]>> wrote:

I'm not sure whether the feature being requested is a) providing the sender with another DMARC tag to specify that they only want aggregate reports to include failures, or b) receivers, at their discretion, being able to choose to only report failures.

On Tue, Jan 21, 2014 at 2:04 PM, Popowycz, Alex <[email-masked]<mailto:[email-masked]>> wrote:
The more that’s left up to the receivers as to what does/does not end up in the aggregate report, the less valuable those reports become. As it stands already, it’s difficult enough to compare the results coming from receivers today let alone having the reports become even more fragmented and less deterministic. Is this suggestion being driven by work effort concerns (in creating a “complete and accurate” aggregate report?

From: dmarc [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Franck Martin
Sent: Tuesday, January 21, 2014 4:21 PM
To: Murray S. Kucherawy
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [dmarc-ietf] Limiting what goes in aggregate reports

_______________________________________________
dmarc mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/dmarc

_______________________________________________
dmarc mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/dmarc

---

From: Andreas Schulze <[email-masked]>
Date: Wed, 22 Jan 2014 13:14:30 +0100

> regarding messages that pass the DMARC test and thus only report on things

I read the initial "feature request" in and other way: The Requester  
complains about domains
requesting aggregated reports but are (temporary) unable to *receive* them.
-> The "queue pollution" was mentioned as a problem.

So maybe the spec could say "the operator of a reporting system may  
decide to temporary not send reports"
The reporting software - opendmarc in this case - has no ability to do  
that today.
Simple solution: create the report and temporary discard.

I also dislike the idea to exclude data regarding messages that pass  
the DMARC test.

Andreas

---

From: Roland Turner <[email-masked]>
Date: Thu, 23 Jan 2014 18:33:46 +0800

Domain owners will of course make use of whatever data they can get 
their hands on, but for many of the likely analyses the less 
contaminated/biased/truncated the data is, the better. DMARC as a whole 
will work better if the only limits to feedback are:

  * Legal constraints on receivers, if/where they exist
  * Distrust by receivers of specific domain owners
  * Throughput problems (domain owner/report receiver's mail-server is
    overloaded/down)

Receivers are of course free to do, or not do, as they wish - even if 
the spec says otherwise - however I suspect that adding knobs to twiddle 
that aren't widely useful will complicate the implementation of DMARC 
for receivers (and indeed, each knob adds decision questions for 
receivers to resolve thereby adding implementation cost; each on its own 
is small, but a large enough number invites implementation oversights 
and errors) and the interpretation of them by domain owners ("in 
addition to any other oddities, some receivers drop reporting of 
successes!") and thus diminish the effectiveness of DMARC overall.

This is not to say that features to suit receivers shouldn't be made to 
receiver-side implementations, but I'm inclined to believe that there's 
a better way to handle this particular issue. If I understand the 
initial report correctly, the concern is not the sending of "all is 
well" reports per se, but rather that domain owners/report receivers 
with broken infrastructure cause reports to get stuck in the 
receiver's/report sender's outbound queue. The solution to this problem 
is obvious: don't queue. Inconveniently, implementing this approach has 
to happen in the receiver's/report sender's outbound MTA unless you're 
willing to add an SMTP client (with DKIM signing!) to opendmarc. 
(Details of this discussion belong in another list, I know.)

- Roland

On 01/22/2014 04:55 AM, Murray S. Kucherawy wrote:
> aggregate reports.  Specifically, the suggestion is not to store any 
> data regarding messages that pass the DMARC test and thus only report 
> on things that fail.
>
> I don't believe the current specification says anything requiring that 
> all messages be recorded, so I think this wouldn't violate the 
> specification, but it might violate the spirit of what was intended or 
> what's desirable to Domain Owners.
>
> Comments?
>
> -MSK
>
>

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Thu, 23 Jan 2014 10:16:00 -0800

I don't think we can say anything in DMARC itself about local operational
problems, like "If the reports can't be delivered for a while, it's OK to
stop generating them."  That's really outside of the scope of a protocol
specification.

On Wed, Jan 22, 2014 at 4:14 AM, Andreas Schulze <[email-masked]>wrote:

> So maybe the spec could say "the operator of a reporting system may decide
> to temporary not send reports"
> I also dislike the idea to exclude data regarding messages that pass the
> DMARC test.
>
> Andreas
>
>

---

From: Roland Turner <[email-masked]>
Date: Fri, 24 Jan 2014 14:40:34 +0800

A concrete example occurred to me of an analysis which benefits from 
receiving reasonably accurate success data in addition to failure data: 
a non-Domain-Owner/non-Sender Report Receiver who is attempting to 
assess the competence of Domain Owner policy assertions will typically 
not have direct information about how many messages were sent in total; 
if success information is excluded by most/all report generators then 
any numerical analysis will be impossible, or at least very inaccurate.

On balance this case may be moot, but for completeness:

  * For a Report Receiver performing such an analysis for a Domain
    Owner, the latter is likely to be willing to provide relevant
    numbers anyway, albeit with greater difficulty and less accuracy.
  * For the same relationship, visibility of the behaviour - successful
    and unsuccessful - of ESPs acting on behalf of the Domain Owner is
    likely to be of particular interest. (This is also true for the case
    where the Report Receiver and Domain Owner are the same entity.)
  * A Report Receiver may face a conflict of interest in publishing
    competence assessments of the same Domain Owners who have requested
    that feedback be delivered to them in the first place, but equally
    they may not. Other things being equal, it would appear desirable to
    foster conditions in which this might happen.

None of these are directly helpful to receivers, but they do contribute 
to the usefulness of DMARC to other honest participants in the email 
system which is likely to indirectly benefit receivers in much the same 
sense that providing feedback does in the first place.

- Roland

On 01/23/2014 06:33 PM, Roland Turner wrote:
> contaminated/biased/truncated the data is, the better. DMARC as a 
> whole will work better if the only limits to feedback are:
>
> the spec says otherwise - however I suspect that adding knobs to 
> twiddle that aren't widely useful will complicate the implementation 
> of DMARC for receivers (and indeed, each knob adds decision questions 
> for receivers to resolve thereby adding implementation cost; each on 
> its own is small, but a large enough number invites implementation 
> oversights and errors) and the interpretation of them by domain owners 
> ("in addition to any other oddities, some receivers drop reporting of 
> This is not to say that features to suit receivers shouldn't be made 
> to receiver-side implementations, but I'm inclined to believe that 
> there's a better way to handle this particular issue. If I understand 
> the initial report correctly, the concern is not the sending of "all 
> is well" reports per se, but rather that domain owners/report 
> receivers with broken infrastructure cause reports to get stuck in the 
> receiver's/report sender's outbound queue. The solution to this 
> problem is obvious: don't queue. Inconveniently, implementing this 
> approach has to happen in the receiver's/report sender's outbound MTA 
> unless you're willing to add an SMTP client (with DKIM signing!) to 
> opendmarc. (Details of this discussion belong in another list, I know.)
>
> - Roland
>
>
>
>> I don't believe the current specification says anything requiring 
>> that all messages be recorded, so I think this wouldn't violate the 
>> specification, but it might violate the spirit of what was intended 
>> or what's desirable to Domain Owners.
>>
>> Comments?
>>
>> -MSK
>>
>>