Subject: [dmarc-ietf] Two new fields in aggregate reports

From: Alessandro Vesely <[email-masked]>
Date: Thu, 24 Oct 2019 19:53:57 +0200

Hi all,

it is difficult to tell what is each aggregate report's record.  It is easier
if the source IP is known.  Mailing lists can be told by their (unaligned) SPF
domain.  Otherwise, it is difficult to tell abuse from legitimate users using
the wrong server.

Getting a failure report for each source IP is not easy, because few mailbox
providers send failure reports.

In order to ease the understanding of aggregate reports, I propose two
additional per-record fields:

*score*:  The average score of the messages in the row; let's say an SA-like
number (< 0 good, > 10 bad, values in between may be worth human inspection).

*list*:  An enumerated type, for example "none", "black", "white", "both",
indicating if the source IP is listed in some public or private DNSxL that the
reporting MTA uses.

They're obviously subjective stuff.  However, most MTAs deploy at least one of
them, and summing up per-IP results every day can bring useful indications.

I haven't added those fields to http://bit.ly/dmarc-rpt-schema, yet.  Let's
discuss.  I hope they will make it to rfc7489bis.

Best
Ale
--

---

From: Brandon Long <[email-masked]>
Date: Thu, 24 Oct 2019 17:53:32 -0700

> number (< 0 good, > 10 bad, values in between may be worth human

This assumes that your spam system is able to make a spamminess score that
approximates
a continuum, I'm not sure that's true.  Also, it's a lot of spamminess
feedback, not sure if that
can be abused or not.

Also, as rejected at smtp time, especially for DMARC, we're unlikely to
have run a full spam
assessment on the message.

> indicating if the source IP is listed in some public or private DNSxL that

At first I was wondering if you were going to have the values of the
list-id header.

That said, though I've never used any DMARC aggregator services, I would
think that
one of the values they're likely to provide is IP address identification,
useful so you can
hunt down what things aren't sending DMARC compliant email yet (ie, looks
like you're sending
mail from ESP A and ESP B, fix those)... they can just as easily use the
various public IP reputation
sources.

Anyways, I doubt we'd expose anything internally... also, this is kind of
the opposite of the
spamminess score, in that it expects that things are black & white, and we
virtually never
classify things as such.

Brandon

---

From: Alessandro Vesely <[email-masked]>
Date: Fri, 25 Oct 2019 10:40:12 +0200

On Fri 25/Oct/2019 02:53:32 +0200 Brandon Long wrote:
> On Thu, Oct 24, 2019 at 10:55 AM Alessandro Vesely wrote:
>>
>> additional per-record fields:>>
>>
>> *score*:  The average score of the messages in the row; let's say an 
>> SA-like number (< 0 good, > 10 bad, values in between may be worth human 
>> inspection).>>
> 
> approximates a continuum, I'm not sure that's true.

The spam score is a rather common concept.  Of course, an MTA which does not
implement any such thing would omit this field.

> Also, it's a lot of spamminess feedback, not sure if that can be abused or
> not.

Correct.  However, it is an average, so a spammer would have a hard time trying
to work out the detail of how the receiver's score is computed.  Again, a
reporter may compute the std deviation along with the average and omit this
field if all messages have equal score.

OTOH, this is indeed a valuable feedback.  Mail sites could build their own
reputation system based on that.  A community IP database.  Revolutionary,
isn't it?

> Also, as rejected at smtp time, especially for DMARC, we're unlikely to have
> run a full spam assessment on the message.

Perhaps we could add a second count.  Or break the record into two rows.

>> *list*:  An enumerated type, for example "none", "black", "white",
>> "both", indicating if the source IP is listed in some public or private
>> DNSxL that the reporting MTA uses.>>
> 

Ah, that way we'd get rid of all those "on behalf of"... :-)

> think that one of the values they're likely to provide is IP address
> identification, useful so you can hunt down what things aren't sending DMARC
> compliant email yet (ie, looks like you're sending mail from ESP A and ESP
> B, fix those)...

Exactly.

> they can just as easily use the various public IP reputation sources.

Yes, could do, but it's more difficult.  Here the reporting MTA already has the
datum, and can include it for free.

This datum would be vouched by the reporting MTA.  It can be internal, like a
fail2ban database.  How about also reporting the source, if external?

> Anyways, I doubt we'd expose anything internally...

I always wonder why Google doesn't publish their internal IP database.

> also, this is kind of the opposite of the spamminess score, in that it
> expects that things are black & white, and we virtually never classify
> things as such.

The aim is to be able to look at an aggregate report and understand it.  For
example, if people can tell an IP is blackish, they'd just smile and be happy
for a good failure case.  What field would you suggest instead?

Best
Ale
--

---

From: Дилян Палаузов <[email-masked]>
Date: Fri, 25 Oct 2019 10:51:43 +0000

Hello Alessandro,

I do not see how this helps for DMARC.  An email either validates DMARC, or fails DMARC and the aggregate repors say per
sending IP server (only direct mail flow is reported), whether DMARC validates or fails.  With this information it is
sufficient to determine, if the DMARC/DKIM implementations on sender and receiver are either both bug-free,  or both
have the same bugs.

I do not see, how the information you ask to add, while interesting, does help DMARC.

What is the purposes of the aggregate and non-aggregate reports?  What are non-goals?  I asked several times here,
nobody answered.  Perhaps a discussion on the goals and non-goal would help.

If it is a goal to reuse the dmarc-reporting mechanism to report also about perceived spam probability, then it can be
discussed in more details how this can be achieved.  My experience is, that asking a provider, why an obviously non-spam 
mail was evaluated as spam, virtually never leads to a useful answer.  So nobody wants to reveal how its spam system
weigths factors and if there is lack of such interest, extending the report format will not help, as nobody will be
willing the report the data.

Exchanging information on hard-coded rules in Spam-Assassing (IP reputation, HTML mime part without text/plain, the
“Nigeria money” phrase), that is not based on filter training, does not help neither, as sender can run the tests on its
own and predict how the recipient will evaluate these set of criteria.

Regards
 Дилян

---

From: "Kurt Andersen (b)" <[email-masked]>
Date: Fri, 25 Oct 2019 08:21:19 -0700

On Fri, Oct 25, 2019 at 3:52 AM Дилян Палаузов <[email-masked]>
wrote:

>
> If it is a goal to reuse the dmarc-reporting mechanism to report also
> about perceived spam probability, then it can be
> discussed in more details how this can be achieved.  My experience is,
> that asking a provider, why an obviously non-spam
> mail was evaluated as spam, virtually never leads to a useful answer.  So
> nobody wants to reveal how its spam system
> weigths factors and if there is lack of such interest, extending the
> report format will not help, as nobody will be

Especially when the answer more or less boils down to "lots of recipients
treated messages like this as spam". There are so many squishy terms there
that it is essentially meaningless.

--Kurt

---

From: Chris Wedgwood <[email-masked]>
Date: Fri, 25 Oct 2019 09:56:26 -0700

> Correct.  However, it is an average, so a spammer would have a hard
> time trying to work out the detail of how the receiver's score is
> computed.

[randomly] permute the input, seen 1000s of messages and look at how
the score varies ... repeat ...

---

From: "John Levine" <[email-masked]>
Date: 25 Oct 2019 13:49:18 -0400

>nobody answered.  Perhaps a discussion on the goals and non-goal would help.

As far as I know, the point of DMARC reports is to help domain owners
understand who is sending mail that purports to be from them.  In a
large organization it can be remarkably hard to track down every mail
server in every department or every subcontractor that might be sending
real mail with the domain in the From: header.

The domain owners use the reports to do things like update SPF records
to include all of the sending hosts, update server configs to add DKIM
signatures, or to fix servers that are adding invalid signatures, and
often to shut rogue servers down that shouldn't have been sending mail
in the first place.

I can't see how spam scores would be of any use for any of these tasks.

R's,
John

---

From: Seth Blank <[email-masked]>
Date: Fri, 25 Oct 2019 10:52:57 -0700

> I can't see how spam scores would be of any use for any of these tasks.

+1000

The point of DMARC reports is to understand what is not authenticating in
an aligned fashion, so that you can get those mailstreams authenticating
properly and verify things are now correct. Spam, nor insight into receiver
mechanisms to combat spam (which change daily, per Brandon), is out of
scope of DMARC reporting.

Seth

-- 

*Seth Blank* | Director, Industry Initiatives
*e:* [email-masked]
*p:* 415.273.8818

This email and all data transmitted with it contains confidential and/or
proprietary information intended solely for the use of individual(s)
authorized to receive it. If you are not an intended and authorized
recipient you are hereby notified of any use, disclosure, copying or
distribution of the information included in this transmission is prohibited
and may be unlawful. Please immediately notify the sender by replying to
this email and then delete it from your system.

---

From: Chris Wedgwood <[email-masked]>
Date: Fri, 25 Oct 2019 10:59:25 -0700

> I can't see how spam scores would be of any use for any of these

i don't think anyone will include those values even if they can/could
for fear of it being used against them

---

From: Dotzero <[email-masked]>
Date: Fri, 25 Oct 2019 14:03:19 -0400

> scope of DMARC reporting.
Absolutely agree with the sentiments expressed by John and Seth.

Michael Hammer

---

From: Alessandro Vesely <[email-masked]>
Date: Fri, 25 Oct 2019 20:35:50 +0200

Hi Dilyan,

On Fri 25/Oct/2019 12:51:43 +0200 Дилян Палаузов wrote:
> 
> I do not see how this helps for DMARC.  An email either validates DMARC, or
> fails DMARC and the aggregate repors say per sending IP server (only direct
> mail flow is reported), whether DMARC validates or fails.  With this
> information it is sufficient to determine, if the DMARC/DKIM implementations
> on sender and receiver are either both bug-free,  or both have the same
> bugs.

Looking at aggregate reports, you cannot tell whether an authentication failure
is a sacrosanct signaling of your domain being abused rather than a legitimate
user going through external forwarders.

> I do not see, how the information you ask to add, while interesting, does
> help DMARC.>
> What is the purposes of the aggregate and non-aggregate reports?  What are
> non-goals?  I asked several times here, nobody answered.  Perhaps a
> discussion on the goals and non-goal would help.

That was probably discussed already.  Now that we have some experience, we can
discuss further.

I know some very acknowledgeable WG participants accumulate aggregate report
values in their own MySQL database (I'm not sure about the details).  Many
people, instead, outsource reports to specialized DMARC analyzers, who display
nice graphical summaries.  I run an XSLT transform of DMARC reports into an
HTML tabular format of one row per record.

In theory, reports can be something more than a debugging aid.  It has the
potential to assemble a community where bad actors are identified and dismissed.

> If it is a goal to reuse the dmarc-reporting mechanism to report also about
> perceived spam probability, then it can be discussed in more details how
> this can be achieved.

Well, spam score usually is hight for phishing too.  To counter phishing is
DMARC core business.

> My experience is, that asking a provider, why an obviously non-spam mail was
> evaluated as spam, virtually never leads to a useful answer.  So nobody
> wants to reveal how its spam system weigths factors and if there is lack of
> such interest, extending the report format will not help, as nobody will be 

This is a problem, indeed.  Large mailbox providers may fear that giving bad
scores to an IP can result in senders complaining against against their
weighting method, requiring more personnel to answer back.  It should be made
clear that reports are given out AS IS, as a favor to senders, without liability.

Anyway, reporting MTAs don't have to reveal the method, just the result.

> Exchanging information on hard-coded rules in Spam-Assassing (IP reputation,
> HTML mime part without text/plain, the “Nigeria money” phrase), that is not
> based on filter training, does not help neither, as sender can run the tests
> on its own and predict how the recipient will evaluate these set of
> criteria.

Changing point of view, perspective also changes.  In addition, by comparing
external scores to internal predictions, one has a chance to evaluate the
goodness of the reporting MTA.

Best
Ale

---

From: "John Levine" <[email-masked]>
Date: 25 Oct 2019 16:13:44 -0400

>user going through external forwarders.

Sure you can, you look at the IP address and see who it is.  In my reports I
see bursts of authentication failures from hosts that are obviously mailing
list servers, and lots of failures in China which are random spambots.

>potential to assemble a community where bad actors are identified and dismissed.

No, that's not what they're for and they don't have the necessary
info.  There are systems that compile data for IP reputation but
that's not what DMARC is.  The point of DMARC is to try to tell "is
this message really from X", not "is this message spam."

R's,
John

---

From: Alessandro Vesely <[email-masked]>
Date: Sat, 26 Oct 2019 12:58:02 +0200

> list servers, and lots of failures in China which are random spambots.

Right, to add a country lookup during XSLT transform is a nice hint.  IP
reputation sites are not quite as handy.

> this message really from X", not "is this message spam."

There are spammers who abuse other domains, and not all of them are phishers.

While hard policies don't seem to be a goal for seldom-abused domains, failed
p=none hardly map to some kind of score.  So, yes, it looks like some necessary
info is missing.  But why would more info hurt?

The biggest obstacle is admins' reluctance to divulge internal data about their
mailflows.  Perhaps we should instead look for something that mail site admins
are eager to communicate to their peers.  I, for one, see many more DMARC
target domains than sources (73 to 13 this morning, most of the latter from my
yesterday's posts to this list).  I guess such ratios are rather common,
possibly worse for large sites.  Correct?

Curiously, in general, people and organizations seem to have a lot to say,
while they are often reluctant to listen.  How come DMARC, as a channel,
induces the opposite behavior?

Just thinking aloud
Ale
--

---

From: Dotzero <[email-masked]>
Date: Sat, 26 Oct 2019 09:32:06 -0400

> Well, spam score usually is hight for phishing too.  To counter phishing is

Absolutely wrong. DMARC does one thing and one thing only - mitigate direct
domain abuse. Bad guys can and have switched to using cousin domains,
homoglyphs and other approaches to engage in spam, phishing and other
abuse. One thing I think we will see more of over time is system compromise
to enable sending of badness as a particular domain. This is one reason I'm
not a huge fan of reputation systems and whitelisting in particular.
Reputation in practice essentially reduces to "What have you done to me
today".

Michael Hammer

---

From: Alessandro Vesely <[email-masked]>
Date: Sun, 27 Oct 2019 11:12:03 +0100

On Sat 26/Oct/2019 15:32:06 +0200 Dotzero wrote:
> abuse.

I conceive that as depicted.  Is it the agreed upon diagram?

Ale