Subject: [dmarc-ietf] Lenient DKIM (new Internet Draft)

From: Rick van Rein <[email-masked]>
Date: Mon, 25 Sep 2017 12:01:42 +0200

Hello DKIM/DMARC list,

I would like to propose a method for "fixing" the real-world problems
that currently break DKIM.  I am aware of the work on ARC, but in
comparison this appriach:

 1. is much simpler, with a simple cryptographic foundation
 2. works without explicit support from message-modifying intermediates
 3. locates message parts that have changed [within practical bounds]
 4. allows training on acceptable changes for given senders/intermediates
 5. pins down rogue senders/intermediates more accurately

The idea isn't completely worked out yet, but the approach should be
crystal clear from the current text.  I believe that this leads to
better email user experiences than possible with ARC.

I am looking forward to your responses.  Please keep me in Cc: if possible?

Best wishes,

Rick van Rein
ARPA2.net / InternetWide.org
> *From:* [email-masked]
> *Date:* 25 September 2017 at 11:49
> *To:* "Rick van Rein" <[email-masked]>
> *Subject:* New Version Notification for draft-vanrein-dkim-lenient-00.txt
> A new version of I-D, draft-vanrein-dkim-lenient-00.txt
> has been successfully submitted by Rick van Rein and posted to the
> IETF repository.
>
> Name:		draft-vanrein-dkim-lenient
> Revision:	00
> Title:		Lenient DKIM
> Document date:	2017-09-25
> Group:		Individual Submission
> Pages:		7
> URL:            https://www.ietf.org/internet-drafts/draft-vanrein-dkim-lenient-00.txt
> Status:         https://datatracker.ietf.org/doc/draft-vanrein-dkim-lenient/
> Htmlized:       https://tools.ietf.org/html/draft-vanrein-dkim-lenient-00
> Htmlized:       https://datatracker.ietf.org/doc/html/draft-vanrein-dkim-lenient-00
>
>
> Abstract:
>    DKIM is a framework for signed messages, especially for email.  While
>    in transit, changes are sometimes made, and these break the DKIM-
>    Signature.  This specification makes DKIM more lenient, without
>    changes to its core.  It adds leniency towards MIME body rewrites,
>    removal of alternatives and annotation with bits of text.  The
>    intention is to allow these changes such that they can be clearly
>    shown to the user, while indicating that the remainder of the
>    signedmessage is still in tact.
>
>                                                                                   
>
>
> Please note that it may take a couple of minutes from the time of submission
> until the htmlized version and diff are available at tools.ietf.org.
>
> The IETF Secretariat
>

---

From: Brandon Long <[email-masked]>
Date: Mon, 25 Sep 2017 09:00:30 -0700

This is an interesting sketch, but it doesn't  include the hard parts, so
it's hard to know if it can provide what is needed.  For example, will the
rolling hash allow for exact boundaries of changes to be determined?
Subjects are pretty short, how does a rolling hash handle that?  I'm not
familiar with rolling hashes, so perhaps my questions are naive.

The mime body canonicalization is odd, what percentage of mail isn't
multipart at this point?  It also requires that mime canonicalization
requires impl of this other spec, which seems more v2ish, though maybe not
in formatting.

I'd say the mime header canonicalization doesn't go far enough.  For
example, most mailing lists that are i18n aware are going to decode rfc2047
subjects before adding a subject prefix, and then reencode, which may not
be in the same charset, especially if the prefix and subject are in
different languages.  Another issue we see is in optional things, such as
quoting in address and parameter headers.  Defining a canonical form for
those headers might be necessary to accomplish what you want.  There's also
smtputf8 downgrades, which would imply taking a utf8 nonencoded subject and
encoding it.  It's also useful if your system uses a non rfc822 format
internally and only gateways to the internet.

I think a proper mime canonicalization would be useful on it's own, we've
talked about it before internally here at Google.

I believe that we discussed something similar to this early on this list,
but the challenge is in the details.

Brandon

On Sep 25, 2017 3:02 AM, "Rick van Rein" <[email-masked]> wrote:

> _______________________________________________
> dmarc mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dmarc
>

---

From: Stan Kalisch <[email-masked]>
Date: Mon, 25 Sep 2017 14:19:22 -0400

> On Sep 25, 2017, at 12:00 PM, Brandon Long <[email-masked]> wrote:
> 
> 
> The mime body canonicalization is odd, what percentage of mail isn't multipart at this point?

iOS Mail (before 11, at least) does 7-bit text/plain mail by default.  (If memory serves, this has something to do with Japanese SMS providers, although Japanese obviously isn't 7-bit.)  I don't know enough to hazard a guess at what percentage of mail this is.

Stan

---

From: Stan Kalisch <[email-masked]>
Date: Mon, 25 Sep 2017 22:17:05 -0400

I apologize for this post.  I was in a car accident, and it's affecting me more than I thought—you're obviously talking about MIME *bodies*.  Again, my apologies to the list.

Stan

> On Sep 25, 2017, at 2:19 PM, Stan Kalisch <[email-masked]> wrote:
> 
>

---

From: Rick van Rein <[email-masked]>
Date: Tue, 26 Sep 2017 16:57:07 +0200

Hi,

Thanks to Brandon for reading, and raking up the past :)

It does sound like the rolling checksum is a new idea, so let me explain
why I think it is useful, and note that you may know it from RSync.  It
helps in efficiently detecting the original text within an extended new
text.

If we have an old text (body or header: size doesn't matter) of size N,
and a new one of size N+M, then computing checksums or hashes over all
possible embeddings of the original in the new text would normally take
O(N*M) elementary computations (such as adding a character to a
checksum).  With a rolling hash, you compute a checksum over the first N
characters, and roll forward by one position in two elementary
computations: addition of the next character and removing the one at the
beginning; the complexity of searching for the text is now O(N+M), so in
the order of the total text size.  This is used as a hint that it would
be useful to attempt a secure hash algorithm on the portion of a text
that matches the checksum.

This explains why the proposed DKIM-Signed-Content header holds:
 - the location of the text (body, header, or something nested)
 - the size N of the signed/original text
 - the outcome of a rolling checksum (perhaps 32 bits)
 - the outcome of a secure hash (to check after the hint)

> will the rolling hash allow for exact boundaries of changes to be
> determined?  Subjects are pretty short, how does a rolling hash handle
> that?

No problems there.  The limitation is that texts may be added before and
after the original text, but nothing can be stripped.
> multipart at this point?

Interesting.  I though that it was quite sensible to reduce the
MIME-representation of binary content to its original binary form before
signing.

The idea of multipart, which I may need to describe more accurately, is
that the individual body parts get their own DKIM-Signature, and are
then combined (following the MIME hierarchy) to form the overall
DKIM-Signature for the message.  By including Content-ID and Message-ID
logically, it is possible to always distinguish between a body part and
the full email.

> It also requires that mime canonicalization requires impl of this
> other spec, which seems more v2ish, though maybe not in formatting.
>
There is no reason why one couldn't sign separately with relaxed/relaxed
and mime/mime canonicalisation.  This would allow for a transition
period.  Even with just mime/mime, there is currently no punishment (but
also no benefit) for mail parties that don't recognise the
DKIM-Signature due to the new canonicalisation algorithm.

The reason why this may not be so v2 as you suggest is that the
dependency on support by intermediate mail facilities is replaced by a
dependency on support in end points.  Note that I'm not talking software
support, but actual deployment.

> I'd say the mime header canonicalization doesn't go far enough.

Thank you, I tend to agree.  I had considered character sets as mere
interpretations of binary content, but was unsure if they were actually
rewritten in passing.

> For example, most mailing lists that are i18n aware are going to
> decode rfc2047 subjects before adding a subject prefix, and then
> reencode, which may not be in the same charset, especially if the
> prefix and subject are in different languages.

A starting charset may be recognised of course, and the number of
original forms would be very limited and may be iterable?  This raises
concerns of potential loss of information due to incomplete / incorrect
rewrites, however.

> Another issue we see is in optional things, such as quoting in address
> and parameter headers.  Defining a canonical form for those headers
> might be necessary to accomplish what you want.

This would be specific to the MIME-headers, I suppose.  Yes, a canonical
form would then be a useful addition to the MIME header canonicalisation
prescription.  The list of headers in use there is limited, so it would
be some work desscribing it, but not impossible, as far as I can tell. 
A predefined order for parameters, always using quotes, only escape when
it is required, never mentioning default values, that sort of thing.

> There's also smtputf8 downgrades, which would imply taking a utf8
> nonencoded subject and encoding it.  It's also useful if your system
> uses a non rfc822 format internally and only gateways to the internet.
These are indeed pesky corner cases :-S so thanks for pointing them out
right away.

I am tempted to think that a limited number of alternative
representations for the Subject are possible, and could be iterated over
to recognise the header properly.  Always mapping characters to the
largest set spanning the current use case could help; that would be
possible in the before and after case and lead to the same canonical
format.  [I may however be naive about i18n and especially what overlap
character sets have.  Please tell me if that is the case.]

> I think a proper mime canonicalization would be useful on it's own,
> we've talked about it before internally here at Google.

Thanks.

> I believe that we discussed something similar to this early on this
> list, but the challenge is in the details.

You certainly convinced me of that.

If you have anything to add to the former, then by all means let me
know.  It is extremely useful to know what I'm heading for when deciding
to get into this or not.

Also, if the list thinks this is not a fruitful endeavour, then I
welcome further critiques :)

Thanks,
 -Rick

---

From: "John Levine" <[email-masked]>
Date: 26 Sep 2017 22:58:56 -0000

>I am looking forward to your responses.  Please keep me in Cc: if possible?

I hate to be totally negative, but this draft revives a lot of things
that we considered and rejected when we did DKIM.

Marking content in an MUA is a WKBI*.  There is no reason to believe
that users would understand content marking or would make reasonable
decisions based on it.  As a general rule, anything that puts security
policy in the hands of end users doesn't work.  Think of all the
browser bad SSL cert warnings you've clicked through.

Also, there are more ways to change content that anyone can describe.
Some of the harder to describe are recoding between 7 and 8 bit and
base64, reducing the size and resolution of images (common on phones)
and reordering MIME parts.

Finally, it is pretty clear from the ARC work that big mail systems
are more interested in telling recipient systems the identities of the
parties that handled a message than how it changed or whether any of
those parties thought the changes were a good idea.

For another rejected approach see my DKIM conditional signatures, which
let senders authorize intermediaries to modify and resign messages.

https://datatracker.ietf.org/doc/draft-levine-dkim-conditional/

R's,
John

* - Well Known Bad Idea

---

From: Brandon Long <[email-masked]>
Date: Thu, 28 Sep 2017 13:18:25 -0700

> browser bad SSL cert warnings you've clicked through.

Knowing what changed is actually something that could go into a spam filter
decision, however.

It may be more complicated than arc.  And we did have an example proposal
that foundered on the deletion case.

https://www.ietf.org/mail-archive/web/dmarc/current/msg01444.html

One could also make a spam/phishing decision and show the original, forcing
the user to click to see the modified version.  In most mailing list cases,
the subject/footer is mostly barely useful (quick, does this mailing list
have a footer?).  Ie, one could spam score both the original and the
modified version, and make choices based on the difference.

> and reordering MIME parts.

There is an obvious question of "handles everything" vs "handles the
99-percentile case better".

Ie, a mechanism that explicitly handled subject prefixes and footers might
do a better job than ARC, it might have better adoption and fewer
deployment headaches, even as it allows some set of modifications to fail.
It might also be harder to

Also, among what you're talking about, I think the 7/8/base64 stuff would
be covered by his MIME canonicalization.  Although I've seen web proxies or
clients which resize photos, I don't think I've ever seen an MTA do it...
which doesn't mean they couldn't, but I'm not sure it's a use case we have
to try and handle.

Ie, I would have been happy to have changed Google Groups to allow DKIM
signed mail to pass through, but we feel that current anti-spam laws mean
we have to include an unsub link.  We would have routed out any of the
other types of changes you're talking about because DKIM passing to get the
mail through is more important.

> those parties thought the changes were a good idea.

Google was the one with the previous diff based proposal, I would have been
happy to avoid the "who" question if we could have
done something more exact.

Brandon

---

From: Rick van Rein <[email-masked]>
Date: Fri, 29 Sep 2017 07:53:50 +0200

Hello,

Thanks to John for reading the draft proposal.

> decisions based on it.

Hmm.  The idea was founded on the common reliance of yellow/green bars
to indicate website safety.  I was not aware that it was considered
WKBI; is there a place where I can learn what others have (so
conclusively) said on the matter?

This takes away one of two reasons to learn through rolling checksums
about what is original and what has changed; automated content judgement
can still benefit from seeing "that same ol' change" be made.

> As a general rule, anything that puts security

I like the control, but I agree that this is not for everyone.

> and reordering MIME parts.

This is the sort of thing I was pitching for with this draft.  Not
everything is so trivial that I would say they need no DKIM or ARC
re-signing.

 * 7/8 bit and base64: I think this can be covered by mapping most body
parts to their binary form; and by canonicalising text/* specially by
mapping it to Unicode in a UTF-8 representation.  This misses out on
mapping between ß and ss, but I am willing to accept that
language-specifics are changes to content that require DKIM re-signing.

 * changes to size and resolution constitute change to the content and
would require DKIM re-signing.  I'd suspect this kind of transformation
to be done near the recipient, so explicit trust in the transforming
party may be setup explicitly?

 * reordering of MIME parts in a multipart/* would be recognised by this
mechanism, because they are independently signed and listed as
sub-parts.  MUAs (or MTA spam filters) may judge that they like this, or
not.

> those parties thought the changes were a good idea.

The ARC draft states clearly that it is difficult to pin down the
culprit if the content does not validate.  This problem does not exist
in my proposal.

It is clear that ARC addresses large mail providers, but it almost seems
like only those are on its mind: parsing ARC is complex, especially
because it involves taking charge over message in reputation
intermediate systems.  When such intermediates check (for spam, say)
less accurately than the end systems on which they land, these
intermediates will be discredited and effectively pushed off the Internet.

I also should add that the cryptographic design of ARC isn't clear to
me.  I pointed out to the ARC authors that there is a lot of *how* but
little or no *why* to underpin what feels to me like a rather complex
set of computations, given its intentions.  After reading the ARC draft,
my crypto-savvy head lacks the usual clear understanding of what is
needed to do ARC well, and that worries me.

> https://datatracker.ietf.org/doc/draft-levine-dkim-conditional/
This proposal is technically different, but maybe you are thinking of
the comments on it?  Please note that the proposed DKIM-Signed-Content
is a helpful extra field, not a mandated extension.

Thanks,
 -Rick

---

From: Rick van Rein <[email-masked]>
Date: Fri, 29 Sep 2017 08:22:41 +0200

Hi,

> Also, among what you're talking about, I think the 7/8/base64 stuff
> would be covered by his MIME canonicalization.

Since most MIME content is binary, it would be helpfulto canonicalise 
individual body parts to their binary/pristine form.  My current
thinking is that even message/* is binary, and I can think of reasons
why message/rfc822 should not be mangled in transit, but I'm not sure if
this is realistic.

Where MIME content is text/*, I had to learn more about i18n.  Unicode
was made to solve these riddles, by offering a place to all the
characters in the world [but not their renderings].  For characters with
more than one code, Unicode has the notion of canonical equivalence. 
And where characters may be decomposed into sequences, it has a more
lenient equivalence notion of compatibility.  The former should be no
issue for DKIM, the latter may be.  Defining equivalence through
compatibility works for human texts but may be dangerous for accurate
content like program fragments and URIs, except that the common
limitation of these uses to ASCII solves these dangers.  The stringprep
profiles, including SASLprep which looks very email-suitable, would be
able to handle all this.  Software support already exists, of course.

Finally, the nested nature of MIME content allows the recognition of
failures.  All that needs to be done is add a DKIM-Signature to the body
parts, and have a mechanism for describing their compositions in a
change-detectable manner (and sign for that mechanism too).

> Although I've seen web proxies or clients which resize photos, I don't
> think I've ever seen an MTA do it... which doesn't mean they couldn't,
> but I'm not sure it's a use case we have to try and handle.

Anything of this kind would know that delivery is to a particular class
of MUA, so it would be near the recipient, where trust has a different
organisation.

-Rick

---

From: "John R Levine" <[email-masked]>
Date: 29 Sep 2017 08:23:13 -0700

> conclusively) said on the matter?

But web sites don't color parts of a page green and parts red.  There 
might be something in a SOUPS proceeding, and I know this came up multiple 
times back when we were rejecting more complex canonicalizations.  The 
relevant question is more like, given that users do a bad job of using a 
single bit of "this web site is dangerous", why is there any reason to 
think they'd do even as well with more complex warnings.

> intermediates will be discredited and effectively pushed off the Internet.

You say that like it's a bad thing.  If your mailing list is so poorly 
configured that it gushes spam, I don't want its mail.

R's,
John

---

From: Scott Kitterman <[email-masked]>
Date: Fri, 29 Sep 2017 11:56:19 -0400

> sub-parts.  MUAs (or MTA spam filters) may judge that they like this, or

Making DKIM signing MIME aware was specifically rejected during DKIM 
development due to implementation complexity.  It didn't magically get easier 
in the mean time.  For the implementation I help maintain, dkimpy, large 
portions of the existing code would have to be thrown out and re-written 
(integrating ARC into the code base was relatively minor compared to this).

If you've made it MIME aware, I think you've changed it enough it's no longer 
DKIM and you should call it something else (and I'm pretty sure I'd never 
implement it).

Scott K

---

From: Elizabeth Zwicky <[email-masked]>
Date: Fri, 29 Sep 2017 09:04:36 -0700

http://www.usablesecurity.org/emperor/

Is the most classic paper on the complete uselessness of icons. You’ll note that browsers have changed to putting up intrusive pages with unobtrusive ways to continue, among other options, instead of showing broken lock icons. 

Elizabeth 

[email-masked]

> On Sep 28, 2017, at 11:22 PM, Rick van Rein <[email-masked]> wrote:
> 
> Hi,
>

---

From: "Kurt Andersen (b)" <[email-masked]>
Date: Fri, 29 Sep 2017 15:19:25 -0700

On Thu, Sep 28, 2017 at 10:53 PM, Rick van Rein <[email-masked]>
wrote:

> set of computations, given its intentions.

Your request for better clarity on this point has not been ignored, but we
are still working on effective ways to express both the intentions and the
design.

--Kurt

---

From: Rick van Rein <[email-masked]>
Date: Sat, 30 Sep 2017 21:58:33 +0200

Hi,

> http://www.usablesecurity.org/emperor/
Thanks for this horror story.  Mind-numbing, but useful work.

I shall remove the reason of "security painting" by MUAs from the text. 
What remains is automated content processing (spam filtering) which
still appears to be a sufficiently good reason on its own.  It still
appears to me that most of the difficult cases can be remedied in the
proposed [but currently somewhat undefined] c=mime/mime specification.

Thanks,
 -Rick

---

From: "John R Levine" <[email-masked]>
Date: 30 Sep 2017 16:07:31 -0400

> proposed [but currently somewhat undefined] c=mime/mime specification.

This really needs some prototype implementations to see if it's 
technically workable.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly