Subject: [apps-discuss] I-D Action: draft-ietf-appsawg-malformed-mail-06.txt

From: [email-masked]
Date: Tue, 18 Jun 2013 00:23:01 -0700

A New Internet-Draft is available from the on-line Internet-Drafts directories.
 This draft is a work item of the Applications Area Working Group Working Group of the IETF.

	Title           : Advice for Safe Handling of Malformed Messages
	Author(s)       : Murray S. Kucherawy
                          Gregory N. Shapiro
	Filename        : draft-ietf-appsawg-malformed-mail-06.txt
	Pages           : 19
	Date            : 2013-06-18

Abstract:
   Although Internet mail formats have been precisely defined since the
   1970s, authoring and handling software often show only mild
   conformance to the specifications.  The distributed and non-
   interactive nature of email has often prompted adjustments to
   receiving software, to handle these variations, rather than trying to
   gain better conformance by senders, since the receiving operator is
   primarily driven by complaining recipient users and has no authority
   over the sending side of the system.  Processing with such
   flexibility comes at some cost, since mail software is faced with
   decisions about whether or not to permit non-conforming messages to
   continue toward their destinations unaltered, adjust them to conform
   (possibly at the cost of losing some of the original message), or
   outright rejecting them.

   A core requirement for interoperability is that both sides of an
   exchange work from the same details and semantics.  By having
   receivers be flexible, beyond the specifications, there can be -- and
   often has been -- a good chance that a message will not be fully
   interoperable.  Worse, a well-established pattern of tolerance for
   variations can sometimes be used as an attack vector.

   This document includes a collection of the best advice available
   regarding a variety of common malformed mail situations, to be used
   as implementation guidance.  It must be emphasized, however, that the
   intent of this document is not to standardize malformations or
   otherwise encourage their proliferation.  The messages are manifestly
   malformed, and the code and culture that generates them needs to be
   fixed.  Therefore, these messages should be rejected outright if at
   all possible.  Nevertheless, many malformed messages from otherwise
   legitimate senders are in circulation and will be for some time, and,
   unfortunately, commercial reality shows that we cannot always simply
   reject or discard them.  Accordingly, this document presents
   alternatives for dealing with them in ways that seem to do the least
   additional harm until the infrastructure is tightened up to match the
   standards.

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-appsawg-malformed-mail

There's also a htmlized version available at:
http://tools.ietf.org/html/draft-ietf-appsawg-malformed-mail-06

A diff from the previous version is available at:
http://www.ietf.org/rfcdiff?url2=draft-ietf-appsawg-malformed-mail-06

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 18 Jun 2013 00:27:08 -0700

On Tue, Jun 18, 2013 at 12:23 AM, <[email-masked]> wrote:

> [...]

Includes changes suggested by John and Ned's recent conversation about the
content.

Anyone else interested in providing feedback on it?  Should I poke
Salvatore to start the machine?

-MSK

---

From: Dave Cridland <[email-masked]>
Date: Tue, 18 Jun 2013 09:56:36 +0100

> Salvatore to start the machine?

§6, last para should explicitly state this is only a suitable fixup for
text/plain and text/html; and even then only in known character encodings
(charsets), such as ASCII, ISO 8859, and UTF-8. I'd hate everyone to
"correct" the 0x0A octets in my JPEGs.

§6 title is misnamed - missing "i" in "terminaton".

I'm concerned with §7.1.4's implication that implementations should be
looking for possibly email addresses in header field comments. I'm happy
that a bare ")" should be treated as a literal, but I'm wary of the
possibility of "hiding" email addresses in comments. Overall, I'm inclined
to say I'd prefer that we advised that an unterminated comment in a header
field value were treated as being terminated at the end of the value.

I have similar concerns with $7.1.6 - more so because I suspect that "Foo
[email-masked] could also be interpreted as "Foo local"@example.net.
Also typo in §7.1.6, example has "&gt" at the end.

I don't think you can show valid syntax and describe it as "Header
Malformations", as §7.4. I think this might be better described as
"Non-normalized Header".

§7.5 mentioned the case of two From fields - it doesn't mention the
possibility of combining the two, so:

From: First <[email-masked]>
From: Second <[email-masked]>

... could become:

From: First <[email-masked]>, Second <[email-masked]>

Obviously you'd need to inject a Sender field as needed (as per §7.6 of
this memo).

I'm not suggesting it has to offer this, but I'm wondering if the lack of
offer is intentional?

§9.1 describes a technique of breaking lines at points that make no
semantic difference. Given the poor NLP capability in the majority of
deployed MTAs, I'm slightly suspect of this advice. Even without any snark
at all, cases such as format=flowed (and other message quoting) could
easily be warped or broken by breaking lines.

Hope that all helps more than hinders,

Dave.

---

From: Ned Freed <[email-masked]>
Date: Tue, 18 Jun 2013 06:10:07 -0700 (PDT)

> "correct" the 0x0A octets in my JPEGs.

Disagree. This section is talking about processing messages as a whole
as received via SMTP, prior to MIME interpretation.

What needs to be made clear is that this is talking about SMTP specificially,
not interpretation of MIME entities. And it probably should note that
it doesn't apply to binary SMTP.

Similar considerations do apply to 7bit and 8bit entities and types under text,
but this section isn't written to cover that. And covering all the permutations
of entity type/encoding/contnent mismatch is a pretty tall order. Do we really
want to get into this so late in the game/

> field value were treated as being terminated at the end of the value.

I'm ambivalent about this one. It's quite true that addresses are sometimes
hidden in comments, but as a practical matter, anyone who does that as opposed
to removing the address outright and then gets the syntax wrong kinda deserves
what they get.

> [email-masked] could also be interpreted as "Foo local"@example.net.

Well, of the three, which do you think is more likely to have been the
intent?

       To: "Joe <[email-masked]>"@example.net
       To: "Joe" <[email-masked]>
       To: "Joe joe"@example.net

Yes, they're all legal, but in practice how often do you see local parts in
valid addresses containing angle brackets and spaces?

> I'm not suggesting it has to offer this, but I'm wondering if the lack of

I hadn't considered this. It's kind of an interesting idea, but given the
issues that arise in practice when more than one address is present in a From:
field, I have doubts about its utility.

> easily be warped or broken by breaking lines.

Strongly disagree. The alternative of encoding oversized lines may not be
practical, so the choices are truncate or wrap. The chances that truncation
is going to produce better results than wrapping are pretty low.

And I'm not even slightly sympathetic about the possible damage done due to
lack of NLP knowledge in MTAs. If you want your format=flowed material to
arrive intact, don't spew out illegal garbage.

				Ned

---

From: Dave Cridland <[email-masked]>
Date: Tue, 18 Jun 2013 14:54:47 +0100

First off, I should say this was my first run-through, and I've not been
following the discussion - if things have been discussed to death already,
then feel free to shoot me down.

> it doesn't apply to binary SMTP.
Right, I'd misunderstood the notion. So yes, it was the paragraph before
derailling me, because it starts discussing ASCII. I think if it merely
said that bare CRs and LFs were uncommon in non-binary messages, we'd be
good here.

   Within modern Internet Mail it is highly unlikely that an isolated CR
   or LF is valid in a message, outside of binary transfers [BINARY].
 Furthermore [MIME] presents
   mechanisms for encoding content that actually does need to contain
   such an unusual character sequence.

> to removing the address outright and then gets the syntax wrong kinda
What's concerning me here is that I wouldn't want to be trying to parse
comments in the hope of finding things there. I'd rather assume that the
error with an unterminated comment was the lack of termination, not that it
might contain something useful I should go hunting for.

> valid addresses containing angle brackets and spaces?
Bear in mind I worked for Isode for years, so I've built up a strong
immunity to local parts with spaces due to MIXER, but in any case the
middle one is likely to be the intent. However I'm again concerned that it
means finding an unterminated phrase and back-tracking to see if the phrase
"looks interesting".

> field, I have doubts about its utility.
Yes, multiple addresses in a From is unusual; but I'd have thought that
collapsing multiple header fields into one would in general be sane?
(Another example might be concatenating two subject header fields into one
longer one).

> arrive intact, don't spew out illegal garbage.

Happy to let your strong disagreement trump my slight suspicion.

Dave.

---

From: Ned Freed <[email-masked]>
Date: Tue, 18 Jun 2013 07:37:16 -0700 (PDT)

> might contain something useful I should go hunting for.

The problem is that assumption leaves you with nothing actionable in the  field
in most if not all cases. (There's still the question of what to do if you hit
a comma - does that terminate the comment?) Since the intent of inserting
the field likely was, with the possible exception of a Bcc:, to actually
communicate one or more addresses, this doesn't seem like a good
interpretation.

> means finding an unterminated phrase and back-tracking to see if the phrase

Quite true. Some of this is undeniably tricky to code, and the document
doesn't really get into all of the issues with actually implementing some
of this stuff.

Nevertheless, I think this is, on balance, reasonable advice for this case.

> collapsing multiple header fields into one would in general be sane?

Absolutely. I'm just concerned with subsequent handling. But the more
I think about it the more I like this appraoch.

> (Another example might be concatenating two subject header fields into one

Also a good idea, although I'd recommend checking to see if they aren't the
same value before doing it.

I guess the meta-question here is whether or not this document is open to
new material at this point. I'd actually like to see both of these added if
it is.

				Ned

---

From: Ned Freed <[email-masked]>
Date: Tue, 18 Jun 2013 07:45:46 -0700 (PDT)

And if we're going to add new material, all this discussion of how to
handle other cases of repeated fields reminded me of another one: Return-path.

I see multiple return-paths in messages pretty regularly. In most cases
the values are the same, but sometimes they are different.

IMO the correct interpretation is that the first one wins since that's most
likely to actually be the MAIL FROM inserted at final delivery. However, we
might want to mention that delivery agents could check to see if there's
already a return-path present with the same value as the one they're about to
insert, and if there is, don't insert an additional one.

				Ned

---

From: Franck Martin <[email-masked]>
Date: Tue, 18 Jun 2013 10:43:03 -0500 (CDT)

----- Original Message -----

> From: "Dave Cridland" <[email-masked]>
> To: "Murray S. Kucherawy" <[email-masked]>
> Cc: "IETF Apps Discuss" <[email-masked]>
> Sent: Tuesday, June 18, 2013 1:56:36 AM
> Subject: Re: [apps-discuss] I-D Action:
> draft-ietf-appsawg-malformed-mail-06.txt

> On Tue, Jun 18, 2013 at 8:27 AM, Murray S. Kucherawy < [email-masked] >
> wrote:

> > Anyone else interested in providing feedback on it? Should I poke Salvatore
> > to start the machine?
> 

> From: First < [email-masked] >
> From: Second < [email-masked] >

> From: First < [email-masked] >, Second < [email-masked] >

> Obviously you'd need to inject a Sender field as needed (as per §7.6 of this
> memo).

> I'm not suggesting it has to offer this, but I'm wondering if the lack of

We should not do that, having more than one From header makes the message non valid (as per RFC). There must be one and one only From: header. Combining makes an invalid message become a valid message. It is not like there is ambiguity on the meaning. The only option here is to reject the email so that the sender can correct its implementation. At least that's my point of view.

---

From: Alessandro Vesely <[email-masked]>
Date: Wed, 19 Jun 2013 09:51:45 +0200

> insert, and if there is, don't insert an additional one.

The abstract and the introduction let the reader guess that the
document is talking about modules of an MTA that strive to make sense
of a message received via SMTP.  However, "receivers" could also mean
IMAP clients downloading from a mailbox.  Some fields ought to be
treated differently in that case.

On SMTP reception, renaming Return-Path to Old-Return-Path is, IMHO, a
safe option.  The "true" Return-Path will be added by the MDA later
on.  On retrieval dices have rolled, and the above heuristic is all
that's left.

And how about Delivered-To?  Some MTAs use it to detect circular
dot-forward recipes, so its presence in the original message can be
disruptive --in some circumstances it can be an attack vector to force
NDNs.

---

From: Ned Freed <[email-masked]>
Date: Wed, 19 Jun 2013 06:49:16 -0700 (PDT)

> treated differently in that case.

It's really talking about message interpretations at various stages,
not necessarily message modifications. In that view, the thing to say,
assuming we say anything, is only look at the top one.

> on.  On retrieval dices have rolled, and the above heuristic is all

The underlying point being that return-path is only supposed to appear
later.

Thinking about it more, the best thing to do on submission is probably
delete the field outright. But this document isn't about submit fixups, so
that's really out of scope.

> disruptive --in some circumstances it can be an attack vector to force

I don't think we should make any recommendations for nonstandard fields.

				Ned

---

From: Alessandro Vesely <[email-masked]>
Date: Thu, 20 Jun 2013 11:45:01 +0200

On Wed 19/Jun/2013 15:49:16 +0200 Ned Freed wrote:
>> IMAP clients downloading from a mailbox.
> 
> [...] Thinking about it more, the best thing to do on submission is
> probably delete the field outright. But this document isn't about
> submit fixups, so that's really out of scope.

Agreed, submission is yet another stage where a message may get
interpreted and possibly modified.  Even if it's out of scope, Section
5 is not conceived as part of an attempt to enumerate those stages,
possibly summarizing, for each one, the pros and cons of committing an
interpretation vs keeping it private to the module that carried it
out.  In that sense, it may seem to contradict Section 4.

> I don't think we should make any recommendations for nonstandard fields.

Correct.  Do you know if there is any good reason why Delivered-To was
never registered?