Subject: Re: [apps-discuss] Review of: draft-ietf-appsawg-malformed-mail-03

From: "Murray S. Kucherawy" <[email-masked]>
Date: Sun, 5 May 2013 23:42:16 -0700

Thanks, Dave.  I'm waiting for my co-author to get back to me on two points
in your review that I can't really answer, and then we'll at long last post
a new version for the WG to review.  I'll try to solicit a couple more
reviews and then suggest Salvatore start WGLC on it.

-MSK, hatless this time

On Wed, Apr 10, 2013 at 8:00 AM, Dave Crocker <[email-masked]> wrote:

>
> Review of:    Advice for Safe Handling of Malformed Messages
>
> I-D:          draft-ietf-appsawg-malformed-**mail-03
>
> Reviewer:     D. Crocker
>
> Review Date:  10 April 2013
>
>
> Summary:
>
>       Internet Mail has always been marked by an unfortunate degree of
> regular and permitted non-conformance to its formal specifications.  The
> current draft seeks to categorize and discuss common types of
> non-conformance and to provide some guidance for how it should be handled.
>  The document is explicit in stating that it does not have the goal of
> standardizing this guidance.
>
>      The document is reasonably clear and complete. I believe a document
> like this can provide very helpful guidance for email developers and
> operators.  It would be useful in its current form, but could greatly
> benefit from some modification.
>
>      One major concern, which is easily remedied, is the draft's use of
> normative language.  The document is often unusually careful to use
> qualifying language that precisely limits the scope of the normative text
> to "a module compliant with this memo".  However I think this is too subtle
> for most readers and that the use of normative language defeats the stated
> limitation of not wanting to define a standard. Hence I changing all such
> language and, instead, using language that is clearly only modest "advice",
> such as with:
>
>    *  a common handling is...
>    *  it is best to...
>    *  it will typically be safe and helpful to...
>
> and so on.
>
>
>
> Detailed Comments:
>
>
>  Abstract
>>
>>    The email ecosystem has long had a very permissive set of common
>>    processing rules in place, despite increasingly rigid standards
>>    governing its components, ostensibly to improve the user experience.
>>
>
>      Although Internet mail formats have been precisely defined since the
> 1970s, authoring and handling software often show only mild conformance to
> the specifications.  The distributed and non-interactive nature of email
> has often prompted adjustments to receiving software, to handle these
> variations, rather than trying to gain better conformance by senders, since
> the receiving operator is primarily driven by complaining recipient users
> and has no authority over the sending side of the system.
>
>
>     The handling of these come at some cost, and various components are
>>
>
>      Processing with such flexibility comes at some cost, since mail
> software is faced with...
>
>
>     faced with decisions about whether or not to permit non-conforming
>>    messages to continue toward their destinations unaltered, adjust them
>>    to conform (possibly at the cost of losing some of the original
>>    message), or outright rejecting them.
>>
>
>      A core requirement for interoperability is that both sides to an
> exchange work from the same details and semantics.  By having receivers be
> flexible, beyond the specifications, there can -- and often has been -- a
> good chance that a message will not be fully interoperable.  Worse, a
> well-established pattern of tolerance for variations can sometimes be used
> as an attack vector.
>
>
>     This document includes a collection of the best advice available
>>    regarding a variety of common malformed mail situations, to be used
>>    as implementation guidance.  It must be emphasized, however, that the
>>    intent of this document is not to standardize malformations or
>>    otherwise encourage their proliferation.  The messages are manifestly
>>    malformed, and the code and culture that generates them needs to be
>>    fixed.  Therefore, these messages should be rejected outright if at
>>    all possible.  Nevertheless, many malformed messages from otherwise
>>    legitimate senders are in circulation and will be for some time, and,
>>    unfortunately, commercial reality shows that we cannot always simply
>>    reject or discard them.  Accordingly, this document presents
>>    alternatives for dealing with them in ways that seem to do the least
>>    additional harm until the infrastructure is tightened up to match the
>>    standards.
>>
>
>
>
>> 1.  Introduction
>>
>> 1.1.  The Purpose Of This Work
>>
>>    The history of email standards, going back to [RFC822] and beyond,
>>
>
> { here I actually suggest citing RFC 733, since it managed to establish
> the solid foundation, with 822 being a relatively small modification. 733
> was not the first formal standard, but the first had poor adoption. /d}
>
>
>     contains a fairly rigid evolution of specifications.  But
>>    implementations within that culture have also long had an
>>    undercurrent known formally as the robustness principle, but also
>>    known informally as Postel's Law: "Be conservative in what you do, be
>>    liberal in what you accept from others."
>>
>
>     Jon Postel's directive is often misinterpreted to mean that any
> deviance from a specification is acceptable.  Rather, it was intended only
> to account for legitimate variations in interpretation /within
> specifications, as well as basic transit errors, like bit errors.  Taken to
> its unintended extreme, excessive tolerance would imply that there are no
> limits to the liberties that a sender might take, while presuming a burden
> on a receiver to "correctly" guess at the meaning of any such variation.
>
> {BTW, I believe Postel's Law was not the motivating reason for email
> format deviations.  Rather, I think that receiver's were accountable to
> their users -- the recipients -- while having no control over the
> misbehaving senders.  So they/we hacked receiving code when necessary, to
> appease the users. /d }
>
>
>     In general, this served the email ecosystem well by allowing a few
>>    errors in implementations without obstructing participation in the
>>    game.  The proverbial bar was set low.  However, as we have evolved
>>    into the current era, some of these lenient stances have begun to
>>    expose opportunities that can be exploited by malefactors.  Various
>>    email-based applications rely on strong application of these
>>    standards for simple security checks, while the very basic building
>>    blocks of that infrastructure, intending to be robust, fail utterly
>>    to assert those standards.
>>
>>    This document presents some areas in which the more lenient stances
>>    can provide vectors for attack, and then presents the collected
>>    wisdom of numerous applications in and around the email ecosystem for
>>    dealing with them to mitigate their impact.
>>
>> 1.2.  Not The Purpose Of This Work
>>
>>    It is important to understand that this work is not an effort to
>>    endorse or standardize certain common malformations.  The code and
>>    culture that introduces such messages into the mail stream needs to
>>    be repaired, as the security penalty now being paid for this lax
>>    processing arguably outweighs the reduction in support costs to end
>>    users who are not expected to understand the standards.  However, the
>>    reality is that this will not be fixed quickly.
>>
>>    Given this, it is beneficial to provide implementers with guidance
>>    about the safest or most effective way to handle malformed messages
>>    when they arrive, taking into consideration the tradeoffs of the
>>    choices available especially with respect to how various actors in
>>    the email ecosystem respond to such messages in terms of handling,
>>    parsing, or rendering to end users.
>>
>> 1.3.  General Considerations
>>
>>    Many deviations from message format standards are considered by some
>>    receivers to be strong indications that the message is undesirable,
>>    i.e., is spam or contains malware.  Such receivers quickly decide
>>
>>
>>
>> Kucherawy & Shapiro      Expires April 12, 2013                 [Page 4]
>>
>> Internet-Draft             Safe Mail Handling               October 2012
>>
>>
>>    that the best handling choice is simply to reject or discard the
>>    message.  This means malformations caused by innocent
>>    misunderstandings or ignorance of proper syntax can cause messages
>>    with no ill intent also to fail to be delivered.
>>
>>    Senders that want to ensure message delivery are best advised to
>>    adhere strictly to the relevant standards (including, but not limited
>>    to, [MAIL], [MIME], and [DKIM]), as well as observe other industry
>>    best practices such as may be published from time to time either by
>>    the IETF or independently.
>>
>> 2.  Document Conventions
>>
>> 2.1.  Key Words
>>
>>    The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
>>    "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
>>    document are to be interpreted as described in [KEYWORDS].  However,
>>    they only have that meaning in this document when they are presented
>>    entirely in upper case.
>>
>
>
> { While the document is clear that its use of normative language is meant
> to apply only to those implementations choosing to conform to this
> document, the document itself says -- appropriately, IMO -- that it is not
> trying to standardize these behaviors.  It's therefore confusing and
> probably counter-productive to use normative language.  I strongly urge
> dropping all such language and, instead, only offering modest "advice" with
> language like:
>
>    *  a common handling is...
>    *  it is best to...
>    *  it will typically be safe and helpful to...
>
>    and so on.   /d}
>
>
>  2.2.  Examples
>>
>>    Examples of message content include a number within braces at the end
>>    of each line.  These are line numbers for use in subsequent
>>    discussion, and are not actually part of the message content
>>    presented in the example.
>>
>>    Blank lines are not numbered in the examples.
>>
>> 3.  Background
>>
>>    The reader would benefit from reading [EMAIL-ARCH] for some general
>>    background about the overall email architecture.  Of particular
>>    interest is the Internet Message Format, detailed in [MAIL].
>>    Throughout this document, the use of the term "messsage" should be
>>
>
> { Freud possibly at work for this missspellling? /d}
>
>
>     assumed to mean a block of text conforming to the Internet Message
>>    Format.
>>
>> 4.  Internal Representations
>>
>>    Any agent handling a message could have one or two (or more) distinct
>>
>
>      As an agent parses and processes a message, it might create a number
> of distinct representations for the message.
>
>
>     representations of a message it is handling.  One is an internal
>>    representation, such as a block of storage used for the header and a
>>    block for the body.  These may be sorted, encoded, decoded, etc., as
>>    per the needs of that particular module.  The other is the
>>    representation that is output to the next agent in the handling
>>    chain.  This might be identical to the version that is input to the
>>
>>
>>
>> Kucherawy & Shapiro      Expires April 12, 2013                 [Page 5]
>>
>> Internet-Draft             Safe Mail Handling               October 2012
>>
>>
>>    module, or it might have some changes such as added or reordered
>>    header fields, body modifications to remove malicious content, etc.
>>
>>    In some cases, advice is provided only for internal representations.
>>    However, there is often occasion to mandate changes to the output as
>>    well.
>>
>
> { What does this last sentence mean?  "Mandate"?  Perhaps what is meant
> is: /d}
>
>      However it is sometimes necessary to make changes between the input
> and output versions, as well.
>
>
>
>> 5.  Invariate Content
>>
>
>      Invariant {?}
>
>
>
>>    Experience has shown that it is beneficial to ensure that, from the
>>    first analysis agent at ingress into the destination Administrative
>>    Management Domain (ADMD; see [EMAIL-ARCH]) to the agent that actually
>>    affects delivery to the end user, the message each agent sees is
>>
>
> { This is an artfully-crafted sentence, but it would be easier to read if
> broken into parts.  Perhaps: /d}
>
>      An especially interesting handling sequence occurs within the
> destination Administrative Management Domain (ADMD; see [EMAIL-ARCH]). From
> ingress to the ADMD, through the boundary agent, until delivery to the end
> user, it is beneficial to ensure that each agent sees an identical form for
> the message.
>
>
>     identical.  Absent this, it can be impossible for different agents in
>>    the chain to make assertions about the content that correlate.
>>
>
>
>      the chain to make consistent assertions about the content.
>
>
>     For example, suppose a handling agent records that a message had some
>>    specific set of properties at ingress to the ADMD, then permitted it
>>    to continue inbound.  Some other agent alters the content for some
>>    reason.  The user, on viewing the delivered content, reports the
>>    message as abusive.  If the report is based on the set of properties
>>
>
>      message as abuse.  However, report processing often takes place at,
> or close to, the original point of ingress and is likely to be based on the
> set of properties recorded there, rather than at the user's system.
>
>
>     recorded at ingress, then the complaint effectively references a
>>    message different from what the user saw, which could render the
>>    complaint inactionable.  Similarly, a message with properties that a
>>    filtering agent might use to reject an abusive message could be
>>    allowed to reach the user if an intermediate agent altered the
>>    message in a manner that alters one of those properties, thwarting
>>    detection of the abuse.
>>
>
> { awkward sentence structure. d/}
>
>
>     Therefore, agents comprising an inbound message processing
>>
>
>      comprising an inbound  -> within an integrated message
>
> {or should this simply say 'within an ADMD'? /d}
>
>
>     environment SHOULD ensure that each agent sees the same content, and
>>    the message reaches the end user unmodified.  An exception to this is
>>    content that is identitfied as certainly harmful, such as some kind
>>    of malicious executable software included in the message.
>>
>
> {the 'exception' sentence is far too specific.  There are, no doubt, many
> reasons for deviating from this recommendation.  Simpler, safer and
> non-normative wording would be:  /d}
>
>      environment will simplify operational concerns by ensuring that each
> agent receives the same content -- except for the usual handling agent
> trace information additions -- and that this is what reaches the end user,
> unmodified.  Various policies, such as special handling for detected
> message abuse, will make exceptions appropriate.
>
>
>  6.  Mail Submission Agents
>>
>>    Within the email context, the single most influential component that
>>    can reduce the presence of malformed items in the email system is the
>>    Mail Submission Agent (MSA).  This is the component that is
>>    essentially the interface between end users that create content and
>>    the mail stream.
>>
>
>      the Mail Handling Service (MHS) [EMAIL-ARCH]
>
>
>     The lax processing described earlier in the document creates a high
>>
>
> {this first sentence is out of place.  the earlier discussion in the
> document established the need for better conformance; it doesn't need to be
> sold here, again. /d}
>
>
>     support and security cost overall.  Thus, MSAs MUST evolve to become
>>    more strict about enforcement of all relevant email standards,
>>    especially [MAIL] and the [MIME] family of documents.
>>
>>
>>
>>
>> Kucherawy & Shapiro      Expires April 12, 2013                 [Page 6]
>>
>> Internet-Draft             Safe Mail Handling               October 2012
>>
>>
>>    Relay Mail Transport Agents (MTAs) SHOULD also be more strict;
>>
>
>      Relay -> Relaying
>
> { This pseudo-normative phrasing does nothing helpful, since it isn't
> actually specifying anything.  Modify to something like: /d}
>
>      More strict conformance by relaying MTAs also will be helpful.
> Although
>
>
>     although preventing the dissemination of malformed messages is
>>    desirable, the rejection of such mail already in transit also has a
>>    support cost, namely the creation of a [DSN] that many end users
>>    might not understand.
>>
>> 7.  Line Terminaton
>>
>>    The only valid line separation sequence in messaging is ASCII 0x0D
>>
>
>      For interoperable Internet Mail messages, the only valid...
>
>
>     ("carriage return", or CR) followed by ASCII 0x0A ("line feed", or
>>    LF), commonly referred to as CRLF.  Common UNIX user tools, however,
>>    typically only use LF for line termination.  This means the protocol
>>    has to convert LF to CRLF before transporting a message.
>>
>
>      for internal line termination.  This means that a protocol engine,
> which converts between Unix and Internet Mail formats, has to convert
> between these two end-of-line representations, before transmitting a
> message or after receiving it.
>
>
>     Naive implementations can cause messages to be transmitted with a mix
>>
>
> { These aren't "naive".  They are quite simply broken!  d/}
>
>      Implementations that do not conform to Internet Mail standards
> sometimes cause messages to be transmitted...
>
>
>     of line terminations, such as LF everywhere except CRLF only at the
>>    end of the message.  According to [SMTP], this means the entire
>>    message actually exists on a single line.
>>
>
> { also RFC 5322! /d}
>
>
>     A "naked" CR or LF in a message has no reasonable justification, and
>>
>
> { this is wrong.  they have legitimate presentation uses, albeit pretty
> archaic at this point.  Better:  /d }
>
>      Within modern Internet Mail it is highly unlikely that an isolated CR
> or LF is valid, in common ASCII text.  Furthermore [MIME]...
>
>
>     furthermore [MIME] presents mechanisms for encoding content that
>>    actually does need to contain such an unusual character sequence.
>>
>>    Thus, handling agents MUST treat naked CRs and LFs as CRLFs when
>>    interpreting the message.
>>
>
>      Thus, it will typically be safe and helpful to treat a naked CR or LF
> as equivalent to a CRLF, when parsing a message.
>
>
>  8.  Header Anomalies
>>
>>    This section covers common syntactical and semantic anomalies found
>>    in headers of messages, and presents preferred mitigations.
>>
>
>      in a message header, and
>
>
>  8.1.  Converting Obsolete and Invalid Syntaxes
>>
>>    There are numerous cases of obsolete header syntaxes that can be
>>    applied to confound agents with variable processing.  This section
>>
>
> { The phrasing of the first sentence sounds as if confounding is a goal.
>  If it's meant that way, say it more clearly.  If it isn't, perhaps:  /d }
>
>      A message using an obsolete header syntax might confound an agent
> that is attempting to be robust in its handling of syntax variations.
>
>
>     presents some examples of these.  Messages including them SHOULD be
>>
>
> { 'of these'?  of which? /d}
>
> { Why reject this particular set?  What about others, outside these
> examples?  Again, phrase this non-normatively. /d}
>
>
>     rejected; where this is not possible, RECOMMENDED internal
>>    interpretations are provided.
>>
>> 8.1.1.  Host-Address Syntax
>>
>>    The following obsolete syntax:
>>
>
>      The following obsolete syntax that attempts to specify source routing:
>
> { explain, or perhaps even cite the old ABNF rule for it /d}
>
>
>
>>        To: <@example.net:[email-masked]**>
>>
>>    should be interpreted as:
>>
>
>      can safely be interpreted as:
>
>
>         To: <[email-masked]>
>>
>>
>>
>> Kucherawy & Shapiro      Expires April 12, 2013                 [Page 7]
>>
>> Internet-Draft             Safe Mail Handling               October 2012
>>
>>
>> 8.1.2.  Excessive Angle Brackets
>>
>>    The following over-use of angle brackets, e.g.:
>>
>>        To: <<<[email-masked]>>>
>>
>>    should be interpreted as:
>>
>
>      can safely be interpreted as:
>
>
>         To: <[email-masked]>
>>
>> 8.1.3.  Unbalanced Angle Brackets
>>
>>    The following use of unbalanced angle brackets:
>>
>>        To: <[email-masked]
>>        To: [email-masked]>
>>
>>    should be interpreted as:
>>
>
>     can usually be treated as:
>
>
>         To: <[email-masked]>
>>        To: [email-masked]
>>
>> 8.1.4.  Unbalanced Parentheses
>>
>>    The following use of unbalanced parentheses:
>>
>>        To: (Testing <[email-masked]>
>>        To: Testing) <[email-masked]>
>>
>>    should be interpreted as:
>>
>>        To: (Testing) <[email-masked]>
>>        To: "Testing)" <[email-masked]>
>>
>> 8.1.5.  Unbalanced Quotes
>>
>>    The following use of unbalanced quotation marks:
>>
>>        To: "Joe <[email-masked]>
>>
>>    should be interpreted as:
>>
>>        To: "Joe <[email-masked]>"@example.net
>>
>
> { WTF??? And why is this a good fixup, especially given concerns about
> display-name attack vectors?  /d}
>
>
>     where "example.net" is the domain name or host name of the handling
>>    agent making the interpretation.
>>
>>
>>
>>
>>
>> Kucherawy & Shapiro      Expires April 12, 2013                 [Page 8]
>>
>> Internet-Draft             Safe Mail Handling               October 2012
>>
>>
>> 8.2.  Non-Header Lines
>>
>>    It has been observed that some messages contain a line of text in the
>>
>
>      Some messages contain a line of...
>
>
>     header that is not a valid message header field of any kind.  For
>>    example:
>>
>>        From: [email-masked] {1}
>>        To: [email-masked] {2}
>>        Subject: This is your reminder {3}
>>        about the football game tonight {4}
>>        Date: Wed, 20 Oct 2010 20:53:35 -0400 {5}
>>
>>        Don't forget to meet us for the tailgate party! {7}
>>
>>    The cause of this is typically a bug in a message generator of some
>>    kind.  Line {4} was intended to be a continuation of line {3}; it
>>    should have been indented by whitespace as set out in Section 2.2.3
>>    of [MAIL].
>>
>>    This anomaly has varying impacts on processing software, depending on
>>    the implementation:
>>
>>    1.  some agents choose to separate the header of the message from the
>>        body only at the first empty line (i.e. a CRLF immediately
>>        followed by another CRLF);
>>
>>    2.  some agents assume this anomaly should be interpreted to mean the
>>        body starts at line {4}, as the end of the header is assumed by
>>        encountering something that is not a valid header field or folded
>>        portion thereof;
>>
>>    3.  some agents assume this should be interpreted as an intended
>>        header folding as described above and thus simply append a single
>>        space character (ASCII 0x20) and the content of line {4} to that
>>        of line {3};
>>
>>    4.  some agents reject this outright as line {4} is neither a valid
>>        header field nor a folded continuation of a header field prior to
>>        an empty line.
>>
>>    This can be exploited if it is known that one message handling agent
>>    will take one action while the next agent in the handling chain will
>>    take another.  Consider, for example, a message filter that searches
>>    message headers for properties indicative of abusive of malicious
>>    content that is attached to a Mail Transfer Agent (MTA) implementing
>>    option 2 above.  An attacker could craft a message that includes this
>>    malformation at a position above the property of interest, knowing
>>    the MTA will not consider that content part of the header, and thus
>>
>>
>>
>> Kucherawy & Shapiro      Expires April 12, 2013                 [Page 9]
>>
>> Internet-Draft             Safe Mail Handling               October 2012
>>
>>
>>    the MTA will not feed it to the filter, thus avoiding detection.
>>    Meanwhile, the Mail User Agent (MUA) which presents the content to an
>>    end user, implements option 1 or 3, which has some undesirable
>>    effect.
>>
>>    It should be noted that a few implementations choose option 4 above
>>    since any reputable message generation program will get header
>>    folding right, and thus anything so blatant as this malformation is
>>    likely an error caused by a malefactor.
>>
>>    The preferred implementation if option 4 above is not employed is to
>>    apply the following heuristic when this malformation is detected:
>>
>>    1.  Search forward for an empty line.  If one is found, then apply
>>        option 3 above to the anomalous line, and continue.
>>
>>    2.  Search forward for another line that appears to be a new header
>>        field, i.e., a name followed by a colon.  If one is found, then
>>        apply option 3 above to the anomalous line, and continue.
>>
>> 8.3.  Unusual Spacing
>>
>>    The following message is valid per [MAIL]:
>>
>>        From: [email-masked] {1}
>>        To: [email-masked] {2}
>>        Subject: This is your reminder {3}
>>         {4}
>>         about the football game tonight {5}
>>        Date: Wed, 20 Oct 2010 20:53:35 -0400 {6}
>>
>>        Don't forget to meet us for the tailgate party! {8}
>>
>>    Line {4} contains a single whitespace.  The intended result is that
>>    lines {3}, {4}, and {5} comprise a single continued header field.
>>    However, some agents are aggressive at stripping trailing whitespace,
>>    which will cause line {4} to be treated as an empty line, and thus
>>    the separator line between header and body.  This can affect header-
>>    specific processing algorithms as described in the previous section.
>>
>>    Ideally, this case simply ought not to be generated.
>>
>
> {This sentence is entirely gratuitous. Replace it with: d/ }
>
>      This example was legal in earlier versions of the Internet Mail
> format standard.
>
>
>>    Message handling agents receiving a message bearing this anomaly MUST
>>    behave as if line {4} was not present on the message, and SHOULD emit
>>    a version in which line {4} has been removed.
>>
>
>      The best handling of this example is for a message parsing engine to
> behave as if line {4} was not present in the message and for a message
> creation engine to emit the message with line {4} removed,.
>
>
>
>>
>>
>>
>> Kucherawy & Shapiro      Expires April 12, 2013                [Page 10]
>>
>> Internet-Draft             Safe Mail Handling               October 2012
>>
>>
>> 8.4.  Header Malformations
>>
>>    There are various malformations that exist.  A common one is
>>
>
> { The first sentence is pretty obvious: there are always lots of ways to
> screw up. I suggest dropping it and beginning with something like:  /d}
>
>    Among the many possible malformations, a common one is...
>
>
>     insertion of whitespace at unusual locations, such as:
>>
>>        From: [email-masked] {1}
>>        To: [email-masked] {2}
>>        Subject: This is your reminder {3}
>>        MIME-Version : 1.0 {4}
>>        Content-Type: text/plain {5}
>>        Date: Wed, 20 Oct 2010 20:53:35 -0400 {6}
>>
>>        Don't forget to meet us for the tailgate party! {8}
>>
>>    Note the addition of whitespace in line {4} after the header field
>>    name but before the colon that separates the name from the value.
>>
>>    The acceptance grammar of [MAIL] permits that extra whitespace, so it
>>    cannot be considered invalid.  However, a consensus of
>>    implementations prefers to remove that whitespace.  There is no
>>    perceived change to the semantics of the header field being altered
>>    as the whitespace is itself semantically meaningless.  Thus, a module
>>    compliant with this memo MUST remove all whitespace after the field
>>    name but before the colon, and MUST emit that version of that field
>>    on output.
>>
>
>      Therefore, it is best to remove all whitespace after the field name
> but before the colon and to emit the field in this modified form.
>
>
>  8.5.  Header Field Counts
>>
>>    Section 3.6 of [MAIL] prescribes specific header field counts for a
>>    valid message.  Few agents actually enforce these in the sense that a
>>    message whose header contents exceed one or more limits set there are
>>    generally allowed to pass; they may add any required fields that are
>>
>
>    ; they typically add any...
>
>
>     missing, however.
>>
>>    Also, few agents that use messages as input, including Mail User
>>    Agents (MUAs) that actually display messages to users, verify that
>>    the input is valid before proceeding.  Two popular open source
>>    filtering programs and two popular Mailing List Management (MLM)
>>
>
> { I suggest changing 'two' to 'some', since the number might change;
> there's no reason to make this document get out of date for such a minor
> issue. /d }
>
>
>     packages examined at the time this document was written select either
>>
>
> { hence, remove "examined at the time this document was written" /d }
>
>
>     the first or last instance of a particular field name, such as From,
>>    to decide who sent a message.  Absent enforcement of [MAIL], an
>>
>
>      Absent strict enforcement
>
>
>     attacker can craft a message with multiple fields if that attacker
>>    knows the filter will make a decision based on one but the user will
>>    be shown the other.
>>
>>    This situation is exacerbated when a claim of message validity is
>>    inferred by something like a valid [DKIM] signature.  Such a
>>    signature might cover one instance of a constrained field but not
>>
>>
>>
>> Kucherawy & Shapiro      Expires April 12, 2013                [Page 11]
>>
>> Internet-Draft             Safe Mail Handling               October 2012
>>
>>
>>    another, and a naive consumer of DKIM's output, not realizing which
>>    one was covered by a valid signature, could presume the wrong one was
>>    the "good" one.  An MUA, for example could show the first of two From
>>    fields as "good" or "safe" while the DKIM signature actually only
>>    verified the second.
>>
>
> { DKIM signatures do not verify addresses outside d=.  While the problem
> you are describing is, of course, real, it's far more complicated than
> you've described here.  Perhaps:  /d }
>
>      when message validity is assessed, such as through enhanced
> authentication methods.  Such methods might cover one instance of a
> constrained field but not another, taking the wrong one as "good" or "safe".
>
>
>     Thus, an agent compliant with this specification MUST enact one of
>>    the following:
>>
>
>      In attempting to counter this exposure, one of the following can be
> enacted:
>
>
>     1.  reject outright or refuse to process further any input message
>>        that does not conform to Section 3.6 of [MAIL];
>>
>>    2.  remove or, in the case of an MUA, refuse to render any instances
>>        of a header field whose presence exceeds a limit prescribed in
>>        Section 3.6 of [MAIL] when generating its output;
>>
>>    3.  alter the name of any header field whose presence exceeds a limit
>>        prescribed in Section 3.6 of [MAIL] when generating its output so
>>        that later agents can produce a consistent result.  Any
>>        alteration likely to cause the field to be ignored by downstream
>>        agents is acceptable.  A common approach is to prefix the field
>>        names with a string such as "BAD-".
>>
>
> { it would help if there were some rationales or analyses of the tradeoffs
> amongst these kinds of choices, to help the implementer/operator decide
> when to use which.  /d }
>
>
>
>  8.6.  Missing Header Fields
>>
>>    Similar to the previous section, there are messages seen in the wild
>>    that lack certain required header fields.  For example, [MAIL]
>>    requires that a From and Date field be present in all messages.
>>
>
> { I think these aren't 'examples' but constitute the entire list.  If
> there are other required fields that can be classed as 'missing', this
> section should list them.  Also, since Message-ID isn't 'required', the
> phrasing here doesn't quite match what's discussed in the section. Might be
> worth distinguishing "required but missing" vs. "optional but really useful
> and worth synthesizing".  Synthesizing the latter probably isn't dangerous.
>  Synthesizing the former always is... d/}
>
>
>
>>    When presented with a message lacking these fields, the MTA might
>>    perform one of the following:
>>
>>    1.  Make no changes
>>
>>    2.  Add an instance of the missing field(s) using synthesized content
>>
>
>      3.  Reject the message
>
>
>     Option 2 is RECOMMENDED for handling this case.  Handling agents
>>
>
> { Wow!  Synthesizing a From: field strikes me as especially dangerous, in
> all cases.  The rationale provided, below, needs to state this and, I
> believe, explain how and why it is worth incurring. The explanation that is
> provided essentially define this hack as an attack vector... /d}
>
>
>     SHOULD add these for internal hanlding if they are missing, but MUST
>>    NOT add them to the external representation.  The reason for this
>>    requirement is that there are some filter modules that would consider
>>    the absence of such fields to be a condition warranting special
>>    treatment (e.g., rejection), and thus the effectiveness of such
>>    modules would be stymied by an upstream filter adding them.
>>
>>    The synthesized fields SHOULD contain a best guess as to what should
>>    have been there; for From, the SMTP MAIL command's address can be
>>    used (if not null) or a placeholder address followed by an address
>>    literal (e.g., unknown@[192.0.2.1]); for Date, a date extracted from
>>
>>
>>
>> Kucherawy & Shapiro      Expires April 12, 2013                [Page 12]
>>
>> Internet-Draft             Safe Mail Handling               October 2012
>>
>>
>>    a Received field is a reasonable choice.
>>
>>    One other important case to consider is a missing Message-Id field.
>>    An MTA that encounters a message missing this field SHOULD synthesize
>>    a valid one using techniques described above and add it to the
>>    external rpresentation, since many deployed tools use the content of
>>    that field as a common unique message reference, so its absence
>>    inhibits correlation of message processing.  One possible synthesis
>>    would be based on based on an encoding of the current date/time and
>>    an internal MTA ID (e.g., queue ID) followed by @ and the fully
>>    qualified hostname of the machine synthesizing the header value.  For
>>    example:
>>
>>        tm = gmtime(&now);
>>        (void) snprintf(buf, sizeof(buf), "%04d%02d%02d%02d%02d.%s@%s",
>>                        tm->tm_year + 1900, tm->tm_mon + 1, tm->tm_mday,
>>                        tm->tm_hour, tm->tm_min, queueID, fqhn);
>>
>> 8.7.  Eight-Bit Data
>>
>>    Standards-compliant mail messages do not contain any non-ASCII data
>>    without indicating that such content is present by means of published
>>    [SMTP] extensions.  Absent that, [MIME] encodings are typically used
>>
>
> Overall, the document sometimes mixes transfer issues with data
> representation (object) issues, in ways that can be confusing.  This
> paragraph is one of those.  It's worth the extra verbosity to label each
> clearly and separately.  So, for example:
>
>      Standards-compliant mail messages that contain non-ASCII data are
> required to self-label this through the use of [MIME].  If the
> representation of the non-ASCII data is in an 8-bit mode (rather than
> special encoding so that it retains a 7-bit base), then this must be
> signaled through the use of [SMTP] extensions.
>
>
> >    without indicating that such content is present by means of published
> >    [SMTP] extensions.
>
>     to convert non-ASCII data to ASCII in a way that can be reversed by
>>    other handling agents or end users.
>>
>>    Non-ASCII data otherwise found in messages can confound code that is
>>    used to analyze content.  For example, a null (ASCII 0x00) byte
>>    inside a message can cause typical string processing functions to
>>    mis-identify the end of a string, which can be exploited to hide
>>    malicious content from analysis processes.
>>
>>    Handling agents MUST reject messages containing null bytes that are
>>    not encoded in some standard way, and SHOULD reject other non-ASCII
>>    bytes that are similarly not encoded.  If rejection is not done, an
>>    ASCII-compatible encoding such as those defined in [MIME] SHOULD be
>>    used.
>>
>
> { Hmmm.  It occurs to me that the document might be helped by an early
> discussion about a/the 'philosophy' that guides choosing whether to reject
> a message versus repair it.  But I don't have any clever text to suggest
> for doing this... /d}
>
>
>
>> 9.  MIME Anomalies
>>
>>    [MIME], et seq, define a mechanism of message extensions for
>>
>
> { perhaps quibbling, but since MIME does a variety of things, including
> this one, I suggest:
>
>      define -> includes
>
>
>     providing text in character sets other than ASCII, non-text
>>    attachments to messages, multi-part message bodies, and similar
>>    facilities.
>>
>>    Some anomalies with MIME-compliant generation are also common.  This
>>    section discusses some of those and presents preferred mitigations.
>>
>>
>>
>>
>> Kucherawy & Shapiro      Expires April 12, 2013                [Page 13]
>>
>> Internet-Draft             Safe Mail Handling               October 2012
>>
>>
>> 9.1.  Header Field Names
>>
>>    [MAIL] permits header field names to begin with "--".  This means
>>    that a header field name can look like a [MIME] multipart boundary.
>>    For example:
>>
>>      --foo:bar
>>
>>    This is a legal header field, whose name is "--foo" and whose value
>>    is "bar".  Thus, consider this header:
>>
>>        From: [email-masked] {1}
>>        To: [email-masked] {2}
>>        Subject: This is your reminder {3}
>>        Date: Wed, 20 Oct 2010 20:53:35 -0400 {4}
>>        MIME-Version: 1.0 {5}
>>        Content-Type: multipart/mixed; boundary="foo:bar" {6}
>>        --foo:bar {7}
>>        Malicious-Content: muahaha {8}
>>
>>    One implementation could observe that line {7} announces the
>>    beginning of the first MIME part while another considers it a part of
>>    the message's header.
>>
>>    If rejection of such messages cannot be done, agents MUST treat line
>>    {7} as part of the message's header block and not a MIME boundary.
>>
>
> { Under what circumstances can rejection /not/ be done??? And what is
> involved in even detecting that it isn't a mime boundary?  d/ }
>
>
>
>> 9.2.  Missing MIME-Version Field
>>
>>    Any message that uses [MIME] constructs is required to have a MIME-
>>    Version header field.  Without them, the Content-Type and associated
>>    fields have no semantic meaning.
>>
>
>      them -> it
>
>
>     It is often observed that a message has complete MIME structure, yet
>>    lacks this header field.
>>
>>    As described at the end of Section 8.2, this is not expected from a
>>
>
>      this -> this omission
>
>
>     reputable content generator and is often an indication of mass-
>>    produced spam or other undesirable messages.
>>
>>    Therefore, an agent compliant with this specification MUST internally
>>    enact one or more of the following in the absence of a MIME-Version
>>    header field:
>>
>>    1.  Ignore all other MIME-specific fields, even if they are
>>        syntactically valid, thus treating the entire message as a
>>        single-part message of type text/plain;
>>
>
> { Offhand, this sounds like a potentially-interesting attack vector.  /d}
>
>
>
>> Kucherawy & Shapiro      Expires April 12, 2013                [Page 14]
>>
>> Internet-Draft             Safe Mail Handling               October 2012
>>
>>
>>    2.  Remove all other MIME-specific fields, even if they are
>>        syntactically valid, both internally and when emitting the output
>>        version of the message;
>>
>> 10.  Body Anomalies
>>
>> 10.1.  Oversized Lines
>>
>>    A message containing a line of content that exceeds 998 characters
>>    plus the line terminator (1000 total) violates Section 2.1.1 of
>>    [MAIL].  Some handling agents may not look at content in a single
>>    line past the first 998 bytes, providing bad actors an opportunity to
>>    hide malicious content.
>>
>>    There is no specified way to handle such messages, other than to
>>    observe that they are non-compliant and reject them, or rewrite the
>>    oversized line such that the message is compliant.
>>
>>    Handling agents MUST take one of the following actions:
>>
>>    1.  Break such lines into multiple lines at a position that does not
>>        change the semantics of the text being thus altered.  For
>>        example, breaking an oversized line such that a [URI] then spans
>>        two lines could inhibit the proper identification of that URI.
>>
>>    2.  Rewrite the MIME part (or the entire message if not MIME) that
>>        contains the excessively long line using a content encoding that
>>        breaks the line in the transmission but would still result in the
>>        line being intact on decoding for presentation to the user.  Both
>>        of the encodings declared in [MIME] can accomplish this.
>>
>> 11.  Security Considerations
>>
>>    The discussions of the anomalies above and their prescribed solutions
>>    are themselves security considerations.  The practises enumerated in
>>    this memo are generally perceived as attempts to resolve security
>>    considerations that already exist rather than introducing new ones.
>>
>
> { Hmmm.  Whereas I think the document introduces quite a few attack
> vectors that probably aren't discussed in other email specifications. /d}
>
>
>
>> 12.  IANA Considerations
>>
>>    This memo contains no actions for IANA.
>>
>>    [RFC Editor: Please remove this section prior to publication.]
>>
>> 13.  References
>>
>>
>>
>>
>>
>>
>> Kucherawy & Shapiro      Expires April 12, 2013                [Page 15]
>>
>> Internet-Draft             Safe Mail Handling               October 2012
>>
>>
>> 13.1.  Normative References
>>
>>    [KEYWORDS]    Bradner, S., "Key words for use in RFCs to Indicate
>>                  Requirement Levels", BCP 14, RFC 2119, March 1997.
>>
>>    [MAIL]        Resnick, P., "Internet Message Format", RFC 5322,
>>                  October 2008.
>>
>> 13.2.  Informative References
>>
>>    [DKIM]        Allman, E., Callas, J., Delany, M., Libbey, M., Fenton,
>>                  J., and M. Thomas, "DomainKeys Identified Mail (DKIM)
>>                  Signatures", RFC 4871, May 2007.
>>
>>    [DSN]         Moore, K. and G. Vaudreuil, "An Extensible Message
>>                  Format for Delivery Status Notifications", RFC 3464,
>>                  January 2003.
>>
>>    [EMAIL-ARCH]  Crocker, D., "Internet Mail Architecture", RFC 5598,
>>                  July 2009.
>>
>>    [MIME]        Freed, N. and N. Borenstein, "Multipurpose Internet
>>                  Mail Extensions (MIME) Part One: Format of Internet
>>                  Message Bodies", RFC 2045, November 1996.
>>
>>    [RFC822]      Crocker, D., "Standard for the Format of Internet Text
>>                  Messages", RFC 822, August 1982.
>>
>>    [SMTP]        Klensin, J., "Simple Mail Transfer Protocol", RFC 5321,
>>                  October 2008.
>>
>>    [URI]         Berners-Lee, T., Fielding, R., and L. Masinter,
>>                  "Uniform Resource Identifier (URI): Generic Syntax",
>>                  RFC 3986, January 2005.
>>
>> Appendix A.  Acknowledgements
>>
>>    The author wishes to acknowledge the following for their review and
>>    constructive criticism of this proposal: Tony Hansen, and Franck
>>    Martin
>>
>> Authors' Addresses
>>
>>    Murray S. Kucherawy
>>
>>    EMail: [email-masked]
>>
>>
>>
>>
>>
>> Kucherawy & Shapiro      Expires April 12, 2013                [Page 16]
>>
>> Internet-Draft             Safe Mail Handling               October 2012
>>
>>
>>    Gregory N. Shapiro
>>
>>    EMail: [email-masked]
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Kucherawy & Shapiro      Expires April 12, 2013                [Page 17]
>>
>>
> --
>  Dave Crocker
>  Brandenburg InternetWorking
>  bbiw.net
>

---

From: Ned Freed <[email-masked]>
Date: Mon, 06 May 2013 14:14:55 -0700 (PDT)

I reviewed this quite some time ago, and as I recall my assessment was mostly
favorable.

I just reviewed it again, and surprised myself considerably by finding this
document to be quite problematic in its present form. It's now my position that
a number of additions and changes need to be made.

The overarching problem here isn't what this document does; it's what it
doesn't do. It presents a small set of malformation issues, a set which is
necessarily only informed by past and present issues encountered in the field.
This is fine as far as it goes, but the fact that things have changed in the
short time since this document was written - we're not seeing several of the
listed malformation nearly as often as we used to and we're starting to see new
ones that aren't on the list - is a good indication that the lack of such
guidance is a serious omission, especially in what is supposed to be a stable
reference for such matters.

So what would such guidance look like? I don't pretend to have the final answer
here, but these three points strike me a a good starting point:

(1) Whenever possible mitigation of syntactic malformations should be guided
    by an assessment of the most likely semantic intent. For example,
    it is reasonable to conclude that multiple sets of angle brackets
    around an address are simply superflous and can simply be dropped.

(2) When the intent is unclear, or alternately, when it is clear but is
    inpractical to change things to express it, mitigation should be limited
    to cases where not doing anything would clearly lead to a worse outcome.

(3) Security issues, when present, need to be addressed and may force
    mitigation strategies that are otherwise suboptimal.

Now let's consider the various recommendations the document currently makes in
light of these guidelines.

Section 5 really needs to distinguish between semantic and syntactic
invariance, especially in light of syntactic variations caused by things like
MIME downgrade/update, to say nothing of the explicit advice given in sections
8.3 and 8.4 of this document.

Line termination (section 7) is fine. True, the intent of a bare CR might be to
overstrike a line, but this is increasingly a minority taste these days and a
failure to deal with bare CRs and LFs likely leads to a worse outcome overall.

Malformed address fields (section 8.1.*) suffers from a problem separate from
consideration of these guidelines: The failure to deal with the possible
presence of multiple addresses. Unfotunately such cases are quite common
and some thought needs to be given to them in at least some of these sections.
For example, in section 8.1.3 it might be good to point out that a comma
may reaonably be interpreted as ending an address:

       To: <[email-masked], [email-masked]> -->
       To: [email-masked], [email-masked]

Beyond that, and now considering the guidlines, I find the advice given in
8.1.5 to be incorrect. I'm sorry, but the chances that something like:

   "Joe <[email-masked]>"@example.net

is going to work are remote in the extreme. All this is going to do is create a
situation where a probem occurs in a context that's far removed from the actual
cause. The only case where I'd actually consider such a "mitigation" would be
where it is essential to pull something out. Otherwise leaving the field alone
is the better bet, because that will make it easier to diagnose the actual
problem.

I'll also note that the failure to consider the fairly common case of 
a field that just says:

   To: Joe

seems to me to be an omission that should be corrected, especially since this
is attempting to cover SUBMIT server fixups.

Non-Header Lines (section 8.2) is OK. (I happen to think the advice given in
8.2 is flat-out wrong, but that's a matter of experience coupled with
implementation practicalities; it's not something I can justify based on
the guidelines.)

Unusual spacing (section 8.3) and Header Malformations (section 8.4) are both
spot on.

Header Field Counts (section 8.5) needs work because it fails to take specific
field semantics into account in what it recommends. It is of course true that
when performing some sort of validation check on an originator field it's
essential to pick one and be consistent about it. But there are many other
fields where such checks are rarely if ever applied, e.g., To: and Cc:. Given
that  there are agents out there that employ a separate field for each address,
surely a viable mitigation is to combine all recipient fields of a given type
into a single field?

Missing header fields (section 8.6) is spot on.

Eight-Bit Data (section 8.7) needs a bunch of work, and once again it's a
failure to consider the semantics of where the 8bit shows up. 8bit appearing in
a header is a very different proposition from 8bit in a body. EAI deserves a
shout-out in the former case because it's new, and the latter case is now so
much of a ho-hum for many agents that a recommendation to reject 8bit is
nothing short of silly.

I'm also far from convinced that rejecting messages because of a single null is
a good idea. I think a fair assessment of the likely intent in this case is the
presence of a null or two is simply a message construction error, and silently
dropping them is a much better bet.

Header Field Names (section 9.1) is certainly cute, but MIME is actually
quite clear that the first place a boundary can occur is in the following
body, not the associated header. I'm not really comfortable with
this document acting on an assessment of intent of material that is in fact
completely valid. My suggestion is therefore to refrain from talking about
rejecting such messages.

I'm afraid "Missing MIME-Version Field" (section 9.2) is flat-out incorrect.
The intent of a message that has a valid content-type and possibly
content-transfer-encoding is pretty darned clear, and perhaps more to the
point, failing to interpret such a message as MIME, given that many other
agents, e.g., metamail, are going to, is exceedingly poor practice from a
security standpoint.

(If we're up for additions at this late date, it might also be good to add a
section on how to handle bogus base64 or quoted-printable.)

And finally, Oversized Lines (section 10.1) is spot on.

Finally, a couple of comments about Dave's review. I'm not especially fussed
about the use or non-use of compliance language. If there's a consensus to
remove it that's fine, if not that's fine with me too.

> > misbehaving senders.  So they/we hacked receiving code when necessary, to

This is more or less correct. Not only do receivers rarely have any way to
correct sender behavior, in a depressingly large number of cases some of the
flagrant violations came from widely deployed software from large companies. In
other cases it's the receiving user agent that's busted, and intermediaries get
stuck with trying to accomodate their bustedness. And both are especially bad
when deployed in hardware with no viable upgrade path.

				Ned

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Sun, 12 May 2013 23:47:12 -0700

Hi Ned, thanks for your comments.  I've worked in most of your suggestions,
but have the following to ask further on some of your points:

> dropping them is a much better bet.

Eight bit data is far from something I'm expert on.  Could I ask you to
write up a paragraph or three to include here, or replace what's here?

> completely valid. My suggestion is therefore to refrain from talking about

It's a real attack seen in the wild.  I see your point about MIME and the
location of boundaries, however, so I think this section can just go.

> agents, e.g., metamail, are going to, is exceedingly poor practice from a

This is also something real I dealt with at a previous employer.  More
often than not it was seen as part of a spam attack that targeted specific
MUAs.  I'm fine with removing this section however if the advice is
controversial; I don't have access to data to back up how important or
useful this change was.

> section on how to handle bogus base64 or quoted-printable.)

Yup, we are.  Do you have anything specific in mind?

-MSK

---

From: "John Levine" <[email-masked]>
Date: 13 May 2013 15:39:32 -0000

>write up a paragraph or three to include here, or replace what's here?

These days, eight bit data in the message body is most likely just
text.  The tricky bit is figuring out what the encoding is.  In my
mail, the eight-bit characters I see most often are angled single and
double quotes and other punctuation, typically in text that has been
pasted in from a web browser or word processor.  I also see accented
characters, either in people's names or just to add élan.  I gather
that simple heuristics that check 8bit characters against various
encodings of stuff likely to appear in text work pretty well, at least
in mailstreams that are primarily in roman alphabet languages.

In headers, you will see unencoded UTF-8 in mail that leaks from EAI
mailstreams into normal mail.  It is my impression that it is rare to
see unencoded 8bit characters in any other encoding in headers.

In both cases, it's much more likely to be a mistake than an attack.

In 8.6, is it really a good idea to add missing date headers in an
MTA?  My impression is that most mail that makes it to your MTA
without a date header isn't mail you want to read.

R's,
John

---

From: Ned Freed <[email-masked]>
Date: Tue, 14 May 2013 10:07:28 -0700 (PDT)

> but have the following to ask further on some of your points:

Great!

> write up a paragraph or three to include here, or replace what's here?

This is going to take a lot more than three paragraphs to do properly. I'd
suggest something like this (apologies in advance - this is *very* rough, but
I'm in the middle of an office move and don't have time right now to wordsmith
the text the way I normally would):

--

8.7 Missing or incorrect charset information

MIME provides the means to include textual material employing charsets other
than US-ASCII. Such material is required to have an identifiable charset.
Charset identification is done using a charset parameter in the content-type
header field, a charset label within the MIME entity itself, or the charset
may be implicitly specified by the content-type [RFC 6657].

It is unfortunately fairly common for required charset information to be
missing or incorrect in textual MIME entities. As such, processing agents
should perform basic sanity checks, e.g.,

(1) US-ASCII is 7bit only so 8bit material is necessarily not US-ASCII.
(2) UTF-8 has a very specific syntactic structure that other 8bit charsets
    are unlikely to follow.
(3) Nulls (ASCII 0x00) are not allowed in either 7bit or 8bit data.
(4) Not all 7bit material is US-ASCII. The presence of the various escape
    sequences used for character switching may be used as an indication
    of the various iso-2022-* charsets.

When a charset error is detected processing agents should:

(a) apply heuristics to determine the most likely charset and if successful
    proceed using that information, or
(b) refuse to process the malformed MIME entity.

A null (ASCII 0x00) byte inside a textual MIME entity can cause typical string
processing functions to mis-identify the end of a string, which can be
exploited to hide malicious content from analysis processes. According, nulls 
require additional special handling.

A few nulls in isolation is likely to be the result of poor message
construction practices. Such nulls should be silently dropped. 

Large numbers of nulls are usually the result of binary material that is
improperly encoded, labelled, or both. Such material is likely
to be damaged beyond the hope of recovery so the best course of action
is to refuse to process it.

Finally, the presence of nulls may be used as indication of possible
malicious intent.

8.8 8bit data

The best way to handle incompliant 8bit material depends on its location.

Incompliant 8bit in MIME entity content should simply be processed as if the
necessary SMTP extensions had been used to transfer the message. Note that
improperly labeled 8bit material in textual MIME entities may require treatment
as described in section 8.7.

Incompliant 8bit in message or MIME entity header fields can be handled
as follows:

(1) Occurrences in unstructured text fields, comments and phrases
    can be converted into encoded-words [RFC 2047] if a likely charset can
    be determined. Alternately, 8bit characters can be removed or replaced
    with some other character.

(2) Occurrences in header fields whose syntax is unknown may be handled
    by dropping the field entirely or by removing/replacing the 8bit
    character as in (1).

(3) Occurrences in addresses are especially problematic. Agents supporting
    [EAI] may, if the 8bit conforms to 8bit syntax, elect to treat the
    messages as an EAI message and process it accordingly. Otherwise it is
    in most cases best to exclude the address from any sort of processing -
    which may mean dropping it entirely - since any attempt to fix it is
    unlikely to help.

--

One thing I'm strongly inclined to add is the option of presenting material
to a user using an interface that lets them select the charset. Like it or not,
this is a successful strategy employed by UAs in the field. 

> location of boundaries, however, so I think this section can just go.

It's an attack using a valid message on egregiously broken software. In
fact it's likely an indicator that an even more fundamental problem is present:
Searching forward into the message for data rather than first picking out the
header and processing it separately.

Given that the underlying problem may well be exploitable in other ways, the
only real solution in such cases is to fix the broken software. Nothing else
really suffices. That said, the issue of exploitable errors in software is
beyond the scope of this document, so I agree the best thing is to remove this
section.

> controversial; I don't have access to data to back up how important or

Not sure what you're talking about here. The section describes no specific
attack of any sort; it simply offers the observation that a message with MIME
structure and no MIME-Version is more likely to be spam than a message that
does  contain a MIME-Version:, and then recommends that MIME structure not be
processed or be removed when this occurs.

It has not been my observation that missing MIME-Version: correlates all that
well with spam; plenty of legitimate agents omit it and spam has evolved to
include it. (This should be seen as natural and inevitable, after all, spam
agents are under more evolutionary pressure than legitimate agents.)

But more to the point, lots of software, including metamail, the original MIME
software, pays no attention to MIME-Version: at all. And the chances of this
changes are, IMO, remote.

As such, the direct consequence of this recommendation is to enable a far worse
attack: One where malicious content sneaks by agents written to comply with
this specification and gets to software that will interpret the MIME structure
and process the malicious content.

I therefore believe that not only is what the section says incorrect, it's
also inappropriate to remove it. The correct thing to do is say that the
MIME structure should always be interpreted when present, especially by
agents checking for malicious material.

You could also add that it may be appropriate to use the lack of a MIME-Version
as an indicator that malicious content may be present and to take additional
precautions in processing the message. I believe that's the best way to address
the concern this section was originally aimed at.

> Yup, we are.  Do you have anything specific in mind?

It's basically the same "exploit different interpretations by different
decoders" issue. For example, the original base64 defined in RFC 1113 -1115
allowed comments inside parentheses. Other implementations will silently ignore
parentheses and interpret whatever is inside as data. The possible exploits of
such differences are obvious.

				Ned

---

From: "John Levine" <[email-masked]>
Date: 15 May 2013 20:26:13 -0000

>the text the way I normally would):

Way better than what I sent.

>changes are, IMO, remote.

I use Alpine which is one of the very few MUAs that take Murray's
advice and ignore MIME headers if the MIME-Version is missing.  It is
a pain in the patoot, since it is invariably wrong.  I ended up using
procmail to add the missing header to mail from known screwups like
the NY Times so I could read it properly.

R's,
John

---

From: Ned Freed <[email-masked]>
Date: Wed, 15 May 2013 13:32:59 -0700 (PDT)

> Way better than what I sent.

FWIW, I agree completely with everything you said, and I hope it's reflected
in what I wrote.

> the NY Times so I could read it properly.

NYT? Wow. I hadn't noticed they were one of the places that does it. The
example I usually think of is one of the sources of political spam^H^H^H^H
legitimate bulk mail which seems to do it intermittently. But that's
nowhere near as good as example as the NYT.

Another amusing thing one of these bulk email generators does is put out
multiparts marked with a quoted-printable CTE. (The multipart is not, in fact
encoded, although there have been other cases in the past where it was.) I
don't know if this one is worth calling out in the doc or not.

				Ned

---

From: "John R Levine" <[email-masked]>
Date: 15 May 2013 21:33:54 -0400

> NYT? Wow. I hadn't noticed they were one of the places that does it.

Yup.  They may have fixed it by now, haven't checked lately since the
procmail rule is harmless either way.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
"I dropped the toothpaste", said Tom, crestfallenly.

---

From: Franck Martin <[email-masked]>
Date: Thu, 16 May 2013 20:47:35 -0500 (CDT)

8.1.5. Unbalanced Quotes

Not sure I agree with that, we should avoid to place a domain name in the display part.

should be interpreted as:
To: "Joe" <[email-masked]>

8.5. Header Field Counts

I see option 1 at the MTA level
option 2 at the MSA
for an MUA, it has to deal with an invalid message, I think 3 could apply, but then the MUA only renders the message why should it alter it, especially if the message will be rendered by different MUA.

I see it differently...

If the email comes from a MSA then add the fields, if the email comes from another MTA, reject the email.

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Thu, 16 May 2013 19:22:15 -0700

> To: "Joe" <[email-masked]>

The document doesn't talk about what end users should be shown.  It's all
about filters and other handling agents.  There's some specific text in
there that talks about "internal representations" and such; see Section 4.

> for an MUA, it has to deal with an invalid message, I think 3 could apply,
> but then the MUA only renders the message why should it alter it,
> especially if the message will be rendered by different MUA.
>

The MSA is not in scope here, other than the fact that the document
mentions that MSAs need to be more strict.  This is all about MTAs and
filters.

> another MTA, reject the email.

See above.

-MSK

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Fri, 17 May 2013 00:07:37 -0700

I've posted a new version of the draft based on all the recent feedback.
Thanks to those that gave some!

With it refreshed and with all the new content, I'd love to have some more
reviews of it in its current form.

-MSK, hatless

On Wed, May 15, 2013 at 6:33 PM, John R Levine <[email-masked]> wrote:

> ______________________________**_________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/**listinfo/apps-discuss<https://www.ietf.org/mailman/listinfo/apps-discuss>
>

---

From: Ned Freed <[email-masked]>
Date: Sat, 18 May 2013 10:39:04 -0700 (PDT)

> Thanks to those that gave some!

This is looking good IMO, but of course I'd say that given how much of my
feedback has been incorporated into the doc.

> reviews of it in its current form.

I concur. Additional eyes are really needed on this.

				Ned

---

From: "John R Levine" <[email-masked]>
Date: 18 May 2013 23:00:24 -0400

>> reviews of it in its current form.

Well, since you asked.

I'm looking at the whole thing and I have to say I don't understand the 
point of sections 4 and 5.  I understand that Exchange and Outlook smash 
messages into an internal format and sort of reconstruct the original, 
sometimes, but I can't tell whether that's what you're referring to or 
something else.

I think sec 4 is trying to say mail software may have a variety of 
internal representations, but all we're talking about here are messages 
interchanged in what purports to be 5322 (or its predecessors) format.

Sec 5 appears to say that if you try to parse and unparse a message, you 
probably won't get back quite the same thing and particularly in the 
presence of DKIM signatures and the like, details matter, so don't do 
that. If you need multiple formats, retain the original so if you need to 
bounce or report something, you can report what you actually received.

If that's not what you mean, I'm totally confused.

This is more a question for Ned: in section 7, how much legitimate mail do 
you still see with bare CR or LF?  It is my impression that the bugs that 
caused those things were fixed a long time ago.

I think 8.6 has TMI, I'd just say to synthesize a unique Message-ID. 
MSAs have been doing that for decades, people know how to do it, and your 
example seems more complicated than enlightening.  (Why use gmtime rather 
than just stick the numeric value of "now" into the ID.  Doesn't matter 
what the encoding is so long as it changes frequently.)

10.1, bottom of p. 17: MUST take?  Even after downcasing the rogue 
2119ism, I'd make it a lot milder than that.  If I am feeding stuff into 
my webmail system, and I happen to know that my display module handles 
long lines OK, I'd just leave the long lines alone.  It is my impression 
that long lines are much more likely to be from sloppy composition 
software than malicious.

R's,
John

---

From: Ned Freed <[email-masked]>
Date: Sun, 19 May 2013 14:24:58 -0700 (PDT)

> caused those things were fixed a long time ago.

Still see bare LFs occasionally. Bare CRs seem to have vanished along with the
use of MacOS versions prior to X, which AFAIK is the only platform of any
significance at all that ever used bare CR as a line terminator.

I considered recommending treating bare CR like null, but I can't see that
it matters all that much.

				Ned

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 28 May 2013 09:57:02 -0700

> interchanged in what purports to be 5322 (or its predecessors) format.

Consider a system that has several independent filtering modules.  It's
important that the modules don't change the content as it passes from one
to the next as doing so might obscure something that the next module might
consider reject-worthy.  Therefore, whatever transformations need to be
done have to be kept internal to the module and not released as the "real"
content.

> presence of DKIM signatures and the like, details matter, so don't do that.
> If you need multiple formats, retain the original so if you need to bounce
> or report something, you can report what you actually received.
>

This is similar, but has a different nuance.  If a message makes it all the
way to an MUA where it's reported as abusive (spam, virus, whatever), then
a report generated based on what the MUA sees might not be useful to the
report receiver because it doesn't match the message that was originally
generated.

In the next revision, I've mushed 4 and 5 together since they were so
similar.  I'll post later today and see what you think.

> I think 8.6 has TMI, I'd just say to synthesize a unique Message-ID. MSAs
> have been doing that for decades, people know how to do it, and your

That's a real-world example; it's what sendmail does.  But sure, simpler is
better.  I'll clean it up.

> 2119ism, I'd make it a lot milder than that.  If I am feeding stuff into my
> webmail system, and I happen to know that my display module handles long
> lines OK, I'd just leave the long lines alone.  It is my impression that
> long lines are much more likely to be from sloppy composition software than
> malicious.
>

That MUST should've come out in the last edit, since we're no longer doing
2119 stuff in here.  I'll just take that out.

Thanks for your comments.

-MSK

---

From: Ned Freed <[email-masked]>
Date: Tue, 28 May 2013 17:50:35 -0700 (PDT)

> done have to be kept internal to the module and not released as the "real"

It's nowhere near this simple. Sometimes you want filtering modules exposed
to unadulterated content. One way to arrange for that is preclude any
changes, but another is to run the modules in parallel. After all, if they're
not talking to each other, why wouldn't you want them to run simultaneously?

But other times you want them to be able to pass information to each other.
Unfortunately that often can only be done by adding headers. And there are
even times when you want much more radical transformations applies, such
as the removal of encryption, forced conversion to MIME format, etc. etc.

There really isn't a one size fits all answer here, and it's silly to think
one can be found.

> report receiver because it doesn't match the message that was originally

And sometimes the opposite is true, e.g., in regards to attaching intermediate
analysis results. Again. it's not a one size fits all situation.

I'm not making any specific suggestion for a change here, more like suggesting
that certain possible changes are best not made.

				Ned

---

From: "John R Levine" <[email-masked]>
Date: 28 May 2013 21:17:24 -0400

> There really isn't a one size fits all answer here, and it's silly to think

Perhaps it's better to describe results than methods, e.g., if the process 
leads to a bounce, what you bounce better be close enough to what you 
received that the other party can recogize it as something they sent.

Regards,

---

From: Ned Freed <[email-masked]>
Date: Tue, 28 May 2013 18:42:13 -0700 (PDT)

> received that the other party can recogize it as something they sent.

Seems reasonable.

				Ned

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 29 May 2013 00:05:54 -0700

> There really isn't a one size fits all answer here, and it's silly to think

The -05 version of this document makes it clear that we're talking about a
system where the handling modules operate in a sequence, since that's the
model I had in mind when I rewrote that text.  I'll add some additional
prose about the points you've made here.

-MSK