Subject: [Emailcore] SMTP-TCP is de facto 8bit (suggested clarity in 5321bis)

From: Sabahattin Gucukoglu <[email-masked]>
Date: Sun, 17 Oct 2021 05:39:10 +0100

5321bis-04 (and especially section 2.4) suggests SMTP-TCP servers corrupt or refuse undeclared 8bit content, and that SMTP clients MUST NOT send undeclared 8bit content. Can I respectfully suggest that in 2021 this is a bad idea that will simply result in corrupted or lost mail?

For a long time now, SMTP-TCP is de facto 8bit: hosts just pass through 8bit content, whether or not it is declared, unless it is MIME-aware, in which case it may or may not perform conversion on 8BITMIME mail that is declared as such or that is auto-detected (but in which case it may corrupt the message or invalidate authentication signatures such as DKIM).

Many implementations do not include MIME support in the SMTP servers. This isn’t unreasonable because it adds complexity. But they will still advertise 8BITMIME, in order to optimise the delivery of 8bit mail. See e.g. Dan Bernstein’s notable rant on the subject:
https://cr.yp.to/smtp/8bitmime.html

Exim enabled 8BITMIME by default in version 4.80. It is “8bit clean”, too.

I think language should be added that sternly admonishes servers against corrupting or refusing undeclared 8bit content, and that downgrades MUST NOT to SHOULD NOT in discussion of client behaviour, suggesting strongly that 8BITMIME is an obligation to perform 7bit downgrade but recognising the necessity of transmitting 8bit where the content cannot definitely be identified as MIME or protected by an authentication scheme (this latter being an especially important consideration for relaying or forwarding systems).

Cheers,
Sabahattin

---

From: John C Klensin <[email-masked]>
Date: Sun, 17 Oct 2021 02:13:10 -0400

Sabahattin,

I am, as usual, willing to modify the text any way the WG tells
me to and I certainly think this issue deserves further
discussion.  However, with all due respect to Dan Bernstein and
the quality of his rants (including both the ones you cite and,
IIR, the arguments in the 1990s that we should simply declare
all older systems obsolete and let anyone who used them bear the
consequences) there are still legacy systems around (I even
still see HELO announcements).  There is also an interesting
question of who or what is sending 8bit content to servers that
do not make announcements.   If the reality is that all
contemporary servers that otherwise conform to 5321 are making
the announcement, then this issue is probably moot.  And if all
of those who might send 8bit materials to servers who don't
announce it are bad actors for other reasons, maybe we should
not be too concerned about dropping their mail.

Finally, as people are thinking about this, please also think
about whether encouraging people to send message body material
that contains what we are calling 8bit content would also
encourage them or others to ignore the more complicated rules
associated with SMTPUTF8 and just assume that good 8-bit-clear
servers will cope.

best,
   john

--On Sunday, October 17, 2021 05:39 +0100 Sabahattin Gucukoglu
<listsebby=[email-masked]> wrote:

> 5321bis-04 (and especially section 2.4) suggests SMTP-TCP
> servers corrupt or refuse undeclared 8bit content, and that
> SMTP clients MUST NOT send undeclared 8bit content. Can I
> respectfully suggest that in 2021 this is a bad idea that will
> simply result in corrupted or lost mail?
> 
> For a long time now, SMTP-TCP is de facto 8bit: hosts just
> pass through 8bit content, whether or not it is declared,
> unless it is MIME-aware, in which case it may or may not
> perform conversion on 8BITMIME mail that is declared as such
> or that is auto-detected (but in which case it may corrupt the
> message or invalidate authentication signatures such as DKIM).
> 
> Many implementations do not include MIME support in the SMTP
> servers. This isn't unreasonable because it adds complexity.
> But they will still advertise 8BITMIME, in order to optimise
> the delivery of 8bit mail. See e.g. Dan Bernstein's notable
> rant on the subject: https://cr.yp.to/smtp/8bitmime.html
> 
> Exim enabled 8BITMIME by default in version 4.80. It is
> "8bit clean", too.
> 
> I think language should be added that sternly admonishes
> servers against corrupting or refusing undeclared 8bit
> content, and that downgrades MUST NOT to SHOULD NOT in
> discussion of client behaviour, suggesting strongly that
> 8BITMIME is an obligation to perform 7bit downgrade but
> recognising the necessity of transmitting 8bit where the
> content cannot definitely be identified as MIME or protected
> by an authentication scheme (this latter being an especially
> important consideration for relaying or forwarding systems).
> 
> Cheers,
> Sabahattin

---

From: "John Levine" <[email-masked]>
Date: 17 Oct 2021 15:34:39 -0400

It appears that John C Klensin  <[email-masked]> said:
>still see HELO announcements). ...

How about making 8BITMIME support mandatory.  Sure, there is a long
tail of dusty MTAs that still say HELO, but the people who run them
won't be reading our specs and if they still send 7bit, nothing will
break.

Every computer on the Internet has used 8 bit bytes for at least 30
years, and at this point it is just perverse not to handle 8 bit
input data.

R's,
John

---

From: John C Klensin <[email-masked]>
Date: Sun, 17 Oct 2021 18:20:01 -0400

--On Sunday, October 17, 2021 15:34 -0400 John Levine
<[email-masked]> wrote:

>> I am, as usual, willing to modify the text any way the WG
>> tells me to and I certainly think this issue deserves further
>> discussion.  However, with all due respect to Dan Bernstein
>> and the quality of his rants (including both the ones you
>> cite and, IIR, the arguments in the 1990s that we should
>> simply declare all older systems obsolete and let anyone who
>> used them bear the consequences) there are still legacy
>> systems around (I even still see HELO announcements). ...
> 
> How about making 8BITMIME support mandatory.  Sure, there is a
> long tail of dusty MTAs that still say HELO, but the people
> who run them won't be reading our specs and if they still send
> 7bit, nothing will break.
> 
> Every computer on the Internet has used 8 bit bytes for at
> least 30 years, and at this point it is just perverse not to
> handle 8 bit input data.

I'd be happier about that solution than about the relaxation
that Sabahattin suggested.  It would also improve the foundation
for SMTPUTF8 which already requires 8BITMIME.

   john

---

From: Ned Freed <[email-masked]>
Date: Wed, 20 Oct 2021 14:21:22 -0700 (PDT)

> > least 30 years, and at this point it is just perverse not to

The concerns of DEC-10 aficionados notwithstanding, 8BITMIME was never about
supporing machines with odd byte sizes. Rather, it was about accommodating
implementations on 8 bit byte machines that handled 8bit data *badly*, e.g.,
clearing (or in a few cases, setting) the high bit unconditionally.

A secondary concern was proper charset labeling in place, but I'd just as soon
not get into that.

We had customers turn on "just send 8" and run into issues as late as the
mid-2000s.

Like it or not, support from the underlying infrastructure doesn't necessarily
translate into application support for something.

I'll also note that personal experience, no matter how extensive, doesn't
translate into comprehensive understanding of everything that's out there.

An anecdote. Like most implementations, at some point we changed the default so
our SMTP server would only accept TLS , not SSL. This seemed like a perfectly
reasonable change to make. However, as luck would have it, a customer had
deployed some sort of software with a version of sendmail that it used to send
out status reports, which were then processed somewhere else. And this version
of sendmail only supported SSL, not TLS.

Even worse, fallback to plaintext didn't work, for reasons I no longer recall.
So the messages stayed in the sendmail queue, which was located on the root
partition with limited disk space.

And this was deployed on 1000s of machines.

> for SMTPUTF8 which already requires 8BITMIME.

Mandating 8BITMIME seems like the right approach to me as well.

				Ned

---

From: "John R Levine" <[email-masked]>
Date: 20 Oct 2021 17:54:23 -0400

On Wed, 20 Oct 2021, Ned Freed wrote:
>>> handle 8 bit input data. ...
>
> clearing (or in a few cases, setting) the high bit unconditionally. ...

> Mandating 8BITMIME seems like the right approach to me as well.

I think we agree.  My point was more that there are no machines left where 
it would be hard to handle 8 bits if the people running it care to do so.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Alessandro Vesely <[email-masked]>
Date: Fri, 22 Oct 2021 19:06:35 +0200

On Wed 20/Oct/2021 23:54:23 +0200 John R Levine wrote:
> I think we agree.

+1, this should neutralize most X-MIME-Autoconverted:.

Best
Ale
--

---

From: Sabahattin Gucukoglu <[email-masked]>
Date: Tue, 26 Oct 2021 17:46:31 +0100

Thanks for the responses.

I agree that 8BITMIME should be made mandatory, although that wasn’t what I was asking for. I agree that this would make the problem of undeclared 8bit a non-issue in most cases, however, in the absence of any definite statement …

I would still be concerned that 8BITMIME carried semantic burdens that were harmful. Specifically, I worry that legitimate mail would be lost or corrupted if receivers were not encouraged to be tolerant of it. There is undoubtedly a “moral hazard” in the relaxation, but I don’t believe it’s worse than the known consequence of not receiving such mail. The goal, I think, should be to minimise rather than prevent such aberration, which is already occurring, and to do this by forbidding senders from *gratuitously* sending avoidably 8bit content to 7bit hosts. Receivers can benefit from 8bit, while being careful only to send 7bit, or relying on a true protocol converter, preferably prior to authenticating outbound messages with signatures, from a submission service. 8BITMIME is still useful if it is simply used to negotiate onward transmission of 8bit mail with complying senders, but it cannot help without some enforcement by receivers (at the very least, checking that the top bit is set in a mail, at the absolute minimum) for onward transmission of non-complying messages, and having essentially variable behaviour in this case wouldn’t be desirable, especially once 8BITMIME becomes mandatory and 8bit transmission is possible in practice and in principle.

As to where undeclared 8bit comes from, often it is just 8bit-unaware senders, e.g. MUAs or scripts that inject 8bit mail without declaring it 8bit in the envelope. That is surprisingly common even in the English-speaking world as well, for example when it is generated by automated systems with interfaces to the mailer that aren’t using SMTP.

I don't intend to die on a hill on this issue. If the consensus is that 8BITMIME should be mandatory but that the obligation to send MIME content that is declared as such remains with the originator of the message, and that any intermediate relays must enforce by rejection or conversion the 8BITMIME requirement not to forward onward, it is still preferable to the status quo. At the very least it would strengthen the case for proper 8BITMIME enforcement , however weakly, to have mandatory 8BITMIME support which senders can rely on in the general case.

Cheers,
Sabahattin

---

From: "John Levine" <[email-masked]>
Date: 26 Oct 2021 22:14:32 -0400

It appears that Sabahattin Gucukoglu  <[email-masked]> said:
>I would still be concerned that 8BITMIME carried semantic burdens that were harmful. Specifically, I worry that legitimate mail would be
>lost or corrupted if receivers were not encouraged to be tolerant of it. ...

IETF standards tell you what to do if you want to interoperate with other people who follow the same standards.

We deliberately stay away from offering advice about what to do when other people do not follow the standards, for
a variety of reasons.  One is that people are infinitely creative when doing things wrong and we could never
anticipate, much less deal with, all of the possible mistakes.  The other is that if you say do so-and-so if someone
makes mistake X, you have now made mistake X a de facto part of the standard.

We all know that there are cruddy mail implementations, some clients send 8 bit characters without looking for
8BITMIME, or following the limitations of 8BITMIME, and it mostly works.  We don't propose to change that, only
to say that if you want to interoperate, your server better support 8BITMIME.

R's,
John