Subject: [Emailcore] Ticket #14: G.7.8. Review different size limits

From: Todd Herr <[email-masked]>
Date: Fri, 9 Jul 2021 15:01:55 -0400

Greetings.

I would like to start a discussion about ticket #14 -
https://trac.ietf.org/trac/emailcore/ticket/14

Section 4.5.3.1 of draft-ietf-emailcore-rfc5321bis reads:

4.5.3.1.  Size Limits and Minimums

   There are several objects that have required minimum/maximum sizes.
   Every implementation MUST be able to receive objects of at least
   these sizes.  Objects larger than these sizes SHOULD be avoided when
   possible.  However, some Internet mail constructs such as encoded
   X.400 addresses (RFC 2156 [26]) will often require larger objects.
   Clients MAY attempt to transmit these, but MUST be prepared for a
   server to reject them if they cannot be handled by it.  To the
   maximum extent possible, implementation techniques that impose no
   limits on the length of these objects should be used.
   Extensions to SMTP may involve the use of characters that occupy more
   than a single octet each.  This section therefore specifies lengths
   in octets where absolute lengths, rather than character counts, are
   intended.

and ticket 14 asks the question:

Given the controversy on the SMTP mailing list between 20191123 and now
about maximum lengths, is the above adequate or is further tuning of the
limit text below needed?

draft-ietf-emailcore-rfc5321bis then goes on to specify length limits in
octets, not characters, in each of the following sections:

4.5.3.1.1. Local-part
4.5.3.1.2. Domain
4.5.3.1.3. Path
4.5.3.1.4. Command Line
4.5.3.1.5. Reply Line
4.5.3.1.6. Text Line
4.5.3.1.7. Message Content
4.5.3.1.8. Recipient Buffer

Thoughts?

-- 

*Todd Herr* | Technical Director, Standards and Ecosystem
*e:* [email-masked]
*m:* 703.220.4153

This email and all data transmitted with it contains confidential and/or
proprietary information intended solely for the use of individual(s)
authorized to receive it. If you are not an intended and authorized
recipient you are hereby notified of any use, disclosure, copying or
distribution of the information included in this transmission is prohibited
and may be unlawful. Please immediately notify the sender by replying to
this email and then delete it from your system.

---

From: Ned Freed <[email-masked]>
Date: Tue, 13 Jul 2021 05:52:51 -0700 (PDT)

> about maximum lengths, is the above adequate or is further tuning of the

The text, especially the part about X.400, is seriously dated at this point.
(For those of you who have never dealt with X.400, X.400 addresses consist of a
list of attributes and values, including but not limited to attributes for
postal address information, and theoretically can be thousands of octets in
length. The 1:1 mapping of this defined by RFC 2156 puts most of this in the
local part in /attr1=/value1/attr2=value2/... form.)

As a practical matter real world X.400 gateways never required longer lengths
on the SMTP/RFC822 side, for the simple reason that they never worked well
enough. (It didn't stop compliance suites from coding tests that required
them...) A combination of restraint on the part of X.400 implementations and
very complicated mappings were used to avoid the problem.

And if a client has a fallback, as the text requires, why not just use it
rather than sending something that may work locally but fail downstream,
where no fallback is possible?

I also note that X.400 has changed from "the mail system we'll all be using
somday" to "only used in limited government/military cases, and even those are
disappearing". Is it really worth mentioning at this point?

Finally, and most importantly, implementation techniques that impose no limits
on object size represent a serious security risk. I think this alone justifies
removing "should support" bit.

> 4.5.3.1.8. Recipient Buffer

Like it or not, the limits in existing implementations are octet, not
character, based. A change to make them character-based is therefore
wholly nonsensical.

				Ned

---

From: Dave Crocker <[email-masked]>
Date: Tue, 13 Jul 2021 08:32:50 -0700

> disappearing". Is it really worth mentioning at this point?

It is not.

> removing "should support" bit.

+1

d/

-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: Alessandro Vesely <[email-masked]>
Date: Tue, 13 Jul 2021 18:30:33 +0200

On Tue 13/Jul/2021 17:32:50 +0200 Dave Crocker wrote:
>> someday" to "only used in limited government/military cases, and even those are

Where do those quotes come from?  Do you have a reference?  When did the change 
occur?

> It is not.

+1,

>> removing "should support" bit.

You mean "should be used"?  ("should support" doesn't appear in that paragraph).

I'm not clear what is the security risk.  Clearly, no implementation can be 
truly limitless, since the Universe itself appears to be limited.  Perhaps:

                                                            To the
    maximum extent possible, implementation techniques that impose no
    *predefined* limits on the length of these objects should be used.

?

Best
Ale
--

---

From: Ned Freed <[email-masked]>
Date: Tue, 13 Jul 2021 11:16:35 -0700 (PDT)

> Where do those quotes come from?

They came from me. I'm paraphrasing what many people said back in the 80s when
X.400 was the new hotness and now when it's old and busted.

It's hard to believe now, but at the time the expectation really was that a
mostly government-run email service based on a universal directory was the
future.

> Do you have a reference?  When did the change occur?

These sorts of transitions rarely involve an abrupt change, but as it happens,
in the case of X.400 I did have an "aha" moment. It was back in the early 90s
at the Electronic Mail Association's Message Attachment Working Group (MAWG)
meeting on profiling the File Transfer Body Part and aligning it with MIME. As
the EMA equivalent of the blue sheet started circulating, Nick Shelness stood
up and (more or less) said, "Please include your Internet address on the sheet,
most of the X.400 ones you've provided in the past don't work."

Since then I've wondered if Nick - among other things, the main author of
Softswitch - knew the significance of what he was saying. Knowing Nick, he
probably did.

> You mean "should be used"?  ("should support" doesn't appear in that

Correct.

>     *predefined* limits on the length of these objects should be used.

That's still a recipe for disaster because people won't know to set them
correctly. Anything without a reasonable limit can  be found by a hacker and
exploited to exhaust resources.

And if you don't believe this happens, I have a couple of very nice bridges to
sell you.

It also does nothing to enhance interoperability. Requiring the minimums but
encouraging everyone to implement more opens the door to implementators coding
stuff that assumes larger limits - because what they had in the lab suppported
more - but then fails badly when faced with a wider range of implementations on
the Internet.

IMO at this point interoperability trumps being able to get away with stuff 
sometimes.

				Ned

---

From: "John Levine" <[email-masked]>
Date: 13 Jul 2021 15:47:35 -0400

>> disappearing". Is it really worth mentioning at this point?

Oh, all right, says cca!ima!johnl

>> removing "should support" bit.

Agreed.  If it's a limit, it's a limit.

Speaking of limits, how do we feel about the 1000 octet limit on line length.  It is my impression
that it is often exceeded in HTML text.

R's,
John

---

From: Todd Herr <[email-masked]>
Date: Wed, 14 Jul 2021 14:38:16 -0400

On Tue, Jul 13, 2021 at 3:47 PM John Levine <[email-masked]> wrote:

> Haven't had occasion to see the issue crop up in a while, but I don't miss
having to explain to a content creator that the limit exists, and that left
angle bracket b r right angle bracket is not the same as carriage return,
line feed.
--

---

From: Viktor Dukhovni <[email-masked]>
Date: Wed, 14 Jul 2021 17:10:22 -0400

> On 13 Jul 2021, at 3:47 pm, John Levine <[email-masked]> wrote:
> 
> Speaking of limits, how do we feel about the 1000 octet limit on line
> length.  It is my impression that it is often exceeded in HTML text.

FWIW, internally Postfix has no message content line length limit.
Long lines (over 2048 bytes) are stored and processed in chunks.

That said, lines in messages sent out via SMTP are by default folded
at 998+CRLF bytes:

	http://www.postfix.org/postconf.5.html#smtp_line_length_limit

Logical (possibly folded across multiple physical lines) *headers* longer
than 100KB are truncated:

	http://www.postfix.org/postconf.5.html#header_size_limit

So it would not be a big deal if the limit were raised, but it would
take some years before most Postfix installations upgraded to a
release with a higher default limit.  If some sites override the
default value, they'd have to update their explicit override if too
small.

-- 
-- 
	Viktor.

---

From: Ned Freed <[email-masked]>
Date: Wed, 14 Jul 2021 18:04:07 -0700 (PDT)

> angle bracket b r right angle bracket is not the same as carriage return,

Purely anecdotal, but I've noticed that in the past couple of years quite
a few HTML senders seem to have newly discovered something called
quoted-printable.

In any case, I don't see how we can possibly increase the 998+CRLF limit at
this point. There are far too many SMTP implementations with fixed buffer sizes
that are never going to be updated and will be in use for a long, long time.
Nor, frankly, do I see any advantage in doing so given the existence of
encoding options like quoted-printable.

				Ned

---

From: Alessandro Vesely <[email-masked]>
Date: Thu, 15 Jul 2021 10:11:53 +0200

> Softswitch - knew the significance of what he was saying. Knowing Nick, he

Thanks for sharing!
(https://en.wikipedia.org/wiki/X.400#cite_note-1)

>> [...]

That's a general safety rule, not an SMTP-specific provision.  Yet, the 
consideration meant by the sentence could be retained.  How about:

                          Up to the point where resource exhaustion
      would be at stake, implementation techniques that impose no
      predefined limits on the length of these objects should be used.

> stuff that assumes larger limits - because what they had in the lab supported

Agreed.  However, removing the sentence conveys a much stronger meaning to the 
size limits given in the spec.  It depends on whether we consider them to be 
hard limits or just reasonable indications given at the time when the spec is 
being published.  Just like X.400 fades out, some trend requiring more space 
might fade in in the future.

Consider the line length limit John mentioned.

Consider the overall message length limit, which the spec does not give, but 
whose range seems to be more or less agreed upon among the vast majority of 
servers.

RCPTLIMIT could impose local hard limits in a more flexible way.

Best
Ale
--

---

From: Alessandro Vesely <[email-masked]>
Date: Thu, 15 Jul 2021 12:43:07 +0200

> 	http://www.postfix.org/postconf.5.html#header_size_limit

Worded as in postconf it's not clear whether you mean a whole header rather 
than a single header field.  In the text above, the phrase up to and including 
the parenthesis suggests the latter interpretation.

Best
Ale
--

---

From: Viktor Dukhovni <[email-masked]>
Date: Thu, 15 Jul 2021 09:45:27 -0400

> the parenthesis suggests the latter interpretation.

It is *a* header field, but perhaps the text could be made more
explicit.  The text in http://www.postfix.org/header_checks.5.html
has a bit more context, but it is I guess possible to still read
it as applying to complete set of message headers as a unit.

       header_checks

       mime_header_checks (default: $header_checks)

       nested_header_checks (default: $header_checks)
              Lookup tables with  content  filter  rules  for  message  header
              lines:  respectively,  these  are applied to the initial message
              headers (not including MIME headers), to the MIME  headers  any-
              where  in  the  message,  and to the initial headers of attached
              messages.

------->      Note: these filters see one logical message header  at  a  time,
------->      even when a message header spans multiple lines. Message headers
------->      that are longer than  $header_size_limit  characters  are  trun-
------->      cated.

Anyway, bottom line, large (even unbounded) message *body* line sizes
work just fine in Postfix, but logical message headers (used in MIME
parsing, header checks, ...) do need a manageable size limit, which is
generously set at 100KiB, with longer (individual) headers truncated to
not exceed that size.

If milters are employed, then the particular milter applications would
have to deal with any larger limits, and I don't know if there's any
general statement that can be made about support for long logical or
physical message header or body lines in milters.

-- 
    Viktor.

---

From: Steffen Nurpmeso <[email-masked]>
Date: Thu, 15 Jul 2021 20:08:57 +0200

Viktor Dukhovni wrote in
 <[email-masked]>:
 |> On 13 Jul 2021, at 3:47 pm, John Levine <[email-masked]> wrote:
 |> 
 |> Speaking of limits, how do we feel about the 1000 octet limit on line
 |> length.  It is my impression that it is often exceeded in HTML text.

Normally MIME should come into play then, and apply
a content-transfer-encoding to break it down.

The problem for (also other) simple-minded software is then to
unfold it when decoding.

 |FWIW, internally Postfix has no message content line length limit.
 |Long lines (over 2048 bytes) are stored and processed in chunks.

That is a desirable approach, congratulations.
I wish i were there already.

 |That said, lines in messages sent out via SMTP are by default folded
 |at 998+CRLF bytes:

There was that UTF-8 RFC from i think a Microsoft guy who soaped
this up to say "in modern times a character is the new byte", so
that this should now be 1000*4, at least for SMTP, not for
5322-bis, though.  Maybe too sloppily said.  (My memory says i had
an angry moment when AlpineLinux added ICU dependency to postfix
in order to support that UTF-8 extension.)

Maybe that is a weakness in the original folding algorithm, that
whitespace at the begin of follow lines is preserved.  Maybe the
(now) usual "line-continuation by escaping the linefeed with
reverse solidus" would have been the better approach, then one
could simply break were desired and it would all not matter.
Like it is now artificial whitespace may have happen in the worst
case, but --- isn't that itself so artificial that extending the
limit of 1000 looks a bit like over-engeneering?
Especially since presence of non 7-bit clean data imposes MIME
encoding, anyway?

 | http://www.postfix.org/postconf.5.html#smtp_line_length_limit
 |
 |Logical (possibly folded across multiple physical lines) *headers* longer
 |than 100KB are truncated:
 |
 | http://www.postfix.org/postconf.5.html#header_size_limit

If i recall correctly i was once stumbling over a dovecot commit
which added header truncation like that, too.

 |So it would not be a big deal if the limit were raised, but it would
 |take some years before most Postfix installations upgraded to a
 |release with a higher default limit.  If some sites override the
 |default value, they'd have to update their explicit override if too
 |small.

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

---

From: "John Levine" <[email-masked]>
Date: 14 Jul 2021 21:51:53 -0400

>encoding options like quoted-printable.

I was wondering if there really are such MTAs or if this is one of those limits
that don't matter to the code any more.

My MTA allocates all the strings dynamically so if its 3 bytes or a megabyte
it dosn't care.  Dunno how common that is.

R's,
John

---

From: "John Levine" <[email-masked]>
Date: 15 Jul 2021 16:16:38 -0400

>that this should now be 1000*4, at least for SMTP, ...

That is just wrong.  RFC 6532 specifically says if you're sending unencoded
UTF-8, the limit is still 1000 octets, not 1000 characters.

>reverse solidus" would have been the better approach, ...

That only applies to headers.  The 1000 octet limit applies to the
whole message.  For text we have flowed text which uses a space at
the end of a line as a signal to rewrap.

>Especially since presence of non 7-bit clean data imposes MIME

Hm.  You might want to investigate the 8BITMIME SMTP extension, defined in
1993 and supported by every MTA I've looked at in this millenium.

R's,
John

PS: I don't feel strongly about lifting the 1000 octet limit but I would like
to understand how widely it's enforced by mail receivers.

---

From: Viktor Dukhovni <[email-masked]>
Date: Thu, 15 Jul 2021 16:58:36 -0400

> On 15 Jul 2021, at 4:16 pm, John Levine <[email-masked]> wrote:
> 

On the receiving side Postfix does not enforce any physical
line length limits on message body lines, but as already
mentioned logical header headers, and thus of course also
any single physical line in a logical header can't exceed
100KiB.

The 1000 octet limit is only enforced when sending.

-- 
	Viktor.

---

From: "John Levine" <[email-masked]>
Date: 15 Jul 2021 18:19:21 -0400

It appears that Viktor Dukhovni  <[email-masked]> said:
>line length limits on message body lines, ...

I realize qmail isn't very popular any more, although you will find
it hiding in the guts of a lot of packaged mail systems.  For incoming
messages, it's all dynamic and there's no length limits.  For outgoing
messages it politely wraps headers to 80 bytes but leaves bodies alone.

---

From: Alessandro Vesely <[email-masked]>
Date: Fri, 16 Jul 2021 10:08:15 +0200

> messages it politely wraps headers to 80 bytes but leaves bodies alone.

Courier has similar limits on submission.  Some 100000 for the whole header 
(configurable since 2007, BOFHHEADERLIMIT) and 5000 bytes line length limit.

Best
Ale
--

---

From: Ned Freed <[email-masked]>
Date: Fri, 16 Jul 2021 08:34:35 -0700 (PDT)

> (configurable since 2007, BOFHHEADERLIMIT) and 5000 bytes line length limit.

It's really quite unfortunate that there's no limit specified on header
size, since sending a huge header is yet another way to exhaust resources
on a poorly implemented MTA.

However, adding one is clearly beyond the scope of what we're allowed to
do in emailcore. Even a change to make the limits we have "real" is pushing
it.

				Ned

> --
> Emailcore mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/emailcore

---

From: Michael Peddemors <[email-masked]>
Date: Fri, 16 Jul 2021 08:52:40 -0700

Vote against changing existing limits, it is not just MTA's that have to 
deal with this, but many email analytics programs etc..

But I agree that we should consider some 'best practices' size limits on 
headers..

Can we start looking at existing limits in more details?

But as I think John pointed out, creators of HTML messages have long be 
operating with poor practices.  Have even seen some messages where the 
whole HTML part was on a single line. But we could have a SHOULD 
indication of max line limits, while a long line is 'technically' not an 
issue, there are many practical examples why it is not preferred, eg 
string matching in long lines, not to mention simple things like viewing 
a message source and how ugly difficult this is when the message creates 
a very long scroll window.

-- 
"Catch the Magic of Linux..."
------------------------------------------------------------------------
Michael Peddemors, President/CEO LinuxMagic Inc.
Visit us at http://www.linuxmagic.com @linuxmagic
A Wizard IT Company - For More Info http://www.wizard.ca
"LinuxMagic" a Registered TradeMark of Wizard Tower TechnoServices Ltd.
------------------------------------------------------------------------
604-682-0300 Beautiful British Columbia, Canada

This email and any electronic data contained are confidential and intended
solely for the use of the individual or entity to which they are addressed.
Please note that any views or opinions presented in this email are solely
those of the author and are not intended to represent those of the company.

---

From: Dave Crocker <[email-masked]>
Date: Fri, 16 Jul 2021 08:56:20 -0700

+1

Unless I missed it in the thread, I didn't see any documentation of a 
serious problem that needs fixing, nevermind a serious benefit to be 
obtained, but it's clear changing limits imposes a systemic effect, 
which the current effort seeks to minimize.

d/

---

From: Steffen Nurpmeso <[email-masked]>
Date: Fri, 16 Jul 2021 18:29:13 +0200

John Levine wrote in
 <[email-masked]>:
 |It appears that Steffen Nurpmeso  <[email-masked]> said:
 |>|> Speaking of limits, how do we feel about the 1000 octet limit on line
 |>|> length.  It is my impression that it is often exceeded in HTML text.
 |
 |>There was that UTF-8 RFC from i think a Microsoft guy who soaped
 |>this up to say "in modern times a character is the new byte", so
 |>that this should now be 1000*4, at least for SMTP, ...
 |
 |That is just wrong.  RFC 6532 specifically says if you're sending unencoded
 |UTF-8, the limit is still 1000 octets, not 1000 characters.

Yes, "octet" was the used term.  In the tradition of preceeding
RFCs.  I surely was too sloppy here.

 |>Maybe that is a weakness in the original folding algorithm, that
 |>whitespace at the begin of follow lines is preserved.  Maybe the
 |>(now) usual "line-continuation by escaping the linefeed with
 |>reverse solidus" would have been the better approach, ...
 |
 |That only applies to headers.  The 1000 octet limit applies to the
 |whole message.  For text we have flowed text which uses a space at
 |the end of a line as a signal to rewrap.

Yes.  (But for headers there is a SHOULD for automatic wrapping
(at 78), whereas bodies i would expect to be left alone unless the
limit of "at least / no lower than 1000" is excessed; for any
software that is not a pure MTA i would expect convertion to MIME
and using a content-transfer-encoding instead.  You very often see
this even enforced, for example on Mailman-driven mailing-lists,
where you are turned to base64 encoding whether useful/necessary
or not.  This started somewhen before Python 3 was truly released,
and it was still like this when i stopped looking at email content
regulary not too far in the past, .. which must have been shorty
before Mailman 2 was deprecated by support for Python 2 has ended,
i think.)

 |>Especially since presence of non 7-bit clean data imposes MIME
 |>encoding, anyway?
 |
 |Hm.  You might want to investigate the 8BITMIME SMTP extension, defined in
 |1993 and supported by every MTA I've looked at in this millenium.

I have these RFCs (i have 6152 locally, it obsoleted 1652).  But
i do not think i have ever encountered usage of this myself.

 |R's,
 |John
 |
 |PS: I don't feel strongly about lifting the 1000 octet limit but I \
 |would like
 |to understand how widely it's enforced by mail receivers.

I would assume transmission is rejected for any such email?
About line reading via fgets(3) and finite buffer .. this is
really more common than one would think, especially in elder
software.  Ever since i "really" program this was a "oh no no no,
do not do that!", so i personally ((basic ->) perl -> Java ->
C++/C) never actively used this i think (when i could make
a decision, ie, for C++/C).  I surely have seen fly by commits of
*BSD base-system software turning over such to dynamic line
readers over the last decade, where still necessary (OpenBSD
thus)!  The Unix console no. 1 MUA mutt turned many many code
places to a dynamic buffer type over the last hm maybe two years
for sure, but it surely did it right before already.  That is,
from the MUA side i would not expect to many restrictions on the
consumption side, at least in Unix i would expect them all to
honour "be liberal in what you expect and strict in what you
produce"; i have collected some test mails that happened to happen
over the last decade, and at least at the beginning i compared how
other Unix console mailers (and once even Apple Mail, of Snow
Leopard, but no) behave ... hm, the longest header line is 1009
bytes; there is a References: that is folded but spans more than
a full screen of a 212x58 screen, hmm, a (maybe manually made
worse) spam mail ("ANGLO AMERICA PLATINUM UK ONLINE
<[email-masked]>") in 54.header-mem-booom.mbox.  This is from 2015,
i have forgotten whether i was still cross-testing by then.

 --End of <[email-masked]>

--steffen
|

---

From: "John Levine" <[email-masked]>
Date: 16 Jul 2021 14:12:57 -0400

>i do not think i have ever encountered usage of this myself.

I see unencoded UTF-8 in mssages bodies all the time. ¡Hola! 😕

> |would like |to understand how widely it's enforced by mail receivers.
>

Not that I've ever seen but I haven't looked very hard.  I think it's
more common to stuff in a \r\n and continue.

Anyway, I think we've agreed that the 1000 octet limit isn't changing.

R's,
John
-- 
Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Steffen Nurpmeso <[email-masked]>
Date: Sat, 17 Jul 2021 18:16:57 +0200

John Levine wrote in
 <[email-masked]>:
 |According to Steffen Nurpmeso  <[email-masked]>:
 |>|>Especially since presence of non 7-bit clean data imposes MIME
 |>|>encoding, anyway?
 |>|
 |>|Hm.  You might want to investigate the 8BITMIME SMTP extension, defined \
 |>|in
 |>|1993 and supported by every MTA I've looked at in this millenium.
 |>
 |>I have these RFCs (i have 6152 locally, it obsoleted 1652).  But
 |>i do not think i have ever encountered usage of this myself.
 |
 |I see unencoded UTF-8 in mssages bodies all the time. ¡Hola! 😕

Your MUA created the according MIME headers, but whether it used
the 8BITMIME SMTP extension when passing this over to a MTA, that
is here the question.  Some may not, the one i took maintainership
of for example surely never did, yet supports 8bit and simply
passed it over, and whereas i changed the default content-
transfer-encoding the SMTP code is more or less unchanged in
released versions.  (That will change with the next, whenever that
is, i am late by half a year at minimum, but 8BITMIME not yet in
there, so you maybe even earned a credit in defiance of your
invalid mail address.)
I have not encountered problems from the SMTP side when actually
trying out aka using these defaults (years ago), so it seems there
are quite liberal SMTP engines around.

 |>|PS: I don't feel strongly about lifting the 1000 octet limit but I \
 |>|would like |to understand how widely it's enforced by mail receivers.
 |>
 |>I would assume transmission is rejected for any such email?
 |
 |Not that I've ever seen but I haven't looked very hard.  I think it's
 |more common to stuff in a \r\n and continue.

You surely do not change body content but by doing the full MIME
dance right.

--steffen
|

---

From: Jeremy Harris <[email-masked]>
Date: Sat, 17 Jul 2021 17:50:28 +0100

On 17/07/2021 17:16, Steffen Nurpmeso wrote:
> is here the question.

See https://cr.yp.to/smtp/8bitmime.html for one opinion, written quite
a long time ago.  Exim has been following that for the last nine years.
-- 
Cheers,
   Jeremy

---

From: Steffen Nurpmeso <[email-masked]>
Date: Sat, 17 Jul 2021 20:13:58 +0200

Jeremy Harris wrote in
 <[email-masked]>:
 |On 17/07/2021 17:16, Steffen Nurpmeso wrote:
 |> Your MUA created the according MIME headers, but whether it used
 |> the 8BITMIME SMTP extension when passing this over to a MTA, that
 |> is here the question.
 |
 |See https://cr.yp.to/smtp/8bitmime.html for one opinion, written quite
 |a long time ago.  Exim has been following that for the last nine years.

I have to say i never quite understood the "an email message must
be human readable text" approach.  It is fine in Plan9 Nupas or
something, were (the individual parts of) your mail messages are
converted to files which are accessible with the normal Unix text
processing tool chain, via a "pseudo" filesystem.  But for over-
the-wire, or even local storage, noone ever cared (mostly) for
binary storage of databases, office documents, calendars or
whatever.  I mean yes of course they do, but mostly not in respect
to "i can use my normal Unix text toolchain on those files".
(Just in case anyone feels RTF or XML or VCARD or JSON or whatever
is something really humanoid.)

I for one hate it that i have to add JSON parsing for SMTP and
other protocols just for the OAUTH authentication handshake, where
K1=V1\0K2=V2\0\0 would do, (i will use jsmn, many open source
people do, also from mailers, take nmh for example, but mind you,
it maps to "object"s with "length"s, 'thus provides the full JSON
power), but get the "we are so much better than you and you just
do not get it right" attitude when i claim that a mail message in
raw form is just a container of some type.  I have no problem with
MBOX and MIME content encoding, and why Maildir is easier to dig
for humans given the file naming scheme than MBOX is, i never
understood.

So technical 7-bit limits have long vanished, and 8-bit
with many NULs (given the UTF-16 common in some worlds, and of
course in binary anyway) is no problem, for message data as such
i never got that message/global and everything is just human text
thing.  Calling the sewer that a real life message is human text
is a cultural bankruptcy and philosophically contemptible.  Not
that engineers ever cared those aspects of life (as long as the
code of conduct is followed), but it may be mentioned once.
And while mentioning things, one or two months ago i created
a Microsoft Outlook account because i wanted to check OAuth
updates that my little MUA needs, and whereas IMAP just worked out
of the box POP3 did not at all even though checked, and SMTP also
did not because i never knew, and when after several unsuccessful
tries i was sending 

  Date: Sat, 05 Jun 2021 23:02:56 +0200
  From: Steffen Nurpmeso <[email-masked]>
  To: [email-masked]
  Subject: Please, i want SMTP for microsoft
  Message-ID: <[email-masked]>

  Yea??

they locked my account and want my phone number to get out of
that, because of policy violation, and so i thought it was time to
quit.  However i was long enough there to find it applicable to
add several _classes_ of mail headers to my ignore list (^IronPort
^MGA ^Spam ^(Accept|Content)-Language ^Thread-) (for message
saving only).

A nice Sunday everybody.

--steffen
|

---

From: "John Levine" <[email-masked]>
Date: 17 Jul 2021 18:09:40 -0400

>the 8BITMIME SMTP extension when passing this over to a MTA, that

The answer is yes.  My MTA is qmail which has never recoded stuff on the fly.
Neither does Postfix or sendmail, at least not with their default settings.

>are quite liberal SMTP engines around.

Everyone still accepts everything they've accepted over the past
30 years.  I tried sending a message with Chinese text from Hotmail/Outlook
and it arrived with a MIME GB2312 encoding.  Yuck, but it worked.

>You surely do not change body content but by doing the full MIME

On outgoing mail, sure.  On ill-formatted incoming mail where your options are limited,
not necessarily.

R's,
John

---

From: "John Levine" <[email-masked]>
Date: 17 Jul 2021 18:17:02 -0400

It appears that Steffen Nurpmeso  <[email-masked]> said:
>be human readable text" approach.

djb wrote that in 1998 when MUAs that support MIME were a lot less ubiquitous
than they are now, and I don't know anyone who objects to base64 encoding
of PDFs and JPEGs.  But 7bit MTAs were gone in 1998 and they're even more
gone now so there is no point in encoding anything that is an 8 bit extended
ASCII like UTF-8 or ISO-8859-x.

R's,
John

---

From: Viktor Dukhovni <[email-masked]>
Date: Sat, 17 Jul 2021 21:56:27 -0400

On Sat, Jul 17, 2021 at 06:09:40PM -0400, John Levine wrote:

> My MTA is qmail which has never recoded stuff on the fly.  Neither
> does Postfix or sendmail, at least not with their default settings.

Actually, given an 8BITMIME message and a nexthop SMTP server that does
not support 8BITMIME, Postfix will in fact (by default) do on the fly
MIME quotable-printable encoding.

    http://www.postfix.org/postconf.5.html#disable_mime_output_conversion

-- 
    Viktor.

---

From: Ned Freed <[email-masked]>
Date: Sat, 17 Jul 2021 19:06:34 -0700 (PDT)

>     http://www.postfix.org/postconf.5.html#disable_mime_output_conversion

Our MTA defaults to this as well. But it's trivial to switch off, and I have no
way of knowing how many sites do so.

				Ned

---

From: John C Klensin <[email-masked]>
Date: Sat, 17 Jul 2021 22:42:41 -0400

--On Saturday, July 17, 2021 19:06 -0700 Ned Freed
<[email-masked]> wrote:

>> Actually, given an 8BITMIME message and a nexthop SMTP server
>> that does not support 8BITMIME, Postfix will in fact (by
>> default) do on the fly MIME quotable-printable encoding.
> 
> Our MTA defaults to this as well. But it's trivial to switch
> off, and I have no way of knowing how many sites do so.

Completely out of curiosity, as the amount of traffic that does
not use Latin script increases (IMO, quoted-printable is far
less useful when most of the characters are outside the ASCII
repertoire), do either of you do the fairly minimal analysis
needed to decide when Base64 would be more compact and no less
useful?  Or provide a switch s.t. if encoding is to be done,
Base64 might be preferred by the configuration instead of
quoted-printable?   It seems like that might be worthwhile if
one were carrying a significant amount of non-Latin-script,
traffic.

of course, none of this has anything to do with changes that
might be needed in rfc5321bis, AFAICT.

best,
   john

---

From: "John Levine" <[email-masked]>
Date: 17 Jul 2021 23:09:14 -0400

>    http://www.postfix.org/postconf.5.html#disable_mime_output_conversion

I believe it, but I don't believe it's useful any more.

IN 2019 I collected some stats for the UASG, going through TLD zone files,
finding hosts with MX records, and sniffing the SMTP EHLO response for 8BITMIME and
SMTPUTF8.  While I didn't find all that many advertising SMTPUTF8, the
number that advertised 8BITMIME rounded to 100% to several digits of precision.

R's,
John

---

From: Viktor Dukhovni <[email-masked]>
Date: Sun, 18 Jul 2021 00:18:38 -0400

On Sat, Jul 17, 2021 at 10:42:41PM -0400, John C Klensin wrote:

> useful?

No, because message sizes are rarely dominated by the text parts.  The
large attachments that dominate message sizes are typically already
Base64 encoded.  Just using quoted-printable is not a problem for the
8bit text, even when not optimal.

-- 
    Viktor.

---

From: John C Klensin <[email-masked]>
Date: Sun, 18 Jul 2021 00:49:06 -0400

--On Sunday, July 18, 2021 00:18 -0400 Viktor Dukhovni
<[email-masked]> wrote:

>> Completely out of curiosity, as the amount of traffic that
>> does not use Latin script increases (IMO, quoted-printable is
>> far less useful when most of the characters are outside the
>> ASCII repertoire), do either of you do the fairly minimal
>> analysis needed to decide when Base64 would be more compact
>> and no less useful?
> 
> No, because message sizes are rarely dominated by the text
> parts.  The large attachments that dominate message sizes are
> typically already Base64 encoded.  Just using quoted-printable
> is not a problem for the 8bit text, even when not optimal.

Makes sense, especially if there are non-text body parts.  And
John Levine's observation that 8BITMIME appears to be
universally supported at this stage makes the whole issue moot
although, perhaps just out of habit, I continue to worry about
people who have much worse Internet connections than those
participating in this discussion..

tnanks,
   john

---

From: Ned Freed <[email-masked]>
Date: Sun, 18 Jul 2021 16:26:17 -0700 (PDT)

> needed to decide when Base64 would be more compact and no less

In our case it depends on when the downgrade occurs. If we're doing it
on the fly with an open SMTP connection we just go with a default; we don't
do the analysis. If we're doing it earlier when we have the data we do
the calculation and pick the better encoding.

> Or provide a switch s.t. if encoding is to be done,

I've had storing the analysis results on the to-do list literally for decades.
But it's never been a sufficient issue that it's ever budged from its position
near the bottom of the list. At this point it's unlikely it ever will.

> might be needed in rfc5321bis, AFAICT.

No, it doesn't.

				Ned

---

From: Steffen Nurpmeso <[email-masked]>
Date: Mon, 19 Jul 2021 15:17:21 +0200

John Levine wrote in
 <[email-masked]>:
 |It appears that Steffen Nurpmeso  <[email-masked]> said:
 |>Jeremy Harris wrote in
 |> <[email-masked]>:
 |>|On 17/07/2021 17:16, Steffen Nurpmeso wrote:
 |>|> Your MUA created the according MIME headers, but whether it used
 |>|> the 8BITMIME SMTP extension when passing this over to a MTA, that
 |>|> is here the question.
 |>|
 |>|See https://cr.yp.to/smtp/8bitmime.html for one opinion, written quite
 |>|a long time ago.  Exim has been following that for the last nine years.
 |>
 |>I have to say i never quite understood the "an email message must
 |>be human readable text" approach.
 |
 |djb wrote that in 1998 when MUAs that support MIME were a lot less \
 |ubiquitous
 |than they are now, and I don't know anyone who objects to base64 encoding
 |of PDFs and JPEGs.  But 7bit MTAs were gone in 1998 and they're even more
 |gone now so there is no point in encoding anything that is an 8 bit \
 |extended
 |ASCII like UTF-8 or ISO-8859-x.

Well the paper may have inspired people to try out and use pure
8-bit paths for good twenty years ago.
..and of course except message generators have to because of the
constraints (length limits, ^From_, invalid bytes) which do exist.
And attachments / signatures and massive header pollution also
make a raw internet message unpleasant to look at.  So but for
debugging maybe any desire or even praising of message/global
seemed a bit artificial to me.

--steffen
|

---

From: Gene Hightower <[email-masked]>
Date: Fri, 23 Jul 2021 07:41:50 -0700

> to understand how widely it's enforced by mail receivers.

I routinely get messages from ups.com that contain an HTML part all in
one line.

With "Content-Type: text/html; charset=UTF-8" and no
Content-Transfer-Encoding. Tens of thousands of bytes, which are UTF-8
code units.

My MTA spits a warning to the logs when a message comes in (via DATA,
not BDAT) containing a line >1000 bytes.

It's not very common, but I do see it. Most email I see nowadays is
Quoted-Printable.