Subject: [Emailcore] Long threads and "References" header (5322 section 3.6.4)

From: Viktor Dukhovni <[email-masked]>
Date: Thu, 11 Mar 2021 03:24:39 -0200

As email threads extend into deep chains on lists such as this one,
the list of message-ids appearing in "References:" headers can get
rather long.

Some MUAs (used to last time I checked, but not admittedly recently,
are they all good citizens now???) flatten the list into a single
physical line, forcing the MSA to attempt to fold the list in order
to stay under the 998 + CRLF byte count limit for physical lines in SMTP.

Such folding is not always particularly cleverly implemented, there
is not always a fancy parser for this embedded in the MTA.

The result is that the "References" header is not infrequently mangled,
with fragments of various message-ids padding out the list.

I'd like to suggest a recommendation that MUAs keep only a short
list of the most recent message-ids when appending a new one, keeping
the overall length of the list below the 998 limit, allowing it to fit
on a single line without folding.

Below is an example of a "starting to get noticeably long" References
list from the TLS WG.  It is nicely folded over multiple lines, so
not mangled, but it risks damage if handled by an MUA that unfolds
the lines, and in any case, it is not clear that quite so many
references are needed to reconstruct the thread.  Just a few should
be enough, unless the recipient is missing many messages in the middle
of the thread:

References: <[email-masked]>
 <[email-masked]>
 <[email-masked]>
 <[email-masked]>
 <[email-masked]>
 <[email-masked]>
 <[email-masked]>
 <[email-masked]>
 <[email-masked]>
 <[email-masked]>
 <[email-masked]>
 <[email-masked]>
 <CABcZeBMAtmPfG0rctvO8UvnhPqY1etk=[email-masked]>
 <[email-masked]>
 <CABcZeBPES99+xo16=[email-masked]>
 <[email-masked]>
 <CABcZeBMeR-kH_P_Lq9X8sOCvZ=u8_tGEOE2QErKX--Tk3cEg=[email-masked]>
 <[email-masked]>
 <CAOgPGoCANLd0hisu5cLtb=[email-masked]>
 <[email-masked]>
 <CAOgPGoA1boAcWyV9cn6_uXvCRPEBkAUeg2f8u8FaPPk=[email-masked]>
 <[email-masked]>
In-Reply-To: <[email-masked]>
From: Joseph Salowey <[email-masked]>
Date: Mon, 1 Feb 2021 13:52:04 -0800
Message-ID: <CAOgPGoChCZhc9BON0yvV=[email-masked]>

-- 
	Viktor.

---

From: "Peter J. Holzer" <[email-masked]>
Date: Thu, 11 Mar 2021 14:12:29 +0100

> on a single line without folding.

RFC 5537 contains such a rule (it's even a MUST, not just a
recommendation):

   If the resulting References header field would, after unfolding,
   exceed 998 characters in length (including its field name but not the
   final CRLF), it MUST be trimmed (and otherwise MAY be trimmed).
   Trimming means removing any number of message identifiers from its
   content, except that the first message identifier and the last two
   MUST NOT be removed.

The References header was originally borrowed from Usenet (IIRC there
was a References header in RFC 822, but with different semantics), so it
would make sense to adopt the trimming rule, too (some MUAs may actually
already do this).

        hp

-- 
   _  | Peter J. Holzer    | Story must make more sense than reality.
|_|_) |                    |
| |   | [email-masked]         |    -- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |       challenge!"

---

From: Pete Resnick <[email-masked]>
Date: Thu, 11 Mar 2021 07:28:52 -0600

> would make sense to adopt the trimming rule, too (some MUAs may 

Trimming was suggested during the DRUMS WG work on 2822 and was rejected 
at the time. Part of the discussion was that Usenet clients were known 
to screw it up, but email clients did not. Unless we have current 
examples of clients that mess this up, I'd be inclined not to re-open 
the issue.

pr
-- 
Pete Resnick https://www.episteme.net/
All connections to the world are tenuous at best

---

From: Viktor Dukhovni <[email-masked]>
Date: Thu, 11 Mar 2021 15:11:17 -0200

> On Mar 11, 2021, at 11:28 AM, Pete Resnick <[email-masked]> wrote:
> 
> Trimming was suggested during the DRUMS WG work on 2822 and was rejected at the time.
> Part of the discussion was that Usenet clients were known to screw it up, but email
> clients did not. Unless we have current examples of clients that mess this up, I'd be
> inclined not to re-open the issue.

I found a recent example, see below. Folded in the middle of the id starting
with "<CACd" at 998 bytes!

-- 
	Viktor.

From: Michael Richardson <[email-masked]>
To: Eric Rescorla <[email-masked]>, "\<tls\@ietf.org\>" <[email-masked]>,
 opsawg <[email-masked]>
In-reply-to: <[email-masked]>
References: <[email-masked]>
 <[email-masked]>
 <[email-masked]>
 <[email-masked]>
 <[email-masked]>
 <CAFpG3gdRUAAYmvV1+m=[email-masked]>
 <CAHbrMsD=BOxYLaJyOkv-t9p+Cm4cEpOui7sQdL9Mmfi=[email-masked]>
 <[email-masked]>
 <CAHbrMsBLrGsg+beMhNadqs+QC9icOsGLxLJYGghEg339=[email-masked]>
 <[email-masked]>
 <[email-masked]>
 <CAHbrMsBOhZ+sMxM3KJYT=[email-masked]>
 <[email-masked]>
 <[email-masked]>
 <[email-masked]>
 <CAFpG3gedM=[email-masked]> <CACd
 [email-masked]>
 <[email-masked]>
Comments: In-reply-to Eric Rescorla <[email-masked]>
 message dated "Wed, 16 Sep 2020 14:02:29 -0700."
X-Mailer: MH-E 8.6+git; nmh 1.7+dev; GNU Emacs 26.3
MIME-Version: 1.0
Date: Fri, 18 Sep 2020 18:12:51 -0400
Message-ID: <107735.1600467171@dooku>
Archived-At: <https://mailarchive.ietf.org/arch/msg/tls/sd-b1Lb4m0keysTetCtCUHxqIjI>
Subject: Re: [TLS] [OPSAWG]  CALL FOR ADOPTION: draft-reddy-opsawg-mud-tls

---

From: Dave Crocker <[email-masked]>
Date: Thu, 11 Mar 2021 09:13:51 -0800

> the list of message-ids appearing in "References:" headers can get

Indeed.  But this is an implementation and/or operations issue, not a 
protocol one.

Some people write overly-long emails.  Shall we specify an RFC5322bis 
trimming rule on the messages?

d/

-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: Viktor Dukhovni <[email-masked]>
Date: Thu, 11 Mar 2021 15:35:03 -0200

> On Mar 11, 2021, at 3:13 PM, Dave Crocker <[email-masked]> wrote:
> 
> Indeed.  But this is an implementation and/or operations issue, not a protocol one.

I very much disagree.  The protocol specifies length limits on physical
lines of headers, and yet it encourages an ever growing "References"
header as messages bounce back and forth in a thread.  With message
ids of roughly ~64 bytes each, it takes just 16 to cross the physical
line limit, and some fraction of clients don't fold (and even unfold)
the references list.

It makes sense to guard against damage.

> Some people write overly-long emails.

That's a bit flippant.  The SMTP protocol has a "SIZE" parameter that
takes care of excessively large message bodies.  (Basically never a
result of "writing" overly long emails, rather overly large or too many
attachments).

In normal threaded conversations message sizes do not grow nearly so
large, and corruption of the message structure does not result from
size limits.

> Shall we specify an RFC5322bis trimming rule on the messages?

No, the strawman fails common sense.

On the other hand, trimming some references, while leaving the oldest and
one or two most recent, preserves all the essential information without
bloating the message headers with an ever-growing fragile payload.

It is basic robustness.

-- 
	Viktor.

---

From: Dave Crocker <[email-masked]>
Date: Thu, 11 Mar 2021 09:39:08 -0800

On 3/11/2021 9:35 AM, Viktor Dukhovni wrote:
> lines of headers, 

That limit is due to concerns over handling or processing limits, not 
over human usage limits.

You have cast your concern in terms of unwieldiness, not in terms 
computing limits.

> It makes sense to guard against damage.

Please document known cases of damage.  This is not a new facility.  If 
there are actual problems with it, documentation of computing problems 
should be readily available.

d/

---

From: Viktor Dukhovni <[email-masked]>
Date: Thu, 11 Mar 2021 15:55:32 -0200

> On Mar 11, 2021, at 3:39 PM, Dave Crocker <[email-masked]> wrote:
> 
> Please document known cases of damage.  This is not a new facility.  If there are actual problems with it, documentation of computing problems should be readily available.

I posted an example where a message id in the references list
was damaged in transit because a physical line length of 999+
bytes resulted from an unfolded References header.

Below is another, where the 3rd from last message id:

  <51E7195E-8B2E-4B83-8 [email-masked]>

got folded and then unfolded and has an extraneous SPACE.
This makes threading inaccurate and bloats mail headers:

From: "Salz, Rich" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Thread-Topic: Birthday attack on long-running 3DES connections
Thread-Index: AQHRoCokd7eX7dzmxE6z5Vd5hLYDWZ+rDftAgIUzH4D//77TgA==
Date: Fri, 29 Jul 2016 13:50:39 +0000
Message-ID: <[email-masked]>
References: <[email-masked]>
 <[email-masked]>
 <[email-masked]>
 <[email-masked]>
 <CAL9PXLyCjj6ha2=[email-masked]>
 <[email-masked]>
 <[email-masked]>
 <CABkgnnViQvFDXGovxmScUTRYxb43LLWY=[email-masked]>
 <[email-masked]>
 <[email-masked]>
 <[email-masked]>
 <[email-masked]>
 <[email-masked]>
 <[email-masked]>
 <51E7195E-8B2E-4B83-8 [email-masked]>
 <[email-masked]>
 <[email-masked]>
In-Reply-To: <[email-masked]>

-- 
	Viktor.

P.S. I set "Reply-To:" to the list.  No need to also Cc: me directly.

---

From: Pete Resnick <[email-masked]>
Date: Thu, 11 Mar 2021 13:52:56 -0600

On 11 Mar 2021, at 11:11, Viktor Dukhovni wrote:

> In-reply-to: 
> Archived-At: 
> <https://mailarchive.ietf.org/arch/msg/tls/sd-b1Lb4m0keysTetCtCUHxqIjI>

Interesting. Bad re-folding. Still parsable under the obsolete syntax, 
but obviously a problem.

Do we know what piece of software did this? Is it Michael's MH? It 
appears the way you have it in the IMAP archive too, so I expect it's 
either Michael's or the IETF server; there doesn't appear to be anything 
in between.

If we do address this, perhaps it's part of an "implementation advice" 
in the A/S instead of in 5322bis. (Yes, there is some implementation 
advice already in 5322bis, but limiting changes to the base spec seems 
prudent.)

pr

---

From: Russ Allbery <[email-masked]>
Date: Thu, 11 Mar 2021 13:07:35 -0800

> examples of clients that mess this up, I'd be inclined not to re-open

My recollection (it's been years now, so this may not be entirely correct)
is that Usenet had a few other problems: References was absolutely central
to the protocol (historically, email is less interested in accurate
threading), most clients did not fold the header, and running over 998
octets was common.  Also, Usenet had the problem that some agents were
doing it entirely wrong: not just misfolding, but trimming by deleting
initial message IDs.

Usenet servers also put the References header into the overview database,
and there are widespread implementations (even INN) that impose hard
limits on the total length of overview lines and will malfunction in
various ways if the References header gets too long.

I don't think most of those concerns apply in the same way to email.

-- 
Russ Allbery ([email-masked])             <https://www.eyrie.org/~eagle/>

---

From: Viktor Dukhovni <[email-masked]>
Date: Thu, 11 Mar 2021 19:16:00 -0200

> On Mar 11, 2021, at 5:52 PM, Pete Resnick <[email-masked]> wrote:
> 
> Interesting. Bad re-folding. Still parsable under the obsolete syntax, but obviously a problem.

The "parseable" status does not always persist, because later in the thread
the SPACE frequently ends up in the middle of a line rather on a line boundary.
I recall seeing much worse mangling resulting from rounds of folding and unfolding
in the past, but these are the examples in my mailbox now.

> Do we know what piece of software did this? Is it Michael's MH?

Plausibly his MH generated a long unfolded input and then it got
folded somewhat crudely (garbage in, garbage out) in transit.

> It appears the way you have it in the IMAP archive too, so I expect
> it's either Michael's or the IETF server; there doesn't appear to
> be anything in between.

Yes, I expect it is both.  Michael unfolded, and the server refolded.

> advice already in 5322bis, but limiting changes to the base spec seems

I'd prefer to have be in whichever document is more likely to be read
by implementors.  Perhaps just talking about it on mailop and ietf-smtp
once there is a published document to lean back on is really the best
way to have it addressed in current implementations.

I still think a change is warranted in 5322, which specifically recommends
to keep appending (regardless of current length).  There's no good reason
to keep growing that header without bounds.

-- 
	Viktor.