Subject: [Emailcore] Ticket #4: Exploders seem to be prohibited from adding List-* header fields

From: Alexey Melnikov <[email-masked]>
Date: Mon, 16 Nov 2020 16:53:16 +0000

Dear collegues,

I would like us to discuss the following ticket for rfc5321bis:

3.9. Mailing Lists and Aliases

     [...] When a message is
     delivered or forwarded to each address of an expanded list form, the
     return address in the envelope ("MAIL FROM:") MUST be changed to be
     the address of a person or other entity who administers the list.
     However, in this case, the message header section (RFC 5322 [11])
     MUST be left unchanged; in particular, the "From" field of the header
     section is unaffected.

As was already pointed out on the mailing list, the text "the message 
header section (RFC 5322 [11]) MUST be left unchanged" seems to prohibit 
addition of List-* header fields.

Strawman proposal how to address this issue:

OLD:

     However, in this case, the message header section (RFC 5322 [11])
     MUST be left unchanged; in particular, the "From" field of the header

NEW:

     However, in this case, the message header section (RFC 5322 [11])

     MUST NOT be modified, except for adding header fields related to

     mailing list processing (e.g. List-* [RFC4021][RFC8058]) and/or trace

     header fields [rfc5322bis]; [...]

Please discuss if the proposed text is an improvement and whether it can 
be further improved.

(I am aware that this depends on ticket #7 "Better definition for trace 
header fields", which includes suggestion to replace "trace header 
field" with something else. But once the ticket #7 is resolved, this 
should hopefully will have little impact on the above proposed text.)

Best Regards,

Alexey, as a co-chair

---

From: Dave Crocker <[email-masked]>
Date: Mon, 16 Nov 2020 09:04:01 -0800

On 11/16/2020 8:53 AM, Alexey Melnikov wrote:
> ...

This document should remove all references to aliases and mailing 
lists.  They are, quite simply, out of scope for basic email transfer, 
which is what SMTP does.

Also  they have become complicated topics that take more than the 
superficial treatment this specification can (or should) provide.

Aliases and Mailing Lists operate /after delivery/.  That is, after SMTP 
has done its job.  They therefore operate at a layer above SMTP.

As a small example, the Must Not stricture is clearly wrong with respect 
to modern handling of the From: field in some cases.

While there is demonstrated resistance to referencing RFC 5598, it's 
still worth considering.

d/

-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: "John Levine" <[email-masked]>
Date: 16 Nov 2020 18:52:18 -0500

>addition of List-* header fields.

It is understanding that the intention here was to prohibit the kind
of rewriting that submission servers do, not to prohibit what mailing
lists have done for decades.

If we change the language I'd say something like it cannot change
existing headers except for the ones whose definition says they can be
replaced, which at this point means Authentication-Results.

R's,
John

---

From: Ned Freed <[email-masked]>
Date: Mon, 16 Nov 2020 23:31:44 -0800 (PST)

> lists.  They are, quite simply, out of scope for basic email transfer,

The problem is that aliases can and do operate at the MTA level without going
through any sort of "delivery" processing. Pretty much every MTA in existence -
including those that do what they consider to be pure relay with nothing they
regard as MDA functionality at all - has the ability to do some form of RCPT TO
address substitution. Or if not substitution, differing interpretations.

I realize that the email architecture document makes a case for treating even
the simple replacement of the RCPT TO value as an MDA thing. But we really are
pretty far out in the rough in regards to what actual implementations do here.

FWIW, I accept part of the blame for the current sorry state of affairs: The
conflation of RCPT TO replacement with delivery goes back to the NOTARY
documents, where we made a distinction between delivery, which includes RCPT TO
replacement and not much else, and final delivery, which includes all the other
stuff like adding Return-Path, success DSNs, and so on. I now think it was a
mistake to call this delivery.

And of course once you want to talk about aliases, you have to distinguish
them from mailing lists. With all that implies.

I don't have any good answers here. But I'm very uncomfortable with a solution
that involves removing this material entirely.

> has done its job.  They therefore operate at a layer above SMTP.

Er, no. Aliases as defined in the NOTARY speficiation do not invoke all aspects
of final delivery processing. And that's how they work in practice.

> While there is demonstrated resistance to referencing RFC 5598, it's

FWIW< RFC 5598 actually isn't entirely clear on this either - it defines
aliases, but doesn't make it clear the thing it's talking about is what's being
done by, say, wholesale replacement of one domain with another in RCPT TO
fields.

				Ned

---

From: Alessandro Vesely <[email-masked]>
Date: Tue, 17 Nov 2020 12:52:05 +0100

> 3.9. Mailing Lists and Aliases

Can we change that title?  It is itself misleading.  I'd propose "Forwarding".

IMHO, aliasing to internal users, such as Sales => Mike, Bob, is very different 
from aliasing to external addresses, such as is done for email address portability.

> As was already pointed out on the mailing list, the text "the message header 
> section (RFC 5322 [11]) MUST be left unchanged" seems to prohibit addition of 
> List-* header fields.

How about subject tags?

> Please discuss if the proposed text is an improvement and whether it can be 
> further improved.

I'd drop the MUST NOT altogether.  The MAIL FROM discussion is pristine. 
Trying to limit what part of a message an MDA can change makes no sense.

Best
Ale
--

---

From: Ned Freed <[email-masked]>
Date: Tue, 17 Nov 2020 05:13:06 -0800 (PST)

> Can we change that title?  It is itself misleading.  I'd propose "Forwarding".

The terms agree with those in the email architecture document, and the NOTARY
documents before that. What would be misleading is to switch to different
terms after ~25 years.

> from aliasing to external addresses, such as is done for email address portability.

In what way is it different? What semantic changes are involved? Please
be very specific.

> How about subject tags?

Nobody is saying that the present text is acceptable. It isn't.

> Trying to limit what part of a message an MDA can change makes no sense.

In the list case, yes, I agree. The alias case is mor subtle, and I for one
think we have a problem with how we've defined it that goes beyond this
txxt.

				Ned

---

From: Dave Crocker <[email-masked]>
Date: Tue, 17 Nov 2020 09:22:14 -0800

Take a seat.  This is going to take awhile...

(I've added the Gatewaying section to the mix, since it, too, is trying 
to cover a complex topic that really is outside the scope of SMTP proper.)

Discussion about Internet Mail architecture is typically muddled by 
inconsistent use of framework and terminology. It also suffers from the 
common malady of conflating software implementation with abstract 
network architecture.  The purpose of RFC 5598 was to attempt to get 
common framework and language provided about the network architecture of 
the service.

SMTP, itself, is a single-hop transfer protocol.  Its specification is 
elaborated for multiple SMTP hops, to cover submission-to-delivery 
normative language, to formulate an end-to-end service within a coherent 
SMTP environment.

What its specification should not and cannot do is cover material that 
is outside the scope of that homogeneous service, especially as such 
material has become far more complex than the SMTP specification can do 
justice to.

That RFC 821 introduced language about aliases and mailing lists was 
reasonable, at that time, but somewhere over the decades, the material 
became grossly inadequate and this cannot be reasonably be fixed with 
the SMTP spec, proper.  Worse, having any meaningful language about 
these topic in this specification gives the impression that the topics 
have been covered; whereas readers need to instead find more extensive 
material.

      A specification like SMTP gets a lot simpler when ways are found to
      divide and conquer, by carving complexities that are not essential
      to the core functions.  A basic test for this is to ask a) will the
      core service's specification suffer by the removal, and b) will
      losing the text in the document adversely affect the operational
      service?

For gatewaying, aliasing, and mailing lists, I believe both answers are 
no.  They are important topics, of course, but they are not important to 
SMTP itself.

To the extent that there is a strong case that the answer to the second 
question is yes, then what is needed is a separate document, not the -- 
by today's perspective -- half-hearted coverage that is possible within 
the main SMTP specification.  These are specialty topics and they need 
special focus.

As far as adding gatewaying to the recommendation for removal...

By definition, gatewaying has to do with circumstances outside of the 
homogeneous SMTP environment.  As such, the SMTP document, itself, 
cannot dictate pretty much of anything about it.

Worse is that gatewaying tends more towards heuristics, rather than 
simple, strict, pervasive rules, by virtue of having to try to connect 
together two services that are, by definition, not the same, and usually 
are quite substantially different.  Sometimes they are quite close, such 
as CSNET had between dial-up and SMTP, but that was because the dial-up 
side was designed to be compatible.  But mostly, gatewaying tries to 
connect two environments that have independent histories and therefore 
distinctly different sets of design and semantics decisions.

So while gatewaying is an extremely important topic, it's not one that 
the SMTP document, itself, can or should cover.

inline...

>> lists.  They are, quite simply, out of scope for basic email
>> transfer, which is what SMTP does.
> 
> The problem is that aliases can and do operate at the MTA level
> without going through any sort of "delivery" processing.

I understand the implementation basis for that view, of course, but the 
hangup is on the definition of delivery.  A simplistic, operational test 
is whether the message got to the address that was specified and did not 
get rejected.  In terms of RFC 5598, the definition of the term 
'delivery' now seems to me weak and well-hidden but not useless, even 
for the current discussion:

    4.3.3.  Mail Delivery Agent (MDA)

    A transfer of responsibility from the MHS to a Recipient's
    environment (mailbox) is called "delivery".

In spite of being implemented down in typical MTA software, the reality 
of an aliasing mechanism is that the RCPTO transformation is an action 
under control of the designated recipient, not the author, nor the basic 
handling service.  However briefly, control passes to the recipient's 
environment, before the message is -- in abstract terms -- re-posted to 
its alternative recipient(s).

Could aliasing be implemented as a streamlined mailing list service? 
Yes.  That is isn't is for efficiency and implementation simplicity 
rather than architectural necessity.

> I realize that the email architecture document makes a case for
> treating even the simple replacement of the RCPT TO value as an MDA
> thing. But we really are pretty far out in the rough in regards to
> what actual implementations do here.

But not in regards to the question of where the responsibility for the 
action lies.  It lies with the original addressee, not the author and, 
really, not the transfer service.

> And of course once you want to talk about aliases, you have to
> distinguish them from mailing lists. With all that implies.

Yup.  And the distinction is useful because aliasing is so constrained 
and mailing lists are so not.  However that does not mean they occupy 
different places in the email service architecture.

One aspect of the lengthy process of developing RFC 5598 that I found 
personally quite educational was the focus on 'responsibility'.  The 
most extreme demonstration of this was the view that emerged -- and 
needed some promoting to convince me -- that MSAs and MDAs have a 
MHS/User transition point /within/ them rather than having that 
responsibility transfer happen between architectural nodes.

> I don't have any good answers here. But I'm very uncomfortable with a
>  solution that involves removing this material entirely.

What happens to the SMTP service if these sections are removed?  What 
would it take to remedy this, outside of the SMTP-proper specification?

>> Aliases and Mailing Lists operate /after delivery/.  That is, after
>> SMTP has done its job.  They therefore operate at a layer above
>> SMTP.
> 
> Er, no. Aliases as defined in the NOTARY speficiation do not invoke
> all aspects of final delivery processing. And that's how they work in
> practice.
> 
>> As a small example, the Must Not stricture is clearly wrong with
>> respect to modern handling of the From: field in some cases.
> 
>> While there is demonstrated resistance to referencing RFC 5598,
>> it's still worth considering.
> 
> FWIW< RFC 5598 actually isn't entirely clear on this either - it
> defines aliases, but doesn't make it clear the thing it's talking
> about is what's being done by, say, wholesale replacement of one
> domain with another in RCPT TO fields.

I'm not in love with the language in RFC 5598's Section 5.1, although 
for the current discussion some of it seems interesting (although 
significantly improveable):

    ...the message continues
    through the transfer service, for delivery to one or more alternate
    addresses.  Although typically implemented as part of an MDA, this
    facility is a Recipient function.  It resubmits the message, although
    all handling information except the envelope Recipient
    (rfc5321.RcptTo) address is retained.

I'd have wished that the 'continues' language emphasized the semantic 
transition, that is only eventually declared by the 'resubmits'.

d/

---

From: Alessandro Vesely <[email-masked]>
Date: Wed, 18 Nov 2020 14:11:17 +0100

> documents before that. What would be misleading is to switch to different

RFC 5598 treats mailing list and aliases as two of five cases of "Mediators". 
In that sense, it is much more general.

The NOTARY documents seem to derive from Section 5.3.6 of RFC 1123, which seems 
to describe some oldish software and is not general at all.  Nowadays mailing 
list managers are well detached from MTA software.

Perhaps, the best resolution is to drop Section 3.9 entirely, as Dave suggested 
in December last year[*].

Otherwise, focusing it on terminology can be useful to straighten out some 
basic email concepts.  "Forwarding", for example, deserves a definition.

For standardization concerns, in neither case any existing implementation would 
be affected by changing that section of the document.

> In what way is it different? What semantic changes are involved? Please

Deliverability.

> In the list case, yes, I agree. The alias case is more subtle, and I for one
> text.

Agreed.

Best
Ale
-- 

[*] bullet 4 in:
https://mailarchive.ietf.org/arch/msg/ietf-smtp/K1MpIrubUzq6-90Ikdl4yuh7n3o

---

From: Russ Allbery <[email-masked]>
Date: Sat, 21 Nov 2020 13:55:17 -0800

> distinguish them from mailing lists. With all that implies.

If there is a reason to retain this section to talk about behavior in the
alias case (and I think there might be, although I'm not certain), then
drawing this distinction, although difficult, seems worthwhile.

Given current practice, I think SMTP should treat a managed mailing list
as a final recipient of a message.  The message is delivered to the
mailing list, at which point SMTP's responsibilities end.  The mailing
list may then choose send a new message with a remarkable similarity to
the original message to all the list members in a way akin to aliasing.
But it may also add new headers, remove headers, rewrite the From and/or
Reply-To headers, change the Subject header, merge multiple messages
together into a digest message, selectively discard some messages, hold
messages for moderation, remove elements from the body of the message, add
elements to the body of the message, decrypt the body and re-encrypt it
with different keys, interpret the message as control commands, and many
other things, all of which are seen in practice and which seem likely to
continue.

In other words, I think the transformations commonly seen in practice (and
desired by mailing list members and administrators) go considerably beyond
the topic of this ticket.

I think it's relatively obvious there's no way to model all of that
behavior in the SMTP standards, particularly given the narrow goal of the
current work.  If someone wanted to document best practices for mailing
lists, that seems much better done as an entirely separate document.

So, therefore, I think the question is whether to retain something like
the current language but limit its scope in some way to clarify that this
is only about "simple aliasing" (which we'd have to define), or decide
that the benefits of describing aliasing behavior outweigh the effort in
distinguishing that from more complex list handling.

-- 
Russ Allbery ([email-masked])             <https://www.eyrie.org/~eagle/>