Subject: Re: [Emailcore] Ticket #8: Need a registry of header fields that are Ok to add after submission

From: Alexey Melnikov <[email-masked]>
Date: Fri, 23 Jul 2021 13:21:51 +0100

Dear WG participants,

I would like to get closure on this ticket, which was briefly discussed 
on the mailing list and also during IETF 109 & 110. Below is a updated 
strawman text:

===============

Add to the IANA Considerations of rfc5321bis:

IANA is requested to create a new subregistry for email header fields 
that can be added to a message header section by a MSA/MTA/MDA. The new 
subregistry would show whether a header field can be added by a "message 
submission", “relay”, “delivery” system or some combination of them. 
Headers appearing in this subregistry SHOULD also be registered in 
<https://www.iana.org/assignments/message-headers/message-headers.xhtml 
<https://www.iana.org/assignments/message-headers/message-headers.xhtml>> 
(whether it is registered as a Permanent Message Header Field Name or as 
a Provisional Message Header Field Name). The registration template has 
the following fields:

1) Name of the header field name

2) Can be added by an MSA?

3) Can be added by an MTA?

4) Can be added by an MDA?

5) Optional reference field that points to one or more document 
describing the header field.

6) Optional comment

Registration policy for this new subregistry is “Expert Review”. 
Designated Experts should only check correctness of the information in 
the registration template without passing any judgement on usuability of 
a specific header field being registered.

Updates to existing entries undergo the same process as registration of 
new header fields.

===============

Let me know your thoughts.

Best Regards,

Alexey

---

From: Alexey Melnikov <[email-masked]>
Date: Fri, 23 Jul 2021 13:43:10 +0100

On 23/07/2021 13:21, Alexey Melnikov wrote:

> Dear WG participants,
>
> I would like to get closure on this ticket, which was briefly 
> discussed on the mailing list and also during IETF 109 & 110. Below is 
> a updated strawman text:
>
> ===============
>
> that can be added to a message header section by a MSA/MTA/MDA. The 
> new subregistry would show whether a header field can be added by a 
> "message submission", “relay”, “delivery” system or some combination 
> of them. Headers appearing in this subregistry SHOULD also be 
> registered in 
> (whether it is registered as a Permanent Message Header Field Name or 
> as a Provisional Message Header Field Name). The registration template 
> has the following fields:
>
> the registration template without passing any judgement on usuability 
> of a specific header field being registered.
>
> Updates to existing entries undergo the same process as registration 
> of new header fields.
>
> ===============
>
Speaking as a participant: I am happy with this text in rfc5321bis or A/S.

---

From: Dave Crocker <[email-masked]>
Date: Fri, 23 Jul 2021 06:01:14 -0700

> 4) Can be added by an MDA?

Shouldn't the list of choices include MUA (ie, either the author or 
possibly even the recipient)?

d/

-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: Alexey Melnikov <[email-masked]>
Date: Fri, 23 Jul 2021 14:43:25 +0100

Hi Dave,

> possibly even the recipient)?

Yes.

Best Regards,

Alexey

---

From: Alessandro Vesely <[email-masked]>
Date: Sun, 25 Jul 2021 12:36:58 +0200

> 4) Can be added by an MDA?

Could we limit this, or split it into two parts?  As MDAs can re-inject 
messages into the mail system, we should distinguish what headers can be added 
or modified during delivery proper (rMDA, in rfc5598 parlance) — such headers 
will never be transmitted via SMTP.

An example could be Munged-From:, the copy of a From: header that had been 
munged by a MLM to mitigate DMARC damage.  This field is created by the MDA 
before restoring the original value of From:.[*]

Best
Ale
-- 

[*] That can be done after recognizing the original signature, and could work 
likewise with ARC.  See the suggested operation in the MTA filter's man page:
http://www.tana.it/sw/zdkimfilter/zdkimfilter.html#mlmtrans

---

From: Dave Crocker <[email-masked]>
Date: Sun, 25 Jul 2021 11:28:14 -0700

On 7/25/2021 3:36 AM, Alessandro Vesely wrote:
> messages into the mail system, we should distinguish what headers can be 
> added or modified during delivery proper (rMDA, in rfc5598 parlance) — 
> such headers will never be transmitted via SMTP.
> 
> An example could be Munged-From:, the copy of a From: header that had 
> been munged by a MLM to mitigate DMARC damage.  This field is created by 
> the MDA before restoring the original value of From:.[*]

If an MDA is 're-injecting' a message, it is acting as an MUA or, at the 
least, an MSA.

As for limiting who can do what, there might be some benefit in such an 
exercise, but I'd expect it to be a long and painful effort...

d/

---

From: "John Levine" <[email-masked]>
Date: 25 Jul 2021 16:12:12 -0400

>possibly even the recipient)?

I'm not necessarily opposed to this, but what problem is it solving?

I assume we agree that the lists would be descriptive rather than normative, since
it's unlikely anyone's going to retroactively change what their MSAs or MTAs do just
because we put headers in a list.

R's,
John

---

From: Dave Crocker <[email-masked]>
Date: Sun, 25 Jul 2021 13:19:28 -0700

On 7/25/2021 1:12 PM, John Levine wrote:
> It appears that Dave Crocker<[email-masked]>  said:

Just to be clear, your question appears to be about Alexey's original 
posting, rather than my suggested addition.

d/

---

From: "John R Levine" <[email-masked]>
Date: 25 Jul 2021 16:36:13 -0400

> posting, rather than my suggested addition.

Well, both, but yes.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: John C Klensin <[email-masked]>
Date: Sun, 25 Jul 2021 19:30:55 -0400

--On Friday, 23 July, 2021 13:21 +0100 Alexey Melnikov
<[email-masked]> wrote:

> discussed on the mailing list and also during IETF 109 & 110.
> Below is a updated strawman text:
> 
> ===============
> 
> IANA is requested to create a new subregistry for email header
> fields that can be added to a message header section by a
> MSA/MTA/MDA. The new subregistry would show whether a header
> field can be added by a "message submission", "relay",
> "delivery" system or some combination of them. Headers
> appearing in this subregistry SHOULD also be registered in
> <https://www.iana.org/assignments/message-headers/message-head
> ers.xhtml>> (whether it is registered as a Permanent Message
> Header Field Name or as a Provisional Message Header Field
> Name). The registration template has the following fields:
> 
> 5) Optional reference field that points to one or more
> document describing the header field.
> 
> 6) Optional comment
> 
> Registration policy for this new subregistry is "Expert
> Review". Designated Experts should only check correctness of
> the information in the registration template without passing
> any judgement on usuability of a specific header field being
> registered.
> 
> Updates to existing entries undergo the same process as
> registration of new header fields.
> 
> ===============
> 

Personal opinion and opinion as co-author of RFC 6409: Much of
the reason for creating a separate Message Submission
specification was to make it clear that the MSA was allowed to
do things that, in a more abstract world, we expected MUAs to do
and discouraged MTAs from doing.  In other words, the boundary
between MUA and SMA is deliberately unclear; the MUA could
incorporate MSA functions (or not) or could communicate with the
MSA using just about whatever protocol or API the two agreed on,
etc.

So, if this proposal intends to classify MUAs and MSAs as
separate, we'd better have a crisp definition of the boundary
that does  not contradict 6409.  I predict that will be hard.

More generally (and with Editor hat on):

Over the year SMTP specifications have deliberately not
distinguished between MUAs and MSAs or between MDAs assorted
message store functions.  All an SMTP server (receiver) knows is
that it got a message from somewhere (could be an MSA, an MUA
with SMA functionality, or a relay) and all an SMTP client
(sender) knows is that it is delivering the message somewhere.
Even the "final delivery server" language of Section 2.3.3 is a
little fuzzy, still points to MTSs getting messages from MUAs,
which should probably be fixed, and explicitly acknowledges that
attempts to use those terms precisely often does not align with
practice.

While I do not object to the idea of a subregistry as proposed,
it seems to me that, if it is going to be specified in 5321bis,
either clear definitions need to be specified in that document
and section 2.3.3 rewritten to stop the handwaving or there need
to be normative references to a standards track document
(preferably an Internet Standard) that contains those
definitions.  FWIW, I think that doing the work in some separate
specification would incur the same problems, but it would not be
my problem to sort out editorially.

    john

---

From: Dave Crocker <[email-masked]>
Date: Sun, 25 Jul 2021 16:42:19 -0700

> MSA using just about whatever protocol or API the two agreed on,

You appear to be saying that Sections 4.2.1 and 4.3.1 of RFC 5598 are 
not sufficiently clear or complete.

Please elaborate with particulars.

d/

---

From: John C Klensin <[email-masked]>
Date: Sun, 25 Jul 2021 20:40:05 -0400

--On Sunday, 25 July, 2021 16:42 -0700 Dave Crocker
<[email-masked]> wrote:

>> do things that, in a more abstract world, we expected MUAs to
>> do and discouraged MTAs from doing.  In other words, the
>> boundary between MUA and SMA is deliberately unclear; the MUA
>> could incorporate MSA functions (or not) or could communicate
>> with the MSA using just about whatever protocol or API the
>> two agreed on, etc.
> 
> 
> You appear to be saying that Sections 4.2.1 and 4.3.1 of RFC
> 5598 are not sufficiently clear or complete.
> 

I am saying that RFC 5598 is an Informational document, perhaps
in part because some of its sections could not get consensus.
If you want to propose to some appropriate AD that it be put out
for Last Call to move it to Proposed Standard or BCP, or
recommend chartering a WG for the purpose (or assigning it to an
existing AD), by all means go for it.

Or you could convince the WG to tell me to go ahead and create a
normative reference to 5598 5598 to and use the exception
procedure, but I imagine some of us would argue that would be
tantamount to moving standards track and hence out of charter.

   john

---

From: "John Levine" <[email-masked]>
Date: 25 Jul 2021 21:25:50 -0400

>tantamount to moving standards track and hence out of charter.

Yikes. If we make this list of headers, it's not going to be
normative, since we can't retroacively clean up 40 years of practice,
and while 5598 is a useful model of the way mail systems work, there's
a lot of places where theory and practice don't exactly match.

Many headers are ideally created by MUAs but in practice added by
MSAs, so they get listed both places. There's also overlap between MTA
and MDA depending on how a mail system is built.

I still don't understand what problem this exercise would be solving.

---

From: John C Klensin <[email-masked]>
Date: Sun, 25 Jul 2021 22:05:10 -0400

--On Sunday, 25 July, 2021 21:25 -0400 John Levine
<[email-masked]> wrote:

>> Or you could convince the WG to tell me to go ahead and
>> create a normative reference to 5598 5598 to and use the
>> exception procedure, but I imagine some of us would argue
>> that would be tantamount to moving standards track and hence
>> out of charter.
> 
> normative, since we can't retroacively clean up 40 years of
> practice, and while 5598 is a useful model of the way mail
> systems work, there's a lot of places where theory and
> practice don't exactly match.\

Exactly.

> Many headers are ideally created by MUAs but in practice added
> by MSAs, so they get listed both places. There's also overlap
> between MTA and MDA depending on how a mail system is built.
> 

Speaking personally only, agreed.  For me that created a
situation in which IANA should insist on my precise definitions
to run even a piece of a registry and getting to those
definitions --with theory and practice matching-- is likely to
be sufficient trouble that the benefits and problem being solved
had best be _very_ clear.

best,
   john

---

From: Dave Crocker <[email-masked]>
Date: Sun, 25 Jul 2021 19:39:51 -0700

> in part because some of its sections could not get consensus.

John,

Since you were there, you know that's not the reason.  You know that 
there was actually very strong rough consensus among email technical 
participants that it should be standards track.  I think you were among 
the very few -- perhaps one other? -- who argued against it.

Given that the document had 5 years of collaborative IETF work, a strong 
rough consensus isn't all that surprising.  You know that what happened 
was that the IESG decided to go against that rough consensus.  For some 
reason.

In any event, my question to you was based on the substance of your 
original comment.  For some reason, you have been unable to engage in 
the substance of RFC 5598.  Again note that it is an IETF-approved 
document, which means there was IETF consensus to publish it.

Your statement that I originally responded to was:

>> between MUA and SMA is deliberately unclear;

The sections of RFC 5598 provide a significant amount of text that 
includes dealing with just this sort of issue.

So again I'll ask, what are the specific deficiencies of those bits of 
substantive text for this bit of presumably-substantive concern that you 
have expressed.

d/

---

From: Dave Crocker <[email-masked]>
Date: Mon, 26 Jul 2021 09:18:06 -0700

On 11/16/2020 9:10 AM, Alexey Melnikov wrote:
> Based on an earlier discussion on the mailing it looks like there is 
> consensus to create (or possible ammend an existing) registry for header 
> fields with extra information about which types of SMTP agents are 
> allowed to add them in transit or before final delivery.

Taking John Levine's pointed challenge on this issue, I went back to 
this start-of-thread message and now find myself also questioning what 
the benefit of this effort will be.

Technical specifications or Applicability Statements, or the like, will 
contain the normative information about acceptable use of a header 
field.  Providing such information in a registry will, therefore, be 
redundant.  (Possibly adding the risk of divergence, over time.)

If that redundancy were expected to be extremely helpful -- such as 
finding appropriate header fields for a given functional component -- 
then it might be useful.

Given the work of maintaining a registry -- and that risk of it getting 
out of sync with the normative specifications -- I'd suggest that the 
expected utility needs to be /very/ high.

Absent such a claim and support for it, this isn't likely to be a 
constructive change.

Registries should be for coordinating against collisions, dealing with 
scarcity, and not much else, absent truly compelling justification.

d/

---

From: Alessandro Vesely <[email-masked]>
Date: Mon, 26 Jul 2021 18:50:18 +0200

On Sun 25/Jul/2021 20:28:14 +0200 Dave Crocker wrote:
>> added or modified during delivery proper (rMDA, in rfc5598 parlance) — such 
>> headers will never be transmitted via SMTP.
>>
> If an MDA is 're-injecting' a message, it is acting as an MUA or, at the least, 
> an MSA.

Right.  Adding that clarification would effectively limit the meaning of "Can 
be added by an MDA".

Best
Ale
--

---

From: Ned Freed <[email-masked]>
Date: Tue, 27 Jul 2021 10:39:02 -0700 (PDT)

> the benefit of this effort will be.

I did so as well, and I see a lot of problems I didn't see before.

For starters, I'm confused by the use of the term "subregistry". This seems to
me to come down to an update to RFC 3864 to add a column where people
can indicate where in email architecture they expect this field 
to be added.

If that's the case, then I don't understand what this has to do with SMTP and
thus why it has any business being in RFC 5321bis. Our architecture covers more
than SMTP, and this is true even if you expand the scope to cover SUBMIT and
LMTP.

If anything this belongs in RFC 5322bis.

If you think about things this way I think most of the issues about
MSA/MDA/MUA/MfooA disappear. The columns is a means of indicating the
point in the architecture where it makes sense for it to be added. Nothing
more. How this corresponds to actual implementations, which we all know
map the architectural elements to software components in complex and
fairly arbitrary ways, is essentially irrelevant.

> finding appropriate header fields for a given functional component --

The value of the information is essentially the same as the value of
registering header fields - it's useful to know what people's intentions
are/were, and why they did what they did. (It's actually even more useful when
what they did doesn't actually make much sense.)

The counter to this is that the header field registry has, regretfully,
for the most part failed to capture this sort of information. Asking for
more information isn't gong to change this, but I don't see how it hurts.

> expected utility needs to be /very/ high.

This makes no sense to me. The registry exists and is being maintained, so
that's a sunk cost. Adding a column to it seems to me to a minimal
cost addtion.

What am I missing here?

> scarcity, and not much else, absent truly compelling justification.

Coordinating use outside of standardize use-cases is also kind of important.

				Ned

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 28 Jul 2021 15:16:37 +0100

Hi Dave,

On 26/07/2021 17:18, Dave Crocker wrote:
>> consensus to create (or possible ammend an existing) registry for 
>> header fields with extra information about which types of SMTP agents 
>> are allowed to add them in transit or before final delivery.
>
> Technical specifications or Applicability Statements, or the like, 
> will contain the normative information about acceptable use of a 
> header field.  Providing such information in a registry will, 
> therefore, be redundant.  (Possibly adding the risk of divergence, 
> over time.)

One of the main purposes of the proposed registry is to be able to find 
a specific RFC that defines a particular header field. So the advice of 
"just find the RFC" is not particularly helpful, considering how many 
there are now.

Also, the registry will contain header fields not necessarily defined in 
an RFC (e.g. header fields already in use in the wild), which would 
serve a useful purpose.

I don't think I agree about the risk of divergence: we will populate the 
registry once for known header fields and then future RFCs will just 
update the registry. As our community will know about existence of this 
new registry, I don't think it will be likely that RFC authors will 
forget to update it.

> finding appropriate header fields for a given functional component -- 
IMHO, this would be useful as well.
>
> Given the work of maintaining a registry -- and that risk of it 
> getting out of sync with the normative specifications -- I'd suggest 
> that the expected utility needs to be /very/ high.
>

Best Regards,

Alexey

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 28 Jul 2021 15:35:57 +0100

Hi Ned,

On 27/07/2021 18:39, Ned Freed wrote:
>> header

You were the original person who motivated creation of this ticket  10 
months ago :-):

----------------------

Ned Freed wrote:

There are many fields that get added after submission and before final
delivery. And not all of them really qualify as "trace" IMO.

Moreover, making a list of these fields is a hopeless task, since all
it takes is one more document to outdate the list.

I see two possible solutions:
(1) Craft some text saying something like, "Only add fields specified as OK
to be added in other specifications."

(2) Create a registry.
I kind of like the idea of a registry, but it is a fair bit more work.

----------------------

> can indicate where in email architecture they expect this field to be 
> added.

No, this is not what I meant. I meant it would be listed on the same 
page as registered permanent/provisional header fields. But even that is 
not required, if people don't think this is a good idea.

And no, definitely not suggesting any changes to RFC 3864, as it is also 
used by HTTP and Usenet, as having extra columns that only apply to 
email would be very confusing to other users of the registry.

Anyway, let's change "subregistry" to "registry".

> If anything this belongs in RFC 5322bis.
That would be Ok with me personally.
> fairly arbitrary ways, is essentially irrelevant.
Ok, so are you suggesting some changes to how columns are named? Or 
their removal altogether? Can you please clarify.

Best Regards,

Alexey

---

From: Dave Crocker <[email-masked]>
Date: Wed, 28 Jul 2021 08:38:54 -0700

> that's a sunk cost. Adding a column to it seems to me to a minimal

It takes effort to get all of the actors who are involved in creating 
and maintaining those entries to get all of the information into the 
registry correctly.

The core information, such as name, authority agent, and specification 
citation, is part of every registry and therefore part of everyone's 
understanding of what to do.

The current topic is about information that isn't ingrained, even among 
subject matter experts.  I suggest that that will make it more likely to 
be handled inconsistently.

d/

---

From: Dave Crocker <[email-masked]>
Date: Wed, 28 Jul 2021 08:56:52 -0700

> On 26/07/2021 17:18, Dave Crocker wrote:
...
> "just find the RFC" is not particularly helpful, considering how many 

Alexey,

I don't recall hearing anyone say anything like 'just find the RFC', so 
I'm not sure what you are referencing.

In the meeting discussion, I noted that I'd neglected to include the 
specification citation in the list of information that goes with the 
registry entry.  The /specific/ specification citation.

On reviewing my above "Technical specification or..." paragraph, above, 
I think I was not at all clear enough.  Attempting to fix that:

      Technical specifications or Applicability statements, or the like 
contain the relevant technical detail, including normative language.  A 
registry needs to point to such documents but should not replicate 
details from them.  Replication of details invites divergence.

> an RFC (e.g. header fields already in use in the wild), which would 

The choice of whether to allow registration entries that do not have a 
citable specification is obviously a strategic choice.  Since a main 
function of registration is to avoid collisions, there an argument for 
being permissive.

But the absence of a specification means that it won't be possible for 
folk to easily find out what the registered entry is used for; that's an 
argument for being more demanding.

For something like header-fields, the namespace is essentially infinite. 
  So my own preference is to require /some/ specification, but not be 
very demanding about the process for producing it.

> update the registry.

Right.  We don't have any issue with redundant details between RFC821 
and RFC822, or their successors.

My concern about divergence isn't really all that unusual.

My own lesson from 821/822, and similar examples, is to specify 
normative information in one place and everywhere else to just cite it 
(albeit somethings offering a terse bit of prose to summarize, but in a 
way that can't be taken as normative.)

> As our community will know about existence of this 

"Our community" is large and diverse and often doesn't know about 
details others consider important.

Hanging around an industry email trade association is educational, for 
seeing what technical information people in the industry do and do not have.

d/

---

From: Ned Freed <[email-masked]>
Date: Wed, 28 Jul 2021 12:16:49 -0700 (PDT)

> You were the original person who motivated creation of this ticket  10

That's ... amusing.

> not required, if people don't think this is a good idea.

OK, I now understand your proposal. But I still think having one list is much
better than having a separate list. Why not simply annotate the existing list
with the  points at which it makes sense to introduce the header field?

> email would be very confusing to other users of the registry.

I believe that's what "n/a" is for. But OK, if you really think that having
mail-only field is too confusing to people viewing the page, then the way to
address that is to have one central list behind it all and generate as many
views of it as you like on the page.

Having to submit to two registries  grots up the process considerably for
anyone wanting to register a header. And given the complaints we're getting
about how hard it is to register stuff, I really think the focus needs to be
on keeping that process as simple as possible.

> Anyway, let's change "subregistry" to "registry".

Except it *is* a sugregistry. Changing the name doesn't change that.

> > than SMTP, and this is true even if you expand the scope to cover
> > SUBMIT and LMTP.

> their removal altogether? Can you please clarify.

No, I am suggesting keeping them the same, and making it clear that we're
referring to the point in the architecture the addition is supposed to be
done, not where it happens to appear in any particular implementation.

And while there may - or may not - be issues with the architecture document
that are worth looking at, I don't think they rise to the level where
citing the basic architectural components of email is problematic.

				Ned

---

From: "John Levine" <[email-masked]>
Date: 28 Jul 2021 23:13:52 -0400

>"just find the RFC" is not particularly helpful, considering how many 

When I look at this registry I see a list of header fields and the RFC or
other reference that defines each one:

https://www.iana.org/assignments/message-headers/message-headers.xhtml

How would this be different?

I can sort of see adding a column to the existing registry with a hint
about where in the mail process it's typically added, but not a whole new registry.

R's,
John

---

From: Dave Crocker <[email-masked]>
Date: Wed, 28 Jul 2021 21:08:56 -0700

> about where in the mail process it's typically added, but not a whole new registry.

A 'hint'?

1. Chosen by whom and through what process?
    Note that that makes it additional work and potentially another
    source of controversy.

2. It will be taken as normative
    Because that's what people do when they see entries like this, in
    places like this.

3. It is an attempt to make a 'registry' be a kind of partial tutorial.
    To do a useful job takes surrounding text.  Typically quite a bit.

d/

---

From: Martin J. Dürst <[email-masked]>
Date: Thu, 29 Jul 2021 13:12:37 +0900

> about where in the mail process it's typically added, but not a whole new registry.

Same here. Another variant would be to allow a fixed range of additions 
to the protocol column. I.e. not just "mail", but e.g. "mail (gateways)" 
or whatever we may decide on.

Regards,   Martin.

---

From: Ned Freed <[email-masked]>
Date: Thu, 29 Jul 2021 07:10:26 -0700 (PDT)

> A 'hint'?

The file extension in the media types registry is just a hint, but even
so it's quite useful.

> 1. Chosen by whom and through what process?

The usual process of adding entries to the registry. This would simply
be an optional item on the registation form.

>     Note that that makes it additional work and potentially another

Really? You actually believe that it's possible to write a clear and cogent
specification for the syntax and semantics for a new email header and
yet not know the places where it can be introduced into the mail system?

And in the unlikely event this does happen, perhaps in this case it would
be good for the specification to explain why it's unclear.

>     Because that's what people do when they see entries like this, in

Decades of experience with various registries says this is not the case.

Like it or not, the "selective inattention" force mediated by the "doesn't
apply to me" boson tends to overpower almost everything.

And as far as I can tell we have yet to detect the "normative" force you
meantion.

>     To do a useful job takes surrounding text.  Typically quite a bit.

Again, I don't see how you can properly specify a header field without
talking about it's relationship to the email architecture. So yes,
actual text is required to do this properly. But providing a indication
of this in the registry doesn't change this one bit.

				Ned