Subject: [Emailcore] Ticket #7: Better definition for trace header fields

From: Alexey Melnikov <[email-masked]>
Date: Mon, 22 Feb 2021 17:03:34 +0000

Hi all,

I didn't realize until John Klensin pointed out that this affects both 
rfc5321bis and rfc5322bis. So below I extracted pretty much all text 
related to trace header fields and in some cases I provide my strawman 
proposals on how to update them:

rfc5322bis:

 >3.6.  Field Definitions
 >
 >   More importantly, the trace header fields and resent
 >   header fields MUST NOT be reordered, and SHOULD be kept in blocks
 >   prepended to the message.  See sections 3.6.6 and 3.6.7 for more
 >   information.

I think the above gives important characteristics of the trace header 
fields and the above text looks fine to me.

 >3.6.7.  Trace Fields
 >
 >   The trace fields are a group of header fields consisting of an
 >   optional "Return-Path:" field, and one or more "Received:" fields.

I think this needs to be updated to make it clear that other header 
fields can be designated as "trace header fields". For example:

    The trace fields are a group of header fields consisting of an
    optional "Return-Path:" field, one or more "Received:" fields
    and other header fields specified in other specifications.

 >   The "Return-Path:" header field contains a pair of angle brackets
 >   that enclose an optional addr-spec.  The "Received:" field contains a
 >   (possibly empty) list of tokens followed by a semicolon and a date-
 >   time specification.  Each token must be a word, angle-addr, addr-
 >   spec, or a domain.  Further restrictions are applied to the syntax of
 >   the trace fields by specifications that provide for their use, such
 >   as [I-D.klensin-rfc5321bis].
 >
 >   trace           =   [return]
 >                       1*received

Again, I think ABNF needs to be clear that other header fields can 
appear in between Return-Path and Received, for example:

    trace           =   [return]
                        *trace-extension-field
                        1*received

   trace-extension-field = optional-field

 >   A full discussion of the Internet mail use of trace fields is
 >   contained in [I-D.klensin-rfc5321bis].
 >   For the purposes of this
 >   specification, the trace fields are strictly informational, and any
 >   formal interpretation of them is outside of the scope of this
 >   document.

 >Appendix B.  Differences from Earlier Specifications

 >The following are changes from [RFC2822].

 >   12.  Allowed optional-field to appear within trace information.

It doesn't look like this actually happened or maybe this was altered in 
RFC 5322.

rfc5321bis:

 >3.7.2.  Received Lines in Gatewaying
 >
 >   "Received:" header fields of messages originating from other
 >   environments may not conform exactly to this specification.  However,
 >   the most important use of Received: lines is for debugging mail
 >   faults, and this debugging can be severely hampered by well-meaning
 >   gateways that try to "fix" a Received: line.  As another consequence
 >   of trace header fields arising in non-SMTP environments, receiving
 >   systems MUST NOT reject mail based on the format of a trace header
 >   field and SHOULD be extremely robust in the light of unexpected
 >   information or formats in those header fields.

This talks about purpose of trace header fields and this text looks fine 
to me.

 >4.1.1.4.  DATA (DATA)
 >
 >   When the SMTP server accepts a message either for relaying or for
 >   final delivery, it inserts a trace record (also referred to

So this introduces "trace record", which is different from "trace header 
fields", as it only related to Received.

 >   interchangeably as a "time stamp line" or "Received" line) at the top
 >   of the mail data.  This trace record indicates the identity of the
 >   host that sent the message, the identity of the host that received
 >   the message (and is inserting this time stamp), and the date and time
 >   the message was received.

 >4.4.  Trace Information
 >
 >   When an SMTP server receives a message for delivery or further
 >   processing, it MUST insert trace (often referred to as "time stamp"

I suggest to add "record" after "trace" to match the above definition.

 >   or "Received" information) beginning of the message content, as
 >   discussed in Section 4.1.1.4.

 >6.4.  Compensating for Irregularities
 >
 >   In all cases, properly operating clients supplying correct
 >   information are preferred to corrections by the SMTP server.  In all
 >   cases, documentation SHOULD be provided in trace header fields and/or
 >   header field comments for actions performed by the servers.

This looks fine to me.

 >7.6.  Information Disclosure in Trace Fields
 >
 >   In some circumstances, such as when mail originates from within a LAN
 >   whose hosts are not directly on the public Internet, trace
 >   ("Received") header fields produced in conformance with this
 >   specification may disclose host names and similar information that
 >   would not normally be available.

I think the above is largely correct, but I suggest inserting "e.g." 
before "Received" in 'trace ("Received") header fields' to make it clear 
that this is not a full list of trace header fields.

 >8.  IANA Considerations
 >
 >   In addition, if additional trace header fields (i.e., in addition to
 >   Return-path and Received) are ever created, those trace fields MUST
 >   be added to the IANA registry established by BCP 90 (RFC 3864) [8]
 >   for use with RFC 5322 [11].

I think the above is fine. We also talked about a separate registry for 
trace header fields in the A/S document.

Best Regards,

Alexey

---

From: John C Klensin <[email-masked]>
Date: Mon, 22 Feb 2021 13:09:28 -0500

Alexey,

I think this is about right.  

As editor: I wonder (it would take some research to figure it
out) where "trace record" entered the vocabulary.  Noting that
"trace" does not appear to occur at all in 821, I suspect it is
just a potentially confusing artifact of the document-merging
process or the editor of 2821 having an attack of stupid
inconsistency rather than the distinction you are making.  From
that perspective, while it would require more work, an
alternative would be to remove that term, use "trace header
field" throughout, and reword the relevant paragraphs to make it
clear that "Received:" is not the only such field.

In addition, the boundary between what information about those
fields is included in 5321bis versus 5322bis is a bit arbitrary
and may not be quite right.  I wonder if it would be worthwhile
for the WG to review those choices and made suggestions that
might reduce having potentially conflicting descriptions in two
different documents.  The latter practice is, as Dave Crocker
recently pointed out, an invitation to confusion.  On the other
hand, if we start tampering with that boundary, we might easily
make things worse.

I'm happy to do whatever the WG decides about the above as long
as I get clear instructions.

    john

--On Monday, February 22, 2021 17:03 +0000 Alexey Melnikov
<[email-masked]> wrote:

> Hi all,
> 
> I didn't realize until John Klensin pointed out that this
> affects both rfc5321bis and rfc5322bis. So below I extracted
> pretty much all text related to trace header fields and in
> some cases I provide my strawman proposals on how to update
> them:
> 
> rfc5322bis:
> 
>  >3.6.  Field Definitions
>  >
> I think the above gives important characteristics of the trace
> header fields and the above text looks fine to me.
> 
> 
>  >3.6.7.  Trace Fields
>  >
> I think this needs to be updated to make it clear that other
> header fields can be designated as "trace header fields". For
> example:
> 
> Again, I think ABNF needs to be clear that other header fields
> can appear in between Return-Path and Received, for example:
> 
>     trace           =   [return]
>                        
> *trace-extension-field
>                         1*received
> 
> It doesn't look like this actually happened or maybe this was
> altered in RFC 5322.
> 
> 
> rfc5321bis:
> 
> This talks about purpose of trace header fields and this text
> looks fine to me.
> 
>  >4.1.1.4.  DATA (DATA)
>  >
> So this introduces "trace record", which is different from
> "trace header fields", as it only related to Received.
> 
> I think the above is largely correct, but I suggest inserting
> "e.g." before "Received" in 'trace ("Received") header fields'
> to make it clear that this is not a full list of trace header
> fields.
> 
> 
>  >8.  IANA Considerations
>  >
> I think the above is fine. We also talked about a separate
> registry for trace header fields in the A/S document.
> 
> Best Regards,
> 
> Alexey

---

From: "John Levine" <[email-masked]>
Date: 22 Feb 2021 14:42:26 -0500

>fields and the above text looks fine to me.

If we want to interoperate well, I think that SHOULD is a MUST.  Also,
how about saying they SHOULD NOT be deleted.  It'd be a MUST NOT except
that there is a special case for Authentication-Results.

>   trace-extension-field = optional-field

Wouldn't that mean that the extension fields are all above the received?  How about this:

    trace           =   [return]
                        1*(received / trace-extension-field)

>trace header fields in the A/S document.

Is it a separate registry or a new column in the existing registry?  It doesn't
seem like a great idea to have two registries where one is supposed to be a subset
of the other.

R's,
John

---

From: "Peter J. Holzer" <[email-masked]>
Date: Mon, 22 Feb 2021 21:20:09 +0100

> >fields and the above text looks fine to me.

I'm a bit worried that someone might read a "MUST be kept in blocks
prepended to the message" as a mandate to move existing trace fields to
the front. 

For example, suppose system D receives this message:

    Unknown-Field: 42
    Received; from B by C
    Delivered-To: x@B
    Received: from A by B
    Received: by A
    Subject: Example
    ...

Instead of just prepending its Received field, making it

    Received: from C by D
    Unknown-Field: 42
    Received; from B by C
    Delivered-To: x@B
    Received: from A by B
    Received: by A
    Subject: Example
    ...

it pulls the recognized trace headers to the front, like this:

    Received: from C by D
    Received; from B by C
    Delivered-To: x@B
    Received: from A by B
    Received: by A
    Unknown-Field: 42
    Subject: Example
    ...

I don't think that would be a good idea. I think an MTA should just
never reorder existing headers (whether they are trace headers or not)
and just prepends any headers it wants to add.

        hp

-- 
   _  | Peter J. Holzer    | Story must make more sense than reality.
|_|_) |                    |
| |   | [email-masked]         |    -- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |       challenge!"

---

From: Pete Resnick <[email-masked]>
Date: Mon, 22 Feb 2021 15:04:00 -0600

I think this is going in an OK direction, though changing ABNF always 
makes me a bit queasy. I'll read over the messages and see if I can come 
up with the 5322bis proposal. On one below question:

On 22 Feb 2021, at 12:09, John C Klensin wrote:

> out) where "trace record" entered the vocabulary.

It appears in 822. See section 4.3. 
https://tools.ietf.org/html/rfc822#section-4.3

pr
-- 
Pete Resnick https://www.episteme.net/
All connections to the world are tenuous at best

---

From: Pete Resnick <[email-masked]>
Date: Mon, 22 Feb 2021 15:32:08 -0600

Combining some answers to John L. and Alexey regarding the 5322bis 
chagnes...

On 22 Feb 2021, at 13:42, John Levine wrote:

> In article <[email-masked]> Alexey 
> Melnikov writes:
>>
>>> 3.6.  Field Definitions
>>>
> If we want to interoperate well, I think that SHOULD is a MUST.

I can go spelunking in the DRUMS archive if needed, but my memory is 
that the reason for the SHOULD was that we knew that certain 
implementations did not (and could not) keep things in prepended blocks 
and we didn't want the MUST to indicate that as a receiver you could so 
strongly depend on them being kept together.

> Also,

All of the "don't change/delete" requirements are in 5321bis; we never 
had any in x22. I'm inclined to keep it that way.

>    and other header fields specified in other specifications.

Yeah, I'll wordsmith that a bit, but I will do something like the above.

> received?  How about this:

Yes, and if I do that, I'll remove the extra optional-field at the top 
of the overall fields syntax in 3.6.

>> It doesn't look like this actually happened or maybe this was altered 
>> in RFC 5322.

It did. I just put it in in 3.6 instead of in 3.6.7.

pr

---

From: John C Klensin <[email-masked]>
Date: Mon, 22 Feb 2021 17:12:55 -0500

--On Monday, February 22, 2021 15:04 -0600 Pete Resnick
<[email-masked]> wrote:

> I think this is going in an OK direction, though changing ABNF
> always makes me a bit queasy. I'll read over the messages and
> see if I can come up with the 5322bis proposal. On one below
> question:
> 

Sorry.  Knew that.  I meant to say "entered the SMTP
vocabulary", but, thinking about it further, I assume the answer
to my question is "when we were trying to harmonize vocabulary
between proto-2822 and proto-2821".  However "trace record" does
not appear in 822 or 2822, so I have to guess that either I had
the distinction that Alexey detected in mind and had completely
forgotten it, or I messed up.  Either way the question now is
what to do going forward.

thanks,
   john

---

From: "John R Levine" <[email-masked]>
Date: 22 Feb 2021 17:47:16 -0500

> I can go spelunking in the DRUMS archive if needed, but my memory is that the 
> reason for the SHOULD was that we knew that certain implementations did not 
> (and could not) keep things in prepended blocks and we didn't want the MUST 
> to indicate that as a receiver you could so strongly depend on them being 
> kept together.

That sounds vaguely familiar.  We need a 2119 word for "do this but don't 
count on other people reading the spec."

> Yes, and if I do that, I'll remove the extra optional-field at the top of the 
> overall fields syntax in 3.6.

OK.  That is not exactly equivalent since it allows a trace block 
consisting of just, say, Authentication-Results with no Received, but I 
don't think that is wrong.

By the way I noticed the reference to errata 2950.  The errata is 
wrong, the resent-fields are indeed optional, and we someday we should 
update the ABNF spec to make it clear that ABNF can and often does provide 
multiple parses of the same valid input which is not a bug and does not 
cause problems for practical parsers.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Ned Freed <[email-masked]>
Date: Mon, 22 Feb 2021 13:51:25 -0800 (PST)

> >> fields and the above text looks fine to me.

I'm afraid it does nothing of the sort. Required handling is not the same
as describing what constitutes a trace field. I suppose you might be able
to work back from the handling to likely uses, but that's not how this
should work.

So a definition is needed. And as it happens, there was one in RFC 822
section 4.3:

     Trace information is used to provide an audit trail of  message 
     handling.   In  addition,  it indicates a route back to the
     sender of the message.

For some reason this bit of text didn't make it into 2822. I don't recall any
discussion of it, and I don't have time to search the archives for a reason why
it was dropped - assuming it was even decided on the list.

Regardless of cause, this definition or something similar needs to be restored.

I also note that Received: and the concept of trace was introduced in RFC 822;
it is not in RFC 733. This parallels the evolution of similar text in the
development of the X.400 standards: FIPS PUB 98 doesn't provide any trace
facilities at all, by DEC's Message Router (AFAIK the first and perhaps only
implementation more or less based on that document) does support trace fields.
And by the time X.400 proper rolled around we have this text in X.411: 

  12.2.1.1.1.3 Trace-information

  This argument documents the actions taken on the message (or probe or report)
  by each MD through which the message (or probe or report) passes as it is
  transferred through the MTS (see 12.3.1). It shall be generated by each MD
  through which the message (or probe or report) passes.

  12.2.1.1.1.4 Internal-trace-information

  This argument documents the actions taken on the message (or probe or report)
  by each MTA through which the message (or probe or report) passes as it is
  transferred within an MD (see 12.3.1). It shall be generated by each MTA
  through which the message (or probe or report) passes within an MD.
  As a matter of local policy, an MTA may (but is not required to) remove
  internal-trace-information relating to other MDs when performing delivery, or
  when transferring to another MD, or on receiving from another MD.

I've often wished that the two types of trace information had made it over to
the Internet side of things, so that calls for all Received: lines to be
removed in the interests of not revealing internal network structure could be
met more easily. Of course it's much too late for that, but it might be
interested to document for each header field containing trace information
whether it's generated by MSAs, MTAs, MDAs, or some combination of the three.

There's probably additional text in the IFIP documents and possibly in GOSIP;
unfortunately that material is in some box or file I can't locate.

However, I think this gives us enough to work with. I actually prefer the
X.400 text to what's in RFC 822; the "actions taken" idea in particular
strikes me as both more general and less formal than "audit trail". So
how about a definition along the lines of:

  Trace information documents actions taken on the message by MSAs, MTAs and
  MDAs. It is provided by an extensible set of header fields.

And then move onto the requirements, current set of fields, and so on.

> strongly depend on them being kept together.

I believe that's correct.

> had any in x22. I'm inclined to keep it that way.

Agreed.

> Yeah, I'll wordsmith that a bit, but I will do something like the above.

Again, you need an actual definition. Suggestions above.,

> of the overall fields syntax in 3.6.

FWIW, I strongly prefer new fields to messing around with Received:. Yes, I'm
aware that this means that some agents may require an update to their list of
fields, but IMO messing around with Received: parsing is a bigger problem.

				Ned

---

From: Alessandro Vesely <[email-masked]>
Date: Tue, 23 Feb 2021 12:39:44 +0100

On Mon 22/Feb/2021 22:32:08 +0100 Pete Resnick wrote:
> Combining some answers to John L. and Alexey regarding the 5322bis chagnes...
> [...]

That ABNF seems to imply there must be at least one Received:.  Why then does 
the table in Section 3.6 show a Min number of 0 trace?

I agree with Peter's worries about mandating the initial block.  The 
optional-field allowed more leeway, as the optional field could happen to be 
whatever, not necessarily a trace field.  Indeed, the requirement to collect 
all trace fields at the top of the header lest the message be syntactically bad 
is a recipe for trouble.

The practice to collect trace fields at the top comes from the fact that it is 
convenient to write down one's 2 cents and then copy the rest without even 
parsing it.  But a mail filter can add a header field holding information that 
semantically does not make it a trace field.  Even then, it is more natural to 
write added fields near the top of the header.  Doing so is also better for 
debugging and forensic analysis, as it is possible to deduce at what stage a 
given field was added.

For example, consider a DKIM filter which tries and reverts a mailing list's 
From: munging.  It records its findings in a trace field, 
Authentication-Results:.  In addition, if the reversion succeeds, it adds an 
Original-From: field, formatted like From:.  Semantically equivalent to From:, 
and eventually swappable with it, the latter is definitely not a trace field. 
Should it be added downward, possibly near to the real From:, or isn't it more 
self-explanatory to add it near the corresponding Authentication-Results:?

Could we relax the header syntax?  For example:

     trace     = [return]
                 *top-field

     top-field = received /
                 trace-extension-field /
                 prowl-field

Where prowl-field can actually be any field, possibly resulting from Sieve or 
any other email filtering.  That syntax still conveys that trace fields tend to 
gather near the top of the header, but doesn't require a strict order.

Best
Ale
--

---

From: "Alexey Melnikov" <[email-masked]>
Date: Tue, 23 Feb 2021 13:39:49 +0000

>                         1*(received / trace-extension-field)

Works for me.

> seem like a great idea to have two registries where one is supposed to 

I don't think adding a new column will work, because this registry is also used by NNTP and HTTP folks. We could however ask IANA to check the second registry for consistency with the first. I believe they have other registries with similar requirements. Anyway, this is a part of ticket #8, so let's defer this discussion until we get to that ticket.

Best Regards,
Alexey

---

From: "Alexey Melnikov" <[email-masked]>
Date: Tue, 23 Feb 2021 13:46:32 +0000

> don't think that is wrong.

I think this is Ok.

Another thought about removing optional-field at the top of the overall fields syntax in 3.6: Does removing it has implication that anything at the top of the header has to be a trace header field or a resent block? If this is too restrictive, then leaving 3.6 syntax as it is is probably the best.

---

From: "Alexey Melnikov" <[email-masked]>
Date: Tue, 23 Feb 2021 13:52:48 +0000

On Mon, Feb 22, 2021, at 9:51 PM, Ned Freed wrote:
> So a definition is needed.

Yes, you are right.

> And as it happens, there was one in RFC 822
> section 4.3:
> 

I like it.

Works for me.

> Emailcore mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/emailcore
>

---

From: "Alexey Melnikov" <[email-masked]>
Date: Tue, 23 Feb 2021 13:56:10 +0000

Hi Peter,

> and just prepends any headers it wants to add.
I generally agree that not reordering is the only sane way. However I am not sure whether we can tighten the requirement in rfc5322bis.

Best Regards,
Alexey

---

From: "Alexey Melnikov" <[email-masked]>
Date: Tue, 23 Feb 2021 14:13:27 +0000

Hi John,

> clear that "Received:" is not the only such field.

I saw Pete's reply, but I am happy with either change.

> hand, if we start tampering with that boundary, we might easily

Right. I think having a definition of trace header fields in rfc5322bis makes sense, as there is already a section there. I don't have strong feelings about moving text between documents.

Best Regards,
Alexey (as a participant)

---

From: "John R Levine" <[email-masked]>
Date: 23 Feb 2021 11:21:42 -0500

> I don't think adding a new column will work, because this registry is also used by NNTP and HTTP folks. We could however ask IANA to check the second registry for consistency with the first. I beli…

There's already a column for which protocol, so I'd say the "trace" column 
only applies to mail headers.  Or maybe change the protocol column so it 
has both mail and mailtrace.

Anything that IANA can handle is fine with me but having done my PhD in 
databases, schemas with two manually maintained tables that are supposed 
to match up make my stomach hurt.

Regards,

PS: The netnews Path header actts a lot like a trace header but we can burn that bridge later.

---

From: Pete Resnick <[email-masked]>
Date: Tue, 23 Feb 2021 11:29:12 -0600

On 22 Feb 2021, at 15:51, Ned Freed wrote:

>>>> header

Well, it does give some characteristics, but I agree, it doesn't give a 
full definition.

> Regardless of cause, this definition or something similar needs to be 

Again without spelunking, I believe I know what happened: We decided 
that we were going to move all of the semantics of trace into 2821 so 
that 2822 didn't have to repeat (potentially slightly different) text 
from 2821. So what we were left with was the specification of the syntax 
in 2822 and left any other discussion to 2821. We can certainly undo 
that decision, but I'm not totally convinced we need to.

> whether it's generated by MSAs, MTAs, MDAs, or some combination of the 

That says to me that it may still be appropriate to let documents other 
than xx22 document the semantics of the trace fields.

>  MDAs. It is provided by an extensible set of header fields.

I think something along those lines is probably safe to put in xx22.

> And then move onto the requirements, current set of fields, and so on.

I think that might be best left for another document.

> fields, but IMO messing around with Received: parsing is a bigger 

I'm not sure I understand this last comment. There is no suggestion to 
mess around with Received:. The proposal is to move optional-field into 
the definition of trace in 3.6.7 rather than have it in the overall 
fields syntax at the top of 3.6. Did I miss something?

pr

---

From: Pete Resnick <[email-masked]>
Date: Tue, 23 Feb 2021 11:34:39 -0600

On 23 Feb 2021, at 5:39, Alessandro Vesely wrote:

>> Yes, and if I do that, I'll remove the extra optional-field at the 
>> top of the overall fields syntax in 3.6.
>
>
> That ABNF seems to imply there must be at least one Received:.  Why 
> then does the table in Section 3.6 show a Min number of 0 trace?

Because the entire trace block is optional according to the ABNF right 
above that table in 3.6.

> optional-field allowed more leeway, as the optional field could happen 
> to be whatever, not necessarily a trace field.  Indeed, the 
> requirement to collect all trace fields at the top of the header lest 
> the message be syntactically bad is a recipe for trouble.

But optional-field is permitted at the bottom of the ABNF of fields in 
3.6, which would allow it to be used for something that is not a trace 
field. (That is, optional-field appears twice in the 3.6 syntax.)

> The practice to collect trace fields at the top comes from the fact 
> that it is convenient to write down one's 2 cents and then copy the 
> rest without even parsing it.  But a mail filter can add a header 
> field holding information that semantically does not make it a trace 
> field.  Even then, it is more natural to write added fields near the 
> top of the header.  Doing so is also better for debugging and forensic 
> analysis, as it is possible to deduce at what stage a given field was 
> added.
>
> For example, consider a DKIM filter which tries and reverts a mailing 
> list's From: munging.  It records its findings in a trace field, 
> Authentication-Results:.  In addition, if the reversion succeeds, it 
> adds an Original-From: field, formatted like From:.  Semantically 
> equivalent to From:, and eventually swappable with it, the latter is 
> definitely not a trace field. Should it be added downward, possibly 
> near to the real From:, or isn't it more self-explanatory to add it 
> near the corresponding Authentication-Results:?
>
> Where prowl-field can actually be any field, possibly resulting from 
> Sieve or any other email filtering.  That syntax still conveys that 
> trace fields tend to gather near the top of the header, but doesn't 
> require a strict order.

Review the ABNF at the top of 3.6 and see if you can't express what you 
want to. Otherwise, I think things are best left alone except for moving 
the first occurrence of optional-field out of 3.6 and putting an 
optional-field (or trace-extension-field) into 3.6.7.

pr

---

From: "Alexey Melnikov" <[email-masked]>
Date: Tue, 23 Feb 2021 17:44:56 +0000

Pete,

> On 22 Feb 2021, at 15:51, Ned Freed wrote:

 [snip]

> I think something along those lines is probably safe to put in xx22.

Can you please advise where to add this text in xx22? (I am not yet convinced that adding this to xx22 is the best course of action, but happy to be proven wrong.)

Thank you,
Alexey

---

From: Pete Resnick <[email-masked]>
Date: Tue, 23 Feb 2021 12:05:57 -0600

On 23 Feb 2021, at 11:44, Alexey Melnikov wrote:

> Can you please advise where to add this text in xx22? (I am not yet 
> convinced that adding this to xx22 is the best course of action, but 
> happy to be proven wrong.)

Here's a shot at the entire set of changes for 3.6.7. (Remember that the 
way xx22 is structured, you get a syntactic description, then the ABNF, 
and finally a semantic description.)

    The trace fields are a group of header fields consisting of an 
optional
    "Return-Path:" field, and one or more "Received:" fields or other 
fields
    (indicated by "optional-field" below) that are defined by other
    specifications as belonging within the trace fields grouping. The
    "Return-Path:" header field contains a pair of angle brackets that 
enclose
    an optional addr-spec.  The "Received:" field contains a (possibly 
empty)
    list of tokens followed by a semicolon and a date- time 
specification.
    Each token must be a word, angle-addr, addr- spec, or a domain.  
Further
    restrictions are applied to the syntax of the trace fields by
    specifications that provide for their use, such as
    [I-D.klensin-rfc5321bis].

    trace           =   [return]
                        1*(received / optional-field)

    return          =   "Return-Path:" path CRLF

    path            =   angle-addr / ([CFWS] "<" [CFWS] ">" [CFWS])

    received        =   "Received:"
                        [1*received-token / CFWS] ";" date-time CRLF

    received-token  =   word / angle-addr / addr-spec / domain

    The trace fields document actions taken as a message moves through 
the
    transport system. A full discussion of the Internet mail use of the
    "Return-Path:" and "Received:" trace fields is contained in
    [I-D.klensin-rfc5321bis]; other specifications describe the use of 
other
    fields that are to be interpreted as trace fields.  For the purposes 
of
    this specification, the trace fields are strictly informational, and 
any
    formal interpretation of them is outside of the scope of this 
document.

pr

---

From: "Alexey Melnikov" <[email-masked]>
Date: Tue, 23 Feb 2021 18:13:52 +0000

>     formal interpretation of them is outside of the scope of this 

Works for me!

---

From: "John R Levine" <[email-masked]>
Date: 23 Feb 2021 13:57:12 -0500

On Tue, 23 Feb 2021, Pete Resnick wrote:
> I'm not sure I understand this last comment. There is no suggestion to mess 
> around with Received:. The proposal is to move optional-field into the 
> definition of trace in 3.6.7 rather than have it in the overall fields syntax 
> at the top of 3.6. Did I miss something?

I'm not Ned but if I were, I would probably have meant that when we want 
to add new trace info, we should add new header fields rather than adding 
new clauses to received headers.  For example, here's the Received line on 
the message I'm replying to, which shoehorns port numbers and TLS 
into the Received header in ways that are OK in 5322 but not 5321.

  Received: from episteme.net (episteme.net [216.169.5.102])
   by mail1.iecc.com ([64.57.183.56])
   with ESMTPS via TCP (port 38756/25) id 670930065
   tls TLS1.2_ECDHE_RSA_AES_256_GCM_AEAD; 23 Feb 2021 17:29:19 -0000

It is my impression that MTAs other than ones from Redmond WA currently 
comply pretty well with 5321, give or take shoehorning like the above.

Regards,

---

From: Alessandro Vesely <[email-masked]>
Date: Tue, 23 Feb 2021 19:58:46 +0100

On Tue 23/Feb/2021 18:34:39 +0100 Pete Resnick wrote:
>>> Yes, and if I do that, I'll remove the extra optional-field at the top of 
>>> the overall fields syntax in 3.6.
>>
> Because the entire trace block is optional according to the ABNF right above 
> that table in 3.6.

Ugh, that way you cannot have a lone Return-Path:?

>> all trace fields at the top of the header lest the message be syntactically 
>> bad is a recipe for trouble.
> 
> But optional-field is permitted at the bottom of the ABNF of fields in 3.6, 
> which would allow it to be used for something that is not a trace field. (That 
> is, optional-field appears twice in the 3.6 syntax.)

optional-field is a poor choice, as it is restricted to header fields not 
specified in the I-D.  That includes fields specified in other RFCs as well as 
unspecified fields.

Consider RFC 5293, where the *addheader* Sieve action is described like so:

    By default, the header field is inserted at the beginning of the
    existing message header.  If the optional flag ":last" is specified,
    it is appended at the end.

>> The practice to collect trace fields at the top comes from the fact that it 
>> is convenient to write down one's 2 cents and then copy the rest without even 
>> parsing it.  But a mail filter can add a header field holding information 
>> that semantically does not make it a trace field.  Even then, it is more 
>> natural to write added fields near the top of the header.  Doing so is also 
>> better for debugging and forensic analysis, as it is possible to deduce at 
>> what stage a given field was added.
>>
>> Original-From: field, formatted like From:.  Semantically equivalent to 
>> From:, and eventually swappable with it, the latter is definitely not a trace 
>> field. Should it be added downward, possibly near to the real From:, or isn't 
>> it more self-explanatory to add it near the corresponding 
>> Authentication-Results:?
>>
>> any other email filtering.  That syntax still conveys that trace fields tend 
>> to gather near the top of the header, but doesn't require a strict order.
> 
> Review the ABNF at the top of 3.6 and see if you can't express what you want 
> to. Otherwise, I think things are best left alone except for moving the first 
> occurrence of optional-field out of 3.6 and putting an optional-field (or 
> trace-extension-field) into 3.6.7.

You're right, I mismatched trace with top.  I gather we want to define trace by 
its semantics and convey that it inhabits the top of the header, but not 
exclusively.

Yet, the term optional-field excludes orig-date, from, and the like, which 
might happen to be thrown there by filter actions.

Perhaps the grammar could be expressed something like the following:

    fields          = *(trace
                        *resent-field /
                        prowl-field)
                      *(elemental-field /
                        optional-field)

    resent-field    = resent-date /
                      resent-from /
                      resent-sender /
                      resent-to /
                      resent-cc /
                      resent-bcc /
                      resent-msg-id

    elemental-field = orig-date /
                      from /
                      sender /
                      reply-to /
                      to /
                      cc /
                      bcc /
                      message-id /
                      in-reply-to /
                      references /
                      subject /
                      comments /
                      keywords

    prowl-field     = elemental-field /
                      optional-field

Best
Ale
--

---

From: Ned Freed <[email-masked]>
Date: Tue, 23 Feb 2021 12:08:18 -0800 (PST)

> I'm not Ned but if I were,

Be happy you're not ;-) ;-)

> I would probably have meant that when we want
> new clauses to received headers.

Exactly what I meant. Sorry for being unclear.

>  For example, here's the Received line on

>    tls TLS1.2_ECDHE_RSA_AES_256_GCM_AEAD; 23 Feb 2021 17:29:19 -0000

Maybe I'm reading it wrong, but there may be another problem here...
The TLS clause falls under the Additional-Registered-Clauses syntax
in 5321:

Additional-Registered-Clauses  = CFWS Atom FWS String

Which leads us to:

String         = Atom / Quoted-string
Atom           = 1*atext

And atext don't include period. Which means TLS1.2_ECDHE_RSA_AES_256_GCM_AEAD
should be quoted, and that takes you into interesting territory... Not 
sure I've ever seen a quoted clause parameter in a Received: field in the wild.

RFC 5322, on the other hand, has:

received        =   "Received:" *received-token ";" date-time CRLF
received-token  =   word / angle-addr / addr-spec / domain

And this takes us to (since word, like String, won't work):

domain          =   dot-atom / domain-literal / obs-domain
dot-atom-text   =   1*atext *("." 1*atext)

Which works syntactically, although TLS1.2_ECDHE_RSA_AES_256_GCM_AEAD is
a pretty strange domain name...

> comply pretty well with 5321, give or take shoehorning like the above.

I think that's to some extent because people don't push the boundaries too
much. When you it's pretty easy to get lost in the syntax.

				Ned

---

From: "John R Levine" <[email-masked]>
Date: 23 Feb 2021 15:48:48 -0500

On Tue, 23 Feb 2021, Ned Freed wrote:
> should be quoted, and that takes you into interesting territory... Not sure 
> I've ever seen a quoted clause parameter in a Received: field in the wild.

You're right, it hadn't occurred to me that the string might contain a 
dot.  I have code that turns forbidden characters in the TLS string to 
underscores so it does that to dots now, too.

Regards,

---

From: Pete Resnick <[email-masked]>
Date: Tue, 23 Feb 2021 15:13:56 -0600

On 23 Feb 2021, at 12:58, Alessandro Vesely wrote:

>> Because the entire trace block is optional according to the ABNF 
>> right above that table in 3.6.
>

No. Is there a reason to ever have one?

>> But optional-field is permitted at the bottom of the ABNF of fields 
>> in 3.6, which would allow it to be used for something that is not a 
>> trace field. (That is, optional-field appears twice in the 3.6 
>> syntax.)
>
> optional-field is a poor choice, as it is restricted to header fields 
> not specified in the I-D.  That includes fields specified in other 
> RFCs as well as unspecified fields.
>

So you're saying that you want it to be syntactically OK (that is, in 
the ABNF) to add one of the non-trace fields at the top of the message? 
There's already a discussion of this at the beginning of 3.6 saying, "It 
is important to note that the header fields are not guaranteed to be in 
a particular order." I'd really rather not change the syntax any more 
that strictly necessary.

>>> rest without even parsing it.

Indeed, not just convenient, but that it's often been damaging to go 
mucking around in the contents of the message. 5321 talks about this.

>>> But a mail filter can add a header field holding information that 
>>> semantically does not make it a trace field.  Even then, it is more 
>>> natural to write added fields near the top of the header.  Doing so 
>>> is also better for debugging and forensic analysis, as it is 
>>> possible to deduce at what stage a given field was added.

I'd like to see more examples, because rewriting From: and inserting 
Original-From: seems more like the gatewaying function that 5321 talks 
about.

>> Review the ABNF at the top of 3.6 and see if you can't express what 
>> you want to. Otherwise, I think things are best left alone except for 
>> moving the first occurrence of optional-field out of 3.6 and putting 
>> an optional-field (or trace-extension-field) into 3.6.7.
>
>
> You're right, I mismatched trace with top.  I gather we want to define 
> trace by its semantics and convey that it inhabits the top of the 
> header, but not exclusively.

I think the "not exclusively" is the part that there is controversy 
about.

> Yet, the term optional-field excludes orig-date, from, and the like, 
> which might happen to be thrown there by filter actions.

Are these filter actions part of submission or delivery? I'm not sure 
there's any need to update the syntax to allow for those at the top of 
the message.

pr

---

From: Ned Freed <[email-masked]>
Date: Tue, 23 Feb 2021 14:31:02 -0800 (PST)

> underscores so it does that to dots now, too.

is-atext-char-p is your friend.

				Ned

---

From: Alessandro Vesely <[email-masked]>
Date: Wed, 24 Feb 2021 13:09:42 +0100

On Tue 23/Feb/2021 22:13:56 +0100 Pete Resnick wrote:
>>>> That ABNF seems to imply there must be at least one Received:.  Why then 
>>>> does the table in Section 3.6 show a Min number of 0 trace?
>>>

You mean if there's a Return-Path then the message must have been Received? 
Hmm... yeah, that makes sense.

>>> which would allow it to be used for something that is not a trace field. 
>>> (That is, optional-field appears twice in the 3.6 syntax.)
>>
>> specified in the I-D.  That includes fields specified in other RFCs as well 
>> as unspecified fields.
>>
> So you're saying that you want it to be syntactically OK (that is, in the ABNF) 
> to add one of the non-trace fields at the top of the message?

If possible, it wouldn't be bad.

> There's already a discussion of this at the beginning of 3.6 saying, "It is
> important to note that the header fields are not guaranteed to be in a
> particular order." I'd really rather not change the syntax any more that
> strictly necessary.

The order of fields is all what the external parentheses are about.  Dropping 
the ")*(" in the middle would be a minimal change, byte-wise, wouldn't it?

It sounds like something of a nuisance to present a syntax by noting that it is 
not actual.

>>>> semantically does not make it a trace field.  Even then, it is more natural 
>>>> to write added fields near the top of the header.  Doing so is also better 
>>>> for debugging and forensic analysis, as it is possible to deduce at what 
>>>> stage a given field was added.
> 
> Original-From: seems more like the gatewaying function that 5321 talks about.

I don't think there's a gateway role.  Email filters are not firewalls 
interposed between client and server.  They're server extensions, rather on the 
MDA side of the SMTP server.

OTOH, one might consider the DMARC-compliant SMTP transmission from a MLM to a 
subscriber's MX as a sort of foreign environment that requires From: rewriting. 
  Restoring the original value of From: would then resume the previous state of 
affairs, just before delivery.

>>> to. Otherwise, I think things are best left alone except for moving the 
>>> first occurrence of optional-field out of 3.6 and putting an optional-field 
>>> (or trace-extension-field) into 3.6.7.
>>
>> You're right, I mismatched trace with top.  I gather we want to define trace 
>> by its semantics and convey that it inhabits the top of the header, but not 
>> exclusively.
> 
> I think the "not exclusively" is the part that there is controversy about.

Exactly.

> Are these filter actions part of submission or delivery? I'm not sure there's 
> any need to update the syntax to allow for those at the top of the message.

The above From: restoration occurs on delivery, after any possible forwarding. 
  I image most Sieve recipes also apply on or after receiving.  Submission 
filters are different.  Albeit they can add signatures or change encoding, in 
addition to MSA proper actions like setting Message-Id: and Date:, or adding 
the domain to addresses that miss it, their actions typically respect the 
correct order and type of header fields.

Stored messages can still be moved by gateways à la fetchmail, which may 
re-inject them via SMTP.  Thus, syntax transgressions that occur only after 
delivery are not bound to remain confined in a given mailbox.  Is it worth to 
consider that elemental fields found at the top of the header are likely to be 
somewhat apocryphal?

Best
Ale
--

---

From: Hector Santos <[email-masked]>
Date: Thu, 25 Feb 2021 23:06:04 -0500

> No. Is there a reason to ever have one?

Only if the backend wanted to give MUAs SPF power.  From UUCP "From " 
stripping design,  the move to SMTP and  stripping Return-Path carried 
on,  once final destination was determined.  Either the router did it 
or the gateway.  However, it was long felt SPF was best done at the 
backend.   The MUA could repeat the process by reading the SPF trace 
field  "Received-SPF"  or "Authentication-Result", if provided.

-- 
Hector Santos,
https://secure.santronics.com
https://twitter.com/hectorsantos

---

From: Ned Freed <[email-masked]>
Date: Sat, 27 Feb 2021 08:48:41 -0800 (PST)

> No. Is there a reason to ever have one?

I can think of at least two:

(1) IMAP APPEND of a (possibly draft) message (thus no Received: fields) where
    you want to say what the MAIL FROM was/is/should be.

(2) Message has had all Received: fields removed for security reasons. (And
    please don't bother telling me this is a bad idea, because you'd be
    barking up the wrong tree. A large number of security reviewers say
    otherwise, and when the choice is between no Received: fields and
    no deployment, guess what wins.)

> mucking around in the contents of the message. 5321 talks about this.

True, but let's keep in mind that RFC 5321 is about message transport, and
RFC 5322 applies more broadly. User agents muck around with the content
of messages all the time. It's kind of what they are for...

				ned

---

From: "John Levine" <[email-masked]>
Date: 28 Feb 2021 22:33:22 -0500

>    no deployment, guess what wins.)

(3) Message is delivered on the same MTA where it originated, and it
was originated by something like an internal sendmail program so it
doesn't go through a submission server.