Subject: Re: [dmarc-ietf] Definition of "value" in RFC8601

From: "John Levine" <[email-masked]>
Date: 1 Apr 2020 12:42:02 -0400

In article <[email-masked]> you write:
>"If the authserv-id is UTF-8, then it might be UTF-8"

Yes.

>This seems equivalent to
>
>> authserv-id = sub-domain *("." sub-domain)
>>    ; Where sub-domain is imported from 6531 for
>>    ; any mail

No, it's 6531 for EAI messages, and 5321 for ASCII messages.

EAI and ASCII mail are separate mail streams. The component that is
parsing the A-R header had better know ahead of time whether your
system is treating them separately.

Since 6531 is a strict superset of 5321, if your system handles EAI
messages (many still don't) then you can treat them all as EAI, give
or take issues with forwarding messages with A-R headers to non-ASCII
recipients.

---

From: Damian Lukowski <[email-masked]>
Date: Wed, 1 Apr 2020 19:52:34 +0200

> No, it's 6531 for EAI messages, and 5321 for ASCII messages.

If they are not equivalent, there must be a counterexample in which a
parser with

>    ; Where sub-domain is imported from 6531 for
and another parser with

>    ; Where sub-domain is imported from 5321 for ASCII mail
>    ; and 6531 for EAI mail.

, given identical input, disagree on the validity of the A-R header of
that input. Can you please show an example?

> system is treating them separately.

Can you elaborate on this paragraph? Know what ahead of time? It sounds
like you agree, that the validity of an A-R header does depend on
external factors, not solely on the A-R content.

---

From: "John Levine" <[email-masked]>
Date: 1 Apr 2020 16:29:22 -0400

>that input. Can you please show an example?

It's the one you found.  If the authserv-id has a non-ASCII UTF-8 character,
it's invalid under 5321 and valid under 6531.

>external factors, not solely on the A-R content.

Right.  ASCII mail and EAI mail are handled as separate streams, even if
they share a mail server.  See this old blog post of mine:

https://jl.ly/Email/i18n3.html

R's,
John

---

From: Damian Lukowski <[email-masked]>
Date: Wed, 1 Apr 2020 22:44:32 +0200

> it's invalid under 5321 and valid under 6531.

I thought we established that as soon as the authserv-id has non-ASCII
UTF-8 characters, the mail is automatically - by definition - a 6531 mail:

> As a close approximation, any message which has an x80 bit set in a
> character in the header is an EAI message.

> they share a mail server.  See this old blog post of mine:

I'll look into it.

---

From: "John Levine" <[email-masked]>
Date: 1 Apr 2020 17:05:49 -0400

>UTF-8 characters, the mail is automatically - by definition - a 6531 mail:

Sorry if it seemed like I said that.  It's either a potentially valid
EAI message or a definitely invalid ASCII message.  If your system
doesn't handle EAI mail, it's invalid.  If there are bytes with the
high bit set but they're not a UTF-8 sequence, or they're not a UTF-8
character allowed in an authserv-id it's invalid even if you handle
EAI.

R's,
John

---

From: Damian Lukowski <[email-masked]>
Date: Wed, 1 Apr 2020 23:26:30 +0200

> doesn't handle EAI mail, it's invalid.

Ok, that is definitely an external factor, isn't it? The information, if
the system which processed the email, handles EAI, is not contained
within the A-R header. Is the information even available in the DATA
portion of an SMTP transaction? Can one look inside an .eml file and
decide if that email has been processed by an EAI capable system?

> If there are bytes with the

That part is fine, because this is decidable from the A-R header itself.

---

From: Damian Lukowski <[email-masked]>
Date: Thu, 2 Apr 2020 01:10:46 +0200

Let's say the system is capable of handling EAI mail. RC6531 sec 3.6:

> An SMTPUTF8-aware SMTP client or server
> that requires accurate knowledge of whether a message is
> internationalized needs to parse all message header fields ...

If I understand the section and your new authserv-id definition
correctly, an EAI capable system can process plain-ASCII messages, such
that with then definition of

>    ; Where sub-domain is imported from 5321 for ASCII mail

, sub-domain comes from RFC5321.

If that is so, then this looks to me like a dependency cycle.

To decide for an authserv-id whether to pull sub-domain from 5321 or
6531, one needs to know if the message is internationalized. But to
obtain that information, one must have parsed all message header fields,
per 6531 sec 3.6, including A-R, that is still waiting to be parsed.

---

From: "John Levine" <[email-masked]>
Date: 1 Apr 2020 19:17:52 -0400

>decide if that email has been processed by an EAI capable system?

Maybe.  The EAI flag is sent on the MAIL TO command.  See RFC 6531,
sec 3.4.  If the recipient system adds the usual Received: header, it
should include a "WITH UTF8SMTP" or "WITH UTF8SMTPS" clause.

I have no idea what's in a .eml file, since the only spec I've seen for 
it is whatever some version of Outlook does.

R's,
John

---

From: "John Levine" <[email-masked]>
Date: 1 Apr 2020 20:32:10 -0400

>per 6531 sec 3.6, including A-R, that is still waiting to be parsed.

In systems that keep ASCII and EAI mail separate, that info will be in
the metadata, whether it arrived in an SMTP session with the SMTPUTF8
option.

---

From: Brandon Long <[email-masked]>
Date: Thu, 2 Apr 2020 16:00:10 -0700

> the metadata, whether it arrived in an SMTP session with the SMTPUTF8

Does anyone implement it that way?

We just upgraded our parser to handle EAI mail, and the parser returns
requested data
in UTF8, decoding 5322 mail or going straight from 6532 mail as necessary,
with
specific accessors needing to know what the possible encodings could be.
We then check
when we go to send mail whether we should specify SMTPUTF8 on the output..
as for input...

We see plenty of mail transferred without SMTPUTF8 that contains UTF8 (or
other
8-bit characters, but the majority of that became UTF8 probably close to a
decade ago), and we
just handle it.

Unfortunately, everybody thinks they can just make their own emali messages
because its a text-like
format, and they're very often wrong.

Brandon

---

From: "John R Levine" <[email-masked]>
Date: 2 Apr 2020 20:10:36 -0400

> Does anyone implement it that way?

I have no idea.  I don't in my code either.

> because its a text-like format, and they're very often wrong.

Surely you're aware that 50% of all programmers are of below average 
skill.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Brandon Long <[email-masked]>
Date: Fri, 3 Apr 2020 13:01:58 -0700

> Surely you're aware that 50% of all programmers are of below average

Clearly.  But it goes beyond that, even programmers who are of above
average skill
aren't expected to be knowledgeable about all of the sharp edges they can
run into.

Brandon