Subject: [Emailcore] SMTP compliance suite?

From: Leo Gaspard <[email-masked]>
Date: Sat, 02 Jan 2021 20:03:32 +0100

Hello world,

With the evolution of mail systems and the RFCs and pre-RFCs that are
related to them, I wonder whether there would be interest from other
people than me for an “SMTP compliance suite 2020” in the XMPP spirit?

The idea would be to take all the RFCs and pre-RFCs that are related to
mail systems, classify the MTAs in a few categories (eg. off the top of
my head, “submission” / “forwarding” / “delivery” would probably be the
three main categories), and explicitly indicate for each category which
RFCs and pre-RFCs should be implemented, and which ones should not.

As a less directly compliance-suite-y extension, maybe it could include
best practices on how to implement said RFCs (eg. “A submitting mail
server that does not expect to send emails to mailing lists SHOULD have
a p=reject DMARC policy” — though I'm not totally sure we'd already be
ready for that, but let's not discuss this particular example in this
thread)

What do you think about this idea?

Cheers,
  Leo

---

From: "John Levine" <[email-masked]>
Date: 2 Jan 2021 17:40:12 -0500

>people than me for an “SMTP compliance suite 2020” in the XMPP spirit?

I wrote some scripts to test EAI compliance in MSA, MTA, and MDA (POP/IMAP)
software for the UASG.  You can find it here:

https://github.com/jrlevine/eaitesttools

The MSA and MTA tests sort of by default check the SMTP compliance of
whatever I'm testing. Not surprisingly, SMTP compliance was excellent
in MSA and MTA software not written in Redmond WA, and there the
errors were largely cosmetic, e.g., minor syntax errors in generated
Received headers.

I would be pretty surprised if any MSA or MTA you ever heard of failed
any test that anyone cared about.  Is there anything in particular you
were hoping to find out?

R's,
John

---

From: Dave Crocker <[email-masked]>
Date: Sat, 2 Jan 2021 16:47:32 -0800

> RFCs and pre-RFCs should be implemented, and which ones should not.

Do you mean something in a style like this:

      https://bbiw.net/clusters/#suitemail

d/

-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: Dave Crocker <[email-masked]>
Date: Sat, 2 Jan 2021 19:59:21 -0800

On 1/2/2021 4:47 PM, Dave Crocker wrote:
> classify the MTAs in a few categories (eg. off the top of
> my head, “submission” / “forwarding” / “delivery”

a further review of the above makes me wonder whether just reading RFC 
5598 would suffice?

d/

---

From: Leo Gaspard <[email-masked]>
Date: Sun, 03 Jan 2021 12:45:32 +0100

Dave Crocker <[email-masked]> writes:
> [and]

Something like the first link sounds indeed like exactly what I'm hoping
could get consensus for a 202x edition, thank you, I didn't know of this
website and it looks awesome! (Unfortunately the link is missing to
mention eg. DMARC, so it is not already what I'm looking for, and would
need at least updating.)

Also, it would maybe be useful to have such a compliance suite also
include documents that were well-known but should not be implemented
(currently thinking of SRS, not mentioned here, and for which I'm
personally still unclear as to whether forwarders SHOULD, SHOULD NOT or
MAY implement it as different destinations appear to have different
recommendations) — this would be going beyond the scope of XMPP
compliance suites, but on the other hand SMTP has a long history of
tried experiments and it's not always clear which ones were ever
formally retired or not.

What I'm thinking is, if we could get such an SMTP compliance suite be
published as an RFC, it might help formally retiring retired
experiments, as well as clarifying which ones are still running. And
maybe even add in best practices for how people nowadays are expected to
implement the quoted RFCs (eg. I'd expect best practices for DMARC
policy to start like “either have p=accept or pct=0 if you want to send
mail to mailing lists” nowadays, and to with time eventually crank up to
“p=reject pct=100” — but this requires the ability to make
evolving-with-time recommendation)

Now… I am not sure RFCs are the proper medium to define such a
compliance suite? But on the other hand, I don't see who else would have
legitimate authority to say “there is consensus and this is the set of
protocols that should be implemented nowadays to make both the mail
ecosystem and your own mail setup work best”

---

From: John C Klensin <[email-masked]>
Date: Sun, 03 Jan 2021 12:38:44 -0500

--On Sunday, January 3, 2021 12:45 +0100 Leo Gaspard
<ietf=[email-masked]> wrote:

>...
> What I'm thinking is, if we could get such an SMTP compliance
> suite be published as an RFC, it might help formally retiring
> retired experiments, as well as clarifying which ones are
> still running. And maybe even add in best practices for how
> people nowadays are expected to implement the quoted RFCs (eg.
> I'd expect best practices for DMARC policy to start like
> "either have p=accept or pct=0 if you want to send mail to
> mailing lists" nowadays, and to with time eventually crank
> up to "p=reject pct=100" — but this requires the ability
> to make evolving-with-time recommendation)
> 
> Now… I am not sure RFCs are the proper medium to define such
> a compliance suite? But on the other hand, I don't see who
> else would have legitimate authority to say "there is
> consensus and this is the set of protocols that should be
> implemented nowadays to make both the mail ecosystem and your
> own mail setup work best"

Leo,

Your original question was, I think, interpreted in several
different ways and I'm glad you found several of those responses
helpful.  I want to address the above, which is not about the
desirability of conformance/compliance suites or where to find
them about about the IETF going into that business.  First, to
clarify a bit more, IETF has explicit provisions for documents
that say "we recommend you do X" or even "start with N and
gradually work up to M".  Depending on how they are written,
what issues are addressed, and how they related to the technical
specifications themselves, those document can be either
"Applicability Statements" (A/Ss) or "Best Current Practice"
(BCP) specifications.  See RFC 2026.  Technical Specifications
and Applicability Statement materials can even be mixed in the
same document and you will find some of that in RFC 5321.

However, there is a long history of standard bodies going into
the actual compliance (or conformance) test business and it is
almost bad.  One reason is that, while the tests almost never
cover everything, they become the real standard with people
paying attention to the tests and what passes them and no to the
text of the standard.  In some cases, that has led to reduced
interoperability because of problems with things the standard
specifies but the tests don't cover.  In others, it has led to
actual damage, sometimes followed by litigation, when a party
has specified that products must pass the conformance tests only
to discover that products can pass the tests but not actually
work in production and under load.  

If a conformance suite is produced by Fred Example and Company,
those issues are rather different: they are that company's
opinion about how to test what conforms and their interpretation
of the standard(s) and how they fit together.  If that company
is sensible, they will not make strong claims that their tests
accurately cover every significant aspect of the standard, only
that they represent their best efforts to do so.   If they make
a strong claim, that, and the problems (and lawyers) that will
likely follow, are on their heads and do not compromise either
the standards or the SDOs.  

One of the examples that has come up actually illustrates this.
If I recall, John Levine and I had a minor disagreement about as
aspect of one of his tests of SMTPUTF8 implementations.  The
issue was minor (minor enough that I don't remember what it
was).   We could agree that it didn't make the tests unsuitable
for the purpose for which they were being used; and we could all
move forward.  But, had the same tests been formally adopted by
the IETF as proving or disproving conformance / compliance, it
would have been incumbent on us to work out those issues and
agree that they were completely consistent with the standards
(that is a stronger requirement than merely having consensus
that the tests were a good idea).

So, please don't even think about IETF-produced or IETF-endorsed
compliance/ conformance test suites.

    john

---

From: "John Levine" <[email-masked]>
Date: 3 Jan 2021 12:55:45 -0500

In article <36863283686B78FFFEA96D00@PSB> you write:
>was). ....

I don't remember what it was either, but there is at least one test
that a package failed, I wrote to its author who disagreed with my
interpretation of the standard. While I think my interpretation is
right, I am not so confident that I would tell people that the
package isn't standards compliant.

>compliance/ conformance test suites.

I assumed he was asking for advice on how to do his own compliance
tests, not to make it an IETF thing.

R's,
John

---

From: John C Klensin <[email-masked]>
Date: Sun, 03 Jan 2021 13:51:09 -0500

--On Sunday, January 3, 2021 12:55 -0500 John Levine
<[email-masked]> wrote:

>> One of the examples that has come up actually illustrates
>> this. If I recall, John Levine and I had a minor disagreement
>> about as aspect of one of his tests of SMTPUTF8
>> implementations.  The issue was minor (minor enough that I
>> don't remember what it was). ....
> 
> I don't remember what it was either, but there is at least one
> test that a package failed, I wrote to its author who
> disagreed with my interpretation of the standard. While I
> think my interpretation is right, I am not so confident that I
> would tell people that the package isn't standards compliant.

And that would be exactly the problem with an IETF-sanctioned or
published suite.  It is also where such things and the
robustness principle (and the "Do The Right Thing" one) come in
because the only real alternative is to try to specify every
detail, preferably in formal language that can be proven
complete and correct... and, in practice, that just doesn't work
with protocols with lots of cases and a need to adapt to, rather
than the ability to completely control, the environment in which
they operate.

>> So, please don't even think about IETF-produced or
>> IETF-endorsed compliance/ conformance test suites.
> 
> I assumed he was asking for advice on how to do his own
> compliance tests, not to make it an IETF thing.

I couldn't tell from the initial message.  But the note to which
I replied contained

>>> What I'm thinking is, if we could get such an SMTP
>>> compliance suite be published as an RFC, it might help
>>> formally retiring >> retired experiments, as well as
>>> clarifying which ones are still running. And maybe even
>>> add in best practices for how people nowadays are
>>> expected to implement the quoted RFCs 
>>> (eg. ...

and that seemed to me to be going rather far in the direction of
"IETF thing", hence the note/warning.

    john

---

From: Leo Gaspard <[email-masked]>
Date: Mon, 04 Jan 2021 00:11:25 +0100

(originally sent this email from a non-subscribed email address, sorry
for the duplicate)

> any test that anyone cared about.  Is there anything in particular you

Thank you for this link! It looks like a useful test suite. That said, I
was more thinking of an “SMTP compliance suite” as a compendium of good
practices and “what should be done to be a well-behaved citizen of the
SMTP system” more than as a test suite.

This is not to diminish the importance of test suites (it'd be great to
have a “SMTP compliance suite 2020 test suite”), but on the other hand
it feels to me like there should first be consensus on what such
compliance requirements would be.