Subject: [Emailcore] Ticket #42: G.12. Extension Keywords Starting in 'X-'

From: Alexey Melnikov <[email-masked]>
Date: Fri, 9 Jul 2021 14:11:31 +0100

Dear WG participants,

I would like to start a discussion on ticket #42 says:

   Section 2.2.2 contains a discussion of SMTP keywords starting in "X".
   Given general experience with such things and RFC 6648, is there any
   reason to not deprecate that practice entirely and remove that text?
   If we do so, should Section 4.1.5 be dropped or rewritten to make
   clear this is an obsolete practice?

We were faced with the same issue in the EXTRA WG when revising 
IMAP4rev1 and we basically removed any text about "X-", so now even an 
extension starting with "X-" can be registered in IMAP.

Below I quote all sections that mention "X" prefix and provide a 
strawman on what to change:

Last 2 paragraphs of Section 2.2.2 ("Definition and Registration of 
Extensions"):

OLD:
    In addition, any EHLO keyword value starting with an upper or lower
    case "X" refers to a local SMTP service extension used exclusively
    through bilateral agreement.  Keywords beginning with "X" MUST NOT be
    used in a registered service extension.  Conversely, keyword values
    presented in the EHLO response that do not begin with "X" MUST
    correspond to a Standard, Standards-Track, or IESG-approved
    Experimental SMTP service extension registered with IANA.  A
    conforming server MUST NOT offer non-"X"-prefixed keyword values that
    are not described in a registered extension.

    Additional verbs and parameter names are bound by the same rules as
    EHLO keywords; specifically, verbs beginning with "X" are local
    extensions that may not be registered or standardized. Conversely,
    verbs not beginning with "X" must always be registered.

Possible NEW:

    Keyword values presented in the EHLO response SHOULD
    correspond to a Standard, Standards-Track, or IESG-approved
    Experimental SMTP service extension registered with IANA.

(Note that I changed original MUST to SHOULD, taking into consideration 
that none of "X" extensions would be registered with IANA.)

OLD:

4.1.5.  Private-Use Commands

    As specified in Section 2.2.2, commands starting in "X" may be used
    by bilateral agreement between the client (sending) and server
    (receiving) SMTP agents.  An SMTP server that does not recognize such
    a command is expected to reply with "500 Command not recognized".  An
    extended SMTP server MAY list the feature names associated with these
    private commands in the response to the EHLO command.

    Commands sent or accepted by SMTP systems that do not start with "X"
    MUST conform to the requirements of Section 2.2.2.

NEW - drop the whole section? I am not sure if any of the above is 
useful if "X" prefix is not special anymore, in particular developers 
can figure out to use "500 Command not recognized" for unrecognized 
commands.

2nd paragraph of Section 8 ("IANA Considerations"):

OLD:
    o  The first, "Simple Mail Transfer Protocol (SMTP) Service
       Extensions" [49], consists of SMTP service extensions with the
       associated keywords, and, as needed, parameters and verbs. As
       specified in Section 2.2.2, no entry may be made in this registry
       that starts in an "X".  Entries may be made only for service
       extensions (and associated keywords, parameters, or verbs) that
       are defined in Standards-Track or Experimental RFCs specifically
       approved by the IESG for this purpose.

NEW:

    o  The first, "Simple Mail Transfer Protocol (SMTP) Service
       Extensions" [49], consists of SMTP service extensions with the
       associated keywords, and, as needed, parameters and verbs.

       Entries may be made only for service
       extensions (and associated keywords, parameters, or verbs) that
       are defined in Standards-Track or Experimental RFCs specifically
       approved by the IESG for this purpose.

Thoughts?

Best Regards,

Alexey

---

From: Dave Crocker <[email-masked]>
Date: Fri, 9 Jul 2021 10:27:55 -0700

> I would like to start a discussion on ticket #42 says:

When X- was suggested, during RFC 822's development -- and I wish I 
could remember who came up with the idea -- it seemed clever and useful. 
  I still think it was clever, but we learned that it had problematic 
downsides.  Hence the later deprecation.

The entire idea of syntactic support for 'private' constructs has proved 
problematic.  And so, yes, it should be removed, wherever possible.

Also wherever possible, we've learned that the simplest and most 
constructive model is for:

1. Extensible naming

2. An ability (but not requirement) for registering names and
    associating a specification, with registration reserving the name and
    its meaning.

3. Standardization of names that are, or are expected to be, important
    to service operation

4. A meta-rule to ignore a name that is not undersstood.

As such, the 'SHOULD' that is suggested here seems to go farther than 
necessary.  SHOULD is supposed to be for things that will break, if you 
don't conform to the should, unless you /really/ know what you are 
doing.  This doesn't seem to be a case of that.

d/
-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: Alexey Melnikov <[email-masked]>
Date: Fri, 9 Jul 2021 21:19:45 +0100

Hi Dave,

> On 9 Jul 2021, at 18:28, Dave Crocker <[email-masked]> wrote:
> 
> When X- was suggested, during RFC 822's development -- and I wish I could remember who came up with the idea -- it seemed clever and useful.  I still think it was clever, but we learned that it had problematic downsides.  Hence the later deprecation.
> 
> The entire idea of syntactic support for 'private' constructs has proved problematic.  And so, yes, it should be removed, wherever possible.
> 
> Also wherever possible, we've learned that the simplest and most constructive model is for:
> 
> 1. Extensible naming
> 

Agreed so far.
> 
> As such, the 'SHOULD' that is suggested here seems to go farther than necessary.  SHOULD is supposed to be for things that will break, if you don't conform to the should, unless you /really/ know what you are doing.  This doesn't seem to be a case of that.

The SHOULD requirement to register extensions with IANA improves interoperability and general reuse of good ideas, so I think it is warranted.

Do you have a counter proposal?

Best Regards,
Alexey

---

From: Dave Crocker <[email-masked]>
Date: Fri, 9 Jul 2021 13:44:43 -0700

On 7/9/2021 1:19 PM, Alexey Melnikov wrote:
>> As such, the 'SHOULD' that is suggested here seems to go farther than necessary.  SHOULD is supposed to be for things that will break, if you don't conform to the should, unless you/really/  know what you are doing.  This doesn't seem to be a case of that.

We don't have a SHOULD for registering header fields, why have one here?

The view that registration improves interoperability has obvious appeal, 
but the counter is that it imposes a burden to adoption.  That is, for 
someone considering creating an enhancement, they face the SHOULD.

Again, a SHOULD makes sense when there is a clear and present danger to 
non-compliance.  But as a general, coercive technique, I think it's 
actually a net-negative.

d/

---

From: "John Levine" <[email-masked]>
Date: 10 Jul 2021 00:06:31 -0400

It appears that Dave Crocker  <[email-masked]> said:
>>> As such, the 'SHOULD' that is suggested here seems to go farther than necessary.  SHOULD is supposed to be for things
>that will break, if you don't conform to the should, unless you/really/  know what you are doing.  This doesn't seem to be a
>case of that.

Presumably to avoid colliding with other people who would use the same
header name for something else. It's not like we're in any danger of
running out of name strings.

It might be interesting to Graham Klyne whether he gets a lot of requests for provisional header registrations 
and how extensive his expert review is.

R's,
John

---

From: Ned Freed <[email-masked]>
Date: Sun, 11 Jul 2021 13:44:14 -0700 (PDT)

> We don't have a SHOULD for registering header fields, why have one here?

Because these are very different cases. Anyone can add a new header field to the
mix with a couple of lines of code in pretty much any email component. That's
all it takes since most headers are for human, not machine, consumption - the
vast majority are never parsed by anything. And most of them are, frankly, junk.
Who cares if they are registered or not?

SMTP extensions are another matter. SMTP clients and servers both have to be
modified in an interoperable way, and to deploy an extension to any significent
extent likely means modifying more than one of each. And for the rare extension
that does deploy and which do something people want, tracking down information
on them in other to "keep up" can be a bit of pain.

Mind you, the only extensions of consequence I can think of have been
authentication extensions that appeared in SMTP courtesy of SASL, not pure SMTP
extensions, and thus aren't in scope here. And requirements language doesn't
seem to have helped in these cases.

But we cannot tell what the future holds, and requirements language is
always somewhat aspirational.

> someone considering creating an enhancement, they face the SHOULD.

Again, very different cases. Assuming there was a way to impose a header
registration requirement that people could not ignore, it really would
be a significant burden given how easy it is to deploy a header.

Developing and deploying an SMTP or authentication is vastly more difficult. The
added cost of a simple registration is not likely to be significent.

> non-compliance.  But as a general, coercive technique, I think it's

Non-compliance isn't the only issue. See above.

				Ned

---

From: Viktor Dukhovni <[email-masked]>
Date: Mon, 12 Jul 2021 15:39:30 -0400

> On 11 Jul 2021, at 4:44 pm, Ned Freed <[email-masked]> wrote:
> 

The Postfix "XFORWARD" and "XCLIENT" extensions seem to have gained
some traction and are implemented in various other software (e.g.
load balancers, and other MTAs).

	http://www.postfix.org/XFORWARD_README.html
	http://www.postfix.org/XCLIENT_README.html

-- 
	Viktor.

---

From: Ned Freed <[email-masked]>
Date: Tue, 13 Jul 2021 05:35:38 -0700 (PDT)

> 	http://www.postfix.org/XCLIENT_README.html

IME proxy protocol seems to be the preferred mechanism for load balancers -
becase it works with any protocol, not just SMTP.

And IME the choice of whether or not to implement Microsoft's client
identification authentication stuff is a far more consequential one.

But this is just quibbling. The SMTP world would be a better place i
XCLIENT were registered and specified. It's been on by to-do list for
years to write a formal specification for it. (Yes, we implement it.)
Maybe someday...

				Ned

---

From: Dave Crocker <[email-masked]>
Date: Tue, 13 Jul 2021 11:58:04 -0700

(Subject line changed, to focus on a basic point of concern.)

On 7/11/2021 1:44 PM, Ned Freed wrote:
> SMTP extensions are another matter. SMTP clients and servers both
> have to be modified in an interoperable way, and to deploy an
> extension to any significent extent likely means modifying more than
> one of each. And for the rare extension that does deploy and which do
> something people want, tracking down information on them in other to
> "keep up" can be a bit of pain.

The IETF has always been a bit confused about enforcement. The use of 
normative language encourages the misguided view that authors have a 
force of law over other folk.  This then extends into the realm of 
registration.  And yet the IETF's success did not come from coercing use 
but from satisfying needs.

When there is limited space for registration, figuring out a 
possibly-restrictive policy for allocations is required.  In other 
cases, it probably isn't.

The view that restrictive registration policies are otherwise necessary 
seems to come from a failure to distinguish tasks.  Standardization is 
where community 'approval' lies.  Not registration.  Registration is for 
disciplined administerion of an identification space.

We invoke a vague fear of potential harm from an allocation that 
involves activity that is not properly vetted.  So we require a 
specification, or even that it be standardized.  But the danger of 
requiring neither lacks a demonstrable danger.  Intuitively, the fear is 
reasonable, but, well... you know... theory vs. practice...

Say a random SMTP extension name is registered, and it has no 
specification.  As long as the meta-rule is "ignore what you don't 
understand", what is the danger?  And note that this case is already 
present, for UNregistered identifiers.  So, really, the question is what 
is the incremental danger from registering such allocations?

Say a random SMTP extension name is registered and has an associated 
specification that is crappy.  For this to cause meaningful damage both 
client and server implementers must fail to see how crappy it is. 
Again, this case already happens, such as with some 'experimental' 
specifications that actually do get IETF approval; and maybe even some 
standards track ones.  Again, what is the danger?

There is an absolute, community good to encouraging registration, 
because it makes it unlikely there will be name collisions; it 
advertises the existing of an extension activity, and it makes clear who 
is in charge of use of the name.

Even requiring a published specification decreases that good.

IETF standards do not succeed because their use is required but because 
the community likes the standards.  Registration should use the same model.

It's not that associating a specification isn't also a good.  And it 
isn't that standardization isn't a good.  It's that making them barriers 
to entry into a registration space is a bad.

d/

---

From: Ned Freed <[email-masked]>
Date: Tue, 13 Jul 2021 13:50:01 -0700 (PDT)

Agreed on all points. Our original "IETF specification required" position for
SMTP extensions was an error. We need to rectify that, and encouraging
registration, hopefully with a pointer to whatever specification does exist, is
IMO the right way forward.

				Ned

---

From: John C Klensin <[email-masked]>
Date: Tue, 13 Jul 2021 21:42:20 -0400

Largely agree with both of you.  However, please note that
extending the same logic too far renders the IETF (and standards
more generally) irrelevant because people should just establish
whatever arrangements they like and hope they get traction,
presumably on a "tell your friends" basis. 

>From that perspective, while some of the vocabulary we use to
describe out requirements is clearly wrong, IETF review and
endorsement should (and usually does) guarantee two things.  One
is that people other than the creators/ authors/ primary
advocates have been able to examine a specification, have done
so, and, depending on the degree of IETF endorsement, improve
upon and/or endorse the spec.   Are those steps worth anything?
That depends on the degree to which the IETF (and its
leadership) takes its role seriously, insists on informed
consensus where appropriate, and so on.   When a specification
is pushed through by a small cabal who are not really interested
in review and general input, much less the possibility of making
changes (and I suggest we have all seen that happen
occasionally), then the endorsement, however presented, is close
to meaningless whether the marketplace recognizes that or not.   

The other is that IETF-endorsed specification are useful in
getting ideas into circulation -- marketing them, if you prefer.
Perhaps in this world in which a relatively small number of
vendors control most of the action in many application areas
(including email), that aspect is inefficient and unnecessary:
it may be far more efficient for those vendors to collaborate,
produce specifications that meet their collective needs, and
then either use their proprietary nature to freeze everyone else
out or to get everyone else to adopt their ideas because, from a
market standpoint, interoperating with the relevant collection
of large gorillas is a condition for being useful.

So, maybe we need to worry less about what is a "standard" and
what isn't and the importance (or lack thereof) about IETF
approval/ consensus.    Instead, we should concentrate on
registration purely to avoid conflicting definitions; try to
figure out a way to avoid DoS and large-scale name-squatting
attacks on that process; stress the importance of published,
easily-accessible, well-vetted, and  clear and comprehensible
documents; and perhaps give out gold stars for the degree of
consensus, document quality, acceptance, and deployment should
people decide to apply for them.

I am not sure how much difference it would make other than,
maybe, reducing the number of unregistered identifiers a bit.
But perhaps that is enough.

best,
   john

--On Tuesday, July 13, 2021 13:50 -0700 Ned Freed
<[email-masked]> wrote:

>> > SMTP extensions are another matter. SMTP clients and
>> > servers both have to be modified in an interoperable way,
>> > and to deploy an extension to any significent extent likely
>> > means modifying more than one of each. And for the rare
>> > extension that does deploy and which do something people
>> > want, tracking down information on them in other to "keep
>> > up" can be a bit of pain.
> 
> 
>> The IETF has always been a bit confused about enforcement.
>> The use of normative language encourages the misguided view
>> that authors have a force of law over other folk.  This then
>> extends into the realm of registration.  And yet the IETF's
>> success did not come from coercing use but from satisfying
>> needs.
> 
>> possibly-restrictive policy for allocations is required.  In
>> other cases, it probably isn't.
> 
>> The view that restrictive registration policies are otherwise
>> necessary seems to come from a failure to distinguish tasks.
>> Standardization is where community 'approval' lies.  Not
>> registration.  Registration is for disciplined administerion
>> of an identification space.
> 
>> We invoke a vague fear of potential harm from an allocation
>> that involves activity that is not properly vetted.  So we
>> require a specification, or even that it be standardized.
>> But the danger of requiring neither lacks a demonstrable
>> danger.  Intuitively, the fear is reasonable, but, well...
>> you know... theory vs. practice...
> 
>> specification.  As long as the meta-rule is "ignore what you
>> don't understand", what is the danger?  And note that this
>> case is already present, for UNregistered identifiers.  So,
>> really, the question is what is the incremental danger from
>> registering such allocations?
> 
>> Say a random SMTP extension name is registered and has an
>> associated specification that is crappy.  For this to cause
>> meaningful damage both client and server implementers must
>> fail to see how crappy it is. Again, this case already
>> happens, such as with some 'experimental' specifications that
>> actually do get IETF approval; and maybe even some standards
>> track ones.  Again, what is the danger?
> 
>> There is an absolute, community good to encouraging
>> registration, because it makes it unlikely there will be name
>> collisions; it advertises the existing of an extension
>> activity, and it makes clear who is in charge of use of the
>> name.
> 
>> IETF standards do not succeed because their use is required
>> but because the community likes the standards.  Registration
>> should use the same model.
> 
>> It's not that associating a specification isn't also a good.
>> And it isn't that standardization isn't a good.  It's that
>> making them barriers to entry into a registration space is a
>> bad.
> 
>> d/
>> --
>> Dave Crocker

---

From: Dave Crocker <[email-masked]>
Date: Tue, 13 Jul 2021 18:57:22 -0700

> presumably on a "tell your friends" basis.

Forgive me, John, but that -- and the rest of your note -- profoundly 
miss the point I made.

Oddly, I'll even claim that my note covered -- and arguably countered -- 
most of the concerns you seem to be raising, except that some seem to be 
entirely outside the scope of my note.

d/