Subject: [Emailcore] 'obsolete' considered misleading

From: Dave Crocker <[email-masked]>
Date: Mon, 19 Apr 2021 08:16:20 -0700

The idea of marking some existing constructs for future deprecation is 
intuitive and appealing.  The experience of doing this at Internet 
scale?  Not so much.

Yet another demonstration of the difference between theory and practice. 
  (cf, "X-").

When something has been marked for deprecation for 25 years but hasn't 
been deprecated, it might mean that the particular choices in assigning 
that status were wrong.

More likely, it means the construct of an obsolescence sequence was wrong.

d/
-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: Pete Resnick <[email-masked]>
Date: Tue, 20 Apr 2021 11:15:45 -0500

On 19 Apr 2021, at 10:16, Dave Crocker wrote:

> Yet another demonstration of the difference between theory and 
> practice.  (cf, "X-").
>
> been deprecated, it might mean that the particular choices in 
> assigning that status were wrong.
>

Luckily, none of 2822, 5322, nor 5322bis say that such constructs are 
"for future deprecation", and there was never any such implication 
AFAICT. They are "obsolete" only in the sense that they are (or at least 
ought to be) no longer produced and are out of date. They are defined 
because implementations must be able to appropriately parse extant 
messages in storage. Perhaps "obsolete" was not the best label, because 
it implies that they are no longer produced at all, and we don't 
actually know that for sure (and there's pretty good reason to believe 
that old implementations appear in the weirdest of places), but 
deprecation was never on the table.

pr

-- 
Pete Resnick https://www.episteme.net/
All connections to the world are tenuous at best

---

From: Dave Crocker <[email-masked]>
Date: Tue, 20 Apr 2021 09:20:39 -0700

> deprecation was never on the table.

If receivers are expected to continue to parse something, then what is 
the claimed benefit of prohibiting their use in generation?

None of the *22 specifications explain/justify this.

d/

ps.  If the construct reasonably must still be supported -- even if only 
by receivers -- the term 'obsolete' is, as noted, misleading.

---

From: Pete Resnick <[email-masked]>
Date: Tue, 20 Apr 2021 11:31:47 -0500

On 20 Apr 2021, at 11:20, Dave Crocker wrote:

>> AFAICT. They are "obsolete" only in the sense that they are (or at 
>> least ought to be) no longer produced and are out of date. They are 
>> defined because implementations must be able to appropriately parse 
>> extant messages in storage. Perhaps "obsolete" was not the best 
>> label, because it implies that they are no longer produced at all, 
>> and we don't actually know that for sure (and there's pretty good 
>> reason to believe that old implementations appear in the weirdest of 
>> places), but deprecation was never on the table.
>

As it says in 
https://www.ietf.org/archive/id/draft-ietf-emailcore-rfc5322bis-01.html#section-1.2.3-6 
:

    The rules of the obsolete syntax are elements that have appeared in
    earlier versions of this specification or have previously been 
widely
    used in Internet messages.  As such, these elements MUST be
    interpreted by parsers of messages in order to be conformant to this
    specification.  However, since items in this syntax have been
    determined to be non-interoperable or to cause significant problems
    for recipients of messages, they MUST NOT be generated by creators 
of
    conformant messages.

> ps.  If the construct reasonably must still be supported -- even if 
> only by receivers -- the term 'obsolete' is, as noted, misleading.

Yes, as I just said, perhaps the term is not ideal. But again, it 
doesn't imply "deprecation", which was what your initial message seemed 
to be about.

pr

---

From: Dave Crocker <[email-masked]>
Date: Tue, 20 Apr 2021 09:38:51 -0700

On 4/20/2021 9:31 AM, Pete Resnick wrote:
>    earlier versions of this specification or have previously been widely
>    for recipients of messages, they MUST NOT be generated by creators of
>    conformant messages.

Ahh. Thanks.  I made the mistake of looking for text by using "obs-".  sigh.

Consider the irony of saying that receivers must support some construct 
and simultaneously claiming the constructs are "determined to be 
non-interoperable or to cause significant problems for recipients of 
messages".

I'm pressing this point because we generally do a very bad job of 
talking about, specifying, and generally dealing with the realities 
about getting rid of bits of protocol usage, over time.  We tend to have 
a reasonable goal, but seem erratic, or worse, at achieving it.

I think we need vastly more care, description, specification, vetting, 
and monitoring of anything we think needs to be expunged over time.

d/

---

From: Alessandro Vesely <[email-masked]>
Date: Wed, 21 Apr 2021 18:31:45 +0200

>>    used in Internet messages.  As such, these elements MUST be

Couldn't that be relaxed to "these elements MAY be interpreted by parsers"?

It is already quite difficult to find correct implementations for the accepted 
grammar.  Requiring that obs- be also implemented can push compliance so far 
that no programmers will want to tackle it.  Perhaps, proposing a reachable 
target can improve parsers quality...

> I think we need vastly more care, description, specification, vetting, and 
> monitoring of anything we think needs to be expunged over time.

+1

Best
Ale
--

---

From: Dave Crocker <[email-masked]>
Date: Sat, 24 Apr 2021 11:06:24 -0700

On 4/21/2021 9:31 AM, Alessandro Vesely wrote:
> It is already quite difficult to find correct implementations for the 
> accepted grammar.  Requiring that obs- be also implemented can push 
> compliance so far that no programmers will want to tackle it.  Perhaps, 
> proposing a reachable target can improve parsers quality...

1. RFC2822, RFC 5322 and the bis draft already contain a normative 
'MUST' for support of the "obs-*" constructs.  Which means that the 
/requirement/ for parsing and semantics support have been in  place for 
(exactly) 20 years.  There is no new burden, here.

2. Possibly moving to a SHOULD, but definitely moving to a MAY, actually 
would effectively deprecate the constructs.  The question will be why 
that has not been ok for 20 years, but now is?  (I'd expect the barrier 
for satisfying that answer to be especially high, given the wg charter.)

3. The fact that support has been required for 20 years fundamentally 
undermines the concept of 'obsolete'.

Again:  it was an entirely reasonable idea.  The fact of continued 
requirement demonstrates a failed semantic.  It probably also represents 
a failed idea.  If it is still required for parsers then it is still 
being generated.

d/

---

From: "John Levine" <[email-masked]>
Date: 24 Apr 2021 15:03:29 -0400

>for satisfying that answer to be especially high, given the wg charter.)

Seems to me it depends on how much mail still uses them.

I can't recall the last time I saw a route in an address, or a space
before the colon in a header in non-spambot mail.

Perhaps there are some archives of mail we could grep through.

R's,
John

---

From: "Peter J. Holzer" <[email-masked]>
Date: Sat, 24 Apr 2021 22:09:49 +0200

On 2021-04-24 11:06:24 -0700, Dave Crocker wrote:
> 1. RFC2822, RFC 5322 and the bis draft already contain a normative 'MUST'
> for support of the "obs-*" constructs.  Which means that the /requirement/
> for parsing and semantics support have been in  place for (exactly) 20
> years.  There is no new burden, here.
> 
> would effectively deprecate the constructs.  The question will be why that
> has not been ok for 20 years, but now is?  (I'd expect the barrier for
> satisfying that answer to be especially high, given the wg charter.)

Well, for one, RFC 5322 was published in 2008, so nobody has checked
whether it was or wasn't ok for 13 years.

It has now been a standards violation to generate messages with these
obsolete forms for 20 years. One could argue that after 20 years
programs which still haven't been updated are so rare that we don't have
to bother. 

I don't know wether that's true. I would certainly want to have access
to a much larger and more diverse corpus of messages than I have to answer
that question.

I also don't remember whether we had any evidence that these forms were
still in use in 2008.

> undermines the concept of 'obsolete'.

Yes. If you call something "obsolete" you should plan to remove it
eventually (typically in the next revision or the one after that).

> If it is still required for parsers then it is still being generated.

That's a non-sequitur.

        hp

-- 
   _  | Peter J. Holzer    | Story must make more sense than reality.
|_|_) |                    |
| |   | [email-masked]         |    -- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |       challenge!"

---

From: Pete Resnick <[email-masked]>
Date: Sat, 24 Apr 2021 15:28:38 -0500

On 24 Apr 2021, at 14:03, John Levine wrote:

> On 24 Apr 2021, at 13:06, Dave Crocker wrote:
>
>>> It is already quite difficult to find correct implementations for 
>>> the accepted grammar.

On what obs- syntax are implementations failing?

>>> Requiring that obs- be also implemented can push compliance so far 
>>> that no programmers will want to tackle it.  Perhaps, proposing a 
>>> reachable target can improve parsers quality...

I don't understand this bit: If we change the document to say that it's 
not required, that would cause people to implement things they haven't 
already?

>> requirement demonstrates a failed semantic.  It probably also 
>> represents a failed idea.  If it is still required for parsers then 
>> it is still being generated.

That doesn't follow: As I said in my earlier messages, one of the 
motivations for specifying the old constructs and expressing the need to 
parse them was extant mail stores, not simply that they are still being 
generated. For example, in the [email-masked] list archive, there are 
messages from 1992 that still use the 3-letter time zones which appear 
in the obs-zone construct and 2-digit years that appear in the obs-year 
construct. My IMAP client (MailMate) parses them fine and sorts the 
messages properly.

> I can't recall the last time I saw a route in an address...

Found some of those from 1992 in the same mailbox. My IMAP client 
handles those fine (though I'm not sure what a server receiving one of 
them will do; probably ignore the route per 5321).

> ...or a space before the colon in a header in non-spambot mail.

Still on the lookout for one of those.

> Perhaps there are some archives of mail we could grep through.

I'm sure we have lots more.

pr

---

From: Pete Resnick <[email-masked]>
Date: Sat, 24 Apr 2021 15:47:29 -0500

On 24 Apr 2021, at 15:09, Peter J. Holzer wrote:

> One could argue that after 20 years

As I replied in Dave's message: Existing message stores are also an 
issue.

> eventually (typically in the next revision or the one after that).

The fact that we chose "obsolete" to describe these may be unfortunate 
(perhaps we should have called it "accept-only") and perhaps it was born 
out of my (our?) thinking that we were in fact the protocol police back 
in my younger days, but it really doesn't change whether we choose to 
keep those constructs around or not. As an interesting example from 
section 4.1:

       |  Note: The "period" (or "full stop") character (".") in obs-
       |  phrase is not a form that was allowed in earlier versions of
       |  this or any other specification.  Period (nor any other
       |  character from specials) was not allowed in phrase because it
       |  introduced a parsing difficulty distinguishing between phrases
       |  and portions of an addr-spec (see section 4.4).  It appears
       |  here because the period character is currently used in many
       |  messages in the display-name portion of addresses, especially
       |  for initials in names, and therefore must be interpreted
       |  properly.

The "." appearing in a phrase was not in the least "obsolete" in the 
true definition of the word; it was a newer existing practice that 
violated the previous standard for parsing messages. But it needed to be 
part of the "accept" syntax because we knew parsers were going to need 
to deal with it.

I just can't get myself excited about worries that the terminology use 
in this case is wrong or misleading. Maybe it is. But (to create a 
Farberism) that's a lot of spilled milk over the bridge and now we've 
got to lie in it.

pr

---

From: Martin J. Dürst <[email-masked]>
Date: Sun, 25 Apr 2021 10:11:17 +0900

Hello Pete, others,

> construct. My IMAP client (MailMate) parses them fine and sorts the 

I'm not at all a mail expert, just a bystander, but could it possible 
that this means that MUAs and similar stuff MUST/SHOULD still be able to 
deal with the obs syntax, but MTAs and such don't? Just a wild guess, 
and even if true, wouldn't be sure it's worth making that distinction.

Regards,   Martin.

---

From: John C Klensin <[email-masked]>
Date: Sat, 24 Apr 2021 21:43:42 -0400

--On Sunday, April 25, 2021 10:11 +0900 "Martin J. Dürst"
<[email-masked]> wrote:

>...
> I'm not at all a mail expert, just a bystander, but could it
> possible that this means that MUAs and similar stuff
> MUST/SHOULD still be able to deal with the obs syntax, but
> MTAs and such don't? Just a wild guess, and even if true,
> wouldn't be sure it's worth making that distinction.

Martin,

Not really the issue here.   First, although many things have
happened (both in how the protocol specs are written in a real
life), 5322 and its predecessors are traditionally entirely
about "MUAs and similar stuff".  The SMTP specification (and
hence MTAs) treat obsolete or deprecated features somewhat
differently.  In particular, several bits in SMTP essentially
prohibit choking on old syntax / features but recommend or
require ignoring them.

More important, in both specs (separately and in different
ways), the general approach has been something like "don't send
that but, because it used to be valid and/or required, you must
continue to accept it".   So everything is required to "deal
with the obs syntax"; the prohibitions are just on generating/
sending it.

Like Pete, I find it very hard to get excited about this.  I
also suspect that we would be far more likely to cause harm by
trying to tinker with how outdated features or syntax are used
than by leaving things alone.

best,
   john

---

From: Dave Crocker <[email-masked]>
Date: Sat, 24 Apr 2021 18:56:41 -0700

On 4/24/2021 1:28 PM, Pete Resnick wrote:
>          3. The fact that support has been required for 20 years
>             fundamentally undermines the concept of 'obsolete'.
> 
>         represents a failed idea. If it is still required for parsers
>         then it is still being generated.
> 
> generated. For example, in the [email-masked] <mailto:[email-masked]> list 
> archive, there are messages from 1992 that still use the 3-letter time 
> zones which appear in the obs-zone construct and 2-digit years that 
> appear in the obs-year construct. My IMAP client (MailMate) parses them 
> fine and sorts the messages properly.

      Merriam-Webster: Obsolete -- "no longer in use or no longer useful"

The term is misapplied here.  The term made sense when there was an
actual expectation of removing the constructs.  My recollection is that
there was.  Again, at the time, it wasn't a silly expectation.

There cannot be a reasonable expectation of that, at this point.  That
makes the terminology wrong.

The use of the term misleads the reader. In spite of the MUST, it 
encourages thinking that the constructs can be ignored, when they can't.

The reality is that removing things from an installed base like the
Internet has consistently been proving somewhere between incredibly
difficult and impossible.  Design work that works from that view stands
a better change of being understood and implemented reasonably well.
Language that purports there will be a future with removal invites
confusion.

d/

---

From: John C Klensin <[email-masked]>
Date: Sat, 24 Apr 2021 22:52:14 -0400

--On Saturday, April 24, 2021 18:56 -0700 Dave Crocker
<[email-masked]> wrote:

>>          3. The fact that support has been required for 20
>>          years fundamentally undermines the concept of
>>             'obsolete'.
>> 
>>         Again: it was an entirely reasonable idea. The fact
>>         of continued requirement demonstrates a failed
>>         semantic. It probably also represents a failed idea.
>> That doesn't follow: As I said in my earlier messages, one of
>> the  motivations for specifying the old constructs and
>> expressing the need to  parse them was extant mail stores,
>> not simply that they are still being  generated. For example,
>> in the [email-masked] <mailto:[email-masked]> list  archive,
>> there are messages from 1992 that still use the 3-letter time 
>> zones which appear in the obs-zone construct and 2-digit
>> years that  appear in the obs-year construct. My IMAP client
>> (MailMate) parses them  fine and sorts the messages properly.
> 
> 
> The use of the term misleads the reader. In spite of the MUST,
> it encourages thinking that the constructs can be ignored,
> when they can't.
> 
> The reality is that removing things from an installed base
> like the Internet has consistently been proving somewhere
> between incredibly difficult and impossible.  Design work that
> works from that view stands a better change of being
> understood and implemented reasonably well. Language that
> purports there will be a future with removal invites confusion.

Dave,

Referring to my response to Martin, It seems to be that the spec
is fairly clear that what is intended is "don't produce/ send
this, but, for compatibility reasons, you need to continue to
accept it".  That seems to me to be both viable and pragmatic.
It, in turn, seems to be consistent with the argument you are
making above: as a particular example, I find the number of
two-digit years we seem to be seeing in dates in recent years to
be rather few although, like Pete, I am concerned enough about
older mail archives that I think it would be unwise to pretend
they were never valid.

Given that, I don't understand what you are suggesting/
requesting.  I trust it is not to go back to recommending the
generation and use of, e.g., two-digit years.  Do you have
better terminology to suggest to replace the uses of "obsolete"
and "obs-" while conveying the "don't send but must handle
appropriately if received" idea and, if so, what would it be?

I have also observed that you have several times made the
argument that (at least as I have understood you) nothing should
be considered in this WG unless their is compelling need for a
change.  While I continue to believe that a narrow
interpretation of that criterion to cut off discussion is
undesirable, because it is an argument you have made I'd like to
understand how you think it applies in this case.  Do you see a
compelling need for change?  Do you have evidence that
presenting older syntax as "obsolete" or at part of "obs-XXX"
has caused or continues to cause harm and/or confusion about the
interpretation of the spec?  

thanks,
   john

---

From: Alessandro Vesely <[email-masked]>
Date: Mon, 26 Apr 2021 12:54:33 +0200

On Sun 25/Apr/2021 04:52:14 +0200 John C Klensin wrote:
> --On Saturday, April 24, 2021 18:56 -0700 Dave Crocker wrote:
>>>> The fact of continued requirement demonstrates a failed semantic. It
>>>> probably also represents a failed idea. If it is still required for
>>>> parsers then it is still being generated. >>>
>>> generated. >>
>> encourages thinking that the constructs can be ignored, when they can't. >>
>> difficult and impossible.  Design work that works from that view stands a
>> better change of being understood and implemented reasonably well.
>> confusion. >
> It seems to be that the spec is fairly clear that what is intended is "don't
> produce/ send this, but, for compatibility reasons, you need to continue to 
> accept it".  That seems to me to be both viable and pragmatic. It, in turn,
> seems to be consistent with the argument you are making above: as a
> particular example, I find the number of two-digit years we seem to be
> seeing in dates in recent years to be rather few although, like Pete, I am
> concerned enough about older mail archives that I think it would be unwise
> to pretend they were never valid.

What is not so clear in the spec is the nature of those compatibility reasons.

After Pete's examples, it is clear that there is a treasure trove of older mail 
archives.  From such perspective, it doesn't sound so misleading to call their 
format "obsolete".  It is an old format, no longer in use.

Specifying the scope of obs- syntax might seem to weaken the MUST somewhat, for 
example like so:

OLD:
    Section 4 of this document specifies an "obsolete" syntax.  There are
    references in section 3 to these obsolete syntactic elements.  The
    rules of the obsolete syntax are elements that have appeared in
    earlier versions of this specification or have previously been widely
    used in Internet messages.  As such, these elements MUST be
    interpreted by parsers of messages in order to be conformant to this
    specification.  However, since items in this syntax have been
    determined to be non-interoperable or to cause significant problems
    for recipients of messages, they MUST NOT be generated by creators of
    conformant messages.

NEW:
    Section 4 of this document specifies an "obsolete" syntax.  There are
    references in section 3 to these obsolete syntactic elements.  The
    rules of the obsolete syntax are elements that have appeared in
    earlier versions of this specification or have previously been widely
    used in Internet messages.  Items in this syntax have been determined
    to be non-interoperable or to cause significant problems for
    recipients of messages, they MUST NOT be generated by creators of
    conformant messages.  However, old mail archives exist and are being
    being actively preserved.  Parsers which may happen to be applied to
    such content MUST be able to interpret obs- elements, and hence be
    fully conformant to this specification.

Best
Ale
--

---

From: Pete Resnick <[email-masked]>
Date: Tue, 27 Apr 2021 17:51:00 -0500

Answering a couple of items in this reply:

On 26 Apr 2021, at 19:47, Keith Moore wrote:

> Regarding the use of the word "obsolete" to describe once common 
> syntax
> that is less favored today: Consider "legacy" as an alternative, and 
> try
> to make it clear that readers need to continue to support that syntax
> indefnintely.   I agree that "obsolete" conveys the wrong 
> impression.

I grant the point that "legacy" (or, as I mentioned before, 
"accept-only", or probably other choices) would have probably been 
better than "obsolete". But I still haven't seen an answer to: Is there 
a proposal to change the terminology and/or the ABNF element names? Or 
is there a proposal to better explain in section 1.2.3 (as quoted by 
Alessandro below) that we chose a misleading or confusing name for these 
things, and we apologize for that, but we're sticking with it to avoid 
even more confusion? Or is there some other proposal for what you all 
want done in the document? This editor needs guidance to come up with 
some proposed text.

As for Alessandro's propsal:

On 26 Apr 2021, at 5:54, Alessandro Vesely wrote:

> After Pete's examples, it is clear that there is a treasure trove of 
> older mail archives.  From such perspective, it doesn't sound so 
> misleading to call their format "obsolete".  It is an old format, no 
> longer in use.
>
> Specifying the scope of obs- syntax might seem to weaken the MUST 
> somewhat, for example like so:
>
> OLD:

To quote Keith and Ned from the other thread:

On 26 Apr 2021, at 20:23, Ned Freed wrote:

> On 26 Apr 2021, at 19:47, Keith Moore wrote:
>
>> MUAs need to be able to read
>> messages that are decades old because some people keep email around
>> forever, and messages can be archived for historical purposes.
>
> And they also select messages from archives and forward them.
> I've seen decades-old mesages resurrected and forwarded.
>
> The fact that this works is actually a tribute to the quality of the 
> design.
> Try doing something similar with decades old word processing files and
> see what happens.
>
> Like it or not, there's no way to prevent essentially any part of
> the email infrastructure from being exposed to old messages.

and:

>> John Levine:
>> I have, on multiple occasions within the past five years, been 
>> required
>> to read copious volumes of email dating from the 1980s, and testify 
>> to
>> its likely authenticity or lack thereof.   I can't recall the last
>> occasion I saw such a message in current email, but it's still 
>> necessary
>> to be able to read email dating from decades ago.   And this will 
>> become
>> even more necessary in the future as more and more of our historical
>> records exist only in digital form.
>
> It would be "today" for me, but that's because we use source routes
> internally. OTOH, I still see %-hacks pretty regularly. Bang paths
> seem to have died, thank goodness.
>
> Spaces before the colon still show up from time to time. We notice
> because they don't interact well with milter.

It goes beyond simply mail archives. It might be forwarding. It might be 
older implementations that are still around. It might be current 
implementations that do silly things, but that you still want to be able 
to accept mail from. The MUST means it is "actually required for 
interoperation". Keith and Ned make a pretty convincing argument that it 
is essential to implement the "obsolete" interpreter to interoperate, 
and I can't figure out what "valid reasons in particular circumstances 
to ignore" the requirement might be to even reduce it to a SHOULD.

pr

---

From: John C Klensin <[email-masked]>
Date: Tue, 27 Apr 2021 18:59:52 -0400

--On Tuesday, April 27, 2021 17:51 -0500 Pete Resnick
<[email-masked]> wrote:

>> Regarding the use of the word "obsolete" to describe once
>> common  syntax
>> that is less favored today: Consider "legacy" as an
>> alternative, and  try
>> to make it clear that readers need to continue to support
>> that syntax indefnintely.   I agree that "obsolete" conveys
>> the wrong  impression.
> 
> "accept-only", or probably other choices) would have probably
> been better than "obsolete". But I still haven't seen an
> answer to: Is there a proposal to change the terminology
> and/or the ABNF element names? Or is there a proposal to
> better explain in section 1.2.3 (as quoted by Alessandro
> below) that we chose a misleading or confusing name for these
> things, and we apologize for that, but we're sticking with it
> to avoid even more confusion? Or is there some other proposal
> for what you all want done in the document? This editor needs
> guidance to come up with some proposed text.

I'm nervous about changes either introducing errors or being
interpreted as more than than are.  Consequently my personal
preference is that we add an explanation to section 1.2.3 along
the lines you describe, probably add a cross-reference to that
explanation early in Section 4, and then move on.  

thanks,
  john

---

From: Ned Freed <[email-masked]>
Date: Tue, 27 Apr 2021 16:00:58 -0700 (PDT)

+1

I'll again repeat that we've managed something fairly remarkable here: A message
written in 1980 is almost certain to work with current agents. Do we really want
to compromise this?

Try reading an X.400-1984 message with an 1988-only version sometime.

				Ned