Subject: [apps-discuss] Adding email capabilities

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 5 Mar 2014 11:00:51 +0000

The discussion during the development of the RRVS draft about SMTP
extensions vs. header fields as mechanisms for adding capabilities has led
me to write this:

https://datatracker.ietf.org/doc/draft-kucherawy-email-caps/

Do people think this is a useful thing to pursue for publication as either
Informational or BCP?  I have a thorough review from Dave Crocker and one
from John Levine, but I'd love to hear from others about whether it's worth
doing and if so, what work needs doing.

Thanks,
-MSK, as just-another-apps-droid

---

From: Tony Finch <[email-masked]>
Date: Wed, 5 Mar 2014 13:07:15 +0000

I think it would be helpful for experts if there was a clear separation of the basic tutorial material from the enumeration of the problem points. I skipped to section 7 but I am not sure what I missed :-)

One thing that was lost with the introduction of ESMTP was a BCC-like mechanism for the various envelope extensions, i.e. a way within the 822 format to specify details of the envelope. This is useful when preparing the message, when saving drafts and saving copies of sent messages. It would be nice if future specs did not leave this out.

Tony.
--
f.anthony.n.finch  <[email-masked]>  http://dotat.at/

> On 5 Mar 2014, at 11:00, "Murray S. Kucherawy" <[email-masked]> wrote:
> 
> The discussion during the development of the RRVS draft about SMTP extensions vs. header fields as mechanisms for adding capabilities has led me to write this:
> 
> Do people think this is a useful thing to pursue for publication as either Informational or BCP?  I have a thorough review from Dave Crocker and one from John Levine, but I'd love to hear from others about whether it's worth doing and if so, what work needs doing.
> 
> Thanks,
> _______________________________________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 5 Mar 2014 13:16:27 +0000

On Wed, Mar 5, 2014 at 1:07 PM, Tony Finch <[email-masked]> wrote:

> I think it would be helpful for experts if there was a clear separation of
> the basic tutorial material from the enumeration of the problem points. I
> skipped to section 7 but I am not sure what I missed :-)
>

You're not the first person to say that, so I'll look at rearranging the
material.  Thanks for the suggestion.

>
> One thing that was lost with the introduction of ESMTP was a BCC-like
> mechanism for the various envelope extensions, i.e. a way within the 822
> format to specify details of the envelope. This is useful when preparing
> the message, when saving drafts and saving copies of sent messages. It
> would be nice if future specs did not leave this out.
>
>
I wasn't around when ESMTP was introduced, so I'm not clear on what you're
talking about here.  Can you either include or point me at an example?

-MSK

---

From: Dave Crocker <[email-masked]>
Date: Wed, 05 Mar 2014 13:15:27 +0000

On 3/5/2014 1:07 PM, Tony Finch wrote:
> I think it would be helpful for experts if there was a clear separation
> of the basic tutorial material from the enumeration of the problem
> points. I skipped to section 7 but I am not sure what I missed :-)

+1

> would be nice if future specs did not leave this out.

huh?  I don't understand.

I used bcc this morning, with thunderbird.

and it has nothing to do with [E]SMTP.

d/
-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: Franck Martin <[email-masked]>
Date: Wed, 5 Mar 2014 08:39:37 -0600 (CST)

I read quickly... 

I think you mention the recording of the envelope from in the message with return-path, but did not mention the record of the envelope to with Original-To: 

Other problems not mentioned that I will crudely name: 
SMTP as FTP: it may impossible to send large emails over unreliable links, as the session as to be restarted from the beginning, and large emails create anti-spam anti-virus scanning issues. 
SMTP as XMPP: long threads (netiquette to remove the old content) or fast exchange of emails: people expect the emails to be delivered within a few seconds (priority short email?) 
Authentication: if I trust you more, I may be nicer with your email and/or provide you more feedback 
Anti-spam is magic: message accepted for delivery sometimes disappear with no feedback to the sender, they are either dropped on the floor or delivered in spam folder. 
Encryption: does STARTTLS and DANE solve everything? 
Privacy: email that even the postmaster cannot read? 
Bounce and abuse processing: through mailbox or specific extensions, so different rules are applied, and user readable DSN are generated (the problem of STMP error one-liner (SMTP-error-tweeting))? 

I'm not saying these problems should be resolved, but they should be mentioned, as they are recurring (mis-)use cases, or expectations from email users. 

----- Original Message -----

From: "Murray S. Kucherawy" <[email-masked]> 
To: "IETF Apps Discuss" <[email-masked]> 
Sent: Wednesday, March 5, 2014 11:00:51 AM 
Subject: [apps-discuss] Adding email capabilities 

Thanks,

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 5 Mar 2014 14:49:12 +0000

On Wed, Mar 5, 2014 at 2:39 PM, Franck Martin <[email-masked]>wrote:

> I read quickly...
>
> I think you mention the recording of the envelope from in the message with
> return-path, but did not mention the record of the envelope to with
> Original-To:
>

There is no registered header field by that name.

Other problems not mentioned that I will crudely name:
> SMTP as FTP: it may impossible to send large emails over unreliable links,
> as the session as to be restarted from the beginning, and large emails
> create anti-spam anti-virus scanning issues.
> SMTP as XMPP: long threads (netiquette to remove the old content) or fast
> exchange of emails: people expect the emails to be delivered within a few
> seconds (priority short email?)
> Anti-spam is magic: message accepted for delivery sometimes disappear with
> no feedback to the sender, they are either dropped on the floor or
> delivered in spam folder.
> Bounce and abuse processing: through mailbox or specific extensions, so
> different rules are applied, and user readable DSN are generated (the
> problem of STMP error one-liner (SMTP-error-tweeting))?
>
> I'm not saying these problems should be resolved, but they should be
> mentioned, as they are recurring (mis-)use cases, or expectations from
> email users.
>
>
I'm confused.  This document's purpose is to educate the reader on cases
where using a header field instead of an SMTP extension (or vice versa) to
establish some capability is actually the wrong approach, and why, and how
we got into these bad habits.  How do the above things relate to that?

-MSK

---

From: Ned Freed <[email-masked]>
Date: Wed, 05 Mar 2014 08:36:45 -0800 (PST)

> I used bcc this morning, with thunderbird.

That's beside the point. Tony is talking about having a way to capture what the
submitting client put in the envelope as part of the message header. For things
like message copies put in a "Sent" folder.

> and it has nothing to do with [E]SMTP.

Perhaps not, but it has everything to do with SUBMIT.

				Ned

---

From: Franck Martin <[email-masked]>
Date: Wed, 5 Mar 2014 12:00:40 -0600 (CST)

----- Original Message -----

> To: "Franck Martin" <[email-masked]>
> Cc: "IETF Apps Discuss" <[email-masked]>
> Sent: Wednesday, March 5, 2014 2:49:12 PM
> Subject: Re: [apps-discuss] Adding email capabilities

> On Wed, Mar 5, 2014 at 2:39 PM, Franck Martin < [email-masked] >
> wrote:

> There is no registered header field by that name.

yes but it is often used, need for standardization? 

> > as
> > the session as to be restarted from the beginning, and large emails create
> > anti-spam anti-virus scanning issues.
> 
> > no
> > feedback to the sender, they are either dropped on the floor or delivered
> > in
> > spam folder.
> 

> > email
> > users.

> I'm confused. This document's purpose is to educate the reader on cases where
> using a header field instead of an SMTP extension (or vice versa) to

Me too ;) did I say I read quickly... 

I thought the purpose was to inventory the landscape of possible capabilities.

---

From: Tony Finch <[email-masked]>
Date: Wed, 5 Mar 2014 18:09:27 +0000

> talking about here.  Can you either include or point me at an example?

So, this way of looking at BCC is you put it in the header when preparing
the message, but on submission it is stripped from the header and used to
construct part of the envelope.

Look at RFC 3461 for example. It specifies some envelope parameters,
particularly NOTIFY and RET, which the user will want to specify when
composing the message, and record in drafts and saved copies. But there is
no standard way to do so as part of the message format, e.g. as stored on
an IMAP server.

See also RFC 2852, RFC 3885, RFC 4865.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Portland, Plymouth, North Biscay: Southerly or southwesterly 4, increasing 5
or 6. Moderate or rough. Occasional rain later. Good, occasionally poor later.

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 5 Mar 2014 18:14:17 +0000

> construct part of the envelope.

Ah, I see.

I was intending only to talk about "I have a capability to add to mail.
Should I do this in an extension or in a header field?"  I don't think
that's the same thing as what you're talking about here.  I take it you
mean this concept would be good to write down as well, and this would be a
good place to put it?

-MSK

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 5 Mar 2014 18:22:53 +0000

On Wed, Mar 5, 2014 at 6:00 PM, Franck Martin <[email-masked]>wrote:

> ------------------------------
>
>

Sure, if you're looking for a project.  :-)

> I thought the purpose was to inventory the landscape of possible
No, more meta than that: If you're going to add a capability, which
extension model is appropriate?

-MSK

---

From: Tony Finch <[email-masked]>
Date: Wed, 5 Mar 2014 18:33:57 +0000

> mean this concept would be good to write down as well, and this would be a

Possibly. I think BCC points to an area where the tension between header
vs envelope is fairly strong. In the past the IETF has kept
transport-related extensions strictly in the envelope without considering
what happens before message submission. I think this makes them more
awkward than necessary. Following the BCC model seems to be a good way to
resolve the tension.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Northwest Forties, Cromarty, Forth: Southerly 5 to 7, becoming variable 4
later. Moderate or rough. Occasional rain. Good, occasionally moderate.

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 5 Mar 2014 18:37:57 +0000

> awkward than necessary. Following the BCC model seems to be a good way to
Got it.  I'll take a look at this and the other cases you suggested and
figure out what's appropriate to include.  Bcc is a pretty obvious case,
but I'm not certain about the others yet.

Thanks!
-MSK

---

From: Dave Crocker <[email-masked]>
Date: Wed, 05 Mar 2014 18:44:00 +0000

> construct part of the envelope.

While, yes, some/many/most/all MUAs use some structured input fields 
that look like 822 fields -- and there certainly have been batch-ish 
packages that do/did it literally by reading a formatted file -- they 
aren't really reading an 822 object.

> See also RFC 2852, RFC 3885, RFC 4865.

What I'm taking from your notes is the thought that there perhaps should 
be some sort of coherent specification of the mapping of 821 envelope 
parameters onto 822 header fields -- obviously just citing existing 
specs where appropriate.

d/

---

From: Ned Freed <[email-masked]>
Date: Wed, 05 Mar 2014 08:48:20 -0800 (PST)

I don't think any of this is relevant to the matter at hand, so FWIW:

> SMTP as FTP: it may impossible to send large emails over unreliable links, as

We wrote RFC 1845 to solve this problem, and to a close first approximation
nobody cared enugh to deploy it.

I'll note that this issue is even worse with FTP itself, because most clients
don't retry automatically, which at least email does. And the checkpont/restart
facilities in FTP haven't been implemented all that often either.

Bottom line: This appears to be a problem nobody cares enough to want to
address.

> seconds (priority short email?)

I don't think this is a standards issue per se, but even so a while back IBM
did a whacking lot of research into how to display such matters in a client.
Once again, nobody cared to implement any of it.

> provide you more feedback

Address-book based whitelisting is widely available, widely used, and
reasonably effective. Not sure how "better" authentication would improve
things, for any value of "better" I can think of.

> Anti-spam is magic: message accepted for delivery sometimes disappear with no
> feedback to the sender, they are either dropped on the floor or delivered in
> spam folder.

Yep. I see no viable solutions to this problem.

> Privacy: email that even the postmaster cannot read?

I've already pointed out a few (but not all) of the serious issues with
widespread  end-to-end encryption on the perpass list. And we have not one, not
two, not three, but four fully specified end-to-end encryption schemes for
email, one of which (S/MIME) has significant client penetration.

And once again, to a first approxication, nobody uses any of them.

> different rules are applied, and user readable DSN are generated (the problem
> of STMP error one-liner (SMTP-error-tweeting))?

Same answer as above: We built the technology, specifically standardized
formats for DSNs and MDNs that are designed to be translated into
other languages, and once again nobody cared enough to implement the
client part.

> mentioned, as they are recurring (mis-)use cases, or expectations from email
> users.

If standardization counts as solving, we've already solved most of these
issues. The problem is getting the solutions deployed. And in most
cases the sticking point is clients, users, or both.

Of course you may argue that we solved them the wrong way. In which case
you are welcome to write up a detailed alternative solution, along with
supporting evidence as to why your solution will deploy whereas what's
currently standardized has not.

				Ned

---

From: Tony Finch <[email-masked]>
Date: Wed, 5 Mar 2014 19:31:20 +0000

Dave Crocker <[email-masked]> wrote:
>
> [MUAs] aren't really reading an 822 object.

They have a user interface for preparing an 822 object, plus whatever
envelope fields the user might want to set.

Note that IMAP does not have a way for storing envelope data for draft
messages, so current MUAs have to invent a non-standard way of
representing envelope-only fields in an 822 object.

Tony.

---

From: Dave Crocker <[email-masked]>
Date: Wed, 05 Mar 2014 19:40:15 +0000

> Perhaps not, but it has everything to do with SUBMIT.

I meant the latter part of that note to touch what I think is that 
issue: standardizing the storage of the full set of envelope parameters 
into the 822 hearder field.  Where a parameter already has an 822 home, 
then just cite it.

Is there any challenge to writing up a mappings spec for this?

d/

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 5 Mar 2014 19:44:43 +0000

On Wed, Mar 5, 2014 at 7:40 PM, Dave Crocker <[email-masked]> wrote:

> I meant the latter part of that note to touch what I think is that issue:
> standardizing the storage of the full set of envelope parameters into the
> 822 hearder field.  Where a parameter already has an 822 home, then just
> cite it.
>
...and should it be in this draft or a different one?  I think the latter.

-MSK

---

From: Dave Crocker <[email-masked]>
Date: Wed, 05 Mar 2014 19:44:47 +0000

> ...and should it be in this draft or a different one?  I think the latter.

separate doc.

d/

---

From: "MH Michael Hammer (5304)" <[email-masked]>
Date: Wed, 5 Mar 2014 19:47:43 +0000

> -----Original Message-----
> From: apps-discuss [mailto:[email-masked]] On Behalf Of
> Dave Crocker
> Sent: Wednesday, March 05, 2014 2:45 PM
> To: Murray S. Kucherawy; Dave Crocker
> Cc: Ned Freed; IETF Apps Discuss

+1 for separate doc.

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 5 Mar 2014 19:55:29 +0000

>> ...and should it be in this draft or a different one?  I think the latter.
Not it!

---

From: Lyndon Nerenberg <[email-masked]>
Date: Wed, 5 Mar 2014 14:32:18 -0800

> representing envelope-only fields in an 822 object.

MH has a long tradition of pushing this message construction meta-data into the headers of (draft) messages, too.  Somewhat coincidentally, the nmh developers are going through an almost parallel discussion of *822 header (ab)use in this context.

--lyndon

---

From: Dave Crocker <[email-masked]>
Date: Thu, 06 Mar 2014 08:01:43 +0000

> MH has a long tradition of pushing this message construction meta-data into the headers of (draft) messages, too.  Somewhat coincidentally, the nmh developers are going through an almost parallel d…

MH is still being worked on?  wow.

    http://bbiw.net/articles/rand-email.pdf

d/

---

From: Scott Brim <[email-masked]>
Date: Thu, 6 Mar 2014 09:44:29 +0000

Yes. Also the Emacs MH-E community is strong. It's a good mail UI.

On Thu, Mar 6, 2014 at 8:01 AM, Dave Crocker <[email-masked]> wrote:
>> MH has a long tradition of pushing this message construction meta-data
>> into the headers of (draft) messages, too.  Somewhat coincidentally, the nmh
>> developers are going through an almost parallel discussion of *822 header
>> (ab)use in this context.
>
>
>