Subject: [Emailcore] Ticket #11: G.7.4. Possible clarification about mail transactions and transaction state

From: Alexey Melnikov <[email-masked]>
Date: Mon, 5 Jul 2021 18:24:21 +0100

draft-ietf-emailcore-rfc5321bis-02 currently says:

3.3.  Mail Transactions

    There are three steps to SMTP mail transactions.  The transaction
    starts with a MAIL command that gives the sender identification.  (In
    general, the MAIL command may be sent only when no mail transaction
    is in progress; see Section 4.1.4.)  A series of one or more RCPT
    commands follows, giving the receiver information.  Then, a DATA
    command initiates transfer of the mail data and is terminated by the
    "end of mail" data indicator, which also confirms the transaction.

    Mail transactions are also terminated by the RSET command
    (Section 4.1.1.5), the sending of an EHLO command (Section 3.2), or
    the sending of a QUIT command (Section 3.8) which terminates both any
    active mail transaction and the SMTP connection.

4.1.4.  Order of Commands

   [...]

    The MAIL command begins a mail transaction.  Once started, a mail
    transaction consists of a transaction beginning command, one or more
    RCPT commands, and a DATA command, in that order.  A mail transaction
    may be aborted by the RSET, a new EHLO, or the QUIT command. There
    may be zero or more transactions in a session.  MAIL MUST NOT be sent
    if a mail transaction is already open, i.e., it should be sent only
    if no mail transaction had been started in the session, or if the
    previous one successfully concluded with a successful DATA command,
    or if the previous one was aborted, e.g., with a RSET or new EHLO.

Ticket #11 suggests that 3.3 can be improved by being more specific 
about where mail transactions begin and end. And section 4.1.4 has a 
note "comment about changing this convoluted discussion to talk about 
'mail transaction' above."

Looking at these 2 sections, I observe that 3.3 talks about how a 
transaction starts, but can be slightly improved by clarifying how it 
terminates. To that end, I propose changing the last sentence of the 1st 
paragraph of 3.3 to read:

OLD:

    Then, a DATA
    command initiates transfer of the mail data and is terminated by the
    "end of mail" data indicator, which also confirms the transaction.

NEW:

    Then, a DATA
    command initiates transfer of the mail data and is terminated by the
    "end of mail" data indicator, which also confirms (and terminates) 
the transaction.

Basically this clarifies that "end of mail" data indicator always 
terminates the transaction. (Paragraph 2 of 3.3 is already clear that 
RSET/EHLO/QUIT also terminate the transaction.

I think the text in 4.1.4 is pretty clear, but it does contain some 
duplication in comparison to 3.3. We can either leave it as is or 
possibly shorten it to avoid duplication.

Here is a strawman proposal to do the latter:

    The MAIL command begins a mail transaction.  Once started, a mail
    transaction consists of a transaction beginning command, one or more
    RCPT commands, and a DATA command, in that order.  A mail transaction
    may be aborted by the RSET, a new EHLO, or the QUIT command. There
    may be zero or more transactions in a session.  MAIL MUST NOT be sent
    if a mail transaction is already open, i.e., it should be sent only
    if no mail transaction had been started in the session, or if the
    previous one successfully concluded, or if the previous one was aborted.

    See Section 3.3. for more details.

Any opinions about the above?

Best Regards,

Alexey

---

From: Jeremy Harris <[email-masked]>
Date: Mon, 5 Jul 2021 19:15:30 +0100

> Any opinions about the above?

Shouldn't the transaction be open until the _response_ to any
of those client-end actions?

-- 
Cheers,
   Jeremy

---

From: Viktor Dukhovni <[email-masked]>
Date: Tue, 6 Jul 2021 17:43:33 -0400

> of those client-end actions?

I was wondering the same thing, but on reflection I think it is clear
that nothing in the server reply can extend the scope of the
transaction.  The transaction may succeed, fail, be terminated, or lead
to an error state (a rejected RSET or EHLO presumably precludes
continuing with subsequent transactions).

So termination here refers to *commands* that terminate a transaction.
The fact that the client typically awaits a response is implicit.
(Clients might not wait for a QUIT response, and might drop a connection
after a successful RSET if despite anticipating more traffic no further
messages are queued for the peer).

-- 
    Viktor.

---

From: John C Klensin <[email-masked]>
Date: Tue, 06 Jul 2021 18:45:12 -0400

> Ticket #11 suggests that 3.3 can be improved by being more
> specific about where mail transactions begin and end. And
> section 4.1.4 has a note "comment about changing this
> convoluted discussion to talk about 'mail transaction' above."
> 
> Looking at these 2 sections, I observe that 3.3 talks about
> how a transaction starts, but can be slightly improved by
> clarifying how it terminates. To that end, I propose changing
> the last sentence of the 1st paragraph of 3.3 to read:
> 
> OLD:
> 
>     Then, a DATA
>     "end of mail" data indicator, which also confirms (and
> terminates) the transaction.
> 
> Basically this clarifies that "end of mail" data indicator
> always terminates the transaction. (Paragraph 2 of 3.3 is
> already clear that RSET/EHLO/QUIT also terminate the
> transaction.

This seems right to me.  It probably makes things a bit better;
clearly it does not make them any worse.

> I think the text in 4.1.4 is pretty clear, but it does contain
> some duplication in comparison to 3.3. We can either leave it
> as is or possibly shorten it to avoid duplication.
> 

> The MAIL command begins a mail transaction.  Once started, a
> mail transaction consists of a transaction beginning command,
> one or more RCPT commands, and a DATA command, in that
> order.  A mail transaction may be aborted by the RSET, a new
> EHLO, or the QUIT command. There may be zero or more
> transactions in a session.  MAIL MUST NOT be sent if a mail
> transaction is already open, i.e., it should be sent only if
> no mail transaction had been started in the session, or if the

> Any opinions about the above?

Alternative if you (and others) want to trim and get the
redundancies out.  This also improves on what is now an odd
relationship between "MUST NOT" and "should"

New:

	The MAIL command begins a mail transaction.  Once
	started, a mail transaction consists of a transaction
	beginning command, one or more RCPT commands, and a DATA
	command, in that order.  Mail transactions are defined
	and terminated as described in Section 3.3. MAIL MUST
	NOT be sent if a mail transaction is already open, i.e.,
	it is permitted only if no mail transaction had been
	started in the session, or if the previous one
	successfully concluded, or was aborted.

I am ok with either but do think the cross-reference is useful.

best,
   john

---

From: Dave Crocker <[email-masked]>
Date: Tue, 6 Jul 2021 15:50:37 -0700

On 7/6/2021 3:45 PM, John C Klensin wrote:
> 	beginning command,

As written, the above text is confusing and I'm not sure what it means.

If the MAIL command begins a mail transaction, then the next thing -- 
what is currently described as "a transaction beginning command" -- 
takes place after the mail transaction has begun.

d/

-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: John C Klensin <[email-masked]>
Date: Tue, 06 Jul 2021 19:19:38 -0400

--On Tuesday, July 6, 2021 17:43 -0400 Viktor Dukhovni
<[email-masked]> wrote:

>> >  beginning command, one or more    RCPT commands, and a
>> >  DATA command, in that order.  A mail transaction    may
>> >  be aborted by the RSET, a new EHLO, or the QUIT command.
>> >  There    may be zero or more transactions in a
>> >  session.  MAIL MUST NOT be sent    if a mail
>> >  transaction is already open, i.e., it should be sent only
>> >     if no mail transaction had been started in the
>> >  session, or if the    previous one successfully
>> >  concluded, or if the previous one was aborted.
>> > 
> I was wondering the same thing, but on reflection I think it
> is clear that nothing in the server reply can extend the scope
> of the transaction.  The transaction may succeed, fail, be
> terminated, or lead to an error state (a rejected RSET or EHLO
> presumably precludes continuing with subsequent transactions).

Exactly.  I suppose we could clarify further by saying that the
server MUST respond to any of those commands/conditions with a
2yz code but Section 4.3 basically already says that.
 
> So termination here refers to *commands* that terminate a
> transaction. The fact that the client typically awaits a
> response is implicit. (Clients might not wait for a QUIT
> response, and might drop a connection after a successful RSET
> if despite anticipating more traffic no further messages are
> queued for the peer).

Actually the current text requires (or is intended to require)
that clients to wait except under unusual circumstances and then
(deliberately) hand-waves about the definition of "unusual".  I
think that just strengthens your point.

So, anyone want to argue that clarification is needed for this
and, if it is, that it is important enough to justify added
text?  If so, proposed text would be appreciated because your
friendly editor thinks it is fairly clear.

    john

---

From: Viktor Dukhovni <[email-masked]>
Date: Tue, 6 Jul 2021 19:55:11 -0400

> 2yz code but Section 4.3 basically already says that.

Well, not always 2yz, since ".", "RSET" or "EHLO" could fail or be
refused.  IIRC it is "QUIT" for which "2yz" is the only expected
outcome.

> think that just strengthens your point.

FWIW, this is another one of those things where practice diverges from
theory.  Clients *DO* disconnect without waiting for a "QUIT" response,
and this is in practice well justified.  Clients also abruptly drop
connections after a successful "RSET" without sending "QUIT", when e.g.
a cached connection times out without new messages to send.

Neither behaviour will change in, e.g., Postfix no matter what the
specification says.  The server must be prepared to handle lost
connections, e.g. because of transport or network layer issues, and the
client may from time to time elect to unilaterally close the transport.

> friendly editor thinks it is fairly clear.

I'm fine with the text re connection termination as-is.

-- 
    Viktor.

---

From: John C Klensin <[email-masked]>
Date: Tue, 06 Jul 2021 20:12:16 -0400

--On Tuesday, July 6, 2021 15:50 -0700 Dave Crocker
<[email-masked]> wrote:

> If the MAIL command begins a mail transaction, then the next
> thing -- what is currently described as "a transaction
> beginning command" -- takes place after the mail transaction
> has begun.

As indicated in various notes in the rfc5321bis draft, I've
never been really happy with the convolutions there.  They
arose, IIR, from concern that someone might someday introduce an
extension that would provide an alternative to the MAIL command
(e.g., a HCTS (Here Comes he Stuff) command) and/or as a nod to
SOML or SAML.   Any of those presumably would start a mail
transaction although naming them and giving more details than
are already present would be out of the scope of
2821/5321/5321bis.

In particular, while Section 2 of RFC 821 discusses mail
transaction initiation only in terms of the MAIL command (just
as 5321 does in different words), Section 3.4 (which defined
SEND, SOML and  SAML) says "used in the mail transaction instead
of the MAIL command".  Since we have deprecated them, that does
not mean much other than this is a very old problem and that, in
principle, nothing would prevent an extension that puts them
back in.  On the other hand, note that Appendix F.6 of RFC 5321/
5321bis allows servers to support them and, if one does so, it
MUST advertise them in the EHLO response.

My conclusion is (1) Despite your confusion and my distaste for
the text, the observation that there have apparently been no
complaints in the last decade or two might tell us something.
(2) We better go very carefully if we decide to start recasting
those definitions.

That said, it occurs to me that the text could probably be
improved by replacing

Old:
	beginning command,...

New (alternative 1):
	The MAIL command begins a mail transaction.  Once a
	mail transaction is thus initiated, it then consists
	of...

  ( or s/thus initiated/begun/ )
	
New (alternative 2):
	A mail transaction begins with a MAIL command and then
	consists of ...

Neither of those is particularly friendly to future extensions
that might introduce new transaction-starting commands, but,
given that there have been, IIR, no serious proposals for such
commands (much less adoption and deployment) since 1993, maybe
it is sensible to ignore that problem and pass the burden of
sorting things out onto anyone who proposes such a thing in the
future.

Do you feel better about either of those alternatives than about
the existing text?  Other suggestions?

thanks,
   john

---

From: Dave Crocker <[email-masked]>
Date: Tue, 6 Jul 2021 18:09:10 -0700

> (2) We better go very carefully if we decide to start recasting

If text makes no sense, then it makes no sense.  The idea that it is in 
any way acceptable to retain text that makes no sense... makes no sense.

The absence of prior complaints, for something like this, means very little.

d/

---

From: John C Klensin <[email-masked]>
Date: Wed, 07 Jul 2021 01:55:19 -0400

Dave,

Some observations (and you may want to ignore the first one)... 

--On Tuesday, July 6, 2021 18:09 -0700 Dave Crocker
<[email-masked]> wrote:

>> My conclusion is (1) Despite your confusion and my distaste
>> for the text, the observation that there have apparently been
>> no complaints in the last decade or two might tell us
>> something. (2) We better go very carefully if we decide to
>> start recasting those definitions.
> 
> If text makes no sense, then it makes no sense.  The idea that
> it is in any way acceptable to retain text that makes no
> sense... makes no sense.
> 

Sure.  Except that the text in question was carefully examined
by a WG (a WG in which, IIR, you participated), went through WG
Last Call and IETF Last Call and then through an IESG that did a
good  deal of nit-picking and then the RFC Editor.  That creates
at least a small presumption that many readers were able to
deduce what the text was intended to mean, not that it "makes no
sense".  Certainly you are entitled to your opinion about the
latter, but that history (including the absence of prior
complaints), does not suggest anything nearly so absolute as a
community conclusion that the text makes no sense.  

Could it be more clear?  I think so, have said so, and suggested
alternate text.  Should it have been caught while YAM was active
or in the subsequent years?  Probably, but, at least IMO,
debating that or trying to figure out not is not particularly
helpful at this stage... except, again, to suggest that there is
not universal agreement that the current wording "makes no
sense".

Finally, I did suggest some options for alternate wording.  Do
you have comments about either of them, or do you just find it
more useful to make assertions about what does or does not make
any sense?

thanks,
   john

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 7 Jul 2021 10:59:47 +0100

Hi John,

On 06/07/2021 23:45, John C Klensin wrote:
>> started, a mail

I like your version better, but I am happy with either.

Alexey (as a participant).

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 7 Jul 2021 11:01:15 +0100

Hi Dave,

> takes place after the mail transaction has begun.

You are right, this can be made clearer. I think it is probably a 
leftover from times when there were more than one way to initiate the 
transaction.

Best Regards,

Alexey

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 7 Jul 2021 11:16:03 +0100

Hi John,

> 	A mail transaction begins with a MAIL command and then
I think this is an improvment and I prefer your 2nd alternative.
> sorting things out onto anyone who proposes such a thing in the

Fair enough.

I briefly considered pointing out that extensions already change how 
transactions are terminated (e.g. BDAT/BURL commands), so maybe this is 
a separate ticket. But not addressing it is fine.

---

From: John C Klensin <[email-masked]>
Date: Wed, 07 Jul 2021 11:44:47 -0400

Alexey,

--On Wednesday, July 7, 2021 11:16 +0100 Alexey Melnikov
<[email-masked]> wrote:

>...
>...

> I think this is an improvment and I prefer your 2nd

I'm going to drop that into the working draft unless someone
objects soon, noting that after having upgraded "confusing and
I'm not sure what it means" to "makes no sense", Dave has not
responded to the suggested alternatives.  

>> sorting things out onto anyone who proposes such a thing in
>> the future.
> 
> Fair enough.
> 
> I briefly considered pointing out that extensions already
> change how transactions are terminated (e.g. BDAT/BURL
> commands), so maybe this is a separate ticket. But not
> addressing it is fine.

Additional suggestion to avoid additional tickets and more
confusion:  How about we add, two sentences to explicitly deal
with those cases,  

Old:
	A mail transaction may be aborted by the RSET, a new
	EHLO, or the QUIT command.

New:
	A mail transaction may be aborted by the RSET, a new
	EHLO, or the QUIT command. SMTP extensions (see Section
	2.2) may create substitutions for any of those commands
	or may define new ones.  Their documentation is expected
	to be very precise about if, or how, they affect the
	mail transaction state.

That paragraph would probably benefit from additional
wordsmithing and suggestions are welcome.   In particular, I
think (at least right now) that text is more useful here than
added to Section 2.2 although a different alternative would be
to add text to Section 2.2.3  explicitly calling out changes to
the transaction and/or session definitions.  Or we could do
both.  I'm not convinced this needs a ticket and a long
discussion, but I'll leave that up to you and Todd.

    best, 
      john

---

From: Ned Freed <[email-masked]>
Date: Wed, 07 Jul 2021 18:50:28 -0700 (PDT)

> responded to the suggested alternatives.

+1 to putting it in the draft.

> wordsmithing and suggestions are welcome.

How about something like:

 	A mail transaction may be aborted by the RSET, a new
 	EHLO, or the QUIT command. SMTP extensions (see Section
 	2.2) may create additional commands that abort or
        end the transaction.

	More generally, any new command MUST clearly document any
  	effect it has on the transaction state.

> In particular, I

it seems like we should be able to nail this down fairly easily.

				Ned

---

From: John C Klensin <[email-masked]>
Date: Thu, 08 Jul 2021 00:00:27 -0400

>   	effect it has on the transaction state.

I like it.  Clearer and more succinct than my fussy-headed first
cut. Copied into the working draft.  

I hope to get that draft up Friday or Saturday at the latest
rather than waiting for the cutoff, so, if anyone objects,
please do so soon if you want your comments reflected in the
next I-D.
 
>> to add text to Section 2.2.3  explicitly calling out changes
>> to the transaction and/or session definitions.  Or we could do

Concur.  My current working editor's hypothesis is is to do this
and not mess with 2.2.3 other than being sure there are adequate
cross references.  Other ideas welcoem.

thanks,
   john

---

From: Alexey Melnikov <[email-masked]>
Date: Thu, 8 Jul 2021 13:41:56 +0100

> cut. Copied into the working draft.
+1.
> cross references.  Other ideas welcoem.
Sounds like a good plan to me.

---

From: Alessandro Vesely <[email-masked]>
Date: Thu, 8 Jul 2021 16:41:12 +0200

>>>       2.2) may create additional commands that abort or

Well, since that mentions aborting and ending the transaction, why don't they 
also mention starting it?

>> cut. Copied into the working draft.

I guess I'm too late...

Best
Ale
--

---

From: Dave Crocker <[email-masked]>
Date: Thu, 8 Jul 2021 08:09:33 -0700

>>>>       2.2) may create additional commands that abort or

Hmm.  Nevermind the usual hassle about using 'may' in a non-normative 
way, it's simple English meaning introduces ambiguity, since it implies 
that the transaction might /not/ be aborted.

More direct and clearer:

      Receipt of an RSET, a QUIT, or a new EHLO terminates the current
      mail transaction.

The second sentence is a classic non-specification that purports to be a 
specification.  It's essentially giving permission for a hypothetical, 
as if the permission were necessary and useful.  It is neither.

Future specifications will do whatever they do.  Whether they are 
'legal' will depend upon what the community decides at the time.

d/

---

From: Dave Crocker <[email-masked]>
Date: Thu, 8 Jul 2021 08:41:11 -0700

On 7/8/2021 8:15 AM, Ned Freed wrote:
> I have no problem with this, although I'm skeptical as to the claim of 
> lack of
> clarity of the original.
> 
> Strongly disagree. The point of saying this is to tell implementors that 
> this
> list of commands that may terminate a transaction is not complete, and they
> need to plan for that eventuality when they code.

1. Saying something akin to your language is more direct and clear than 
the existing language.  If it is really felt to be that important to 
tell implementers that things can change with future specification, then:

     The list of actions that can terminate a mail transaction is
     extensible and subject to expansion with future specifications.

> 
> Decades of experience with implementors getting this sort of thing wrong 
> and
> then saying "nothing in the specification said this was possible" says this
> *is* useful information.

2.  Do we have any evidence, over 50 years of protocol specification, 
that such guidance has useful effect on the bulk of implementers?  I 
suspect not, but would be delighted to hear otherwise.  Yes they get it 
wrong, but that's different from saying we have any hope of getting them 
to do better with bits of language like this.  As for their using the 
excuse the cite, they will always find an excuse.  Trying to write 
specifications that anticipate every hypothetical implementation design 
error is a lost cause.

> True but irrelevant. The purpose of the text is not to give permission, 
> it's to
> remind people of a possibility. Of course at this point we're lucky if
> implementors even bother to get a copy of the specification, let alone 
> read it,
> but that's not an excuse to fail to make the point.

I suspect it would then be far more effective to show a state diagram, 
with extensibility made explicit.

d/

---

From: "Alexey Melnikov" <[email-masked]>
Date: Thu, 08 Jul 2021 16:54:58 +0100

>       Receipt of an RSET, a QUIT, or a new EHLO terminates the current

Yes, this is better.

> as if the permission were necessary and useful.  It is neither.

Actually, I disagree, as we have existing extensions that can end transactions (BDAT/BURL). So pointing out that this should be considered by readers of the base spec is a good thing.

Best Regards,
Alexey

---

From: Dave Crocker <[email-masked]>
Date: Thu, 8 Jul 2021 09:01:20 -0700

...
> Actually, I disagree, as we have existing extensions that can end transactions (BDAT/BURL). So pointing out that this should be considered by readers of the base spec is a good thing.

Ahh, yeah, I missed that completely.  But I still think the proposed 
sentence does not serve the intended goal sufficiently.

Perhaps:

     The list of actions that terminate a mail transactions is 
extensible; it currently includes some SMTP extensions and might be 
expanded by later extensions.

d/

---

From: Ned Freed <[email-masked]>
Date: Thu, 08 Jul 2021 08:51:00 -0700 (PDT)

> that such guidance has useful effect on the bulk of implementers?

MIME has a lot of such notes. Feedback on those has been uniformly positive. To
the extent there has been negative feedback, it's been to say we needed more of
them.

I myself find such notes quite useful - they have saved me from making
invalid simplifying assumptions on more than one occasion.

> I

Yes, but there's a difference between trying to list every eventuality and
trying to hit the really high points. This is a really high point.

> with extensibility made explicit.

As luck would have it, I recently stumbled across an SMTP state diagram that I
drew many years ago when this idea occurred to me. It's a lot more complicated
than you'd suspect, so much so it's ability to inform is limited. This is true
even when you limit things to the transaction part of the protocol.

And representing extensibility in such a diagram means either adding
additional boxes or adding text to existing boxes. Neither works very well IME.

				Ned

---

From: "Alexey Melnikov" <[email-masked]>
Date: Thu, 08 Jul 2021 17:17:47 +0100

> expanded by later extensions.

That would work as well.

---

From: Alessandro Vesely <[email-masked]>
Date: Thu, 8 Jul 2021 18:26:30 +0200

>      extensible and subject to expansion with future specifications.

That's clear and clean!  Maybe s/can terminate/can operate on/ so as to include 
beginning and middle stuff.

> Yes they get it wrong, but that's different from saying we have any hope of
> getting them to do better with bits of language like this.

That text is also useful to inspire future SMTP extension designers.

Best
Ale
--

---

From: Ned Freed <[email-masked]>
Date: Thu, 08 Jul 2021 08:15:48 -0700 (PDT)

>       Receipt of an RSET, a QUIT, or a new EHLO terminates the current

I have no problem with this, although I'm skeptical as to the claim of lack of
clarity of the original.

> as if the permission were necessary and useful.  It is neither.

Strongly disagree. The point of saying this is to tell implementors that this
list of commands that may terminate a transaction is not complete, and they
need to plan for that eventuality when they code.

Decades of experience with implementors getting this sort of thing wrong and
then saying "nothing in the specification said this was possible" says this
*is* useful information.

> 'legal' will depend upon what the community decides at the time.

True but irrelevant. The purpose of the text is not to give permission, it's to
remind people of a possibility. Of course at this point we're lucky if
implementors even bother to get a copy of the specification, let alone read it,
but that's not an excuse to fail to make the point.

				Ned

---

From: Ned Freed <[email-masked]>
Date: Thu, 08 Jul 2021 08:25:56 -0700 (PDT)

> also mention starting it?

Very true. I would suggest changing the text to say "start, abort, or end" and
moving the sentence to a separate paragraph.

				Ned

> --
> Emailcore mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/emailcore

---

From: John C Klensin <[email-masked]>
Date: Sat, 10 Jul 2021 01:11:32 -0400

--On Thursday, July 8, 2021 16:41 +0200 Alessandro Vesely
<[email-masked]> wrote:

>>>>       A mail transaction may be aborted by the RSET, a
>>>> new       EHLO, or the QUIT command. SMTP extensions
>>>> (see Section       2.2) may create additional commands
>>>> that abort or          end the transaction.
> 
> 
> Well, since that mentions aborting and ending the transaction,
> why don't they also mention starting it?

Nit successfully picked; text changed.

   john

---

From: John C Klensin <[email-masked]>
Date: Sat, 10 Jul 2021 03:46:07 -0400

Ok.

I have tried to capture this thread to the best of my ability.
Once the -03 version of the I-D appears, please review and make
additional suggestions as needed, subject to two observations
from the editor:

* As concerns the text about what future specifications can and
cannot do, the intent is to make a specific warning about a very
specific type of change that other sections of the document call
out (the SMTP extension mechanism) and to impose an additional
requirement on any extensions that change the transaction model.
IMO (see below), at least some of the suggestions to further
generalize that text would move it toward the meaningless
handwaving to which other suggestions objected.

* Both in the hope of helping the WG move forward and to
preserve my sanity as editor (I value the latter, opinions of
others may differ), proposed substitutions of new text for text
I believe is already clear and that do not suggest changes with
substantive, rather than editorial, effect are likely to be
ignored unless explicitly endorsed by at least one person other
than the author of the suggestion.  Those who make suggestions
but disagree with my treatment of them are free to (and probably
should) ask the co-chairs to either assign tickets, make
consensus calls, or both.

FWIW, my criterion for "substantive" is fairly low.  For
example, I believe noticing that the text had talked about
aborting and ending but not starting was a good catch.  On the
other hand, I'm not convinced that trying to find-tune text to
make it extra clear that aborting a session also ends it would
be a good use of people's time.

The latter indication about how I intend to proceed is, of
course, not limited to this particular topic.

    john (as editor)

---

From: John C Klensin <[email-masked]>
Date: Sat, 10 Jul 2021 03:49:19 -0400

--On Tuesday, July 6, 2021 19:55 -0400 Viktor Dukhovni
<[email-masked]> wrote:

>> > is clear that nothing in the server reply can extend the
>> > scope of the transaction.  The transaction may succeed,
>> > fail, be terminated, or lead to an error state (a rejected
>> > RSET or EHLO presumably precludes continuing with
>> > subsequent transactions).
>> 
>> Exactly.  I suppose we could clarify further by saying that
>> the server MUST respond to any of those commands/conditions
>> with a 2yz code but Section 4.3 basically already says that.
> 
> Well, not always 2yz, since ".", "RSET" or "EHLO" could fail
> or be refused.  IIRC it is "QUIT" for which "2yz" is the only
> expected outcome.

I don't understand how RSET could fail unless the server went
outside the spec and considered it equivalent to QUIT, something
that would leave things in an odd state wrt the session/
Connection.  I suppose a server that paid no attention to the
conventional wisdom or our advice about a second EHLO without an
intervening RSET could respond to a second EHLO with "503 Bad
sequence of commands".

However, please note that, if we are going to start looking for
those edge cases (I'm somewhat agnostic about whether we
should), we are going to have to open and reexamine the
command-reply sequences of Section 4.3.2.  If you are convinced
that is desirable (or necessary) I recommend opening a ticket.

>> > response, and might drop a connection after a successful
>> > RSET if despite anticipating more traffic no further
>> that clients to wait except under unusual circumstances and
>> then (deliberately) hand-waves about the definition of
>> "unusual".  I think that just strengthens your point.
> 
> FWIW, this is another one of those things where practice
> diverges from theory.  Clients *DO* disconnect without waiting
> for a "QUIT" response, and this is in practice well justified.
> Clients also abruptly drop connections after a successful
> "RSET" without sending "QUIT", when e.g. a cached connection
> times out without new messages to send.
> 
> Neither behaviour will change in, e.g., Postfix no matter what
> the specification says.  The server must be prepared to handle
> lost connections, e.g. because of transport or network layer
> issues, and the client may from time to time elect to
> unilaterally close the transport.

But I think the text already says that that server must be
prepared for such things.  As to the client,  Whether we should
change, e.g., the description of QUIT from the current text that
strongly suggests that the client should wait for a reply to
QUIT.  Perhaps that stems from 821 being written in the time of
a much more fragile network, but, while I imagine that few of us
on this list regularly use fragile networks, they do still exist
around the world.

> I'm fine with the text re connection termination as-is.

Thanks.  I think it has been improved somewhat since you posted
your note, but the improvements are, IMO, relatively small.

best,
   john

---

From: Alessandro Vesely <[email-masked]>
Date: Mon, 12 Jul 2021 20:01:40 +0200

On Sat 10/Jul/2021 09:49:19 +0200 John C Klensin wrote:
> --On Tuesday, July 6, 2021 19:55 -0400 Viktor Dukhovni wrote:
>> 

The onset of bots and abusive behavior had the side effect of making networks 
appear more fragile than they formally are.  Several mail sites use fail2ban or 
equivalent software, which can drop network connection abruptly, based on log 
files or received messages, possibly sending a tcp-reset to the local server 
and leaving the client timeout on its own.

Of course, that is outside SMTP.  No need to mention it in the spec, but let's 
not assume that improved hardware implies better reliability.

Best
Ale
--

---

From: Viktor Dukhovni <[email-masked]>
Date: Mon, 12 Jul 2021 15:20:58 -0400

> On 10 Jul 2021, at 3:49 am, John C Klensin <[email-masked]> wrote:
> 

Non-success responses to RSET are not uncommon, when the server
is unwilling to continue the client session beyond the current
transaction.  Clients need to be prepared to receive a 4XX
(perhaps 421) response to RSET and should then disconnect, b/c
as you observe the session is no longer in a recoverable state.

> that is desirable (or necessary) I recommend opening a ticket.

Sadly, I'm not very good at following through on opening tickets,
so if that's what it takes, the issue might get lost...

> on this list regularly use fragile networks, they do still exist

The issue is that the response to QUIT is of no interest to the
client, it closes the connection either way.  Keeping an open
connection to the server waiting for a useless reply can be
expensive if it ties up connection slots that could be used
to deliver mail (in MTAs with carefully managed concurrency
limits).

So sending QUIT and closing the channel is the best choice
in practice.  PIPELINING helps here, because in that case
we have:

	C> ... msg body ...
	C> .<CRLF>QUIT<CRLF>
	S> 250 OK<CRLF>221 Goodbye<CRLF>
	C> ... closes connection ...

So whether or not the client actually reads the "221 Goodbye" is
not observable, it waits for the "250 Ok" after the combined "."+QUIT
and then disconnects.  It is only when PIPELINING is not offered
that disconnect before QUIT response is potentially visible to the
server.

-- 
	Viktor.

---

From: Michael Peddemors <[email-masked]>
Date: Mon, 12 Jul 2021 12:38:24 -0700

> that disconnect before QUIT response is potentially visible to the

For the record, if you do NOT wait for a response to the QUIT, you will 
probably be more likely to be marked as a spammer..

Not that there is a response to QUIT that represents, "don't hang up 
yet, I haven't accepted responsibility for the message(s) you sent in 
this transaction yet.. ", but waiting a 'suitable' time interval before 
terminating the connection is a reasonable ask. The server MAY want to 
provide you with a more detailed response in it's 250 ok

-- 
"Catch the Magic of Linux..."
------------------------------------------------------------------------
Michael Peddemors, President/CEO LinuxMagic Inc.
Visit us at http://www.linuxmagic.com @linuxmagic
A Wizard IT Company - For More Info http://www.wizard.ca
"LinuxMagic" a Registered TradeMark of Wizard Tower TechnoServices Ltd.
------------------------------------------------------------------------
604-682-0300 Beautiful British Columbia, Canada

This email and any electronic data contained are confidential and intended
solely for the use of the individual or entity to which they are addressed.
Please note that any views or opinions presented in this email are solely
those of the author and are not intended to represent those of the company.

---

From: Viktor Dukhovni <[email-masked]>
Date: Mon, 12 Jul 2021 15:55:06 -0400

> On 12 Jul 2021, at 3:38 pm, Michael Peddemors <[email-masked]> wrote:
> 

Postfix has well over 20% of the MTA market share, and even higher by
message volume.  So declaring all Postfix MTAs to be spammers is generally
not a winning choice.  The risk is exceedingly low.

> Not that there is a response to QUIT that represents, "don't hang up yet,
> I haven't accepted responsibility for the message(s) you sent in this
> transaction yet.. ", but waiting a 'suitable' time interval before terminating
> the connection is a reasonable ask. The server MAY want to provide you with
> a more detailed response in it's 250 ok

This is not going to happen.  Postfix MTAs default to hanging up without
waiting for a QUIT response.  This is only observable in the absence of
PIPELINING, and even then typically not seen because the server closes
the connection after handing off the QUIT response to the local TCP
stack, and never sees the RST if the client happened to already have
also closed its TCP read-side.

What might make close before QUIT response more prominent is use of TLS.
The server's TLS stack might shutdown the write side upon receipt of the
client's TLS close notify, returning an error for attempts to return
anything other than a reciprocal close-notify in the reverse direction.

Whether the early disconnect is observed or not depends on various
timing and implementation details, but if observed from one Postfix
system, it will typically be observed from most.

No apologies for this, the implementation is well motivated, and will
not change for mere adherence to the letter of the specification.

-- 
	Viktor.