Subject: [Emailcore] Ticket #5, rfc5321bis Section 4.5.3.1.10, and G.5

From: John C Klensin <[email-masked]>
Date: Tue, 09 Mar 2021 16:34:40 -0500

Hi.
This is about the "Too Many Recipients Code" transition
described in earlier versions of SMTP and the question of
whether the transition text should be removed from 5321bis.

The more I think about this, the more I wonder about whether 821
was right in the first place.  Assuming the current spec,
including that minimum of 100 recipients, was actually followed
(see below) and the sender comes in with its 101st RCPT command.
If the server comes back with 452, which is shown in the list in
Section 4.2.2 as "insufficient storage" and with the implication
of all 4yz codes of "try again".  If the sender tries again with
the same recipient list and gets the same response -- especially
if the server uses the "insufficient system storage" message in
the spec without a supplemental code -- we are getting close to
Einstein's definition of insanity.   The presumably correct
response from the sender would be to cut the number of
recipients and try again, but the 452 response gives no hint of
that, nor any way to distinguish what should have been "45x
Requested action not taken: too many recipients, will not accept
more than NN" from "452 Requested action not taken: insufficient
system storage".

And, before someone else points it out, telling the sender the
maximum both violates the general principle of 5321 and its
predecessors that the response text does not carry substantive
information except for a small number of specific exceptions --
a principle that is violated by documenting both "too many
recipients" and "insufficient storage" for 452 -- and may make
things more efficient for an attacker.  

This is further complicated if we consider lowering the
requirement for the minimum number of recipients a server is
expected to accept (now 100) as suggested on list and (vaguely)
covered in Ticket #14, Section G.7.8.

I'm not sure if this is a serious suggestion but, if we retain
the 452 code, do we need should we have an new code (borrowing
an analogy from interpretations of "666"), e.g.,

 566 You tried that more times, with the same parameters
	and address list, than I will tolerate, please go
	straight to an appropriate hot place and stay there

Probably not a serious suggestion, at least in that form, but
you get the point.

And, of course, if we made any change at all in this area, would
anyone pay any attention to it?

    john

---

From: "Peter J. Holzer" <[email-masked]>
Date: Wed, 10 Mar 2021 09:06:45 +0100

On 2021-03-09 16:34:40 -0500, John C Klensin wrote:
> Einstein's definition of insanity.

That's not how senders handle this case in my (admittedly limited)
experience.

> The presumably correct response from the sender would be to cut the
> number of recipients and try again,

I'm not sure that 'cut' is the right word to describe current (and I
think correct) behaviour.

For illustrative purposes I'll assume that the receiving MTA has a limit
of 3 instead of 100. The sender tries to send a message to 8 recipients.

[EHLO, etc.]
MAIL FROM:<[email-masked]>
250 OK
RCPT TO:<[email-masked]>
250 OK
RCPT TO:<[email-masked]>
250 OK
RCPT TO:<[email-masked]>
250 OK
RCPT TO:<[email-masked]>
452 OK                          # The limit is now exceeded, the
                                # receiver starts sending 452 replies
RCPT TO:<[email-masked]>
452 OK
RCPT TO:<[email-masked]>
452 OK
RCPT TO:<[email-masked]>
452 OK
RCPT TO:<[email-masked]>
452 OK
DATA                            # The sender knows that the message will
                                # be delivered to 3 recpipients, so it
                                # sends the content
354 Start mail input; end with <CRLF>.<CRLF>
Blah blah blah...
.
250 OK
QUIT
221

At this point, the message has been delivered to 3 recipients, but not
yet to the remaining 5. The sender doesn't really know why those got
temporary errors, nor does it care. It just requeues the message with
those 5 recpients and retries some time later:

[EHLO, etc.]
MAIL FROM:<[email-masked]>
250 OK
RCPT TO:<[email-masked]>
250 OK
RCPT TO:<[email-masked]>
250 OK
RCPT TO:<[email-masked]>
250 OK
RCPT TO:<[email-masked]>
452 OK
RCPT TO:<[email-masked]>
452 OK
DATA
354 Start mail input; end with <CRLF>.<CRLF>
Blah blah blah...
.
250 OK
QUIT
221

Two recpients still left, requeue, wait and try again:

[EHLO, etc.]
MAIL FROM:<[email-masked]>
250 OK
RCPT TO:<[email-masked]>
250 OK
RCPT TO:<[email-masked]>
250 OK
DATA
354 Start mail input; end with <CRLF>.<CRLF>
Blah blah blah...
.
250 OK
QUIT
221

Done.

> 	straight to an appropriate hot place and stay there

As I understand the RFC (and obviously implementors of SMTP clients as
well), each response to a RCPT command pertains only to that recipient.
So after getting a mixture of 2xx and 4xx responses a sender would not
retry with the same address list. Some addresses have already been
successful, so they aren't tried again. (Similary, if there were also
some 5xx responses, those addresses won't be tried again, either.)

Maybe that should be stated explicitely somewhere (if it isn't already),
but that's how I always understood the RFC.

        hp

-- 
   _  | Peter J. Holzer    | Story must make more sense than reality.
|_|_) |                    |
| |   | [email-masked]         |    -- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |       challenge!"

---

From: Ned Freed <[email-masked]>
Date: Wed, 10 Mar 2021 00:17:05 -0800 (PST)

> > Section 4.2.2 as "insufficient storage"

Yes, but there is no other code in that list that seems appropriate
for "Too many recipients".

> > and with the implication of all 4yz codes of "try again".

The meaning of a 4YZ repsonse to RCPT TO is more subtle. (I don't
put much stock in interpreting the second and third digits.) It could
be any of:

(1) A problem resolving this specific recipient,
(2) An administrative limit has been reached, or
(3) An actual storage limit has been hit.

> > If the sender tries again with

Not necessarily. Transient problems do go away. LDAP servers come back up.

> Done.

My experience agrees with yours - this is how things work almost all of the
time. There is, however, another possibility:

[EHLO, etc.]
MAIL FROM:<[email-masked]>
250 OK
RCPT TO:<[email-masked]>
250 OK
RCPT TO:<[email-masked]>
250 OK
RCPT TO:<[email-masked]>
250 OK
RCPT TO:<[email-masked]>
452 Too many recipients         # The limit is now exceeded, the
                                # receiver starts sending 452 replies
RCPT TO:<[email-masked]>
452 Too many recipients
RCPT TO:<[email-masked]>
452 Too many recipients
RCPT TO:<[email-masked]>
452 Too many recipients
RCPT TO:<[email-masked]>
452 Too many recipients
DATA                            # The sender assumes the 3 good recipients
                                # will work, but...
452 Recipient limit exceeded, try again with fewer recipients

True storage limit errors are rare given the capacity of modern systems; the
more likely cause here is a badly implemented administrative limit. And in such
a case repeating the same transaction is unlikely to ever work. So what we do
in this case is requeue as multiple messages so that we won't get to the point
of receiving 452s responses to RCPT TO the next time.

Now, I freely admit we're going to a lot of trouble to deal with what is
arguably a broken server, but (a) It turned out to be easy to implement and (b)
There was a point were a fairly popular server was doing this often enough to
make it worthwhile.

I also note that the message "Too many recipients" is ambiguous as to cause.

> well), each response to a RCPT command pertains only to that recipient.

Exactly.

> but that's how I always understood the RFC.

Me too. I also think that removing permitted code/response pairs is a really
bad idea. Of course it's wrong for clients to be overly restrictive in the code
combinations they allow, but given that there are widely used implementations
that do this (I'm looking at you, JavaMail) removing things is IMO a really bad
idea.

				Ned

---

From: Alessandro Vesely <[email-masked]>
Date: Wed, 10 Mar 2021 19:29:13 +0100

> (3) An actual storage limit has been hit.

4yz can also be used to split the recipient list into a set that has DATA 
filtering enabled and one that has it disabled.  A DATA filter can either 
accept or reject a message based on its content.  In case it rejects, 
recipients who have disabled such filtering have to receive the message in a 
separate transaction.  I'm not sure this case is part of (1) above.

Let me also note that by checking the first digit only, 552 and 550 are treated 
in the same way.

>> think correct) behaviour.

'Split'?

> of receiving 452s responses to RCPT TO the next time.

Here the last digit matters.  It's obviously different from:

451 Requested action aborted: DATA filter had a hiccup.

>> well), each response to a RCPT command pertains only to that recipient.

Perhaps, that can be made explicit in Section 3.3 Mail Transactions.

> Me too.

+1

> I also think that removing permitted code/response pairs is a really bad
> idea. Of course it's wrong for clients to be overly restrictive in the code 
> combinations they allow, but given that there are widely used
> implementations that do this (I'm looking at you, JavaMail) removing things
> is IMO a really bad idea.

If not removed, RCPT/552 deserves at least being discouraged.  For example, by 
noting that a 552 reply code to RCPT likely has the same effect as a 550.

Frankly, I don't understand the discussion in Section 4.5.3.1.10.  In 
particular (i) the two intents:

A    prohibit messages with more than a site-specified number of recipients

and

B    merely limit the number of recipients in a given mail transaction

seem to only differ by the fact that A applies systematically to all 
transactions while B might look transient, which hardly sounds like the correct 
meaning.  Then, (ii) Ned's implementation of storing separate queue entries may 
overcome what he calls "a badly implemented administrative limit", but is based 
on 4xy negative replies.  Therefore, the option to:

     simply return the 503 after DATA
     without returning any previous negative response

won't work in any case.

jm2c
Ale
--

---

From: Ned Freed <[email-masked]>
Date: Wed, 10 Mar 2021 11:08:09 -0800 (PST)

> separate transaction.  I'm not sure this case is part of (1) above.

It isn't from the server's point of view, but it's the same as far as the 
client is concerned. However, in the case you raised, AFAICT there's no
ambiguity to how to handle it: A 5YZ response return the message for the
recipients you accepted, and retry the other set later.

> Let me also note that by checking the first digit only, 552 and 550 are
> treated in the same way.

If you're referring to the workaround for the RFC 821 bug, aren't we close to
consensus that it is no longer necessary?

> 451 Requested action aborted: DATA filter had a hiccup.

Obviously different to the server, but not, I think, in terms of what the client
should do. Given a situation where some of the recipients were tempfailed and
the DATA command also tempfailed, it does no harm to limit the number of
recipients the next time around no matter what the cause. Arguably an even
better approach would be to split into two groups based on the specific
responses, but that's asking quite a lot of the client.

> noting that a 552 reply code to RCPT likely has the same effect as a 550.

Aside from avoiding 552 because of the bug in RFC 821, I'm afraid I don't see
rationale for this. We can and do check for persistent over quota conditions at
RCPT TO time, and if that's the case return a 5YZ error. If the RFC 821 bug is
no longer a concern, the only reason for not returning 552 now is to avoid the
workaround.

> won't work in any case.

It will work, for some value of the word "work", because we treat 5YZs as the
permanent failures they are and don't attempt to second-guess the server. We
only engage in recipient list limiting when given a subsequent 4YZ.

That said, given the number of clients that use a separate transaction for each
recipient, the idea that you can limit the number of reicpients of a message in
this way is sufficiently dumb that I see no reason to mention it as useful
thing to do.

				Ned

---

From: John C Klensin <[email-masked]>
Date: Wed, 10 Mar 2021 16:11:40 -0500

--On Wednesday, March 10, 2021 09:06 +0100 "Peter J. Holzer"
<[email-masked]> wrote:

>> The more I think about this, the more I wonder about whether
>> 821 was right in the first place.  Assuming the current spec,
>> including that minimum of 100 recipients, was actually
>> followed (see below) and the sender comes in with its 101st
>> RCPT command. If the server comes back with 452, which is
>> shown in the list in Section 4.2.2 as "insufficient storage"
>> and with the implication of all 4yz codes of "try again".  If
>> the sender tries again with the same recipient list and gets
>> the same response -- especially if the server uses the
>> "insufficient system storage" message in the spec without a
>> supplemental code -- we are getting close to Einstein's
>> definition of insanity.
> 
> That's not how senders handle this case in my (admittedly
> limited) experience.
> 
>> The presumably correct response from the sender would be to
>> cut the number of recipients and try again,
> 
> I'm not sure that 'cut' is the right word to describe current
> (and I think correct) behaviour.
> 
> For illustrative purposes I'll assume that the receiving MTA
> has a limit of 3 instead of 100. The sender tries to send a
> message to 8 recipients.
> 
> [EHLO, etc.]
> 452 OK                          # The limit is now exceeded,
> the                                 # receiver starts sending
> 452 replies RCPT TO:<[email-masked]>
> 452 OK
> RCPT TO:<[email-masked]>
> 452 OK
> RCPT TO:<[email-masked]>
> 452 OK
> RCPT TO:<[email-masked]>
> 452 OK
> DATA                            # The sender knows that the
> message will                                 # be delivered to
> 3 recpipients, so it                                 # sends
> the content
> At this point, the message has been delivered to 3 recipients,
> but not yet to the remaining 5. The sender doesn't really know
> why those got temporary errors, nor does it care. It just
> requeues the message with those 5 recpients and retries some
> time later:
> 
> [EHLO, etc.]

This makes much more sense than the scenario I was thinking of,
but runs into another issue or two. Your scenario probably
explains why we wanted to make the change from 552 to 452 for
this case [1].  Although neither is precisely aligned with what
the spec says about the difference between 4yz and 5yz codes,
the former seems much more likely to be interpreted by an unwary
implementation in the way your scenario suggests -- discard the
additional recipients, handle the other recipients and message
as normal, and then resend with the recipients that got the 452
code, rather than interpreting a 5yz code as "just stop right
now and send QUIT".

Based on the discussion during the meeting a few hours ago (I
only saw your note just now) that indicated that almost no one
is following the 5321 advice and transitioning to 452, it looks
as if probably the advice about switching should be removed, the
more general advice about the code should be switched to SHOULD
or MAY, and, especially if the latter, the whole paragraph
removed and any comments on the subject moved to the A/S.

It is probably time for me to take another nap before I have an
additional attack of stupid.

 thanks,
    john

[1] I have normally made notes in the source files to remind me
of why we made a particular change but didn't in this case.
Murphy's Law, I suppose.

---

From: John C Klensin <[email-masked]>
Date: Wed, 10 Mar 2021 16:47:25 -0500

--On Wednesday, March 10, 2021 00:17 -0800 Ned Freed
<[email-masked]> wrote:

>> > The more I think about this, the more I wonder about
>> > whether 821 was right in the first place.  Assuming the
>> > current spec, including that minimum of 100 recipients, was
>> > actually followed (see below) and the sender comes in with
>> > its 101st RCPT command. If the server comes back with 452,
>> > which is shown in the list in Section 4.2.2 as
>> > "insufficient storage"
> 
> Yes, but there is no other code in that list that seems
> appropriate for "Too many recipients".
> 
> The meaning of a 4YZ repsonse to RCPT TO is more subtle. (I
> don't put much stock in interpreting the second and third
> digits.) It could be any of:
> 
>> > the same recipient list and gets the same response --
>> > especially if the server uses the "insufficient system
>> > storage" message in the spec without a supplemental code --
>> > we are getting close to Einstein's definition of insanity.
> 
>> the
>>                                 # receiver starts sending 452
>>                                 # replies
>> RCPT TO:<[email-masked]>
>> 452 OK
>> RCPT TO:<[email-masked]>
>> 452 OK
>> RCPT TO:<[email-masked]>
>> 452 OK
>> RCPT TO:<[email-masked]>
>> 452 OK
>> message will
>>                                 # be delivered to 3
>>                                 # recpipients, so it sends
>>                                 # the content
>> At this point, the message has been delivered to 3
>> recipients, but not yet to the remaining 5. The sender
>> doesn't really know why those got temporary errors, nor does
>> it care. It just requeues the message with those 5 recpients
>> and retries some time later:
> 
>> [EHLO, etc.]
> My experience agrees with yours - this is how things work
> almost all of the time. There is, however, another possibility:
> 
> [EHLO, etc.]
> 452 Too many recipients         # The limit is now exceeded,
> DATA                            # The sender assumes the 3
> good recipients                                 # will work,
> but...
> True storage limit errors are rare given the capacity of
> modern systems; the more likely cause here is a badly
> implemented administrative limit. And in such a case repeating
> the same transaction is unlikely to ever work. So what we do
> in this case is requeue as multiple messages so that we won't
> get to the point of receiving 452s responses to RCPT TO the
> next time.
> 
> Now, I freely admit we're going to a lot of trouble to deal
> with what is arguably a broken server, but (a) It turned out
> to be easy to implement and (b) There was a point were a
> fairly popular server was doing this often enough to make it
> worthwhile.
> 
>> > I'm not sure if this is a serious suggestion but, if we
>> > retain the 452 code, do we need should we have an new code
>> > (borrowing an analogy from interpretations of "666"), e.g.,
>> > 
>> As I understand the RFC (and obviously implementors of SMTP
>> clients as well), each response to a RCPT command pertains
>> only to that recipient.
> 
> Exactly.
> 
>> So after getting a mixture of 2xx and 4xx responses a sender
>> would not retry with the same address list. Some addresses
>> have already been successful, so they aren't tried again.
>> (Similary, if there were also some 5xx responses, those
>> addresses won't be tried again, either.)
> 
>> Maybe that should be stated explicitely somewhere (if it
>> isn't already), but that's how I always understood the RFC.
> 
> Me too. I also think that removing permitted code/response
> pairs is a really bad idea. Of course it's wrong for clients
> to be overly restrictive in the code combinations they allow,
> but given that there are widely used implementations that do
> this (I'm looking at you, JavaMail) removing things is IMO a
> really bad idea.

That suggests to me that:

(1) We don't remove either code (easy, because they can be used
for other purposes, such as running out of of storage, although
I agree that is unlikely with today's systems.

(2) We note for both the 552 and 452 codes that they can be used
for the "too many addresses" case as well as the more general
storage-related one.

(3) Probably (but subject to further discussion) we leave the
first paragraph of 4.5.3.1.10 alone because all it really says
is that a 552 code in response to RCTP is reasonably treated as
temporary rather that the "just stop and send QUIT or RSET"
behavior normally expected in response to a 5yz code.

(4) We reconsider the MUST in the third paragraph of 4.5.3.1.10.
because it is being widely ignored and figure out if that
paragraph needs to be rewritten.

(5) We put text in the A/S exploring this issue and other issues
where an implementation exceeds either the 5321 minimum limits
or smaller limits imposed by the implementation or configuration
as "operationally necessary.

--and-- 

we consider whether there should be a separate code or codes for
"implementation limit exceeded" and what they should be.  Then I
think that we drop the idea because it is too late and because
the observation that, in the 20 years since 2821 was published,
very few systems have adopted the instructions of 2821 and 5321
to send 452 rather than 552 suggests that such instructions are
hopeless

Does that seem like a way forward?

   john

---

From: Viktor Dukhovni <[email-masked]>
Date: Thu, 11 Mar 2021 02:44:40 -0200

> On Mar 10, 2021, at 7:47 PM, John C Klensin <[email-masked]> wrote:
> 

Correct.

> for the "too many addresses" case as well as the more general

Perhaps so, but 552 is very unwise if the only reason for rejecting
the recipient is a ceiling on the number of recipients per envelope
that is short of 100, or a desire to force different "classes" of
recipients into separate envelopes, by selectively rejecting (that
is tempfailing) some.

> behavior normally expected in response to a 5yz code.

Absolutely, 100%, unconditionally, ... NO WAY.  The text about
treating 5XX as though it were a 4XX reply MUST go.  There is
no mechanism for the client to divine the exact reason why the
server choice 552 and not some other 5XX code.

The bottom line is that a recipient that got a 5XX code WILL be
bounced, and a recipient got a 4XX code will be deferred.

When at least one recipient is accepted, but some are deferred
(4XX) an MTA doing SMTP relaying (unlike an MUA doing SUBMIT)
MUST NOT immediately defer the accepted recipients. MTAs are
expected to be able to deliver a portion of the message envelope
in one transaction and the remainder in one or more additional
transactions.

Indeed when a destination has more than one MX host, and the
first transaction defers some recipients, Postfix will by
default immediately open a connection to a second MX (by
default tries at most two per envelope, before deferring
for later) and try the remaining 4XX recipients there.

> paragraph needs to be rewritten.

There's not much to reconsider, no clients treat a 5XX as 4XX,
and none are likely to start now.

-- 
	Viktor.

---

From: Alessandro Vesely <[email-masked]>
Date: Fri, 12 Mar 2021 11:16:52 +0100

> no longer a concern, the only reason for not returning 552 now is to avoid the

I assume you mean 452.

> permanent failures they are and don't attempt to second-guess the server.

There is no point in giving 503 rather than 550.  Number of recipient doesn't 
play any role.  I'd strike that sentence, as Viktor suggests.

> We only engage in recipient list limiting when given a subsequent 4YZ.

Yes.

> recipient, the idea that you can limit the number of recipients of a message in

It still has to be programmed for.

Best
Ale
--

---

From: Alessandro Vesely <[email-masked]>
Date: Fri, 12 Mar 2021 11:20:05 +0100

> server choice 552 and not some other 5XX code.

+1, 552 can be be understood as 550.

> in one transaction and the remainder in one or more additional

Hm... a MUA shouldn't get a 452 on submit, should it?

I'm not clear what it means "immediately defer", verb and adverb sound incongruous.

> for later) and try the remaining 4XX recipients there.

Indeed, some mention this as one of the reasons to have multiple MXes.

It would make sense to enable this feature, quick retry, also for mail sites 
that only sport a single MX at a single address.  Since there are multiple 
codes, we could convene that some 4yz want a real delay while others just need 
a separate transaction.  I understand such distinction cannot be made in 
5321bis, but maybe the A/S?

Best
Ale
--