Subject: [Emailcore] Status of Greylisting (i'd wish MessageID were part of SMTP prologue)

From: Steffen Nurpmeso <[email-masked]>
Date: Thu, 06 Jan 2022 21:29:00 +0100

Hello.

It is not only IPv6, it also cloud services.
And it is that ever changing TTL in times of DNS responsibility
indirection a.k.a. content delivery.  Worse, if i bypass my local
DNS cache and for example say "dig XXX @8.8.8.8" then for the same
host i do get the same IP, but the TTL is 159, 208, 134.
These rotations surely have an impact on RFC 6647, 5.,

   1.  Implement greylisting based on a tuple consisting of (IP address,
       RFC5321.MailFrom, and the first RFC5321.RcptTo).

that was not forseeable to this extent in 2012?
It seems greylisting now mask /8 for IPv4 and /64 for IPv6.
It seems to me when generating X different messages in the
delay/retry time frame Y, the message X+1 "unlocks" sent before.

As i am in the process of writing a greylisting policy server (for
postfix policy protocol) i wonder how future proof this can be,
without adding more identification tokens to early stages of the
SMTP protocol?
With the caveat that i (am) personally (not an administrator and)
sit in a small corner of the internet with only a *very* limited
view onto the overall SMTP traffic.  How useful is greylisting on
overall and/or on this scale today, and tomorrow?

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

---

From: John C Klensin <[email-masked]>
Date: Thu, 06 Jan 2022 15:58:43 -0500

--On Thursday, January 6, 2022 21:29 +0100 Steffen Nurpmeso
<[email-masked]> wrote:

> indirection a.k.a. content delivery.  Worse, if i bypass my
> local DNS cache and for example say "dig XXX @8.8.8.8" then
> for the same host i do get the same IP, but the TTL is 159,
> 208, 134. These rotations surely have an impact on RFC 6647,
> 5.,
> 
>    1.  Implement greylisting based on a tuple consisting of
> (IP address,        RFC5321.MailFrom, and the first
> RFC5321.RcptTo).
> 
> As i am in the process of writing a greylisting policy server
> (for postfix policy protocol) i wonder how future proof this
> can be, without adding more identification tokens to early
> stages of the SMTP protocol?
> With the caveat that i (am) personally (not an administrator
> and) sit in a small corner of the internet with only a *very*
> limited view onto the overall SMTP traffic.  How useful is
> greylisting on overall and/or on this scale today, and
> tomorrow?

Steffen,

Am I correct to suggest that this is a topic for either a
6647bis or for comments on the subject in the developing
EMAILCORE A/S that might update 6647bis -- neither of which is
obviously in scope with this WG at the moment-- and not
something you believe should be addressed in 5321bis?

thanks,
   john

---

From: "John Levine" <[email-masked]>
Date: 6 Jan 2022 17:06:59 -0500

>that was not forseeable to this extent in 2012?

Item 5 in section 5 says:

       To accommodate those senders that have clusters of outgoing mail
       servers, greylisting servers MAY track CIDR blocks of a size of
       its own choosing, such as /24, rather than the full IPv4 address.
       (Note, however, that this heuristic will not work for clusters
       having machines on different networks.)  A similar grouping
       capability MAY be established based on the domain name of the
       mail server if one can be determined.

Is this the problem you are encountering or something else?

In my experience, allowing matches within a /24 in IPv4 or a /64
in IPv6 largely addresses this problem.

>How useful is greylisting on

My small system recently greylisted 21238 sending hosts of which 12745
retried and 8770 didn't. Once a host retries, it isn't greylisted
again unless it hasn't sent any mail for over a month. Spot checks
show that the 40% of hosts that don't retry are almost all spambots,
so it's useful, but not so much we'd change the protocol.

Apropos jck's question, while we might consider revising 6447, this has nothing
to do with 5321 so replies are directed to the ietf-smtp list.

R's,
John

---

From: Steffen Nurpmeso <[email-masked]>
Date: Fri, 07 Jan 2022 00:22:54 +0100

John C Klensin wrote in
 <001C904B1B67B2632182BF8D@PSB>:
 |--On Thursday, January 6, 2022 21:29 +0100 Steffen Nurpmeso
 |<[email-masked]> wrote:
 ..
 |> 208, 134. These rotations surely have an impact on RFC 6647,
 |> 5.,
 ...
 |Steffen,
 |
 |Am I correct to suggest that this is a topic for either a
 |6647bis or for comments on the subject in the developing
 |EMAILCORE A/S that might update 6647bis -- neither of which is
 |obviously in scope with this WG at the moment-- and not
 |something you believe should be addressed in 5321bis?

I well heared the respect from everyone that you managed to
publish draft 8 with all the formal bells and whistles that IETF
imposes before new year's eve, i am too much living on a sideshow
to have even considered to say "Congratulations" or "Thank you"
(though especially the latter would really be due).
(It was just that idea to "throw a problem into a forum".)

--steffen
|

---

From: Dave Warren <[email-masked]>
Date: Fri, 7 Jan 2022 13:45:23 -0700

On 2022-01-06 15:06, John Levine wrote:
> It appears that Steffen Nurpmeso<[email-masked]>  said:

In my experience, SPF is really useful too as you can treat multiple 
delivery attempts that achieve a SPF:PASS to be identical, such that I 
don't have to care whether the sender uses a cluster across multiple IP 
Addresses.

This would seem to be a handy way to identify a "similar grouping 
capability" for senders that use SPF without manual effort.

---

From: Hector Santos <[email-masked]>
Date: Thu, 13 Jan 2022 10:48:25 -0500

> something you believe should be addressed in 5321bis?

 From a technical standpoint, RFC5321bis should take into account 
basic implementation considerations used in the "pseudo-standard" 
greylist protocol that are at least 19+ years old showing two 
essential SMTP ideas:

- Timely delivery with minimal delays,
- Meaningful structured retry responses to advance intelligent SMTP 
clients.

The draft SMTP GreyList extension addresses these two basic SMTP 
advancement ideas:

      https://tools.ietf.org/id/draft-santos-smtpgrey-02.html

    GREYLIST is a SMTP extension to formalize the widely supported
    Greylisting mail filtering method and to help support SMTP rejected
    transports by following a new formal structured 4yz server temporary
    rejection response by including a "retry=time-delay" tag string which
    SMTP clients can use to optimize the rescheduling of the mail
    delivery attempts.  With adoption, network overhead reduction in
    wasteful mail delivery attempts will be realized.

Perhaps updating RFC5321bis examples showing real responses that exist 
in the market that include include "Retry Hints" as outlined in the 
draft.

There is running code that use the GreyList SMTP Extension retry hints.

Note: The informational RFC6647 was being drafted at the time the SMTP 
GreyList extension draft was being written.  I took it as a rebuttal 
against my greylisting work and in fact, it provided an opinion 
(without any stats to back it up) that "Retry Hints" presented a 
security risk to DoS attacks.  There was no evidence of this then, nor 
is there any evidence of this today. But the damage was done for any 
further IETF work for the proposed SMTP extension to help formalized 
Greylisting. All responsible systems, regardless of GreyListing, is 
already handling DoS attacks in some secured fashion.

Whether the improved RFC5321bis "SMTP Handbook" should codify 
important modern SMTP transport implementation ideas used in practice, 
I leave to others to decide.

I do agree, 100%, RFC6647bis should remove its guidance against 
structured Greylist reply text.

-- 
Hector Santos,
https://winserver.com/public/spamstats.wct

---

From: Steffen Nurpmeso <[email-masked]>
Date: Sat, 15 Jan 2022 15:17:47 +0100

Hello.

Sorry for the late reply, i wanted to read the draft first.
(The thread continued on ietf-smtp@ due to request.) 

Hector Santos wrote in
 <[email-masked]>:
 |On 1/6/2022 3:58 PM, John C Klensin wrote:
 |> Am I correct to suggest that this is a topic for either a
 |> 6647bis or for comments on the subject in the developing
 |> EMAILCORE A/S that might update 6647bis -- neither of which is
 |> obviously in scope with this WG at the moment-- and not
 |> something you believe should be addressed in 5321bis?
 |
 | From a technical standpoint, RFC5321bis should take into account 
 |basic implementation considerations used in the "pseudo-standard" 
 |greylist protocol that are at least 19+ years old showing two 
 |essential SMTP ideas:
 |
 |- Timely delivery with minimal delays,
 |- Meaningful structured retry responses to advance intelligent SMTP 
 |clients.
 |
 |The draft SMTP GreyList extension addresses these two basic SMTP 
 |advancement ideas:
 |
 |      https://tools.ietf.org/id/draft-santos-smtpgrey-02.html

I think the idea of a retry=[MIN:]SEC member in the deferral
response is interesting, even though i do not know how MTAs manage
their queues .. but likely they could adapt to this easily.
It could be used to avoid useless traffic, which in today's
greylisting practice as i experience it happens, even multiple
times, before the message gets through.

 |    GREYLIST is a SMTP extension to formalize the widely supported
 |    Greylisting mail filtering method and to help support SMTP rejected
 |    transports by following a new formal structured 4yz server temporary
 |    rejection response by including a "retry=time-delay" tag string which
 |    SMTP clients can use to optimize the rescheduling of the mail
 |    delivery attempts.  With adoption, network overhead reduction in
 |    wasteful mail delivery attempts will be realized.
 |
 |Perhaps updating RFC5321bis examples showing real responses that exist 
 |in the market that include include "Retry Hints" as outlined in the 
 |draft.
 |
 |There is running code that use the GreyList SMTP Extension retry hints.
 |
 |Note: The informational RFC6647 was being drafted at the time the SMTP 
 |GreyList extension draft was being written.  I took it as a rebuttal 
 |against my greylisting work and in fact, it provided an opinion 
 |(without any stats to back it up) that "Retry Hints" presented a 
 |security risk to DoS attacks.  There was no evidence of this then, nor 
 |is there any evidence of this today. But the damage was done for any 
 |further IETF work for the proposed SMTP extension to help formalized 
 |Greylisting. All responsible systems, regardless of GreyListing, is 
 |already handling DoS attacks in some secured fashion.
 |
 |Whether the improved RFC5321bis "SMTP Handbook" should codify 
 |important modern SMTP transport implementation ideas used in practice, 
 |I leave to others to decide.
 |
 |I do agree, 100%, RFC6647bis should remove its guidance against 
 |structured Greylist reply text.

--steffen
|

---

From: John C Klensin <[email-masked]>
Date: Sat, 15 Jan 2022 15:10:53 -0500

Steffen (and Hector),

I don't mean to pick on you and hope this note is not taken as
doing that, but...

Unless one of you, or someone else, wants to make a strong case
for discussing Greylisting in 5321bis (or 5322bis) I hope there
are no further discussions on this list.

Procedurally, discussing it in either of those documents would
require either:

 * Publishing one or both of them at Proposed Standard,
	countering the main justification for this WG.

or

 * Pausing the WG's work while an updated, preferably
	Standards Track, version of RFC 6647 is written, works
	its way through the system and then, IIR no less that
	six months after publication, an implementation report
	is generated that justifies moving it to full standard.
	Or new specifications, documenting new ideas, are moved
	through the system on the same basis.

I would not feel so strongly about this except that, with a
grand total of zero comments on the changes and questions raised
in draft-ietf-emailcore-rfc5321bis-08 since it was posted two
weeks ago, I am feeling very concerned that, except for
discussions on new ideas like these or arguments about document
organization, the WG is running out of energy already and, more
important, that there may not be sufficient energy to do the
comprehensive reviews that Internet Standard Versions of these
documents should have.

I hope I'm wrong, but would feel a lot better if the WG's
mailing list delivered a posting that started "I have carefully
read all the way through draft-ietf-emailcore-rfc5321bis-08
and..."

Just my opinion of course but I am at the point where I almost
wish the co-chairs would treat starting this type of thread or
responding to it as disruptive.

   john

---

From: "John Levine" <[email-masked]>
Date: 20 Jan 2022 23:32:34 -0500

It appears that John C Klensin  <[email-masked]> said:
>weeks ago, ...

FWIW, I've been going through the draft and so far haven't found anything
that I'd want to change.

In this case (albeit not in general I realize) silence actually does mean consent.

R's,
John

---

From: John C Klensin <[email-masked]>
Date: Fri, 21 Jan 2022 00:18:25 -0500

--On Thursday, January 20, 2022 23:32 -0500 John Levine
<[email-masked]> wrote:

>> grand total of zero comments on the changes and questions
>> raised in draft-ietf-emailcore-rfc5321bis-08 since it was
>> posted two weeks ago, ...
> 
> FWIW, I've been going through the draft and so far haven't
> found anything that I'd want to change.
> 

John,

The effort and calibration are very much appreciated.  And I
appreciate your speaking up.

Talk with you (and I hope everyone else) tomorrow.  

For everyone else, and with a bit under twelve hours to go
before the interim, I'd strongly encourage people to take a
careful look at Section 2.3.5 and Section 5.  I've moved some
material between the two sections and done some rewriting but
concluded I need feedback from the WG before either stopping or
going further, remembering that such rewrites risk introducing
errors.  Those two sections are on the agenda, but doubt that it
is possible to understand enough about what is going on by
looking at them in real time during the meeting.

thanks,
   john