Subject: RE: Activation points and callout modes

From: "Martin Stecher" <[email-masked]>
Date: Wed, 2 Feb 2005 17:33:44 +0100

Hi Hilarie,

> 
> I don't see how D (email body modification) "falls back" to A (command
> modification), but as long as it can be done, I don't care what letter
> gets assigned to it.

On Jan 20 you replied to an earlier message saying:
  > Yes, I think that operating on the DATA portion fits into the
  > architecture without the need for special consideration.

That is what I wanted to describe here again. The email message body is
the data coming after the DATA command. It can be seen also as command
modification. No special consideration necessary -> D "falls back" to A.

> 
> I'd not thought that the OPES processor would be maintaining queues, [...]

And I assumed that most MTAs maintain something like queues in order to
store the messages they receive before they forward them (or maybe to store
if they cannot forward immediatly).
But I may be wrong.

Has anybody a clear view on how many MTAs (percent of real world deployment)
can only work in a non-storing proxy mode (only accepting a message if they
already successfully forwarded to the next MTA in a parallel SMTP dialog)?
And how many MTAs do always store a message before forwarding?
And how many try a direct forward and store only if that does not work out?

Regards
Martin

---

From: Tony Finch <[email-masked]>
Date: Wed, 2 Feb 2005 17:07:48 +0000

> if they cannot forward immediatly).

It's practially required by the SMTP architecture. See RFC 2821 section
4.5.4.1 Sending Strategy.

> already successfully forwarded to the next MTA in a parallel SMTP dialog)?

I doubt any proper MTAs work in this way, but some firewalls implement
SMTP proxies (pejoratively known as "SMTP fux-up mode" because of the
gratuitous incompetence of many implementations). These are probably in
the target area for OPES.

> And how many try a direct forward and store only if that does not work out?

I don't think there are any. That would imply attempting to do a
complete delivery of the message to its next hop in the time between
CRLF.CRLF and its response, which is against the following text from RFCs
1123 and 2821:

   To avoid receiving duplicate messages as the result of timeouts, a
   receiver-SMTP MUST seek to minimize the time required to respond to
   the final <CRLF>.<CRLF> end of data indicator.  See RFC 1047 [28] for
   a discussion of this problem.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
LUNDY FASTNET IRISH SEA: NORTHWEST BACKING WEST, 3 OR 4, OCCASIONALLY 5 AT
FIRST. MAINLY FAIR. GOOD.

---

From: "The Purple Streak, Hilarie Orman" <[email-masked]>
Date: Wed, 2 Feb 2005 10:29:08 -0700

Thanks for reminding me what "D falls back to A" means.

For the SMTP queues, it had seemed an interesting possibility to have
the callout server maintain the queues.  That's a radical change to
the normal SMTP implementation, admittedly, and it might not make
sense from the perspective of I/O efficiency.  But it would facilitate
a very lean SMTP implementation.

Hilarie

---

From: "The Purple Streak, Hilarie Orman" <[email-masked]>
Date: Wed, 2 Feb 2005 10:36:58 -0700

An implementation can certainly cache a message during simultaneous
receive/send and defer storage until it is clear that the delivery
won't complete?

---

From: Tony Finch <[email-masked]>
Date: Wed, 2 Feb 2005 17:56:42 +0000

> a very lean SMTP implementation.

Existing callout protocols try to avoid any queueing in the policy server.

Postfix has support for a pre-reception SMTP proxy, effectively mode 1 but
implemented like mode 4.
http://www.postfix.org/SMTPD_PROXY_README.html

It also has a simpler policy protocol, which is more like an OPES mode 1
callout protocol.
http://www.postfix.org/SMTPD_POLICY_README.html

The Sendmail milter protocol is another mode 1 example, but it's defined
by its implementation rather than in an open fashion.
http://www.milter.org/milter_api/

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
CULLERCOATS: THE STRONG SOUTHWEST WINDS WILL AFFECT SOUTHERN AREAS ON
SATURDAY, WHILE THE WINDS EASE FOR A TIME IN THE NORTH.

---

From: Tony Finch <[email-masked]>
Date: Wed, 2 Feb 2005 18:02:53 +0000

> receive/send and defer storage until it is clear that the delivery

In principle, yes, but timeouts become much more likely (e.g. because of
anti-spam measures on the next hop server) which greatly increases the
likelihood of accidental message duplication. There's a huge amout of
added complexity if the message has multiple recipients and the MTA isn't
relaying to a single destination.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
DOGGER FISHER GERMAN BIGHT: WEST OR NORTHWEST 5 OR 6, DECREASING 4. OCCASIONAL
DRIZZLE. MODERATE OR GOOD.

---

From: "The Purple Streak, Hilarie Orman" <[email-masked]>
Date: Wed, 2 Feb 2005 11:18:58 -0700

A callout server could offer a specific service: queue.

Even though this isn't part of the use case discussion, I'm interested
in understanding why caching isn't a viable option for mail delivery.
For the simple case of single recipient, surely it makes sense?

Hilarie

---

From: "Hector Santos" <[email-masked]>
Date: Wed, 2 Feb 2005 13:38:33 -0500

----- Original Message -----
From: "Martin Stecher" <[email-masked]>
To: <[email-masked]>
Sent: Wednesday, February 02, 2005 11:33 AM
Subject: RE: Activation points and callout modes

> if they cannot forward immediatly).

As Tony mentioned, it is pretty much a design requirement to queue your
outbound mail.  Undeliverable may must be bounce after the specific system
policy of "retries" is exhausted.  Simply throwing them away would not be
viewed as an acceptable
operation.  Of course, a growing number of systems (I say isolated small
operators) are adding filtering to the mail so it is quite conceivable that
mail can be throw away.  However, it is still a standard practice to bounce
undelivable mail that was accepted by the SMTP process.  If a transaction is
rejected at the transaction level, then no BOUNCE is required.

The idea is simple:

The expectation of deliver or the reasons for rejection is naturally
expected by the user. This is legal precedence for this in covered by
provisions in US ECPA governing "user expectations."  A dynamic rejection
provided the instant notificaiton. If the SMTP accepts the message, the
either delivery or a delay rejection notification is expected.  This is
based on the original non-store and forward Online Mail Hosting days, where
users were online creating messages:  Once the user press the "Save" button,
the system either accepted it or rejected it immediately for some policy
reason.  To accept the message and never make it deliverable or readable,
constituted a form of censorship.

This changed in 1998 with the USCA Section 230 provisions passed by Congress
that now give ISP for the most part, a near 100% immunity in areas of
content editing, including having editorial power to not post the user
message at all.   However, there is still a good samaritan provision in 230
in honoring user expectations.

Even with a PROXY concept here, I believe it still needs to follow the
current SMTP design expectations.  If the OPES device is implemented at the
DATA stage, this falls in line with the "instant notification" concept
satisfying the user expectation.   If the OPES device accepts the message,
then I believe it is now the SMTP operator responsibility (ISP) on what he
will do with the message.

> already successfully forwarded to the next MTA in a parallel SMTP dialog)?

That wouldn't make sense.  A SMTP proxy design can not break the expected
operations of a normal SMTP server.  It wouldn't be a good proxy if that was
the case.

> And how many MTAs do always store a message before forwarding?

All of them.

> And how many try a direct forward and store only if that does not work
out?

Yes and maybe no. This may depend on the features of the SMTP router.

By design, most, if not all, will go directly to the EMAIL address MX host.

The basic rule of thumb is this:

1) Get the DOMAIN of the email address (example.com)

2) Lookup the MX record(s) of the domain example.com.

2.a) This will provide the list of IP address to try to connect to.  So for
example,  if the MX records result in
10 IP addresses, then each one is tried in a round robin fashion.  If the
first IP fails, it goes to the next
and so on.

2.b) If the domain has NO MX record, then the basic rule is to use the A
record of the domain and try atleast
1 time to connect to the remote mail host server.

So to answer this question.

Yes, if there is an MX record.

Maybe no, if there is no MX record.

It could be that the DOMAIN results in a NXDOMAIN which is a no-domain exist
in DNS.  This could
be a candidate for an immediate "try once" only.

But it depends on the sysop and the features offered by the SMTP software.
If DNS is down or your internet connection is down, you certainly want to
requeue this until DNS/network is back up.

Sincerely,

Hector Santos, CTO
Santronics Software, Inc.
http://www.santronics.com
305-431-2846 Cell
305-248-3204 Office

---

From: "Hector Santos" <[email-masked]>
Date: Wed, 2 Feb 2005 14:32:15 -0500

Hi Tony,

In practice, SMTP clients also follow the specs from a timeout perspective,
which I believe the current recommendation is 5 minutes depending on the
state.

But in any case, for us, one of the anti-spoof callout features we added was
the CBV (call back verifier).  Over time, we discovered we needed to offer
an option to set a lower timeout for the SMTP-based CBV callout.  I believe
it was set to 1 minute by default and since then, we have not had report of
problems with this CBV setting.

Hmmm, I just thought of this. Here is an suggestion to solve this problem.

In our FTP server, we had a similar callout timeout problem.

Not to get into details,  the general issue was that most, if not all, FTP
clients (per FTP specifications)  will timeout if it doesn't get a DATA
channel response within X seconds for file transfers (in particular
uploads).  This was discovered to be around 30-35 seconds.

FTP servers which have a 1 to 1 upload to storage location design do not see
this problem.

For a virtual FTP server like ours, a large upload (150 megs or larger), our
FTP server will move the uploaded file from temporary storage to a final
backend operator defined storage location.  The FTP server can be in New
York City, and a virtual directory setup in a file server in Japan!

But even with a LAN,  if a virtual directory was defined with a storage
location across the network machines,  this creating the potential a huge
network I/O delays and thus a delay in the FTP server response to the FTP
client as the server was copying/moving the file to permanent storage.

As a result, the FTP Client (again, per FTP RFC specification) assumed a
TIMEOUT and closed the connection socket.  This timeout is about 30-35
seconds after the final block is sent by the client and was now in a state
waiting for the final FTP server file transfer (PUT) response.

In order to solve this problem, we issue a FTP client response as a continue
line. So the client would see this:

    226-Processing file please wait (10 megs...)
    226-Processing file please wait (20 megs...)
    226-Processing file please wait (30 megs...)
    226-Processing file please wait (40 megs...)
    226-Processing file please wait (50 megs...)
    ....
    226 Transfer complete (150.4 K/s).

This solved the timeout problem for the FTP client expecting a response
within 30/35 seconds.  The problem was generic to most FTP clients and this
solution solved it across the board.

We can probably do the same thing for SMTP.

A integrated OPES callout can issue continuation lines at a particular state
to help prevent a timeout.  I wonder if the 1xx can be used here. The RFC
2821 specs say:

  1yz   Positive Preliminary reply

      The command has been accepted, but the requested action is being
      held in abeyance, pending confirmation of the information in this
      reply.  The SMTP client should send another command specifying
      whether to continue or abort the action.  Note: unextended SMTP
      does not have any commands that allow this type of reply, and so
      does not have continue or abort commands.

So I can see the following examples:

    150-Process OPES Callout. Please wait....
    150-Process OPES Callout. Please wait....
    150-Process OPES Callout. Please wait....
    150-Process OPES Callout. Please wait....
    250 Accepted Message

    150-Process OPES Callout. Please wait....
    150-Process OPES Callout. Please wait....
    150-Process OPES Callout. Please wait....
    150-Process OPES Callout. Please wait....
    550 Sorry Message not acceptable.

The only issue I see is that I think some servers will expect the same
response code on each line.

I think it is worth while to explore this, and if need be the OPES specs
provide an 2821 Update provision to make it work.

Sincerely,

Hector Santos, CTO

---

From: "Hector Santos" <[email-masked]>
Date: Wed, 2 Feb 2005 17:08:17 -0500

Good News!

I just explored the idea of keeping an SMTP client alive with a lengthy
callout using the SMTP "1xx" response code.

I used Outlook Express (an obvious popular MUA) as the test to see how it
will behave.  I ran 4 test:

Test #1 - Does Outlook honor the "1xx " (no dash) response?
Test #2 - Does Outlook honor the "1xx-" (with dash) response?
Test #3 - Find the Timeout period for a non-responsive DATA command.
Test #4- Keep the client alive for over 6 minutes.

In test #1 and #2, I found out Outlook requires the dash version of the
continuation.

So test #1 failed:

    C: DATA
    S: 354 Start mail input; end with <CRLF>.<CRLF>
    S: 150 Please Wait

As soon as Outlook saw the "150 " (w/ space) response, a Popup Error was
displayed.

The "150-" was required, so test #2 was successful:

    C: DATA
    S: 354 Start mail input; end with <CRLF>.<CRLF>
    S: 150-Please Wait
    S: 250 Message accepted for delivery. (bytes: 2573)
    C: QUIT

Outlook accepted the "150-" without any error and waited for the 250
response.

For Test #3, I wanted to see what was the timeout when Outlook did not get
any response.

To quickly test this, I created a DATA callout with a passive (silent)
delay:

        for (int n = 0; n < 10; n++) {
            Sleep(40000);  // sleep 40 seconds
        }

This would be a total delay of 6 minutes and 40 seconds.

Within 1 minute, Outlook popped up an "Timeout" error dialog.

So in Test #4, I changed the callout to issue the "150-" response between
delays:

        for (int n = 0; n < 10; n++) {
            Send("150-Hello There #%d\r\n",n);
            Sleep(40000);  // sleep 40 seconds
        }

Eureka!!

I was able to keep Outlook alive for the entire duration!  No errors!

Here is the conversation and the 40 second time intervals:

16:39:48 C: DATA
16:39:48 S: 354 Start mail input; end with <CRLF>.<CRLF>
16:39:48 S: 150-Hello there #0.
16:40:28 S: 150-Hello there #1.
16:41:08 S: 150-Hello there #2.
16:41:48 S: 150-Hello there #3.
16:42:28 S: 150-Hello there #4.
16:43:08 S: 150-Hello there #5.
16:43:48 S: 150-Hello there #6.
16:44:28 S: 150-Hello there #7.
16:45:08 S: 150-Hello there #8.
16:45:48 S: 150-Hello there #9.
16:46:28 S: 250 Message accepted for delivery. (bytes: 2573)
16:46:28 C: QUIT

This is great news!  It will add life to extended applications simply by
keeping the client alive using the 1xx response code.

Of course, this has to be tested with other mail clients/MUAs.

And of course, in my opinion,  if we expect an increase of such activity,
then there should be some
recommendation for the maximum wait time. It should not a "Forever Thing."

Sincerely,

Hector Santos, CTO

----- Original Message -----
From: "Hector Santos" <[email-masked]>
To: "Tony Finch" <[email-masked]>; "The Purple Streak, Hilarie Orman"
<[email-masked]>
Cc: <[email-masked]>
Sent: Wednesday, February 02, 2005 2:32 PM
Subject: Re: Activation points and callout modes

---

From: jfcm <[email-masked]>
Date: Wed, 02 Feb 2005 20:26:46 +0100

>modification. No special consideration necessary -> D "falls back" to A.

This is true with unstructured text. But not with structured text messages 
(XML, ASN.1, proprietary formats, signals, etc.).

>if they cannot forward immediatly). But I may be wrong.

This is standardizing, not market study for a type of response. Also, you 
cannot consider existing systems when defining something whioch may change 
their architecture. All the point I make are part of the brain storming we 
make on a new (compatible) generation mail server.
jfc