Subject: Re: Start on OPES protocol work

From: "Martin Stecher" <[email-masked]>
Date: Sun, 16 Feb 2003 13:57:10 +0100

> 
> >  Defining it this way should guarantee a very strict and 
> clear protocol
> >  message syntax that avoids those complicated phrases like 
> "after sending
> >  this, the client has to wait until blah blah" and this 
> should then have
> >  a positive effect on protocol implementations that should 
> avoid that
> >  messages on an open persistent connection run out-of sync.
> 
> "out-of sync"?  I'm not sure what the problem is.  Could you 
> elaborate,
> because it's important that we all understand what the 
> requirements mean.

With "out-of-sync" I mean that either no more request can be sent on an open connection because client and server are waiting endless time for an event that the other peer will not trigger (again) or a connection shutdown because one peer detected a syntax error by peeking into a wrong part of the data.

I know from ICAP that these things could happen. Two examples:

Due to the preview feature and the 204 response an ICAP server wants to stop a message transaction early; it sends the 204 response and is prepared for the next request that comes on this connection. Unfortunately it was too lazy in being patient to wait and acknowledge the rest of the data from the first message that it does not longer care about but the ICAP client is still forwarding. The ICAP server tries to interpret that later part of the message data as the next ICAP request header and fails: Server is not longer in sync.

Or another example from ICAP:
Due to the little complicated "ieos extension" indicating that a message is not larger than the preview which was sent, an ICAP server could be buggy in a way that waits for more data which is not coming. ICAP client and server wait endless time for each other. I saw an implementation where this happened only for the case in which HTTP body size was exactly the size of the preview.

> 
> >  2.
> >  The same question from Markus points me to another feature 
> that got on
> >  my wish list last year: Although I know that we want to 
> concentrate on
> >  HTTP, section 3.11 encourages us to define a protocol that 
> could be used
> >  for any application-layer protocol. And unlike most application
> >  protocols that have one source and one destination, SMTP 
> has one sender
> >  but many recipients. An OPES protocol that is prepared to 
> encapsulate
> >  SMTP messages should be aware that a callout service may 
> have different
> >  results for different recipients.
> >  Therefore I'd welcome if the OPES protocol is able (or can 
> be easily
> >  extended) to send multiple replies on one request.
> 
> Hmmmm - I'm not sure that there's a clear need for multiple replies,
> interesting as it is.  Is this example for OPES located between the
> sender's mail agent and the SMTP server?  Or for OPES located between
> an SMTP server and the users, working on a message sent to a local
> alias that is a list?  Somehow the callout server (or the OPES box?)
> knows the expansion of the alias?

This example is for an OPES box that works as an intermediate SMTP gateway. It could be the sender's SMTP server itself ot a later gateway in the forwarding chain.
If this box receives message for 20 users of a single domain, it is a bad idea to create 20 copies of the message first and sending one-sender-one-recipient-pairs to the callout service when maybe 18 copies are filtered with the same settings but 2 copies have a different filter result.
I'd would like a protocol that allows the box to forward the SMTP message as is and the callout server to reply "Here is a modified message for 18 recipients and a second message copy for the other two recipients following right after."

[...]

> 
> >  I think about a very lightweight but still powerfull and flexible
> >  protocol, building on components of HTTP/HTTPS.
> >  Only knowing a little about SOAP and BEEP, I think that 
> they are too
> >  complex to define a direct base for the OPES protocol, 
> although there
> >  seem to be very interesting concepts which could be integrated.
> 
> SOAP might be considered complicated, but BEEP seems to be an entirely
> different and simpler kind of thing.  Possibly simpler than HTTP.

I would like an even simpler basic message structure. Please see my argumentation in my response to Robert.

Martin

---

From: "The Purple Streak, Hilarie Orman" <[email-masked]>
Date: Sun, 16 Feb 2003 13:56:52 -0700

1. I believe that the protocol requirements stipulate the use of
independent "channels", and those were put in exactly for the the case
you are discussing for preview.  If channels don't solve the problem,
we need to know why.

2. For the SMTP case, let's suppose that the site policy requires virus
scanning, but some subscribers to the OPES box want potential viruses
removed, and some simply want them flagged.  Assume a message has been
sent to several recipients of a domain with an OPES virus scanning
service, and its body is denoted as "A" (identified by its hash, I
suppose).  If it's the OPES processor that holds the user preferences,
then it can handle the process of sending A to a callout server for
virus scanning and asking for both removal and flagging, possibly in
one request, but at worst in two.  Ah, I see, you'd like to eliminate
the two requests, and have a request type that asks for multiple
services and multiple replies.  This is different from asking for
multiple services and one reply.  A notation for this might be something
like:

  Request(data=D; service=A AND service=B)
  Reply(data=A(B(D)))

  Request(data=D; service=A; service=B)
  Reply(data=A(D); data=B(D))

Have I captured the requirement correctly?

3. I know the arguments for simple formats, but please give something
specific about your objection to BEEP.  My objection to XML encoding
is that it requires many instructions to parse it into tag pairs and
parameters, to create a parse tree, and to traverse the tree.  When
comparing this to direct indexing, or even scans for end-of-header,
end-of-line, XML is very slow.  But, experience has shown that there
are distinct limits to what can be accomplished with fixed parameters
positions and bit encodings, and even TCP has had a great deal of
trouble staying within its header limitations.  Up at the application layer,
I think some compromise will be necessary.  I'd thought BEEP was actually
a good point to begin looking for that compromise point.  What is so
awful about it?

Hilarie

---

From: "Reinaldo Penno" <[email-masked]>
Date: Sun, 16 Feb 2003 21:06:29 -0500

----- Original Message -----
From: "Martin Stecher" <[email-masked]>
To: "OPES Group" <[email-masked]>
Sent: Sunday, February 16, 2003 7:57 AM
Subject: Re: Start on OPES protocol work

> With "out-of-sync" I mean that either no more request can be sent on an
open connection because client and server are waiting endless time for an
event that the other peer will not trigger (again) or a connection shutdown
because one peer detected a syntax error by peeking into a wrong part of the
data.
> Due to the preview feature and the 204 response an ICAP server wants to
stop a message transaction early; it sends the 204 response and is prepared
for the next request that comes on this connection. Unfortunately it was too
lazy in being patient to wait and acknowledge the rest of the data from the
first message that it does not longer care about but the ICAP client is
still forwarding. The ICAP server tries to interpret that later part of the
message data as the next ICAP request header and fails: Server is not longer
in sync.

Hello Martin,

I guess that a server could also end a transaction early for security
reasons...say, wouldn't sequence numbers + handshake + timers policy ala TCP
take care of this. It seems to me ( if I captured your thoughts right) kind
of the same problem we have in TCP when one end closes a connection and
there are still packets in flight.

regards,

Reinaldo

> Due to the little complicated "ieos extension" indicating that a message
is not larger than the preview which was sent, an ICAP server could be buggy
in a way that waits for more data which is not coming. ICAP client and
server wait endless time for each other. I saw an implementation where this
happened only for the case in which HTTP body size was exactly the size of
the preview.
> This example is for an OPES box that works as an intermediate SMTP
gateway. It could be the sender's SMTP server itself ot a later gateway in
the forwarding chain.
> If this box receives message for 20 users of a single domain, it is a bad
idea to create 20 copies of the message first and sending
one-sender-one-recipient-pairs to the callout service when maybe 18 copies
are filtered with the same settings but 2 copies have a different filter
result.
> I'd would like a protocol that allows the box to forward the SMTP message
as is and the callout server to reply "Here is a modified message for 18
recipients and a second message copy for the other two recipients following
right after."