Subject: RE: capability negotiations

From: "Martin Stecher" <[email-masked]>
Date: Mon, 6 Oct 2003 23:12:09 +0200

Hi Alex,

> 
> 	I like simplifications and understand what you are trying to
> do. I agree that there is a complexity problem here. I think your
> solution works OK on a local level. I am, however, concerned about
> overall consistency and symmetry of protocol mechanisms. Let me try to
> explain with a couple of examples and observations.
> 
> 	You propose to use Optional-Parts, Skip-Parts using named
> headers as opposed to [named] feature parameters. Once we separate a
> feature ID and its parameters, how will agents report their [multiple]
> capabilities in a [single] "I Support" (i-can) response?
> 
> 	For example, consider a callout service that wants to always
> receive request-headers in RESPMOD mode and wants to receive
> request-body in RESPMOD mode if and only if that body is chunked. You
> cannot express that condition using a single feature structure or a
> single feature ID with attached named parameters. You need two
> structures OR two messages each with its own group of named
> parameters.
> 
> 	Do you think that the above scenario is too esoteric to
> support?

Yes, I think it is too esoteric.
So far, I am not aware of an existing filter application that needs request-bodies in RESPMOD, but probably there are some useful filters that like to have this meta info. But why should the interest in that data depend on the transfer encoding between client and proxy? I am lacking phantasy here.

What I understand is your desire to keep the protocol as unrestricted and powerful as possible. Not restricting any protocol feature we have, because it may be useful now or later in some more or less esoteric environment. That's fine with me.

On the other hand I am concerned about negotiation complexity. I think that it is unlikely that two agents find a perfect match or the nearly perfect second choice even if you list all features in a complicated decision matrix. Agents need to workaround lacks of capabilities of the other side and so they could work around a slightly restricted negotiation procedure.

> 
> 	When negotiating/announcing encryption, would not the same
> problem arise? How to express "RSA with 512bit keys" and "DES with
> 1024bit keys" in a single NO or i-can message?

I do not think that encryption is transaction dependend, it works on connections.
Certainly an agent can respond to a "which encryption method do you support" with a list of encryptions in an i-can message.
Even if it was a feature of an NO message, the value of the named parameter could be a list of course, such as my example
	Optional-Parts: request-header, request-body
which should better read
	Optional-Parts: (request-header, request-body)
to be consistent and symetric, sorry for that mistake.

> 
> 	Will we never need to offer two features at once?
> 
> 	Do we want to eliminate i-can mechanism altogether?
> 
> 	Once we start using named parameters for features, we cannot
> describe multiple conflicting features within one OCP message. A
> general solution is a list of structures (with [named] members).  I
> would even argue that we can make everything symmetric if named
> members are using name:value MIME syntax OR if named parameters adopt
> a name=value non-MIME syntax instead.
> 
> 	This generalization allows us to, essentially, list OCP
> statements/messages (such as negotiation offers or i-can responses) in
> a single OCP message:
> 
> 	NO ({
> 		"feature-id1"
> 		param1: value1a
> 		param2: value2a
> 	},{
> 		"feature-id1"
> 		param1: value1b
> 		param2: value2b
> 	})
> 	Applies-to-whole-message-name1: value1
> 	Applies-to-whole-message-name2: value2
> 	...
> 
> where "Applies-to-whole-message-name" stands for a property that
> applies to the whole NO message and not any feature being negotiated
> in particular. SG, for example, is one of such named parameters.

We can go this way, if you insist.
I think usage of named items (either parameters or members of structures) is the key.
Whether structures are needed for (some) features depends on the complexity we believe is necessary for a powerful protocol vs. the complexity that makes the protocol hard to handle (for a human being).

> 
> As you can see, this leads to a very symmetric/consistent syntax
> because the whole message prefix becomes nothing but an OCP structure
> (just without the surrounding {} which perhaps should be added)! This
> simplifies parsing and internal representation, IMO: the same
> code/structures are used to store and manage whole messages and
> message parts.
> 
> We are now kind of stuck in the middle: we have structures but we are
> afraid of them. This is probably the worst position to be in!
> 
> I like symmetry and consistency. The idealist in me says "a feature
> should be a structure OR we should not have structures at all". Or, in
> other words, "either we commit to support structures in any context or
> we do not support them at all".

For me this is too black and white.
It is great to have structures. Structures with named members are even better.
But that does not imply automatically that structures need to be used everywhere, IMO.
Symmetry is great but is it the number 1 design rule that must never break?

> 
> Do you see what I am getting at? Would you prefer to get rid of
> structures completely OR use them extensively? Do you see any other
> consistent design options?

If you (still) insist in black and white decision, I vote for extensive usage.
I assume that you then also like to continue support for multiple application profile features in NR messages? No restrictions because the syntax is there to make it possible. So be it.
For my full understanding, could you please write the structured NO/NR messages for my specific example that I gave in the previous message; and also how it will look like when adding the esoteric condition you suggested?

Regards
Martin

> 
> On Mon, 29 Sep 2003, Martin Stecher wrote:
> 
> >
> > Hi,
> >
> > here are some thoughts about the capability negotiations.
> >
> >
> > Current status:
> > HTTP/OCP uses NO and NR messages to negotiate one of the 
> two defined application profiles (request or response) plus 
> optional message parts.
> > Futher information exchange about capabilities for 
> transfer- and content-encodings needs to be defined and done.
> >
> >
> > Question:
> > Can we simplify the application profiles negotiation and 
> how to implement the encodings stuff?
> >
> >
> > Compare with ICAP/1.0:
> > ICAP defines the OPTIONS method; in a response to an 
> OPTIONS request, the ICAP server returns various information, 
> including support for the main methods (REQMOD/RESPMOD). But 
> which method to use is not negotiated and ICAP does not know 
> something about optional message parts, skipped parts or 
> information about supported encodings.
> > If to be implemented it would probably be done in the 
> OPTIONS request.
> >
> >
> > Do we really need to negotiate the HTTP profile?
> > Depending on the vectoring point, an OPES processor will 
> either send a HTTP request or an HTTP response via OCP to the 
> callout server. It makes not much sense to offer both 
> profiles and let the callout server select.
> > All ICAP implementations I know and the rules language 
> show: The decision which profile to use is being done in the 
> OPES processor.
> > Still the information whether a callout server at all 
> supports this profile is important; but it could be done in a 
> "has feature" message, rather than in the negotiation offer.
> > So, I vote for restricting the NO message to send a single 
> application profile, not a list of profiles.
> > The callout server can then either accept or decline.
> > Having a list of profiles can make the negotiation offer 
> very long if additional parameters are offered as an option. 
> In the current draft we offer additional message parts as 
> options and adding more paramters would make this much too long.
> >
> > Other parameters
> > Client and server need to exchange information about (or 
> negotiate on):
> >   1. wanted additional optional message parts
> >        server asks client
> >        e.g. please add original HTTP request headers to 
> HTTP response)
> >   2. standard message parts not needed
> >        server tells client
> >        e.g. do not need long HTTP request body, only 
> interested in headers
> >   3. supported transfer encodings (or HTTP version)
> >        client and server need to tell each other
> >        e.g. is it ok for the server to add chunked transfer 
> encoding?
> >   4. supported content encodings
> >        client and server need to tell each other
> >        e.g. does the client need to decode gzip encoding 
> before sending message to server?
> >   5. maybe additional parameters, specific for the callout server
> >        e.g. number of parallel connections supported
> >   6. maybe additional parameters, specific to the service to use
> >        e.g. please add more meta information such as user info
> >
> > I think it is not a good idea to create separate messages 
> (negotiation offer or has-feature) for each parameter 
> although several messages could be sent in a single TCP frame.
> >
> > So, what about adding these paramters as named parameters 
> to the NO and NR messages. If the NO message has only a 
> single application profile it works that way.
> >
> > Example:
> >   NO "38:http://iana.org/opes/ocp/HTTP/response"
> >   SG: 10
> >   Optional-Parts: request-header, request-body
> >   Transfer-Encodings: chunked
> >   Content-Encodings: gzip, compress
> >
> >   NR "38:http://iana.org/opes/ocp/HTTP/response"
> >   SG: 10
> >   Optional-Parts: request-header
> >   Skip-Parts: response-body
> >   Transfer-Encodings: chunked
> >
> > It means:
> >   Client wants to do response modification with services in 
> group ten.
> >   It offers two additional optional message parts.
> >   Client supported chunked transfer encoding and gzip and 
> compress content encoding
> >   Server confirms that it can do response modification and 
> asks for the request-header as an optional message part.
> >   If the client is able skip the response body part, it may do so.
> >   Chunked transfer encoding is also supported by the server 
> but if the message has a content encoding, the client has 
> either to decode it or the server won't understand it (not a 
> problem in this example because Skip-Parts header indicates 
> that it does not need the body at all.
> >
> >
> > What do you think about this? Can we do it that way?
> >
> > Regards
> > Martin
> >
> >
> 
> ------------------------------------------------------------
> This mail has been scanned by debian3-smtp
> (WebWasher 4.4.1 Build 652)
> ------------------------------------------------------------
>

---

From: Alex Rousskov <[email-masked]>
Date: Mon, 6 Oct 2003 16:45:04 -0600 (MDT)

On Mon, 6 Oct 2003, Martin Stecher wrote:

> On the other hand I am concerned about negotiation complexity. I
> think that it is unlikely that two agents find a perfect match or
> the nearly perfect second choice even if you list all features in a
> complicated decision matrix. Agents need to workaround lacks of
> capabilities of the other side and so they could work around a
> slightly restricted negotiation procedure.

This is my concern as well. Perhaps we can address both concerns
(clean design and negotiation complexity) without too many sacrifices.

> I do not think that encryption is transaction dependend, it works on

Yes, I was using connection encryption to provide a better example.
But it makes no difference since negotiation _mechanisms_ for
connection- and transaction-scope should be the same, right?

> Certainly an agent can respond to a "which encryption method do you
> support" with a list of encryptions in an i-can message.

Unless I am missing something, using your simplified format, the agent
will _not_ be able to offer the above choices in NO and will not be
able to respond with them in an i-can response. Given your single
feature option, an agent can only say:

	NO "opes:connection-encryption"
	Algorithms: (RSA)
	Key-Sizes: (512);

or

	NO "opes:connection-encryption"
	Algorithms: (DES)
	Key-Sizes: (1024)

The agent cannot say:

	NO "opes:connection-encryption"
	Algorithms: (RSA,DES)
	Key-Sizes: (512,1024)

because the agent does not offer/support RSA with 1024bit keys and DES
with 512bit keys. With current, more complex scheme, the agent can
express both options (I will use named members):

	NO ({
		"opes:connection-encryption"
		Algorithms: (RSA)
		Key-Sizes: (512)
	   },{
		"opes:connection-encryption"
		Algorithms: (DES)
		Key-Sizes: (1024)
	   })
	SG: 10
	;

And the recipient can make a choice knowing all available options
(there are two options available in this example). Do you see what I
am getting at?

All of the above applies to i-can responses as well, of course: Only
lists of structures can express RSAx512 and DESx1024 ability without
implying RSAx1024 and DESx512 support.

> Even if it was a feature of an NO message, the value of the named
> parameter could be a list of course, such as my example

Of course. That's the way I interpreted it. The problem is that name
parameter lists create a full mesh of choices and do not allow for
specific subsets (see example above). The lists should still be
allowed. The above example can be extended to show a more powerful DES
implementation:

        NO ({
                "opes:connection-encryption"
                Algorithms: (RSA)
                Key-Sizes: (512)
           },{
                "opes:connection-encryption"
                Algorithms: (DES)
                Key-Sizes: (1024,2048,8192)
           })
        SG: 10
        ;

> Whether structures are needed for (some) features depends on the
> complexity we believe is necessary for a powerful protocol vs. the
> complexity that makes the protocol hard to handle (for a human
> being).

Exactly. It seems that the final decision depends on whether we think
that supplying more than one feature in a NO and responding with more
than one feature in i-can is essential to support. If the answer is
"yes", then we should support the lists of structures in NO messages.
If the answer is "no, not essential". Then we can limit NO messages to
a single feature (with a full mesh of named parameters). Do you agree
that this is the choice we face?

Did the examples above convince you that we need to support multiple
features in some negotiations?

Also, let's compare the complexity of the two approaches:

	a) Multiple structures with named members. A minimal
	   implementation would require parsing each structure,
	   checking whether the feature matches capabilities,
	   until the first matching feature is found.

	b) Single feature with named parameters. A minimal
	   implementation would require parsing the message
	   headers and checking whether the feature matches
	   capabilities

So the only difference is in the extra iteration in case (a). Is this
the added complexity you are concerned with? Or did I miss any other
complications?

> It is great to have structures. Structures with named members are
> even better. But that does not imply automatically that structures
> need to be used everywhere, IMO.

True. They only need to be used where the information cannot be
represented using simpler elements. I do not think we disagree on
whether or how to use structures. I think the only thing we need to
agree on is whether we need multiple non-meshed features in a single
negotiation offer or i-can response.

> I assume that you then also like to continue support for multiple
> application profile features in NR messages?

Not necessarily. Again, I like your simplification. I just want to
make sure it is still general _enough_ :-).

> For my full understanding, could you please write the structured
> NO/NR messages for my specific example that I gave in the previous
> message;

 >   Transfer-Encodings: chunked

becomes this:

     NO ({
	"38:http://iana.org/opes/ocp/HTTP/response"
	Optional-Parts: (request-header,request-body)
	Transfer-Encodings: (chunked)
	Content-Encodings: (gzip, compress)
     })
     SG: 10

     NR {
	"38:http://iana.org/opes/ocp/HTTP/response"
	Optional-Parts: (request-header)
	Skip-Parts: (response-body)
	Transfer-Encodings: (chunked)
     }
     SG: 10

That is, one feature offered (but there could have been more) and one
more specific feature is accepted (there could not have been more than
one).

> message; and also how it will look like when adding the esoteric
> condition you suggested?

Well, now you are pushing it! :-)

     i-can ({
	"38:http://iana.org/opes/ocp/HTTP/response"
	Optional-Parts: (request-header)
     },{
	"38:http://iana.org/opes/ocp/HTTP/response"
	Optional-Parts: (request-header,request-body)
	Transfer-Encodings: (chunked)
     })

See also encryption negotiation examples above.

Again, there seems to be only one important choice here: multiple
structures versus one; all in the context of negotiations and
capability reporting. Did more examples and clarifications change your
preferences in any way?

Thank you,

Alex.