Subject: Re: [domainrep] Possible changes to the standard (HTTP)

From: "Murray S. Kucherawy" <[email-masked]>
Date: Fri, 21 Oct 2011 15:16:27 -0700

Hi Steve,

I’ve been working on a first implementation of the proposed system using the long-form (HTTP) query/reply format.  Finally I have something to say about your message from back in July.  ☺

The new specification (-02 posted a couple of days ago) makes the query format variable based on a template the server provides.  Basically, a client configured to use your server would hit a reserved, well-known URI at your server to download the URI template it should use to make queries to your service.  Then with values the client computes, it constructs the actual query URI and makes the query.

In your example, you used a vocabulary for sender reputation, “sender-rep”.  So in the context of the REPUTE framework, you would create a vocabulary for email sender reputation (see the “vocab” drafts for an example) that creates “sender-rep” as a REPUTE application, and specify what assertions and extensions are defined within it.  This basically gives you the format that the reply will take.

The query’s format is specified by the template.  In the current HTTP draft, the four mandatory values are “application”, “scheme”, “subject” and “service”.  In your application, you’d want to include additional mandatory parameters to accommodate what you’re doing here.  This means I’ll have to modify the HTTP document to mention that vocabularies defined by other documents may also add other mandatory query parameters.  And it looks like either the “model” or “media type” document needs to create the vocabulary registry and specify registration procedures, including listing other mandatory query parameters.

All this talk of query templates and so forth is starting to make me wonder if a lightweight DNS-based mechanism is even plausible, since the only way to encode parameters is to stuff them into the query name somehow.  Rather, doing some other UDP thing might be more plausible, or perhaps dropping the lightweight thing altogether is the way to go.

-MSK

From: [email-masked] [mailto:[email-masked]] On Behalf Of Steve Allam
Sent: Monday, July 18, 2011 6:22 AM
To: [email-masked]
Subject: [domainrep] Possible changes to the standard (HTTP)

All,

The other interface we use to connect to our 'reputation engine' (the LogiQ product) uses an HTTP GET query.  A typical query would be:

http://logiq.server.com:8000/check_sender?i=193.4.5.2&s=[email-masked]&r=[email-masked]&d=inbound<http://logiq.server.com:8000/check_sender?i=193.4.5.2&s=[email-masked]&r=[email-masked]&spf=pass&dkim=pass&v=spam&subject=test>

and this can be extended using a variety of other parameters such as:

http://logiq.server.com:8000/check_sender?i=193.4.5.2&s=[email-masked]&r=[email-masked]&spf=pass&dkim=pass&v=spam&subject=test_message&d=inbound<http://logiq.server.com:8000/check_sender?i=193.4.5.2&s=[email-masked]&r=[email-masked]&spf=pass&dkim=pass&v=spam&subject=test>

If we try and encode that in the current proposed standard, we can get as far as perhaps:

http://logiq.server.com:8000/email/here.com/sender-rep<http://logiq.server.com:8000/email/here.com/sender-rep?i=193.4.5.2&s=[email-masked]&r=[email-masked]&spf=pass&dkim=pass&v=spam&subject=test>

But then need to define the sender and other parameters.  This is my main comment - there needs to be a mechanism for people to add extensible parameters to the basic query.

When we decided on the query were were going to use, we took a number of factors into consideration:

1.  Where will the url parsing take place?
    - we decided to keep the url fixed for all queries, and pass the variable data in via parameters.  Mainly this was due to having many parameters to parse, and keeping the http engine for what it was good at, passing all the parameters to the application to handle.  We do however do some work in the HTTP server (nginx) based on the params - for instance, depending on whether we get d=inbound or d=outbound, we set different rate limiting values.  However, the basic premise was - keep the application separate from the web server - although clearly an application is quite capable of parsing a URL as well as a param stream.

2. Do we use a GET or a POST to provide params?
    - we decided that as we weren't sending extended parameter data, we could live without the extra overhead of a POST command - one of the purposes of our engine is to work as fast as possible after all.
    - parameter names were again simply made as small as possible to ensure the query fitted into as few packets as possible - a little thing perhaps....

3.  Responses....
    - LogiQ is capable of returning responses in three formats:
        - XML:  we wanted to use this as it is nicely extensible.  In practice, not many of our interfaces use it; partly because when people start using it they load up an XML parser and process the result....whoops...that's gone slow for some reason....
        - Text:   As the response line is usually a single line, its simple.....
        - HTTP response code - this is the fastest and used directly in out interface with the Cloudmark Gateway - as it can support direct HTTP get queries in its rules the LogiQ interface needs no extra modules.  In order to get the quickest integration, the gateway merely uses the response code to determine reputation, and can drill down into the response body if more information is needed.  I don't expect this to be considered for a standard however.  I can't remember the exact responses we use but its something like 201 for good reputation, 404 for unknown and 503 for a bad reputation.

I know perhaps I have rambled a bit here - but I hope it gives an idea of how we are using similar mechanisms for reputation query/response - please let me know if you have any questions.

Regards,

Steve
--

Steve Allam | Chief Technology Officer | TrustSphere

3 Phillip Street, #13-03 Commerce Point, Singapore, 048693
Tel: +65 6536 5203 | Fax: +65 6536 5463
[email-masked]<mailto:[email-masked]> | www.trustsphere.com<http://www.trustsphere.com>

---

From: Chris Lewis <[email-masked]>
Date: Sat, 22 Oct 2011 00:58:58 +0200

On 11-10-22 12:16 AM, Murray S. Kucherawy wrote:

> All this talk of query templates and so forth is starting to make me
> wonder if a lightweight DNS-based mechanism is even plausible, since the
> only way to encode parameters is to stuff them into the query name
> somehow.  Rather, doing some other UDP thing might be more plausible, or
> perhaps dropping the lightweight thing altogether is the way to go.

Some DNSBLs already do encode parameters into the query name - the 
multi-subzone types.  Eg: the various SURBLs, spamhaus' offerings etc. 
If there's not many, and the lengths are kept in bounds, it should still 
be doable.

An alternate approach is to have the template for the lightweight 
mechanism specify the interpretation of the result, rather than the 
formation of the query.  Which is to a great extent simply providing a 
machine description of the return values that existing systems do as a 
web page intended for humans to read.

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Fri, 21 Oct 2011 17:05:26 -0700

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Chris Lewis
> Sent: Friday, October 21, 2011 3:59 PM
> To: [email-masked]
> Subject: Re: [domainrep] Possible changes to the standard (HTTP)
> 

In the implementation I imagine, effectively there's a template to constructing the query, like there is with URIs but shorter.  The more elements there are in the generated name, though, the more constrained the space is; if you have to spend 40 characters encoding query parameters and another ten for the name of the service, one could prevent queries from ever happening by registering a domain name that's 207 bytes long.  That's an extreme example, but if we try to pack a lot of query parameters in, it becomes more plausible that one could hide.

> web page intended for humans to read.

I think that's more plausible in the long run, except that it may not be too lightweight anymore since the reply might have to encode quite a bit of information rather than just a "score" and "confidence", which was the original target, and the specification gets more and more complex (and thus prone to errors).

I don't really know right now which is ultimately better as I haven't really taken a run at it.  Just cogitating so far.

---

From: Steve Allam <[email-masked]>
Date: Tue, 25 Oct 2011 15:18:17 +0100

Murray,

That approach sounds plausible - having to look up a template in order 
to get instructions on how to build a valid URI wouldn't have to happen 
very often - perhaps the template should also have a TTL value in it so 
that clients know how long the template is expected to be valid for - 
for our system, I would expect a template to be modified on software 
version changes only, but I guess other systems might wish to use a 
changing template (can't think why).

I shall have a look at the drafts (before July!) and comment further.

Regarding the lightweight DNS/UDP mechanisms, - the same system I think 
would work - it seems to me that it might still be worth pursuing, on 
the basis that the client can still look up the template to be used for 
the DNS mechanism, so that a proper query can be formed - again, the 
template lookup need only happen infrequently, so it wouldn't over 
burden the client application.
     - Also, it could perhaps share the same template lookup mechanism 
as the long-form query, to save inventing a DNS-based template lookup 
mechanism, but I guess you had that bit already.
     - We will continue to use our own DNS based reputation lookup, and 
there is a possibility we might be extending it soon - I'll let you know 
how that goes!

I hope you are all enjoying Paris - ironic that this is the first Maawg 
I've missed in 4 years I think, and its the closest to me (London) !

Regards,

Steve

On 21/10/2011 11:16 PM, Murray S. Kucherawy wrote:
>
> Hi Steve,
>
> I've been working on a first implementation of the proposed system 
> using the long-form (HTTP) query/reply format.  Finally I have 
> something to say about your message from back in July. J
>
> The new specification (-02 posted a couple of days ago) makes the 
> query format variable based on a template the server provides.  
> Basically, a client configured to use your server would hit a 
> reserved, well-known URI at your server to download the URI template 
> it should use to make queries to your service.  Then with values the 
> client computes, it constructs the actual query URI and makes the query.
>
> In your example, you used a vocabulary for sender reputation, 
> "sender-rep".  So in the context of the REPUTE framework, you would 
> create a vocabulary for email sender reputation (see the "vocab" 
> drafts for an example) that creates "sender-rep" as a REPUTE 
> application, and specify what assertions and extensions are defined 
> within it.  This basically gives you the format that the reply will take.
>
> The query's format is specified by the template.  In the current HTTP 
> draft, the four mandatory values are "application", "scheme", 
> "subject" and "service".  In your application, you'd want to include 
> additional mandatory parameters to accommodate what you're doing 
> here.  This means I'll have to modify the HTTP document to mention 
> that vocabularies defined by other documents may also add other 
> mandatory query parameters.  And it looks like either the "model" or 
> "media type" document needs to create the vocabulary registry and 
> specify registration procedures, including listing other mandatory 
> query parameters.
>
> wonder if a lightweight DNS-based mechanism is even plausible, since 
> the only way to encode parameters is to stuff them into the query name 
> somehow.  Rather, doing some other UDP thing might be more plausible, 
> or perhaps dropping the lightweight thing altogether is the way to go.
>
> -MSK
>
> *From:[email-masked] [mailto:[email-masked]] 
> *On Behalf Of *Steve Allam
> *Sent:* Monday, July 18, 2011 6:22 AM
> *To:* [email-masked]
> *Subject:* [domainrep] Possible changes to the standard (HTTP)
>
> All,
>
> The other interface we use to connect to our 'reputation engine' (the 
> LogiQ product) uses an HTTP GET query.  A typical query would be:
>
> http://logiq.server.com:8000/check_sender?i=193.4.5.2&s=[email-masked]&r=[email-masked]&d=inbound 
> <http://logiq.server.com:8000/check_sender?i=193.4.5.2&s=[email-masked]&r=[email-masked]&spf=pass&dkim=pass&v=spam&subject=test>
>
> http://logiq.server.com:8000/check_sender?i=193.4.5.2&s=[email-masked]&r=[email-masked]&spf=pass&dkim=pass&v=spam&subject=test_message&d=inbound 
> <http://logiq.server.com:8000/check_sender?i=193.4.5.2&s=[email-masked]&r=[email-masked]&spf=pass&dkim=pass&v=spam&subject=test>
>
> http://logiq.server.com:8000/email/here.com/sender-rep 
> <http://logiq.server.com:8000/email/here.com/sender-rep?i=193.4.5.2&s=[email-masked]&r=[email-masked]&spf=pass&dkim=pass&v=spam&subject=test>
>
> But then need to define the sender and other parameters.  This is my 
> main comment - there needs to be a mechanism for people to add 
> extensible parameters to the basic query.
>
>     - we decided to keep the url fixed for all queries, and pass the 
> variable data in via parameters.  Mainly this was due to having many 
> parameters to parse, and keeping the http engine for what it was good 
> at, passing all the parameters to the application to handle.  We do 
> however do some work in the HTTP server (nginx) based on the params - 
> for instance, depending on whether we get d=inbound or d=outbound, we 
> set different rate limiting values.  However, the basic premise was - 
> keep the application separate from the web server - although clearly 
> an application is quite capable of parsing a URL as well as a param 
> stream.
>
>     - we decided that as we weren't sending extended parameter data, 
> we could live without the extra overhead of a POST command - one of 
> the purposes of our engine is to work as fast as possible after all.
>     - parameter names were again simply made as small as possible to 
> ensure the query fitted into as few packets as possible - a little 
> thing perhaps....
>
> 3.  Responses....
>         - XML:  we wanted to use this as it is nicely extensible.  In 
> practice, not many of our interfaces use it; partly because when 
> people start using it they load up an XML parser and process the 
> result....whoops...that's gone slow for some reason....
>         - HTTP response code - this is the fastest and used directly 
> in out interface with the Cloudmark Gateway - as it can support direct 
> HTTP get queries in its rules the LogiQ interface needs no extra 
> modules.  In order to get the quickest integration, the gateway merely 
> uses the response code to determine reputation, and can drill down 
> into the response body if more information is needed.  I don't expect 
> this to be considered for a standard however.  I can't remember the 
> exact responses we use but its something like 201 for good reputation, 
> 404 for unknown and 503 for a bad reputation.
>
>
> I know perhaps I have rambled a bit here - but I hope it gives an idea 
> of how we are using similar mechanisms for reputation query/response - 
> please let me know if you have any questions.
>
> Regards,
>
> Steve
>
> -- 
> *
> **Steve Allam | *Chief Technology Officer *| TrustSphere *
>
> [email-masked] <mailto:[email-masked]> | 
> www.trustsphere.com <http://www.trustsphere.com>
>
>
>
> _______________________________________________
> domainrep mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/domainrep

-- 
*
Steve Allam | * Chief Technology Officer *| TrustSphere *

3 Phillip Street, #13-03 Commerce Point, Singapore, 048693
Tel: +65 6536 5203 | Fax: +65 6536 5463
[email-masked] | www.trustsphere.com