Subject: [scim] evaluating filters: true, false, and undefined

From: Dale Olds <[email-masked]>
Date: Tue, 11 Sep 2012 18:51:07 -0700

section 3.2.2.1 of the API doc describes filtering expressions and 
bindings. It is very clear about what should happen if an operator is 
not defined (the example is 'regex'), but I do not see anything which 
explicitly states what happens if an attribute in the expression is not 
defined. The closest statement is in the first paragraph: "The 
expression language that is used in the filter parameter supports 
references to attributes and literals." I suppose that could mean a 
reference to an unknown attribute is not supported, but it doesn't 
really say that, and it doesn't say undefined attributes are an error.

In filtering code for similar systems I've seen implementations that did 
not require attributes to be defined. This can be quite natural but does 
make the implementation more interesting. Consider this filter of rooms:

True if contains someone with hat size > 7 or wearing a read shirt.

There may not be anyone wearing a hat in any room, but it would still 
find rooms with red shirts. IOW, it only takes one true component of an 
'or' expression to make it true, even if one of the components is 
undefined.

"True if someone with foobar > 7 or wearing a read shirt" works just as 
well.

This is not a big issue for us, we're just checking out implementation 
options. What do most implementations currently do?

--Dale

---

From: Samuel Erdtman <[email-masked]>
Date: Wed, 12 Sep 2012 08:21:57 +0200

In the scimproxy we will probably do it by interpreting undefined as
false i.e. the statements would be evaluated as you describe it.
However currently we only support one filter operation i.e. no 'and'
'or' stuff yet. Have actually not tested much filtering during the
interops.

If you would want to require foobar to be precent before evaluating
rest of the expression you could state it as folows:
?filter=PR foobar AND (foobar GT 7 OR shirtColor EG red)
(PR means present i.e. not undefined)

Regards
//Samuel

On Wed, Sep 12, 2012 at 3:51 AM, Dale Olds <[email-masked]> wrote:
> section 3.2.2.1 of the API doc describes filtering expressions and bindings.
> It is very clear about what should happen if an operator is not defined (the
> example is 'regex'), but I do not see anything which explicitly states what
> happens if an attribute in the expression is not defined. The closest
> statement is in the first paragraph: "The expression language that is used
> in the filter parameter supports references to attributes and literals." I
> suppose that could mean a reference to an unknown attribute is not
> supported, but it doesn't really say that, and it doesn't say undefined
> attributes are an error.
>
> In filtering code for similar systems I've seen implementations that did not
> require attributes to be defined. This can be quite natural but does make
> the implementation more interesting. Consider this filter of rooms:
>
> There may not be anyone wearing a hat in any room, but it would still find
> rooms with red shirts. IOW, it only takes one true component of an 'or'
> expression to make it true, even if one of the components is undefined.
>
> _______________________________________________
> scim mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/scim
>

---

From: Dale Olds <[email-masked]>
Date: Fri, 14 Sep 2012 18:17:59 -0700

I would expect that if "PR foobar" on a resource evaluates to false it 
means "this resource has no value for attribute foobar", which is not 
quite the same as "this server does not know what foobar means". 
Nevertheless, if the server does not know what an attribute means no 
resource will have a value for that attribute, so perhaps it's close 
enough. I can't think of an example that's not overly contrived where 
the difference would matter.

One of the reasons I brought this issue up is because there are places 
we'd like to add our own scim extensions, and we'd like to be able to 
write queries so that they work whether a particular scim server 
supports that extension or not. I also need to know how to handle 
undefined attributes in our own server.

Would the working group support this additional sentence to the first 
paragraph of section 3.2.2.1?

"Filters that contain references to attributes unknown to the server do 
not cause an error, but are evaluated such that those attributes are not 
present on each resource."

--Dale

---

From: Kelly Grizzle <[email-masked]>
Date: Mon, 17 Sep 2012 16:53:46 +0000

> "Filters that contain references to attributes unknown to the server do not
> cause an error, but are evaluated such that those attributes are not present
> on each resource."

I would not support this.  IMO filters that reference attributes that are unknown by the server should return an error, otherwise the client may get unexpected results.

> we'd like to be able to write queries so that they work whether a particular
> scim server supports that extension or not.

The client should be able to use the /Schemas resource to determine whether the server supports the extension.

In general, I am in favor of making things simpler for the client, but this seems like a case where it would be beneficial for the server to be more strict.

--Kelly

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Dale Olds
Sent: Friday, September 14, 2012 8:18 PM
To: Samuel Erdtman
Cc: [email-masked]
Subject: Re: [scim] evaluating filters: true, false, and undefined

I would expect that if "PR foobar" on a resource evaluates to false it means "this resource has no value for attribute foobar", which is not quite the same as "this server does not know what foobar means". 
Nevertheless, if the server does not know what an attribute means no resource will have a value for that attribute, so perhaps it's close enough. I can't think of an example that's not overly contrived where the difference would matter.

One of the reasons I brought this issue up is because there are places we'd like to add our own scim extensions, and we'd like to be able to write queries so that they work whether a particular scim server supports that extension or not. I also need to know how to handle undefined attributes in our own server.

Would the working group support this additional sentence to the first paragraph of section 3.2.2.1?

"Filters that contain references to attributes unknown to the server do not cause an error, but are evaluated such that those attributes are not present on each resource."

--Dale

On 09/11/2012 11:21 PM, Samuel Erdtman wrote:
> ?filter=PR foobar AND (foobar GT 7 OR shirtColor EG red) (PR means 
> present i.e. not undefined)
>
> Regards
> //Samuel
>
>> It is very clear about what should happen if an operator is not 
>> defined (the example is 'regex'), but I do not see anything which 
>> explicitly states what happens if an attribute in the expression is 
>> not defined. The closest statement is in the first paragraph: "The 
>> In filtering code for similar systems I've seen implementations that 
>> did not require attributes to be defined. This can be quite natural 
>> but does make the implementation more interesting. Consider this filter of rooms:
>>
>> find rooms with red shirts. IOW, it only takes one true component of an 'or'
>> "True if someone with foobar > 7 or wearing a read shirt" works just 
>> as well.
>>
>> This is not a big issue for us, we're just checking out 
>> implementation options. What do most implementations currently do?
>>
>> --Dale
>>
>>
>>
>>
>>
>>

---

From: Dale Olds <[email-masked]>
Date: Thu, 20 Sep 2012 17:59:27 -0700

Kelly, thanks for the reply. I'm more interested in what is compatible 
with other implementations than a particular outcome, and your position 
is certainly implementable. Just in case there are others that might 
agree with my reasoning, I will reply inline below, but I'm more 
interested in determining if the wg feels this issue should be clarified.

replies inline:

> I would not support this.  IMO filters that reference attributes that are unknown by the server should return an error, otherwise the client may get unexpected results.

I think it depends on the client and how much the scim schema is 
extended in practice. If clients are tightly coupled to the server then 
they can expect specific attributes to be known and anything else is an 
error. If the scim server supports extensions (as our implementation 
does, and as do others represented on this list), then IMO it's easier 
with fewer unexpected results if they can make queries that work in a 
reasonable fashion to different servers.

> The client should be able to use the /Schemas resource to determine whether the server supports the extension.

Of course, but that means that an application needs to query the server 
first and will need to support multiple queries based on what extensions 
are supported. As I said above, it depends on the client application and 
how often there are variations in the server schema, but in my 
experience it's crucial that schema extensions can be added without 
requiring clients and servers to be in sync.

> In general, I am in favor of making things simpler for the client, but this seems like a case where it would be beneficial for the server to be more strict.

Complete agreement on keeping things simpler for the client, I just 
think it's simpler to make logical queries rather than querying for 
schema support.

I prefer: does joe have a hat or a foobar?

rather than: if (server supports the foobar extension) then (does joe 
have a hat or a foobar?) else (does joe have a hat?)

--Dale

---

From: "Morteza Ansari (moransar)" <[email-masked]>
Date: Fri, 21 Sep 2012 16:37:49 +0000

You have raised a good question and we should clarify this in the spec.
We should capture this in the issue tracker.

Cheers,
Morteza

On 9/20/12 5:59 PM, "Dale Olds" <[email-masked]> wrote:

>> I would not support this.  IMO filters that reference attributes that
>>are unknown by the server should return an error, otherwise the client
>>may get unexpected results.
>
>> The client should be able to use the /Schemas resource to determine
>>whether the server supports the extension.
>
>> In general, I am in favor of making things simpler for the client, but
>>this seems like a case where it would be beneficial for the server to be
>>more strict.
>
>>>>bindings.