Subject: Re: [scim] #10: Add ability to mark attributes as sensitive in the schemas

From: Kelly Grizzle <[email-masked]>
Date: Tue, 10 Sep 2013 13:42:12 +0000

+1 for the "returned" attribute and its values.

Regarding "index", this might be overkill.  Filters support different operators (eq, sw, co) to do exact and substring matching.  The index just indicates server-side behavior about the most optimal way to query.  While this information isn't harmful, it may be more than what we need.

--Kelly

-----Original Message-----
From: scim issue tracker [mailto:[email-masked]] 
Sent: Monday, September 09, 2013 3:54 PM
To: [email-masked]; Kelly Grizzle; [email-masked]
Cc: [email-masked]
Subject: Re: [scim] #10: Add ability to mark attributes as sensitive in the schemas

#10: Add ability to mark attributes as sensitive in the schemas

Comment (by [email-masked]):

 I would like to propose a series of settings per attribute to cover this  issue.  Under /schemas, each attribute would have the following
 attributes:

 "index"="none"|"substring"|"exact"
 "returned"="always"|"never"|"default"|"request"

 For example, password would have   "index"="exact" since the client would
 have to do an exact match, and "returned"="never" since the value is  hashed and should not be returned.

 Note:  the index type will also depend on the value of caseExact. For  example caseExact = false means the search index is case insensitive.
 caseExact="true" means index is case sensitive.

 "always" is used for attributes that must always be returned such as "id"
 "never" is used for write-only attributes that cannot or should not be  returned.
 "default" is used by most attributes and means are returned if the  attribute param is missing. In other words the attribute is returned on  default.
 "request" means that the attribute will only be returned if requested.
 Note: this is similar to operational attributes in LDAP.

-- 
-------------------------+----------------------------------------------
-------------------------+---
 Reporter:               |       Owner:  draft-ietf-scim-core-
  [email-masked]|  [email-masked]
     Type:  enhancement  |      Status:  new
 Priority:  major        |   Milestone:
Component:  core-schema  |     Version:
 Severity:  -            |  Resolution:
 Keywords:               |
-------------------------+----------------------------------------------
-------------------------+---

Ticket URL: <http://trac.tools.ietf.org/wg/scim/trac/ticket/10#comment:2>
scim <http://tools.ietf.org/scim/>

---

From: Phil Hunt <[email-masked]>
Date: Tue, 10 Sep 2013 09:02:18 -0700

Kelly,

The problem is that if the index is anything other than "substring", then the client has to restrict the type of filter it generates, since the index may not be able to match using the filter specified. If the filter result become unspecified (because index doesn't support filter type) and a match should not occur, what should the server do?
A.  Throw an error
B.  Simply not match the entry

Because of REST style and the public nature of these APIs, I would rather avoid complex error messages and go with B. This means that clients need discovery to find out what query filters are supported.

It goes without saying I would be open to expanding index to multi-valued and being specific as to what search filters are supported.  When I wrote the message below, I was thinking about the exact match case and that it is different from substring.

Phil

@independentid
www.independentid.com
[email-masked]

On 2013-09-10, at 6:42 AM, Kelly Grizzle <[email-masked]> wrote:

> _______________________________________________
> scim mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/scim

---

From: Samuel Erdtman <[email-masked]>
Date: Wed, 11 Sep 2013 18:18:51 +0100

I agree with Kelly.
+1 for returned
And for index I cannot se the benefit. What is the use case that needs this?

Cheers
//Samuel

On Tue, Sep 10, 2013 at 5:02 PM, Phil Hunt <[email-masked]> wrote:

> Kelly,
>
> The problem is that if the index is anything other than "substring", then
> the client has to restrict the type of filter it generates, since the index
> may not be able to match using the filter specified. If the filter result
> become unspecified (because index doesn't support filter type) and a match
> should not occur, what should the server do?
> A.  Throw an error
> Because of REST style and the public nature of these APIs, I would rather
> avoid complex error messages and go with B. This means that clients need
> discovery to find out what query filters are supported.
>
> It goes without saying I would be open to expanding index to multi-valued
> and being specific as to what search filters are supported.  When I wrote
> the message below, I was thinking about the exact match case and that it is
> different from substring.
>
> Phil
>
> @independentid
> www.independentid.com
> [email-masked]
>
>
>
>
>
>
>
> > Regarding "index", this might be overkill.  Filters support different
> operators (eq, sw, co) to do exact and substring matching.  The index just
> indicates server-side behavior about the most optimal way to query.  While
> this information isn't harmful, it may be more than what we need.
> >
> > --Kelly
> >
> >
> > I would like to propose a series of settings per attribute to cover this
>  issue.  Under /schemas, each attribute would have the following
> > attributes:
> >
> > Note:  the index type will also depend on the value of caseExact. For
>  example caseExact = false means the search index is case insensitive.
> > "default" is used by most attributes and means are returned if the
>  attribute param is missing. In other words the attribute is returned on
>  default.
> > Ticket URL: <http://trac.tools.ietf.org/wg/scim/trac/ticket/10#comment:2
> >

---

From: Phil Hunt <[email-masked]>
Date: Wed, 11 Sep 2013 11:04:30 -0700

Samuel,

Regarding use case for index discovery….

Building a user self-service component that allows searching for colleages/friends, whatever.  The UI needs to know what attributes are searchable and what search queries are allowed (exact, startsWith, endsWith, isLike, substring, etc).

The UI could be client side tool that allows an administrator to query the data each of the cloud service providers they use.

The implication from an inter-op perspective is that either all SCIM SPs support substring indexing of all attributes, OR deployers can set specific indexes based on their performance needs and abilities.  If the latter, the client has to know in order to know what searches are possible.

It is doubtful that many of us can support substring indexing of all attributes. Putting aside the performance issue, there are issues with some attributes where privacy or security would block partial match searches and only exact match searches would be allowed - very much acting like an 'Identity Oracle'.

Phil

@independentid
www.independentid.com
[email-masked]

---

From: Samuel Erdtman <[email-masked]>
Date: Thu, 12 Sep 2013 20:18:44 +0100

Hi Phil

Are you saying that some backend systems might not allow some filter
operations and that it therefore would be good to indicate which so that
the client does not need to run into an error?

Or is it just to indicate that certain searches might take a bit longer?
this is how Kelly interprets it.

If the server has search query limitations it would have to define the
error any way since the client might ignore the not allowed information.

I would vote for simplicity i.e. try the operation and adapt according to
error message. And if it is just a speed indication I cannot se any reason
for it, it just adds complexity.

Cheers
//Samuel

On Wed, Sep 11, 2013 at 7:04 PM, Phil Hunt <[email-masked]> wrote:

> Building a user self-service component that allows searching for
> colleages/friends, whatever.  The UI needs to know what attributes are
> searchable and what search queries are allowed (exact, startsWith,
> endsWith, isLike, substring, etc).
>
> The UI could be client side tool that allows an administrator to query the
> data each of the cloud service providers they use.
>
> The implication from an inter-op perspective is that either all SCIM SPs
> support substring indexing of all attributes, OR deployers can set specific
> indexes based on their performance needs and abilities.  If the latter, the
> client has to know in order to know what searches are possible.
>
> It is doubtful that many of us can support substring indexing of all
> attributes. Putting aside the performance issue, there are issues with some
> attributes where privacy or security would block partial match searches and
> only exact match searches would be allowed - very much acting like an
> 'Identity Oracle'.
>
> Phil
>
> @independentid
> www.independentid.com
> [email-masked]
>
>
>
>
>
>
>
>> > I would like to propose a series of settings per attribute to cover
>> this  issue.  Under /schemas, each attribute would have the following
>> > attributes:
>> >
>> > Ticket URL: <
>> http://trac.tools.ietf.org/wg/scim/trac/ticket/10#comment:2>

---

From: Phil Hunt <[email-masked]>
Date: Thu, 12 Sep 2013 13:51:11 -0700

Inline

Phil

On 2013-09-12, at 12:18, Samuel Erdtman <[email-masked]> wrote:

> Hi Phil
> 
> Are you saying that some backend systems might not allow some filter operations and that it therefore would be good to indicate which so that the client does not need to run into an error?
[ph] yes. And so that clients never receive an error. 
> 
> Or is it just to indicate that certain searches might take a bit longer? this is how Kelly interprets it.
No
> 
> If the server has search query limitations it would have to define the error any way since the client might ignore the not allowed information.
> 
[ph] errors need to be avoided in a restful environment. I have put text in issue 48(?) to cover processing rules for invalid filters. 
> I would vote for simplicity i.e. try the operation and adapt according to error message. And if it is just a speed indication I cannot se any reason for it, it just adds complexity.

[ph] errors mean huge complexity. Remember x.500?

---

From: Phil Hunt <[email-masked]>
Date: Thu, 12 Sep 2013 13:54:54 -0700

Also. Check rfc2251 filter processing rules. I think this is an example of a proven model for filter processing. 

Phil

---

From: Kelly Grizzle <[email-masked]>
Date: Thu, 12 Sep 2013 22:17:47 +0000

IMO, silently ignoring invalid queries leads to more problems than the client having to handle errors.  It seems like an error 500 with a reasonable error message would be easier for a client to be able to understand why their query did not work the way they intended.

We could return the supported index type information in the schema so that servers can be more self-documenting.  This would be especially useful if you are trying to build a generic client that has no a priori knowledge of the server.

On another note, to me an “index” (and most coming from the RDBMS world) indicates whether querying over a column will run quickly or not.  If we add this to the schema, maybe this should be “supportedFilters” instead.

--Kelly

From: Phil Hunt [mailto:[email-masked]]
Sent: Thursday, September 12, 2013 3:55 PM
To: Samuel Erdtman
Cc: [email-masked]; [email-masked]; scim issue tracker; Kelly Grizzle
Subject: Re: [scim] #10: Add ability to mark attributes as sensitive in the schemas

Phil

On 2013-09-12, at 13:51, Phil Hunt <[email-masked]<mailto:[email-masked]>> wrote:
Inline

Phil

On 2013-09-12, at 12:18, Samuel Erdtman <[email-masked]<mailto:[email-masked]>> wrote:
Hi Phil

Cheers
//Samuel

On Wed, Sep 11, 2013 at 7:04 PM, Phil Hunt <[email-masked]<mailto:[email-masked]>> wrote:
Samuel,

Phil

@independentid
www.independentid.com<http://www.independentid.com>
[email-masked]<mailto:[email-masked]>

On 2013-09-11, at 10:18 AM, Samuel Erdtman <[email-masked]<mailto:[email-masked]>> wrote:

I agree with Kelly.
+1 for returned

Cheers
//Samuel

On Tue, Sep 10, 2013 at 5:02 PM, Phil Hunt <[email-masked]<mailto:[email-masked]>> wrote:
Kelly,

Phil

@independentid
www.independentid.com<http://www.independentid.com/>
[email-masked]<mailto:[email-masked]>

On 2013-09-10, at 6:42 AM, Kelly Grizzle <[email-masked]<mailto:[email-masked]>> wrote:

> From: scim issue tracker [mailto:[email-masked]<mailto:[email-masked]>]
> To: [email-masked]<mailto:[email-masked]>; Kelly Grizzle; [email-masked]<mailto:[email-masked]>
> Cc: [email-masked]<mailto:[email-masked]>
> Comment (by [email-masked]<mailto:[email-masked]>):
>
>  [email-masked]<mailto:[email-masked]>|  [email-masked]<mailto:[email-masked]>
> [email-masked]<mailto:[email-masked]>

_______________________________________________
scim mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/scim

_______________________________________________
scim mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/scim

_______________________________________________
scim mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/scim

---

From: Phil Hunt <[email-masked]>
Date: Thu, 12 Sep 2013 17:53:44 -0700

Sorry Kelly, I have to disagree.  Let's think this through carefully….

REST APIs need to have simplistic error signalling.  We have to assume cases where no detailed error message will be returned. 

When you state that any unresolved filter must fail the entire operation it often may mean no query is possible. For example a filter with multiple filter terms that are OR'd together would have to fail if even one term could not be resolved.

What happens when one attribute has an attribute and another does not?  How does the filter term resolve?

IMPORTANT CASE: when we extended get to work at the root level, the assumption is that for some namespaces some filters aren't resolvable. The assumption was that items would not be matched but that processing would return only those items that match (unresolvable items are ignored). So if a SCIM searches throws error, we have to then re-write how top level searching works since most of those would fail because different branches have different attributes. The only queries that could succeed are where attributes are common across all resources - rendering the feature useless.

Phil

@independentid
www.independentid.com
[email-masked]

---

From: Leif Johansson <[email-masked]>
Date: Fri, 13 Sep 2013 08:47:00 +0200

On 09/13/2013 02:53 AM, Phil Hunt wrote:
> Sorry Kelly, I have to disagree.  Let's think this through carefully....
>
... with the chair switch firmly locked in the OFF-position
> REST APIs need to have simplistic error signalling.  We have to assume
> cases where no detailed error message will be returned. 
>
> When you state that any unresolved filter must fail the entire
> operation it often may mean no query is possible. For example a filter
> with multiple filter terms that are OR'd together would have to fail
> if even one term could not be resolved.
>
There is a *big* difference between being able to process a filter and
not matching because
data isn't available and not being able to process a filter because you
don't understand the
filter definition. Filtering on givenName=Foo when then entry doesn't
have a givenName
attribute clearly should resolve to False.

The case when you don't have '=' in your set of filter operations should
result in a 400 (bad
request) imo

        Cheers Leif

---

From: Phil Hunt <[email-masked]>
Date: Fri, 13 Sep 2013 00:33:12 -0700

Leif,

I think a little more detail of the detailed error response is known.  Returning 400 won't allow clients to figure out why their search didn't work.  It seems to me, this has only been superficially thought through. 

I believe the text I put in is non-breaking where as changing processing rules to throwing bad request errors is a major change.

The example I gave was what happens across different data sets processed in a single query.  It turns out to not be a problem based on the algorithm that LDAP uses.

Phil

@independentid
www.independentid.com
[email-masked]

On 2013-09-12, at 11:47 PM, Leif Johansson <[email-masked]> wrote:

> There is a *big* difference between being able to process a filter and not matching because
> data isn't available and not being able to process a filter because you don't understand the 
> filter definition. Filtering on givenName=Foo when then entry doesn't have a givenName 
> The case when you don't have '=' in your set of filter operations should result in a 400 (bad
> request) imo
> 
>         Cheers Leif
> 
>

---

From: Kelly Grizzle <[email-masked]>
Date: Fri, 13 Sep 2013 18:42:08 +0000

I'm with Leif here - missing attribute handling is fine as-is, but a server refusing to query because of an unsupported operation is quite different.

Regarding detailed errors, the spec already has facilities for this:

   The SCIM Protocol uses the response status codes defined in HTTP [12]
   to indicate operation success or failure.  In addition to returning a
   HTTP response code implementers MUST return the errors in the body of
   the response in the client requested format containing the error
   response and, per the HTTP specification, human-readable
   explanations.  Error responses are identified using the following
   URI: 'urn:scim:schemas:core:2.0:Error'.

In addition to this, issue #46 was opened to address allowing non-HTTP response codes (ie - SCIM-specific error codes) to be returned in addition to the HTTP response code to give more information.

IMO, returning an error here would not be a breaking change.  I think that the current specs are just largely deficient around possible error conditions.

--Kelly

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Phil Hunt
Sent: Friday, September 13, 2013 2:33 AM
To: Leif Johansson
Cc: [email-masked]
Subject: Re: [scim] #10: Add ability to mark attributes as sensitive in the schemas

Leif,

Phil

@independentid
www.independentid.com
[email-masked]

On 2013-09-12, at 11:47 PM, Leif Johansson <[email-masked]> wrote:

> not matching because data isn't available and not being able to 
> process a filter because you don't understand the filter definition. 
> Filtering on givenName=Foo when then entry doesn't have a givenName attribute clearly should resolve to False.
>

---

From: Phil Hunt <[email-masked]>
Date: Fri, 13 Sep 2013 12:21:20 -0700

I am open to considering a proposal, but it needs to be fully defined with detailed error messages.

This is a major change for us.

Phil

@independentid
www.independentid.com
[email-masked]