Subject: [scim] Persistent ID:s across services in SCIM

From: Palle Girgensohn <[email-masked]>
Date: Wed, 20 Sep 2017 00:16:38 +0200

Hello all,

I have a question about how to maintain persistent id:s for objects through differents systems if we use SCIM to sync data between them.

Say for example that we have school student register system where students, classes, study groups etc is managed. Say also that we extend SCIM with some resource types that define how to decribe time tables for the school.

We have two services that need information from the register; a time table software (A) and an attendance software (B).

The time table software (A) adds information about the time tables and uses reference ("value" & "$ref") to point out groups and users from the register for the time table entries. Of course, B needs this information.

Simple setup:

   Register

    |    |
    v    v

    A -> B

Now, according to SCIM, the id property that may be sent from Register to A and B should be ignored by A & B. Instead, A & B should make up their own id:s and return them to the register. This means that A and B cannot communicate about these objects since they don't hava any common id to reference. The externalId in SCIM is just for filtering, it cannot be used in reference attributes.

How is this supposed to work? I think that maybe I'm missing something here? Why can't the Register decide on the id for the object, it seems so much more logical?

A similar question arises even in a simple setup with a register and a service provider. If the register first provisions a user in the service provider, it gets the service provider's made up ID in the respoonse. Then it creates a group and populate it with the user. Should it then use the id from the service provider in the "members": {"value" and "$ref" ? It seems a bit backward to me?

We need the same "objects" to be synced between more than just two systems, and to me it is confusing how to keep the id:s intact. Is this not a job for SCIM at all?

Please advice :-)

Palle Girgensohn

---

From: "Vella, Shon" <[email-masked]>
Date: Tue, 19 Sep 2017 16:40:08 -0600

I believe what you are looking for is externalId for storing the
common id. It could be the id from one of the systems or it could be
something else completely - whatever the orchestration/synchronization
glue needs it to be.

Shon Vella
Identity Automation

On Tue, Sep 19, 2017 at 4:16 PM, Palle Girgensohn <[email-masked]> wrote:
> _______________________________________________
> scim mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/scim
>

---

From: "Phil Hunt (IDM)" <[email-masked]>
Date: Tue, 19 Sep 2017 16:01:15 -0700

The id in scim is a local instance identifier. Its purpose is to give permanence to the id. Ie. It never changes which gives referential integrity. The timetable system can count on the uri for the student. 

Technically the client cannot assign the id, but the server can choose to use an attribute provided (eg the studentid) for the id value.  The challenge is not to choose an id value that might change for any reason. It seems to me student ids are stable.  

The point is if you were accessing a student in another school you can't count on id being the student id. Nor that you will be able to keep them the same. 

When this happens you can use externalid to establish the linkage. 

Phil

> On Sep 19, 2017, at 3:40 PM, Vella, Shon <[email-masked]> wrote:
> 
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_scim&d=DwICAg&c=RoP1YumCXCgaWHvlZYR8PQcxBKCX5YTpkKY057SbK10&r=JBm5biRrKugCH0FkITSeGJxPEivzjWwlNKe4C_lLIGk&m=TdZHqR-ToFuh_fDFofMC8m9xMtSQi_LtTBpJmvsKWw8&s=mrJ6_FjoMs25rlq59xNH4oqI3ObIV-7Gy2R4hvH1x-8&e= 
>> 
> 
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_scim&d=DwICAg&c=RoP1YumCXCgaWHvlZYR8PQcxBKCX5YTpkKY057SbK10&r=JBm5biRrKugCH0FkITSeGJxPEivzjWwlNKe4C_lLIGk&m=TdZHqR-ToFuh_fDFofMC8m9xMtSQi_LtTBpJmvsKWw8&s=mrJ6_FjoMs25rlq59xNH4oqI3ObIV-7Gy2R4hvH1x-8&e=

---

From: "Phil Hunt (IDM)" <[email-masked]>
Date: Tue, 19 Sep 2017 16:29:59 -0700

Also. There may be privacy implications of using the student id as the scim id because it will be revealed in browser histories etc. 

Some implementations also assign a different id for each client. This prevents two separate clients from doing correlation. This minimizes the PII repercussions of the scim id value. 

Phil

---

From: Palle Girgensohn <[email-masked]>
Date: Wed, 20 Sep 2017 09:23:59 +0200

In a general case, there is a strong argument for assigning a different id for each client. Totally agree.

In our scenario the owner of the PII have good control over all services, so this is not an issue.

> 20 sep. 2017 kl. 01:29 skrev Phil Hunt (IDM) <[email-masked]>:
>

---

From: Samuel Erdtman <[email-masked]>
Date: Wed, 20 Sep 2017 10:29:55 +0200

Hi Palle!

Great to see you here on the list and that you are making progress with the
SCIM work.

I would opt for the solution I think Phil is describing, i.e. having the
*register* send an identifier (e.g. student identifier) in the create
request and then have the *time table software* and the *attendance
software* use that id as the id. I think it would be best to define a new
attribute for this id and document that when present this should become the
id in the returned object. In this way you would get okay interoperability,
not awesome but okay.

Cheers
//Samuel

On Wed, Sep 20, 2017 at 9:23 AM, Palle Girgensohn <[email-masked]>
wrote:

> > Also. There may be privacy implications of using the student id as the
> scim id because it will be revealed in browser histories etc.
> >
> > Some implementations also assign a different id for each client. This
> prevents two separate clients from doing correlation. This minimizes the
> PII repercussions of the scim id value.
> >
> > Phil
> >
> >>> I have a question about how to maintain persistent id:s for objects
> through differents systems if we use SCIM to sync data between them.
> >>>
> >>> Say for example that we have school student register system where
> students, classes, study groups etc is managed. Say also that we extend
> SCIM with some resource types that define how to decribe time tables for
> the school.
> >>>
> >>> We have two services that need information from the register; a time
> table software (A) and an attendance software (B).
> >>>
> >>> The time table software (A) adds information about the time tables and
> uses reference ("value" & "$ref") to point out groups and users from the
> register for the time table entries. Of course, B needs this information.
> >>>
> >>> Simple setup:
> >>>
> >>>
> >>>  Register
> >>>
> >>>   |    |
> >>>   v    v
> >>>
> >>>   A -> B
> >>>
> >>>
> >>> Now, according to SCIM, the id property that may be sent from Register
> to A and B should be ignored by A & B. Instead, A & B should make up their
> own id:s and return them to the register. This means that A and B cannot
> communicate about these objects since they don't hava any common id to
> reference. The externalId in SCIM is just for filtering, it cannot be used
> in reference attributes.
> >>>
> >>> How is this supposed to work? I think that maybe I'm missing something
> here? Why can't the Register decide on the id for the object, it seems so
> much more logical?
> >>>
> >>> A similar question arises even in a simple setup with a register and a
> service provider. If the register first provisions a user in the service
> provider, it gets the service provider's made up ID in the respoonse. Then
> it creates a group and populate it with the user. Should it then use the id
> from the service provider in the "members": {"value" and "$ref" ? It seems
> a bit backward to me?
> >>>
> >>> We need the same "objects" to be synced between more than just two
> systems, and to me it is confusing how to keep the id:s intact. Is this not
> a job for SCIM at all?
> >>>
> >>> Please advice :-)
> >>>
> >>> Palle Girgensohn
> >>>
> >>>
> >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.
> ietf.org_mailman_listinfo_scim&d=DwICAg&c=RoP1YumCXCgaWHvlZYR8PQcxBKCX5Y
> TpkKY057SbK10&r=JBm5biRrKugCH0FkITSeGJxPEivzjW
> wlNKe4C_lLIGk&m=TdZHqR-ToFuh_fDFofMC8m9xMtSQi_LtTBpJmvsKWw8&s=mrJ6_
> FjoMs25rlq59xNH4oqI3ObIV-7Gy2R4hvH1x-8&e=
> >>>
> >>
> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.
> ietf.org_mailman_listinfo_scim&d=DwICAg&c=RoP1YumCXCgaWHvlZYR8PQcxBKCX5Y
> TpkKY057SbK10&r=JBm5biRrKugCH0FkITSeGJxPEivzjW
> wlNKe4C_lLIGk&m=TdZHqR-ToFuh_fDFofMC8m9xMtSQi_LtTBpJmvsKWw8&s=mrJ6_
> FjoMs25rlq59xNH4oqI3ObIV-7Gy2R4hvH1x-8&e=
> >

---

From: Palle Girgensohn <[email-masked]>
Date: Wed, 20 Sep 2017 12:16:59 +0200

Hi Samuel,

Yes, we are making good progress, thanks!

My take was to use the externalId and require that the internal id *should* be the same if the scim server needs to interact, using scim, with other service providers.

I get that for interoperability reasons, a custom id *could* be better, but I can't really see the difference from using the scim built in eternalId?

Palle

> 20 sep. 2017 kl. 10:29 skrev Samuel Erdtman <[email-masked]>:
> 
> Hi Palle!
> 
> Great to see you here on the list and that you are making progress with the SCIM work.
> 
> I would opt for the solution I think Phil is describing, i.e. having the register send an identifier (e.g. student identifier) in the create request and then have the time table software and the attendance software use that id as the id. I think it would be best to define a new attribute for this id and document that when present this should become the id in the returned object. In this way you would get okay interoperability, not awesome but okay.
> 
> Cheers
> //Samuel
> 
> 
> 
> 
> On Wed, Sep 20, 2017 at 9:23 AM, Palle Girgensohn <[email-masked]> wrote:

---

From: Palle Girgensohn <[email-masked]>
Date: Wed, 20 Sep 2017 13:06:30 +0200

Our scope it not a solution for a single school but a Swedish national standard for information exchange between systems within schools. Hence we shouldn't really step away from the technical rules in the scim standard, as it would be confusing.

In our general use case, the register would probably act both as a client and as as a (mostly read-only) scim server. Services that don't require their own storage could be implemented to just GET objects on demand and toss them once the user logs out.

But for now, lets focus on the use case where the student register is a scim client. Since it is still not crystal clear to me how this is supposed to work, lets try with an example.

The systems below are register.example.com, a.example.com and b.example.com.

First the register creates a new user:

register -> a

   POST /Users  HTTP/1.1
   Host: a.example.com
   Accept: application/scim+json
   Content-Type: application/scim+json
   Authorization: Bearer h480djs93hd8
   Content-Length: ...

   {
     "schemas":["urn:ietf:params:scim:schemas:core:2.0:User"],
     "userName":"bjensen",
     "externalId":"a7ff200e-8cca-4532-9e44-56721ee554e6",
     "name":{
       "formatted":"Ms. Barbara J Jensen III",
       "familyName":"Jensen",
       "givenName":"Barbara"
     }
   }

a responds:

   HTTP/1.1 201 Created
   Content-Type: application/scim+json
   Location:
    https://a.example.com/v2/Users/b24dbd91-a874-465f-b916-8c7003403bcb
   ETag: W/"e180ee84f0671b1"

   {
     "schemas":["urn:ietf:params:scim:schemas:core:2.0:User"],
     "id":"b24dbd91-a874-465f-b916-8c7003403bcb",
     "externalId":"a7ff200e-8cca-4532-9e44-56721ee554e6",
     "meta":{
       "resourceType":"User",
       "created":"2011-08-01T21:32:44.882Z",
       "lastModified":"2011-08-01T21:32:44.882Z",
       "location":
   "https://a.example.com/v2/Users/b24dbd91-a874-465f-b916-8c7003403bcb",
       "version":"W\/\"e180ee84f0671b1\""
     },
     "name":{
       "formatted":"Ms. Barbara J Jensen III",
       "familyName":"Jensen",
       "givenName":"Barbara"
     },
     "userName":"bjensen"
   }

and also the register creates the resource in b.example.com using the exact same call as above.

and b responds:

   HTTP/1.1 201 Created
   Content-Type: application/scim+json
   Location:
    https://b.example.com/v2/Users/713a3079-99dc-4295-9800-2d6bfcb66202
   ETag: W/"e180ee84f0671b1"

   {
     "schemas":["urn:ietf:params:scim:schemas:core:2.0:User"],
     "id":"713a3079-99dc-4295-9800-2d6bfcb66202",
     "externalId":"a7ff200e-8cca-4532-9e44-56721ee554e6",
     "meta":{
       "resourceType":"User",
       "created":"2011-08-01T21:32:44.882Z",
       "lastModified":"2011-08-01T21:32:44.882Z",
       "location":
   "https://b.example.com/v2/Users/713a3079-99dc-4295-9800-2d6bfcb66202",
       "version":"W\/\"e180ee84f0671b1\""
     },
     "name":{
       "formatted":"Ms. Barbara J Jensen III",
       "familyName":"Jensen",
       "givenName":"Barbara"
     },
     "userName":"bjensen"
   }

Now we have three stable id:s for the user, one for each system. All three also have a common stable "externalId".

Now the register creates a group:

register -> a & b

   POST /Groups  HTTP/1.1
   Host: a.example.com
   Accept: application/scim+json
   Content-Type: application/scim+json
   Authorization: Bearer h480djs93hd8
   Content-Length: ...

   {
     "schemas": ["urn:ietf:params:scim:schemas:core:2.0:Group"],
     "externalId": "e9e30dba-f08f-4109-8486-d5c6a331660a",
     "displayName": "Tour Guides",
     "members": [
       {
         "value": "a7ff200e-8cca-4532-9e44-56721ee554e6",
         "$ref":
   "https://register.example.com/v2/Users/a7ff200e-8cca-4532-9e44-56721ee554e6",
         "display": "Babs Jensen"
       }
     ]
   }

This does make sense if the client is also a scim server. It would be possible to search the register using scim, and the id:s and references would be the same. The user's $ref points to the *source*, that is the student register. In the more narrow use case where the register is only a client, the reference to the member Babs is not correct here, right? There is no correspondance to the externalId? a.example.com has never really seen this id before, so how can it understand that this is the Babs that was just created, unless we explicitally introduce a correlation between externalId and id?

How should a respond? Should it use it's own id:s for the user and the group, like this:

a -> register:

   HTTP/1.1 201 Created
   Content-Type: application/scim+json
   Location:
    https://a.example.com/v2/Groups/bfc4daf4-845b-42d8-89c9-d27b2fb87ad9
   ETag: W/"e180ee84f0671b1"

   {
     "schemas": ["urn:ietf:params:scim:schemas:core:2.0:Group"],
     "id": "bfc4daf4-845b-42d8-89c9-d27b2fb87ad9",
     "displayName": "Tour Guides",
     "members": [
       {
         "value": "b24dbd91-a874-465f-b916-8c7003403bcb",
         "$ref":
   "https://a.example.com/v2/Users/b24dbd91-a874-465f-b916-8c7003403bcb",
         "display": "Babs Jensen"
       }
     ],
     "meta": {
       "resourceType": "Group",
       "created": "2010-01-23T04:56:22Z",
       "lastModified": "2011-05-13T04:42:34Z",
       "version": "W\/\"3694e05e9dff592\"",
       "location":
   "https://a.example.com/v2/Groups/bfc4daf4-845b-42d8-89c9-d27b2fb87ad9"
     }
   }

In this case, it seems to me it that the server first has to ask the client what it means by member "a7ff200e-8cca-4532-9e44-56721ee554e6", since it has never seen it before. This becomes almost a catch-22.

Another way is to let the client create the group but references a:s id:s for referenced resources. This is how scim is supposed to work:

register -> a

   POST /Groups  HTTP/1.1
   Host: a.example.com
   Accept: application/scim+json
   Content-Type: application/scim+json
   Authorization: Bearer h480djs93hd8
   Content-Length: ...

   {
     "schemas": ["urn:ietf:params:scim:schemas:core:2.0:Group"],
     "externalId": "e9e30dba-f08f-4109-8486-d5c6a331660a",
     "displayName": "Tour Guides",
     "members": [
       {
         "value": "b24dbd91-a874-465f-b916-8c7003403bcb",
         "$ref":
   "https://a.example.com/v2/Users/b24dbd91-a874-465f-b916-8c7003403bcb",
         "display": "Babs Jensen"
       }
     ]
   }

But if the register is also a scim server, this is rather confusing.

I believe we could, for our profile of the scim standard, require externalId to bu uuid:s and to never change for any reason, and also require that the externalId and the "local" id should be treated as identical. This would require a scim server to always use the externalId as its own id, and we would be "home safe". Samuel just suggested a custom attribute instead of externalId. I can't really see how it would much in terms of interoperability, but I guess you could argue that it becomes more obvious what it is?

Thoughts?

Thanks,
Palle

> 20 sep. 2017 kl. 01:01 skrev Phil Hunt (IDM) <[email-masked]>:
>

---

From: Chris Phillips <[email-masked]>
Date: Wed, 20 Sep 2017 12:00:39 +0000

Palle,

When it comes to identifiers in Sweden's education sector I would
encourage you to take a look here:

https://wiki.swamid.se/display/SWAMID/Attribute+Profile

One suggestion is to be consistent with your region's educational sector
data dictionary which appears to use eduPersonPrincipalName(ePPN) as an
identifier. This appears to be permissible to link across systems but
check with SWAMID about it.

Storing ePPN in externalId may be your easiest way unless you want to
extend the schema for the eduPerson schema which may be beneficial if you
are looking at circulating educationally focused attributes.

eduPerson schema details:
https://www.internet2.edu/products-services/trust-identity/eduperson-eduorg
/

I see norEduPersonNIN as another identifier but classified as sensitive.
While it looks really useful, I encourage you to take care as it would be
PII and may have other constraints around it, possibly legal implications
in your region. 

Hope this helps..

C

P.s. Quick googling on SCIM and eduperson shows some work be PennState in
the US: https://github.com/PennState/SCIMple-Identity
And some group management SCIM work by Internet2 as well:
https://spaces.internet2.edu/display/Grouper/Grouper+TIER+SCIM+server that
may be useful to take a glance at to see if there's previous work to
leverage.

On 2017-09-20, 7:06 AM, "scim on behalf of Palle Girgensohn"
<[email-masked] on behalf of [email-masked]> wrote:

>Our scope it not a solution for a single school but a Swedish national
>standard for information exchange between systems within schools. Hence
>we shouldn't really step away from the technical rules in the scim
>standard, as it would be confusing.
>
>In our general use case, the register would probably act both as a client
>and as as a (mostly read-only) scim server. Services that don't require
>their own storage could be implemented to just GET objects on demand and
>toss them once the user logs out.
>
>But for now, lets focus on the use case where the student register is a
>scim client. Since it is still not crystal clear to me how this is
>supposed to work, lets try with an example.
>
>Now we have three stable id:s for the user, one for each system. All
>three also have a common stable "externalId".
>
>This does make sense if the client is also a scim server. It would be
>possible to search the register using scim, and the id:s and references
>would be the same. The user's $ref points to the *source*, that is the
>student register. In the more narrow use case where the register is only
>a client, the reference to the member Babs is not correct here, right?
>There is no correspondance to the externalId? a.example.com has never
>really seen this id before, so how can it understand that this is the
>Babs that was just created, unless we explicitally introduce a
>correlation between externalId and id?
>
>
>In this case, it seems to me it that the server first has to ask the
>client what it means by member "a7ff200e-8cca-4532-9e44-56721ee554e6",
>since it has never seen it before. This becomes almost a catch-22.
>
>
>Another way is to let the client create the group but references a:s id:s
>for referenced resources. This is how scim is supposed to work:
>
>
>register -> a
>
>   POST /Groups  HTTP/1.1
>   Host: a.example.com
>I believe we could, for our profile of the scim standard, require
>externalId to bu uuid:s and to never change for any reason, and also
>require that the externalId and the "local" id should be treated as
>identical. This would require a scim server to always use the externalId
>as its own id, and we would be "home safe". Samuel just suggested a
>custom attribute instead of externalId. I can't really see how it would
>much in terms of interoperability, but I guess you could argue that it
>becomes more obvious what it is?
>
>Thoughts?
>
>Thanks,
>Palle
>
>
>> The id in scim is a local instance identifier. Its purpose is to give
>>permanence to the id. Ie. It never changes which gives referential
>>integrity. The timetable system can count on the uri for the student.
>> 
>> Technically the client cannot assign the id, but the server can choose
>>to use an attribute provided (eg the studentid) for the id value.  The
>>challenge is not to choose an id value that might change for any reason.
>>It seems to me student ids are stable.
>> 
>> The point is if you were accessing a student in another school you
>>can't count on id being the student id. Nor that you will be able to
>>keep them the same.
>> 
>>>>students, classes, study groups etc is managed. Say also that we
>>>>extend SCIM with some resource types that define how to decribe time
>>>>tables for the school.
>>>> 
>>>> The time table software (A) adds information about the time tables
>>>>and uses reference ("value" & "$ref") to point out groups and users
>>>>from the register for the time table entries. Of course, B needs this
>>>>information.
>>>> 
>>>> Simple setup:
>>>> 
>>>> 
>>>>  Register
>>>> 
>>>>   |    |
>>>>   v    v
>>>> 
>>>>   A -> B
>>>> 
>>>> 
>>>> Now, according to SCIM, the id property that may be sent from
>>>>Register to A and B should be ignored by A & B. Instead, A & B should
>>>>make up their own id:s and return them to the register. This means
>>>>that A and B cannot communicate about these objects since they don't
>>>>hava any common id to reference. The externalId in SCIM is just for
>>>>filtering, it cannot be used in reference attributes.
>>>> 
>>>> How is this supposed to work? I think that maybe I'm missing
>>>>something here? Why can't the Register decide on the id for the
>>>>object, it seems so much more logical?
>>>> 
>>>> A similar question arises even in a simple setup with a register and
>>>>a service provider. If the register first provisions a user in the
>>>>service provider, it gets the service provider's made up ID in the
>>>>respoonse. Then it creates a group and populate it with the user.
>>>>Should it then use the id from the service provider in the "members":
>>>>{"value" and "$ref" ? It seems a bit backward to me?
>>>> 
>>>>systems, and to me it is confusing how to keep the id:s intact. Is
>>>>this not a job for SCIM at all?
>>>> 
>>>> Please advice :-)
>>>> 
>>>> Palle Girgensohn
>>>> 
>>>> 
>>>>https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailm
>>>>an_listinfo_scim&d=DwICAg&c=RoP1YumCXCgaWHvlZYR8PQcxBKCX5YTpkKY057SbK10
>>>>&r=JBm5biRrKugCH0FkITSeGJxPEivzjWwlNKe4C_lLIGk&m=TdZHqR-ToFuh_fDFofMC8m
>>>>9xMtSQi_LtTBpJmvsKWw8&s=mrJ6_FjoMs25rlq59xNH4oqI3ObIV-7Gy2R4hvH1x-8&e=
>>>> 
>>> 
>>>https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailma
>>>n_listinfo_scim&d=DwICAg&c=RoP1YumCXCgaWHvlZYR8PQcxBKCX5YTpkKY057SbK10&r
>>>=JBm5biRrKugCH0FkITSeGJxPEivzjWwlNKe4C_lLIGk&m=TdZHqR-ToFuh_fDFofMC8m9xM
>>>tSQi_LtTBpJmvsKWw8&s=mrJ6_FjoMs25rlq59xNH4oqI3ObIV-7Gy2R4hvH1x-8&e=
>>

---

From: Palle Girgensohn <[email-masked]>
Date: Wed, 20 Sep 2017 14:16:17 +0200

Hi Chris,

Thanks for your reply. I am well aware of Swamid's work as I do a lot of work with higher education as well.

eppn in the externalId is definitely a good choice. A UUID is another alternative. We could really allow both, as long as it is something universally unique. eppn:s are sometimes reused in schools, but this is of course only due to bad internal processes. There's also the risk that they change when a user changes surname, but this should really not be allowed either. We need the eppn anyway to interoperate with SAML Identity Providers. Our thought is to require eppn in the username attribute.

norEduPersonNIN is sadly not a stable identifier in schools, mainly since refugees have the right to attend school from day one, but the will not get a norEduPersonNIN until months later. And, as you say, it is too sensitive.

> 20 sep. 2017 kl. 14:00 skrev Chris Phillips <[email-masked]>:
> 
> Palle,
>