Subject: [scim] SCIM Attribute Validation

From: Shelley <[email-masked]>
Date: Tue, 21 Oct 2014 14:36:28 -0500

The SCIM specification expects service providers should be able to
canonicalize provided data into a common format (e.g. RFC 3966 for
phoneNumbers), yet it does not explicitly describe input requirements or
validation for attributes such as these.* Is it expected that values that
cannot accurately be canonicalized are rejected by the SCIM service or that
the SCIM service accepts this invalid data and merely persists and returns
this bad data to consumers?*

As a SCIM Service Provider, we are currently performing validation on
attributes such as phoneNumbers and emails to ensure that they contain
well-formed, valid values, and returning a 400 with a description of the
error if validation fails. One of the benefits of SCIM is that data may
come from multiple sources and may be consumed by multiple clients, and
SCIM provides a repository of structured and well-formed data so that
consumers can reliably derive meaning and use from this data. Without
attribute validation and a guarantee of canonicalized values, this goal
does not seem feasible and puts the burden on consumers to make sense of
potentially invalid data.

The downside to attribute validation is that it requires the SCIM consumers
sending data to ensure valid data as well. As these consumers are closest
to the source data, though, this seems like the most reasonable approach
rather than putting the burden on the service provider and/or other SCIM
consumers to make sense of the data. Aside from just "bad data" (e.g.
sending "not a phone number" or "000-0000" for a phone number), for
example, if the SCIM consumer sending source data sends phone numbers
without area codes or sends only internal extensions, another SCIM consumer
may not be able to make sense of this data. It seems reasonable to require
the consumer sending the data to ensure that this data is well-formed and
canonicalized/canonicalizable.

A *good *example in the SCIM specification is the country attribute, which
clearly indicates the attribute's requirements. A *bad *example is the
phoneNumbers attribute, which doesn't describe the expected format to be
provided by consumers, yet recommends that the service provider
canonicalize the value even though the input may not be able to be
canonicalized.

I'd be interested in hearing others' opinions on this matter, and whether
the SCIM specification should clarify further restrictions on attribute
values.

---

From: Alex Redston <[email-masked]>
Date: Tue, 21 Oct 2014 20:53:54 +0100

Of course the challenge here comes down to the extent to which that 
particular data can be standardised and validated. Taking phone numbers 
as the example you have sip calls, invalid numbers, country specific 
shortcodes, this is potentially a minefield.

Well formed and valid values are two distinct concepts and with multiple 
carrier technologies the anatomy of simple +4412345678901 style numbers 
is no longer fit for purpose. For this reason I do not think that it is 
feasible for this particular data type to be rejected as bad data 
because the only way to know if a phone number is valid is to dial it.

What do you think?

Alex

Alex Redston

CEO
Switch Identity Governance Ltd
07973 320 795
020 3434 3888

On 21/10/2014 20:36, Shelley wrote:
> phoneNumbers), yet it does not explicitly describe input requirements 
> or validation for attributes such as these.*Is it expected that values 
> that cannot accurately be canonicalized are rejected by the SCIM 
> service or that the SCIM service accepts this invalid data and merely 
> persists and returns this bad data to consumers?*
>
> well-formed, valid values, and returning a 400 with a description of 
> the error if validation fails. One of the benefits of SCIM is that 
> data may come from multiple sources and may be consumed by multiple 
> clients, and SCIM provides a repository of structured and well-formed 
> data so that consumers can reliably derive meaning and use from this 
> data. Without attribute validation and a guarantee of canonicalized 
> values, this goal does not seem feasible and puts the burden on 
> consumers to make sense of potentially invalid data.
>
> The downside to attribute validation is that it requires the SCIM 
> consumers sending data to ensure valid data as well. As these 
> consumers are closest to the source data, though, this seems like the 
> most reasonable approach rather than putting the burden on the service 
> provider and/or other SCIM consumers to make sense of the data. Aside 
> from just "bad data" (e.g. sending "not a phone number" or "000-0000" 
> for a phone number), for example, if the SCIM consumer sending source 
> data sends phone numbers without area codes or sends only internal 
> extensions, another SCIM consumer may not be able to make sense of 
> this data. It seems reasonable to require the consumer sending the 
> data to ensure that this data is well-formed and 
> A /good /example in the SCIM specification is the country attribute, 
> which clearly indicates the attribute's requirements. A /bad /example 
> is the phoneNumbers attribute, which doesn't describe the expected 
> format to be provided by consumers, yet recommends that the service 
> provider canonicalize the value even though the input may not be able 
> to be canonicalized.
>
> I'd be interested in hearing others' opinions on this matter, and 
> whether the SCIM specification should clarify further restrictions on 
> attribute values.

---
This email is free from viruses and malware because avast! Antivirus protection is active.
http://www.avast.com

---

From: Shelley <[email-masked]>
Date: Wed, 22 Oct 2014 09:50:00 -0500

The simplest approach would be to require SCIM consumers to provide phone
numbers in the RFC 3966 format, which at least helps to guarantee the
validity of the format and canonicalizes it into a standard global format
consumable by both external systems and users. This also accounts for some
of the scenarios you mentioned, such as sip calls.

As far as validating whether the phone number is actually a potentially
valid number (based on country, area code, etc.), this can be discussed
further/separately as well. I agree that validating whether a phone number
is actually in service is probably not feasible but there are some
standards and libraries that *could *potentially be used to deter
completely bogus numbers [1], but the value in this may be debatable.

The overall problem is just that without centralized data validation in the
service, SCIM consumers cannot rely on the syntax/validity of the data and
therefore the burden of validating and normalizing falls to them.

[1] https://code.google.com/p/libphonenumber/

On Tue, Oct 21, 2014 at 2:53 PM, Alex Redston <[email-masked]>
wrote:

> particular data can be standardised and validated. Taking phone numbers as
> the example you have sip calls, invalid numbers, country specific
> carrier technologies the anatomy of simple +4412345678901 style numbers is
> no longer fit for purpose. For this reason I do not think that it is
> feasible for this particular data type to be rejected as bad data because
> the only way to know if a phone number is valid is to dial it.
>
> What do you think?
>
> Alex
>
> Alex Redston
>
> CEO
> consumers sending data to ensure valid data as well. As these consumers are
> closest to the source data, though, this seems like the most reasonable
> approach rather than putting the burden on the service provider and/or
> other SCIM consumers to make sense of the data. Aside from just "bad data"
> (e.g. sending "not a phone number" or "000-0000" for a phone number), for
> A *good *example in the SCIM specification is the country attribute,
> which clearly indicates the attribute's requirements. A *bad *example is
> the phoneNumbers attribute, which doesn't describe the expected format to
> be provided by consumers, yet recommends that the service provider
> ------------------------------
>    <http://www.avast.com/>
>
> This email is free from viruses and malware because avast! Antivirus
> <http://www.avast.com/> protection is active.
>
>

---

From: Shelley <[email-masked]>
Date: Wed, 22 Oct 2014 10:41:44 -0500

I'd also like to clarify/highlight this problem/challenge with the current
SCIM specification - the spec recommends that service providers
canonicalize phone numbers to RFC 3966, yet, if there are no requirements
for the data provided by the consumer, this may be impossible.

In the extreme scenario, how can a SCIM SP canonicalize "not a phone
number" to RFC 3966? In an incomplete scenario, how can an SP canonicalize
"222-2222" without a country/area code? And in an ambiguous scenario, how
can an SP canonicalize "201-5399 123" (where 123 is assumed to be an
extension)?

While it may be possible for the SP to canonicalize *some *values under
some *assumptions *(e.g. country code may be assumed, and assumptions based
on patterns, etc), it is still possible that the canonicalization would be
incorrect and in some cases, impossible.

How are other SPs handling this? Is anyone doing validation or
canonicalization?

Likewise, how are SCIM clients consuming phone numbers (especially those
that may be read-only consumers of data and don't have visibility/control
over the source data)? Is anyone doing any normalizing, validating,
canonicalizing, or just leaving the data as-is regardless of the
format/validity?

On Wed, Oct 22, 2014 at 10:05 AM, Alex Redston <[email-masked]>
wrote:

>  Thanks - interesting link.
>
> Alex Redston
>
> CEO
> On 22/10/2014 15:50, Shelley wrote:
>
>  The simplest approach would be to require SCIM consumers to provide
> phone numbers in the RFC 3966 format, which at least helps to guarantee the
>  The overall problem is just that without centralized data validation in
> the service, SCIM consumers cannot rely on the syntax/validity of the data
> and therefore the burden of validating and normalizing falls to them.
>
> On Tue, Oct 21, 2014 at 2:53 PM, Alex Redston <[email-masked]
> > wrote:
>
>> validation for attributes such as these.* Is it expected that values
>> that cannot accurately be canonicalized are rejected by the SCIM service or
>> that the SCIM service accepts this invalid data and merely persists and
>> returns this bad data to consumers?*
>>
>> the phoneNumbers attribute, which doesn't describe the expected format
>> to be provided by consumers, yet recommends that the service provider

---

From: Phil Hunt <[email-masked]>
Date: Wed, 22 Oct 2014 09:39:56 -0700

Shelley,

Thanks for your comments and bringing this to the groups attention.

Overall, I would like to strike a balance in favour of being flexible in what is accepted.  If the SP can understand the value, it may accept it. Further it may modify the value to conform to RFC3966 and return the modified value in the HTTP response. For example a service provider should feel free to correct something in telephone-subscriber format to be returned as a URI instead.
—> maybe Postel’s Law is one of the general principles of SCIM that we should make more clearer?

I think one of the problems with 3966 is that it seems quite broadly defined. Enforcement may be quite valid. But will it improve quality of information given the complication it will cause for clients (e.g. when a request fails due to format, what is the corrective action)?  Many service providers would rather accept the POST and simply ignore the value than to loose the transaction.

I agree we should adjust the language a bit. For example, are we expecting a telephone-uri or a telephone-subscriber as the value?  The spec is not clear.  

There are also some other parts to 3966 which duplicate what we are doing in multi-valued attributes. For example the ABNF “parameter” enables the following example: 
  tel:7042;phone-context=example.com
In this case, “phone-context” could be seen as duplicative of the function of “type”.

Maybe it is not only that we want to following RFC3966, but we actually want to fully specify how 3966 is mapped to phoneNumber’s multi-valued JSON structure?

Phil

@independentid
www.independentid.com
[email-masked]

On Oct 22, 2014, at 8:41 AM, Shelley <[email-masked]> wrote:

> I'd also like to clarify/highlight this problem/challenge with the current SCIM specification - the spec recommends that service providers canonicalize phone numbers to RFC 3966, yet, if there are no requirements for the data provided by the consumer, this may be impossible.
> 
> In the extreme scenario, how can a SCIM SP canonicalize "not a phone number" to RFC 3966? In an incomplete scenario, how can an SP canonicalize "222-2222" without a country/area code? And in an ambiguous scenario, how can an SP canonicalize "201-5399 123" (where 123 is assumed to be an extension)?
> 
> While it may be possible for the SP to canonicalize some values under some assumptions (e.g. country code may be assumed, and assumptions based on patterns, etc), it is still possible that the canonicalization would be incorrect and in some cases, impossible.
> 
> How are other SPs handling this? Is anyone doing validation or canonicalization? 
> 
> Likewise, how are SCIM clients consuming phone numbers (especially those that may be read-only consumers of data and don't have visibility/control over the source data)? Is anyone doing any normalizing, validating, canonicalizing, or just leaving the data as-is regardless of the format/validity?
> 
> 
> On Wed, Oct 22, 2014 at 10:05 AM, Alex Redston <[email-masked]> wrote:
>> The simplest approach would be to require SCIM consumers to provide phone numbers in the RFC 3966 format, which at least helps to guarantee the validity of the format and canonicalizes it into a standard global format consumable by both external systems and users. This also accounts for some of the scenarios you mentioned, such as sip calls.
>> 
>> As far as validating whether the phone number is actually a potentially valid number (based on country, area code, etc.), this can be discussed further/separately as well. I agree that validating whether a phone number is actually in service is probably not feasible but there are some standards and libraries that could potentially be used to deter completely bogus numbers [1], but the value in this may be debatable.
>> 
>> The overall problem is just that without centralized data validation in the service, SCIM consumers cannot rely on the syntax/validity of the data and therefore the burden of validating and normalizing falls to them.
>> 
>> On Tue, Oct 21, 2014 at 2:53 PM, Alex Redston <[email-masked]> wrote:
>> Of course the challenge here comes down to the extent to which that particular data can be standardised and validated. Taking phone numbers as the example you have sip calls, invalid numbers, country specific shortcodes, this is potentially a minefield.
>> 
>> Well formed and valid values are two distinct concepts and with multiple carrier technologies the anatomy of simple +4412345678901 style numbers is no longer fit for purpose. For this reason I do not think that it is feasible for this particular data type to be rejected as bad data because the only way to know if a phone number is valid is to dial it.
>> 
>> What do you think?
>> 
>> Alex
>> Alex Redston
>> 
>> CEO
>>> The SCIM specification expects service providers should be able to canonicalize provided data into a common format (e.g. RFC 3966 for phoneNumbers), yet it does not explicitly describe input requirements or validation for attributes such as these. Is it expected that values that cannot accurately be canonicalized are rejected by the SCIM service or that the SCIM service accepts this invalid data and merely persists and returns this bad data to consumers?
>>> 
>>> As a SCIM Service Provider, we are currently performing validation on attributes such as phoneNumbers and emails to ensure that they contain well-formed, valid values, and returning a 400 with a description of the error if validation fails. One of the benefits of SCIM is that data may come from multiple sources and may be consumed by multiple clients, and SCIM provides a repository of structured and well-formed data so that consumers can reliably derive meaning and use from this data. Without attribute validation and a guarantee of canonicalized values, this goal does not seem feasible and puts the burden on consumers to make sense of potentially invalid data.
>>> 
>>> The downside to attribute validation is that it requires the SCIM consumers sending data to ensure valid data as well. As these consumers are closest to the source data, though, this seems like the most reasonable approach rather than putting the burden on the service provider and/or other SCIM consumers to make sense of the data. Aside from just "bad data" (e.g. sending "not a phone number" or "000-0000" for a phone number), for example, if the SCIM consumer sending source data sends phone numbers without area codes or sends only internal extensions, another SCIM consumer may not be able to make sense of this data. It seems reasonable to require the consumer sending the data to ensure that this data is well-formed and canonicalized/canonicalizable.
>>> 
>>> A good example in the SCIM specification is the country attribute, which clearly indicates the attribute's requirements. A bad example is the phoneNumbers attribute, which doesn't describe the expected format to be provided by consumers, yet recommends that the service provider canonicalize the value even though the input may not be able to be canonicalized.
>>> 
>>> I'd be interested in hearing others' opinions on this matter, and whether the SCIM specification should clarify further restrictions on attribute values.
>> 
>> 
>> 
>>  	
> _______________________________________________
> scim mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/scim

---

From: Shelley <[email-masked]>
Date: Wed, 22 Oct 2014 11:44:10 -0500

While I'd prefer to put more requirements on the data provided by SCIM
clients given all the aforementioned concerns, here is a potential
temporary mitigation/compromise
given the *current *state of the spec:

   - Strongly recommend SCIM clients provide data in specific formats (e.g.
   RFC3966 phone numbers)
   - SCIM SP performs no validation, accepts any values
   - SCIM SP attempts to canonicalize the value in the response
      - Use the "value" sub-attribute to return the raw value
      - Use the "display" sub-attribute to return the canonicalized value,
      if canonicalization is possible; otherwise, return the raw value

This helps limit the burden on SCIM consumers of the data, though they may
still have to deal with bad values and inconsistent formats. The use of the
display and value sub-attributes helps bring better visibility to cases
where the SP may have canonicalized a value using different assumptions.
(For example, if a country code is assumed, or in the case of "201-5399
123" where 123 is assumed to be an extension, but the SCIM SP assumes this
is a 10-digit phone number.)

Is this usage of the display and value sub-attributes consistent with the
intended usage in SCIM?

 "phoneNumbers": [
    {
      "value": "555-555-5555",
      "display": "tel:+1-555-555-5555",
      "type": "work"
    },
    {
      "value": "tel:+1-555-555-5555",
      "display": "tel:+1-555-555-5555",
      "type": "work"
    },
    {
      "value": "not a phone number",
      "display": "not a phone number",
      "type": "mobile"
    },
    {
      "value": "555-5555",
      "display": "555-5555",
      "type": "mobile"
    },
    {
      "value": "555-5555 123",
      "display": "tel:+1-555-555-5123",
      "type": "mobile"
    }
  ]

On Wed, Oct 22, 2014 at 10:41 AM, Shelley <[email-masked]> wrote:

> some *assumptions *(e.g. country code may be assumed, and assumptions
> based on patterns, etc), it is still possible that the canonicalization
> would be incorrect and in some cases, impossible.
>
>>> or validation for attributes such as these.* Is it expected that values
>>> which clearly indicates the attribute's requirements. A *bad *example
>>> provider canonicalize the value even though the input may not be able to be
>>> canonicalized.
>>>

---

From: Shelley <[email-masked]>
Date: Wed, 22 Oct 2014 12:22:35 -0500

Thanks for the response, Phil.

 If the SP can understand the value, it may accept it.

Right, but what values can/should SPs attempt to "understand" since these
aren't described in the spec? What assumptions, if any, can be made about
the format provided by consumers? Further, what happens if the SP doesn't
"accept" the value: 400 response? value is silently ignored? value is
stored, but not canonicalized?

Further it may modify the value to conform to RFC3966 and return the
> modified value in the HTTP response.

The problem here is that modifying the value is not always possible or may
be done incorrectly depending on the format provided by the consumer.

Many service providers would rather accept the POST and simply ignore the
> value than to loose the transaction.
>

Silently ignoring unknown values seems a bit dangerous as well. I agree
that the corrective action may be difficult to determine automatically if
the transaction fails fast, but at least it brings some form of visibility
to the problem.

On Wed, Oct 22, 2014 at 11:39 AM, Phil Hunt <[email-masked]> wrote:

> Overall, I would like to strike a balance in favour of being flexible in
> what is accepted.  If the SP can understand the value, it may accept it.
> Further it may modify the value to conform to RFC3966 and return the
> modified value in the HTTP response. For example a service provider
> should feel free to correct something in telephone-subscriber format to be
> returned as a URI instead.
> I think one of the problems with 3966 is that it seems quite broadly
> defined. Enforcement may be quite valid. But will it improve quality of
> information given the complication it will cause for clients (e.g. when a
> request fails due to format, what is the corrective action)?  Many service
> providers would rather accept the POST and simply ignore the value than to
> loose the transaction.
>
> I agree we should adjust the language a bit. For example, are we expecting
> a telephone-uri or a telephone-subscriber as the value?  The spec is not
> clear.
>
> There are also some other parts to 3966 which duplicate what we are doing
> in multi-valued attributes. For example the ABNF “parameter” enables the
> following example:
> Maybe it is not only that we want to following RFC3966, but we actually
> want to fully specify how 3966 is mapped to phoneNumber’s multi-valued JSON
> structure?
>
> Phil
>
> @independentid
> www.independentid.com
> [email-masked]
>
>
>

---

From: Phil Hunt <[email-masked]>
Date: Wed, 22 Oct 2014 10:37:04 -0700

Shelley,

I think we are pretty much in agreement. Would you mind providing some new text for the definition of phoneNumbers?  I like where you are going with value vs. displayName etc.

It would be good to see how the group feels on a some new proposed text.

Thanks,

Phil

@independentid
www.independentid.com
[email-masked]

On Oct 22, 2014, at 10:22 AM, Shelley <[email-masked]> wrote:

> Right, but what values can/should SPs attempt to "understand" since these aren't described in the spec? What assumptions, if any, can be made about the format provided by consumers? Further, what happens if the SP doesn't "accept" the value: 400 response? value is silently ignored? value is stored, but not canonicalized?
> 
> Further it may modify the value to conform to RFC3966 and return the modified value in the HTTP response.
> 
> The problem here is that modifying the value is not always possible or may be done incorrectly depending on the format provided by the consumer.
> 
> Many service providers would rather accept the POST and simply ignore the value than to loose the transaction.
> 
> Silently ignoring unknown values seems a bit dangerous as well. I agree that the corrective action may be difficult to determine automatically if the transaction fails fast, but at least it brings some form of visibility to the problem.
>  
>

---

From: Grahame Grieve <[email-masked]>
Date: Thu, 23 Oct 2014 05:11:17 +1100

In healthcare, we defined some global standards that required all telephone
numbers to be in rfc3966 format

In practice, implementers have not conformed to this - they simply ignore
that part of the specification. Not only do they have vast amounts of
existing non-conformant data that they can't rectify automatically, there
is very little value in it - telephone numbers are almost exclusively used
by people to communicate with people, and conforming to rfc 3966 is
actually a negative value.

Yes, there are some limitations to this - systems that have integrated
pabxs have the load of doing the data validation on the fly. But if they're
using a telephone, they almost always have a person to mediate that process.

Mobile phone numbers, and email addresses, on the other hand, the use case
for valid data is extremely clear, and the validation is already generally
performed for this reason. So I suggest that a distinction be made in this
regard in the spec - through this is definitely a policy issue, not a
conformance one. Still, the use cases are sufficiently different that I
argue it's worth mentioning

Grahame

On Thursday, October 23, 2014, Shelley <[email-masked]> wrote:

> On Wed, Oct 22, 2014 at 11:39 AM, Phil Hunt <[email-masked]
> <javascript:_e(%7B%7D,'cvml',[email-masked]');>> wrote:
>
>> Shelley,
>>
>> I agree we should adjust the language a bit. For example, are we
>> expecting a telephone-uri or a telephone-subscriber as the value?  The spec
>> is not clear.
>>
>> <javascript:_e(%7B%7D,'cvml',[email-masked]');>
>>
>>
>>
>> On Oct 22, 2014, at 8:41 AM, Shelley <[email-masked]
>> <javascript:_e(%7B%7D,'cvml',[email-masked]');>> wrote:
>>
>> I'd also like to clarify/highlight this problem/challenge with the
>> current SCIM specification - the spec recommends that service providers
>> On Wed, Oct 22, 2014 at 10:05 AM, Alex Redston <
>> [email-masked]
>> <javascript:_e(%7B%7D,'cvml',[email-masked]');>> wrote:
>>
>>>  As far as validating whether the phone number is actually a
>>> potentially valid number (based on country, area code, etc.), this can be
>>> discussed further/separately as well. I agree that validating whether a
>>> phone number is actually in service is probably not feasible but there are
>>> some standards and libraries that *could *potentially be used to deter
>>>  The overall problem is just that without centralized data validation
>>> in the service, SCIM consumers cannot rely on the syntax/validity of the
>>> data and therefore the burden of validating and normalizing falls to them.
>>>
>>> <javascript:_e(%7B%7D,'cvml',[email-masked]');>> wrote:
>>>
>>>> Well formed and valid values are two distinct concepts and with
>>>> multiple carrier technologies the anatomy of simple +4412345678901 style
>>>> numbers is no longer fit for purpose. For this reason I do not think that
>>>> it is feasible for this particular data type to be rejected as bad data
>>>> or validation for attributes such as these.* Is it expected that
>>>> values that cannot accurately be canonicalized are rejected by the SCIM
>> [email-masked] <javascript:_e(%7B%7D,'cvml',[email-masked]');>

-- 
-----
http://www.healthintersections.com.au / [email-masked]
/ +61 411 867 065

---

From: Shelley <[email-masked]>
Date: Wed, 22 Oct 2014 14:13:50 -0500

Before any specific text is proposed, I'd like to get a general
understanding for what the agreement/direction is. It seems there are still
some major open questions that I'd appreciate some input on.

   - Should consumers be required to provide values in specific formats?
      - For example, for phone numbers, must consumers provide RFC 3966
      format, an international format, any of multiple common/pre-defined
      formats, or is any value allowed? If not in RFC/international format, is
      the country/area code assumed?
   - Should service providers perform attribute validation or accept any
   values?
      - If service providers perform validation, what is the expected
      behavior when validation fails? (400 / ignore the value / store the value
      as-is?)
   - Particularly if there are no requirements for data input and/or no
   validation is performed:
      - What are the assumptions that service providers can make during
      canonicalization?
      - What should service providers / consumers do if canonicalization is
      not possible?
      - Should the use of both the "value" and "display" attributes be
      recommended?

There are minimally two major variations of possible changes on the table,
with various flavors in between:

   - Require consumers to provide values in specified formats and SPs
   perform validation, rejecting invalid values
      - SCIM serves as a repository of well-formed, valid data. This
      results in the most usable data for SCIM consumers *reading *data,
      but puts some burden on SCIM consumers *providing *the data to
      conform.

*-OR-*

   - Allow consumers to provide values in any format, SP performs no
   validation though it may *attempt *perform canonicalization
   - SCIM serves as a repository of data from third parties, data may not
      be consistent, valid, or normalized. This results in the least effort for
      SCIM consumers *providing *data, but puts some burden on SCIM
      consumers *reading *the data to make sense of the data.
      - SCIM SPs may *attempt *to normalize some of the data, but there may
      be no guarantees with respect to the consistency or accuracy of these
      normalizations.

On Wed, Oct 22, 2014 at 12:37 PM, Phil Hunt <[email-masked]> wrote:

> Shelley,
>
> I think we are pretty much in agreement. Would you mind providing some new
> text for the definition of phoneNumbers?  I like where you are going with
> value vs. displayName etc.
>
>> [email-masked]> wrote:
>>

---

From: Shelley <[email-masked]>
Date: Wed, 22 Oct 2014 14:14:56 -0500

Thanks for your input, Grahame. To clarify, it sounds like you are in favor
of the following SCIM SP behavior? Please correct me if I've misunderstood.

   - Performing validation on email addresses
   - Performing validation on mobile phone numbers
   - Not performing validation on other phone numbers
   - Not canonicalizing phone numbers??

Are you currently implementing this approach in a SCIM implementation? If
you are in favor of performing validation on mobile numbers, would you
recommend/require the use of a certain format? What happens when validation
fails?

Personally, I don't know that we should be differentiating required formats
or validation based on the attribute "type" (i.e. treating mobile numbers
differently than other numbers). The use cases for these may often be the
same, and further, some numbers may not even specify a type, so drawing a
line here would be ambiguous.

On Wed, Oct 22, 2014 at 1:11 PM, Grahame Grieve <
[email-masked]> wrote:

> In healthcare, we defined some global standards that required all
> telephone numbers to be in rfc3966 format
>
>> The problem here is that modifying the value is not always possible or
>> may be done incorrectly depending on the format provided by the consumer.
>>
>>> There are also some other parts to 3966 which duplicate what we are
>>> doing in multi-valued attributes. For example the ABNF “parameter” enables
>>> the following example:
>>>>> attributes such as phoneNumbers and emails to ensure that they
>>>>> contain well-formed, valid values, and returning a 400 with a description
>>>>> of the error if validation fails. One of the benefits of SCIM is that data
>>>>> may come from multiple sources and may be consumed by multiple clients, and

---

From: Grahame Grieve <[email-masked]>
Date: Thu, 23 Oct 2014 06:42:14 +1100

>
> Thanks for your input, Grahame. To clarify, it sounds like you are in
> favor of the following SCIM SP behavior? Please correct me if I've
> misunderstood.
>

I'm in favor of it, but not in favor of requiring it - there are too many
incompatible business processes out there, and it harms adoption and
outcomes if you require changes to business practice in these things

> Are you currently implementing this approach in a SCIM implementation?
>
no. I have done that approach in an enterprise information system, but not
a SCIM implementation. My SCIM implementation doesn't validate any
addresses. but since my SCIM implementation lives in prototype land, I
wouldn't think that my implementing validation or not would teach anything.

> If you are in favor of performing validation on mobile numbers, would you
if we require anything, it would make sense to require RFC 3966. It would
be possible for clients with a known nation to remediate prefixes if people
wanted (e.g. switch out +1 in USA for display)

> Personally, I don't know that we should be differentiating required
> formats or validation based on the attribute "type" (i.e. treating mobile
> numbers differently than other numbers). The use cases for these may often
> be the same, and further, some numbers may not even specify a type, so
> drawing a line here would be ambiguous.
>
well, no type would mean no validation. But I wasn't recommending that the
spec require validation. I think it should
(a) say that the recommended canonical representation is RFC 3966
(b) say that validation is not required, but may be implemented, and is
likely to be so more often for mobile phone numbers and email addresses
(c) say that if values fail validation, the error code should be 422. In my
health RESTful spec, we define a structured error format to be returned in
cases like this, SCIM could consider this, but this is borderline. (We have
compelling cases in other contexts where handling errors correctly is a
life/death matter, and inherit that in these simple cases)
(d) define a way to describe what validation occurs in the schema

Grahame

>> Mobile phone numbers, and email addresses, on the other hand, the use
>> case for valid data is extremely clear, and the validation is
>> already generally performed for this reason. So I suggest that a
>> distinction be made in this regard in the spec - through this is definitely
>> a policy issue, not a conformance one. Still, the use cases are
>> sufficiently different that I argue it's worth mentioning
>>
>> Grahame
>>
>>
>>> Right, but what values can/should SPs attempt to "understand" since
>>> these aren't described in the spec? What assumptions, if any, can be made
>>> about the format provided by consumers? Further, what happens if the SP
>>> doesn't "accept" the value: 400 response? value is silently ignored? value
>>> is stored, but not canonicalized?
>>>
>>>> Overall, I would like to strike a balance in favour of being flexible
>>>> in what is accepted.  If the SP can understand the value, it may accept it.
>>>> While it may be possible for the SP to canonicalize *some *values
>>>> under some *assumptions *(e.g. country code may be assumed, and
>>>> assumptions based on patterns, etc), it is still possible that the
>>>> canonicalization would be incorrect and in some cases, impossible.
>>>>
>>>> Likewise, how are SCIM clients consuming phone numbers (especially
>>>> those that may be read-only consumers of data and don't have
>>>> visibility/control over the source data)? Is anyone doing any normalizing,
>>>> validating, canonicalizing, or just leaving the data as-is regardless of
>>>> the format/validity?
>>>>
>>>>
>>>>> some standards and libraries that *could *potentially be used to
>>>>> deter completely bogus numbers [1], but the value in this may be debatable.
>>>>>
>>>>>> phoneNumbers), yet it does not explicitly describe input
>>>>>> requirements or validation for attributes such as these.* Is it
>>>>>> expected that values that cannot accurately be canonicalized are rejected
>>>>>> by the SCIM service or that the SCIM service accepts this invalid data and
>>>>>> merely persists and returns this bad data to consumers?*
>>>>>>
>> http://www.healthintersections.com.au /
>> [email-masked] / +61 411 867 065
>>
>
>

---

From: Phil Hunt <[email-masked]>
Date: Wed, 22 Oct 2014 12:46:06 -0700

Comments inline…

Phil

@independentid
www.independentid.com
[email-masked]

On Oct 22, 2014, at 12:13 PM, Shelley <[email-masked]> wrote:

> Before any specific text is proposed, I'd like to get a general understanding for what the agreement/direction is. It seems there are still some major open questions that I'd appreciate some input on.
> Should consumers be required to provide values in specific formats?
> For example, for phone numbers, must consumers provide RFC 3966 format, an international format, any of multiple common/pre-defined formats, or is any value allowed? If not in RFC/international format, is the country/area code assumed?
> Should service providers perform attribute validation or accept any values?
> If service providers perform validation, what is the expected behavior when validation fails? (400 / ignore the value / store the value as-is?)
> Particularly if there are no requirements for data input and/or no validation is performed:
> What are the assumptions that service providers can make during canonicalization?
> What should service providers / consumers do if canonicalization is not possible?
> Should the use of both the "value" and "display" attributes be recommended?
> There are minimally two major variations of possible changes on the table, with various flavors in between:
> 
> Require consumers to provide values in specified formats and SPs perform validation, rejecting invalid values
> SCIM serves as a repository of well-formed, valid data. This results in the most usable data for SCIM consumers reading data, but puts some burden on SCIM consumers providing the data to conform.
> -OR-
> Allow consumers to provide values in any format, SP performs no validation though it may attempt perform canonicalization
> SCIM serves as a repository of data from third parties, data may not be consistent, valid, or normalized. This results in the least effort for SCIM consumers providing data, but puts some burden on SCIM consumers reading the data to make sense of the data.
> SCIM SPs may attempt to normalize some of the data, but there may be no guarantees with respect to the consistency or accuracy of these normalizations.
[PH - <individualHat/>] I prefer this because it aligns with Postel’s Law.

I recommend stipulating that any modification to the supplied values (e.g. canonicalization) by the service providers MUST be reflected in the HTTP Response JSON representation (e.g. after a POST request). 

I think we also need to be clear about how 3966 ABNF elements should map to SCIM's multi-valued attribute representation for phoneNumbers. The earlier suggestion you made is on the right track (mapping to display, type, value, etc).  This might get a bit complex if for example the telephone number includes embedded parameters *and* the client already specifies type, value, and display representations.  My feeling is that the SP should always honour any assigned “type” value by the client.  However, if the client does not assign a “type”, the SP could assign one based on parsing parameters. For example, say a client provides a value of "tel:7042;phone-context=internal”.  The server might canonicalize/map as:

“type”:”internal”,
“display”:”7042”,
“value”:"tel:7042;phone-context=internal”

More generically, here is one possible mapping based on the ABNF from RFC3966 to phoneNumbers attribute:

“value”: “{telephone-uri}”
“display”:”{global-number-digits / local-number-digits}”
“type”:”{descriptor}”

SP’s may attempt to canonicalize “value” and normalize it. SPs may assign values to “display” and “type” if the client has not asserted values for these sub-attribues.

Regarding mapping of context and descriptor to SCIM type, we would be profiling 3966 to say that the ABNF  “context” component is used and interpreted as a type. (can anybody confirm I have interpreted use of context in 3966 correctly?)