Subject: [scim] SCIM Synchronization Problem

From: Danny Mayer <[email-masked]>
Date: Wed, 18 Aug 2021 10:34:21 -0400

I decided that this needs it's own thread and not be part of the meeting 
minutes.

I have had a great deal of experience dealing with the user account 
synchronization problem. Here's my view of the problems.

I will be calling one system Management Server and the other system 
Application Server. I found client/server labels confusing. The 
Management Server is what I am defining to be the server that sends 
updates to add/update/remove users and groups to the Application server 
whose account, groups and access permissions are being managed.

First some definitions of user accounts. There are usually more than one 
of each of these:
1. Builtin accounts
2. Special-purpose accounts
3. Employee
4. Contractor
5. Agent
6. Customer

There may be more.

1. Builtin accounts: These are accounts that applications have and there 
may be more than one. There is always an admin account which can do 
anything, for example the administrator account in Active Directory or a 
database admin account. The application may have more accounts for other 
purposes.

2. Special-purpose accounts: These may be set up to provide access to 
other applications, for example a SCIM request to a SCIM REST API should 
be handled by a special account which cannot be used to login via a UI 
interface and only be able to perform certain functions. In addition 
there may be accounts set up to listen for topics or queues on a message 
queue among other possibilities. Keeping separate accounts like this are 
important for tracking in logs and applications.

3. Employee: These are accounts that employees may login to the application.

4. Contractors: These accounts that a contractor performing work for the 
company may use to log into an application. Unlike Employee accounts 
these would have an expiration date.

5. Agent: Accounts like this are for external users who may need to 
manage information for their own customers. An example of this is an 
insurance agent logging in to handle an insurance policy for their clients.

6. Customers: These are where the customers are using the application 
directly. For a bank it's likely to be millions of customers. The 
management platform should not be involved in managing these accounts.

Let's now look at a few example applications.

1. Helpdesk
All employees and contractors will need to be able to log into a 
helpdesk application and enter tickets. This means loading information 
about all employees and contractors. For a company with only 1000 
employees that's manageable. For a company with 100K employees, it's a 
bigger challenge.

2. Customer Support
Only employees or contractors in the department providing customer 
support need access plus a few other employees. In addition identified 
customers may need accounts.

3. Expenses
Not all employees or contractors will be submitting expenses so it may 
not be necessary to have accounts for all possible users. This is 
something that the application owner needs to decide.

Now let's look at logistics.

Bulk load:
Each application will need an initial set of accounts set up and for 
something like a helpdesk this could involve load 1000-100,000 accounts. 
The information needed could come from either the management server or 
separately, say from an HR system. Many servers that I have encountered 
limit the number of records to something like 1000, so the pagination 
requirement is needed for this. Even when dealing with a limited subset 
of employees or contractors you can run into this need.

Synchronization
An application that is bulk-loaded above may need to be synchronized to 
the management server if the data did not come from the management server.

Change Management
This is really a synchronization issue as well. Changes happen all the 
time and new employees/contractors need to be added, terminated ones 
removed and updates happen all the time. The best way of dealing with 
this may be to set up a message queue that each application can 
subscribe to and they can take the needed action when it's convenient 
for that application. It's not the only method but it's the one I found 
to be the most helpful. There are two ways of doing that: 1. send the 
complete user information for new accounts, send just the change for 
updating accounts, send the ID for terminated accounts along with some 
meta information. The other method which I have used is just to send the 
ID and whether it's new, updated or terminated.

I hope this is helpful to the discussion.

Danny

---

From: "Matt Peterson (mpeterso)" <[email-masked]>
Date: Thu, 19 Aug 2021 17:33:33 +0000

Thank you, Danny, for creating a new thread.  The following are my comments from meeting minutes (some edits):

There appears to be two use cases for "synchronization": 

  1.  Constructing and enforcing Application authorization models -  An application (acting as a SCIM client) caches users/groups data from the IdP in order to present "user/group pickers" when presenting screens used to configure the application  authorization rules (RBAC, Policy, or ACL).  Also, when the application enforces authorization, user and group data needs to be immediately available so that authorization decisions can be made quickly.

  2.  Identity Management and Governance systems - implement a canonical identity model" where all accounts and groups are represented.  This model is used to build provisioning rules and calculate separation of duty violations, attestations, and approvals etc.   Management-time evaluation of the model needs to be done efficiently without external calls to the SCIM service provider.

Both above use cases, are "client-cache" use cases that need only a "one-way" sync (from the SCIM server to the SCIM client).   To accomplish there are two distinct steps:  a) download the initial results (users/groups) and, b) keep the cached copy of these initial results (user/groups) up to date with changes that are being made on the SCIM Service provider.   

I think it could help us narrow down the list of suitable approaches if we could agree that "one-way" sync (keeping a client-side cache up to date) is our target.

--
Matt Peterson

-----Original Message-----
From: scim <[email-masked]> On Behalf Of Danny Mayer
Sent: Wednesday, August 18, 2021 8:34 AM
To: SCIM WG <[email-masked]>
Subject: [scim] SCIM Synchronization Problem

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

I decided that this needs it's own thread and not be part of the meeting minutes.

I have had a great deal of experience dealing with the user account synchronization problem. Here's my view of the problems.

I will be calling one system Management Server and the other system Application Server. I found client/server labels confusing. The Management Server is what I am defining to be the server that sends updates to add/update/remove users and groups to the Application server whose account, groups and access permissions are being managed.

First some definitions of user accounts. There are usually more than one of each of these:
1. Builtin accounts
2. Special-purpose accounts
3. Employee
4. Contractor
5. Agent
6. Customer

There may be more.

1. Builtin accounts: These are accounts that applications have and there may be more than one. There is always an admin account which can do anything, for example the administrator account in Active Directory or a database admin account. The application may have more accounts for other purposes.

2. Special-purpose accounts: These may be set up to provide access to other applications, for example a SCIM request to a SCIM REST API should be handled by a special account which cannot be used to login via a UI interface and only be able to perform certain functions. In addition there may be accounts set up to listen for topics or queues on a message queue among other possibilities. Keeping separate accounts like this are important for tracking in logs and applications.

4. Contractors: These accounts that a contractor performing work for the company may use to log into an application. Unlike Employee accounts these would have an expiration date.

5. Agent: Accounts like this are for external users who may need to manage information for their own customers. An example of this is an insurance agent logging in to handle an insurance policy for their clients.

6. Customers: These are where the customers are using the application directly. For a bank it's likely to be millions of customers. The management platform should not be involved in managing these accounts.

1. Helpdesk
All employees and contractors will need to be able to log into a helpdesk application and enter tickets. This means loading information about all employees and contractors. For a company with only 1000 employees that's manageable. For a company with 100K employees, it's a bigger challenge.

2. Customer Support
Only employees or contractors in the department providing customer support need access plus a few other employees. In addition identified customers may need accounts.

3. Expenses
Not all employees or contractors will be submitting expenses so it may not be necessary to have accounts for all possible users. This is something that the application owner needs to decide.

Bulk load:
Each application will need an initial set of accounts set up and for something like a helpdesk this could involve load 1000-100,000 accounts.
The information needed could come from either the management server or separately, say from an HR system. Many servers that I have encountered limit the number of records to something like 1000, so the pagination requirement is needed for this. Even when dealing with a limited subset of employees or contractors you can run into this need.

Synchronization
An application that is bulk-loaded above may need to be synchronized to the management server if the data did not come from the management server.

Change Management
This is really a synchronization issue as well. Changes happen all the time and new employees/contractors need to be added, terminated ones removed and updates happen all the time. The best way of dealing with this may be to set up a message queue that each application can subscribe to and they can take the needed action when it's convenient for that application. It's not the only method but it's the one I found to be the most helpful. There are two ways of doing that: 1. send the complete user information for new accounts, send just the change for updating accounts, send the ID for terminated accounts along with some meta information. The other method which I have used is just to send the ID and whether it's new, updated or terminated.

Danny

_______________________________________________
scim mailing list
[email-masked]
https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Fscim&amp;data=04%7C01%7Cmatt.peterson%40quest.com%7C1fb1fe803b8c439026bd08d9625553ba%7C91c369b51c9e439c989c1867ec606603%7C0%7C0%7C637648940889425726%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=riJQy3jEHltc6ZFCaEe%2B6%2F2ecXN0SWBXOxIZAKLxum4%3D&amp;reserved=0

---

From: "Matt Peterson (mpeterso)" <[email-masked]>
Date: Thu, 19 Aug 2021 17:42:53 +0000

Danny,

To help me understand in your post is the "SCIM Client" and what is the "SCIM Server" can you tell me which of your components (the Management Server or the Application Server) implements SCIM endpoints?   

The component that acts as a "SCIM Server" is the component that provides the of WebAPI endpoints that conform to the SCIM spec.  For example, the  /user, /group SCIM endpoints. 

The component(s) that uses these endpoints to query (GET)  users/groups or to create (POST) user/groups is the "SCIM client".

--
Matt 

-----Original Message-----
From: scim <[email-masked]> On Behalf Of Danny Mayer
Sent: Wednesday, August 18, 2021 8:34 AM
To: SCIM WG <[email-masked]>
Subject: [scim] SCIM Synchronization Problem

There may be more.

1. Helpdesk

2. Customer Support

3. Expenses

Bulk load:

Synchronization

Change Management

Danny

---

From: Phillip Hunt <[email-masked]>
Date: Thu, 19 Aug 2021 11:05:43 -0700

This is good high level case information. 

Matt, you make a good case for access control standards. It brings out some questions…

Is syncing users and groups enough or must access policy rules/data be synchronized (assuming a common standard for policy)?  How close must separate domains be to work this well together? Is it true that access control is the same across domains?

These issues are what led me to believe client/server RESTful “command”s like POST/PUT/PATCH are insufficient. Instead, what if an event in one domain is published and a subscriber decides how to process the event? In command mode the requestor has to know the command will work. In event mode, the receiver has context to interpret and translate into local commands. 

Maybe we can do a con call and go through security events and how they support arms length cross domain relationships. 

Phil

> On Aug 19, 2021, at 10:33 AM, Matt Peterson (mpeterso) <Matt.Peterson=[email-masked]> wrote:
> 
> https://www.ietf.org/mailman/listinfo/scim

---

From: Danny Mayer <[email-masked]>
Date: Fri, 20 Aug 2021 10:46:15 -0400

It's not that simple. If an administrator goes in an adds a new account 
or updates an existing account and adds permissions to it, how will the 
management side know? The way I did it was to have the application 
monitor for those changes and notify the management server of the 
changes. I did this via a message queue for our internal applications. 
The management server needs to decide how to handle these violations.

Danny

---

From: Danny Mayer <[email-masked]>
Date: Fri, 20 Aug 2021 10:54:18 -0400

I decided to avoid the client/server naming convention because I find it 
confusing. The system responsible for managing the user accounts and 
groups I have declared to be the Management server. I believe you 
describe this as the "SCIM Client" and is used to GET/POST/PATCH, etc. 
to the Application server to maintain the user accounts and groups. The 
application server is what I believe you call the "SCIM Server". This 
provides the SCIM API's needed.

I find the use of client/server not helpful to understand their role in 
the protocol.

Danny

---

From: Danny Mayer <[email-masked]>
Date: Fri, 20 Aug 2021 11:04:50 -0400

You cannot assume that the groups are the same across different 
application domains. In our case each group was totally different and 
had different permissions associated with it. In addition you can have 
different permissions associated individually to a user account. 
Microsoft's NTFS access controls are a good example of this.

The SCIM Config information should include information on what commands 
are available on that system. In addition the application needs to 
notify the requestor that if something went wrong why it went wrong. For 
example, in my cases if a request for a group to be added to a user 
cannot be fulfilled because the group does not exist then the failure 
needs to be reported back. Also for our builtin and special accounts, 
the requestor cannot make any changes.

Does this help?

Danny

---

From: Phillip Hunt <[email-masked]>
Date: Fri, 20 Aug 2021 08:31:43 -0700

To be clear, the specs are written as protocol profiling HTTP.  The terms client and server stem from HTTP’s notion that a client initiates a request and a server responds. 

It is ok to develop a higher level in the information layer.  For example we had to adopt publish and subscribe for security events flow but the delivery specs talk about whether subscribers act as http clients or servers depending in whether events are pushed to them or whether they pull events from the publisher. 

Some of Danny’s requirements strike me as access control needs in scim service.  For example a client that authenticates with the correct scope may be allowed to return unlimited results in a request or may be allowed frequent polling requests. Another client may only be allowed to update or read specific claims at a certain rate.   Access control is something we can talk about in the charter (I have a proposal I can put together quickly in a draft)) but I would not support conflating access control with synchronization.  

The cases Danny is describing sound like a hybrid (of replica vs coordinated) where two domains are unequal but under a common authority.  This of course should be supported. 

The event model helps responsiveness in that subscribers can register to listen for changes as they occur.  As I mentioned previously I am happy to give a presentation on synchronous command (ie REST) vs async events and why this aids cross-domain synchronization performance and accuracy. 

Phil

> On Aug 20, 2021, at 8:04 AM, Danny Mayer <[email-masked]> wrote:
> 
> You cannot assume that the groups are the same across different application domains. In our case each group was totally different and had different permissions associated with it. In addition you can have different permissions associated individually to a user account. Microsoft's NTFS access controls are a good example of this.
> 
> The SCIM Config information should include information on what commands are available on that system. In addition the application needs to notify the requestor that if something went wrong why it went wrong. For example, in my cases if a request for a group to be added to a user cannot be fulfilled because the group does not exist then the failure needs to be reported back. Also for our builtin and special accounts, the requestor cannot make any changes.
> 
> Does this help?
> 
> Danny
>

---

From: Danny Mayer <[email-masked]>
Date: Fri, 20 Aug 2021 12:44:45 -0400

The cases I was describing were exactly access and permissions cases and 
was needed for audit and compliance purposes. In our cases the groups 
that were associated with the user accounts provided the permissions 
needed for each user.

Danny

---

From: "Matt Peterson (mpeterso)" <[email-masked]>
Date: Mon, 23 Aug 2021 23:30:33 +0000

Thanks for the clarification.  

"Client" and "Server" are as used dozens of times in the existing RFCs which makes these as useful in understanding the SCIM protocol as they are for understanding the HTTP protocol.  In a SCIM protocol exchange there is a thing that provides a resource (the server) and a thing that makes a HTTP request (GET,POST,PUT,PATCH,DELETE) on the resource (the client).  

I think it would be helpful to try and use this generally accepted HTTP terminology if we can -- especially when talking about this synchronization topic.  

Management server <-- SCIM Client
Application Server <-- SCIM Server

With this clarification, seems to me like your Management Server is using SCIM to manage accounts on the Application Server, and that the Management Server would benefit from having an up-to-date cache of accounts that are on the Application Server(s) even for cases where account changes/additions/deletions are made "out of band"  (e.g. directly on the Application Server, not initiated by the Management server.)

Did I understand correctly?

-----Original Message-----
From: Danny Mayer <[email-masked]> 
Sent: Friday, August 20, 2021 8:54 AM
To: Matt Peterson (mpeterso) <[email-masked]>; SCIM WG <[email-masked]>
Subject: Re: [scim] SCIM Synchronization Problem

I decided to avoid the client/server naming convention because I find it confusing. The system responsible for managing the user accounts and groups I have declared to be the Management server. I believe you describe this as the "SCIM Client" and is used to GET/POST/PATCH, etc.
to the Application server to maintain the user accounts and groups. The application server is what I believe you call the "SCIM Server". This provides the SCIM API's needed.

I find the use of client/server not helpful to understand their role in the protocol.

Danny

On 8/19/21 1:42 PM, Matt Peterson (mpeterso) wrote:
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.
> ietf.org%2Fmailman%2Flistinfo%2Fscim&amp;data=04%7C01%7CMatt.Peterson%
> 40oneidentity.com%7C93c364afa0274e3fb35b08d963ea643e%7C91c369b51c9e439
> c989c1867ec606603%7C0%7C1%7C637650680638761908%7CUnknown%7CTWFpbGZsb3d
> 8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C
> 3000&amp;sdata=BqZusI4HsTB%2FyPo9SCrTMW6ZdIQNyVrPxGG%2BiiXx2fs%3D&amp;
> reserved=0
>

---

From: "Matt Peterson (mpeterso)" <[email-masked]>
Date: Mon, 23 Aug 2021 23:49:22 +0000

The management server in your scenario is a SCIM client.  

If an application administrator goes in to the Application Server  and adds, removes, or modifies account,  the Management Server  has one option when using SCIM -- which is to query the Application Server (the SCIM server) for all accounts and then resolve the differences between this call and a previous call where all accounts were fetched.   This is very inefficient.  Especially when it comes to detecting deleted accounts, there is no other way in standard SCIM except to have your Management Server re-load all the accounts from your Application server.  

What you've implemented in your Management Server *is* a pure client-side cache scenario as I described it.  Except that you've recognized that the SCIM spec makes it very inefficient to keep your Management Serve up-to-date with changes happening "out-of-band" on the Application Servers.  

So... you implemented a notification system whereby your Management Server (SCIM Client) can be notified of changes that just happened on the Application Server (the SCIM server).
 
Your implementation, where the Application Server (SCIM server) notifies your Management Server (SCIM client) that a change has occurred *is* one of the proposed technical approaches identified by @Phil in a previous post to this list and was also mentioned in the BOF.  

I believe that an approach (similar to your implementation ) whereby a SCIM client can subscribe to be notified of changes on the SCIM server is an intuitive approach for solving the "Syncronization problem"

--
Matt

-----Original Message-----
From: Danny Mayer <[email-masked]> 
Sent: Friday, August 20, 2021 8:46 AM
To: Matt Peterson (mpeterso) <[email-masked]>; SCIM WG <[email-masked]>
Subject: Re: [scim] SCIM Synchronization Problem

It's not that simple. If an administrator goes in an adds a new account or updates an existing account and adds permissions to it, how will the management side know? The way I did it was to have the application monitor for those changes and notify the management server of the changes. I did this via a message queue for our internal applications.
The management server needs to decide how to handle these violations.

Danny

On 8/19/21 1:33 PM, Matt Peterson (mpeterso) wrote:
> ietf.org%2Fmailman%2Flistinfo%2Fscim&amp;data=04%7C01%7CMatt.Peterson%40oneidentity.com%7C0502a1bbffb94f6220b808d963e944e9%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637650675822236228%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=ol7Mrjj%2Bx2SGTgQegrvwVNlVeMODf6pxEOc5PTjkK1g%3D&amp;reserved=0
>

---

From: Phil Hunt <[email-masked]>
Date: Mon, 23 Aug 2021 18:03:11 -0700

+1 Matt

Ironically when developing the Security Event Token spec, we listed a hypothetical SCIM example in RFC8417.  For example in Section 2.2.1 (https://datatracker.ietf.org/doc/html/rfc8417#section-2.1.1), a password reset event.

  {
     "iss": "https://scim.example.com",
     "iat": 1458496025,
     "jti": "3d0c3cf797584bd193bd0fb1bd4e7d30",
     "aud": [
       "https://jhub.example.com/Feeds/98d52461fa5bbc879593b7754",
       "https://jhub.example.com/Feeds/5d7604516b1d08641d7676ee7"
     ],
     "sub": "https://scim.example.com/Users/44f6142df96bd6ab61e7521d9",
     "events": {
       "urn:ietf:params:scim:event:passwordReset": {
         "id": "44f6142df96bd6ab61e7521d9"
       },
       "https://example.com/scim/event/passwordResetExt": {
         "resetAttempts": 5
       }
     }
   }

Notice under the “events” attribute is a somewhat SCIM like structure consisting of an Event ID (urn:ietf:params:scim:event:passwordReset), a payload (the id). The example has an extension where additional data is provided which in this case is “resetAttempts”.

We could also have a set of *simple* IDM events indicating:  CREATE, MODIFY, DELETE and an id.  The events don’t even have to contain data. You could then have the receiver simply do a GET on the resource allowing it to reconcile the current resource.  This enables access control restrictions to process and keeps event payloads simple in the event.

SET has two approved RFCs for exchanging events, namely SET PUSH (RFC8935) and POLLING (RFC8936).  

Polling mode is great when a management server behind a firewall.  It can use HTTP LONG POLLING in order to receive events in real time. Polling is also good because you can theoretically re-use the same credential to pick up events.  Push on the other hand requires a credential be placed in the publishing side to POST to the event receiver.

Push ends up being preferred in situations where a single event publisher has many 1000s of subscribers (this was the downfall of pubsubhubub).  I think for most SCIM cases, we are looking at more equal relationships of publishers to subscribers.

Phil Hunt
@independentid
[email-masked]

> On Aug 23, 2021, at 4:49 PM, Matt Peterson (mpeterso) <Matt.Peterson=[email-masked]> wrote:
> 
> From: Danny Mayer <[email-masked] <mailto:[email-masked]>> 
> To: Matt Peterson (mpeterso) <[email-masked] <mailto:[email-masked]>>; SCIM WG <[email-masked] <mailto:[email-masked]>>
>> ietf.org <http://ietf.org/>%2Fmailman%2Flistinfo%2Fscim&amp;data=04%7C01%7CMatt.Peterson%40oneidentity.com <http://40oneidentity.com/>%7C0502a1bbffb94f6220b808d963e944e9%7C91c369b51c9e439c989c1867ec606603%7C0%7C1%7C637650675822236228%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=ol7Mrjj%2Bx2SGTgQegrvwVNlVeMODf6pxEOc5PTjkK1g%3D&amp;reserved=0
>> 
> 
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/scim <https://www.ietf.org/mailman/listinfo/scim>

---

From: Craig McClanahan <[email-masked]>
Date: Mon, 23 Aug 2021 18:43:42 -0700

Several years ago (early in the lifetime of the SCIM specification), I was
involved in exactly this kind of situation.  I was trying to extract the
Identity portion of a monolith out into a separate service.  Naturally, as
you'll see in lots of monoliths, the User database information was used for
*lots* of things, not just authentication and authorization, so we couldn't
just remove it from the monolith's database.

>From a SCIM terminology perspective, life was actually pretty clear -- the
monolith was the SCIM server (the source of truth), and the new Identity
Service was the SCIM client.  Creating the initial download mechanism was
no big deal -- one humongous batch transfer (or done in pieces if need be)
and the Identity Service is up to date with the initial snapshot.  But what
happens when:

   - User changes their name (or a bunch of other profile fields)?  Not
   security sensitive, but definitely UI sensitive.  If they are part of the
   data transferred from the SCIM server to the SCIM client, it's definitely
   relevant.
   - User logs on or logs off?  (Groan ... the monolith cared a lot about
   this, because it affects UI like "is this person currently logged in" ...
   even worse, the monolith updated a field in the actual user database row,
   which triggered a gazillion "user change" events (originally internal to
   the monolith) documenting a change that had no other impact than to change
   the last logged in timestamp ... sigh ... performance ... sigh.)
   - User changes their password?  We probably could have absorbed that
   function, but would have required modifications to a bunch of apps out of
   my team's purview, so we didn't.
   - For that matter, *any* change happens between when the snapshot was
   taken, and the Identity Service caught up.

Anyone who has ever tried this trick has most likely run into the same
kinds of issues -- how do you deal with incremental changes that need to be
communicated from the server to the client?  Some of those changes
(passwords, ACLs, account deactivations, etc) are very much time critical.

Our team chose to try to use webhooks from the monolith (SCIM server) back
to the Identity Service (SCIM client), totally out-of-band to anything
defined by SCIM.  This would have worked OK if the webhook technology was
actually reliable and incorporated things like guaranteed forwarding of the
change event messages.  It doesn't work very well, of course, when the
Identity Service hasn't fully processed the initial snapshot yet but
receives a realtime update from the monolith.  In retrospect, this approach
was probably a mistake -- a different messaging technology would have been
better for a narrow point-to-point requirement like this, but would not
have addressed all of the issues.  And what about a more broad-based
notification requirement?

Could we have turned the whole thing around, and made the new Identity
Service the SCIM server, and the monolith the SCIM client?  I suppose, but
it would have required the Identity Service to be involved in a gazillion
things that were not authentication or authorization related, and the need
for some sort of out-of-band "incremental change" event notifications would
have remained, just going in the other direction.

For the SCIM specification, It's the "incremental change" thing that is the
hard nut to crack, IMHO.  But it's actually a bigger problem than that,
endemic to any scenario where you are trying to tear apart a monolith.

Craig McClanahan

On Mon, Aug 23, 2021 at 4:30 PM Matt Peterson (mpeterso) <Matt.Peterson=
[email-masked]> wrote:

> "Client" and "Server" are as used dozens of times in the existing RFCs
> which makes these as useful in understanding the SCIM protocol as they are
> for understanding the HTTP protocol.  In a SCIM protocol exchange there is
> a thing that provides a resource (the server) and a thing that makes a HTTP
> request (GET,POST,PUT,PATCH,DELETE) on the resource (the client).
>
> I think it would be helpful to try and use this generally accepted HTTP
> terminology if we can -- especially when talking about this synchronization
> topic.
>
> With this clarification, seems to me like your Management Server is using
> SCIM to manage accounts on the Application Server, and that the Management
> Server would benefit from having an up-to-date cache of accounts that are
> on the Application Server(s) even for cases where account
> changes/additions/deletions are made "out of band"  (e.g. directly on the
> Application Server, not initiated by the Management server.)
>
> CAUTION: This email originated from outside of the organization. Do not
> follow guidance, click links, or open attachments unless you recognize the
> sender and know the content is safe.
>
>
> confusing. The system responsible for managing the user accounts and groups
> I have declared to be the Management server. I believe you describe this as
> the "SCIM Client" and is used to GET/POST/PATCH, etc.
> > To help me understand in your post is the "SCIM Client" and what is the
> "SCIM Server" can you tell me which of your components (the Management
> Server or the Application Server) implements SCIM endpoints?
> >
> > The component that acts as a "SCIM Server" is the component that
> provides the of WebAPI endpoints that conform to the SCIM spec.  For
> example, the  /user, /group SCIM endpoints.
> >
> > The component(s) that uses these endpoints to query (GET)  users/groups
> or to create (POST) user/groups is the "SCIM client".
> >
> > --
> > Matt
> >
> > CAUTION: This email originated from outside of the organization. Do not
> follow guidance, click links, or open attachments unless you recognize the
> sender and know the content is safe.
> >
> >
> Application Server. I found client/server labels confusing. The Management
> Server is what I am defining to be the server that sends updates to
> add/update/remove users and groups to the Application server whose account,
> groups and access permissions are being managed.
> >
> other applications, for example a SCIM request to a SCIM REST API should be
> handled by a special account which cannot be used to login via a UI
> interface and only be able to perform certain functions. In addition there
> may be accounts set up to listen for topics or queues on a message queue
> among other possibilities. Keeping separate accounts like this are
> company may use to log into an application. Unlike Employee accounts these
> would have an expiration date.
> >
> about all employees and contractors. For a company with only 1000 employees
> that's manageable. For a company with 100K employees, it's a bigger
> challenge.
> >
> > 2. Customer Support
> not be necessary to have accounts for all possible users. This is something
> that the application owner needs to decide.
> >
> requirement is needed for this. Even when dealing with a limited subset of
> employees or contractors you can run into this need.
> >
> > Synchronization
> removed and updates happen all the time. The best way of dealing with this
> may be to set up a message queue that each application can subscribe to and
> they can take the needed action when it's convenient for that application.
> It's not the only method but it's the one I found to be the most helpful.
> There are two ways of doing that: 1. send the complete user information for
> new accounts, send just the change for updating accounts, send the ID for
> terminated accounts along with some meta information. The other method
> which I have used is just to send the ID and whether it's new, updated or
> terminated.
> >

---

From: Danny Mayer <[email-masked]>
Date: Tue, 24 Aug 2021 11:33:38 -0400

Matt,

What I'm trying to accomplish here is to see the goals of the effort. 
The protocols is just a means to implement the goals. This is the higher 
level of what we are doing.

The answer to your last paragraph is yes, that's what I was trying to 
get to.

Danny

---

From: Danny Mayer <[email-masked]>
Date: Tue, 24 Aug 2021 11:35:04 -0400

Exactly right.

---

From: Danny Mayer <[email-masked]>
Date: Tue, 24 Aug 2021 11:48:12 -0400

I've been there and done that. We were lucky enough to have a way to 
differentiate between someone logging in and out and when an update is 
made to something that the SCIM client cares about. We preferred to send 
a notification to a queue that something has changed for the specific ID 
and application and leave it up to the "SCIM client" to query the 
application for the given ID. This had to be a manual operation as the 
SCIM client had no way of reading the queue. The queue was necessary for 
compliance and audit purposes so it wasn't a waste.

Danny

On 8/23/21 9:43 PM, Craig McClanahan wrote:
> Several years ago (early in the lifetime of the SCIM specification), I 
> was involved in exactly this kind of situation.  I was trying to 
> extract the Identity portion of a monolith out into a separate 
> service.  Naturally, as you'll see in lots of monoliths, the User 
> database information was used for *lots* of things, not just 
> authentication and authorization, so we couldn't just remove it from 
> the monolith's database.
>
> From a SCIM terminology perspective, life was actually pretty clear -- 
> the monolith was the SCIM server (the source of truth), and the new 
> Identity Service was the SCIM client.  Creating the initial download 
> mechanism was no big deal -- one humongous batch transfer (or done in 
> pieces if need be) and the Identity Service is up to date with the 
> initial snapshot.  But what happens when:
>
>   * User changes their name (or a bunch of other profile fields)?  Not
>     security sensitive, but definitely UI sensitive.  If they are part
>     of the data transferred from the SCIM server to the SCIM client,
>     it's definitely relevant.
>   * User logs on or logs off?  (Groan ... the monolith cared a lot
>     about this, because it affects UI like "is this person currently
>     logged in" ... even worse, the monolith updated a field in the
>     actual user database row, which triggered a gazillion "user
>     change" events (originally internal to the monolith) documenting a
>     change that had no other impact than to change the last logged in
>     timestamp ... sigh ... performance ... sigh.)
>   * User changes their password?  We probably could have absorbed that
>     function, but would have required modifications to a bunch of apps
>     out of my team's purview, so we didn't.
>   * For that matter, *any* change happens between when the snapshot
>     was taken, and the Identity Service caught up.
>
> kinds of issues -- how do you deal with incremental changes that need 
> to be communicated from the server to the client?  Some of those 
> changes (passwords, ACLs, account deactivations, etc) are very much 
> time critical.
>
> Our team chose to try to use webhooks from the monolith (SCIM server) 
> back to the Identity Service (SCIM client), totally out-of-band to 
> anything defined by SCIM.  This would have worked OK if the webhook 
> technology was actually reliable and incorporated things like 
> guaranteed forwarding of the change event messages.  It doesn't work 
> very well, of course, when the Identity Service hasn't fully processed 
> the initial snapshot yet but receives a realtime update from the 
> monolith.  In retrospect, this approach was probably a mistake -- a 
> different messaging technology would have been better for a narrow 
> point-to-point requirement like this, but would not have addressed all 
> of the issues.  And what about a more broad-based notification 
> requirement?
>
> Service the SCIM server, and the monolith the SCIM client?  I suppose, 
> but it would have required the Identity Service to be involved in a 
> gazillion things that were not authentication or authorization 
> related, and the need for some sort of out-of-band "incremental 
> change" event notifications would have remained, just going in the 
> other direction.
>
> For the SCIM specification, It's the "incremental change" thing that 
> is the hard nut to crack, IMHO.  But it's actually a bigger problem 
> than that, endemic to any scenario where you are trying to tear apart 
> a monolith.
>
> Craig McClanahan
>
>
>
> On Mon, Aug 23, 2021 at 4:30 PM Matt Peterson (mpeterso) 
> <Matt.Peterson=[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     "Client" and "Server" are as used dozens of times in the existing
>     RFCs which makes these as useful in understanding the SCIM
>     protocol as they are for understanding the HTTP protocol.  In a
>     SCIM protocol exchange there is a thing that provides a resource
>     (the server) and a thing that makes a HTTP request
>     (GET,POST,PUT,PATCH,DELETE) on the resource (the client).
>
>     I think it would be helpful to try and use this generally accepted
>     HTTP terminology if we can -- especially when talking about this
>     synchronization topic.
>
>     With this clarification, seems to me like your Management Server
>     is using SCIM to manage accounts on the Application Server, and
>     that the Management Server would benefit from having an up-to-date
>     cache of accounts that are on the Application Server(s) even for
>     cases where account changes/additions/deletions are made "out of
>     band"  (e.g. directly on the Application Server, not initiated by
>     the Management server.)
>
>     To: Matt Peterson (mpeterso) <[email-masked]
>     <mailto:[email-masked]>>; SCIM WG <[email-masked]
>     <mailto:[email-masked]>>
>     CAUTION: This email originated from outside of the organization.
>     Do not follow guidance, click links, or open attachments unless
>     you recognize the sender and know the content is safe.
>
>
>     I decided to avoid the client/server naming convention because I
>     find it confusing. The system responsible for managing the user
>     accounts and groups I have declared to be the Management server. I
>     believe you describe this as the "SCIM Client" and is used to
>     GET/POST/PATCH, etc.
>     to the Application server to maintain the user accounts and
>     groups. The application server is what I believe you call the
>     "SCIM Server". This provides the SCIM API's needed.
>
>     > To help me understand in your post is the "SCIM Client" and what
>     is the "SCIM Server" can you tell me which of your components (the
>     Management Server or the Application Server) implements SCIM
>     endpoints?
>     >
>     provides the of WebAPI endpoints that conform to the SCIM spec. 
>     For example, the  /user, /group SCIM endpoints.
>     >
>     > The component(s) that uses these endpoints to query (GET) 
>     users/groups or to create (POST) user/groups is the "SCIM client".
>     >
>     > --
>     > Matt
>     >
>     > From: scim <[email-masked]
>     <mailto:[email-masked]>> On Behalf Of Danny Mayer
>     > To: SCIM WG <[email-masked] <mailto:[email-masked]>>
>     > CAUTION: This email originated from outside of the organization.
>     Do not follow guidance, click links, or open attachments unless
>     you recognize the sender and know the content is safe.
>     >
>     >
>     > I have had a great deal of experience dealing with the user
>     account synchronization problem. Here's my view of the problems.
>     >
>     > I will be calling one system Management Server and the other
>     system Application Server. I found client/server labels confusing.
>     The Management Server is what I am defining to be the server that
>     sends updates to add/update/remove users and groups to the
>     Application server whose account, groups and access permissions
>     are being managed.
>     >
>     > 1. Builtin accounts: These are accounts that applications have
>     and there may be more than one. There is always an admin account
>     which can do anything, for example the administrator account in
>     Active Directory or a database admin account. The application may
>     have more accounts for other purposes.
>     >
>     > 2. Special-purpose accounts: These may be set up to provide
>     access to other applications, for example a SCIM request to a SCIM
>     REST API should be handled by a special account which cannot be
>     used to login via a UI interface and only be able to perform
>     certain functions. In addition there may be accounts set up to
>     listen for topics or queues on a message queue among other
>     possibilities. Keeping separate accounts like this are important
>     for tracking in logs and applications.
>     >
>     > 4. Contractors: These accounts that a contractor performing work
>     for the company may use to log into an application. Unlike
>     Employee accounts these would have an expiration date.
>     >
>     > 5. Agent: Accounts like this are for external users who may need
>     to manage information for their own customers. An example of this
>     is an insurance agent logging in to handle an insurance policy for
>     their clients.
>     >
>     > 6. Customers: These are where the customers are using the
>     application directly. For a bank it's likely to be millions of
>     customers. The management platform should not be involved in
>     managing these accounts.
>     >
>     helpdesk application and enter tickets. This means loading
>     information about all employees and contractors. For a company
>     with only 1000 employees that's manageable. For a company with
>     100K employees, it's a bigger challenge.
>     >
>     > 2. Customer Support
>     > Only employees or contractors in the department providing
>     customer support need access plus a few other employees. In
>     addition identified customers may need accounts.
>     >
>     > 3. Expenses
>     > Not all employees or contractors will be submitting expenses so
>     it may not be necessary to have accounts for all possible users.
>     This is something that the application owner needs to decide.
>     >
>     > Each application will need an initial set of accounts set up and
>     for something like a helpdesk this could involve load 1000-100,000
>     accounts.
>     > The information needed could come from either the management
>     server or separately, say from an HR system. Many servers that I
>     have encountered limit the number of records to something like
>     1000, so the pagination requirement is needed for this. Even when
>     dealing with a limited subset of employees or contractors you can
>     run into this need.
>     >
>     > Synchronization
>     > An application that is bulk-loaded above may need to be
>     synchronized to the management server if the data did not come
>     from the management server.
>     >
>     > Change Management
>     > This is really a synchronization issue as well. Changes happen
>     all the time and new employees/contractors need to be added,
>     terminated ones removed and updates happen all the time. The best
>     way of dealing with this may be to set up a message queue that
>     each application can subscribe to and they can take the needed
>     action when it's convenient for that application. It's not the
>     only method but it's the one I found to be the most helpful. There
>     are two ways of doing that: 1. send the complete user information
>     for new accounts, send just the change for updating accounts, send
>     the ID for terminated accounts along with some meta information.
>     The other method which I have used is just to send the ID and
>     whether it's new, updated or terminated.
>     >
>     https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww
>     <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww>.
>     > ietf.org
>     <http://ietf.org>%2Fmailman%2Flistinfo%2Fscim&amp;data=04%7C01%7CMatt.Peterson%
>     > 40oneidentity.com
>     <http://40oneidentity.com>%7C93c364afa0274e3fb35b08d963ea643e%7C91c369b51c9e439
>     >
>     <https://www.ietf.org/mailman/listinfo/scim>
>

---

From: Phillip Hunt <[email-masked]>
Date: Tue, 24 Aug 2021 09:10:19 -0700

So a requirement here is to indicate what schema is of interest for sync. Any notification method needs to be filterable on at least schema-id and potentially specific attributes. Eg a receiver may only be interested in events related to roles. 

The also shows why pre-conditioned HEAD queries couldn’t be used. These types of queries can ask: does this url exist? Has this resource changed since this etag or date.  Obviously any change to content changes the hash and mod-dates. 

Obviously one of the downsides to attaching session information to the user resource is affecting change rates. Why not have separate session resources point to users?  Their change rate and lifecycle are much different than that of a user. A best practice would be to keep this data separate from the user. 
IOW -> how much do we need to accommodate in sync?  Should we be having a separate session schema discussion?

Phil

> On Aug 24, 2021, at 8:49 AM, Danny Mayer <[email-masked]> wrote:
> 
> 
> I've been there and done that. We were lucky enough to have a way to differentiate between someone logging in and out and when an update is made to something that the SCIM client cares about. We preferred to send a notification to a queue that something has changed for the specific ID and application and leave it up to the "SCIM client" to query the application for the given ID. This had to be a manual operation as the SCIM client had no way of reading the queue. The queue was necessary for compliance and audit purposes so it wasn't a waste.
> 
> Danny
> 
>> Several years ago (early in the lifetime of the SCIM specification), I was involved in exactly this kind of situation.  I was trying to extract the Identity portion of a monolith out into a separate service.  Naturally, as you'll see in lots of monoliths, the User database information was used for *lots* of things, not just authentication and authorization, so we couldn't just remove it from the monolith's database.
>> 
>> From a SCIM terminology perspective, life was actually pretty clear -- the monolith was the SCIM server (the source of truth), and the new Identity Service was the SCIM client.  Creating the initial download mechanism was no big deal -- one humongous batch transfer (or done in pieces if need be) and the Identity Service is up to date with the initial snapshot.  But what happens when:
>> User changes their name (or a bunch of other profile fields)?  Not security sensitive, but definitely UI sensitive.  If they are part of the data transferred from the SCIM server to the SCIM client, it's definitely relevant.
>> User logs on or logs off?  (Groan ... the monolith cared a lot about this, because it affects UI like "is this person currently logged in" ... even worse, the monolith updated a field in the actual user database row, which triggered a gazillion "user change" events (originally internal to the monolith) documenting a change that had no other impact than to change the last logged in timestamp ... sigh ... performance ... sigh.)
>> User changes their password?  We probably could have absorbed that function, but would have required modifications to a bunch of apps out of my team's purview, so we didn't.
>> For that matter, *any* change happens between when the snapshot was taken, and the Identity Service caught up.
>> Anyone who has ever tried this trick has most likely run into the same kinds of issues -- how do you deal with incremental changes that need to be communicated from the server to the client?  Some of those changes (passwords, ACLs, account deactivations, etc) are very much time critical.
>> 
>> Our team chose to try to use webhooks from the monolith (SCIM server) back to the Identity Service (SCIM client), totally out-of-band to anything defined by SCIM.  This would have worked OK if the webhook technology was actually reliable and incorporated things like guaranteed forwarding of the change event messages.  It doesn't work very well, of course, when the Identity Service hasn't fully processed the initial snapshot yet but receives a realtime update from the monolith.  In retrospect, this approach was probably a mistake -- a different messaging technology would have been better for a narrow point-to-point requirement like this, but would not have addressed all of the issues.  And what about a more broad-based notification requirement?
>> 
>> Could we have turned the whole thing around, and made the new Identity Service the SCIM server, and the monolith the SCIM client?  I suppose, but it would have required the Identity Service to be involved in a gazillion things that were not authentication or authorization related, and the need for some sort of out-of-band "incremental change" event notifications would have remained, just going in the other direction.
>> 
>> For the SCIM specification, It's the "incremental change" thing that is the hard nut to crack, IMHO.  But it's actually a bigger problem than that, endemic to any scenario where you are trying to tear apart a monolith.
>> 
>> Craig McClanahan
>> 
>> 
>> 
>> On Mon, Aug 23, 2021 at 4:30 PM Matt Peterson (mpeterso) <Matt.Peterson=[email-masked]> wrote: