Know Your Baselines: The importance of knowing what ‘good’ looks like when recovering from outages
On Friday, May 17 at approximately 9:56 UTC, the Salesforce “modify all” issue began when a Salesforce database script accidentally assigned the Modify All permission on objects and profiles across a number of customer environments. While we describe the impact of the Modify All permission in more detail HERE, this article will describe the process of assessing your impact and recovering back to a known-good baseline.
As Salesforce restored configurations starting on Friday, May 17 and continuing through the week of Monday, May 20, Salesforce asked customers to validate the correctness of their restored account configurations and open support cases for any identified problems.
The reason this created an unusual situation is because many customers treat their production Salesforce implementations as the authoritative “systems of record” for configurations, with their sandboxes being representatively configured clones and potential backups. The recovery scenario where both production and sandbox configurations could be incorrect is rarely considered in our experience. Validating that access is restored to normal requires knowledge of a known-good baseline and the intended access of various roles, users, and applications. In other words, a company must know what “good” looks like in order to validate whether their cloud/SaaS applications are currently in a correct state.
There are two types of baselines that must be considered: functional and security. Functional baselines define the “must have” access levels in the application. that must be met for users to do their jobs. A good baseline should include not only the high level activities a user needs, but also the application-specific interaction flows that constitute that access. For example, a particular support persona must be able to view incoming cases, take ownership of a case, make updates to the case, and eventually close or re-assign the case. The ability to perform these flows is often dependant on a number of different application permissions and settings being correctly configured.
Security baselines are different. They define the access to data that users must not have. For example, external customer support users must not have access to the data of other customers. Defining a set of access rights or permissions that should be blocked or prevented is often referred to as a block list or blacklist. Alternatively, a more mature security baseline would express the totality of access available to an external customer support user. This may include access such as the ability to create cases associated with their account, edit open cases associated with their account, and view published knowledge base articles. Defining a set of comprehensive access rights or permissions that a user should have is often referred to as an allow list or whitelist. It is important to note that a whitelist is meant to be all inclusive. Conversely, a blacklist defines certain exclusions and is not meant to be comprehensive.
In practice, assessment of these baselines is what most companies perform during their release cycle. Certain baselines, such as automated tests for SaaS-hosted software (e.g. Salesforce Force.com) can be easily run in an automated fashion. Many companies assess their functional baselines using a labor-intensive process involving representatives from various business groups in the company spending considerable time clicking through a user acceptance testing (UAT) environment to make sure they can still perform all required actions.