On Friday, May 17 at approximately 9:56 UTC, the Salesforce “modify all” issue began with a Salesforce deployment script accidentally assigning the Modify All permission for all objects to all profiles and permission sets in Salesforce orgs that have ever been enabled for the Pardot feature set. Enabling Modify All for all objects is tantamount to enabling the global Modify All Data (MAD) system permission on these profiles and permission sets. While we describe the impact of the MAD permission in more detail HERE, this article will describe the issue from the perspective of an AppOmni customer.
First, a quick primer on the AppOmni feature sets most responsive to this issue. In the AppOmni platform, customers configure their intended effective data access business logic to describe who can see and do what across their cloud/SaaS applications. AppOmni then creates 360 degree data access models for each of those cloud/SaaS applications and evaluates those data access models to ensure that user access is consistent with the customer’s intent. This intended access can include not only read access to data, but write or delete access as well.
Access can be described based on high level business categories or be as specific as intended field access, allowable access by third parties, data classifications, and other SaaS application-specific concepts. Additionally, the AppOmni platform manages configurations for both data access security (maximum allowable or explicitly disallowed access) and functional data access (required access for business groups to perform their jobs).
The Salesforce issue included two main phases: the initial security phase and a transition to functional outages as Salesforce restored the original configuration from backups. These phases impacted both AppOmni customer security and functional configurations, respectively.
The security phase lasted approximately 5 hours before Salesforce initiated containment measures, with at least one smaller scale event on Monday, May 20 where excessive permissions were briefly enabled again on 3 Salesforce server instances. During this period, all impacted data was exposed in full read/write mode to internal and external users. Due to the way the Salesforce data access model works, field-level restrictions and those based on system permissions were still in place.
Shortly after the assignment of the MAD permission across all objects, the automated AppOmni cloud/SaaS monitoring detected the greatly increased access and the violations of customer security configurations indicating intents such as “External Users may not access internal Customer Data”, “Contractors may not read or write Forward Looking Financials”, or simply that users were suddenly able to access data beyond their allowed baseline.
Immediately following this detection, AppOmni customers received alerts via email and notification integrations such as Salesforce Support Cloud, Jira, and PagerDuty. On their dashboards, the view of their cloud/SaaS security posture showed a sharp spike in security violations that looked like the top chart line in this screenshot:
Approximately 5 hours later, Salesforce initiated containment measures by removing access from all users. These actions were recorded by the AppOmni platform as the open security violations being successfully resolved via access removal.
At the same time, AppOmni detected violations across customers’ functional configurations (shown by the smaller line in the chart since the required functionality is typically a small subset of the overall possible access within an enterprise Salesforce environment). These violations were due to the specific mechanism in which Salesforce contained the incident, which was to remove read, create, update, and delete access for all objects from all users. As the majority of customer functional configurations involve interacting with data, the Salesforce containment measures correctly registered as functional outages within AppOmni. Salesforce later instituted recovery procedures by taking server instances offline and restoring configurations from backups.
For the majority of Salesforce customers, functional outages were resolved over the weekend or early on Monday, with AppOmni recording the closure of most functional configuration violations. Many customers report ongoing difficulties around managed packages and certain profiles or permission sets not being correctly restored, especially on NA53, NA57, and the NA59 server instances. Salesforce is requesting that any administrators experiencing problems open a support ticket for additional remediation. At the time of this writing, restoration of sandbox configurations was lower priority for Salesforce and may or may not be performed.
For many customers, the current need to validate the completeness and the correctness of their restored Salesforce accounts caught them off guard with no way to validate the current state of their account and whether it was restored correctly. We’ll cover that topic in our article on the importance of knowing your baselines with cloud and SaaS applications.
Contributor: Brian Soby, CTO | May 22, 2019