Introducing the SaaS Event Maturity Matrix (EMM)

A Threat Detection framework for assessing SaaS audit logging and a resource for the cybersecurity community, developed by AppOmni

By David Tocco, Lead Threat Detection Engineer, AppOmni

SaaS is the new operating system for businesses. This is largely due to SaaS’ ability to offer unprecedented flexibility, scalability, and integration capabilities for multiple business use cases — from people management and finance to customer management and collaboration.

While SaaS brings significant benefits, organizations must also address a new reality: SaaS applications store, maintain, and process critical business data and workflows, including intellectual property, customer data, employee PII, and much more on a daily basis.

The need to comprehensively and continuously monitor SaaS apps for malicious and anomalous activities and mitigate the threat of data compromise has never been greater.

A key challenge when it comes to monitoring SaaS against cyber threats centers on the lack of uniformity of SaaS APIs and audit logs. To address the challenge, the AppOmni Threat Detection team has developed an industry-first, open source, SaaS Event Maturity Matrix (EMM).

The EMM is built to provide a framework for assessing SaaS audit logging from different SaaS platforms commonly used in the enterprise. This will enable cybersecurity teams to organize and catalog audit logging capabilities, resulting in improved SaaS threat detection efficacy.

Read on to understand the rationale for, and utility of, developing the SaaS Event Maturity Matrix.

SaaS Adoption and Cyber Risk is Growing in Scale and Complexity

Organizations are continuously adopting SaaS often at-will and out of purview from cybersecurity teams. It’s typical now for organizations to leverage hundreds, and in some cases, thousands of SaaS apps, each with their own intricacies in data collection, audit log structure, and visibility.

There’s an additional dimension to this attack surface risk that’s represented by SaaS-to-SaaS connections.

AppOmni has determined that on average a single instance of SaaS has 256 SaaS-to-SaaS connections, many of which are no longer in use, but still have excessive permissions into core business apps such as Salesforce, Okta and GitHub among others.

As the security threats targeting SaaS become increasingly expansive in scope and scale, security practitioners need to evolve their strategies to meet these new threats preemptively and as they arise.

In traditional on-premise environments, endpoint, network, and infrastructure audit logs were the priority, and security teams had a wide range of resources to guide their security monitoring strategies.

But in a decentralized IT operating model – where SaaS is often accessed from non-corporate networks and devices – these traditional methods are insufficient for monitoring threats targeting SaaS at scale.

The challenge for securing SaaS is real, and the path isn’t paved with well-established standards or ready-made guides. Security and risk leaders are, however, becoming aware of the imperative to secure their SaaS, with SaaS cybersecurity now identified as a top 3 cybersecurity priority.

SaaS Audit Log Challenges

SaaS audit logs are essential for security and compliance, but they can also be challenging to manage. This is largely due to the variance in SaaS app APIs and the quality of audit logs.

• While APIs provide a means to access audit logs and other data from SaaS apps, they are not standardized across platforms. Each SaaS provider may offer a different set of APIs, each with its own authentication methods, data retrieval methods, and rate limits. This complicates the process of developing and maintaining integration scripts or connectors to collect audit logs from various SaaS sources.
• Each SaaS app often employs its own audit log format and data structure, which can vary significantly from one application to another. Additionally, SaaS audit logs vary wildly in visibility context, often excluding critical user operations or metadata about the activity.

In summary, the diverse range of SaaS apps, varying audit log formats, and the complexities of collecting logs through APIs emphasize the need for robust and adaptable strategies in SaaS audit log management. The diversity and variances make it challenging for security teams to identify and respond to security incidents consistently.

A Threat Detection Framework for Assessing SaaS Audit Logging

At AppOmni, we grapple with these challenges on a daily basis. But in response, our Threat Detection team developed a framework for providing a normalized means of organizing and cataloging audit logging capabilities from different SaaS platforms.

AppOmni is proud to announce the release of the SaaS Event Maturity Matrix (EMM).

The goal of the EMM is to provide a knowledge base devoted to SaaS audit logging, and serve as a resource for the security community. By utilizing this knowledge base, security professionals can quickly understand:

• The different audit log sources provided by a given SaaS application, including links to vendor documentation regarding log schemas and the APIs needed for collection.
• Audit log availability, including factors such as licensing requirements, latency, and the retention policies maintained by SaaS vendors.
• How a particular SaaS application’s audit logs align with fundamental event categories typically required by security operations teams, with real-world examples further illustrating this alignment.

How Does It Work?

The EMM was developed with security operations in mind. Whether your responsibilities involve threat detection, threat hunting, or incident response on SaaS, the Matrix revolves around a central theme: providing context tailored to security monitoring scenarios.

The EMM consists of the following concepts:

• Products: A distinct SaaS platform.
• Event Sources: The different audit log sources available for analysis or collection.
• Event Categories: Top-level categories of the EMM, used to organize Event Types.
• Event Types: Pre-defined, security-relevant, types of audited activity by a SaaS platform.
• Event Attributes: The individual fields or keys that may be available within different Event Types. A real-world example log is included as a reference.

Within the EMM, each Product is associated with at least one Event Source. An Event Source describes a Product’s associated audit logs. This includes information such as audit log schema, retention, latency, licensing, and other related external references.

Because each SaaS platform is unique, with its own use cases and logging language, we designed the Matrix to provide a baseline of what we consider to be the critical activities, Event Types, and fields, Event Attributes, that should be exposed regardless of the system.

• Event Types were chosen based on their value and relevance to security operations. They emphasize audit operations such as create, read, update, delete (CRUD) actions and add, change, remove (ACR) for authorization-related actions. Event Types are also aligned with MITRE ATT&CK to help illustrate how a SaaS platform’s audit logs can be utilized by security operations.
• Event Attributes represent the visibility and context within each logged audit event. Event Attributes highlight the depth and comprehensiveness of each audit event, aiding in its utilization for security operations.

The ultimate goal is to reveal a SaaS platform’s audit logging capabilities and assist security teams in enhancing detection and response on the SaaS platform. In fact, at AppOmni, we have also used EMM to drive audit logging improvements in our own product. After assessing AppOmni against the EMM, we quickly realized the need to expand on the scope of different user operations that are logged, as well as provide greater user and change context within audit events.

How does the EMM Benefit Cybersecurity Teams?

If you’re engaged in security operations on SaaS, you might have asked these questions:

• During a security incident, what can my incident response team expect when needing to access audit logs from a potentially compromised SaaS platform?
• What visibility can my threat detection team expect regarding activities performed on a SaaS platform, in order to create robust detections?
• When evaluating SaaS platforms, how does the audit visibility of a particular platform measure against our company’s requirements? Does this change the risk associated with using a specific vendor?
• Are there other audit log sources or intricacies that should be researched via SaaS vendor documentation to improve our visibility?

The EMM aims to provide an answer to these questions and more, with a net result seeking to improve the operational efficacy of SaaS focussed SecOps teams.

What to Expect Next

This initial release of the SaaS Event Maturity Matrix is just the beginning. The SaaS EMM currently supports Salesforce, Okta, Box, Github, Slack, and AppOmni. We have plans to add more SaaS platforms, starting with Microsoft 365 and Google Workspace as the next top priorities.

We also plan to continually reassess and improve the EMM, reevaluating every aspect to maximize value and relevance. For example, this may include fine-tuning Event Types and Event Attributes to ensure accuracy and better capture edge cases. It may also include expanding Event Categories, to better support the distinct classes of the myriad of SaaS platforms.

We’re also looking for ways to better capture the intricacies and subtle nuances associated with each audit log source and event, shedding light on those potential ‘gotchas’ that can impact SaaS security strategies including SecOps’ ability to effectively respond.

Lastly, we value community expertise and experiences, so we encourage you to join us in shaping the EMM’s future by submitting feedback directly to our Github repo.

We’re excited to shape the future of SaaS audit logging together.

---

Related Resources