Unpacking (and Preventing) the CircleCI Data Breach

How One Compromised Laptop Led to Exposed Encryption Keys and SaaS Data

By Drew Gatchell, Director of Threat Detection Engineering, AppOmni

On Jan. 13, 2023, CircleCI released its incident report for a security breach disclosed in early January. In this article, we’ll delve into the nature of the attack, how this breach occurred, and steps you can take to protect your organization and SaaS data from similar threats.

To begin, CircleCI, which advertises its CI/CD tools as being used by one million engineers worldwide, first acknowledged the breach in a security alert posted on Jan. 4, 2023. At that time, the company advised all of its customers to immediately:

• Rotate all “secrets” (passwords or private keys).
• Replace project API tokens, which the company has invalidated.
• Audit internal logs for unauthorized access between Dec. 21, 2022 and Jan. 4, 2023.

The next day, CircleCI stated its confidence that the risk leading to the breach was eliminated and that customers could resume their builds. Over the next week, the company provided updates on the progress of rotating GitHub, Bitbucket, and AWS tokens, along with confirmation that this breach was unrelated to a recent reliability update.

But the nature of the breach was not revealed until the incident report was published on Jan. 13.

How Was CircleCI Data Accessed and Exfiltrated?

According to the incident report, the breach stemmed from a CircleCI engineer’s laptop becoming infected with malware on Dec. 16, 2022. While most breaches involving successful theft of MFA-backed tokens relate to social engineering — such as the phishing scam employed in the 0ktapus breach — the specifics of how malware was installed on this employee’s laptop have not been revealed.

This threat actor successfully stole a valid, multi-factor authenticated (MFA) SSO session. CircleCI’s antivirus software did not detect the malware. And due to the engineer’s ability to generate product access tokens, the threat actor was able to access and exfiltrate data that included customer environment variables, tokens, and keys. Data exfiltration occurred on Dec. 22.

CircleCI became aware of these events by:

• A customer reporting suspicious GitHub OAuth activity on Dec. 29. How this customer discovered the anomaly hasn’t been revealed, although it might be related to finding “repo.download_zip” events associated with the CircleCI token in their audit logs.
• A day later, CircleCI discovered the same customer’s GitHub OAuth token was compromised.
• On Dec. 31, CircleCI began rotating their GitHub OAuth tokens.
• Roughly a week later on Jan. 4, CircleCI publicly announced the breach and advised customers to take immediate action as described in the introduction.

How Can My Team Prevent a Similar Breach?

As the exact nature of the malware infection is unknown, we cannot offer definitive guidance for preventing that form of attack. But most cases along these lines are due to employees becoming victims of social engineering, particularly phishing attempts.

To protect against the increasingly sophisticated phishing attacks, we recommended continuous training that ensures your employees:

• Check and double check the URL of any site requesting login credentials. SaaS app admins and similar users with highly privileged access should receive in-depth training and support.
• Never click on any URL from a questionable source. If an employee suspects the link is legitimate, they should ask your Security team for guidance.
• Change their password immediately if they suspect their user credentials have been compromised.

Regularly auditing logs for anomalies and unusual activity should be a part of your regular security practices, not an ad-hoc approach to incidences. And to limit the impact of a breach and your attack surface, we recommend several of the actions CircleCI has taken, including:

• Limiting access to production environments to key employees, and ensuring those employees must pass additional security measures, such as step-up authentication. This additional step requires a user to authenticate once more before accessing your production environments or other sensitive areas.
• Implementing monitoring and alerts for suspicious behaviors and events, especially those across third-party apps.

Consider leveraging Canarytokens as well. They enable you to, essentially, embed traps in your production systems. If a threat actor metaphorically falls into one of those traps, you’ll receive an alert. Canarytokens are free, relatively easy to implement, and versatile.

How Can AppOmni Help My Team Detect and Minimize This Type of Attack?

Even the best defenses against malware and phishing attempts have their limits. Organizations must prepare for threat actors to compromise an employee’s credentials as CircleCI experienced.

To limit a compromised account’s reach, SaaS Security Posture Management (SSPM) solutions like AppOmni can help organizations take a proactive approach. Configuration management capabilities show whether users have been granted highly privileged roles or other excessive permissions across all SaaS systems. Our solution detects drift if security groups or roles have been modified. Least privilege access drastically reduces the scope of malware or an attacker’s capabilities.

You can improve your threat-hunting processes by automating and normalizing event log data in AppOmni. This provides a regular pattern for checking logs and SaaS security settings, along with alerting your team of abnormal, concerning entries. You can route normalized logs into your existing workflows in SIEM, SOAR, and security data lakes.

Organizations should also employ threat detection capabilities to continuously monitor activity and provide alerts. AppOmni’s threat detection rules can flag suspicious events for immediate investigation. Examples of rules include:

• Data exfiltration detection: When data is downloaded from your SaaS systems quickly or over a given time period, it can indicate data exfiltration. Rules for these activities help you see what data (and how much of it) has been downloaded to determine what, if any, risk exists.
• Resource shared externally: Any resource that’s been shared outside your environment can be flagged so you understand who is on the receiving end.
• Custom rules: Default rules may not be enough for some organizations. Creating custom rules allows you to monitor your unique business processes, for example. Or you can monitor for specific activities, such as a user or group frequently accessing or downloading privileged data.

Based on the details provided in its incident report, CircleCI appears to have taken smart steps to prevent and limit a similar attack. We strongly recommend other organizations follow their lead.

---

Related Resources