CircleCI Security Incident - Jan 2023

Key points

• This incident is now closed, you can read CircleCI's own incident report here.
• We have reviewed audit logs and found no evidence that our systems, external data integrations or any customer data has been compromised.
• No action is required by Multitudes' customers.
• On January 5th 2023, CircleCI, Multitudes' CI/CD platform, notified customers of a security breach that occurred from December 19th 2022-January 5th 2023. An engineer's machine was compromised and an unauthorised actor gained access to production environments that contained customer secrets and encryption keys.
• Customers were advised on January 5th 2023 to rotate any and all secrets stored in CircleCI, which our engineering team actioned immediately.
• Access logs of all Multitudes systems were reviewed for any suspicious activity, including the provided attacker IP addresses and none was found.
• CircleCI permissions and keys stored in CircleCI were reviewed and reduced to ensure that, if this were to happen in the future, that our systems would be more protected.
• If you have any issues accessing the Multitudes app, please log out and back in and clear your cache.

‍

Overview

Multitudes' CI/CD platform provider, CircleCI, was compromised on December 19th 2022

On January 5th 2023, CircleCI notified customers of an ongoing security incident and advised, out of an abundance of caution, for customers to rotate any and all secrets stored in CircleCI. They are confident that "there are no unauthorized actors active in our systems” after January 5th 2023.

On 13th January 2023, CircleCI released their incident report and stated that an engineer's computer compromised by an unauthorized third party. This actor was able to gain access to production systems that included customer secrets and environment variables stored in CircleCI. Although these variables were encrypted at rest, the attacked was able to obtain encryption keys and exfiltrate customer keys.

As per CircleCI’s security policies, all secrets and environment variables are encrypted in transit and at rest. At the time of the initial alert, we were uncertain as to whether said variables were decrypted and exposed. Therefore we treated this as a high severity security incident and all secrets stored in CircleCI were rotated on January 5th 2023.

Multitudes users do not need to take any action.

We avoid, where possible, storing sensitive secrets in CircleCI. The only secrets stored were keys to access AWS and Auth0 for deployments. Additionally, we found a legacy API key to Stripe. These were all revoked one hour after the original notification from CircleCI.

We do not store any keys that provide access to customer data through our integrations (GitHub, Slack, Linear) in CircleCI. We have audited access logs across CircleCI, AWS, Auth0 and Stripe and have found no evidence of any suspicious or unauthorized activity. CircleCI provided source IP addresses of the threat actor and we found no trace of these in any of our systems.

Therefore, we have no evidence to suggest that any Multitudes customer data was accessed or any Multitudes system was compromised by unauthorized persons. This includes both access to our customer's data via integrations in addition to metadata and processed data stored on our own systems.

This incident has now been closed, you can read CircleCI's own incident report here.

If you have any issues accessing the Multitudes app, please log out and back in and clear your cache. Or contact us on support@multitudes.co.

‍

Timeline

Please note that all times are stated in New Zealand Daylight Time, the local timezone of our engineering team.

January 5 2023

3pm - CircleCI notified all customers via email that they were investigating a security incident that had occurred from 21 December 2022 until January 4 2023 (PST). Out of an abundance of caution, they asked customers to rotate any and all secrets stored in CircleCI.

4pm - The Multitudes engineering team responded to the incident and immediately rotated all secrets stored in CircleCI which included AWS and Auth0 deployment keys, and an old Stripe API key. Production containers were restarted with new keys.

5pm - AWS, Auth0 and Stripe access logs were reviewed and the AWS GuardDuty status checked for any potentially suspicious activity. None was found.

Deployment to all AWS environments was halted until further notice.

‍

January 6 2023

Reviewed logs across CircleCI, Auth0, Snowflake, GitHub and Stripe for any suspicious or unexpected activity. None was found.

SSH deployment keys between CircleCI and our own GitHub repos were rotated.

Auxiliary AWS accounts to the production environment were also reviewed.

‍

January 7 2023

CircleCI rotates customer’s GitHub OAuth keys.

‍

January 9 2023

Multitudes customers first notified of our response to the incident via email and Slack.

‍

January 10 2023

Policies and permissions of theCircleCI user in our AWS accounts revised and some permissions removed.

Deployments to Multitudes AWS environments resumed.

‍

January 12 2023

CircleCI announced they are working with AWS to notify customers via email if their AWS Access keys have been exposed. As yet, we have not received an email from AWS informing us of such.
‍

‍
January 13 2023

CircleCI released the official incident report stating the cause of the incident and the steps they are taking to improve security going forward. Additionally, a list of source IP addresses and VPN providers of the attacker were given to assist customers in their own incident response.

‍
January 16 2023

Multitudes audited systems logs again (from 16th December - January 5th) for the specified attacker IP addresses and no access was found.

The incident was then closed by us and marked as resolved. Customers were notified.

‍

‍