Introduction
When hackers breached one of Braintrust's AWS accounts, they made off with AI provider secrets stored inside the company's systems. The incident forced an immediate API key rotation to lock out the attackers. This real‑world case highlights a critical truth: no organization is immune to cloud credential theft. In this guide, you'll learn exactly how to respond to a similar breach — from detection to full rotation — and how to harden your secrets management for the future.
What You Need
- Cloud console access (AWS, Azure, or GCP) with permissions to manage IAM users, roles, and service accounts.
- Inventory list of all API keys, access keys, and secrets currently in use by your AI providers and internal applications.
- Incident response plan (or a documented runbook for key rotation).
- Backup or fallback mechanism to avoid service downtime during rotation (e.g., side‑by‑side key deployment).
- Communication channels to alert your team, users, and external partners.
- Monitoring tool (CloudTrail, GuardDuty, or custom logging) to detect further unauthorized activity.
Step 1: Confirm and Contain the Breach
Before rotating any keys, you must verify that a breach actually occurred and limit its spread. In Braintrust's case, the attackers compromised one AWS account. Your first action is to identify exactly which account or environment was hit.
- Check your cloud provider's security notifications and access logs for unusual API calls or login attempts.
- Isolate the affected account by temporarily revoking permissions or applying a service control policy (SCP) that blocks all write actions.
- Document the incident: note the timestamp, IP addresses, and the resources that were accessed — including any AI provider secrets that may have been exfiltrated.
Step 2: Identify All Compromised Secrets
The breach at Braintrust involved secrets stored in the company's internal systems. You need a complete map of every API key and secret that might have been exposed.
- List all secrets that were stored in the breached AWS account, including environment variables, parameter store entries, and secrets manager items.
- Cross‑reference with your AI provider integrations — e.g., OpenAI, Anthropic, or custom model endpoints — whose keys may now be in the hands of attackers.
- Use a secrets scanning tool to quickly detect any keys that are still active and tied to the breached account.
Step 3: Generate New API Keys
Now it's time to create fresh credentials. Braintrust's response centered on rotating keys. Follow these best practices to ensure the new keys are secure.
- For each compromised key, generate a new one from the cloud console (e.g., AWS IAM > Users > Security credentials > Create access key).
- Use a naming convention that makes the key's purpose clear (e.g.,
production-ai-key-v2). - Assign the new key to the same IAM policy or service role as the old one, but do not delete the old key yet — you'll need to test the new one first.
Step 4: Deploy New Keys and Validate
Simply generating new keys doesn't stop the breach; you must swap them into your running applications without breaking functionality.
- Update your configuration files, CI/CD pipelines, and secret stores (like AWS Secrets Manager) to reference the new keys.
- Use a gradual deployment strategy: update one service at a time and monitor for errors.
- Run automated tests that call your AI provider endpoints using the new key to confirm it works.
- Set a short observation window — e.g., 15–30 minutes — to ensure no downtime or permission issues arise.
Step 5: Revoke Old Keys
Once every service has been successfully migrated to the new keys, you can disable the compromised ones. This is the step that actually locks out the attackers.
- In your cloud console, deactivate the old access keys (for AWS, set state to Inactive).
- After 24 hours of no reported issues, delete the old keys permanently.
- If the old keys were used in third‑party tools (e.g., a Slack bot or monitoring service), ensure those locations have been updated as well.
Step 6: Monitor and Audit
A rotation alone is not enough — you need to verify that the breach has been stopped and that no new malicious activity is occurring.
- Enable detailed logging for all API calls (e.g., AWS CloudTrail) and set up alerts for any use of the old (now revoked) keys.
- Check your AI provider dashboards for any unexpected requests that might indicate the attacker exfiltrated data before rotation.
- Conduct a post‑incident review to understand how the AWS account was compromised in the first place — was it a leaked IAM key? A misconfigured S3 bucket? Patch the root cause.
Tips for Long‑Term Security
- Use a secrets manager. Never hardcode API keys in code or configuration files. Services like AWS Secrets Manager or HashiCorp Vault automate rotation and access control.
- Implement automatic rotation policies. Many providers allow you to set a rotation schedule (e.g., every 90 days) to reduce the window of exposure if a key is leaked.
- Limit key permissions. Apply the principle of least privilege: each key should only have the permissions it needs, and for AI services, restrict to specific endpoints.
- Train your team. Regularly run tabletop exercises that simulate a data breach so everyone knows the rotation process without panic.
- Review third‑party integrations. The Braintrust incident involved AI provider secrets; ensure that any external services you connect to have proper security measures and that you don't share more than necessary.
By following these steps, you can respond to a cloud data breach the way Braintrust did — with swift, organized API key rotation that minimizes damage. Prepare now, and your organization will be ready when the unexpected happens.