Cloud Cost Anomaly Detection: Lessons From Real Incidents

Cloud Security Incident Patterns: GCP vs. AWS

Google Cloud incidents tracked almost entirely to one thing: the explosion of Gemini API and Vertex AI usage over the past year, and how easy it turned out to be to leave an API key unrestricted while building fast. We posted about the fix for Gemini API key abuse on the blog here. AWS incidents skewed toward a more traditional pattern: leaked IAM keys turning into compute abuse, plus a handful of straight-up account takeovers. Across both platforms, the root cause was almost never a platform vulnerability. It was almost always something a human did, or didn't do, with a credential.

GCP Security Incidents: Leaked API Keys Are the No. 1 Cause

Root cause	Share
Leaked or unrestricted API keys (Gemini, Vertex, AI Studio)	77%
Leaked service account key used for crypto mining or unauthorized Compute Engine fleets	19%
Admin or session hijack leading to billing manipulation or admin lockout	2%
Data exfiltration via a compromised service account	2%

The top row is doing almost all the work. 77% of GCP incidents we saw in the past six months came down to an API key, usually a Firebase browser key or a Maps/Gemini key embedded in client-side code, that had no HTTP referrer restriction, no IP restriction, and no API scoping. Attackers scan public repos and exposed frontend bundles for exactly this pattern, and once they find a working key, they point automated scripts at the Generative Language API and let it run. We've seen these incidents range from a few hundred dollars to over $200,000 in unauthorized Gemini charges, generated in a matter of hours.

The second cluster, leaked service account keys, plays out differently but starts the same way: a long-lived key sitting in a GitHub repo, a CI/CD config, or a developer's laptop. Once an attacker has it, they don't run up API bills, they spin up compute. In one case we worked, a leaked GitLab CI service account key let an attacker launch more than 10,500 crypto-mining VMs in a single GCP project overnight. And refund requests from the cloud provider aren't always approved. That's the part worth sitting with: cloud providers are getting stricter about "runaway spend" credits, especially on repeat or preventable incidents. Containment and prevention matter more than they used to, because the safety net of a billing dispute is not guaranteed.

AWS Security Incidents: Leaked IAM Keys and Account Takeover

Root cause	Share
Leaked or unrestricted IAM access key (dev machines, CI/CD, code) leading to unauthorized EC2, ECS, Fargate, or Bedrock usage	56%
Root account or admin session takeover leading to billing manipulation or admin lockout	15%
Compromised workload or application (exposed service, misconfigured trust policy) leading to lateral movement or secrets exfiltration	15%
Platform abuse via compromised customer accounts, phishing or spam sent through Amazon SES	7%
DDoS or network-layer attack with no credential compromise	7%

The AWS story looks similar to what we see with GCP: leaked credentials account for over half of everything. IAM access keys leaked from developer machines or CI pipelines turn into unauthorized EC2 or Fargate fleets running crypto-miners, or, increasingly, unauthorized Bedrock API calls racking up model inference charges.

The root account takeover cases are the ones that should worry security leaders most, because they're the hardest to walk back. In one incident, an attacker reset the password on a customer's AWS root account, enrolled their own MFA device, and locked the real administrators out entirely. From there, they ran large-scale Bedrock API calls that generated more than $500,000 in charges before the account was recovered. Recovery required working directly with AWS to re-verify ownership and rebuild access from scratch. No amount of anomaly detection alerts you fast enough if the people who'd normally respond to them can't get into the account.

We also saw a smaller but real cluster of workload-level compromises: an internet-facing load balancer exposing a pod, a misconfigured OIDC trust policy, and an attacker moving laterally to exfiltrate thousands of secrets from Secrets Manager. These are less about a single leaked key and more about the accumulated debt of a permissive network and IAM configuration.

Identity Compromise: Phishing, OAuth Abuse, and Account Takeover

The remaining incidents followed a different, but related, pattern: phishing campaigns that harvested credentials, then got used to hijack accounts and send spam through a compromised sender identity, or pivot into connected third-party tools via OAuth. One customer's compromised Workspace account was used to add unauthorized manager links to their Google Ads account and rack up close to $50,000 in fraudulent ad spend. Another had an employee's login used to grant OAuth access to third-party tools, resulting in unauthorized purchases.

The common thread across every platform: the cloud provider's infrastructure wasn't the weak point. Identity was.

The bill is the smoke, not the fire

It's tempting to treat a cost spike as the problem. However, the cost spike is often the first visible symptom of a credential compromise. Fix the bill and you've addressed the symptom. Fix the credential hygiene and get serious about anomaly detection, and you address the root cause.

That's why our recommendation has two halves that need to work together: a posture review that closes the gaps before they're exploited, and monitoring that catches the exploitation fast if a gap slips through anyway.

How to Prevent Cloud Security Incidents: Closing the Gap

DoiT's two-part recommendation includes the following:

Step 1: Review Your Cloud Security Posture

Most of the incidents above were preventable with practices that show up in a standard cloud security posture review: restricting API keys to specific referrers or IP ranges, rotating and retiring long-lived service account keys, moving from static credentials to workload identity federation or short-lived tokens wherever possible, enforcing MFA and session length limits at the identity provider, and locking down who can create or use owner-level IAM bindings. None of this is exotic. It's the boring, unglamorous work of access hygiene, and it's exactly the work that keeps showing up as missing in incident after incident.

Step 2: Set Up Cloud Cost Anomaly Detection and Notifications

DoiT Cloud Intelligence™ Real-Time Anomaly Detection (available in our Enhanced and Enterprise tiers) is built for the exact gap that shows up across nearly every incident in this dataset: the delay between something going wrong and someone finding out. Most billing-based tools catch a spend spike hours or days after it happened, because they're reading exports that update on a lag. Real-Time Anomaly Detection reads runtime usage data directly, blended with historical spend patterns, and can flag unusual activity within minutes, not days. Every alert comes with severity scoring and an AI-generated breakdown of the affected service, SKU, and resource, so your team isn't starting the investigation from zero.

For customers in our Essentials tier, our out-of-the-box cloud Anomaly Detection can and should still be configured, with appropriate notifications set to identify and flag anomalies. To upgrade to Real-Time Anomaly Detection, reach out to your DoiT Account Manager or submit a ticket via Expert Inquiry in the DoiT console.

To get the most out of your anomaly detection:

1. Confirm anomaly detection notifications are configured for the right people via the right channels at the right time. Cost anomaly notifications require the Cloud Analytics permission, and by default they go to Admins, Power Users, Finance Users, and Standard Users, but it's worth checking who on your team actually has this configured, and that they understand the importance of responding quickly to notifications.
2. Don't stop at the default severity threshold. If your organization runs high-velocity workloads (CI/CD, GenAI inference, autoscaling fleets), tune your review cadence so a mid-severity anomaly at 2 a.m. still gets looked at before morning.
3. Leverage your DoiT Customer Success Manager to help configure (or validate) your detection and notifications. DoiT's CSMs are happy to help and can be the expert eyes needed to ensure you are set up correctly.
4. Set up notifications for spend spikes and new SKU usage. The new SKU anomaly detection and notification is underutilized yet key: it tells you the moment your organization starts using a service or SKU it's never used before, which is exactly what happens when an attacker with a stolen key starts hitting the Generative Language API or spinning up Bedrock inference for the first time. A legitimate new SKU shows up occasionally when your team ships something new. An unexplained one showing up at an unexpected time is worth a look immediately.
5. Consider configuring notifications on quota exhaustion or credit usage. While built for a different failure mode, this rounds out the picture of unexpected behaviors in your spend.
6. Evaluate using automations. Configure DoiT Cloud Intelligence™ CloudFlow to run on cost anomaly triggers. You can automate incident mitigation tasks like posting to Slack, creating tickets, or running remediation steps. Not sure where to begin? Your DoiT Customer Success Manager can coordinate a working session with a DoiT Forward Deployed Engineer to get your CloudFlows created.

For more information on configuring anomaly detection and notifications, see the DoiT Help Documentation for anomaly detection and notifications, or reach out to your DoiT Customer Success Manager.