by benterix
16 subcomments
- > We had a budget alert (€80) and a cost anomaly alert, both of which triggered with a delay of a few hours
> By the time we reacted, costs were already around €28,000
> The final amount settled at €54,000+ due to delayed cost reporting
So much for the folks defending these three companies that refused to provide hard spending cap ("but you can set the budget", "you are doing it wrong if you worry about billing", "hard cap it's technically impossible" etc.)
by JohnScolaro
5 subcomments
- > We had a budget alert (€80) and a cost anomaly alert, both of which triggered with a delay of a few hours. By the time we reacted, costs were already around €28,000.
I had a similar experience with GCP where I set a budget of $100 and was only emailed 5 hours after exceeding the budget by which time I was well over it.
It's mind boggling that features like this aren't prioritized. Sure it would probably make Google less money short term, but surely that's more preferable to providing devs with such a poor experience that they'd never recommend your platform to anyone else again.
by embedding-shape
4 subcomments
- Considering the amount of repositories on public GitHub with hard-coded Gemini API tokens inside the shared source code (https://github.com/search?q=gemini+%22AIza%22&type=code), this hardly comes as a surprise. Google also has historically treated API keys as non-secrets, except with the introduction of the keys for LLM inference, then users are supposed to treat those secretly, but I'm not sure everyone got that memo yet.
Considering that the author didn't share what website this is about, I'd wager they either leaked it accidentally themselves via their frontend, or they've shared their source code with credentials together with it.
- As others have said, this is a "feature" for Google, not a bug. There is no easy way to set a hard cap on billing on a project. I spent the better time of an hour trying to find it in the billing settings in GCP, only to land on reddit and figuring out that you could set a budget alert to trigger a Pub/Sub message, which triggers a Cloud Function to disable billing for the project. Insanity.
- I read the following [0] and immediately went to my firebase project to downgrade my plan. This is horrific.
> Yes, I’m looking at a bill of $6,909 for calls to GenerativeLanguage.GenerateContent over about a month, none of which I made. I had quickly created an API key during a live Google training session. I never shared it with anyone and it’s not pushed to any public (or private) repo or website.
0 - https://discuss.ai.google.dev/t/unexpected-gemini-api-billin...
- It is scary building on the public cloud as a solo dev or small team. No real safety net, possibly unbounded costs, etc. A large portion of each personal project I do is spent thinking about how to prevent unexpected costs, detect and limit them, and react to them. I used to just chuck everything onto a droplet or VPS, but a lot of the projects I am doing lately need services from Google or AWS. I tend to prefer GCP at this point because at least I can programmatically disconnect the billing account when they get around to tripping the alert.
- These are all poorly designed systems from a CX perspective (the billing systems).
Billing is usually event driven. Each spending instance (e.g. API call) generates an event.
Events go to queues/logs, aggregation is delayed.
You get alerts when aggregation happens, which if the aggregation service has a hiccup, can be many hours later (the service SLA and the billing aggregator SLA are different).
Even if you have hard limits, the limits trigger on the last known good aggregate, so a spike can make you overshoot the limit.
All of these protect the company, but not the customer.
If they really cared about customer experience, once a hard limit hits, that limit sets how much the customer pays until it is reset, period, regardless of any lags in billing event processing.
That pushes the incentive to build a good billing system. Any delays in aggregation potentially cost the provider money, so they will make it good (it's in their own best interest).
by naturalauction
1 subcomments
- We had this exact same problem (the key initially wasn’t a secret but became a secret once we enabled Gemini API with no warnings).
We managed to catch it somewhat early through alerting, so the damage was only $26k.
We asked our Google cloud support rep for a refund - they initially came back with a no but now the case is under further consideration.
I’d escalate this up the chain as much as possible.
- Hey folks, I just wanted to drop a quick note here that there’s a way to stop billing in an emergency that’s officially documented on the Google Cloud documentation site: https://docs.cloud.google.com/billing/docs/how-to/disable-bi... . You can see the big red warning that this could destroy resources that you can’t get back even if you reconnect a billing account, but this is a way to stop things before they get out of control. This billing account disconnect goes all the way to implement a full on “emergency hand brake” that “unplugs the thing from the wall” (or whatever analogy you prefer) without you having to affirmatively do it yourself.
https://docs.cloud.google.com/billing/docs/how-to/modify-pro... and https://docs.cloud.google.com/billing/docs/how-to/budgets-pr... are other documented alternatives to receive billing alerts without the billing account disconnect.
The billing account disconnect obviously shouldn’t be used for any production apps or workloads you’re using to serve your own customers or users, since it could interrupt them without warning, but it’s a great option for internal workloads or test apps or proof of concept explorations.
Hope this helps!
- Two things that should be default on any GCP project touching generative-AI APIs:
1 API-key restrictions by HTTP referrer AND by API (`generativelanguage.googleapis.com` only),
2 a billing budget with a Pub/Sub "cap" action, not just an email alert. Neither is on by default, and almost nobody sets them before shipping. 13 hours is actually fast for detection. most teams find out at end-of-month reconciliation.
by alibarber
6 subcomments
- Forgive my ignorance - but what's the payoff for fraudsters in getting access to a generative AI service for a short-ish period of time, before they get cut off?
With EC2 / GCC credentials, I could understand going all out on bitcoin mining - but what are they asking the AI to do here that's worth setting up some kind of botnet or automation to sift the internet for compromised keys?
- Related: https://news.ycombinator.com/item?id=47156925
by turtlebits
0 subcomment
- Slightly off-topic, but Backblaze B2 has usage caps that actually work. I have $0 cap on API requests, and yesterday when litestream burned through the free tier (defaults to replicating every second), I got a notice and requests stopped working until I upped my cap.
- > Are there recommended safeguards beyond ... moving calls server-side?
This implies the API calls originated in the client, suggesting the client may have had they API key.
- I think the logistics of calculating cost in real time is something that is extremely hard. I don't think there is one big cloud service provider that has hard limits instead of alerts.
As long as they revert the charge when notified of scenarios like this , and they have historically done so for many cases, it's fine. It's an acceptable workaround for a hard problem and the cost of doing business ( just like Credit Cards accept a certain amount of loss to fraud as part of business)
by riteshkew1001
0 subcomment
- This story is almost quaint. The version we're about to see is a coding agent running in CI with an API key, hitting a transient 429, retrying in a tight loop because the prompt told it to "be persistent." Firebase had at least a human typing the query. Caps aren't a nice-to-have once the caller is autonomous.
- The spend-cap discussion is the right instinct but misses a more fundamental fix available to Firebase projects: restricting the API key itself. In Google Cloud Console → APIs & Services → Credentials, you can edit your Firebase browser key and set API restrictions to only allow specific Firebase services (Firestore, Authentication, Storage, etc.). This prevents the key from being usable with Gemini or any other GCP API entirely—so even if the key is exposed, it can't incur AI billing costs.
Most Firebase 'add AI to your app' tutorials skip this step because Firebase's initialization flow doesn't prompt you to configure it, and Firebase Security Rules only gate Firebase-specific services, not the key's broader GCP API access scope.
by linkregister
0 subcomment
- From the response from the Gemini product lead:
> We are moving to disable the usage of unrestricted API keys in the Gemini API, should have more updates there soon.
It's unacceptable the contract for client-side keys was broken in this manner, and doubly bad that it's taken so long for Google to remediate this issue. The Gemini team needs to publish a postmortem to explain what broke down in the engineering process to allow this to happen.
context: https://news.ycombinator.com/item?id=47156925
- It's incredible that in 2026 your best bet for getting support from Google is still posting to HN and hoping a Product Owner at Google takes pity on you (or feels shamed...)
- The scarier version of this problem is coming. Imagine AI agents making API calls across multiple vendors. Each vendor tracks usage in isolation. But no one knows what the agent is spending across all vendors. Nobody building the cross-vendor spending cap today.
- Don’t use GCP (and other big clouds) until they sort out their safeguards.
All three of the big cloud subreddits have stories like this on a regular basis
- For personal projects, is there a cloud service that has actual working spend caps? I would perhaps try using a cloud service if I wasn't exposing myself to a risk of losing my yearly income by a small mistake. Or is renting a VPS the only sensible option?
by luanmuniz
1 subcomments
- Unfortunately, yet just another story like this. One of these unexpected usage charges in the thousands appears every month, and with the same automatic denied too. This is one of the reasons I just stopped using these kinds of pay-per-usage cloud services long ago. At best, I still use services that have hard-bounded usage limits, like EC2 from AWS, where one instance can never go beyond 24h/day usage and is always capped, with shutdowns when exceeded, and limited credit cards, too.
It's super frustrating that this is the only option to realistically deal with this issue, since all stories end up the same way: The cloud company just saying "f* you, we don't care, pay up." and legal fees are always expensive :(
- Surprised they don’t have usage limits. E.g. you can’t get many IPs from AWS for your region until you request a limit increase. The UX for these kinds of things seems like it should default to low and allow easy increasing.
by william0353
0 subcomment
- Hi I am just curious the reason behind it as I have a firebase app with firebase ai logic service as well.
Is that the apiKey below which was used for web sdk init?
const firebaseConfig = {
apiKey: XXXX,
authDomain: XXX
Did Zanbezi enabled app check?
This is kind worrying...
};
by noplace1ikegone
0 subcomment
- Google responded to your post so that’s good news. We all know the nature of APIs, but a secure transaction system is non-negotiable from Google and its peers for LLM API use. Right now LLM APIs are like unencrypted credit card numbers floating around.
- That's fucking bonkers that nothing in the system could see this as unusual and worthy of throttling. The embarrassment of this -- that a company LITERALLY SELLING machine learning services and expertise -- cannot spot such a thing... This should have led them to deal with this internally and refund it. Just... Wow Google.
by thedangler
1 subcomments
- Also, can't you tie a key to a domain or IP address to help stop unauthorized usage?
- Does the blog post explain how this happened exactly? Did he leak his API key in frontend code somehow, or was his project itself vulnerable to misuse? I'm curious how someone racked up 30k in a few hours.
by ozlikethewizard
0 subcomment
- The top comment on the post physically hurt me. We've moved past the era of keep env files in code bases and are now actually serving them lol.
by comrade1234
7 subcomments
- Can you pre-load money into your account and have that be used until it's zero, at which time you have to load more? Deepseek does it this way.
- Oh the days when these scenarios just lead to a slowdown of the database server, felt by everyone, not just accounting.
by DaedalusII
0 subcomment
- on the one hand if you play with petrol you cant complain about burning down your garage
on the other hand hetzner sell ipv4 instance with no security on by default, just raw ubuntu 24.x
within 3-4 days of deploying one, it will be hacked and have crypto miners installed unless additional special config is added. i do wonder what % of hetzner vps instances are compromised
- Crude Oil Futures, Natural Gas Futures, Google Cloud API keys.
the widow-maker list increases.
by bossyTeacher
0 subcomment
- As always, you will need to make lots of noise on here and similar channels visited by influential people so stuff can get actioned.
Leading tech companies in 2026, folks.
by bcjdjsndon
0 subcomment
- I thought the pricing model was meant to be a benefit of the cloud? All of a sudden, shock horror, paying by the minute turns out to be no cheaper and maybe even more expensive than just doing it yourself
by chinathrow
0 subcomment
- Take them to court.
- on a more positive note, you saved a few bucks not running your own server or database.
- Anthropic and Claude are running circles around Google / Gemini for me these days. Anthropic was quite helpful for a while but strange limit issues started popping up. The final thread was a bug that essentially broke my ability to develop. I moved over to Claude Code full time and haven't looked back. Opus 4.6 is awesome for accelerating probabilistic programming!
- It's terrible that giant cloud providers such as Google or AWS doesn't allow for hard cap at project levels or prepaid. And that especially because alerts are delayed as author stated "We had a budget alert (€80) and a cost anomaly alert, both of which triggered with a delay of a few hours. By the time we reacted, costs were already around €28,000.".
by ButlerianJihad
0 subcomment
- Back in 2020–21, we were teaching our students how to stand up and configure cloud services, and I decided that a good extension of my homelab would be in AWS, so that I could learn basic cloud administration.
I was using the Free Tier for starters, of course. I managed to start up a working MediaWiki server on a Linux machine in EC2. I began to explore some of the more esoteric options and always checked out the extra-secure methods, and IAM and so forth.
My MediaWiki had a lot of spammers registering accounts. They weren't actually able to make edits, but I couldn't seem to stop them from creating user accounts. And I always felt rather... naked in terms of securing the Linux system itself. It seemed like the entire Internet had an Eye of Sauron focused on my open TCP ports and they were port-scanning it and running pentests 24/7. I honestly couldn't keep up!
Ultimately, I did realize that I could never constrain the budget to an affordable $20 or $30. Signs seemed to indicate that any malicious traffic could crank up my network egress costs alone! There were some rudimentary controls but they would never permit a full-scale shutdown of all services that could actually cost money.
So I shut down the cloud services and abandoned my Amazon AWS account. Migration to the cloud might seem like a good value proposition for any business that can't handle its own machine rooms or its own I.T. team to manage physical infrastructure. But it's an unconstrained cost nightmare waiting to happen for basically anyone of any scale. I would never recommend it, for personal or business use, until that aspect is somehow brought under control.
- I said this when this finding was originally posted and I'll say it again: This is by far the worst security incident Google has ever had, and that's why they aren't publicly or loudly responding to it. It's deeply embarrassing. They can't fix it without breaking customer workflows. They really, really want it to just go away and six months from now they'll complete their warning period to their enterprise contracts and then they can turn off this automated grant. Until then they want as few people to know about it as possible, and that means if you aren't on anyone's big & important customer list internally, and you missed the single 40px blurb they put on a buried developer documentation site, you're vulnerable and this will happen to you.
Disgusting behavior.
- good
- [dead]
- [dead]
by twoshotsai
0 subcomment
- [dead]
- [dead]
by ajaystream
7 subcomments
- [flagged]
by lukewarm707
2 subcomments
- i have seen this so many times...
i'm thinking it's time we replaced api keys.
some type of real time crypto payment maybe?
- With AI there is NO justification in NOT DOING IT BY YOURSELF. Why use firebase or <technology-x> if you can generate <the-thing> by yourself and deploy to hardware you own or rent.
Hacker News