- The interesting and yet-to-be-explained part is why google flagged the account?
Put all the timestamps you want in the post mortem about what you observed, but you haven't addressed the root cause.
The "this doesn't make sense" part of the story likely has a real explanation that nobody wants to reveal yet.
- Railway has not had the best month in the tech press have they? And in both cases it was an automated process belonging to some other party that put them there, damaging their reputation.
I was going to talk to our google rep about their killing the Gemini cli but this is way more concerning.
by majdalsado
3 subcomments
- Unfortunately we had to make emergency migration off to Azure yesterday due to this. Thankfully our DB was not hosted on Railway and we were back up in a couple hours.
As much as we loved the simplicity they provided us, there's just been too many mishaps and shortcomings for us to continue running a B2B enterprise app on their infrastructure.
Sad day :(
by ryanSrich
17 subcomments
- Question: for a smaller SaaS tool, or even internal product. If a team doesn't want to manage AWS or another IaaS provider, what are the best alternatives for the following
1.) Vercel - having a bad month
2.) Supabase - having a bad month
3.) Railway - now having a bad month
- > May 19, 22:10 UTC - Our automated monitoring detected API health check failures and paged our on-calls, who started investigating the issue.
> At 22:20 UTC on May 19, Google Cloud placed Railway’s production account into a suspended status incorrectly, as part of an automated action.
If the timestamps are accurate, what was causing the errors 10 minutes before the account was suspended?
The simplest explanation is just that one or the other of these timestamps is wrong, which wouldn't be a big deal. But if the timestamps aren't known with certainty, it seems very odd to include them in the writeup as though they are certain, even though they are very obviously inconsistent with each other.
by shwetanshu21
13 subcomments
- This should be a warning to anyone running GCP. They suspend accounts left right and centre without even thinking about what they're doing. It seems like they use Gemini 3.1 Pro to run their production decisions.
TK has a history of absolutely destroying the culture of the place like in OCI and has done something similar in GCP from what I've heard. GCP and Google are completely different entities with how they work. Don't expect Google quality from the name. It's just like those old brands which now have cheap licensed products like Nokia (An exaggeration I know but not far from truth).
Not only that they are known to shut off their services randomly giving you like 6 months to migrate. They have lots of engineers not doing anything, so they put them on migrating internal users off those services, most of their clients don't. There was a brilliant article on this by an ex-GCP employee that I can't find right now.
Avoid GCP like plague if you are serious about your business.
Edit: Gemini (unironically) found the article on this, a very good read: https://steve-yegge.medium.com/dear-google-cloud-your-deprec...
by AlfieJones
0 subcomment
- This isn’t the first time Google Cloud has seriously messed with a customer’s account: https://cloud.google.com/blog/products/infrastructure/detail...
by shwetanshu21
1 subcomments
- "Railway owns our vendor choices, and we ultimately own this one. Your customers don't care whether the failure was Google or Railway; they see your product. Your uptime is our responsibility, and we'll keep delivering on it."
Kudos to them for acknowledging it and not doing PR speak. It shows it was an architectural failure from their part of trusting GCP, and they are working to fix it. Should they have seen it coming? Yes. But better late than never.
- "Finally, we are in planning to remove Google Cloud services from our data plane’s hot path, and keeping them only for secondary/failover."
That's pretty clear. Google can no longer be trusted as a B2B service provider.
by dantillberg
2 subcomments
- What drives Google to apply these actions so completely and immediately, versus a more deliberate approach, with notification and delay before action, manual review for paying customers, or a warning to resolve within X hours/days? Once or twice could be errors or bad implementation, but these can't explain away the pattern.
It would seem that Google's counsel has deemed that whenever _____ is detected, the company must immediately and completely sever the business relationship. What is that driving concern? Is it sanctions enforcement? CSAM? Something else?
- > As a side effect, Terms-of-service acceptance records were also reset, prompting users to re-accept on their next visit to the dashboard.
Don't get me wrong- the rest of this mess falls pretty clearly on Google Cloud, but this one feels like something Railway did to themselves.
- I've read all the threads and their main page and I still don't really understand what this service is. Is this like a commercial alternative to Gerrit? What do people use this for?
I'm not a developer, just curious what this is.
- "Your customers don't care whether the failure was Google or Railway; they see your product. Your uptime is our responsibility, and we'll keep delivering on it." - Thanks Claude!
- It's highly unlikely that GCP banning their account without telling them is true, but GCP is probably not going to go public with the real reason.
by beauregardener
0 subcomment
- Sadly, My Railway project is still having issues 24 hours later. Already started emergency migration away from Railway backend :(
- Duplicate of:
https://news.ycombinator.com/item?id=48201484
- All my services have lost their databases. All their AI agent is offering me is to delete all the data (thousands of data) and create a new database. It insists there's no other way. But I have clients, I have hundreds of courses on the platform!
- The RCA and preventive measures was a pleasant read. I got a lot of respect for companies putting a lot of effort into incident reports like these. Makes them appear very professional rather than just blaming the cloud provider outright.
- Had similar experience with GCP. Terminated VMs six times, and responded zero times.
by mellosouls
0 subcomment
- Even if it ultimately turns out to be "Google's fault" (as this report seems to be saying), Railway say they own the incident but make no apology here.
- Unfortunately we've also had a litany of problems with our GCP deployment and chose to remove them completely as a service provider.
- It's reassuring to know they will ban a million dollar enterprise customer just like they will ban your GMail of 20 years.
by in_a_society
1 subcomments
- Google has a culture problem. This is not something that can change easily nor will it change when it’s not recognized as being an issue within their organization.
Between my peer c-suites, the conversation is that GCP cannot even be in the consideration set until such a time as a several-year period has elapsed without this kind of incident.
by theredleft
1 subcomments
- back to on-prem
- How many trains were delayed or incorrectly routed as a result?
- Google, the new Microsoft!
- I will definitely not be signing up on GCP because of this.
by FajitaNachos
0 subcomment
- 19 minutes from detection to getting the google account restored is pretty awesome honestly.
by 1970-01-01
0 subcomment
- They forgot to get reimbursement for downtime. A free month of GCP is better than nothing.
- Flagged by some AI automation.
- Now given the logic that you can't be dependent on any one service to run your SaaS, how does Railway convince its customers to run their SaaS on a single service?
by siliconc0w
1 subcomments
- Why would you use an infrastructure provider on top of another infrastructure provider? It adds cost and risk, it's always going to be a leaky abstraction, and it's not hard to learn how to use GCP or AWS correctly - especially with agents.
- > Your customers don't care whether the failure was Google or Railway; they see your product.
Refreshing. So tired of businesses blaming their vendors. Oh it wasn't us spamming you text messages and emails, it was Shopify. Oh, our delivery guarantee said 2 days and it's been a week? That's not us, it's UPS.
I don't care. I didn't pay UPS or Shopify. I paid you.
- I don’t understand why Google still has TK helming GCP when its obviously not achieved the kind of success it should. Google infra is some of the best in the world yet GCP is meh. It continues to underperform and seems content to be a distant 3rd behind AWS and Azure.
- I've been getting serious, recently, about moving all my workloads to equipment that I control in datacenters with which I have professional relationships. It's less expensive, easier, and this kind of nonsense doesn't happen. These cloud providers need to step back and observe how terrible they've made these products. Footguns everywhere, pricing that is impossible to forecast or reason about, broken APIs, and automated self destruction. Then you have third-party providers sitting on top of them, adding another layer of each antifeature. Crazy.
by AtNightWeCode
0 subcomment
- So, what was the reason for the account suspension. Why did it happen? I know Google can be a bit stupid with their automatons but I am bit skeptical here. There are sites more critical than Railway hosted on GCP.
by phendrenad2
0 subcomment
- Nov 28, 2012, user fbuilesv posted: Google Compute Engine is on Limited Preview right now. If you're planning to offer a service that you care about you should consider this.
Do we know if GCP has ever left limited preview..??!
by kittikitti
0 subcomment
- It's not my proudest moment, but at least being banned and suspended all the time brings some wisdom.
by charcircuit
2 subcomments
- >Google Cloud placed Railway’s production account into a suspended status incorrectly, as part of an automated action.
There is no justification given on why this action was incorrect. It's possible they actually did something wrong.
- Major infra provider -> has no backups/game plan if GCP goes down
- > Railway’s production account into a suspended status incorrectly, as part of an automated action.
Be it individuals or companies, this time is the best time to ditch all dependence on anything clouds or SaaS since all are using automated AI, more and more of these incidents will occur.
by ChrisArchitect
0 subcomment
- Related discussion during the incident:
https://news.ycombinator.com/item?id=48201484
- Does this qualify for a list entry on killedbygoogle.com ?
- Perfect reminder that it's time to use Google Takeout while I still can.
- tldr: AI suspended an almost a billion dollar startup account.
Hacker News